Document ID: 116348
Updated: Jul 23, 2013
Contributed by Austin Peacock and Carlo Schmidt, Cisco TAC Engineers.
This document describes a problem with the Cisco Nexus 6000 connection to the Cisco Nexus 2248PQ Fabric Extender (FEX) and offers a solution.
Cisco recommends that you have basic knowledge of these topics:
- Cisco Nexus 2000 Series FEX
- Cisco Nexus 6000 Series FEX Configuration
The information in this document is based on these software and hardware versions:
- Cisco Nexus 2248PQ-10GE
- Cisco Nexus 6004
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
The Cisco Nexus 2248PQ FEX has sixteen 10-Gigabit Ethernet fabric interfaces via four Quad (4-channel) Small Form-factor Pluggable (QSFP) ports and forty-eight 10-Gigabit Ethernet host interfaces. All host interfaces use all of the available fabric interfaces. (Static pinning is not supported. Port-channel mode is only supported on fabric interfaces.) When all host interfaces send traffic to all fabric interfaces, the maximum oversubscription ratio for the Cisco Nexus 2248PQ is 3:1.
When you bring up the link on the Cisco Nexus 6000 FEX fabric ports that connects to the Cisco Nexus 2248PQ-10GE, the link on the Cisco Nexus 6000 shows as Link not connected, and the link lights on the port of the Cisco Nexus 2248PQ are green or off:
Nexus6004-A# show int eth 2/7
Ethernet2/7 is down (Link not connected)
Belongs to Po125
Hardware: 40000 Ethernet, address: 5897.1ef0.4856 (bia 5897.1ef0.4856)
MTU 1500 bytes, BW 40000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Port mode is fex-fabric
auto-duplex, 40 Gb/s, media type is 40g
In order to bring this link up, the Cisco Nexus 2248PQ needs to see the QSFP cable as four separate 10G links (recall the previous description of the uplinks as sixteen 10G links via four QSFP ports). The same cable is used, but you need to enter this command on the Cisco Nexus 6000 ports that you plan to break into four 10G connections:
(config)# interface breakout slot 2 port 7-9 map 10g-4x
Once you enter this command, you need to reload the module for the changes to take effect:
(config)# poweroff module 2
(config)# no poweroff module 2
It takes a few minutes for the module to come online. When it does, you should see your link divide into four 10G subports:
Nexus6004-A# show int status | i 2/7
Eth2/7/1 -- connected 1 full 10G --
Eth2/7/2 -- connected 1 full 10G --
Eth2/7/3 -- connected 1 full 10G --
Eth2/7/4 -- connected 1 full 10G --
At this point, you need to bundle the ports into a port-channel, and the normal FEX configuration can be applied:
(config)# fex 270
(config)# int ethernet 2/7/1-4
(config-if-range)# switchport mode fex-fabric
(config-if-range)# channel-group 270
(config)# interface port-channel 270
(config-if)# switchport mode fex-fabric
(config-if)# fex associate 270
(config-if)# no shut
The Cisco Support Community is a forum for you to ask and answer questions, share suggestions, and collaborate with your peers.
Refer to Cisco Technical Tips Conventions for information on conventions used in this document.