Nexus 2248PQ FEX Connection Problem with a Nexus 6000 40G QSFP Port

Introduction

This document describes a problem with the Cisco Nexus 6000 connection to the Cisco Nexus 2248PQ Fabric Extender (FEX) and offers a solution.

Prerequisites

Requirements

Cisco recommends that you have basic knowledge of these topics:

• Cisco Nexus 2000 Series FEX
• Cisco Nexus 6000 Series FEX Configuration

Components Used

The information in this document is based on these software and hardware versions:

• Cisco Nexus 2248PQ-10GE
• Cisco Nexus 6004

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Background Information

The Cisco Nexus 2248PQ FEX has sixteen 10-Gigabit Ethernet fabric interfaces via four Quad (4-channel) Small Form-factor Pluggable (QSFP) ports and forty-eight 10-Gigabit Ethernet host interfaces. All host interfaces use all of the available fabric interfaces. (Static pinning is not supported. Port-channel mode is only supported on fabric interfaces.) When all host interfaces send traffic to all fabric interfaces, the maximum oversubscription ratio for the Cisco Nexus 2248PQ is 3:1.

Problem

When you bring up the link on the Cisco Nexus 6000 FEX fabric ports that connects to the Cisco Nexus 2248PQ-10GE, the link on the Cisco Nexus 6000 shows as Link not connected, and the link lights on the port of the Cisco Nexus 2248PQ are green or off:

Nexus6004-A# show int eth 2/7
Ethernet2/7 is down (Link not connected)
 Dedicated Interface
  Belongs to Po125
  Hardware: 40000 Ethernet, address: 5897.1ef0.4856 (bia 5897.1ef0.4856)
  MTU 1500 bytes, BW 40000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is fex-fabric
  auto-duplex, 40 Gb/s, media type is 40g

Solution

In order to bring this link up, the Cisco Nexus 2248PQ needs to see the QSFP cable as four separate 10G links (recall the previous description of the uplinks as sixteen 10G links via four QSFP ports). The same cable is used, but you need to enter this command on the Cisco Nexus 6000 ports that you plan to break into four 10G connections:

(config)# interface breakout slot 2 port 7-9 map 10g-4x

Note: You have to divide the ports into groups of three, and you must divide the entire group, not just one member. So your options are 1-3, 4-6, 7-9, 10-12, or any combination that includes all members of the previous groups. For example, 1-6, 7-12, 4-9.

Once you enter this command, you need to reload the module for the changes to take effect:

(config)# poweroff module 2
(config)# no poweroff module 2

It takes a few minutes for the module to come online. When it does, you should see your link divide into four 10G subports:

Nexus6004-A# show int status | i 2/7
Eth2/7/1      --                 connected 1         full    10G     --
Eth2/7/2      --                 connected 1         full    10G     --
Eth2/7/3      --                 connected 1         full    10G     --
Eth2/7/4      --                 connected 1         full    10G     --

At this point, you need to bundle the ports into a port-channel, and the normal FEX configuration can be applied:

(config)# fex 270
(config)# int ethernet 2/7/1-4
(config-if-range)# switchport mode fex-fabric
(config-if-range)# channel-group 270
(config)# interface port-channel 270
(config-if)# switchport mode fex-fabric
(config-if)# fex associate 270
(config-if)# shut
(config-if)# no shut

Related Information

• Breakout Command in the Configuration Guide
• Nexus 6000 FEX Configuration Guide
• Nexus 2000 Data Sheet