QSFP+ Ports Operations and Switch Fabric Mode

This chapter describes the application-specific integrated circuit (ASIC) mapping and configuration of quad small form-factor pluggable plus (QSFP+) ports on the Cisco Nexus 2248PQ Fabric Extender (FEX), the Cisco Nexus 5500 with QSFP+ Generic Expansion Module (N55-M4Q) switch, the Cisco Nexus 6004 switch and the Cisco Nexus 6004 switch fabric mode.

This chapter contains the following sections:

Quad Small Form-Factor Pluggable Plus

The quad small form-factor pluggable plus (QSFP+) is a transceiver module that offers high-density 40 Gigabit Ethernet connectivity options for datacenter and high-performance computing networks. The QSFP+ transceiver module is a hot-swappable, parallel fiber-optical/copper module with four independent optical transmit and receive channels. These channels can terminate in another 40 Gigabit Ethernet QSFP+ transceiver, or the channels can be broken out to four separate 10 Gigabit Ethernet SFP+ transceivers. The QSFP+ transceiver module connects the electrical circuitry of the system with either a copper or an optical external network. For more information about the Cisco 40 Gigabit Ethernet modules, see:

http://www.cisco.com/en/US/partner/products/ps11708/tsd_products_support_model_home.html.

Cisco Nexus 2248PQ Fabric Extender

The Cisco Nexus 2248PQ is a 1 rack unit (1 RU) Fabric Extender(FEX) with 48 SFP+ host ports and 4 QSFP+ uplink ports. The host ports can be operated in either 1 Gigabit Ethernet or 10 Gigabit Ethernet mode. The 4 QSFP+ uplink ports can operate only in 4 x10 Gigabit Ethernet mode. The uplink ports do not support the native 40 Gigabit Ethernet mode.

You can connect the Cisco Nexus 2248PQ FEX to its parent switch in the following two ways:

  • QSFP+ to QSFP+
  • QSFP+ to SFP+

Figure 1-1 shows the cabling options for a Cisco Nexus 2248PQ FEX to Cisco Nexus 5500 and Cisco Nexus 6004 Series switches.

Figure 1-1 Connection Cable Options for a Cisco Nexus 2248PQ FEX to Cisco Nexus 5500 and Cisco Nexus 6004 Series Switches


Note For the Cisco Nexus 6004 switch, the default mode of the QSFP+ ports is native 40 Gigabit Ethernet. You must configure QSFP+ ports on the Cisco Nexus 6004 that are connected to the Cisco Nexus 2248PQ in 4x10 Gigabit Ethernet mode. See the Configuring QSFP+ Ports section for more details about the configuration.


Cisco Nexus 5500 Series with QSFP+ GEM

The QSFP+ GEM (N55-M4Q) is a generic expansion module (GEM) that provides 16x10 Gigabit Ethernet SFP+ in a 4 x QSFP+ form factor. This module does not support native 40 Gigabit Ethernet on the QSFP+ port. This module is supported on all Cisco Nexus 5500 chassis (Cisco Nexus 5548P, Cisco Nexus 5548UP, Cisco Nexus 5596UP, and Cisco Nexus 5596T). This module is suitable for any expansion slot on the Cisco Nexus 5500 Series and you can populate all the expansion slots on the Cisco Nexus 5596 chassis with N55-M4Qs. There is no additional configuration needed to specify the operation mode of the QSFP+ ports because all the ports only operate in 4 x10 Gigabit Ethernet mode. The numbering of the ports uses the following convention(see Figure 1-2):

interface ethernet [chassis_ID/] slot/ QSFP_port [/.subintf_ port-no]


Note The argument, chassis_ID, is not optional when addressing the host interfaces of a Cisco Nexus 2000 Series Fabric Extender.


Figure 1-2 QSFP+ Slot Number on N55-M4Q Module

 

 

This example shows how to display the information about the module and interface on the Cisco Nexus 5596 switch:

Nexus5596# show module 3
Mod Ports Module-Type Model Status
--- ----- --------------------------------- ---------------------- ------------
3 16 N5K QSFP+ Expansion Module N55-M4Q ok
 
Mod Sw Hw World-Wide-Name(s) (WWN)
--- -------------- ------ --------------------------------------------------
3 6.0(2)N1(1) 2.0 --
 
Mod MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
3 547f.eed8.b4a0 to 547f.eed8.b4af FOC16417WYT
Nexus5596# show running-config | section Ethernet3
interface Ethernet3/1/1
interface Ethernet3/1/2
interface Ethernet3/1/3
interface Ethernet3/1/4
interface Ethernet3/2/1
interface Ethernet3/2/2
interface Ethernet3/2/3
interface Ethernet3/2/4
interface Ethernet3/3/1
interface Ethernet3/3/2
interface Ethernet3/3/3
interface Ethernet3/3/4
interface Ethernet3/4/1
interface Ethernet3/4/2
interface Ethernet3/4/3
interface Ethernet3/4/4
Nexus5596# show interface e3/1/1
Ethernet3/1/1 is up
Dedicated Interface
Hardware: 1000/10000 Ethernet, address: 547f.eed8.b4a0 (bia 547f.eed8.b4a0)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is access
full-duplex, 10 Gb/s, media type is 40G
Beacon is turned off
Input flow-control is off, output flow-control is off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 00:00:35
Last clearing of "show interface" counters never
2 interface resets
30 seconds input rate 400 bits/sec, 0 packets/sec
30 seconds output rate 1104 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 80 bps, 0 pps; output rate 432 bps, 0 pps
RX
0 unicast packets 68719489350 multicast packets 0 broadcast packets
12614 input packets 1056826 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 660284 multicast packets 0 broadcast packets
24065 output packets 1919643 bytes
0 jumbo packets
0 output errors 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
 
Nexus5596#

Cisco Nexus 6004 Switch and QSFP+ Ports

The Cisco Nexus 6000 Series platform uses Cisco's ASIC technology. It provides line rate throughput and low latency performance. The ASICs on the Cisco Nexus 6000 Series platform are unified port controllers (UPCs) and switch fabric (SF). The UPCs provide data-plane processing and the SF cross-connects the UPCs. Each UPC can be operated as either 3 ports of 40G Ethernet or 12 ports of 10G Ethernet.

The Cisco Nexus 6004 switch is a 4 RU unit chassis with 8 vertical slots. The first 4 slots are fixed with 12 QSFP+ ports per slot. The remaining 4 slots are for linecard expansion modules(LEM). A QSFP+ LEM is a 12 QSFP+ port, field-replaceable module that has the same functionality as the fixed slots. These 4 slots provide up to 48 additional 40 Gigabit Ethernet ports. Figure 1-3 shows the Cisco Nexus 6004 switch port numbering and port mapping with UPCs in a slot.

This section includes the following topics:

Figure 1-3 Cisco Nexus 6004 Switch Port Numbering and Port Mapping with UPCs in a Slot

 

Configuring QSFP+ Ports

QSFP+ ports can operate as either 40 Gigabit Ethernet or 4x10 Gigabit Ethernet modes on the Cisco Nexus 6004 switch. The default mode is 40 Gigabit Ethernet for the fixed QSFP+ ports and QSFP+ ports on a LEM. The port numbering of the QSFP+ port operating in 40 Gigabit Ethernet mode contains two levels. The first level is the slot number. There are eight slots in the Cisco Nexus 6004 switch. The farthest left slot is slot 1 and the farthest right is slot 8. The second level is the QSFP+ port number. There are 16 ports in the built-in slot and 16 ports in the QSFP+ LEM. The QSFP+ port numbering system is from top to bottom and left to right as indicated in Figure 1-3. The QSFP+ port reference on the Cisco Nexus 6004 switch is as follows:

interface ethernet [chassis_ID/] slot/ QSFP_port [/.subintf_ port-no]

To change from the default mode to 4x10 Gigabit Ethernet mode, you have to change it on a per port group basis. A port group is the ports that are managed by the same UPC. The table in Figure 1-3 shows the four available port groups within a given slot. To change from the default mode of 40 Gigabit Ethernet to 4x10 Gigabit Ethernet mode for any port group, follow these steps:

SUMMARY STEPS

1. Configure terminal

2. interface breakout slot slot-number port port-range map 10g-4x

3. power off module module

4. no power off module module

5. (Optional) show interface brief

DETAILED STEPS

Command
Purpose

Step 1

configure terminal

 
Example:
N6004-TME3# config t N6004-TME3(config)#

Enters global configuration mode.

Step 2

interface breakout slot slot-number port port-range map 10g-4x

Example:
N6004-TME3(config)# interface breakout slot 2 port 1-3 map 10g-4x

Enables you to configure the linecard expansion module in 10g mode.

Step 3

power off module module

Example:
N6004-TME3(config)# poweroff module 2

Enables you to power off the module.

Step 4

no power off module module

Example:
N6004-TME3(config)# no poweroff module 2

Enables you to return power to the module.

Step 5

(Optional) Show interface brief

Example:
N6004-TME3# show interface brief | incl Eth2
 

(Optional) Displays brief information about the interface.

Examples

This example shows how to change QSFP+ ports 1 to 3 and 7 to 12 to 4 x10 Gigabit Ethernet mode on slot 2:

N6004-TME3# show interface brief | incl Eth2
Eth2/1 1 eth access down SFP not inserted 40G(D) --
Eth2/2 1 eth access down SFP not inserted 40G(D) --
Eth2/3 1 eth access down SFP not inserted 40G(D) --
Eth2/4 1 eth access down SFP not inserted 40G(D) --
Eth2/5 1 eth access down SFP not inserted 40G(D) --
Eth2/6 1 eth access down SFP not inserted 40G(D) --
Eth2/7 1 eth access down SFP not inserted 40G(D) --
Eth2/8 1 eth access down SFP not inserted 40G(D) --
Eth2/9 1 eth access down SFP not inserted 40G(D) --
Eth2/10 1 eth access down SFP not inserted 40G(D) --
Eth2/11 1 eth access down SFP not inserted 40G(D) --
Eth2/12 1 eth access down SFP not inserted 40G(D) --
N6004-TME3# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
N6004-TME3(config)# interface breakout slot 2 port 1-3 map 10g-4x
N6004-TME3(config)# interface breakout slot 2 port 7-12 map 10g-4x
N6004-TME3(config)#
 
 
 
 

This example shows you how to power off a module:

N6004-TME3(config)# poweroff module 2
N6004-TME3(config)# 2013 Jan 2 23:30:40 N6004-TME3 %$ VDC-1 %$ %PFMA-2-MOD_REMOVE: Module 2 removed (Serial number FOC16422P28)
N6004-TME3(config)#
 
This example shows you how to power on a module:
 
N6004-TME3(config)# no poweroff module 2
N6004-TME3(config)# show module 2
Mod Ports Module-Type Model Status
--- ----- --------------------------------- ---------------------- ------------
2 39 Norcal Ethernet Module N6K-FIXED-LEM ok
 
Mod Sw Hw World-Wide-Name(s) (WWN)
--- -------------- ------ --------------------------------------------------
2 6.0(2)N1(1) 1.0 --
 
Mod MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
2 a44c.11e8.3a10 to a44c.11e8.3a1f FOC16422P28
N6004-TME3(config)# show interface brief | incl Eth2
Eth2/1/1 1 eth access down SFP not inserted 10G(D) --
Eth2/1/2 1 eth access down SFP not inserted 10G(D) --
Eth2/1/3 1 eth access down SFP not inserted 10G(D) --
Eth2/1/4 1 eth access down SFP not inserted 10G(D) --
Eth2/2/1 1 eth access down SFP not inserted 10G(D) --
Eth2/2/2 1 eth access down SFP not inserted 10G(D) --
Eth2/2/3 1 eth access down SFP not inserted 10G(D) --
Eth2/2/4 1 eth access down SFP not inserted 10G(D) --
Eth2/3/1 1 eth access down SFP not inserted 10G(D) --
Eth2/3/2 1 eth access down SFP not inserted 10G(D) --
Eth2/3/3 1 eth access down SFP not inserted 10G(D) --
Eth2/3/4 1 eth access down SFP not inserted 10G(D) --
Eth2/4 1 eth access down SFP not inserted 40G(D) --
Eth2/5 1 eth access down SFP not inserted 40G(D) --
Eth2/6 1 eth access down SFP not inserted 40G(D) --
Eth2/7/1 1 eth access down SFP not inserted 10G(D) --
Eth2/7/2 1 eth access down SFP not inserted 10G(D) --
Eth2/7/3 1 eth access down SFP not inserted 10G(D) --
Eth2/7/4 1 eth access down SFP not inserted 10G(D) --
<SNIP>
Eth2/12/1 1 eth access down SFP not inserted 10G(D) --
Eth2/12/2 1 eth access down SFP not inserted 10G(D) --
Eth2/12/3 1 eth access down SFP not inserted 10G(D) --
Eth2/12/4 1 eth access down SFP not inserted 10G(D) --
N6004-TME3(config)#
 

After the module is back inline, the port naming changes to three levels. The change takes effect only after you have entered the power off module and power on module commands. The port reference to the 10 Gigabit Ethernet ports changes to the following:

interface ethernet [chassis_ID/] slot/ QSFP_port/ breakout_port [/.subintf port-no]

To change the ports configuration back to 40 Gigabit Ethernet mode, use the no form of the interface breakout slot command as follows:

no interface breakout slot slot-number port port-range map 10g-4x

After the module is back inline, the port naming returns to two levels. The change takes effect only after you have entered the power off module and power on module commands. The port reference to the 40 Gigabit Ethernet ports changes back to the following:

interface ethernet [chassis_ID/] slot/ QSFP_port/ [/.subintf_ port-no]

Cisco Nexus 6004 Switch Fabric Mode

Switch fabric mode refers to the single link speed configuration on the Cisco Nexus 6004 switch. The following are the characteristics of switch fabric mode:

  • There are two user configurable switch fabric modes: 40 Gigabit Ethernet and 10 Gigabit Ethernet. The default switch fabric mode is 40 Gigabit Ethernet.
  • The switch fabric mode is a system-wide configuration.
  • Changing the switch fabric mode requires a chassis reload.
  • The total bandwidth between UPCs and switch fabric remain the same regardless of the switch fabric mode configuration.
  • The switch fabric mode configuration is independent of the QSFP+ port speed configuration (40 Gigabit Ethernet or 4 x10 Gigabit Ethernet).

We recommend that you use the default switch fabric mode of 40 Gigabit Ethernet. You can change to the 10 Gigabit Ethernet mode if most of the front panel ports are operating in 10 Gigabit Ethernet mode and the application desires a lowest latency between 10 Gigabit Ethernet ports. For this software release, when the fabric mode is set to 10 Gigabit Ethernet, an In Service Software Upgrade (ISSU) is disabled. For future software releases, when the fabric mode is set to 10 Gigabit Ethernet, an ISSU will be enabled.

To change the switch fabric mode, use the fabric-mode command. You must perform a system reload for the change to take place.

SUMMARY STEPS

1. Configure terminal

2. fabric-mode {10g | 40g}

3. (Optional) copy runing-config startup-config

DETAILED STEPS

Command
Purpose

Step 1

configure terminal

 
Example:
N6004-TME3# config t N6004-TME3(config)#

Enters global configuration mode.

Step 2

fabric-mode {10g | 40g}

Example:
N6004-TME3(config)# fabric-mode 10g

Enables you to set the fabric mode.

Step 3

(Optional) copy runing-config startup-config
 
Example:
N6004-TME3(config)# copy running-config startup-config

(Optional) Copies the running configuration to the startup configuration

EXAMPLES

This example shows how to select a fabric mode of 10 Gigabit Ethernet:

switch# configure terminal
N6004-TME3(config)# fabric-mode 10g
Fabric mode set to 10G. Please copy the configuration and reload the switch
N6004-TME3(config)# copy running-config startup-config
 

This example shows how to select a fabric mode of 40 Gigabit Ethernet:

switch# configure terminal
N6004-TME3(config)# fabric-mode 40g
Fabric mode set to 10G. Please copy the configuration and reload the switch
N6004-TME3(config)# copy running-config startup-config