Configure Unified and Breakout Ports on the 3rd Generation UCS Fabric Interconnect

Introduction

This document describes how the 3rd generation Cisco Unified Computing System Fabric Interconnect (UCS-FI-6332-16UP) contains 16x10GE unified ports,18x40GE breakout capable ports, and 6x40GE ports. Certain configuration changes require a reboot of the fabric interconnect. Multiple reboots consume valuable time during initial system setup. It explains a process to complete all configuration changes with a single reboot of each fabric interconnect. Specifically this document covers configuring System QoS, fibre channel interfaces, and 4x10GE breakout interfaces. You may as well wish to add additional steps of setting Ethernet and Fibre Channel (FC) switching modes.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Cisco Unified Computing System (UCS) Manager 
• Cisco Unified Computing System (UCS) Manager Command Line Interface (CLI)

Components Used

The information in this document is based on these software and hardware versions:

• UCS Manager version 3.1(1) or later
• UCS-FI-6332-16UP
• UCS-FI-6332  (Does not support Unified Ports)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Configure

Network Diagram

These are the configurations of the document:

• Set the best-effort class for MTU 9216.
• Configure ports 1/1-1/6 as Fibre Channel.
• Configure ports 1/31-1/34 as 4x10GE Breakout Ports.

System QoS Policy

Configure the best-effort system QoS policy for jumbo framing. Unlike previous fabric interconnect generations the 6300 series requires a reload when modifying global QoS policies.

UCS-6332-A# scope eth-server
UCS-6332-A /eth-server # scope qos
UCS-6332-A /eth-server/qos # scope eth-best-effort 
UCS-6332-A /eth-server/qos/eth-best-effort # set mtu 9216
UCS-6332-A /eth-server/qos/eth-best-effort *# top

Unified Ports for Fibre Channel

Unified Port are numbered 1/1-16 on the 6332-16UP. Fibre Channel port modes must be configured such that the last FC port occupies port id 6,12, or 16.This results in a minimum of six FC ports and a maximum of sixteen.

Valid port combinations are:

• Ethernet: 1/1-1/16
• FC: 1/1-1/6 Ethernet: 1/7-1/16
• FC: 1/1-1/12 Ethernet: 1/13-1/16
• FC: 1/1-16 

UCS-6332-A *# scope fc-uplink 
UCS-6332-A /fc-uplink *# scope fabric a
UCS-6332-A /fc-uplink/fabric* # create interface 1 1
UCS-6332-A /fc-uplink/fabric/interface* # up
UCS-6332-A /fc-uplink/fabric* # create interface 1 2
UCS-6332-A /fc-uplink/fabric/interface* # up
UCS-6332-A /fc-uplink/fabric* # create interface 1 3
UCS-6332-A /fc-uplink/fabric/interface* # up
UCS-6332-A /fc-uplink/fabric* # create interface 1 4
UCS-6332-A /fc-uplink/fabric/interface* # up
UCS-6332-A /fc-uplink/fabric* # create interface 1 5
UCS-6332-A /fc-uplink/fabric/interface* # up
UCS-6332-A /fc-uplink/fabric* # create interface 1 6
UCS-6332-A /fc-uplink/fabric/interface* # top

Breakout Ports 4x10GE Connectivity

Breakout ports are used to connect 40GE interfaces to 10GE capable equipment. These can be used as Uplink ports connecting to a 10G switch or as Server ports connecting to a 22XX IOM module or as FCoE ports.

• Ethernet 1/17-1/34 are capable of breakout configuration.
• Ethernet 1/35-1/40 are fixed at 40GE.
• An individual 1x10GE interface in a breakout port is capable of FCoE.

Note: If jumbo QoS is configured globally then only 4 physical ports can be configured for breakout.

UCS-6332-A *# scope cabling
UCS-6332-A /cabling *# scope fabric a 
UCS-6332-A /cabling/fabric *# create breakout 1 31
Warning: Port breakout create action reboots FI and any existing configurations on 40G port will be erased.!
UCS-6332-A /cabling/fabric/breakout* # up
UCS-6332-A /cabling/fabric # create breakout 1 32
UCS-6332-A /cabling/fabric/breakout* # up
UCS-6332-A /cabling/fabric # create breakout 1 33
UCS-6332-A /cabling/fabric/breakout* # up
UCS-6332-A /cabling/fabric # create breakout 1 34
UCS-6332-A /fc-uplink/fabric/interface* # top

Save the Configuration

Warning: The Fabric Interconnect will immediately reload when the changes are committed.

UCS-6332-A* # commit-buffer

*The switch will now reboot.

Repeat the changes on Fabric Interconnect B.

Verify

Confirm ports FC 1/1-6 are configured for Fibre Channel and ports Ethernet 1/31-34 are in Breakout mode. Notice the breakout ports now have 4 subinterfaces.  In this example, ports 1/33/1 and 1/33/2 are configured for FCoE and 1/33/3-4 as Uplink interfaces.

UCS-6332-A# scope fabric-interconnect a
UCS-6332-A /fabric-interconnect # show port

Ether Port:

Slot  Aggr Port  Port  Oper State       Mac                  Role    Xcvr
----- ---------- ----- ---------------- -------------------- ------- ----
[...]
    1          0    30 Sfp Not Present  8C:60:4F:BC:C4:D0    Unknown N/A
    1          0    35 Sfp Not Present  8C:60:4F:BC:C4:E4    Unknown N/A
    1          0    36 Sfp Not Present  8C:60:4F:BC:C4:E5    Unknown N/A
    1          0    37 Sfp Not Present  8C:60:4F:BC:C4:E6    Unknown N/A
    1          0    38 Sfp Not Present  8C:60:4F:BC:C4:E7    Unknown N/A
    1          0    39 Sfp Not Present  8C:60:4F:BC:C4:E8    Unknown N/A
    1          0    40 Sfp Not Present  8C:60:4F:BC:C4:E9    Unknown N/A
    1         31     1 Sfp Not Present  8C:60:4F:BC:C4:D4    Unknown N/A
    1         31     2 Sfp Not Present  8C:60:4F:BC:C4:D5    Unknown N/A
    1         31     3 Sfp Not Present  8C:60:4F:BC:C4:D6    Unknown N/A
    1         31     4 Sfp Not Present  8C:60:4F:BC:C4:D7    Unknown N/A
    1         32     1 Sfp Not Present  8C:60:4F:BC:C4:D8    Unknown N/A
    1         32     2 Sfp Not Present  8C:60:4F:BC:C4:D9    Unknown N/A
    1         32     3 Sfp Not Present  8C:60:4F:BC:C4:DA    Unknown N/A
    1         32     4 Sfp Not Present  8C:60:4F:BC:C4:DB    Unknown N/A
    1         33     1 Up               8C:60:4F:BC:C4:DC    Fcoe Uplink
                                                                     QSFP 40G SR4
    1         33     2 Up               8C:60:4F:BC:C4:DD    Fcoe Uplink
                                                                     QSFP 40G SR4
    1         33     3 Up               8C:60:4F:BC:C4:DE    Network N/A
    1         33     4 Up               8C:60:4F:BC:C4:DF    Network N/A
    1         34     1 Sfp Not Present  8C:60:4F:BC:C4:E0    Unknown N/A
    1         34     2 Sfp Not Present  8C:60:4F:BC:C4:E1    Unknown N/A
    1         34     3 Sfp Not Present  8C:60:4F:BC:C4:E2    Unknown N/A
    1         34     4 Sfp Not Present  8C:60:4F:BC:C4:E3    Unknown N/A

FC Port:

Slot  Port  Oper State       Wwn
----- ----- ---------------- ---
    1     1 Up               20:01:8C:60:4F:BC:C4:80
    1     2 Up               20:02:8C:60:4F:BC:C4:80
    1     3 Sfp Not Present  20:03:8C:60:4F:BC:C4:80
    1     4 Sfp Not Present  20:04:8C:60:4F:BC:C4:80
    1     5 Sfp Not Present  20:05:8C:60:4F:BC:C4:80
    1     6 Sfp Not Present  20:06:8C:60:4F:BC:C4:80

In NXOS, breakout ethernet ports are named  as br-ethernet x/y/z.

UCS-6332-A# # connect nxos a
UCS-6332-A(nxos)# show int br-ethernet 1/33/1
Br-Ethernet1/33/1 is up
 Dedicated Interface 
  Hardware: 10000 Ethernet, address: 8c60.4fbc.c4dc (bia 8c60.4fbc.c4dc)
  Description: C: FcoeUplink
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
[...]

Troubleshoot

There is currently no specific troubleshooting information available for this configuration.