Migrate to 40 Gbps with Cisco UCS Fabric Interconnects

The third-generation Cisco^® Unified Fabric for the Cisco Unified Computing System^™ (Cisco UCS^®) delivers higher speeds for both Ethernet and Fibre Channel connectivity. It provides high performance, consistent latency, and a lossless fabric to address the needs of our customers deploying high-capacity data centers.

The Cisco UCS fabric interconnect is one of the main components of Cisco UCS. It provides the communication backbone for Cisco UCS B-Series Blade Servers and C-Series Rack Servers. All the servers that are attached to the interconnect are part of a single, highly available management domain. By supporting unified fabric, it provides uniform access to both network and storage connectivity for all the servers within its domain. This architecture provides simlified management and greater orchestration for lower total cost of ownership (TCO). The following are the main features that help reduce TCO:

●   Centralized management: Cisco UCS Manager provides open and extensible embedded management of all software and hardware components of Cisco UCS across multiple chassis, rack servers, composable infrastructure, and virtulal machines through policy- and model-based management.

◦     Because the solution has only a few management touch points, deployment and provisioning processes are faster.

◦     The solution offers stateless computing that allows the use of any resource at any time.

●   Converged infrastructure: The lossless fabric can support both Ethernet and storage traffic through flexible port configuration.

◦     The use of less cabling allows more efficient use of power and cooling resources.

◦     The solution supports both Fibre Channel and Ethernet uplink connectivity.

●   High performance and scale: Cisco Fabric Extender Technology (FEX Technology) enables organizations to scale up to 160 servers in a single unified system, and Cisco Data Center Virtual Machine FEX (VM-FEX) helps provide a consistent operational model and visiblity between the physical and virtual environments.

 

The third-generation unified fabric interconnects—Cisco UCS 6332 and 6332-16UP Fabric Interconnects—make the transition to 40 Gigabit Ethernet and 16-Gbps Fibre Channel smooth and cost effective. The new components provide investment protection; they are backward compatible with second-generation components, so they can be swapped into exisiting systems or provisioned with newer infrastructure instances.

●   Cisco UCS 6300 Series Fabric Interconnects provide 40 Gigabit Ethernet connectivity upstream to access-layer switches and downstream to Cisco UCS servers.

◦     The Cisco 6332 is a 32-port 40-Gbps fabric interconnect in a 1-rack-unit (1RU) that supports both Ethernet and Fibre Channel over Ethernet (FCoE). The 40-Gbps port can also be converted to 10 Gigabit Ethernet ports through either a breakout cable that converts a Quad Small Form-Factor Pluggable (QSFP) port to four SFP ports (called a QSFP-to-4xSFP breakout cable) or a QSFP-to-SFP adapter (QSA), which converts a 40-Gbps port into a single 10-Gbps port.

◦     The Cisco 6332-16UP is 24-port 40-Gbps fabric interconnect with 16 fixed unified ports (UPs) supporting 1 and 10 Gigabit Ethernet or 4-, 8-, or 16-Gbps Fibre Channel connections in a 1RU form factor. The 40-Gbps port can also be converted to 10-Gbps ports through either a QSFP to 4xSFP breakout cable or a QSA, which converts a 40-Gbps port into a single 10-Gbps port.

●   The Cisco UCS 2304 Fabric Extender is a third-generation I/O module (IOM) that brings 40-Gbps fabric to the blade server chassis. It provides four 40-Gbps connections to the fabric interconnet and eight 40-Gbps connections to the server through either a native 40-Gbps link or a port channel consisting of four 10-Gbps links. The Cisco UCS 2304 can be inserted into an existing blade chassis.

●   The Cisco Nexus^® 2348UPQ 40GE Fabric Extender connects up to 40 Gigabit Ethernet links to the fabric interconnects and supports 1 and 10 Gigabit Ethernet connections downstream to the rack servers.

Audience

The target audience for this document consists of system architects, system engineers, and any other technical personnel who are responsible for planning or upgrading fabric interconnects. Although every effort has been made to make this document appeal to the widest possible audience, it assumes that the audience has an understanding of Cisco UCS hardware, terminology, and configuration.

Objective

This document describes the process of upgrading from Cisco UCS 6200 Series Fabric Interconnects to fabric interconnects based on the Cisco UCS 6332. After reading this document, the reader should have a complete understanding of the upgrade process and any factors that need to be taken into consideration. Table 1 summarizes the various upgrade paths between fabric interconnects, fabric extenders and IOMs, and Cisco Nexus 2000 Series Fabric Extenders. The Cisco UCS C-Series servers are connected in a single-wire managemet configuration. Direct-connect disruption varies depending on the number of adapters that are connected to the fabric interconnect. A single-adapter configuration is less disruptive than a dual-adapter configuration. A single-adapter configuration does not require a server reacknowledgment, whereas a dual-adapter configuration does. A server reacknowledgment is also required for a direct server connection when the new port connection is different from the original.

For a Cisco UCS C-Series server located behind a Cisco Nexus 2000 Series Fabric Extender, in most cases a FEX acknowledgment is all that is required to reestablish the link. In some cases, however, a server reacknowledgment is required.

Note:    Two server acknowledgments will be required: one for each fabric interconnect exchanged. To reduce the number of server acknowledgments, the acknowledgment process can be performed after tha last fabric interconnect has been upgraded to the Cisco UCS 6332 platform.

For Cisco UCS B-Series servers, a server acknowledgment is not required for any extraneous connection cases for either a fabric interconnect upgrade or a fabric interconnect upgrade plus IOM upgrade from the Cisco UCS 2200 Series to the Cisco UCS 2304.

Table 1.       Migration Support Matrix for Infrastructure Hardware

 

The minimum Cisco UCS software release required for Cisco UCS 6332 platform fabric interconnects and Cisco UCS 2304 IOMs is Release 3.1(1). Therefore, you must update the Cisco UCS 6200 Series fabric interconnects and the IOMs, fabric extenders, and servers to Release 3.1(1) before you can start the upgrade process (Table 2).

Table 2.       Software Migration from Cisco UCS 6200 Series Fabric Interconnect to Cisco UCS 6332 Platform Support Matrix for Host Firmware

 

Preplanning

Before beginning the migration process, be sure that your system is ready.

Inventory Check

Because the Cisco UCS 6332 platform fabric interconnects are first supported in Cisco UCS Release 3.1(1), the existing Cisco UCS domain needs to be upgraded to Release 3.1(1) before you can start the upgrade process. However, before you upgrade the existing Cisco UCS domain to Release 3.1(1), you should inventory the system. Cisco UCS Release 3.1(1) deprecates older hardware: specifically, generation-one hardware. Tables 3 and 4 list the hardware supported in Release 3.1(1). If the hardware is not listed, then it will be not recognized when you upgrade to Release 3.1(1), thus preventing successful migration to the Cisco UCS 6332 platform.

Table 3.       Server Support for Cisco UCS Release 3.1(1)

Table 4.       I/O and Fabric Extender Support for Cisco UCS Release 3.1(1)

 

Port and Cabling Planning

Port schemes for the two different generations are vastly different. Thus, port mapping is an important step in the preplanning process to help ensure a smooth transition from a fabric based on the Cisco UCS 6200 Series to a fabric based on the Cisco UCS 6332 platform. The first step in port planning is to map the Fibre Channel ports, first asking:

●   Will Fibre Channel be used, or are you moving to FCoE?

●   How many Fibre Channel ports are needed?

 

Figures 1, 2, and 3 show the port numbering schemes.

Note:    Fibre Channel ports are available only on the Cisco UCS 6332-16UP. On the 16UP, the ports can be converted from left to right from Ethernet to Fibre Channel. The first block is ports 1 to 6, the second block is ports 1 to 12, and the third block is ports 1 to 16.

After the Fibre Channel ports are set, you need to map the Ethernet connections. If you need 10 Gigabit Ethernet connections, whether for appliance ports, direct connections, or Cisco UCS 2200XP IOMs, then you need to consider additional factors. All the 40-Gbps ports can be converted to 10-Gbps ports through either a QSFP-to-4xSFP 40-Gbps breakout cable or a QSA module that converts the 40-Gbps port to a single 10-Gbps port. Table 5 provides a support matrix for conversion from 40-Gbps to 10-Gbps ports.

Table 5.       Support Matrix for Conversion from 40-Gbps to 10-Gbps Ports

 

Note:    The maximum number of 40-Gbps ports that break out to four 10-Gbps ports with a second no-drop class and jumbo maximum transmission unit (MTU) configuration is four. Otherwise, there is no limit on the number of ports that can be broken out to four 10-Gbps ports.

VSAN Considerations

Depending on the type of Cisco UCS 6332 platform fabric interconnect to which you are upgrading, the number of VSANs may need to be taken into consideration. The Cisco UCS 6332-16UP model supports only 15 VSANs, and if the current environment has more than 15, then VSAN reduction is required. The Cisco UCS 6332 model supports 32 VSANs, and thus no action is required.

Migration Procedure

After taking the appropriate steps to move workloads and decommission any old hardware, use the steps presented here to perform the upgrade process.

Note:    Although the upgrade can be performed while the system is active, Cisco recommends that you perform the upgrade during a maintenance window.

Step 1.    If you are upgrading to the Cisco UCS 6332-16UP, verify that the environment does not exceed 15 VSANs.

Step 2.    Download Cisco UCS Release 3.1(1) from the Cisco Support site at https://software.cisco.com/download/navigator.html and upgrade the existing Cisco UCS 6200 Series domain to Release 3.1(1).

Step 3.    Fail over traffic from the subordinate server to the primary server. You can do this in several ways, as listed in Table 6.

Table 6.       Traffic Failover Method

 

Note:    For Cisco UCS domains that use blades only, Cisco recommends the fabric evacuation method. This approach provides the fastest way to validate proper traffic failover. It also lets you back out the failover simply by turning the fabric evacuation off.

a.     If Cisco UCS C-Series servers are directly attached to the fabric interconnect, then manually disable either the port or the uplink for Ethernet, Fibre Channel, and FCoE.

Step 4.    Verify that traffic is flowing over the primary fabric interconnect.

a.     If the Cisco Nexus 2232PP 10GE is being retired and replaced by Cisco Nexus 2348UPQ 10GE, then decommission and remove old fabric extender from Cisco UCS Manager.

Step 5.    As mentioned in the “Port and Cabling Planning” section, port schemes are vastly different, so you should unconfigure all the ports on the subordinate interconnect and reconfigure the ports on the new fabric interconnect after it joins the cluster.

Note:    You do not need to delete the port-channel groups because they can be reused, but you should verify that the individual ports are removed from LAN and SAN port channels, VLAN groups, etc.

Step 6.    Remove the cable from the downed subordinate fabric interconnect.

a.     If you are replacing the IOM and Cisco Nexus 2232 with the Cisco UCS 2304 and Cisco Nexus 2348-UPQ, then this is the time to make the replacement.

Step 7.    Replace with the new Cisco UCS 6332 platform fabric interconnect and connect the L1 to L2 connections between the Cisco UCS 6200 Series primary interconnect to the Cisco UCS 6332 platform subordinate interconnect.

Step 8.    Reconnect the components (IOM, Cisco Nexus 2200 and 2300 Series, and Cisco UCS C-Series servers) to the Cisco UCS 6332 platform subordinate interconnect according to the port planning table.

Step 9.    Power up the Cisco UCS 6332 platform subordinate interconnect. If it is properly cabled with the correct software version, then the subordinate interconnect will recognize that it is connecting to an existing cluster (Figure 5).

 

Step 10.     Enter show cluster extended-state to verify the cluster state.

B26-A# show cluster extended-state

Cluster Id: 0xac919450846211e5-0x833f002a6a923061

 

Start time: Wed Jan 27 22:43:01 2016

Last election time: Thu Jan 28 03:40:02 2016

 

A: UP, PRIMARY

B: UP, SUBORDINATE

 

A: memb state UP, lead state PRIMARY, mgmt services state: UP

B: memb state UP, lead state SUBORDINATE, mgmt services state: UP

   heartbeat state PRIMARY_OK

 

INTERNAL NETWORK INTERFACES:

eth1, UP

eth2, UP

 

HA READY

Detailed state of the device selected for HA storage:

Chassis 1, serial: FOX1734GXB3, state: active

Server 1, serial: FCH1735V06K, state: active

Step 11.   Configure the ports (Fibre Channel ports, server ports, appliance ports, breakout ports, uplink ports, etc.) on the Cisco UCS 6332 platform subordinate fabric interconnect (Figure 6).

The system will reboot when you change ports from Ethernet to Fibre Channel or from 40-Gbps to 10-Gbps or vice versa. To reduce the number of reboots, apply changes to multiple ports instead of one by one.

 

 

Step 12.   Verify server discovery. The IOM discovery process on the Cisco UCS 6332 platform subordinate interconnect may take a few minutes to complete and become operational. In certain situations, you may need to acknowledge the IOM (choose Equipment > Chassis > Chassis X > IO Modules > IO Module X), as shown in Figure 7.

a.     Cisco UCS C-Series servers that are directly connected with dual adapters or are connected to a different port require server acknowledgment to properly update the port mapping. To reduce the number of server reboots to one, you can perform this step after the other fabric interconnect has been replaced.

b.     For servers connected to the Cisco Nexus 2200 platform fabric extenders, in most cases a fabric extender reacknowledgment will reestablish a connection. Otherwise, a server reacknowledgment is needed. To verify the connection, use the command shown here.

B26-A# connect adapter 2/1

adapter 0/2/1 # connect

<output truncated>

adapter 0/2/1 (top):1# attach-mcp

<output truncated>

adapter 0/2/1 (mcp):1# vnic

<output truncated>

------------------------------------------ --------- --------------------------

                 v n i c                     l i f             v i f

id  name           type    h:bb:dd.f state lif state uif  ucsm   idx vlan state

--- -------------- ------- --------- ----- --- ----- --- ----- ----- ---- -----

15 vnic_1         enet    0:05:00.0 UP      3 UP    - 0   790   110  180 UP

                                                     =>1   789    21  180 UP

Step 13.   After all port configuration processes are complete, including the process of adding ports back into LAN and SAN port channels and VLAN groups, you need to validate LAN and SAN connectivity. Use the appropriate show commands, such as the following:

●   show interface status

●   show port-channel summary

●   show interface trunk

●   show flogi database

●   show fcns database

 

Note that a Cisco UCS M4 server with a Cisco UCS VIC 1340 and a port expander that is connected to a Cisco UCS 2304 IOM will be connected as a native 40-Gbps port. You can verify this connection by entering the show interface status command in the Cisco NX-OS Software shell:

 

B26-B(nxos)# show interface status

<output truncated>

--------------------------------------------------------------------------------

Ethernet       VLAN   Type Mode   Status  Reason                  Speed     Port

Interface                                                                   Ch #

--------------------------------------------------------------------------------

Eth1/1/1       1       eth  access down    Member port of 40G         10G(D) --

Eth1/1/2       1       eth  access down    Member port of 40G         10G(D) --

Eth1/1/3       1       eth  access down    Member port of 40G         10G(D) --

Eth1/1/4       1       eth  vntag  up      none                       40G(D) –

<output truncated>

 

Step 14.   Reestablish flow on the subordinate fabric interconnect, the Cisco UCS 6332 platform. In the case shown here, fabric evacuation will be turned off (Figure 8).

a. If other method used other than fabric evacuation, then enable uplinks for Ethernet, Fibre Channel, and FCoE.

Note:    Some backplane ports may report link down because no service profile is associated with that server or the host is powered off.

Step 15.   Verify that traffic is flowing normally on the Cisco UCS 6332 platform subordinate interconnect.

Step 16.   After verifying traffic flow, promote the Cisco UCS 6332 platform subordinate interconnect to primary status by entering the cluster lead b command in the local-mgmt shell on the primary fabric interconnect:

B26-A(local-mgmt)# cluster lead b

If the system is at 'infrastructure firmware' auto-install 'pending user Ack' stage, please check the outstanding faults (scope monitoring <enter> show new-faults) and make sure the data-paths on FI-B are established properly before making it primary to ensure there is no data outage.

Do you want to continue? (yes/no):yes

Cluster Id: 0xac919450846211e5-0x833f002a6a923061

Step 17.   Enter the show cluster extended-state command to verify that the primary role has switched to the Cisco UCS 6332 platform and that high availability (HA) is in the ready state. Figure 9 shows the results in Cisco UCS Manager.

B26-A(local-mgmt)# show cluster extended-state

Cluster Id: 0xac919450846211e5-0x833f002a6a923061

 

Start time: Wed Jan 27 22:43:01 2016

Last election time: Thu Jan 28 06:59:27 2016

 

A: UP, SUBORDINATE

B: UP, PRIMARY

 

A: memb state UP, lead state SUBORDINATE, mgmt services state: UP

B: memb state UP, lead state PRIMARY, mgmt services state: UP

   heartbeat state PRIMARY_OK

 

INTERNAL NETWORK INTERFACES:

eth1, UP

eth2, UP

 

HA READY

Detailed state of the device selected for HA storage:

Chassis 1, serial: FOX1734GXB3, state: active

Server 1, serial: FCH1735V06K, state: active

 

Step 18.   Repeat the steps 3 through 14 to replace the other Cisco UCS 6200 Series Fabric Interconnect. Figure 10 shows the results.

Step 19.   (Optional) Promote fabric interconnect A to the primary role with the command cluster lead a in the local-mgmt shell of the primary fabric interconnect (B).

Step 20.   Check for faults from old configurations, policies, software packages, etc. Remove any noncompliant configurations, policies, and software packages to clear faults.

Transitioning from Cisco UCS 6100 Series Fabric Interconnect to Cisco UCS 6332 Platform

In-service upgrade from the Cisco UCS 6100 Series to Cisco UCS 6332 platform fabric interconnects is not supported. The upgrade path requires the existing Cisco UCS instance to use Cisco UCS Software Release 3.1(1), which does not support first-generation hardware. Thus, the better approach is to set up a new pod with the current hardware and migrate the workload to that.

You can export and import the configuration from the Cisco UCS 6100 Series to the 6300 Series, maintain the pools, policies, and service profiles. Any irrelevant or dated policies and configurations will be ignored by Cisco UCS Manager with Release 3.1(1).

If you want to perform in-service migration, this will require a multiple-upgrade path: from the Cisco UCS 6100 and 2100 Series to the Cisco UCS 6200 and 2200 Series, and then to the Cisco UCS 6332 platform and 2200 Series or the Cisco UCS 6332 and 2300 Series, with the assumption that the server hardware has been verified to support this migration.

For More Information

●   Read more about Cisco UCS products at
http://www.cisco.com/c/en/us/products/servers-unified-computing/index.html.

●   Read more about Cisco UCS 6332 platform fabric interconnects at http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-6300-series-fabric-interconnects/index.html.

●   Read more about the Cisco UCS 2304 Fabric Extender at http://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-6300-series-fabric-interconnects/data_sheet_c78-675243.html?cachemode=refresh.