Have an account?

  •   Personalized content
  •   Your products and support

Need an account?

Create an account

Migrate to Cisco UCS 6400 Fabric Interconnect with 10- to 25-Gbps Ports White Paper

White Paper

Available Languages

Download Options

  • PDF
    (2.0 MB)
    View with Adobe Reader on a variety of devices
Updated:April 7, 2020

Available Languages

Download Options

  • PDF
    (2.0 MB)
    View with Adobe Reader on a variety of devices
Updated:April 7, 2020
 

 

Cisco Unified Computing System overview

The Cisco Unified Computing System (Cisco UCS®) is a next-generation data center platform that unites computing, networking, storage access, and virtualization resources into a cohesive system designed to reduce Total Cost of Ownership (TCO) and increase business agility.

The Cisco UCS 6400 series FI is the replacement for the UCS 6200 series FI to deliver higher speeds for both Ethernet and Fibre Channel. The Cisco UCS 6400 series FI provides high performance, consistent latency, and lossless fabric to address the needs of customers deploying today’s modern data centers.

Product overview

The Cisco UCS 6400 series uses a cut-through architecture, supporting deterministic, low-latency, line-rate 10/25/40/100 Gigabit Ethernet ports.

The fourth-generation Cisco UCS 6454 Fabric Interconnect is a one-rack-unit device with high-density, nonblocking, low-power 10- and 25-Gbps ports. The switch has twenty-eight 10/25-Gbps Ethernet ports, four 1/10/25-Gbps Ethernet ports, six 40/100-Gbps Ethernet uplink ports, and sixteen unified ports that can support 10/25-Gbps Ethernet or 8/16/32-Gbps Fibre Channel ports. All Ethernet ports are capable of supporting FCoE. (See Figure 1.)

Cisco UCS 6454 Fabric Interconnect (rear view)

Figure 1.            

Cisco UCS 6454 Fabric Interconnect (rear view)

Cisco UCS 6454 Fabric Interconnect have two power supplies (hot swappable, 1+1) and four variable-speed fans (hot swappable, 3+1) that insert into the front of the chassis with fan-side intake and port-side exhaust. Server-facing ports are at the rear of the chassis to simplify connections to servers and fabric extenders. Serial, management, and USB ports are provided at the front of the chassis for easy access at the front of the rack. (See Figure 2.)

Cisco UCS 6454 Fabric Interconnect (front view)

Figure 2.            

Cisco UCS 6454 Fabric Interconnect (front view)

The Cisco UCS 64108 Fabric Interconnect is a two-rack-unit device with seventy-two 10/25-Gbps Ethernet ports, eight 1/10/25- Gbps Ethernet ports, twelve 40/100-Gbps Ethernet uplink ports, and sixteen unified ports that can support 10/25-Gbps Ethernet or 8/16/32-Gbps Fibre Channel ports. All Ethernet ports are capable of supporting FCoE. (See Figure 3.)

Cisco UCS 64108 Fabric Interconnect (rear view)

Figure 3.            

Cisco UCS 64108 Fabric Interconnect (rear view)

Cisco UCS 64108 Fabric Interconnect has two power supplies (hot swappable, 1+1) and three variable-speed fans (hot swappable, 2+1) that insert into the front of the chassis with front-to-back cooling. Server-facing ports are at the rear of the chassis to simplify connections to servers and fabric extenders. Serial, management, and USB ports are provided at the front of the chassis for easy access at the front of the rack. (See Figure 4.)

Cisco UCS 64108 Fabric Interconnect (front view)

Figure 4.            

Cisco UCS 64108 Fabric Interconnect (front view)

Audience

The audience for this document consists of system architects, system engineers, and any other technical staff who are responsible for planning or upgrading the fabric interconnects. Although every effort has been made to make this document appeal to the widest possible audience, the document assumes that readers have an understanding of Cisco UCS hardware, terminology, and configurations.

Objective

The Cisco UCS 6400 Series Fabric Interconnects are hardware switches that use the latest Cisco silicon Switch-on-Chip (SoC) ASIC technology. Because the hardware is new, you may need to make changes in port and software configurations to experience a successful migration.

This document discusses the process for upgrading from Cisco UCS 6200 Series Fabric Interconnects to Cisco UCS 6400 Series Fabric Interconnects. After reading this document, you should understand the upgrade process and any precautions you should consider to achieve a successful migration.

Planning: precheck steps

This document only describes the process of migrating from the Cisco UCS 6200 series FI to the Cisco UCS 6400 series FI. (Table 1) It summarizes various upgrade paths between fabric Interconnect, fabric extenders and IOMs, and Nexus 2000 Series Fabric Extenders.

Table 1.        Migration support matrix for infrastructure hardware

From

To

State

Cisco UCS 6100 Series Fabric Interconnects and 2100XP Fabric Extender

Cisco UCS 6400 series Fabric Interconnect, 2204/2208/2408 and 2232PP/2232TM-E Fabric Extenders

Not supported

Cisco UCS 6200 Series Fabric Interconnects, 2204/2208 and 2232PP/2232TM-E Fabric Extenders

Supported

Cisco UCS 6332 and 6332-16UP Fabric Interconnect and 2200XP Fabric Extender

Not supported

Cisco UCS 6324 Fabric Interconnect and 2300XP Fabric Extender

Not supported

Cisco UCS 6454 Fabric Interconnect, 2204/2208/2408 and 2232PP/2232TM-E Fabric Extenders

Cisco UCS 64108 Fabric Interconnect, 2204/2208/2408 and 2232PP/2232TM-E Fabric Extenders

Supported

Precheck 1: Back up and export Cisco UCS Manager configuration.

Precheck 2: Take an inventory of the Cisco UCS domain and remove any unsupported hardware.

Cisco UCS 6200 Series Fabric Interconnect migration to Cisco UCS 6454 Fabric Interconnect is supported from Cisco UCS Manager (UCSM) Release 4.0(1), and the 6200 series migration to the 64108 Fabric Interconnect is supported with UCSM Release 4.1(1). The focus of this white paper is from the perspective of UCSM Release 4.1(1). Tables 2 and 3 list the supported hardware for Cisco UCSM Release 4.1(1). Note that any server prior to the M3 platform will cause the upgrade to fail on the Cisco UCS 6200 Series Fabric Interconnects.

Table 2.        Server migration support for Cisco UCSM 4.1(1)

Cisco UCS B-Series Blade Servers

M3

M4

M5

Cisco UCS B200

Yes

Yes

Yes

Cisco UCS B260

-

Yes

-

Cisco UCS B420

Yes

Yes

-

Cisco UCS B460

-

Yes

-

Cisco UCS B480

-

-

Yes

 

Cisco UCS C-Series Rack Servers

M3

M4

M5

Cisco UCS C24

Yes

-

-

Cisco UCS C220

Yes

Yes

Yes

Cisco UCS C240

Yes

Yes

Yes

Cisco UCS C420

Yes

-

-

Cisco UCS C460

-

Yes

-

Cisco UCS C480

-

-

Yes

Table 3.        I/O and fabric extender migration support for Cisco UCSM 4.1(1)

Adapter for Cisco UCS B-Series Blade Servers

Fabric extenders

Cisco UCS VIC 1240

Cisco UCS 2204/2208

Cisco UCS VIC 1280

Cisco UCS 2204/2208

Cisco UCS VIC 1340

Cisco UCS 2204/2208, 2408

Cisco UCS VIC 1340 + port-expander

Cisco UCS 2204/2208, 2408

Cisco UCS VIC 1380

Cisco UCS 2204/2208, 2408

Cisco UCS VIC 1440

Cisco UCS 2204/2208, 2408

Cisco UCS VIC 1480

Cisco UCS 2204/2208, 2408

 

Adapter for Cisco UCS C-Series Rack Servers

Fabric extenders

Cisco UCS VIC 1225 and 1225T

Cisco Nexus® 2232PP, 2232TM-E

Cisco UCS VIC 1227 and 1227T

Cisco Nexus 2232PP, 2232TM-E

Cisco UCS VIC 1285

Cisco Nexus 2232PP, 2232TM-E

Cisco UCS VIC 1385

Cisco Nexus 2232PP, 2232TM-E

Cisco UCs VIC 1387

Cisco Nexus 2232PP, 2232TM-E

Cisco UCS VIC 1455

Cisco Nexus 2232PP, 2232TM-E

Cisco UCS VIC 1457

Cisco Nexus 2232PP, 2232TM-E

Note:     Cisco UCS Virtual Interface Card (VIC) 1400 series and IOM 2408 are supported only from UCSM releases 4.0(4a) and 4.0(4e), respectively. The VIC 1495/1497 cannot be connected to fabric interconnects and hence is not part of the adapter list.

Precheck 3: Confirm that the data VLAN does not overlap with the VLANs reserved for the Cisco UCS 6400 Series Fabric Interconnects. If it does, then identify a block of 128 contiguous VLANs. With the 6400 series, the number of active VLANs increases from 2000 in the predecessor switches to 3000 in the new switches. This document describes how to change the reserved VLAN range in a later step (step 8). Table 4 lists the reserved VLAN ranges.

Table 4.        Cisco UCS and fabric interconnect reserved VLANs

Cisco UCS 6200 Series Fabric Interconnects

Cisco UCS 6400 Series Fabric Interconnects

VLANs 4030 to 4047

VLANs 3915 to 4042 (reserved for Cisco® NX-OS)

 

VLANs 4030 to 4047 (reserved for Cisco UCS Manager)

 

VLANs 4093 to 4098 (reserved for Cisco NX-OS)

If you have an active VLAN mapped to ID 4093, this VLAN needs to be remapped to another VLAN. VLAN 4093 is reserved for the fabric interconnect, and this VLAN ID cannot be changed.

Figure 3 shows an example from the Cisco UCS Manager installer in which active VLANs overlapped with the default VLANs reserved for the Cisco UCS 6400 series.

Example showing active VLANs overlapping with the default VLANs reserved for the Cisco UCS 6400 Series Fabric Interconnects

Figure 5.            

Example showing active VLANs overlapping with the default VLANs reserved for the Cisco UCS 6400 Series Fabric Interconnects

Precheck 4: Validate software features and configurations. Cisco UCS 6400 Series Fabric Interconnects do not support some software features that were allowed with the 6200 series. Some of these features will become available at a later software release.

Table 5.        Features that need special attention prior to upgrading

Feature

Remediation

Chassis and fabric extender, I/O port channel

Select a port channel to the I/O Module (IOM).

Multicast optimization

Verify that multicast optimization is not enabled under the LAN Quality-of-Service (QoS) system classes.

NetFlow

Unconfigure NetFlow.

MAC Security

Select Allow for MAC security.

VM-FEX

Remove port profiles and UCSM ESXi/SCVMM related configurations.

Dynamic vNIC connection policies

Set the dynamic vNIC connection policy in the vNIC profile to “Not set.”

Cisco Switched Port Analyzer (SPAN)

Use receive (RX) direction only. The installer will change SPAN to the RX direction and send an alert indicating that this setting is being changed.

Failure to comply with these remediation steps will result in a migration warning alert during the migration process and prevent the fabric interconnects from synchronizing. On changing the “Chassis/FEX discovery policy” to port-channel, the chassis needs to be re-acknowledged before proceeding with the migration. Figure 4 shows an example of a migration warning.

Sample migration warning

Figure 6.            

Sample migration warning

Precheck 5: The Cisco UCS 6200 Series Fabric Interconnects are modular, whereas the 6400 series are fixed, as shown in Figures 7 and 8. Thus, the port schemes change when you migrate from the 6200 series to the 6400 series. Tables 6 and 7 list the supported port capabilities of the 6400 Series. Use the information in this table to plan the port migration.

Note:     As a best practice, you should label the cables.

Ports on the Cisco UCS 6454 Fabric Interconnect

Figure 7.            

Ports on the Cisco UCS 6454 Fabric Interconnect

Table 6.        Port capabilities for the Cisco UCS 6454 Series Fabric Interconnects in Cisco UCSM 4.1(1)

Port no.

Interface type

Protocol support

Port role

 

1 Gigabit Ethernet

10/25 Gigabit Ethernet

40/100 Gigabit Ethernet

8/16/32-Gbps Fibre Channel

4 x 10/25-Gbps breakout

Ethernet

Fibre Channel

FCoE

Server

Uplink: Ethernet

Uplink: Fibre Channel

Uplink: FCoE

Appliance port

Storage port (Fibre Channel)

Storage port (and FCoE)

1 to 16

 

X

 

X

 

X

X

X

X

X

X

X

X

X

X

17 to 44

 

X

 

 

 

X

 

X

X

X

 

X

X

 

X

45 to 48

X

X

 

 

 

X

 

X

X

X

 

X

X

 

X

49 to 541

 

 

X

 

X

X

 

X

 

X

 

X

 

 

X

1 QSFP+ will support port roles other than the uplink role in a later software release.

Ports on the Cisco UCS 64108 Fabric Interconnect

Figure 8.            

Ports on the Cisco UCS 64108 Fabric Interconnect

Table 7.        Port capabilities for the Cisco UCS 64108 Series Fabric Interconnects in Cisco UCSM 4.1(1)

Port no.

Interface type

Protocol support

Port role

 

1 Gigabit Ethernet

10/25 Gigabit Ethernet

40/100 Gigabit Ethernet

8/16/32-Gbps Fibre Channel

4 x 10/25-Gbps breakout

Ethernet

Fibre Channel

FCoE

Server

Uplink: Ethernet

Uplink: Fibre Channel

Uplink: FCoE

Appliance port

Storage port (Fibre Channel)

Storage port (FCoE)

1 to 16

 

X

 

X

 

X

X

X

X

X

X

X

X

X

X

17 to 88

 

X

 

 

 

X

 

X

X

X

 

X

X

 

X

89 to 96

X

X

 

 

 

X

 

X

X

X

 

X

X

 

X

97 to 1081

 

 

X

 

X

X

 

X

 

X

 

X

 

 

X

1 QSFP+ will support port roles other than the uplink role in a later software release.

Precheck 6: All the components need to be upgraded to Cisco UCSM release 4.1(1 – Fabric Interconnect, IOM, and servers.

Upgrade steps

After the prerequisite steps are complete, the upgrade process can commence.

Step 1: Move the traffic to the primary fabric interconnect. This change can be accomplished in several ways. This document describes two of these approaches: fabric evacuation and uplink disablement. Use the flowchart in Figure 6 to choose between the two based on your Cisco UCS domain server and connectivity.

Flowchart for choosing the best method for moving traffic

Figure 9.            

Flowchart for choosing the best method for moving traffic

Figure 10 shows the fabric evacuation method.

Fabric evacuation

Figure 10.         

Fabric evacuation

Figure 11 shows the uplink disablement method.

Uplink disablement

Figure 11.         

Uplink disablement

Step 2: Verify that all traffic has failed over to the primary fabric interconnect.

Note:     If you decide that you want to unconfigure all the ports of the subordinate fabric interconnect and reconfigure them after you add the new subordinate fabric interconnect, this is the time to do so.

Step 3: Power down the old subordinate fabric interconnect and disconnect the power and the L1/L2 cables.

Step 4: If the new fabric interconnect is being racked in the same location as the old one, disconnect the server cables. Un-rack the old subordinate fabric interconnect and rack the new subordinate fabric interconnect.

Note:     As a best practice, you should label the cables.

Step 5: Connect the L1/L2 cables and the server ports according to your port mapping plan.

Step 6: Power up the new fabric interconnect. If it is connected correctly, the new subordinate fabric interconnect will detect that it is being added to an existing cluster (Figure 9).

Powering up the new fabric interconnect

Figure 12.         

Powering up the new fabric interconnect

Step 7: Enter the IP address information for the new subordinate fabric interconnect (Figure 12).

Entering the IP address information

Figure 13.         

Entering the IP address information

Step 8: After the management IP address is configured for the new subordinate fabric interconnect, the installer will perform a configuration and VLAN check.

If any misconfiguration is found (incorrect switch mode, VM-FEX, non-port-channel IOM, etc.), the installer will provide a list of objects that are out of compliance, as shown in the example in Figure 14.

Checking for misconfigurations

Figure 14.         

Checking for misconfigurations

Any objects that are out of compliance will need to be fixed as described in Table 5 before you can proceed with the migration. Corrections can be performed while in a mixed fabric interconnect connection.

If the active VLANs overlapped with the default reserved VLANs for the Cisco UCS 6400 Series Fabric Interconnect, the warning message shown in Figure 15 will be displayed.

Warning message for overlapping VLANs

Figure 15.         

Warning message for overlapping VLANs

If VLANs overlap, change the reserved VLANs after the new subordinate fabric interconnect joins the cluster. The change can be implemented through the GUI or the Command-Line Interface (CLI), as shown in Figures 16 and 17.

Changing the reserved VLANs in the GUI

Figure 16.         

Changing the reserved VLANs in the GUI

Changing the reserved VLANs in the CLI

Figure 17.         

Changing the reserved VLANs in the CLI

After you change the reserved VLAN block, you will need to reboot the new subordinate fabric interconnect. If you need to configure Fibre Channel, then wait to reboot until after you have configured the Fibre Channel ports. Otherwise, reboot the new subordinate fabric interconnect and skip step 9 and proceed to step 10.

Note:     Do not reboot the primary fabric interconnect. The change in the reserved VLAN range is a global configuration, and when the primary fabric interconnect is replaced, the fabric interconnect will inherit the changes.

Step 9: Convert Ethernet ports to Fibre Channel ports and reboot (Figure 18). The default speed will be 16 Gbps. If 8-Gbps Fibre Channel optics are used, then the port will be error-disabled. Manually set the speed to 8 Gbps and disable and reenable the port. If the system is connected to a Cisco MDS 9000 Family switch, then you need to configure the “fill-pattern IDLE speed 8000” setting.

Note:     When you convert from Ethernet ports to Fibre Channel ports, a reboot is required. Fibre Channel ports are converted in blocks of four.

Converting the ports

Figure 18.         

Converting the ports

Step 10: After bootup, verify that the new subordinate fabric interconnect joined the cluster successfully.

Step 11: Configure the ports based on your port-mapping worksheet. If you disabled the uplinks (Ethernet or Fibre Channel) in step 1, then make sure to disable the uplinks after the configuration is complete to help ensure that the traffic is still flowing on the primary fabric interconnect.

Note:     If the Cisco UCS domain is configured for disjointed networks, then validate VLANs to uplink.

If IOM 2204/2208 migration to IOM 2408 is also planned, this would be the time to replace the IOMs toward the subordinate fabric interconnect. Unconfigure server ports on the Cisco UCS 6400 Series Fabric Interconnect before proceeding with replacing the IOMs. Note that IOM 2408 connects to a 6400 series fabric interconnect over 25 GbE links. After connecting the IOM 2408, reconfigure the server ports on the 6400 series fabric interconnect and wait for the IOM 2408 auto upgrade to complete.

Step 12: Neither an IOM nor a fabric extender require re-acknowledgement, but if they do not become operational, re-acknowledgement may be required. You can perform this step on a per-device basis, so re-acknowledge just the IOMs and fabric extenders that were migrated to the 6400 series fabric interconnect (Figure 19).

For directly connected rack servers, re-acknowledgment is needed to help ensure that the Virtual Interfaces (VIFs) have been built correctly. Re-acknowledgment for rack servers is disruptive because it reboots the server.

Re-acknowledging fabric extender and IOM

Figure 19.         

Re-acknowledging fabric extender and IOM

Step 13: Move the traffic back to the new subordinate fabric interconnect. If you used the fabric evacuation method, then deselect fabric evacuation. If you disabled the uplinks (Ethernet and Fibre Channel), then reenable the uplinks. Verify that traffic is flowing correctly on the new subordinate fabric interconnect.

Step 14: After verifying that traffic is flowing on the subordinate fabric interconnect, promote the new subordinate fabric interconnect to make it the primary fabric interconnect. Verify that the primary fabric interconnect is now the 6400 series (Figure 20).

Verifying that the Cisco UCS 6400 series is the primary fabric interconnect

Figure 20.         

Verifying that the Cisco UCS 6400 series is the primary fabric interconnect

Step 15: Repeat these steps for the other fabric-interconnect replacements to complete the migration. Steps 1–15 detailed the migration of Cisco UCS 6200 series to 6400 series fabric interconnects. The same steps can be followed to migrate from 6454 fabric interconnect to the higher-port-count 64108 fabric interconnect. For 6454 to 64108 migration, one additional check needs to be taken care of at Step 2, unconfiguring ports 49–54 on the 6454 fabric interconnects and then reconfiguring these ports on the 64108s after completing the migration of each fabric interconnect.

For more information

Read more about Cisco UCS products:
https://www.cisco.com/c/en/us/products/servers-unified-computing/index.html.

Read more about the Cisco UCS 6400 Series Fabric Interconnect:
https://www.cisco.com/c/en/us/products/servers-unified-computing/fabric-interconnects.html.

Read more about the Cisco UCS VIC 1400:
https://www.cisco.com/c/en/us/products/interfaces-modules/unified-computing-system-adapters/index.html.

Learn more