Migrating to Cisco UCS 6400 Series Fabric Interconnect

Migrating Fabric Interconnects

Fabric Interconnect Migration Considerations

Ensure that the following prerequisites are met before beginning any procedures in this section:


Caution

Cisco UCS Manager Release 4.0 is the bare minimum version that provides support for Cisco UCS 6454 Fabric Interconnects. To migrate from Cisco UCS 6200 Series to Cisco UCS 6454 Fabric Interconnects:

  • Cisco UCS 6200 Series Fabric Interconnects must be on Cisco UCS Manager Release 4.0(1) or a later release.

  • Cisco UCS 6454 Fabric Interconnects must be loaded with the same build version that is on the Cisco UCS 6200 Series Fabric Interconnect that it will replace.

  • Licenses from Cisco UCS 6200 Series Fabric Interconnects are not transferable to Cisco UCS 6454 Fabric Interconnects. You must obtain licenses for the Cisco UCS 6454 Fabric Interconnects before you upgrade.

  • During migration, the Cisco UCS 6200 Series Fabric Interconnect and the Cisco UCS 6454 Fabric Interconnect must use the same allowed SSL protocol, either default or Only TLSv1.2, to successfully complete compatibility checks.

  • Cisco UCS 6454 Fabric Interconnects use the IDLE fill pattern for FC uplink ports and FC storage ports when using 8 Gbps speed.

    When migrating to Cisco UCS 6454 Fabric Interconnects and configuring FC Uplink Ports or FC Storage Ports at 8Gbps speed, ensure that the fill pattern is set as IDLE on the corresponding FC switch ports and the direct-attached FC storage array ports. If the fill pattern is not set as IDLE, FC uplink ports and FC storage ports operating at 8 Gbps might go to an errDisabled state, lose SYNC intermittently, or receive errors or bad packets.

    Cisco UCS 6454 Fabric Interconnects supports 8 Gbps only with fill-pattern set to IDLE for direct-attached FC connectivity (FC uplink ports or FC storage ports). This limitation is not applicable for 6454 Fabric Interconnects with FC ports at 16 Gbps and 32 Gbps. When migrating to Cisco UCS 6454 Fabric Interconnects from Cisco UCS 6200 Series Fabric Interconnects for direct-attached storage arrays that don’t support IDLE fill-pattern at 8 Gbps do one of the following:

    • Use a SAN switch between the Cisco UCS 6454 Fabric Interconnect and the storage array with 8 GB FC connectivity.

    • Upgrade the storage array to 16 GB or 32 GB FC connectivity.

  • Before migrating from Cisco UCS 6200 Series Fabric Interconnects to Cisco UCS 6454 Fabric Interconnects, ensure that you unconfigure the unified ports on the Cisco UCS 6200 Series Fabric Interconnects.

    After migrating to Cisco UCS 6454 Fabric Interconnects, reconfigure the unified ports based on their location on the Cisco UCS 6454 Fabric Interconnects, and reacknowledge the newly configured ports. For example, a unified port on a UCS 6248 Fabric Interconnect should be reconfigured on any port between 1 and 16 on a Cisco UCS 6454 Fabric Interconnect.

  • Upgrading the fabric interconnect should be done before upgrading to a new FEX or virtual interface card.

  • During fabric interconnect migration, image synchronization between fabric interconnects is not allowed. This is done to prevent incompatible images from getting synchronized. We recommend that you download B-Series and C-Series server software bundles again after migration is complete.

  • Do not attempt to implement new software features from the new Cisco UCS software version until all required hardware is installed.

  • Changes to the topology, such as the number of servers or uplink connections, should be performed after the fabric interconnect migration is complete.

  • Make a detailed record of the cabling between FEXes and fabric interconnects. You must preserve the physical port mapping to maintain the server pinning already configured and minimize down time.

  • For a cluster configuration, both fabric interconnects must have symmetrical connection topologies between fabric interconnect and FEXes.

  • Cisco UCS VIC 1455 and 1457 adapters support cables of 10G and 25G speed. However, the cables connecting Cisco UCS VIC 1455 or 1457 adapter ports to each 6454 fabric interconnect must be of uniform speed-either all 10G or all 25G cables. If you connect these adapter ports to a 6454 fabric interconnect through a mix of 10G and 25G cables, UCS rack-mount server discovery fails and ports may go to a suspended state.

  • Standalone installations should expect down time. Migrating or upgrading a fabric interconnect is inherently traffic disruptive.

  • A best practice would be to perform a full configuration and software backup before performing this hardware upgrade.

  • A WWN pool can include only WWNNs or WWPNs in the ranges from 20:00:00:00:00:00:00:00 to 20:FF:00:FF:FF:FF:FF:FF or from 50:00:00:00:00:00:00:00 to 5F:FF:00:FF:FF:FF:FF:FF. All other WWN ranges are reserved. When fibre channel traffic is sent through the UCS infrastructure the source WWPN is converted to a MAC address. You cannot use WWPN pool which can translate to source multicast MAC addresses. To ensure the uniqueness of the Cisco UCS WWNNs and WWPNs in the SAN fabric, Cisco recommends using the following WWN prefix for all blocks in a pool: 20:00:00:25:B5:XX:XX:XX

Validating Feature Configurations before Upgrade

Cisco UCS 6454 Fabric Interconnect does not support some software features that were allowed with Cisco UCS 6200 Fabric Interconnect.

Table 1. Features that needs special attention prior to upgrading

Feature

Remediation

Chassis and fabric extender I/O port channel

Select a port channel to the I/O module (IOM).

Multicast optimization

Verify that multicast optimization is not enabled under the LAN quality-of-service (QoS) system classes

Fabric forwarding mode for Ethernet

Verify that the Ethernet forwarding mode is set to End Host Mode Only.

Fabric forwarding mode for Fibre Channel

Verify that Fibre Channel forwarding mode is set to End Host Mode or FC Switching Mode.

Cisco NetFlow

Unconfigure NetFlow.

MAC Security

Select Allow for MAC security.

VM-FEX

Remove port profiles and Cisco UCS Manager ESXi or SCVMM related configurations.

Dynamic vNIC connection policies

Set the dynamic vNIC connection policy in the vNIC profile to Not set.

Cisco Switched Port Analyzer (SPAN)

Use receive (RX) direction only. The installer will change SPAN to the RX direction and send an alert indicating that this setting is being changed.

Port Mapping for Upgrades

The upgrade described here is primarily for upgrading a Cisco UCS 6248 fabric interconnect to a Cisco UCS 6454. The same considerations will also apply when upgrading a Cisco UCS 6296 fabric interconnect to a Cisco UCS 6454.


Note

If you are using appliance ports for direct attached storage, you must add VLANs to the ethernet uplinks. This will ensure that vNICS can properly pin on boot.

Fixed Ports

On the UCS 6248 fabric interconnect, you can separate the 32 physical ports in slot one into two contiguous pools, low numbered ports being Ethernet ports and high numbered ports being Fibre Channel ports. On the UCS 6454 fabric interconnect, you can configure the first 16 ports as Fibre Channel ports.

Because a UCS 6248 has 32 ports in slot 1 and a UCS 6454 has all ports in slot 1, any ports on GEM slots will be removed during the hardware upgrade process.


Caution

If you ever need to change the pool sizes for slot 1, you must reboot the fabric interconnect which can lead to a service disruption. If you ever need to change the pool sizes for slot 2, you must reset the expansion module in slot 2. To minimize disruption, plan to have at least a few Ethernet uplink and Fibre Channel uplink ports configured on slot 1. Implement this fail safe after the upgrade is complete and the system restabilizes.

Figure 1. Cisco UCS 6454 Port Numbering

1

Ports 1-16 (Unified Ports 10/25 Gbps Ethernet or FCoE or 8/16/32 Gbps Fibre Channel)

Note

 

When using Cisco UCS Manager releases earlier than 4.0(4), only ports 1-8 are Unified Ports.

2

Ports 17-44 (10/25 Gbps Ethernet or FCoE)

Note

 

When using Cisco UCS Manager releases earlier than 4.0(4), ports 9-44 are 10/25 Gbps Ethernet or FCoE.

3

Ports 45-48 (1/10/25 Gbps Ethernet or FCoE)

4

Uplink Ports 49-54 (40/100 Gbps Ethernet or FCoE)

Each of these ports can be 4 x 10/25 Gbps Ethernet or FCoE uplink ports when using an appropriate breakout cable.

Migrating from UCS 6200 Series Fabric Interconnects to UCS 6454 Fabric Interconnects

You can migrate from a UCS 6200 Series Fabric Interconnect to a UCS 6454 Fabric Interconnect. However, you cannot migrate back to a UCS 6200 Series Fabric Interconnect after you have migrated to a UCS 6454 Fabric Interconnect.

The UCS 6454 Series Fabric Interconnect does not support a few software features that were supported on UCS 6200 Series Fabric Interconnect in Cisco UCS Manager, Release 3.2 and earlier releases. For more information, see Software Feature Configuration.

The UCS 6454 Series Fabric Interconnect supports only port-channel mode for chassis-discovery. On changing the chassis or FEX discovery policy to port-channel, the chassis needs to be re-acknowledged before proceeding with the migration. If the chassis is not re-acknowledged, the migration will fail.

The UCS 6454 Fabric Interconnect is intended as a replacement for the UCS 6200 Series Fabric Interconnect, but not as a replacement for the higher speed (or 40Gb) UCS 6332/6332-16UP Fabric Interconnect. Therefore, Cisco has not tested or published a plan to migrate from UCS 6332/6332-16UP Fabric Interconnects to UCS 6454 Fabric Interconnects.

Unless otherwise noted, for more information about how to perform configuration procedures in Cisco UCS Manager for a particular step, see the appropriate Cisco UCS Manager configuration guide for Cisco UCS Manager.

Procedure

Step 1

Download Cisco UCS Manager, Release 4.0 or later versions to the UCS 6200 Series Fabric Interconnects and upgrade to this version.

Step 2

Evacuate traffic from the subordinate fabric interconnect to ensure there is no data traffic impact during migration.

For more information, see the Fabric Interconnect Traffic Evacuation section in the Guidelines and Prerequisites chapter of the Cisco UCS Manager Firmware Management Guide.

Step 3

When migrating from UCS 6200 Series Fabric Interconnects to UCS 6400 Series Fabric Interconnects with B-Series servers and S-Series servers, ensure that the port channel is enabled before migration.

Step 4

Unconfigure all the unified ports on the subordinate fabric interconnect.

Step 5

Power down the subordinate fabric interconnect by unplugging it from the power source.

If you are monitoring the migration using a KVM session, you may need to reconnect the KVM session when you power down the fabric interconnect.

Step 6

Mount the replacement UCS 6454 fabric interconnect into either the same rack or an adjacent rack.

Refer to the Cisco UCS 6454 Installation Guide for details.

Step 7

Disconnect the cables from the chassis FEXes or fabric extenders to the subordinate fabric interconnect ports in slot 1 on the UCS 6200 Series Fabric Interconnect.

Step 8

Connect these cables into the corresponding ports on slot 1 of one of the new Cisco UCS 6454 fabric interconnects, using the connection records to preserve the port mapping and the configured server pinning. To change the port mapping, especially while reconfiguring FC ports, you must reacknowledge the newly configured ports.

Step 9

Connect the L1/L2 cables that were disconnected onto the new Cisco UCS 6454 fabric interconnect.

L1 connects to L1, L2 connects to L2.

Step 10

Connect the server and uplink cables.

Refer to the Cisco UCS 6454 Installation Guide for details.

Step 11

Connect the power to the new Cisco UCS 6454 fabric interconnect, it will automatically boot and run POST tests. If it reboots itself, this is a normal behavior.

Important

 

Directly connect the console port to a terminal and observe the boot sequence. You should at some point see the Basic System Configuration Dialog, where you will configure the switch as a subordinate interconnect. If you do not see this dialog, you either have different builds of software on your old primary and new subordinate, or the new subordinate has previously been part of a cluster and will need to have all configuration information wiped before it can be added to a cluster as a subordinate. In either case, immediately disconnect the L1 and L2 connections and complete the bringup as a standalone fabric interconnect, then correct the issue before proceeding further.

Step 12

Configure the server and uplink ports on the new Cisco UCS 6454 fabric interconnect.

Step 13

The new subordinate Cisco UCS 6454 fabric interconnect will automatically synchronize the configuration and database/state information from the primary UCS 6200 Series Fabric Interconnect.

Synchronization between primary and subordinate fabric interconnects can take several minutes. You may see an error message that will persist until the server ports are enabled.

The port configuration is copied from the subordinate switch to the new hardware.

Step 14

Reconfigure the server ports that had been unconfigured in Step 4.

  1. If you have changed port mappings, you may need to reacknowledge the IOM, FEX, or direct-connect rack server connected to the subordinate fabric interconnect.

  2. Verify and if necessary reconfigure Ethernet ports as server ports.

Step 15

Verify that the data path is ready.

For more information, see the Verifying that Dynamic vNICs Are Up and Running section in the Guidelines and Prerequisites chapter of the Cisco UCS Manager Firmware Management Guide.

Ensure that all faults are resolved before proceeding.

  1. Verify and if necessary reconfigure the SAN pin group for FC ports in the associated service profile.

  2. Verify and if necessary reconfigure the LAN pin group for Ethernet ports in the associated service profile.

  3. Verify and if necessary reconfigure the port channel for uplink Ethernet ports.

Step 16

Restart stopped traffic flows by disabling fabric evacuation.

Step 17

Promote the subordinate fabric interconnect to primary, and repeat the process on the second new Cisco UCS 6454 fabric interconnect.

Cable the second new fabric interconnect identically to the first, and allow the reconfiguration done to be applied to the second new fabric interconnect as well.

Cisco UCS 6400 Series Fabric Interconnect Migration Considerations

Beginning with Cisco UCS Manager release versions 4.0, you can migrate from Cisco UCS 6200 Series Fabric Interconnects to Cisco UCS 6454 Fabric Interconnects respectively.

To migrate from Cisco UCS 6200 Series Fabric Interconnects to 6454 Fabric Interconnects:

  • Cisco UCS 6200 Series Fabric Interconnects must be on Release 4.0(1) or a later release to migrate to Cisco UCS 6454 Fabric Interconnects.

  • Cisco UCS 6454 Fabric Interconnects must be loaded with the same infrastructure firmware version as on the Cisco UCS 6200 Series Fabric Interconnect that it will replace.

Prerequisites

Before performing the migration from Cisco UCS 6200 series fabric interconnects to Cisco UCS 6454 fabric interconnects, ensure that the following prerequisites are met for a successful migration:

  • Back up and export Cisco UCS Manager configuration before initiating the upgrade. Ensure that the backup file from the 6200 Series Fabric Interconnects is compatible with the 6454 Fabric Interconnects. You can follow any of the backup methods:

    • Online Migration: This method allows for backing up the configuration while the Fabric Interconnect is operational. It ensures minimal disruption and maintains continuous availability during the migration process.

    • Offline Migration: This method involves taking the fabric interconnect offline to perform the backup, which may result in downtime.


      Note

      For more information, see .

  • Take an inventory of the Cisco UCS domain and remove any unsupported hardware.

  • Ensure to enable the cluster failover.

  • Do not attempt to implement new software features from the new Cisco UCS software version until all required hardware is installed.

  • Validate the software features and configurations between Cisco UCS 6200 series and UCS 6400 series. Ensure that the 6200 series fabric interconnect is reconfigured to only have features that are supported with 6400 series fabric interconnect.


    Note

    This is one of the most common reasons for migration failures.

  • Make sure both Cisco UCS 6200 series and 6400 series fabric Interconnects are on the same UCSM build before migration.

  • Licenses from Cisco UCS 6200 series fabric interconnects are not transferable to Cisco UCS 6400 series fabric interconnects.

  • Standalone installations should expect down time. In a cluster configuration, migrating the fabric interconnects can result in a small traffic disruption when the traffic fails over from one Fabric Interconnect to another. To avoid that there is no permanent traffic loss during migration, ensure that there is redundancy in the UCS domain on both fabric interconnects before migration and test the redundancy before starting the migration.

  • Ensure that the fill pattern is set as IDLE on the corresponding FC switch ports and the direct-attached FC storage array ports when configuring FC Uplink Ports or FC Storage Ports at 8Gbps speed. If the fill pattern is not set as IDLE, FC Uplink ports and FC Storage ports operating at 8 Gbps might go to an errDisabled state, lose SYNC intermittently, or receive errors or bad packets.

    Cisco UCS 6400 series fabric interconnect supports 8 Gbps only with fill-pattern set to IDLE for direct-attached FC connectivity (FC uplink ports or FC storage ports). This limitation is not applicable for Cisco UCS 6400 series fabric interconnects with Fibre Channel (FC) ports at 16 Gbps and 32 Gbps. When migrating to Cisco UCS 6400 series fabric interconnects for direct-attached storage arrays that don’t support IDLE fill-pattern at 8 Gbps do one of the following:

    • Use a SAN switch between the Cisco UCS 6400 series fabric interconnects and the storage array with 8 GB FC connectivity.

    • Upgrade the storage array to 16 GB or 32 GB FC connectivity.

  • Ensure the latest firmware bundle is downloaded and upgraded through GUI or CLI. Incase of attempting to upgrade the firmware bundle using other methods (loader prompt/erase configuration) can result in missing package version.

  • Before migration, make sure that the FC Speed is 8Gbps on Cisco UCS 6200 fabric interconnects or the connected switch supports 8Gbps speed.


    Note

    Post migration, you can configure the FC Port (Scalability Port) speed on Cisco UCS 6400 fabric interconnects.

  • Migrating to different IOM models can result in peer communication issue between IOMs of the primary and secondary fabric interconnects.

  • Make a detailed record of the cabling between FEX and fabric interconnects. You must preserve the physical port mapping to maintain the server pinning already configured and minimize down time.

  • For a cluster configuration, both fabric interconnects must have symmetrical connection topologies between fabric interconnect and FEX.

  • Use the same speed cables on all the adapter ports that are connected to same Fabric Interconnect. Cisco UCS VIC adapter ports connected to Cisco UCS 6400 series fabric interconnect through a mix of 10G and 25G cables can result in UCS rack-mount server discovery failure and ports moving to suspended state.

  • A WWN pool can include only WWNNs or WWPNs in the ranges from 20:00:00:00:00:00:00:00 to 20:FF:00:FF:FF:FF:FF:FF or from 50:00:00:00:00:00:00:00 to 5F:FF:00:FF:FF:FF:FF:FF. All other WWN ranges are reserved. When fibre channel traffic is sent through the UCS infrastructure the source WWPN is converted to a MAC address. You cannot use WWPN pool which can translate to source multicast MAC addresses. To ensure the uniqueness of the Cisco UCS WWNNs and WWPNs in the SAN fabric, Cisco recommends using the following WWN prefix for all blocks in a pool: 20:00:00:25:B5:XX:XX:XX

  • Unconfigure the fibre channel ports on the migrating subordinate Cisco UCS 6200 series Fabric Interconnect and reconfigure on the Cisco UCS 6454 fabric interconnects.

  • When migrating from Cisco UCS 6296 Fabric Interconnect to Cisco UCS 6454 Fabric Interconnect, the ports on the 64108 Fabric Interconnect that are connected to the extended module on the 6296 Fabric Interconnect must be reacknowledged.

  • Before migrating from Cisco UCS 6200 series fabric interconnects to Cisco UCS 6454 fabric interconnects, ensure that you unconfigure the unified ports on the Cisco UCS 6200 Series fabric interconnects.

    After migrating to Cisco UCS 6454 fabric interconnects, reconfigure the unified ports based on their location on the Cisco UCS 6454 fabric interconnects, and reacknowledge the newly configured ports. For example, a unified port on a UCS 6248 fabric interconnect should be reconfigured on any port between 1 and 16 on a Cisco UCS 6454 fabric interconnect.


    Note

    You should login directly to the new Cisco UCS 6454 Fabric Interconnect GUI to configure the unified ports during the migration.

  • Upgrading the fabric interconnect should be done before upgrading to a new FEX or virtual interface card.

Recommendations

Following are the best practices for a successful migration:

  • For minimal disruption during migration, ensure that there is redundancy for Ethernet and FC traffic from the servers in the UCS domain across both 6200 series fabric interconnects before migration.

  • Changes to the topology, such as the number of servers or uplink connections, should be performed after the fabric interconnect migration is complete.

  • During the migration of Fabric Interconnects, ensure the Cluster ID is not changed.

  • During the migration, image synchronization between fabric interconnects is not allowed. This is done to prevent incompatible images from getting synchronized. It is necessary to download B-Series, C-Series, and S-Series server software bundles again after migration is complete.

  • During the migration, ensure that VLAN is not created in the range of 3915 to 4042 which are the reserved VLAN range for Cisco UCS 6400 series fabric interconnects.

  • After migrating from Cisco UCS 6200 to 6400 series fabric interconnects, ensure to assign unique VLAN IDs while reconfiguring the Fibre Channel (FC) ports. It is critical to prevent the virtual SAN's FCoE VLAN IDs from overlapping with the Cisco UCS 6400 fabric interconnect's reserved VLAN IDs, as this overlap can result the FC ports to shut down (Admin down state).

  • The Cisco UCS 6400 series fabric interconnect supports only port-channel mode for chassis-discovery. Hence, change the Chassis/FEX discovery policy on the Cisco UCS 6200 series Fabric Interconnect to port-channel and immediately re-acknowledge the Cisco UCS 5108 chassis.


    Note

    The migration can fail when the chassis is not immediately re-acknowledged.

Important

For more information on migrating Cisco UCS 6200 Series Fabric Interconnects to Cisco UCS 6400 Series Fabric Interconnect with UCS Central, see Considerations for migrating Cisco UCS 6200 Series Fabric Interconnects to Cisco UCS 6400 Series Fabric Interconnects with Cisco UCS Central.

Considerations for migrating Cisco UCS 6200 Series Fabric Interconnects to Cisco UCS 6400 Series Fabric Interconnects with Cisco UCS Central

To successfully migrate from Cisco UCS 6200 Series to Cisco UCS 6400 Series Fabric Interconnects using Cisco UCS Central, ensure the following prerequisites are met:

  • Backup Configurations: Complete a full backup of Cisco UCS Manager and UCS Central configurations before initiating the migration.

  • Chassis/FEX Discovery Policy

    • Set the Link Grouping Preference to PORT CHANNEL .

    • Set the Multicast Hardware Hash to DISABLED.

  • Dynamic vNIC Connection Policy: Ensure the Dynamic vNIC Connection Policy is not applied.

  • QOS System Class: Disable the QOS System Class.

  • Policy Resolution Control: To avoid any configuration issues during migration, ensure these policies are set to Local in Cisco UCS Manager and maintain this setting post-migration. Changing the settings from Local to Global after migration may result in configuration issues.

    • Infrastructure and Catalog Firmware Policy

    • Equipment Policy

    • Port Configuration Policy

  • Reserved VLAN IDs: Ensure the reserved VLAN IDs are not utilized to avoid any issues.


Note

For any additional information, see Cisco UCS 6400 Series Fabric Interconnect Migration Considerations.