Implement Best Practices for Fabric Interconnect Migrations

Introduction

This document describes the best practices for Fabric Interconnects migrations.

Prerequisites

Requirements

Cisco recommends knowledge of these topics:

• Fabric Interconnects (FI)
• Cisco Unified Computing System (UCS)
• Virtual Local Area Network (VLAN) configuration
• Storage Area Network (SAN) configuration

Components Used

The information in this document is based on these software and hardware versions:

• UCS FI 6248UP
• UCS infrastructure version 4.1(3a)A
• Cisco Unified Computing System Manager (UCSM)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

During a Fabric Interconnect migration, there are some aspects that are needed to take into account, otherwise, some issues can arise and impact the process and time expected for this activity.

This document collects relevant information about the changes in the 6400 and 6500 Series Fabric interconnect and the best practices for a successful migration so it can be a smooth and efficient transition.

Fabric interconnect compatibility

The fabric interconnects migration can be only done in these combinations:

• UCS 6200 to UCS 6400 Series Fabric Interconnects.
• UCS 6200 or 6300 to UCS 6500 Series Fabric Interconnects.

Best practices

• Ensure that the hardware in your previous domain is compatible with the new Fabric interconnect you want to install.
• Verify the software compatibility, the upgrade order (either upgrade the old Fabric interconnect or downgrade the new Fabric interconnect), and the appropriate path. Cisco recommends to have installed a recommended release. In case your old Fabric interconnect is in a deferred version you can not downgrade the new Fabric interconnect to that version because deferred versions cannot be downloaded from Cisco Software Central .
• Ensure that the high availability is completely up.
• Review your configuration prior to the migration.

Software considerations

Cisco UCS 6400 and UCS 6500 Series Fabric Interconnects do not support a few software features that were supported on UCS 6200 and UCS 6300 Series Fabric Interconnects in Cisco UCS Manager.

Review this configuration features before proceeding with the migration:

Chassis Discovery Policy and Chassis Connectivity Policy

This policy needs to be set to Port Channel instead of None. You can use a port channel even with only one port. To apply the change you have to reacknowledge the chassis. In UCSM you can find that option in Equipment > Policies > Global Policies > Chassis/FEX Discovery Policy > Link Grouping Preference.

Chassis Discovery Policy location in UCSM

Port Configuration Mismatch

Take note that the port mapping from the 6200 and 6300 Series differs from the 6400 and 6500 Series Fabric Interconnects. Ensure that the previous port configuration is appropriately mapped to match the destination port disposition.

Multicast Hardware Hash

To balance the load effectively and avoid bandwidth issues, multicast hardware hashing is used. Enabling multicast hardware hashing allows all links between the IOM and the Fabric Interconnect in a port channel to handle multicast traffic. This feature is no longer available in the 6400 and 6500 series.

You can disable it under Equipment > Policies > Global Policies > Chassis/FEX Discovery Policy > Multicast Hardware Hash.

Multicast Hardware Hash setting location in UCSM

VLAN Port Count Optimization

Enabling VLAN port count optimization allows the consolidation of the status of multiple VLANs into a unified internal state. When VLAN port count optimization is activated, Cisco UCS Manager intelligently groups VLANs according to their port VLAN membership, thereby expanding the limit for port VLAN counts. 

On 6400 and 6500 FIs the port count optimization is done through port VLAN (VP) grouping when the PV is surpasses the 16000 limit. On this event, the Migration Warnings page informs that if the migration process continues the VP Grouping is enabled automatically. This table shows the PV count with VLAN port count optimization enabled and disabled on 6200, 6300, 6400, and 6500 Series FI.


Table 1. VLAN Port Count Optimization Enabled and Disabled on 6200, 6300, 6400, and 6500 Series Fabric Interconnects

Multicast Optimized for Quality of Service (QoS)

Multicast optimization is no longer available in 6400 and 6500 series FI. You can disable it in UCSM at LAN > QoS.

Multicast optimization on QoS policy in UCSM

NetFlow Configuration

NetFlow is a network protocol that collects and exports IP traffic data. It defines flows based on shared characteristics of IP packets, which are then sent to external NetFlow Collectors for analysis and application-specific processing. This has to be unconfigured when migrating to 6400 or 6500 Series FI.

In UCSM you can find the option to disable it in LAN > Netflow Monitoring > General. Do not forget to click Save Changes.

Netflow configuration in UCSM

VMM Integration

This feature is no longer available in 6400 and 6500 Series FI. If you proceed with the migration all configurations related to port profiles and Distributed Virtual Switches (DVSes) are automatically deleted.


Dynamic vNIC Connection Policy

This policy determine how the configuration between Virtual Machines (VMs) and virtual Network Interface Cards (vNICs) is configured. It is required for Cisco UCS domains that have servers with VIC adapters on which are VMs with dynamic vNICs configured. Dynamic vNICs are not supported on 6400 and 6500 Series FIs. If you proceed with the migration the policies and dynamic vNICs created are deleted.

Set the dynamic vNIC connection policy to not set. In UCSM you can find that setting under Server Profile > Network > vNICs > Modify > Connection Policies.

Dynamic vNIC Connection Policy in UCSM

Reserved VLANs

For 6400 and 650 Series FI the reserved VLANs are in the range of 3915 to 4042. Verify that your 6200 or 6300 Series FI do not have VLANs created within that range prior to the migration.

The Migration Warnings page reports if any configuration is deleted if the migration process continues.

Target Cisco UCS FI must be loaded with the same Infrastructure Firmware version on the FI that replaces. You can either downgrade the 6400 and 6500 Series FI firmware or upgrade the 6200 and 6300 Series FI. The downgrade/upgrade must respect the minimum firmware version.

UCS 6200 to UCS 6454 Fabric Interconnects Migration Details

The minimum UCSM version for 6454 FI is 4.0 and for 64108 FI is 4.1.

The Cisco UCS 6454 Fabric Interconnect supports:

• The fabric interconnect supports a maximum of 8 Fibre Channel over Ethernet (FCoE) port channels.

• Alternatively, it can support 4 SAN port channels.

• Another option is to have a maximum of 8 port channels that are a combination of SAN and FCoE (4 each).

• Ports 1-16 on the fabric interconnect support Unified Ports with options for 10/25 Gbps Ethernet, FCoE, or 8/16/32 Gbps Fibre Channel.

• Uplink Ports 49-54 on the fabric interconnect support 40/100 Gbps Ethernet or FCoE.

• The fabric interconnect is compatible with this IOMs: UCS 2204, UCS 2208, and UCS 2408.

• It is also compatible with this Fabric Extender (FEX) models: Cisco Nexus 2232PP and Cisco Nexus 2232TM-E.

• UCS 6454 Fabric Interconnects support only Port-Channel mode.

• Prior to Cisco UCS Manager Release 4.0(1), Ethernet or Fibre Channel (FC) switching modes were not supported on Cisco UCS 6454 Fabric Interconnects.

UCS 6200 & 6300 to UCS 6536 Fabric Interconnects Migration Details

The Cisco UCS Manager version 4.2(3b) introduces support for the 5th generation of Cisco UCS 6536 Fabric Interconnect (UCS FI 6536).

• The fabric interconnect has thirty-six QSFP28 ports, including 4 unified ports (33-36), with support for 100G.
• Ethernet ports (1-32) and unified ports (33-36) can be broken out into multiple ports using breakout cables.
• The Dynamic Ethernet Breakout feature allows on-the-fly conversion of a standard Ethernet port to a breakout port without rebooting the fabric interconnect. It also supports converting breakout ports back to standard Ethernet ports without a reboot.
• Unified Ports (33-36) can be configured as Fibre Channel breakout ports, but all four FC breakout ports must have the same speed.
• Breakout ports enable the Fabric Interconnect to support a maximum of 16 FC ports for the Fibre Channel.
• Changes to the breakout type from Ethernet to FC or vice versa require a reboot or reload of the fabric interconnect.
• FCoE storage ports are not supported.
• Ensure that all fan and power supply modules have the same airflow direction to prevent overheating and potential shutdown of the fabric interconnect.
• Cisco 22xx IOMs are not supported by 65xx fabric interconnects.

Common Issues

Port Configuration Mismatch

On UCS 6454 Fabric Interconnects, the Unified Port capability is restricted to the first 16 ports. Only ports 1/1-1/16 can be configured as FC. The FC ports must be contiguous, followed by contiguous Ethernet ports.

On UCS 6200 Series Fabric Interconnects, all ports have the Unified Port capability. All ports can be configured as Ethernet or FC. The Ethernet ports must be contiguous, followed by contiguous FC ports. FC ports appear toward the end of the module. During cluster addition, the ports that are mismatched are automatically unconfigured.

Reserved VLANs

UCS 6400 and 6500 Fabric Interconnects reserve more VLANs for internal use than UCS 6200 or 6300 Series Fabric Interconnects. During migration, the Migration Warnings page contains the list of VLANs that could potentially conflict with the default reserved VLAN range. If you proceed with migration, the Reserved VLAN range is configured but VLANs found in the conflicting range are not.

FC Uplink Ports Not Coming Up

The 6400 and 6500 use the IDLE fill pattern for FC uplinks and FC storage ports when the speed is 8Gbps. Make sure that the fill pattern is set as IDLE in the connected FC switch and the direct-attached FC ports. In case that configuration is not set these ports can go to an errDisabled state, lose SYNC intermittently, and receive errors or bad packets.

If the IDLE fill-pattern is not supported in your domain you can use a SAN switch between the FIs and the storage array or upgrade the storage array to have 16GB or 32 GB FC connectivity.

Related information

• Migrating from UCS 6200 to UCS 6454 Fabric Interconnects
• Migrating to Cisco UCS 6500 Series Fabric Interconnects
• Cisco Technical Support & Downloads