Guest

Cisco UCS 6100 Series Fabric Interconnects

UCS 6140XP-40-port Fabric Interconnect Replacement or Upgrade for a UCS 6120XP-20-port Fabric Interconnect

Document ID: 111830

Updated: Feb 08, 2012

   Print

Introduction

This document describes the High Availability (HA) process for the installation of a Cisco Unified Computing System (UCS) 6140XP-40-port Fabric Interconnect Chassis as a replacement or upgrade for a Cisco UCS 6120XP-20-port Fabric Interconnect.

The UCS 6140XP Fabric Interconnect provides an integrated access layer for the many chassis of server blades that can be connected. The fabric interconnect also provides a single point of connectivity to Storage networks, Ethernet networks, and management networks.

Refer to Cisco UCS 6140XP Fabric Interconnect for more information.

Prerequisites

Requirements

Ensure that you meet these requirements before you attempt this configuration:

Components Used

The information in this document is based on these software and hardware versions:

  • Cisco UCS 6120XP and 6140XP Fabric Interconnects, Release version 4.1.3 N2 (1.1j)

  • Cisco UCS Manager, Release version 4.1.3 N2 (1.1j)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Related Products

This configuration can also be used with this hardware version:

  • Linux OS BSD installed on Cisco UCS M71KR-Q QLogic Converged Network Adapter

Conventions

Refer to the Cisco Technical Tips Conventions for more information on document conventions.

Background Information

IOM—This refers to the IO Module channel located on the Cisco USC 5108 Backplane Chassis where the 2104XP FEX inserts. There are four 10GbE links that extend from the 2104 FEX to the Cisco 6100XP Series Fabric Interconnect (FI). In general IOM-1 is the connection to Path A, which connects to the Primary (FI-A), and IOM-2 connects to Path B (FI-B).

FI —This term is an abbreviation for Fabric Interconnect and is connected to Fabric A/B

FEX—This is an abbreviation for Fabric Extender Cisco UCS 6100XP Series or 2104XP

HA—High Availability L1/L2-HA port(s) connections between two Cisco UCS 6100XP Series

GEM—Fiber Module used for SAN connectivity

Before you can perform these tasks you must have a valid Cisco 6100 Series HA License downloaded from the Download Software Center.

Cisco UCS has a Fabric based failover feature built into the system. Each adapter in UCS is a dual port adapter that connects to both the fabrics (A and B). The two fabrics in UCS provide failover in order to protect against planned or unplanned downtime of a component in one of the fabrics.

These are the two states when you cluster together Cisco 6100 Series Fabric Interconnects over their L1/L2 cable connections:

  • ACTIVE

  • SUBORDINATE

The management plane uses the ACTIVE state path to send Heartbeat signaling in order to monitor the health of the link and communicate with the UCS Manager for database synchronization. The SUBORDINATE Fabric Interconnect path takes over in the event of a failed link down of the Active Fabric Interconnect.

For IOM and Failover (FO), this is something that you must be aware of before you attempt an FO in order to replace a Cisco 6120XP with a Cisco 6140XP, which is also called Server Pinning. With Server interface to Fabric port pinning, there is a default association with Server ports to Fabric ports. A Server interface is defined as EthernetChassis/IOM/Slot and represents the Chassis, for example, 1, 2, 3, and so forth, of the UCS. The IOM-1 (Left), or IOM-2 (Right) are the slot insertion points into the Backplane Chassis for the 2104XP (FEX) and the Slot is the Blade slot of the Server Blade, such as Menlo or Palo Adapter Cards.

This means if you use the connect nxos command and perform a show running interface e1/1/8, the fabric interface says Eth1/4. A continued show running interface Eth1/4 shows that this interface is connected to FEX Fabric. Therefore, Blade 8 is pinned to the Ethernet port 1/4 of the 6100XP Fabric Interconnect.

Cisco FEX 6100 Series Fabric Interconnect only supports one, two, and four link topologies. If there is a link failure on one of the four links, UCS falls back to two links with regards to blade to fabric port mapping. The Cisco UCS FEX does not support three links, and falls back to a two link topology in case the link is down. In this case you need to re-acknowledge the chassis, which requires manual intervention in order to re-map the fabric ports.

It is recommended during the replacement of a Cisco 6100XP Series with another that you have all four ports connected for the Fabric Interconnect before the configuration sync. Refer to Cisco UCS 6100 Series Fabric Interconnect Hardware Installation Guide for more information.

Configure

This document shows the replacement of a Cisco 6120XP Fabric Interconnect with a Cisco UCS 6140XP Fabric Interconnect. In this lab there is a Cisco UCS Manager, two Dual 6120XP FI-A and FI-B connected to two 2104XP FEX IOM-1 and IOM-2, as shown in the Network Diagram, and Cisco 6100XP Series Lab Logical Topology Setup. The objective is to replace the Cisco UCS 6120XP-20-port FI-B with a UCS 6140XP-40-port into a FI-B topology. The FI-A 6120XP remains unaffected with this and still continues to pass traffic from the UCS Blade Server 5.

Complete these steps in order to replace a Cisco UCS 6120XP FI-B with a Cisco UCS 6140XP FI-B:

  1. Begin the configuration with Cisco UCS 6120XP HA connected Fabric Interconnects.

  2. Begin the configuration with Cisco UCS 6140XP HA connected Fabric Interconnect. Ensure that the new Cisco UCS 6140XP is running the same firmware version as that on 6120XP. If not, complete the steps mentioned in Firmware Upgrades in order to upgrade or downgrade the firmware.

  3. Connect to the Cisco UCS 6120XP FI-A, and verify the state of the cluster HA FI-B.

  4. Power down the SUBORDINATE FI-B in the HA Setup.

  5. Disconnect the four 10GbE cables on FI-B Cisco UCS 6120XP.

  6. Place the four 10GbE cables onto the corresponding ports FI-B Cisco UCS 6140XP.

  7. Disconnect the HA L1/L2 cables on the FI-B Cisco UCS 6120XP.

  8. Place the HA L1/L2 cables that were disconnected onto the FI-B Cisco UCS 6140XP.

  9. Disconnect the Uplink Ethernet cables FI-B.

  10. Place the Uplink Ethernet cables onto FI-B Cisco UCS 6140XP corresponding ports.

  11. Disconnect the Uplink FC cable(s) FI-B.

  12. Place the Uplink FC cables onto FI-B Cisco UCS 6140XP corresponding ports.

  13. Optional: Disconnect the Management Console Cisco UCS 6120XP FI-B and connect onto 6140XP FI-B console port.

  14. Remove the GEM (FC Module) from the Cisco UCS 6120XP FI-B.

  15. Place the GEM (FC Module) into the Cisco UCS 6140XP FI-B.

  16. Return to the FI-B Cisco UCS 6140XP, and start the configuration process.

  17. After you enter the GUI, wait for the configuration to synchronize Cisco UCS 6140XP FI-B with Cisco UCS 6120XP FI-A.

  18. Wait until HA is ready for Cisco UCS 6120XP FI-B with Cisco UCS 6120XP FI-A.

  19. The HA for Cisco UCS 6140XP FI-B connected to Cisco UCS 6120XP FI-A is now installed.

  20. Use the Cisco UCS Manager in order to view the new HA topology for FI-A and FI-B in Equipment > Main Topology View.

Network Diagram

This document uses this network setup:

Cisco 6100XP Series Lab Logical Topology Setup

fabric-interconnect-upgrade-01.gif

Configuration with Cisco UCS 6120XP HA connected Fabric Interconnects

  1. Connect to the Cisco UCS Manger http://ipaddress.

  2. Click the Equipment tab.

  3. Verify Main Topology View.

Main Topology View

fabric-interconnect-upgrade-02.gif

Verify the connectivity for Ethernet to IOM, L1/L2 ports, Fiber Connection and Operating System of FI-A

Complete these steps:

  1. Click the Equipment tab.

  2. Click the Fabric Interconnect A tab.

UCSM Fabric Interconnect A

fabric-interconnect-upgrade-03.gif

Verify the connectivity for Ethernet to IOM, L1/L2 ports, FC and OS of FI-B

Complete these steps:

  1. Click the Equipment tab.

  2. Click the Fabric Interconnect B tab.

UCSM Fabric Interconnect B

fabric-interconnect-upgrade-04.gif

Configure

UCS 6140XP Replacement for 6120XP FI-B

Configuration with Cisco UCS 6140XP HA connected Fabric Interconnect

Note: The new Cisco UCS 6140XP should be running the same firmware version as that on the Cisco UCS 6120XP. If not, complete the steps mentioned in Firmware Upgrades in order to upgrade or downgrade the firmware.

Complete these steps:

  1. Connect to the console port (RS232) of the UCS 6140XP.

  2. After you connect to the console port, issue the connect local-mgmt command in order to connect to the local-mgmt CLI.

  3. Issue the erase configuration command.

  4. Choose yes in order to Reboot.

  5. After the reboot, you are prompted with configuration method console/gui . Do not make any entry.

UCS 6140XP FI-B Preparation

6140XP- FI-B (local-mgmt) #erase configuration 
All UCS configurations will be erased and system will reboot. 
Are you sure? (yes/no):yes
Removing all the configuration. Please wait....
Configurations are cleaned up. Rebooting.... writing reset reason 9, 
Booting kickstart image: bootflash:/installables/switch/
ucs-6100-k9-kickstart.4.1.3.N2.1.1j.bin....
....................................................................
Image verification OK
Starting kernel...
Usage: init 0123456SsQqAaBbCcUu
INIT: version 2.85 booting
Starting Nexus5020 POST...
  Executing Mod 1 1 SEEPROM Test......done
  Executing Mod 1 1 GigE Port Test.....
         ---- Basic System Configuration Dialog ----
This setup utility will guide you through the basic configuration 
of the system.Only minimal 
configuration including IP connectivity to the Fabric interconnect 
and its clustering mode is performed through these steps.
Type Ctrl-C at any time to abort configuration and reboot system.
To back track or make modifications to already entered values, 
complete input till end of 
section and answer no when prompted to apply configuration.
Enter the configuration method. (console/gui)?

Connect to the Cisco UCS 6120XP FI-A and Verify the State of the Cluster HA FI-B

Complete these steps:

  1. Issue the connect local-mgmt command after you console or Secure Shelling (SSH) into the 6120XP FI-A.

  2. Issue the show cluster state command on the 6120XP FI-A.

This command gives you the output of the HA state, and the states are either A/B “Primary or Subordinate” or B/A “Primary or Subordinate”

6120XP-FI-A#show cluster state
Cluster Id: 0x3c29b5b8d32d11de-0xb75a000dec6dc084
B: UP, SUBORDINATE    This is the 6120XP FI-B 
A: UP, PRIMARY        This is the 6120XP FI-A
HA READY              This means HA 6120XP FI A/B is Active

Power down the “SUBORDINATE” FI-B in the HA Setup

  • Power down FI-B Cisco UCS 6120XP

    Note: You can either use the AC power switch located on the UCS 6120XP or remove the AC power connector cables shown in diagram 6.

Disconnect the Four 10GbE Cables on FI-B Cisco UCS 6120XP

Complete these steps:

  1. Disconnect the Ethernet cable(s).

  2. Replace the first four 10GbE cables connected downstream to the Cisco UCS 5108 FEX IOM.

    The four 10GbE cables (cable group) located on the 20 port Ethernet Connectors go to the UCS 2104 FEX.

Place the Four 10GbE Cables onto the Corresponding Ports FI-B Cisco UCS 6140XP

Complete these steps:

  1. Place the four Ethernet cable(s) onto the replacement 6140XP FI-B.

  2. Reconnect the cables removed from the Cisco UCS 6120XP FI-B, and connect them onto the Cisco UCS 6140XP FI-B.

Disconnect the HA L1/L2 Cables on the FI-B Cisco UCS 6120XP

Complete these steps:

  1. Remove the SAM cables L1/L2. This L1/L2 refers to ports not layers

    These are the L1/L2 HA cables.

  2. Remove the L1/L2 HA SAM Cluster cable connections from the Cisco UCS 6120XP FI-B.

    fabric-interconnect-upgrade-05.gif

    The previous diagram shows the SAM Cluster L1/L2 HA ports for Ethernet 10/100/1000.

Place the HA L1/L2 Cables Disconnected onto the FI-B Cisco UCS 6140XP

Complete these steps:

  1. Reconnect L1/L2 cables.

  2. Reconnect the L1/L2 HA FI-B cables onto Cisco UCS 6140XP FI-B.

  3. Reconnect onto SAM Cluster ports L1/L2, as shown in this diagram.

    fabric-interconnect-upgrade-06.gif

Disconnect the Uplink Ethernet cables FI-B

Disconnect Uplink Ethernet connector cable(s).

For this example, the Uplink cable was connected to port 20 of the Cisco UCS 6120XP FI-B and was disconnected.

Place the Uplink Ethernet Cables onto FI-B Cisco UCS 6140XP Corresponding Ports

Complete these steps:

  1. Places the cable(s) removed onto the 6140XP FI-B.

  2. Place the cable(s) on the Uplink corresponding ports. In this example, the Uplink cable is placed onto port 20 for the Cisco UCS 6140XP FI-B.

Disconnect the Uplink FC Cable(s) FI-B

Complete these steps:

  1. Disconnect Fiber Connectors for SAN Uplinks.

  2. Disconnect the Uplink FC port cable(s) that connects to the Cisco UCS 6120XP FI-B SAN Network and connect onto Cisco UCS 6140XP FI-B GEM.

    Note: These are the Uplink FC port cable(s) on the UCS 6120XP FI-B slot 2 GEM.

Place the Uplink FC Cables onto FI-B Cisco UCS 6140XP Corresponding Ports

Complete these steps:

  1. Place the SAN connectors onto the 6140XP GEM FI-B.

  2. Place the Uplink Fiber Connector cable(s) removed from 6120XP FI-B onto Cisco UCS 6140XP FI-B.

    Note: These steps are not necessary if the GEM is removed from the Cisco UCS 6120XP and reinserted into the Cisco UCS 6140XP FI-B. See Remove the GEM (FC Module) from the Cisco UCS 6120XP FI-B.

Optional: Disconnect the Management Console Cisco UCS 6120XP FI-B and Connect onto 6140XP FI-B Console Port

Remove the Management Ethernet connectors.

Note: The Management RS232 console is required for out-band management and has no impact on this HA processes. Reconnect the Management cables onto the Cisco UCS 6140XP FI-B after removal from UCS 6120XP FI-B.

Remove the GEM (FC Module) from the Cisco UCS 6120XP FI-B

Remove your GEM from Slot 2.

If you do not have a GEM located in slot 2, you need to remove the GEM that is in the Cisco UCS 6120XP FI-B for reinsertion onto 6140XP FI-B.

Place the GEM (FC Module) into the Cisco UCS 6140XP FI-B

Complete these steps:

  1. Replace the GEM into slot 2 6140XP FI-B.

  2. Insert the GEM into Slot 2 of the Cisco UCS 6140XP FI-B.

    Note: Only complete this step if there is no GEM in the Cisco UCS 6140XP FI-B.

Return to the FI-B Cisco UCS 6140XP and Start the Configuration Process

You are now ready to return to the configuration prompt of the 6140XP FI-B.

Return to the Cisco UCS 6140XP FI-B you physically cabled in the previous steps, which is now the FI-B.

  1. Assure you are still connected to the console port of the 6140XP FI-B.

  2. Enter "gui".

  3. After you enter “gui”, wait for the configuration to synchronize Cisco UCS 6140XP FI-B with Cisco UCS 6120XP FI-A.

    Proceed with the configuration of the Cisco UCS 6140XP FI-B.

    Enter the configuration method. (console/gui)? gui
     Switch can now be configured from GUI. 
    Use http://10.193.23.111 or https://10.193.23.111 and click
     on 'Express Setup' link. If you want to cancel the configuration 
    from GUI and go back,
    press the 'X' key. Press any other key to see the installation 
    progress from GUI
       *** Synchronizing_Images ***
      Configuration file – Ok
    

Wait until HA is Ready for Cisco UCS 6120XP FI-B with Cisco UCS 6120XP FI-A

Complete these steps:

  1. Connect to Cisco UCS Manager http://ipaddress of your Management Network.

  2. Click Express Setup .

  3. Click Enable clustering.

  4. Verify Switch Fabric is Fabric B.

  5. Optional: Under System setup, set the Admin Password of Master.

  6. Wait for the Install SUCCESS message to appear.

    When the Configuration file –OK message appears, the database is in sync, and you are ready to perform Express Configuration of the UCS Manager.

HTTP UCS Manager

fabric-interconnect-upgrade-07.gif

Initial Express Setup Dialog

fabric-interconnect-upgrade-08.gif

Management IP Addressing

fabric-interconnect-upgrade-09.gif

Install SUCCESS

fabric-interconnect-upgrade-10.gif

Verify

Verify from the Cisco UCS Manager that HA FI-A and FI-B is installed and the database is synchronized.

  1. Connect to the UCS Manger http://ipaddress.

  2. Click the Equipment tab.

  3. Look at the Main Topology View.

    This is the new Main Topology View for the Cisco UCS 6140XP FI-B now connected to 6120XP FI-A.

    fabric-interconnect-upgrade-11.gif

Verification with ICMP Ping

During the replacement process for the Cisco UCS 6120XP, FI-B ICMP runs on UCS Server Blade 5.

On a Cisco UCS Blade Sever, run ICMP to an Upstream Network for Connectivity validation.

  • Ping Snapshot while replacement of the Cisco UCS 6120XP FI-B is completed.

    fabric-interconnect-upgrade-12.gif

  • Ping Snapshot after replacement of the Cisco UCS 6120XP FI-B to 6140XP FI-B.

    fabric-interconnect-upgrade-13.gif

These ICMP Ping results show that no connectivity loss occurred during the replacement of the Cisco UCS 6120XP FI-B with an upgrade to a 6140XP FI-B.

Troubleshoot

Troubleshooting Commands

The Output Interpreter Tool (registered customers only) (OIT) supports certain show commands. Use the OIT to view an analysis of show

cluster state

cluster extended-state

Note: Refer to Important Information on Debug Commands before you use debug commands.

Related Information

Updated: Feb 08, 2012
Document ID: 111830