Replacing the Fabric Interconnect and Components

Replacing Fabric Interconnects

Preparing a Fabric Interconnect for Removal

Removing a subordinate (standby) fabric interconnect is non-disruptive in a redundant, high-availability Cisco UCS configuration. Removing a primary (active) fabric interconnect will cause the standby fabric interconnect to become active with minimal or no disruption.

When powering down and removing clustered fabric interconnects, remove the subordinate fabric interconnect first, and then remove the primary fabric interconnect.


Caution

Be aware that if you remove both the primary and subordinate fabric interconnects, or the single fabric interconnect from a standalone system, you are shutting down the entire Cisco UCS domain.

Procedure

Step 1

Use Cisco UCS Manager to perform the following tasks:

  1. Back up your Cisco UCS Manager configuration.

  2. Shut down the OS on all servers in the Cisco UCS domain.

  3. Disable the Smart Call Home feature in the Cisco UCS domain.

  4. Decommission every attached chassis in the Cisco UCS domain.

For details, see the Configuration Guide for the version of Cisco UCS Manager that you are using. The configuration guides are available at this URL: http://www.cisco.com/c/en/us/support/servers-unified-computing/ucs-manager/products-installation-and-configuration-guides-list.html.

Step 2

Power down every attached chassis as described in the Cisco UCS 5108 Server Chassis Hardware Installation Guide.

Shutting Down a Fabric Interconnect

Use the following procedure when you need to replace a single fabric interconnect with the same model of fabric interconnect.

Use Cisco UCS Manager, either the GUI or the CLI, to perform the software-related tasks mentioned in the following procedure. For additional information, refer to the Cisco UCS Manager Getting Started Guide, the Cisco UCS Manager Infrastructure Management Guide, and the instructional videos available at this URL: http://www.cisco.com/c/en/us/support/servers-unified-computing/ucs-manager/products-installation-and-configuration-guides-list.html

As a best practice, perform a full configuration backup before replacing the fabric interconnect.


Note

Verify that all vNICs are either redundant or that the fabric failover is enabled.

Procedure

Step 1

Contact Cisco TAC or your Cisco Sales representative to have them transfer the license to the replacement fabric interconnect.

Step 2

Label the ports and the cables that you using so that you can refer to this information later. The cabling and port numbering should be same for the replacement fabric interconnect.

Step 3

Log into UCS Manager and verify the state of the A and B fabric interconnects, either UP or DOWN. From the CLI, enter the show cluster extended-state command. High availability should be running in UCS Manager, as indicated by HA is READY.

Example:

FI-A# show cluster extended-state 
Cluster Id: 0x537d0580bf9911e0-0x8955000decd07984

A: UP, PRIMARY
B: UP, SUBORDINATE

A: memb state UP, lead state PRIMARY, mgmt services state: UP
B: memb state UP, lead state SUBORDINATE, mgmt services state: UP
   heartbeat state PRIMARY_OK 

INTERNAL NETWORK INTERFACES:
eth1, UP
eth2, UP

HA READY   <<<<<<<<<<<<<←---------------- HA is READY
Detailed state of the device selected for HA storage:
Chassis 1, serial: FOX1344G1R1, state: active
Chassis 2, serial: FOX1318GDKR, state: active
FI-A#

Step 4

Back up the software configuration.

Step 5

Use the fabric evacuation procedure on the subordinate fabric interconnect to ensure there is no data traffic impact during the hardware replacement.

Step 6

Power down the subordinate fabric interconnect by unplugging it from the power source.

Example:

FI-A# show cluster extended-state 
Cluster Id: 0x537d0580bf9911e0-0x8955000decd07984

A: UP, PRIMARY
B: DOWN, INAPPLICABLE

A: memb state UP, lead state PRIMARY, mgmt services state: UP
B: memb state DOWN, lead state INAPPLICABLE, mgmt services state: DOWN
   heartbeat state SECONDARY_FAILED

INTERNAL NETWORK INTERFACES:
eth1, DOWN
eth2, DOWN

HA NOT READY
Peer Fabric Interconnect is down
Detailed state of the device selected for HA storage:
Chassis 1, serial: FOX1344G1R1, state: active
Chassis 2, serial: FOX1318GDKR, state: active
FI-A#

Step 7

Remove the fabric interconnect from the rack. Follow the instructions in Removing a Cisco UCS Fabric Interconnect From a Rack.

Step 8

Install the replacement fabric interconnect to the rack. Follow the instructions in Installing the Cisco UCS FI 6454 in a Cabinet or Rack.

Step 9

Connect the management and console cables to the replacement fabric interconnect.

Step 10

Connect the L1/L2 cables that were disconnected to the replacement fabric interconnect.

Step 11

Connect the data cable according to the labels that you created in Step 2.

Step 12

Connect the power cable to the fabric interconnect and it will automatically boot and run POST tests. 

         ---- Basic System Configuration Dialog ----

  This setup utility will guide you through the basic configuration of
  the system. Only minimal configuration including IP connectivity to
  the Fabric interconnect and its clustering mode is performed through these steps.

  Type Ctrl-C at any time to abort configuration and reboot system.
  To back track or make modifications to already entered values,
  complete input till end of section and answer no when prompted
  to apply configuration.

  
  Enter the configuration method. (console/gui) ? console

  Installer has detected the presence of a peer Fabric interconnect. 
  This Fabric interconnect will be added to the cluster. Continue (y/n) ? y

  Enter the admin password of the peer Fabric interconnect: 
    Connecting to peer Fabric interconnect... done
    Retrieving config from peer Fabric interconnect... done
    Peer Fabric interconnect Mgmt0 IP Address: 122.255.252.2
    Peer Fabric interconnect Mgmt0 IP Netmask: 255.255.255.0
    Cluster IP address          : 122.255.252.1

  Physical Switch Mgmt0 IPv4 address : 122.255.252.3


  Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no): yes
  Applying configuration. Please wait.
Configuration file - Ok


Cisco UCS 6454 Fabric Interconnect

FI-B login:

Step 13

If necessary, upgrade the UCS Manager software. If the replacement fabric interconnect is not running the same firmware version as the cluster, the setup utility can upgrade the firmware.

Example:

---- Basic System Configuration Dialog ----

  This setup utility will guide you through the basic configuration of
  the system. Only minimal configuration including IP connectivity to
  the Fabric interconnect and its clustering mode is performed through 
  these steps.

  Type Ctrl-C at any time to abort configuration and reboot system.
  To back track or make modifications to already entered values,
  complete input till end of section and answer no when prompted
  to apply configuration.


  Enter the configuration method. (console/gui) ? console

  Installer has detected the presence of a peer Fabric interconnect. 
  This Fabric interconnect will be added to the cluster. Continue (y/n) ? y

  Enter the admin password of the peer Fabric interconnect:
    Connecting to peer Fabric interconnect... done
    Retrieving config from peer Fabric interconnect... done
    Installer has determined that the peer Fabric Interconnect is 
    running a different firmware version than the local Fabric. 
    Cannot join cluster.

    Local Fabric Interconnect
      UCSM version     : 3.1(2c)
      Kernel version   : 5.0(3)N2(3.12b)
      System version   : 5.0(3)N2(3.12b)
      local_model_no   : UCS-FI-6454

    Peer Fabric Interconnect
      UCSM version     : 3.1(2c)
      Kernel version   : 5.0(3)N2(3.12c)
      System version   : 5.0(3)N2(3.12c)
      peer_model_no    : UCS-FI-6454


  Do you wish to update firmware on this Fabric Interconnect to the 
  Peer's version? (y/n): y
  Updating firmware of Fabric Interconnect....... 
  [ Please don't press Ctrl+c while updating firmware ]

 Updating images
 Please wait for firmware update to complete.
 
 <output truncated>
 
 Install has been successful.
 Firmware Updation Successfully Completed. Please wait to enter the IP address
    Peer Fabric interconnect Mgmt0 IPv4 Address: xx.xx.xx.xx
    Peer Fabric interconnect Mgmt0 IPv4 Netmask: xx.xx.xx.xx
    Cluster IPv4 address          : xx.xx.xx.xx

    Peer FI is IPv4 Cluster enabled. Please Provide Local 
    Fabric Interconnect Mgmt0 IPv4 Address

  Physical Switch Mgmt0 IP address : xx.xx.xx.xx
  
  Apply and save the configuration (select 'no' if you want to re-enter)? 
  (yes/no): yes
  Applying configuration. Please wait.

  Fri Dec  9 19:57:15 UTC 2016
  Configuration file - Ok

Step 14

Enter the show cluster extended-state command to revalidate the cluster state and verify that HA is ready.

Example:

FI-A# show cluster extended-state 
Cluster Id: 0x537d0580bf9911e0-0x8955000decd07984

A: UP, PRIMARY
B: UP, SUBORDINATE

A: memb state UP, lead state PRIMARY, mgmt services state: UP
B: memb state UP, lead state SUBORDINATE, mgmt services state: UP
   heartbeat state PRIMARY_OK 

INTERNAL NETWORK INTERFACES:
eth1, UP
eth2, UP

HA READY   <<<<<<<<<<<<<←---------------- HA is READY
Detailed state of the device selected for HA storage:
Chassis 1, serial: FOX1344G1R1, state: active
Chassis 2, serial: FOX1318GDKR, state: active
FI-A#

Step 15

Disable fabric evacuation and verify that traffic is flowing in both fabric interconnects.

Step 16

Apply the port license to the new fabric interconnect.

Removing a Cisco UCS Fabric Interconnect From a Rack


Caution

Support both sides of the fabric interconnect when removing it from the rack. The slider rail and front rack-mount brackets do not have a stop mechanism. If the front of the chassis is unfastened from the rack and the chassis slides forward on the slider rails, it may slip off the end of the rails and fall out of the rack.

Procedure

Step 1

Ensure that the weight of the Cisco UCS Fabric Interconnect is fully supported and that the chassis is being held by another person.

Step 2

Remove the two screws holding the grounding cable to the chassis (if installed).

Step 3

Disconnect the power cords and any console cables.

Step 4

Disconnect all cables that are connected to SFP28 transceivers.

Step 5

Remove the screws fastening the front rack-mount brackets to the mounting rails.

Step 6

Gently slide the Cisco UCS Fabric Interconnect toward you, off of the slider rails and out of the rack.

Repacking the Cisco UCS Fabric Interconnect for Return Shipment

If you need to return the fabric interconnect, remove the fabric interconnect from the rack and repack it for shipment. If possible, use the original packing materials and container to repack the unit. Contact your Cisco customer service representative to arrange for return shipment to Cisco.

Replacing Components

Replacing a Fan Module

You can replace a fan module while the switch is operating, as long as you perform the replacement within one minute. If you cannot perform the replacement within one minute, leave the original fan module in the chassis to maintain the designed airflow until you have the replacement fan module on hand and can perform the replacement.


Caution

If you are replacing a module during operations, verify the replacement fan module has the correct direction of airflow. This means that it has the same airflow direction as the other modules in the chassis. Also, verify that the airflow direction takes in air from a cold aisle and exhausts air to a hot aisle. Otherwise, the switch can overheat and shutdown.

If you are changing the airflow direction of all the modules in the chassis, shutdown the switch before replacing all the fan and power supply modules with modules using the other airflow direction. During operations, all of the modules must have the same direction of airflow.

Removing a Fan Module


Caution

The fans might still be turning when you remove the fan assembly from the chassis. Keep fingers, screwdrivers, and other objects away from the openings in the fan assembly's housing.

Procedure

Step 1

On the fan module that you are removing, press the two sides of the fan module handle together, and pull on the handles enough to unseat it from its connectors.

Step 2

Holding the handle, pull the module out of the chassis.

Caution

 

Do not touch the electrical connectors on the back side of the module and prevent anything else from coming into contact with and damaging the connectors.

Installing a Fan Module

Before you begin

  • A fan slot must be open and ready for the new fan module to be installed.

  • If the switch is operating, you must have a new fan module on hand and ready to install within one minute of removing the original fan module.

  • The new fan module must have the same airflow direction as the other fan and power supply modules installed in the switch.

Procedure

Step 1

Holding the fan module by its handle, align the back of the fan module (the side with the electrical connectors) to the open fan slot in the chassis.

Step 2

Slide the fan module into the slot until it clicks in place.

Step 3

Verify that the Status (STS) LED turns on and becomes green.

Replacing a Power Supply Module

The switch requires two power supplies for redundancy. With one power supply providing the necessary power for operations, replace the other power supply during operations as long as the new power supply has the same airflow direction as the other modules in the chassis.

Replace a power supply with another supported power supply that has the same power source type as the other installed power supply. Additionally, the airflow direction of the power supply must match or conform to the airflow direction of the installed fan modules. For the airflow direction used by the switch, see the coloring of the fan modules.

Replacing an AC Power Supply

Replace an AC power supply during operations as long as the other power supply provides power to the switch.

Before you begin


Note

Determine the airflow direction by looking at the coloring of the latch on each power supply. AC power supplies with burgundy latches have port-side intake airflow direction AC power supplies with blue latches have port-side exhaust airflow direction.

  • The replacement power supply must have the same wattage and airflow direction as the power supply being replaced. Do not mix AC, DC, HVAC/HVDC power supplies in the same switch.

  • An AC power source must be within reach of the power cable that will be used with the replacement power supply. If you are using n+n power redundancy, there must be a separate power source for each power supply installed in the chassis.

  • There must be an earth ground connection to the chassis that you are installing the replacement module. AC power supplies connected to AC power sources are automatically grounded through their power cable.

Procedure

Step 1

Remove an AC power supply.

  1. Holding the plug for the power cable, pull the plug out from the power receptacle on the power supply and verify that both power supply LEDs are off.

  2. Grasp the power supply handle while pressing the colored release latch towards the power supply handle.

  3. Place your other hand under the power supply to support it while you slide it out of the chassis.

    Note

     

    Do not touch the electrical connections on the back side of the module and prevent anything else from coming into contact with and damaging the connectors.

Step 2

Install the replacement power supply.

  1. Holding the replacement power supply with one hand underneath the module and the other hand holding the handle, turn the power supply so that its release latch is on the right side and align the back end of the power supply (the end with the electrical connections) to the open power supply slot before carefully sliding the power supply all the way into the slot until it clicks into place.

    Note

     

    If the power supply does not fit into the open slot, turn the module over before sliding it carefully into the open slot.

  2. Test the installation by trying to pull the power supply out of the slot without using the release latch.

    If the power supply does not move out of place, it is secured in the slot. If the power supply moves, carefully press it all the way into the slot until it clicks in place.

  3. Attach the power cable to the electrical outlet on the front of the power supply.

  4. Make sure that the other end of the power cable is attached to the appropriate power source for the power supply.

    Note

     

    Depending on the outlet receptacle on your power distribution unit, you might need the optional jumper cable to connect the switch to your outlet receptacle.

  5. Verify that the power supply is operational by making sure that the power supply LED is green.

Replacing a High Voltage (HVAC/HVDC) Power Supply

Replace an HVAC/HVDC power supply during operations as long as the other power supply provides power to the switch.

Before you begin

  • The replacement power supply must have the same wattage and airflow direction as the power supply being replaced. Do not mix AC, DC, HVAC/HVDC power supplies in the same switch.


    Note

    Determine the airflow direction by looking at the coloring of the latch on each power supply. HVAC/HVDC power supplies have white latches. White latches indicate that they have dual airflow directional capability. Those power supplies automatically use the same airflow direction as the fan modules installed in the same switch.

  • An HVAC/HVDC power source must be within reach of the power cable that will be used with the replacement power supply. If you are using n+n power redundancy, there must be a separate power source for each power supply installed in the chassis.

  • There must be an earth ground connection to the chassis in which you are installing the replacement power supply. HVAC/HVDC power supplies connected to AC power sources are automatically grounded by their power cable when connected to the power supply and AC power source. HVAC/HVDC power supplies connected to DC power sources have Saf-D-Grid power cables with three connectors on the power source end--you connect one of those connectors to the earth ground.

Procedure

Step 1

Remove an HVAC/HVDC power supply.

  1. Turn off the circuit breaker for the power feed to the power supply that you are replacing.

    Be sure that the LEDs turn off on the power supply that you are removing.

  2. Remove the power cable from the power supply by pressing the tab on the top of the Anderson Power SAF-D-Grid connector and pull the cable and connector out of the power supply.

  3. Grasp the power supply handle while pressing the colored release latch towards the power supply handle.

  4. Place your other hand under the power supply to support it while you slide it out of the chassis.

    Caution

     

    Do not touch the electrical connections on the back side of the module and prevent anything else from coming into contact with and damaging the connectors.

Step 2

Install the replacement power supply.

  1. Holding the replacement power supply with one hand underneath the module and the other hand holding the handle, turn the power supply so that its release latch is on the right side and align the back end of the power supply (the end with the electrical connections) to the open power supply slot before carefully sliding the power supply all the way into the slot until it clicks into place.

    Note

     

    If the power supply does not fit into the open slot, turn the module over before sliding it carefully into the open slot.

  2. Test the installation by trying to pull the power supply out of the slot without using the release latch.

    If the power supply does not move out of place, it is secured in the slot. If the power supply moves, carefully press it all the way into the slot until it clicks in place.

  3. Attach the Saf-D-Grid end of the power cable to the electrical outlet on the front of the power supply.

  4. Make sure that the other end of the power cable is attached to the appropriate power source for the power supply.

    • For an HVAC power source, plug the other end of the power cable into the power source.

    • For a HVDC power source, verify that the circuit breaker is turned off and then connect each of the three cable connectors to the appropriate DC and grounding terminals on the power source. If there is a cover plate for the DC terminals, install the plate to prevent accidental contact with the terminals.

  5. If using an HVDC power source, turn on the circuit breaker for the power source.

  6. Verify that the power supply is operational by making sure that the power supply LED is green.

Replacing a DC Power Supply

Replace a DC power supply during operations so long as the other power supply provides power to the switch.

Before you begin

  • Replace a power supply with the same wattage and airflow direction as the power supply being replaced. Do not mix AC, DC, HVAC/HVDC power supplies in the same switch.


    Note

    Determine the airflow direction by looking at the coloring of the latch on each power supply. AC power supplies with burgundy latches have port-side intake airflow direction. AC power supplies with blue latches have port-side exhaust airflow direction.

  • A DC power source must be within reach of the power cables that will be used with the replacement power supply. If you are using n+n power redundancy, there must be a separate power source for each power supply installed in the chassis.

  • There must be an earth ground connection to the chassis in which you are installing the replacement power supply. If the switch is mounted properly in a grounded rack then there is no need to ground the PSU. Some DC power supplies connected to DC power sources have three power cables (two for DC power and one for grounding). Some, like the (NXA-PDC-1100W) only have two power cables (both for DC power).

  • We recommend 8-AWG wire for DC installation in the U.S.

  • All DC power supplies have reverse polarity protection. When you inadvertently connect the input power (+) to the DC PSU’s – terminal and the input power – to the DC PSU’s (+) terminal, the PSU will not be damaged and will operate fine after the input power feeds are correctly wired.

Procedure

Step 1

Remove a DC power supply.

  1. Turn off the circuit breaker for the power feed to the power supply that you are replacing.

    Be sure that the LEDs turn off on the power supply that you are removing.

  2. Remove the DC power connector block from the power supply.

    1. Push the orange plastic button on the top of the connector block inward toward the power supply.

    2. Pull the connector block out of the power supply.

  3. Grasp the power supply handle while pressing the release latch towards the power supply handle.

  4. Place your other hand under the power supply to support it while you slide it out of the chassis.

    Caution

     

    Do not touch the electrical connections on the back side of the module and prevent anything else from coming into contact with and damaging the connectors.

Step 2

Install the replacement power supply.

  1. Holding the replacement power supply with one hand underneath the module and the other hand holding the handle, turn the power supply so that its release latch is on the right side and align the back end of the power supply (the end with the electrical connections) to the open power supply slot before carefully sliding the power supply all the way into the slot until it clicks into place.

    Note

     

    If the power supply does not fit into the open slot, turn the module over before sliding it carefully into the open slot.

  2. Test the installation by trying to pull the power supply out of the slot without using the release latch.

    If the power supply does not move out of place, it is secured in the slot. If the power supply moves, carefully press it all the way into the slot until it clicks in place.

  3. Attach the power connector block end of the power cable to the electrical outlet on the front of the power supply.

  4. Turn on the circuit breaker for the power source.

  5. Verify that the power supply is operational by making sure that the power supply LED is green.