Installing the Cisco UCS X580p PCIe Node

This chapter contains the following topics:

Installation Guides and Limitations

Be aware of the following guidelines and limitations when installing the Cisco UCS X580p PCIe Node.

  • The Cisco UCS X580p PCIe Node connects with the Cisco UCS X9516 X-Fabric Module to provide PCIe Gen5 connectivity. However, Cisco also offers PCIe Gen4 connectivity through the Cisco UCS X440p PCIe Node and Cisco UCS X6416 X-Fabric Modules. You cannot mix these products within the same Cisco UCS X9508 chassis. One X9508 chassis can support either the Gen5 or Gen4 products, but not both.

  • The Cisco UCS X580p PCIe Node is supported in the Cisco UCS X9508 chassis only. Do not attempt to install the PCIe node in any other UCS server chassis.

  • The companion compute node's host power should be off before attempting any removal or insertion procedures. Hot removal or insertion of the PCIe node while the host is on is not supported.

  • When installing the Cisco UCS X580p PCIe Nodes and Cisco UCS X9516 X-Fabric Modules, make sure to install the XFMs before installing the PCIe nodes.

Recommended Configuration


Note

Each new Cisco UCS X9508 chassis will ship with the Cisco UCS X580p PCIe node(s) pre-installed. However, for existing deployments, the Cisco UCS X580p PCIe node can be shipped by itself. If you are installing PCIe nodes in an existing Cisco UCS X9508 deployment, make sure to install the PCIe nodes as documented below.

Each PCIe node must be paired with an X-Series M8 compute node, and therefore, has specific configurations based on the type of M8 compute node.

For single-slot M8 compute nodes (for example, the Cisco UCS X210c M8), the best practice is install PCIe and M8 compute nodes as shown.

In this configuration, the PCIe Nodes are installed in slots 3/4 and 7/8. The companion M8 compute nodes are installed in chassis slots 1/2 and 5/6. Be aware of these configurations, especially when removing and installing M8 compute nodes and PCIe nodes in your deployment.

Figure 1. Recommended Configuration

Supported Configuration

The following additional configuration is supported, but not recommended.

In this configurations, the PCIe node is installed in slots 1/2 and 5/6, and the M8 compute nodes are supported in slots 3/4 and 7/8.


Note

Although this configuration is supported, the best practice is to install PCIe and M8 compute nodes as shown in the Recommended Configuration.

Figure 2. Supported Configuration

PCIe Node Companion Module

The PCIe node must be installed to pair with one or two M8 compute nodes. The PCIe node and M8 compute node(s) work together to provide compute and GPU processing acceleration.

Each pairing of PCIe node and M8 compute node connect to a UCS X9516 X Fabric switching module at the rear of the Cisco UCS X9508 chassis. Both XFMs serve all nodes in the chassis, but the following PCIe mappings are based on the specific XFM.

  • XFM 1 (the top module) supports PCIe device mapping to CPU1 on the M8 compute node(s). On the PCIe node, PCIe slots 1 and 2 are supported by XFM 1 in the X9508 chassis and CPU1 on the companion M8 compute node.

  • XFM 2 (the bottom module) supports PCIe device mapping to CPU2 on the M8 compute node(s). On the PCIe node, PCIe slots 3 and 4 are supported by XFM 2 in the X9508 chassis and CPU2 on the companion M8 compute node.

When you install, replace, or upgrade the PCIe node, make sure that you install it next to a supported companion modules, such as a Cisco UCS X210c M8 compute node.


Note

The Cisco X-Fabric solution does not limit or restrict slot usage when PCIe nodes are not required. Cisco compute nodes can be installed in any slot without restriction when PCIe nodes are not used.

Replacing the Cisco UCS X580P PCIe Node

The Cisco UCS X580p PCIe Node is tool-less, field replaceable hardware that can be installed into the Cisco UCS X9508 chassis. You can install or remove the node by using the node's integrated ejectors.


Caution

Hot swap of the PCIe node is not supported while the host is on. The paired M8 compute node must be powered down first before removing or inserting the PCIe node.

For proper operation, if you remove a PCIe node and will not install a new one, you must cover the PCIe node slot with a filler panel. Do not operate the Cisco UCS X9508 chassis with any empty, uncovered module slot.

To replace the PCIe node, use the following tasks:

Installing the PCIe Node

Use this task to install the PCIe node.


Caution

Before inserting a PCIe node, make sure that its paired M8 compute node is powered off.

Before you begin

The PCIe node is a tools-free module that you install and secure to the chassis through ejectors.

Procedure

Step 1

Orient and align the PCIe node so that the ejectors are on the right vertical side of the slot.

Step 2

Using one hand to support the PCIe node, hold the node level and slide it into the chassis slot.

When the node is almost all the way in the slot, you might feel some resistance. This resistance is normal. It comes from the connector on the rear of the module meeting its socket on the inside of the chassis.

Step 3

Press on both ends of the faceplate to seat the module, then push the ejectors inward to secure the node in the slot.

The node is fully seated when the ejectors click into place and are flush with the node's faceplate.

Step 4

Using Cisco management software, such as Cisco Intersight, power on the PCIe node's paired M8 compute node.

Step 5

After the paired M8 compute node is completely powered on, use Cisco's management software, such as Cisco Intersight, to rediscover the PCIe node.

Removing the PCIe Node

After the M8 compute node and PCIe node are powered down, use this task to physically remove the PCIe node from the chassis.

Before you begin

You can remove each Cisco UCS X580p PCIe Node directly from the front of the chassis by using the node's ejectors. You do not need any tools to remove the node.


Note

The PCIe node can be heavy. Use your other hand to support the module while you are removing it.

Caution

The X580p PCIe Node itself is hot swappable, but the paired M8 compute node must be powered down first before hot swapping the PCIe node.

Before removing the PCIe node, you must properly decommission the M8 compute node(s) associated with the PCIe node. A brief summary follows, but make sure to follow the correct power off procedures as documented for your software management platform.

  • Using a Cisco management tool, such as Cisco Intersight, gracefully power down M8 compute nodes that may have been connected to the PCIe node.

  • Verify that all zones are deleted and associated M8 compute nodes are powered down. Doing so ensures that the GPUs are all powered off and can be safely removed and ensure no workloads are interrupted.

  • Verify that the PCIe node's Health Status LED is blinking green, which indicates that the PCIe node is safe to remove.

Procedure

Step 1

After the paired M8 compute node is completely powered off, push the PCIe node's ejector button (1) between the two ejectors to disconnect the ejectors from the sheet metal.

When you press the ejector button, the ejector handles will pop up slightly from the face of the node.

Step 2

Gently swing each ejector vertically to disconnect the PCIe node from the server (2).

Step 3

Using one hand to support the PCIe node, pull the node towards you to slide it out of the server chassis (3).

What to do next

Reinsert the PCIe node. See Installing the PCIe Node.

PCIe Node Configuration

The Cisco UCS X580p PCIe Node can be configured and managed using the Cisco Intersight management platform. For details, see the Cisco Intersight Managed Mode Configuration Guide, which is available at: Cisco Intersight Managed Mode Configuration Guide.