Cisco UCS C460 Server Installation and Service Guide

Cisco Integrated Management Interface (CIMC)

You can monitor the server inventory, health, and system event logs by using the built-in Cisco Integrated Management Controller (CIMC) GUI or CLI interfaces. See the user documentation for your firmware release at the following URL:

http://www.cisco.com/en/US/products/ps10739/products_installation_and_configuration_guides_list.html

Server Configuration Utility

Cisco has also developed the Cisco Server Configuration Utility for C-Series servers, which can aid and simplify the following tasks:

• Monitoring server inventory and health
• Diagnosing common server problems with diagnostic tools and logs
• Setting the BIOS booting order
• Configuring some RAID configurations
• Installing operating systems

This utility is shipped with new servers on CD. You can also download the ISO from Cisco.com. See the user documentation for this utility at the following URL:

http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/ucsscu/user/guide/20/SCUUG20.html

Front Panel LEDs

Figure 3-1 shows the front panel LEDs.

Figure 3-1 Front Panel LEDs

Operations Panel LEDs and Buttons

Figure 3-2 shows the operations panel LEDs and buttons.

Figure 3-2 Operations Panel LEDs and Buttons

Rear Panel LEDs and Buttons

Figure 3-3 shows the rear panel LEDs and buttons.

Figure 3-3 Rear Panel LEDs and Buttons

Required Equipment

The following equipment is used to perform the procedures in this chapter:

• Number 2 Phillips-head screwdriver
• Number 1 Phillips-head screwdriver
• Needle-nose pliers
• Electrostatic discharge (ESD) strap or other grounding equipment such as a grounded mat

Shutting Down and Powering Off the Server

The server can run in two power modes:

• Main power mode—Power is supplied to all server components and any operating system on your hard drives can run.
• Standby power mode—Power is supplied only to the service processor and the cooling fans and it is safe to power off the server from this mode.

You can invoke a graceful shutdown or an emergency shutdown (hard shutdown) by using either of the following methods:

• Use the CIMC management interface.
• Use the Power button on the server front panel. To use the Power button, follow these steps:

Step 1 Check the color of the Power Status LED (see the “Operations Panel LEDs and Buttons” section).

• Green indicates that the server is in main power mode and must be shut down before it can be safely powered off. Go to Step 2.
• Off indicates that the server is already powered off or is in standby mode and can be safely powered off. Go to Step 3.

Step 2 Invoke either a graceful shutdown or a hard shutdown:

Caution To avoid data loss or damage to your operating system, you should always invoke a graceful shutdown of the operating system.

---

• Graceful shutdown—Press and release the Power button. The operating system will perform a graceful shutdown and the server goes to standby mode, which is indicated by the Power Status LED being off.
• Emergency shutdown—Press and hold the Power button for 4 seconds to force the main power off and immediately enter standby mode.

Step 3 Disconnect the power cords from the power supplies in your server to completely power off the server.

Removing and Replacing the Server Top Cover

Use the following procedure to remove or replace the top cover of the server:

Tip You do not have to remove the cover to replace hard drives or power supplies.

Step 1 Remove the top cover:

a. Simultaneously press the two green release buttons. See Figure 3-4.

b. Push the cover toward the server rear about three inches, until it stops.

c. Lift the cover straight up from the server and set it aside.

Step 2 Replace the top cover:

a. Place the cover on top of the server about 3 inches behind the front of the chassis. The cover should sit flat when the cover flanges are sitting in the grooves in the chassis.

b. Slide the cover toward the front of the server until it stops at the front panel and the green buttons lock.

Figure 3-4 Removing the Top Cover

Replaceable Component Locations

This section shows the locations of the components that are discussed in this chapter. The view in Figure 3-5 is from the top down, with the top cover and internal CPU cage removed.

Figure 3-5 Replaceable Component Locations

Replacing Power Supplies

The server can have two or four power supplies. Four power supplies are required for four-CPU configurations.

The qualified and supported part numbers for this component are subject to change over time. For the most up-to-date list of replaceable components, see the following URL and then scroll to Technical Specifications :

http://www.cisco.com/en/US/products/ps10493/products_data_sheets_list.html

This section contains the following topics:

• Replacement Procedure
• Power Supply Cold Redundancy

Replacement Procedure

To replace or install a power supply, follow these steps:

Note If you have ordered a server with power supply redundancy (at least four power supplies), you do not have to power off the server to replace power supplies. Also see the note below about 1+1 redundancy.

Step 1 Remove the power supply that you are replacing or a blank panel from an empty bay. See Figure 3-6:

a. Do one of the following actions:

– If your server has only two power supplies, shut down and power off the server as described in the “Shutting Down and Powering Off the Server” section. See the following note.

Note You can use 2 power supplies with a 2-CPU configuration. In this case, 1+1 redundancy is supported only if your server’s overall power consumption can be supported by a single 850W power supply during the failure and hot-swap. For more information about your server’s power consumption, consult with your Cisco sales representative or use the power calculator accessible at the Unified Computing System Partner Resource Center:

http://www.ciscoprc.com/resourcelib.asp?id=937

– If your server has four power supplies, you do not have to shut down the server.

b. Remove the power cord from the power supply that you are replacing.

a. Grasp the power supply handle while squeezing the release lever towards the handle.

b. Pull the power supply out of the slot.

Step 2 Install a new power supply:

a. Grasp the power supply handle and insert the power supply into the power supply bay.

b. Push the power supply into the bay until the release lever locks.

c. Replace the power cord to the new power supply.

d. Press the Power button to return the server to main power mode.

Figure 3-6 Removing and Replacing Power Supplies

Power Supply Cold Redundancy

Depending on the power being drawn by the server, one or more power supplies might actively provide all power to the system while the remaining power supplies are put into a standby state. This is known as cold redundancy.

The server’s CIMC interface reports the power supply redundancy based on the number of supplies that are plugged in with AC power applied. For example:

• 2 PSUs: Redundancy lost
• 3 PSUs: Redundancy degraded
• 4 PSUs: Full redundancy

With cold redundancy, depending on the server’s power usage, one or more supplies might actively provide all power to the system while the remaining supplies are put into a standby state. For example, if you have two supplies connected to AC power, but the power consumption can be satisfied by power supply 1, then power supply 2 is put into a standby state.

Replacing the I/O Riser

The I/O riser is module that connects to the motherboard and that provides the ports for rear panel connectivity. To install or replace the I/O riser, follow these steps:

Step 1 Remove the I/O riser you are replacing. See Figure 3-7:

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Disconnect all cables from the ports on the I/O riser.

Tip Label the cables when you remove them to aid in identifying them for replacement.

c. Slide the server out the front of the rack far enough so that you can remove the top cover.

Caution If you cannot safely view and access the component, remove the server from the rack.

---

d. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

e. Pinch and lift up the green retaining clip that secures the I/O riser to the chassis rear panel.

f. Lift the riser straight up from the motherboard connector and out of the chassis.

Step 2 Install a new I/O riser:

a. Align the I/O riser with the empty I/O riser connector on the motherboard.

b. Push down evenly on both ends of the I/O riser until it is fully seated in the motherboard connector.

c. Ensure that the I/O riser rear panel sits flat against the chassis back panel opening.

d. Push the green retaining clip down until it locks over the top of the I/O riser.

e. Replace the top cover.

f. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Figure 3-7 Removing and Replacing the I/O Riser

Replacing an Internal Flash eUSB Drive

The eUSB drive is a small solid-state flash drive card that connects to the motherboard in either of two eUSB drive connectors.

The qualified and supported part numbers for this component are subject to change over time. For the most up-to-date list of replaceable components, see the following URL and then scroll to Technical Specifications :

http://www.cisco.com/en/US/products/ps10493/products_data_sheets_list.html

This section contains the following topics:

• Replacement Procedure
• Enabling or Disabling the Internal USB Port

Replacement Procedure

To install or replace the eUSB drive, follow these steps:

Step 1 Remove the eUSB drive you are replacing. See Figure 3-8:

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover.

Note You might have to detach cables from the rear panel to provide clearance.

Caution If you cannot safely view and access the component, remove the server from the rack.

---

c. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

d. Remove the I/O riser to provide clearance. See Replacing the I/O Riser.

e. Locate the eUSB drive and remove the single screw that secures it to the motherboard standoff.

f. Lift straight up on both ends the eUSB drive to disengage it from the motherboard connector.

Step 2 Install a new eUSB drive.

a. Align the connector on the underside of the eUSB drive with the empty drive connector on the motherboard and push on both ends of the drive evenly to seat the connector.

b. Replace the single mounting screw that secures the eUSB drive to the motherboard standoff.

c. Replace the I/O riser. See Replacing the I/O Riser.

d. Replace the top cover.

e. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Figure 3-8 Removing and Replacing the eUSB Drive

Enabling or Disabling the Internal USB Port

The factory default is for all USB ports on the server to be enabled. However, the internal USB port can be enabled or disabled in the server BIOS. To enable or disable the internal USB port, follow these steps:

Step 1 Enter the BIOS Setup utility by pressing the F2 key when prompted during bootup.

Step 2 Navigate to the Advanced tab.

Step 3 On the Advanced tab, select USB Configuration.

Step 4 On the USB Configuration page, select USB Ports Configuration.

Step 5 Scroll to USB Port: Internal, press Enter, and then select either Enabled or Disabled from the pop-up menu.

Step 6 Press F10 to save and exit the utility.

Replacing a SAS Riser (RAID Controller)

The SAS riser is a RAID controller card that has a designated position and motherboard connector inside the server.

RAID Card Firmware Compatibility

Firmware on the RAID controller must be verified for compatibility with the current Cisco IMC and BIOS versions that are installed on the server. If not compatible, upgrade or downgrade the RAID controller firmware accordingly using the Host Upgrade Utility (HUU) for your firmware release to bring it to a compatible level.

See the HUU guide for your Cisco IMC release for instructions on downloading and using the utility to bring server components to compatible levels: HUU Guides

Replacement Procedure

Step 1 Remove the SAS riser you are replacing. See Figure 3-9:

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover.

Note You might have to detach cables from the rear panel to provide clearance.

Caution If you cannot safely view and access the component, remove the server from the rack.

---

c. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

d. Pinch and lift up the green retaining clip that secures the SAS riser to the chassis rear panel.

e. Lift the SAS riser straight up from the motherboard connector.

Note Lift up on both ends of the card evenly to avoid damaging its connector.

f. Disconnect the SAS cables and any battery backup unit (BBU) cable from the SAS riser. See Figure 3-10.

Tip Label the SAS cables when you disconnect them to aid correct connection to the new SAS riser.

Step 2 Install a new SAS riser:

a. Connect the SAS cables and BBU cable to the new SAS riser. See Figure 3-10.

b. Align the SAS riser with the empty SAS riser connector on the motherboard.

c. Push down evenly on both ends of the SAS riser until it is fully seated in the motherboard connector.

d. Ensure that the SAS riser rear panel sits flat against the server back panel opening.

e. Push the green retaining clip down until it locks over the top of the SAS riser.

f. Replace the top cover.

g. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Step 3 Restore the RAID configuration on your drives to the new mass storage controller.

See Restoring RAID Configuration After Replacing a RAID Controller.

Figure 3-9 Removing and Replacing a SAS Riser

Figure 3-10 SAS Riser Card Connectors

Replacing the SAS Riser Battery Backup Unit

Note This optional battery backup unit (BBU) is available only when using the optional LSI 9260-8i SAS RAID controller card as the SAS riser. This BBU provides approximately 72 hours of battery backup for the disk write-back cache DRAM in the case of sudden power loss.

Note LSI recommends that you replace the LSI BBU once per year or after 1,000 recharge cycles, whichever comes first. Verify whether BBU replacement is required by looking in the CIMC. Log in to CIMC for the server, then click Server—Inventory—Storage—Battery Backup Unit. If the Battery Replacement Required field says, “True,” then you must purchase a replacement BBU and replace it.

Warning There is danger of explosion if the battery is replaced incorrectly. Replace the battery only with the same or equivalent type recommended by the manufacturer. Dispose of used batteries according to the manufacturer’s instructions.
Statement 1015

The qualified and supported part numbers for this component are subject to change over time. For the most up-to-date list of replaceable components, see the following URL and then scroll to Technical Specifications :

http://www.cisco.com/en/US/products/ps10493/products_data_sheets_list.html

To replace the SAS riser BBU, follow these steps:

Step 1 Remove the BBU that you are replacing. See Figure 3-11.

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover.

Note You might have to detach cables from the rear panel to provide clearance.

Caution If you cannot safely view and access the component, remove the server from the rack.

---

c. Remove the top cover as described in the “Removing and Replacing the Server Top Cover” section.

d. Remove the memory riser that sits closest to the BBU. See Replacing Memory Risers.

e. Slide the battery assembly toward the front of the chassis to disengage its retaining clips from the chassis wall slots.

f. Disconnect the cable that is attached to the BBU.

Step 2 Install a new BBU:

Note When you install the cable from the LSIBBU06, LSIBBU07 or LSIBBU08 BBU (varying capacities) to the LSI 9260-8i RAID controller, make sure to align the red dot on the cable with the red dot on the connector that is on the card to ensure correct polarity.

a. Connect the cable from the SAS riser to the replacement BBU.

b. Insert the two retaining clips on the rear of the BBU into the chassis wall slots and slide the BBU toward the chassis rear until it locks into place.

c. Replace the memory riser that you removed for clearance. See Replacing Memory Risers.

d. Replace the top cover.

e. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Figure 3-11 Removing and Replacing the RAID Battery Assembly

Replacing the Motherboard CMOS Battery

Warning There is danger of explosion if the battery is replaced incorrectly. Replace the battery only with the same or equivalent type recommended by the manufacturer. Dispose of used batteries according to the manufacturer’s instructions.
Statement 1015

The CMOS battery retains system settings when the server is disconnected from power.

The qualified and supported part numbers for this component are subject to change over time. For the most up-to-date list of replaceable components, see the following URL and then scroll to Technical Specifications :

http://www.cisco.com/en/US/products/ps10493/products_data_sheets_list.html

To replace or install the motherboard CMOS battery, follow these steps:

Step 1 Remove the CMOS battery. See Figure 3-12:

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover.

Note You might have to detach cables from the rear panel to provide clearance.

Caution If you cannot safely view and access the component, remove the server from the rack.

---

c. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

d. Remove the I/O riser to provide clearance. See Replacing the I/O Riser.

e. Locate the CMOS battery.

f. Bend the battery retaining clip away from the chassis wall and pull the battery from the socket.

Step 2 Install a CMOS battery:

a. Bend the retaining clip away from the chassis wall and insert the battery in the socket.

Note The positive side of the battery marked “+” should face the chassis wall.

b. Push the battery into the socket until it is fully seated.

Note Ensure that the retaining clip clicks over the top of the battery.

c. Replace the I/O riser. See Replacing the I/O Riser.

d. Replace the top cover.

e. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Figure 3-12 Removing and Replacing the Motherboard CMOS Battery

Replacing CPUs and Heatsinks

This server can operate with 2-, 3-, or 4-CPU configurations. Each CPU supports two memory risers (four memory buffers) connected by serial memory interface (SMI). Each memory buffer has two DDR3 memory bus interfaces.

The qualified and supported part numbers for this component are subject to change over time. For the most up-to-date list of replaceable components, see the following URL and then scroll to Technical Specifications :

http://www.cisco.com/en/US/products/ps10493/products_data_sheets_list.html

Note The minimum CPU configuration is that CPU1 and CPU3 must be installed. See Figure 3-13, which has a view shown facing the front of the server. Only CPU1 and CPU3 are connected to the internal PCIe hub. With CPU1 and CPU3 installed, any other combination operates.

Figure 3-13 CPUs and Memory Risers

This section contains the following topics:

• Additional CPU-Related Parts To Order With RMA Replacement Motherboards
• CPU Replacement Procedure

Additional CPU-Related Parts To Order With RMA Replacement Motherboards

When a return material authorization (RMA) of the motherboard or CPU is done on a Cisco UCS C-series server, there are additional parts that might not be included with the CPU or motherboard spare bill of materials (BOM). The TAC engineer might need to add the additional parts to the RMA to help ensure a successful replacement.

• Scenario 1—You are re-using the existing heatsinks:

– Heat sink cleaning kit (UCSX-HSCK=)

– Thermal interface pad for C460 (RC-460-TIM=)

• Scenario 2—You are replacing the existing heatsinks:

– Heat sink (RC460-BHTS1=)

– Heat sink cleaning kit (UCSX-HSCK=)

A CPU heatsink cleaning kit is good for up to four CPU and heatsink cleanings. The cleaning kit contains two bottles of solution, one to clean the CPU and heatsink of old thermal interface material and the other to prepare the surface of the heatsink.

New heatsink spares have preinstalled thermal interface material covered by a small sheet of plastic. It is important to clean the old thermal interface material off of the CPU prior to installing the heatsinks. Therefore, when ordering new heatsinks it is still necessary to order the heatsink cleaning kit at a minimum.

CPU Replacement Procedure

To install or replace a CPU heatsink and CPU, follow these steps:

Step 1 Remove the CPU and heatsink that you are replacing:

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover.

Note You might have to detach cables from the rear panel to provide clearance.

Caution If you cannot safely view and access the component, remove the server from the rack.

---

c. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

d. Remove all memory risers to uncover the CPU cage screws. See Replacing Memory Risers.

e. Remove all the memory riser dividers by lifting them straight up and sliding them free of the CPU cage. See Figure 3-14.

f. Remove the CPU cage by loosening the six captive screws that secure the cage to the motherboard. See Figure 3-14 for the screw locations.

g. Loosen the two captive screws that secure the heatsink and lift it off of the CPU. See Figure 3-14 and Figure 3-15.

Note Alternate loosening each screw evenly to avoid damaging the heatsink or CPU.

h. Unclip the CPU retaining lever and lift the CPU retaining lid. See Figure 3-15.

i. Lift the CPU out of the socket and set it aside on an antistatic mat or in an antistatic bag.

Step 2 Install a new CPU:

a. Insert the replacement CPU in the socket with the arrow on the CPU pointing toward the arrow on the socket.

b. Close the CPU retaining lid and clip down the CPU retaining lever.

Step 3 Install a heatsink:

Caution The heatsink must have a new, undamaged thermal pad on the heatsink-to-CPU surface to ensure proper cooling. If you are replacing a heatsink that was previously installed, you must remove the old thermal pad. If you are installing a new heatsink, skip to step d. below.

---

a. Apply the supplied cleaning solution to the old thermal pad and let it soak for a least 15 seconds.

b. Wipe all of the old thermal pad off the heatsink using a soft cloth that will not scratch the heatsink surface.

Caution Damage to the heatsink surface can damage the heat transferring properties of the heatsink.

---

c. Apply the supplied preparation solution to the center bottom of the heatsink, where the new thermal pad will be applied.

d. Apply the new thermal pad to the center bottom of the heatsink.

e. Align the heatsink captive screws with the motherboard standoffs, then tighten the captive screws evenly, until the screws stop against the captive springs.

Note Alternate tightening each screw evenly to avoid damaging the heatsink or CPU.

f. Replace the CPU cage. Align the six captive screws with the holes in the motherboard, then tighten each screw evenly.

g. Replace the memory riser dividers. Slide each one into the slots on the CPU cage and the chassis.

h. Replace all memory risers. See Replacing Memory Risers.

i. Replace the top cover.

j. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Figure 3-14 Locations of CPU Cage Screws

Figure 3-15 Removing a CPU and Heatsink

Replacing Memory Risers

The memory risers connect to the motherboard and each riser provides eight DIMM slots. The memory riser is hot-swappable when you use the Attention button, as described in the following procedure.

Figure 3-16 Memory Riser LEDs (Top View)

The qualified and supported part numbers for this component are subject to change over time. For the most up-to-date list of replaceable components, see the following URL and then scroll to Technical Specifications :

http://www.cisco.com/en/US/products/ps10493/products_data_sheets_list.html

To replace or install a hot-swappable memory riser, follow these steps:

Step 1 Remove the memory riser that you are replacing: See Figure 3-17:

a. Slide the server out the front of the rack far enough so that you can remove the top cover.

b. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

c. Press the attention button (ATTN BUTTON) on the top of the memory riser (see Figure 3-16).

d. Wait until the attention LED (ATTN) and the power LED turn off.

e. Simultaneously press both green release buttons on the top of the memory riser to release the riser retaining latches. The latches open up to a 45-degree angle when they are released.

f. Grasp the open retaining latches and lift the memory riser straight up and out of the motherboard connector.

g. If you are installing or replacing DIMMs on the memory riser, use the instructions in Replacing DIMMs.

Step 2 Install (hot-add) a new memory riser:

a. Ensure that the riser release latches are in the open position.

a. Align the riser with the empty motherboard connector.

b. Push the riser into the connector until it is seated and the open release levers engage.

c. Simultaneously press down on each release lever to put them in the locked position. This ensures that the riser is properly seated in the motherboard connector.

d. Press the attention button, then wait until the attention LED turns off.

e. Replace the top cover.

f. Replace the server in the rack and replace any cables.

Figure 3-17 Removing and Replacing Memory Risers

Replacing DIMMs

This section includes the following sections:

• Memory Performance Guidelines and Population Rules
• DIMM Installation Procedure

Note To ensure the best server performance, it is important that you are familiar with memory performance guidelines and population rules before you install or replace memory modules.

Memory Performance Guidelines and Population Rules

This section describes the type of memory that the server requires and its effect on performance. The following topics are covered:

• Supported DIMMs
• Memory Channels
• DIMM Population Rules
• DIMM and Rank Sparing

Supported DIMMs

The qualified and supported part numbers for this component are subject to change over time. For the most up-to-date list of replaceable components, see the following URL and then scroll to Technical Specifications :

http://www.cisco.com/en/US/products/ps10493/products_data_sheets_list.html

Memory Channels

Each CPU supports two memory risers. Figure 3-18 shows the placement of the CPUs and their corresponding memory risers. The view shown is facing the front of the server. This numbering is also inscribed on the top of the CPU cage.

Figure 3-18 CPUs and Memory Risers

Each memory riser contains two memory buffers that are connected to the CPU by serial memory interface (SMI) channels. Each memory buffer has two channels, each containing a pair of DDR3 DRAM slots.

In Figure 3-19, the buffers and channels are:

• Buffer 1, channel 1: slots 1B and 1D
• Buffer 1, channel 2: slots 1A and 1C
• Buffer 2, channel 1: slots 2B and 2D
• Buffer 2, channel 2: slots 2A and 2C

Figure 3-19 DIMM Slots and Memory Buffers

DIMM Population Rules

Following are the DIMM population rules:

• The minimum configuration for the server is, at least one matched DIMM pair installed in a memory riser on either CPU1 or CPU2 (see Figure 3-18). All four CPUs can run from a single DIMM pair.
• DIMMs are required to be populated in pairs. DIMMs for this server are sold as two-DIMM kits.
• The DIMMs in any given pair must be identical.
• Any DIMM installed in a memory riser corresponding to an empty CPU slot becomes inaccessible.
• For optimal performance, distribute DIMMs evenly across all installed CPUs and memory buffers.
• DIMMs within a channel are populated starting with the DIMMs farthest from the memory buffer in a fill-farthest approach.

For example, the order that you should populate the four channels on a memory riser is as follows (see also Figure 3-19):

1. Slots 1B and 1D

2. Slots 1A and 1C

3. Slots 2B and 2D

4. Slots 2A and 2C

DIMM and Rank Sparing

DIMM and rank sparing can be enabled in the BIOS configuration utility.

Sparing involves utilizing one of the DIMM pairs or rank pairs within each memory riser as a spare unit. When any of the other DIMM pairs within the same memory riser experiences errors beyond a pre-defined threshold, it fails over to the spare DIMM pair. Spared DIMMs and ranks are hidden from the user and the OS so that the BIOS can migrate to a spare unit when it finds degrading DIMMs.

When sparing is enabled, the available system memory is lesser than the total installed memory.

• When using DIMM sparing, the available memory equals total installed memory minus the size of spared DIMMs.
• When using rank sparing, available memory equals total installed memory minus the size of the spared ranks. Rank size equals DIMM size divided by the number of ranks.

DIMM Installation Procedure

This section contains the following topics:

• Identifying a Faulty DIMM
• DIMM Replacement Procedure

Identifying a Faulty DIMM

The memory riser has LEDs on it supper surface that can assist you in isolating a faulty DIMM. The faulty DIMMs are indicated by the DIMM fault LEDs, which light amber to indicate which DIMMs are faulty. See Figure 3-20.

Figure 3-20 Memory Riser LEDs (Top View)

DIMM Replacement Procedure

To install a DIMM pair, follow these steps:

Note DIMM risers are hot-swappable when you use the attention button, so you do not have power off the server or disconnect power cords. Use the following procedure.

Step 1 Remove the DIMMs that you are replacing: See Figure 3-21:

a. Slide the server out the front of the rack far enough so that you can remove the top cover.

Note You might have to detach cables from the rear panel to provide clearance.

Caution If you cannot safely view and access the component, remove the server from the rack.

---

b. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

c. Press the attention button (ATTN BUTTON) on the top of the memory riser that contains the faulty DIMM (see Figure 3-20).

d. Wait until the attention LED (ATTN) and the power LED turn off.

e. Simultaneously press both green release buttons on the top of the memory riser to release the riser retaining latches. The latches open up to a 45-degree angle when they are released.

f. Lift the memory riser straight up and out of the motherboard connector.

g. Locate the faulty DIMM and remove it from the connector on the memory riser.

Step 2 Install a new DIMM:

a. Insert the DIMM assembly into the connector on the riser.

b. Push the DIMM into the connector until it is seated properly, and the white clips on either side of the connector lock into place.

c. Push the hot-addable memory riser into the motherboard connector until it is seated and the open release levers engage the chassis and the CPU cage.

d. Simultaneously press down on each release lever to put them in the closed position. This ensures that the riser is properly seated in the motherboard connector.

e. Press the attention button on the top of the memory riser, then wait until the attention LED turns off (see Figure 3-20).

f. Replace the top cover.

g. Replace the server in the rack and replace any cables.

Figure 3-21 Removing and Replacing DIMMs

Replacing Fan Modules

The eight fan modules in the server are numbered as follows when you are facing the front of the server. Each fan module has a fault LED that lights amber when the fan module fails.

Figure 3-22 Fan Module Numbering

The qualified and supported part numbers for this component are subject to change over time. For the most up-to-date list of replaceable components, see the following URL and then scroll to Technical Specifications :

http://www.cisco.com/en/US/products/ps10493/products_data_sheets_list.html

To replace or install a hot-pluggable fan module, follow these steps:

Caution You do not have to shut down or power off the server to replace fan modules because they are hot- pluggable. However, to maintain proper cooling, do not operate the server for more than one minute with any fan module removed.

---

Step 1 Remove the fan module that you are replacing: See Figure 3-23:

a. Slide the server out the front of the rack far enough so that you can remove the top cover.

Note You might have to detach cables from the rear panel to provide clearance.

Caution If you cannot safely view and access the component, remove the server from the rack.

---

b. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

c. Insert your thumb and forefinger in the two green release latches on top of the fan module.

d. Squeeze the release latches together and lift out the fan module.

Step 2 Install a new fan module:

a. Grasp the fan module by the release latches and align it with the empty fan bay and the motherboard connector.

Note As you face the front of the server, the connector on underside of the fan module should be oriented on the right-bottom side of the fan module. See Figure 3-23.

b. Press down on the top corners of the fan module until the connector is fully seated and the release latches lock in place.

c. Replace the top cover.

d. Replace the server in the rack.

Figure 3-23 Removing and Replacing Fan Modules

Replacing Hard Drives or Solid State Drives

Tip You do not have to shut down or power off the server to replace hard drives or SSDs because they are hot pluggable.

Note You can mix hard drives and solid state drives (SSDs) in the same server. However, You cannot configure a logical volume (virtual drive) that contains a mix of hard drives and SSDs. That is, when you create a logical volume, it must contain all hard drives or all SSDs.

Note Cisco recommends following the industry-standard practice of using drives of the same capacity when configuring RAID volumes. If you use drives of different capacities, the usable portion of the smallest drive will be used on all drives of the RAID volume.

Note The 500 GB SATA drives (A03-D500GC3) and the 1 TB SATA drives (A03-D1TBSATA) that are sold with the UCS C460 M2 server are supported at full 6G speeds.

The qualified and supported part numbers for this component are subject to change over time. For the most up-to-date list of replaceable components, see the following URL and then scroll to Technical Specifications :

http://www.cisco.com/en/US/products/ps10493/products_data_sheets_list.html

To replace or install a hot-pluggable hard drive, follow these steps:

Step 1 Remove the drive that you are replacing or remove a blank panel from an empty bay (See Figure 3-25):

a. Press the release button on the face of the hard drive.

b. Grasp the ejector lever and pull the hard drive tray out of the slot.

c. If you are replacing an existing drive, remove the four drive tray screws that secure the drive to the tray and then lift the drive out of the tray.

Step 2 Install a new drive:

a. Place a new hard drive in the empty drive tray and replace the four drive tray screws.

b. Insert the drive tray into the empty drive bay.

c. Push the tray into the slot until the drive connectors are fully seated in the backplane.

d. Press the ejector lever flat to lock the drive and tray in place.

Figure 3-24 Drive Numbering and Physical Orientation, Facing Server Front

Figure 3-25 Removing and Replacing Hard Drives

Replacing a DVD Drive

Warning Class 1 laser product.
Statement 1008

The qualified and supported part numbers for this component are subject to change over time. For the most up-to-date list of replaceable components, see the following URL and then scroll to Technical Specifications :

http://www.cisco.com/en/US/products/ps10493/products_data_sheets_list.html

To replace or install a DVD drive, follow these steps:

Step 1 Remove the DVD drive that you are replacing. See Figure 3-26:

a. Power off the server as described in Shutting Down and Powering Off the Server.

b. Slide the server out the front of the rack far enough so that you can remove the top cover.

Caution If you cannot safely view and access the component, remove the server from the rack.

---

c. Remove the top cover as described in Removing and Replacing the Server Top Cover.

d. Remove all memory risers. See Replacing Memory Risers.

e. Remove all memory riser dividers.

f. Remove the black plastic cover from the floor of the chassis by removing the four screws that secure it. This strip covers the width of the chassis floor behind the fan modules.

g. Push the release button on the rear of the DVD drive, and then push the DVD drive out the front panel.

Step 2 Install a new DVD drive:

a. Insert the rear of the new DVD drive into the empty DVD drive bay on the front panel.

b. Push the drive inward until it seats in its connector and the release button locks in place.

c. Replace the black strip to the floor of the chassis by replacing its four screws.

d. Replace all memory riser dividers.

e. Replace all memory risers.

f. Replace the top cover.

g. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Figure 3-26 Removing and Replacing the DVD Drive

Replacing a PCIe Card

Note If you are installing a Cisco UCS Virtual Interface Card, there are prerequisite considerations. See Special Considerations for Cisco UCS Virtual Interface Cards.

This server has 10 PCIe expansion slots. See Figure 3-27 and Table 3-4 for information about the slots and which slots are hot-swappable. The replacement procedures differ depending on whether the PCIe slot is hot-swappable or not. This section contains the following topics:

• PCIe Slots
• Replacing a PCIe Card in a Non Hot-Swappable Slot
• Replacing a PCIe Card in a Hot-Swappable Slot
• Special Considerations for Cisco UCS Virtual Interface Cards
• Special Considerations for Cisco UCS Fusion ioDrive2 Storage Accelerator Cards
• Installing Multiple PCIe Cards and Resolving Limited Resources

PCIe Slots

Figure 3-27 PCIe Slots

Note CPU3 must be installed to support PCIe slots 5, 6, 7, 9, and 10.

Note Legacy I/O devices like video cards are only supported on slots 1, 2, 3, 4 and 8.

Replacing a PCIe Card in a Non Hot-Swappable Slot

The qualified and supported part numbers for this component are subject to change over time. For the most up-to-date list of replaceable components, see the following URL and then scroll to Technical Specifications :

http://www.cisco.com/en/US/products/ps10493/products_data_sheets_list.html

To install or replace a PCIe card in slots 3, 4, 5, 8, 9, or 10, follow these steps:

Step 1 Remove a PCIe card:

a. Shut down and power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover.

Note You might have to detach cables from the rear panel to provide clearance.

Caution If you cannot safely view and access the component, remove the server from the rack.

---

c. Remove the top cover as described in the “Removing and Replacing the Server Top Cover” section.

d. Remove any cables from the ports of the PCIe card that you are replacing.

Tip Label the cables when you disconnect them to aid correct connection to the new card.

e. Pinch and lift up the green retaining clip that secures the card to the chassis rear panel.

f. Lift the card straight up from the motherboard connector.

Note Lift up on both ends of the card evenly to avoid damaging its connector.

Step 2 Install a PCIe card:

a. Align the PCIe card with the empty PCIe connector on the motherboard.

b. Push down evenly on both ends of the card until it is fully seated in the motherboard connector.

c. Ensure that the card rear panel sits flat against the server back panel opening.

d. Push the green retaining clip down until it locks over the top of the card.

e. Replace the top cover.

f. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Replacing a PCIe Card in a Hot-Swappable Slot

The qualified and supported part numbers for this component are subject to change over time. For the most up-to-date list of replaceable components, see the following URL and then scroll to Technical Specifications :

http://www.cisco.com/en/US/products/ps10493/products_data_sheets_list.html

To install or replace a PCIe card in slots 1, 2, 6, or 7, follow these steps:

Step 1 Remove a hot-swappable PCIe card:

a. Slide the server out the front of the rack far enough so that you can remove the top cover.

Note You might have to detach cables from the rear panel to provide clearance.

Caution If you cannot safely view and access the component, remove the server from the rack.

---

b. Remove the top cover as described in the “Removing and Replacing the Server Top Cover” section.

c. Remove any cables from the ports of the PCIe card that you are replacing.

Tip Label the cables when you disconnect them to aid correct connection to the new card.

d. Press the lightpipe switch that is on the top of the plastic divider for the hot-swappable PCIe slot.

Note Wait for the lightpipe switch LED to turn off before removing the card in the next step.

e. Pinch and lift up the green retaining clip that secures the card to the chassis rear panel.

f. Lift the card straight up from the motherboard connector.

Note Lift up on both ends of the card evenly to avoid damaging its connector.

Step 2 Install a hot-swappable PCIe card:

a. Align the PCIe card with the empty PCIe connector on the motherboard.

b. Push down evenly on both ends of the card until it is fully seated in the motherboard connector.

c. Ensure that the card rear panel sits flat against the server back panel opening.

d. Push the green retaining clip down until it locks over the top of the card.

e. Press the lightpipe switch that is on top of the plastic divider for the hot-swappable PCIe slot. Wait for the LED to turn on to ensure that the PCIe slot receives power.

f. Replace the top cover.

g. Replace the server in the rack and replace cables.

Special Considerations for Cisco UCS Virtual Interface Cards

Table 3-5 describes the requirements for the supported Cisco UCS virtual interface cards (VICs).

Special Considerations for Cisco UCS Fusion ioDrive2 Storage Accelerator Cards

Table 3-6 describes the requirements for the supported Cisco UCS Fusion ioDrive2 cards.

Note PCIe slots 8 and 9 are Gen1 x4 electrical lane width slots and provide limited bandwidth if used with these cards.

Installing Multiple PCIe Cards and Resolving Limited Resources

When a large number of PCIe add-on cards are installed in the server, the system may run out of the following resources required for PCIe devices:

• Option ROM memory space
• 16-bit I/O space

The topics in this section provide guidelines for resolving the issues related to these limited resources.

• Resolving Insufficient Memory Space to Execute Option ROMs
• Resolving Insufficient 16-Bit I/O Space

Resolving Insufficient Memory Space to Execute Option ROMs

The system has very limited memory to execute PCIe legacy option ROMs, so when a large number of PCIe add-on cards are installed in the server, the system BIOS might not able to execute all of the option ROMs. The system BIOS loads and executes the option ROMs in the order that the PCIe cards are enumerated (Slot 1, Slot 2, Slot 3, etc.).

If the system BIOS does not have sufficient memory space to load any PCIe option ROM, it skips loading that option ROM, reports a system event log (SEL) event to the CIMC controller and reports the following error in the Error Manager page of the BIOS Setup utility:

To resolve this issue, disable the Option ROMs that are not needed for system booting. The BIOS Setup Utility provides the setup options to enable or disable the Option ROMs at the PCIe slot level for the PCIe expansion slots and at the port level for the onboard NICs. These options can be found in the BIOS Setup Utility Advanced → PCI Configuration page (see Figure 3-28).

Figure 3-28 BIOS Setup Utility Advanced > PCI Configuration Page

• Guidelines for RAID controller booting:

If the RAID controller does not appear in the system boot order, even if the option ROMs for the RAID controller slots are enabled, the RAID controller option ROM might not have sufficient memory space to execute. In that case, disable other option ROMs that are not needed for the system configuration to free up some memory space for the RAID controller option ROM.

• Guidelines for onboard NIC PXE booting:

If the system is configured to primarily perform PXE boot from onboard NICs, make sure that the option ROMs for the onboard NICs to be booted from are enabled in the BIOS Setup Utility. Disable other option ROMs that are not needed to create sufficient memory space for the onboard NICs.

Resolving Insufficient 16-Bit I/O Space

The system has only 64 KB of legacy 16-bit I/O resources available. This 64 KB of I/O space is divided between two I/O Hubs (IOH0 and IOH1) in the system.

The default BIOS setting is to allocate 40 KB to IOH0 and 24 KB to IOH1. Every PCIe device that is connected to IOH0 has to share the 40 KB of I/O. Every PCIe device connected to IOH1 must share that 24 KB of I/O.

When a large number of PCIe cards are installed in the system, the system BIOS might not have sufficient I/O space for some PCIe devices. If the system BIOS is not able to allocate the required I/O resources for any PCIe devices, the following symptoms have been observed:

• The BIOS might appear to hang while initializing PCIe devices.
• The PCIe option ROMs might take excessive time to complete, which appears to lock up the system.
• PCIe boot devices might not be accessible from the BIOS.
• PCIe option ROMs might report initialization errors. These errors are seen before the BIOS hands control to the operating system.
• The keyboard might not work.

To work around this problem, rebalance the load on each IOH using the following methods:

1. Adjust the IOH IO Resource Allocation ratio. This option is included in the BIOS Setup Utility Advanced → PCI Configuration page (see Figure 3-28).

2. Physically remove any unused PCIe cards.

3. Physically rearrange the PCIe cards to create a balance between IOH0 and IOH1.

In this server, the PCIe slots are connected to IOH0 and IOH1 as follows:

– Slot 1: IOH 0

– Slot 2: IOH 0

– Slot 3: IOH 0

– Slot 4: IOH 0

– Slot 5: IOH 1

– Slot 6: IOH 1

– Slot 7: IOH 1

– Slot 8: IOH 0

– Slot 9: IOH 1

– Slot 10: IOH 1

Installing an NVIDIA Tesla C2050 GPU Card

Note The NVIDIA Tesla C2050 GPU card is supported in Generation M2 servers only. This card is supported only in PCIe slot 5 of the server (see Figure 3-27). When installed in PCIe slot 5, this double-wide card physically covers two slots, slot 5 and slot 6.

Note The DVI connector on the rear panel of the GPU card is disabled. Connect your monitor to one of the VGA connectors on the server instead.

Cisco ships an accessory kit (UCSC-GPU_N01-C460) that includes two adapters that you must use to customize the card for installation in the Cisco C460 M2 server. Each kit includes the following components (see Figure 3-29):

• One rear-panel faceplate
• One front bracket
• One power cable adapter

Figure 3-29 Cisco Adapters For NVIDIA Tesla C2050 GPU Card

To install or replace an NVIDIA Tesla C2050 GPU card, follow these steps:

Step 1 Prepare the server for installation of the card:

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Disconnect all power cords from the power supplies.

c. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

d. Remove the top cover as described in the “Removing and Replacing the Server Top Cover” section.

e. Open the two green PCIe retainer latches on PCIe slots 5 and 6 (see Figure 3-30).

f. Remove any PCIe cards or blanking panels from PCIe slots 5 and 6.

g. Remove the plastic divider assembly that is over PCIe slots 6 and 7 to make clearance for the card.

Pull up on the pin that attaches this divider assembly to the motherboard, then lift the assembly out of the server. This pin is on the front end of the divider assembly where it touches the motherboard (see Figure 3-30).

Figure 3-30 Cisco C2050 GPU Card Slots

Step 2 Install adapters on the GPU card to prepare for installation:

a. Remove the default rear-panel faceplate from the rear end of a new card. Remove the two securing screws and set them aside.

b. Install the new rear-panel faceplate to the rear end of the card (see Figure 3-31). Install the two securing screws that you removed in the prior step.

c. Install the front bracket to the front end of the card (see Figure 3-31). Insert the pins on the bracket into the corresponding holes in the card and then tighten the single securing thumbscrew.

d. Connect the power cable adapter to the card. Insert the six-pin connector on the cable into the six-pin connector on the front end of the card.

Figure 3-31 Assembling Cisco C2050 GPU Card Components

Step 3 Install a new GPU card to the server:

a. Align the card so that its bottom-edge connector is over PCIe slot 5.

b. Tilt the front end of the card downward as you lower it into the chassis so that the attached power cable goes under the CPU cage (behind CPU3). The two thumbscrews on the front bracket that you installed must align with the two threaded holes in the CPU cage (see Figure 3-32).

c. Perform the following two actions simultaneously:

– Insert the two tabs on the rear-panel faceplate of the card into the two chassis rear-panel openings for slots 5 and 6.

– Carefully push down on both ends of the card to seat its connector in the PCIe slot 5 motherboard connector.

d. Close the two green PCIe retainer latches for slots 5 and 6 over the top of the rear-panel faceplate.

e. Tighten the two thumbscrews on the front bracket of the card to secure it to the CPU cage.

Step 4 Install the power cable adapter:

a. Disconnect the PCIe power cable from the motherboard. This is 10-pin connector J166 (see Figure 3-32).

b. Connect the male 10-pin connector of the power cable adapter to the motherboard connector J166.

c. Connect the PCIe power cable into the female 10-pin connector on the power cable adapter, which you attached to the card in Step 2.

Step 5 Replace the top cover.

Step 6 Replace the server in the rack, replace power cords and any other cables, and then power on the server by pressing the Power button.

Figure 3-32 Installing and Cabling the GPU Card

Installing and Enabling a Trusted Platform Module

The trusted platform module (TPM) is a small circuit board that connects to a motherboard socket and is secured by a one-way screw.

Installing the TPM Hardware

Note For security purposes, the TPM is installed with a one-way screw. It cannot be removed with a standard screwdriver.

Step 1 Verify that the Server has a valid UUID. This can be seen from the Cisco IMC Server Summary page.

Step 2 Prepare the server:

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Disconnect all power cords from the power supplies.

c. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

d. Remove the top cover as described in the “Removing and Replacing the Server Top Cover” section.

Step 3 Verify that the jumper that disables onboard TPM is in place before you begin to install the add-on TPM module (see Figure 3-33 for the location).

Note The default from the factory is that this two-pin jumper comes installed and onboard TPM is disabled. The jumper must be in place before you can enable an add-on TPM module.

Step 4 Install a TPM (see Figure 3-33 for the header location):

a. Align the connector that is on the underside of the TPM with the TPM header pins on the motherboard, and then press firmly on both ends of the TPM to seat it.

b. Install the securing screw that holds the TPM to the motherboard.

c. Replace the top cover.

d. Replace the server in the rack, replace power cords and any other cables, and then power on the server by pressing the Power button.

Step 5 Continue with Enabling TPM Hardware Support and the TXT Feature in the BIOS.

Figure 3-33 Installing a TPM

Enabling TPM Hardware Support and the TXT Feature in the BIOS

Step 1 When the server boots, press F2 when prompted to enter the BIOS menu.

Step 2 Enter the administrator password or set up one (Security > Set Administrator Password).

Step 3 Enable TPM (Security > TPM Administrative Control: Turn ON)

Step 4 Enable TXT (Advanced > Processor Configuration > Intel Trusted Execution Technology: Enabled).

Step 5 Save and Reboot (press F10).

Step 6 Enter the BIOS menu Clear TPM (Security > TPM Administrative Control: Clear Ownership).

Step 7 Save and Reboot (press F10).

Step 8 Enter the BIOS menu Enable TPM (Security > TPM Administrative Control: Turn ON).

Step 9 Save and Reboot (press F10).

Step 10 Install your OS.

Note It is required that you perform the three reboots. At this point the TPM is ready to be used by ESXi, which takes ownership during the first boot. If you want to boot another OS on the same server, you must repeat the process above to clear the ownership of the TPM before booting that new OS.