Cisco UCS C240 M3 Server Installation and Service Guide

Cisco Integrated Management Interface (CIMC)

You can monitor the server inventory, health, and system event logs by using the built-in Cisco Integrated Management Controller (CIMC) GUI or CLI interfaces. See the user documentation for your firmware release at the following URL:

http://www.cisco.com/en/US/products/ps10739/products_installation_and_configuration_guides_list.html

Server Configuration Utility

Cisco has also developed the Cisco Server Configuration Utility for C-Series servers, which can aid and simplify the following tasks:

• Monitoring server inventory and health
• Diagnosing common server problems with diagnostic tools and logs
• Setting the BIOS booting order
• Configuring some RAID configurations
• Installing operating systems

You can download the ISO from Cisco.com. See the user documentation for your version of the utility at the following URL:

http://www.cisco.com/en/US/products/ps10493/products_user_guide_list.html

Front Panel LEDs

Figure 3-1 shows the front panel LEDs. Table 3-1 defines the LED states.

Figure 3-1 Front Panel LEDs

Rear Panel LEDs and Buttons

Figure 3-2 shows the rear panel LEDs and buttons. Table 3-2 defines the LED states.

Figure 3-2 Rear Panel LEDs and Buttons

Internal Diagnostic LEDs

The server is equipped with a SuperCap voltage source that can activate internal component fault LEDs up to one half-hour after AC power is removed. The server has internal fault LEDs for fan modules and DIMMs.

To use these LEDs to identify a failed component, press the front or rear Identification button (see Figure 3-1 or Figure 3-2) with AC power removed. An LED lights amber to indicate a failed component.

See Figure 3-3 for the locations of these internal LEDs.

Figure 3-3 Internal Diagnostic LED Locations

Required Equipment

The following equipment is used to perform the procedures in this chapter:

• Number 2 Phillips-head screwdriver
• Electrostatic discharge (ESD) strap or other grounding equipment such as a grounded mat

Shutting Down and Powering Off the Server

The server can run in two power modes:

• Main power mode—Power is supplied to all server components and any operating system on your drives can run.
• Standby power mode—Power is supplied only to the service processor and the cooling fans and it is safe to power off the server from this mode.

You can invoke a graceful shutdown or an hard shutdown by using either of the following methods:

• Use the CIMC management interface.
• Use the Power button on the server front panel. To use the Power button, follow these steps:

Step 1 Check the color of the Power Status LED (see the “Front Panel LEDs” section).

• Green—the server is in main power mode and must be shut down before it can be safely powered off. Go to Step 2.
• Amber—the server is already in standby mode and can be safely powered off. Go to Step 3.

Step 2 Invoke either a graceful shutdown or a hard shutdown:

• Graceful shutdown—Press and release the Power button. The operating system performs a graceful shutdown and the server goes to standby mode, which is indicated by an amber Power Status LED.
• Emergency shutdown—Press and hold the Power button for 4 seconds to force the main power off and immediately enter standby mode.

Step 3 Disconnect the power cords from the power supplies in your server to completely power off the server.

Removing and Replacing the Server Top Cover

To remove or replace the top cover of the server, follow these steps:

Tip You do not have to remove the cover to replace hard drives or power supplies.

Step 1 Remove the top cover (see Figure 3-4).

a. Loosen the captive thumbscrew that secures the rear edge of the cover to the chassis.

b. Press the release button.

c. Using the rubber finger pads, push the top cover toward the server rear about one-half inch (1.27 cm), until it stops.

d. Lift the top cover straight up from the server and set it aside.

Step 2 Replace the top cover:

a. Place the cover on top of the server about one-half inch (1.27 cm) behind the lip of the chassis front cover panel. The cover should sit flat.

Note The rear of the cover has a wrap-around flanged edge that must be correctly aligned with the chassis rear edge when sliding the cover forward.

b. Slide the top cover toward the front cover panel until it stops and the release button locks.

c. Tighten the captive thumbscrew that secures the rear edge of the cover to the chassis.

Figure 3-4 Removing the Top Cover

Replaceable Component Locations

This section shows the locations of the components that are discussed in this chapter. The view in Figure 3-5 is from the top down with the top cover and air baffles removed.

Figure 3-5 Replaceable Component Locations

The Technical Specifications Sheet for this server, which includes component part numbers, is at: http://www.cisco.com/en/US/prod/collateral/ps10265/ps10493/C240M3_SFF_SpecSheet.pdf .

Serial Number Location

The serial number for the server is printed on a label on the top of the server, near the front.

Color-Coded Touch Points

This server has color-coded touch points that indicate thumbscrews and latches on replaceable and hot-swappable components.

• Hot-swappable components have green plastic touch points. This includes the internal cooling fans and the power supplies. (An exception is the drive trays on the front panel, which are hot-swappable but not green).
• Some replaceable but non-hot-swappable components have light-blue plastic touch-points.

Replacing Hard Drives or Solid State Drives

This section includes the following information:

• Drive Population Guidelines
• Drive Replacement Procedure

Drive Population Guidelines

The server is orderable in three different versions, each with one of three different front panel/backplane configurations:

• Cisco UCS C240 (small form-factor (SFF) drives, with 24-drive backplane and expander).
  Holds up to twenty-four 2.5-inch hard drives or solid state drives.
• Cisco UCS C240 (small form-factor (SFF) drives, with 16-drive backplane, no expander).
  Holds up to sixteen 2.5-inch hard drives or solid state drives.

Note When the server has the 16-drive backplane, only the first 16 drive bays are used.

• Cisco UCS C240 (large form-factor (LFF) drives, with 12-drive backplane and expander).
  Holds up to twelve 3.5-inch hard drives.

The drive-bay numbering is shown in Figure 3-6 and Figure 3-7.

Figure 3-6 Drive Numbering, Small Form-Factor Drives

Figure 3-7 Drive Numbering, Large Form-Factor Drives

Observe these drive population guidelines for optimal performance:

• When populating drives, add drives in the lowest numbered bays first (populate HDD1 to HDD24).

Note Sixteen-drive backplane option: If your SFF drives server has the 16-drive backplane installed, only the first 16 drive bays are used, with population order HDD1 to HDD16. Keep blanking trays in the empty eight bays to ensure optimal air flow and cooling.

• Keep an empty drive blanking tray in any unused bays to ensure optimal air flow and cooling.
• You can mix hard drives and solid state drives in the same server. However, You cannot configure a logical volume (virtual drive) that contains a mix of hard drives and SSDs. That is, when you create a logical volume, it must contain all hard drives or all SSDs.

Note The large form-factor drives version of the server does not support 3.5-inch solid state drives.

Drive Replacement Procedure

To replace or install a hot-pluggable hard drive, follow these steps:

Tip You do not have to shut down or power off the server to replace hard drives or solid state drives (SSDs) because they are hot-pluggable.

Step 1 Remove the drive that you are replacing or remove a blank drive tray from an empty bay:

a. Press the release button on the face of the drive tray. See Figure 3-8.

b. Grasp and open the ejector lever and then pull the drive tray out of the slot.

c. If you are replacing an existing drive, remove the four drive-tray screws that secure the drive to the tray and then lift the drive out of the tray.

Step 2 Install a new drive:

a. Place a new drive in the empty drive tray and replace the four drive-tray screws.

b. With the ejector lever on the drive tray open, insert the drive tray into the empty drive bay.

c. Push the tray into the slot until it touches the backplane, then close the ejector lever to lock the drive in place.

Figure 3-8 Replacing Hard Drives

Replacing a Drive Backplane

Note The Small Form-Factor (24-drive or 16-drive) and Large Form-Factor (12-drive) backplanes and their corresponding chassis drive bays are factory-configurable options. When replacing a backplane, you must replace it with the same version of the backplane.

To install or replace a drive backplane, follow these steps:

Note The 16-drive backplane option does not use a SAS expander. You are instructed to skip steps that involve the SAS expander in the following procedure, if your server has the 16-drive backplane.

Step 1 Prepare the server for component replacement:

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

Step 2 Remove all drives and any empty drive trays from the server.

Tip Label the drive trays as you remove them to aid replacement.

Step 3 Remove the fan tray from the server (see Figure 3-9).

a. Pinch the blue-plastic lever at each end of the fan tray to release the lever from the chassis.

b. Simultaneously rotate both levers up 90-degrees. The cam action of the levers lifts the fan tray up and disengages the six fan modules from their connectors on the motherboard.

c. Lift the fan tray straight up and remove it from the chassis.

Step 4 Disconnect all cables from the backplane.

Step 5 If your server has a SAS expander, disconnect cables from this expander. (The SFF 16-drive backplane option does not use an expander.)

Step 6 Disconnect the first backplane power harness cable from the motherboard to provide clearance. This is motherboard connector BACKPLANE POWER1, between fan connectors 5 and 6 (see Figure 3-9).

Step 7 Loosen the two captive thumbscrews that secure the backplane to the chassis (see Figure 3-9).

Step 8 Lift the backplane assembly, including steel tray and any expander card straight up from the chassis and set it on an antistatic mat.

Note If your SFF server has the 16-drive backplane, it does not use a SAS expander. Skip to Step 11.

Step 9 Remove the SAS expander card from the backplane assembly:

a. Use a #2 Phillips-head screwdriver to remove the two screws that secure the SAS expander to the backplane assembly steel tray (see Figure 3-10).

b. Pull the SAS expander from the sockets on the drive backplane and then set the SAS expander aside on an antistatic mat.

Step 10 Install the SAS expander card to the new backplane assembly:

a. Push the two connectors on the SAS expander into the two sockets on the backplane assembly.

b. Use a #2 Phillips-head screwdriver to install the two screws that secure the SAS expander to the backplane assembly steel tray (see Figure 3-10).

Step 11 Align the backplane assembly steel tray with the guides on the chassis walls, and then lower it evenly to the chassis floor.

Step 12 Tighten the two captive thumbscrews that secure the backplane to the chassis.

Step 13 Reconnect the power harness cable to the motherboard connector BACKPLANE POWER1.

Step 14 If your server has a SAS expander, reconnect SAS cables to the SAS expander. (The SFF 16-drive backplane option does not use an expander.)

Step 15 Reconnect all cables to the backplane.

Step 16 Replace all drives and drives trays to the drive bays.

Step 17 Replace the fan tray.

a. With the blue-plastic lever at each end of the fan tray in the upright and open position, set the fan tray in place in the chassis. Use the chassis guides at each end of the fan tray to keep the fan tray level and straight.

b. Rotate each blue-plastic lever down to the locked position. Stop when the levers click and lock.

Step 18 Replace the top cover.

Step 19 Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Figure 3-9 Replacing the Drive Backplane

Replacing a SAS Expander

The SAS expander is a card that plugs directly into the drive backplane. See Appendix C, “RAID Controller Considerations” for more information about supported RAID controllers.

Note The SAS expander is required for the SFF 24-drive option and the LFF 12-drive option.
The SFF 16-drive option does not use the SAS expander.

To install or replace a SAS expander, follow these steps:

Step 1 Prepare the server for component replacement:

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

Step 2 Remove the fan tray from the server (see Figure 3-10).

a. Pinch the blue-plastic lever at each end of the fan tray to release the lever from the chassis.

b. Simultaneously rotate both levers up 90-degrees. The cam action of the levers lifts the fan tray up and disengages the six fan modules from their connectors on the motherboard.

c. Lift the fan tray straight up and remove it from the chassis.

Step 3 Disconnect all cables from the SAS expander.

Step 4 Remove the SAS expander:

a. Use a #2 Phillips-head screwdriver to remove the two screws that secure the SAS expander to the backplane assembly steel tray (see Figure 3-10).

b. Pull on both corners of the SAS expander to disengage it from the sockets on the drive backplane and then set the SAS expander aside on an antistatic mat.

Step 5 Install the new SAS expander:

a. Push the two board-edge connectors on the new SAS expander into the two sockets on the backplane.

b. Use a #2 Phillips-head screwdriver to install the two screws that secure the SAS expander to the backplane assembly steel tray (see Figure 3-10).

Step 6 Reconnect SAS cables to the new SAS expander.

Step 7 Replace the fan tray.

a. With the blue-plastic lever at each end of the fan tray in the upright and open position, set the fan tray in place in the chassis. Use the chassis guides at each end of the fan tray to keep the fan tray level and straight.

b. Rotate each blue-plastic lever down to the locked position. Stop when the levers click and lock.

Step 8 Replace the top cover.

Step 9 Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Figure 3-10 Replacing the SAS Expander

Replacing Fan Modules

The six hot-pluggable fan modules in the server are numbered as follows when you are facing the front of the server.

Figure 3-11 Fan Module Numbering

Tip There is a fault LED on the top of each fan module that lights amber if the fan module fails. To operate these LEDs from the SuperCap power source, remove AC power cords and then press the Identification button. See also Internal Diagnostic LEDs.

To replace or install a hot-pluggable fan module, follow these steps:

Step 1 Remove a fan module that you are replacing (see Figure 3-12):

a. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

b. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

c. Grasp and squeeze together the two plastic finger-latches on the top of the fan module and then lift straight up to disengage the fan connector from the motherboard.

Step 2 Install a new fan module:

a. Set the new fan module in place, aligning the connector on the bottom of the fan module with the connector on the motherboard (see Figure 3-12).

b. Press down gently on the fan module until the finger-latches click and lock in place.

c. Replace the top cover.

d. Replace the server in the rack.

Figure 3-12 Replacing Fan Modules

Replacing the Motherboard RTC Battery

Warning There is danger of explosion if the battery is replaced incorrectly. Replace the battery only with the same or equivalent type recommended by the manufacturer. Dispose of used batteries according to the manufacturer’s instructions. [Statement 1015]

The real-time clock (RTC) battery retains system settings when the server is disconnected from power. The battery type is CR2032. Cisco supports the industry-standard CR2032 battery, which can be purchased from most electronic stores.

Step 1 Remove the RTC battery (see Figure 3-13):

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

d. Remove the fan tray from the server (see Figure 3-13).

1. Pinch the blue-plastic lever at each end of the fan tray to release the lever from the chassis.

2. Simultaneously rotate both levers up 90-degrees. The cam action of the levers lifts the fan tray up and disengages the six fan modules from their connectors on the motherboard.

3. Lift the fan tray straight up and remove it from the chassis.

e. Gently remove the battery from its holder on the motherboard. Use a small screwdriver or pointed object under the battery to gently pry upward.

Step 2 Install an RTC battery:

a. Insert the battery into its holder and press down until it clicks in place.

Note The positive side of the battery marked “3V+” should face upward.

b. Replace the fan tray.

1. With the blue-plastic lever at each end of the fan tray in the upright and open position, set the fan tray in place in the chassis. Use the chassis guides at each end of the fan tray to keep the fan tray level and straight.

2. Rotate each blue-plastic lever down to the locked position. Stop when the levers click and lock.

c. Replace the top cover.

d. Replace the server in the rack, replace cables, and power on the server by pressing the Power button.

Figure 3-13 Replacing the Motherboard RTC Battery

Replacing DIMMs

This section includes the following topics:

• Memory Performance Guidelines and Population Rules
• DIMM Replacement Procedure

Note To ensure the best server performance, it is important that you are familiar with memory performance guidelines and population rules before you install or replace memory.

Memory Performance Guidelines and Population Rules

This section describes the type of memory that the server requires and its effect on performance. The section includes the following topics:

• DIMM Slot Numbering
• DIMM Population Rules
• Memory Mirroring

DIMM Slot Numbering

Figure 3-14 shows the numbering of the DIMM slots.

Figure 3-14 CPUs and DIMM Slots on Motherboard

DIMM Population Rules

Observe the following guidelines when installing or replacing DIMMs:

• Each CPU supports four memory channels.

– CPU1 supports channels A, B, C, and D.

– CPU2 supports channels E, F, G, and H.

• Each channel has three DIMM slots (for example, channel A = slots A1, A2, and A3).

– A channel can operate with one, two, or three DIMMs installed.

– If a channel has only one DIMM, populate slot 1 first (the blue slot).

• When both CPUs are installed, populate the DIMM slots of each CPU identically.

– Fill blue #1 slots in the channels first: A1, E1, B1, F1, C1, G1, D1, H1

– Fill black #2 slots in the channels second: A2, E2, B2, F2, C2, G2, D2, H2

– Fill black #3 slots in the channels third: A3, E3, B3, F3, C3, G3, D3, H3

• Any DIMM installed in a DIMM socket for which the CPU is absent is not recognized. In a single-CPU configuration, populate the channels for CPU1 only.
• Although 1600 MHz DIMMs can be run in Power Saving Mode (low-voltage 1.35 V operation), 1600 MHz operation is supported only when the DDR mode is set to Performance Mode (see “Enabling Low-Voltage DIMM Operation.”) A 1600 MHz DIMM with DDR mode set to Power Saving Mode operates at 1066 MHz.
• Observe the DIMM mixing rules shown in Table 3-4 .
• Memory mirroring reduces the amount of memory available by 50% because only one of the two populated channels provides data. When memory mirroring is enabled, DIMMs must be installed in sets of 4, 6, 8, or 12 as described in Memory Mirroring.
• Note the following restrictions when using UDIMMs. Even though a UDIMM might be rated at 1600 MHz, actual operational speeds are slower because of the Intel implementation.

– UDIMMs do not support 3 DIMMs per channel (3 DPC) configurations.

– In Performance Mode (1.5 V operation), UDIMMs run at 1333 MHz in 1 DPC and 2 DPC configurations.

– In Power Saving Mode (1.35 V operation), UDIMMs run at 1066 MHz in 1 DPC and 2 DPC configurations.

Enabling Low-Voltage DIMM Operation

You can enable low voltage (1.35 V) DIMM operation for all DIMMs in the server. There is a setting in the BIOS Setup utility that you can use to change the DDR memory mode to Power Saving mode, as described in the following procedure:

Step 1 Enter the BIOS setup utility by pressing the F2 key when prompted during bootup.

Step 2 Select the Advanced tab.

Step 3 Select Low Voltage DDR Mode.

Step 4 In the pop-up window, select either Power Saving or Performance Mode:

• Power Saving Mode–Prioritizes low-voltage memory operation.
• Performance Mode–Prioritizes performance memory operation. If you mix low-voltage DIMMs with standard DIMMs, the system defaults to this setting.

Step 5 Press F10 to save your changes and exit the setup utility.

Memory Mirroring

When memory mirroring is enabled, the memory subsystem simultaneously writes identical data to two channels. If a memory read from one of the channels returns incorrect data due to an uncorrectable memory error, the system automatically retrieves the data from the other channel. A transient or soft error in one channel does not affect the mirrored data, and operation continues unless there is a simultaneous error in exactly the same location on a DIMM and its mirrored DIMM. Memory mirroring reduces the amount of memory available to the operating system by 50% because only one of the two populated channels provides data.

Note You must choose to enable memory mirroring in the server BIOS setup utility, under
Advanced > Memory Configuration > Mirroring.

If you choose to enable memory mirroring, populate the DIMM slots in the order shown in Table 3-5 .

DIMM Replacement Procedure

This section includes the following topics:

• Identifying a Faulty DIMM
• Replacing DIMMs

Identifying a Faulty DIMM

Each DIMM slot has a corresponding DIMM fault LED. See Figure 3-3 for the locations of these LEDs.

The LEDs light amber to indicate a faulty DIMM. To operate these LEDs from the SuperCap power source, remove AC power cords and then press the Identification button. See also Internal Diagnostic LEDs.

Replacing DIMMs

To install a DIMM assembly, follow these steps:

Step 1 Remove the DIMMs that you are replacing:

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

d. Identify the faulty DIMM by observing the DIMM slot fault LEDs on the motherboard (see Figure 3-3).

e. Open the ejector levers at both ends of the DIMM slot, then lift the DIMM out of the slot.

Step 2 Install a new DIMM:

Note Before installing DIMMs, refer to the population guidelines. See Memory Performance Guidelines and Population Rules.

a. Align the new DIMM with the empty slot on the motherboard. Use the alignment key in the DIMM slot to correctly orient the DIMM.

b. Push down evenly on the top corners of the DIMM until it is fully seated and the ejector levers on both ends lock into place.

c. Replace the top cover.

d. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Replacing CPUs and Heatsinks

This section contains the following topics:

• Special Information For Upgrades to Intel E5-2600 v2 Series CPUs
• CPU Configuration Rules
• CPU Replacement Procedure
• Additional CPU-Related Parts To Order With RMA Replacement Motherboards

Special Information For Upgrades to Intel E5-2600 v2 Series CPUs

The minimum software and firmware versions required for the server to support Intel E5-2600 v2 Series CPUs are as follows:

Do one of the following actions:

• If your server’s firmware and/or Cisco UCS Manager software are already at the required levels shown in Table 3-6 , you can install the CPU hardware by using the procedure in this section.
• If your server’s firmware and/or Cisco UCS Manager software is earlier than the required levels, use the instructions in the Cisco UCS C-Series Servers Upgrade Guide for Intel E5-2600 v2 Series CPUs to upgrade your firmware. After you upgrade, return to the procedure in this section to install the CPU and heatsink hardware.

CPU Configuration Rules

This server has two CPUs. Each CPU supports four DIMM channels (12 DIMM slots). See Figure 3-14.

• The server can operate with one CPU or two identical CPUs installed.
• The minimum configuration is that the server must have at least CPU1 installed. Install CPU1 first, then CPU2.

Note The following restrictions apply when using a single-CPU configuration: the maximum number of DIMMs is twelve, the internal mezzanine card slot is unavailable, and PCIe riser 2, which contains PCIe slots 4 and 5, is unavailable.

CPU Replacement Procedure

To install or replace a CPU heatsink and CPU, follow these steps:

Step 1 Remove the CPU and heatsink that you are replacing:

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

d. Use a Number 2 Phillips-head screwdriver to loosen the four captive screws that secure the heatsink and then lift it off of the CPU.

Note Alternate loosening each screw evenly to avoid damaging the heatsink or CPU.

e. Unclip the first CPU retaining latch labeled with the icon, then unclip the second retaining latch labeled with the icon. See Figure 3-15.

f. Open the hinged CPU cover plate.

Figure 3-15 CPU Heatsink and Socket

Step 2 Remove a protective cap or an old CPU from the socket (if present):

• If you are removing an old CPU, skip to Step 3.
• If you are installing a new CPU to a socket that was shipped empty, the socket has a protective cap that is intended to prevent bent contact pins. Use the tool as shown in Figure 3-16 to grasp the protective cap and then pivot to remove the cap.

Figure 3-16 Protective Cap Removal Tool

Step 3 Remove an old CPU:

a. Set the Pick-and-Place tool on the CPU in the socket, aligning the arrow on the tool with the registration mark on the socket (the small triangular mark). See Figure 3-17.

b. Press the top button on the tool to grasp the installed CPU.

c. Lift the tool and CPU straight up.

d. Press the top button on the tool to release the old CPU on an anti-static surface.

Figure 3-17 Removing or Inserting a CPU

Step 4 Insert the new CPU into the Pick-and-Place tool:

a. Remove the new CPU from the packaging and place it on the pedestal that is included in the kit. Align the registration mark on the corner of the CPU with the arrow on the corner of the pedestal (see Figure 3-18).

b. Press down on the top button of the tool to lock it open.

c. Set the Pick-and-Place tool on the CPU pedestal, aligning the arrow on the tool with the arrow on the corner of the pedestal. Make sure that the tabs on the tool are fully seated in the slots on the pedestal.

d. Press the side lever on the tool to grasp and lock in the CPU.

e. Lift the tool and CPU straight up off the pedestal.

Figure 3-18 CPU and Pick-and-Place Tool on Pedestal

Step 5 Install a new CPU:

a. Set the Pick-and-Place tool with CPU over the empty CPU socket on the motherboard.

Note Align the arrow on the top of the tool with the registration mark (small triangle) that is stamped on the metal of the CPU socket, as shown in Figure 3-17.

b. Press the top button on the tool to set the CPU into the socket. Remove the empty tool.

c. Close the hinged CPU cover plate.

d. Clip down the CPU retaining latch with the icon, then clip down the CPU retaining latch with the icon. See Figure 3-15.

Step 6 Install a heatsink:

a. Apply an alcohol-based cleaning solution to the old thermal pad and let it soak for a least 15 seconds.

b. Wipe all of the old thermal pad off the old heatsink using a soft cloth that will not scratch the heatsink surface.

c. Apply thermal grease from an included syringe to the top of the CPU.

Apply about 2 cubic centimeters of grease (about half the syringe contents) to the top of the CPU in the pattern that is shown in Figure 3-19.

Note CPU spares come with two syringes of thermal grease; one with a blue cap and one with a red cap. The syringe with the red cap is UCS-CPU-GREASE2=, which is used with this server.

Note If you do not have a syringe of thermal grease, you can order a spare (UCS-CPU-GREASE2=).

Figure 3-19 Thermal Grease Application Pattern

d. For a new heatsink, peel the protective film from the thermal pad that is on the bottom of the new heatsink.

Note Do not apply a syringe of thermal grease if you are installing a new heatsink that already has a pre-applied thermal pad.

e. Align the heatsink captive screws with the motherboard standoffs, then use a Number 2 Phillips-head screwdriver to tighten the captive screws evenly.

Note Alternate tightening each screw evenly to avoid damaging the heatsink or CPU.

f. Replace the top cover.

g. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Additional CPU-Related Parts To Order With RMA Replacement Motherboards

When a return material authorization (RMA) of the motherboard or CPU is done on a Cisco UCS C-series server, there are additional parts that might not be included with the CPU or motherboard spare bill of materials (BOM). The TAC engineer might need to add the additional parts to the RMA to help ensure a successful replacement.

• Scenario 1—You are re-using the existing heatsinks:

– Heat sink cleaning kit (UCSX-HSCK=)

– Thermal grease kit for C240 (UCS-CPU-GREASE2=)

– Intel CPU Pick-n-Place tool: UCS-CPU-EP-PNP= for 10-, 8-, 6-, 4-, or 2-core CPUs (green); or UCS-CPU-EP2-PNP= for v2 12-core CPUs (purple)

• Scenario 2—You are replacing the existing heatsinks:

– Heat sink (UCSC-HS-C240M3=)

– Heat sink cleaning kit (UCSX-HSCK=)

– Intel CPU Pick-n-Place tool: UCS-CPU-EP-PNP= for 10-, 8-, 6-, 4-, or 2-core CPUs (green); or UCS-CPU-EP2-PNP= for v2 12-core CPUs (purple)

A CPU heatsink cleaning kit is good for up to four CPU and heatsink cleanings. The cleaning kit contains two bottles of solution, one to clean the CPU and heatsink of old thermal interface material and the other to prepare the surface of the heatsink.

New heatsink spares have preinstalled thermal interface material covered by a small sheet of plastic. It is important to clean the old thermal interface material off of the CPU prior to installing the heatsinks. Therefore, when ordering new heatsinks it is still necessary to order the heatsink cleaning kit at a minimum.

Replacing a Mezzanine Card

The server has a dedicated socket on the motherboard for a mezzanine-style RAID controller card.

Note See RAID Controller Considerations for more information about supported cards and cabling.

Note After you replace a component such as an adapter card or a RAID controller card, use the Cisco Host Upgrade Utility to update all Cisco components to compatible firmware levels (Cisco Host Upgrade Utility User Guides). If you install third-party components, install the firmware or drivers supplied by the third-party vendor.

To install or replace a mezzanine card, follow these steps:

Step 1 Remove the mezzanine card that you are replacing (see Figure 3-20):

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

d. Remove PCIe riser 2 from the server to provide clearance (see Figure 3-20).

e. Disconnect all cables from the mezzanine card.

Tip Label the cables before you disconnect them to aid replacement.

f. Use a #2 Phillips-head screwdriver to remove the two screws that secure the card to the motherboard standoffs.

g. Pull the mezzanine card straight up to disengage it from its motherboard socket, and then remove the card from the chassis.

Step 2 Install a new mezzanine card:

a. Set the new mezzanine card in place on the motherboard, aligning its connector with the motherboard socket.

b. Push down on both ends of the mezzanine card evenly to engage the card’s connector with the socket on the motherboard.

c. Use a #2 Phillips-head screwdriver to install the two screws that secure the card to the motherboard standoffs.

d. Reconnect cables to the new mezzanine card. See RAID Controller Considerations for more information about supported cards and cabling.

e. Replace PCIe riser 2 to the server.

f. Replace the top cover.

g. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

h. If this was a replacement card, continue with Restoring RAID Configuration After Replacing a RAID Controller.

Figure 3-20 Replacing the Mezzanine Card (Shown With PCIe Riser 2 Removed)

Replacing a PCIe Riser

Note The older PCIe risers UCSC-PCIE-RL-C240= and UCSC-PCIE-RR-C240= are compatible only with the older version V01 chassis, and have been obsoleted. These parts have been replaced by a new single item (UCSC-GPUKIT1-C240=), which contains left and right risers along with additional hardware to facilitate installation of NVIDIA GPU cards. Order the new UCSC-GPUKIT1-C240= if riser replacement is required on any version of the C240 chassis, or if a version V01 chassis must be upgraded to support an NVIDIA GPU card. See also How to Determine Your Server Version.

The server contains two toolless PCIe risers for horizontal installation of PCIe cards. PCIe riser 2 also includes two internal sockets for SD cards. See Replacing a PCIe Card for specifications of the PCIe slots on the risers.

Figure 3-21 Rear Panel, Showing PCIe Slots

To install or replace a PCIe riser, follow these steps:

Step 1 Remove the PCIe riser that you are replacing (see Figure 3-20):

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the top cover as described in “Removing and Replacing the Server Top Cover” section.

d. Disconnect external cables from any PCIe cards that are installed in the PCIe riser.

e. Use the finger holes to lift straight up on both ends of the riser to disengage its circuit board from the socket on the motherboard. Set the riser on an antistatic mat.

f. If the riser has a card installed, remove the card from the riser.

To remove a card, push down on the retaining clip on the hinged card retainer and then swing open the retainer to free the rear-panel tab of the card (see Replacing a PCIe Card).

Step 2 Install a new PCIe riser:

a. If you removed a card from the old PCIe riser, install the card to the new riser (see Replacing a PCIe Card).

b. Position the PCIe riser over its socket on the motherboard and over its alignment slots in the chassis (see Figure 3-22).

c. Carefully push down on both ends of the PCIe riser to fully engage its circuit board connector with the socket on the motherboard.

d. Connect external cables to any PCIe cards installed in the riser.

e. Replace the top cover.

f. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Figure 3-22 Replacing the PCIe Riser

Replacing a PCIe Card

Note After you replace a component such as an adapter card or a RAID controller card, use the Cisco Host Upgrade Utility to update all Cisco components to compatible firmware levels (Cisco Host Upgrade Utility User Guides). If you install third-party components, install the firmware or drivers supplied by the third-party vendor.

This section includes the following topics:

• PCIe Slots
• RAID Card Firmware Compatibility
• Replacing a PCIe Card
• Special Considerations for Cisco UCS Virtual Interface Cards
• Special Considerations for Cisco UCS Fusion ioDrive2 Storage Accelerator Cards
• RAID Controller Card Cable Routing
• Installing Multiple PCIe Cards and Resolving Limited Resources

PCIe Slots

The server contains two toolless PCIe risers for horizontal installation of PCIe cards. See Figure 3-23 and Table 3-7 .

Figure 3-23 Rear Panel PCIe Slots

RAID Card Firmware Compatibility

If the PCIe card that you are installing is a RAID controller card, firmware on the RAID controller must be verified for compatibility with the current Cisco IMC and BIOS versions that are installed on the server. If not compatible, upgrade or downgrade the RAID controller firmware accordingly using the Host Upgrade Utility (HUU) for your firmware release to bring it to a compatible level.

See the HUU guide for your Cisco IMC release for instructions on downloading and using the utility to bring server components to compatible levels: HUU Guides

Replacing a PCIe Card

Note For the list of supported PCIe adapters and other components, see the Technical Specifications Sheet at http://www.cisco.com/en/US/prod/collateral/ps10265/ps10493/C240M3_SFF_SpecSheet.pdf.

Note If you are installing a Cisco UCS Virtual Interface Card, there are prerequisite considerations. See Special Considerations for Cisco UCS Virtual Interface Cards.

Note If you are installing a RAID controller card, see RAID Controller Considerations for more information about supported cards and cabling.

To install or replace a PCIe card, follow these steps:

Step 1 Remove a PCIe card (or a blank filler panel) from the PCIe riser assembly:

a. Shut down and power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the top cover as described in the “Removing and Replacing the Server Top Cover” section.

d. Disconnect cables from any PCIe cards that are installed in the PCIe riser.

Tip Label the cables when you disconnect them to aid correct connection to the new card.

e. Use the finger holes to lift straight up on both ends of the riser to disengage its circuit board from the socket on the motherboard. Set the riser on an antistatic mat.

f. Push down on the securing clip on the hinged card retainer and then swing open the retainer to free the rear-panel tab of the existing card (or blanking panel). See Figure 3-24.

Note Slot 5 on PCIe riser 2 has an additional plastic retaining clip that stabilizes the front end of a card. Push down on this clip before pulling the card from the riser socket (see Figure 3-24).

g. Pull evenly on both ends of the PCIe card to disengage it from the socket on the PCIe riser (or remove a blanking panel) and then set the card aside.

Step 2 Install a PCIe card:

a. Align the new PCIe card with the empty socket on the PCIe riser.

b. Push down evenly on both ends of the card until it is fully seated in the socket.

Ensure that the card rear panel tab sits flat against the PCIe riser rear panel opening.

c. Close the hinged card retainer over the rear panel tab of the card and push in on the retainer until its clip clicks into place to secure the card.

d. Position the PCIe riser over its socket on the motherboard and over its alignment features in the chassis (see Figure 3-22).

e. Carefully push down on both ends of the PCIe riser to fully engage its circuit board connector with the socket on the motherboard.

f. Connect cables to the PCIe card. See RAID Controller Considerations for more information about supported cards and cabling.

g. Replace the top cover.

h. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

i. If you replaced a RAID controller card, continue with Restoring RAID Configuration After Replacing a RAID Controller.

Figure 3-24 PCIe Riser card Retainers (Slot 5 on PCIe Riser 2 Shown)

Special Considerations for Cisco UCS Virtual Interface Cards

Table 3-8 describes the requirements for the supported Cisco UCS virtual interface cards (VICs).

Special Considerations for Cisco UCS Fusion ioDrive2 Storage Accelerator Cards

Table 3-9 describes the requirements for the supported Cisco UCS Fusion ioDrive2 cards.

Enabling Higher Power-Draw for Fusion ioDrive2 Cards

Some Fusion ioDrive2 accelerator cards such as the 3.0 TB card draw 25 W by default, but can achieve higher performance when the PCIe slot is configured to provide more than 25 W for the device. This topic describes the slots that are capable of providing more than 25W and the commands to enable that higher power draw.

PCIe slots that can be configured to support more than 25 W power draw: PCIE1, PCIE3, PCIE5.

These three slots can support the higher device power draw of 55 W (75 W maximum) when configured with the override parameter in the IO Accelerator VSL software.

How to Enable Higher Power Draw For a Device With the IO Accelerator Override Parameter

The override parameter in the IO Accelerator VSL software (in the /usr/modprove.d/iomemory-vsl.conf file) overrides the setting that prevents devices from drawing more than 25 W from the PCIe slot. The parameter is enabled by device, using the device serial numbers.

Note The override parameter persists in the server and enables full power draw on an enabled device even if the device is removed and then placed in a different slot within the same system. If the device is moved to a slot that is not rated to provide 55 W of power, the server hardware could experience a power drag.

Note The override parameter is a setting for the IO Accelerator VSL software by server and is not stored in the device. When moved to a new server, the device defaults to the 25 W power limit until an external power cable is added or the override parameter is enabled for that device in the new server.

Use the following procedure to enable the override parameter for a Fusion device that is installed in a slot capable of >25 W power draw:

Step 1 Determine the serial number of the installed Fusion device by using the fio-status command in the IO Accelerator VSL software. Sample output is shown here:

In this example, 1149D0969 is the adapter serial number. If you have multiple Fusion ioDrive2 devices installed in your system, use the fio-beacon command to verify where each device is physically located.

Note There is also a serial number label on the card, but it is recommended that you use the fio-status command to confirm the serial number is an adapter serial number (FIO SN in the sample output above).

Step 2 Set the override parameter by editing the /usr/modprove.d/iomemory-vsl.conf file, which is installed when you install the device drivers.

Change the value for the external_power_override parameter, where the <value> for this parameter is a comma-separated list of adapter serial numbers.

For example:

Step 3 Reboot the server or unload and then load the drivers to enforce the parameter changes.

RAID Controller Card Cable Routing

If the PCIe card that you are installing or replacing is a RAID controller card, see RAID Controller Considerations for cable routing and other guidelines.

Installing Multiple PCIe Cards and Resolving Limited Resources

When a large number of PCIe add-on cards are installed in the server, the system may run out of the following resources required for PCIe devices:

• Option ROM memory space
• 16-bit I/O space

The topics in this section provide guidelines for resolving the issues related to these limited resources.

• Resolving Insufficient Memory Space to Execute Option ROMs
• Resolving Insufficient 16-Bit I/O Space

Resolving Insufficient Memory Space to Execute Option ROMs

The system has very limited memory to execute PCIe legacy option ROMs, so when a large number of PCIe add-on cards are installed in the server, the system BIOS might not able to execute all of the option ROMs. The system BIOS loads and executes the option ROMs in the order that the PCIe cards are enumerated (Slot 1, Slot 2, Slot 3, etc.).

If the system BIOS does not have sufficient memory space to load any PCIe option ROM, it skips loading that option ROM, reports a system event log (SEL) event to the CIMC controller and reports the following error in the Error Manager page of the BIOS Setup utility:

To resolve this issue, disable the Option ROMs that are not needed for system booting. The BIOS Setup Utility provides the setup options to enable or disable the Option ROMs at the PCIe slot level for the PCIe expansion slots and at the port level for the onboard NICs. These options can be found in the BIOS Setup Utility Advanced → PCI Configuration page.

• Guidelines for RAID controller booting:

If the server is configured to boot primarily from RAID storage, make sure that the option ROMs for the slots where your RAID controllers installed are enabled in the BIOS, depending on your RAID controller configuration.

If the RAID controller does not appear in the system boot order even with the option ROMs for those slots are enabled, the RAID controller option ROM might not have sufficient memory space to execute. In that case, disable other option ROMs that are not needed for the system configuration to free up some memory space for the RAID controller option ROM.

See also Factory-Default Option ROM Settings.

• Guidelines for onboard NIC PXE booting:

If the system is configured to primarily perform PXE boot from onboard NICs, make sure that the option ROMs for the onboard NICs to be booted from are enabled in the BIOS Setup Utility. Disable other option ROMs that are not needed to create sufficient memory space for the onboard NICs.

Resolving Insufficient 16-Bit I/O Space

The system has only 64 KB of legacy 16-bit I/O resources available. This 64 KB of I/O space is divided between the CPUs in the system because the PCIe controller is integrated into the CPUs. This server BIOS has the capability to dynamically detect the 16-bit I/O resource requirement for each CPU and then balance the 16-bit I/O resource allocation between the CPUs accordingly during the PCI bus enumeration phase of the BIOS POST.

When a large number of PCIe cards are installed in the system, the system BIOS might not have sufficient I/O space for some PCIe devices. If the system BIOS is not able to allocate the required I/O resources for any PCIe devices, the following symptoms have been observed:

• The system might get stuck in an infinite reset loop.
• The BIOS might appear to hang while initializing PCIe devices.
• The PCIe option ROMs might take excessive time to complete, which appears to lock up the system.
• PCIe boot devices might not be accessible from the BIOS.
• PCIe option ROMs might report initialization errors. These errors are seen before the BIOS passes control to the operating system.
• The keyboard might not work.

To work around this problem, rebalance the 16-bit I/O load using the following methods:

1. Physically remove any unused PCIe cards.

2. If the system has one or more Cisco virtual interface cards (VICs) installed, disable the PXE boot on the VICs that are not required for the system boot configuration by using the Network Adapters page in the CIMC WebUI to free up some 16-bit I/O resources. Each VIC uses a minimum 16 KB of 16-bit I/O resource, so disabling PXE boot on Cisco VICs would free up some 16-bit I/O resources that can be used for other PCIe cards that are installed in the system.

Installing an NVIDIA GRID or Tesla GPU Card

Note After you replace a component such as an adapter card or a RAID controller card, use the Cisco Host Upgrade Utility to update all Cisco components to compatible firmware levels (Cisco Host Upgrade Utility User Guides). If you install third-party components, install the firmware or drivers supplied by the third-party vendor.

This section contains the following topics:

• Overview of Software Requirements
• Configuration Rules
• How to Determine Your Server Version
• Installing the GPU Card in Version 01 Servers
• Installing the GPU Card in Version 02 and Later Servers
• Installing Drivers to Support the NVIDIA GPU Cards

Overview of Software Requirements

Server Firmware

Table 3-10 lists the minimum server firmware versions for the GPU cards.

Note The NVIDIA Tesla K40 GPU card is not supported for management by Cisco UCS Manager at this time.

Configuration Rules

The rules for configuring the server with GPUs differ, depending on the server version and other factors. This section includes the following topics:

• General NVIDIA GPU Card Configuration Rules
• NVIDIA GRID GPU Slot Population Rules
• NVIDIA Tesla GPU Slot Population Rules

General NVIDIA GPU Card Configuration Rules

• Do not mix different GPU card types (GRID, Tesla) in the same server. (You can mix GRID K1 and GRID K2 GPU cards in the same server.)
• All GPU cards require two CPUs and two 1200 W power supplies in the server.
• It is not possible to use dual NVIDIA GPU cards and a Cisco virtual interface card (VIC) at the same time. This is because dual NVIDIA GPUs must be installed in slots 2 and 5 of the server, and a Cisco VIC must be installed in either slot 2 or slot 5. If you require two GPU cards and 10-Gb Ethernet connectivity, you must chooses a different supported adapter that can be used in a different slot. For supported adapters, see the Technical Specifications Sheet for the Cisco UCS C240 M3 server (Small Form Factor or Large Form Factor) at: http://www.cisco.com/en/US/products/ps10493/products_data_sheets_list.html

NVIDIA Tesla K40 Requirements

The NVIDIA Tesla K40 GPU card requires the following minimum items:

• Intel E5-2600 v2 Series CPUs in the server
• Cisco IMC and BIOS firmware minimum 1.5(7) or later.

NVIDIA GRID GPU Slot Population Rules

Table 3-11 shows the rules for populating NVIDIA GRID K1 or K2 GPU cards in the server.

NVIDIA Tesla GPU Slot Population Rules

Table 3-12 shows the rules for populating NVIDIA Tesla K20, K20X, and K40 GPU cards in the server.

Note The NVIDIA Tesla K40 GPU card requires Intel E5-2600 v2 Series CPUs in the server.

How to Determine Your Server Version

The version of your server can be seen in the version ID on the PID/VID label. The PID/VID label is on the top of the server (see Figure 3-25).

• A version ID with “V01” designates version 01 of the server.
• A version ID with “V02” or later designates version 02 or later of the server.

V02 and later versions of the server support GPU card installation without the GPU upgrade kit.

Figure 3-25 Label and Version ID

Installation Procedures

Use one of the following two procedures, depending on your server version:

• If you have a version 01 server, use the procedure in Installing the GPU Card in Version 01 Servers. You are required to upgrade the server with a GPU upgrade kit before you can install the GPU card.
• If you have a version 02 or later server, use the procedure in Installing the GPU Card in Version 02 and Later Servers. In this case, the GPU upgrade kit is not required.

See How to Determine Your Server Version if you do not know your server version.

Installing the GPU Card in Version 01 Servers

The procedure in this section requires that you use the GPU upgrade kit to ready the server for GPU card installation.

• GPU Upgrade Kit
• Installation Procedure

GPU Upgrade Kit

The GPU upgrade kit includes the following components (see Figure 3-26):

• Two replacement PCIe risers
• One replacement chassis mid-brace
• One 8-to-8 pin power-cable adapter (connects between the motherboard and the GPU power cable)
• One straight GPU power cable (for a single GPU)
• One “Y” GPU power cable (for dual GPUs)

Figure 3-26 Cisco UCS C240 GPU Kit

Installation Procedure

Summary of procedure steps:

1. Prepare the server for component installation.

2. Replace the chassis mid-brace.

3. Connect the 8-to-8 pin power cable adapter and the power cable to the motherboard.

4. Install the GPU card into the new replacement riser.

5. Install the new replacement riser into the chassis.

6. Connect the power cable to the GPU card.

Use the following procedure to make the necessary chassis and cabling changes to a version 01 server before you install the GPU card.

Step 1 Prepare the server for component installation:

a. Shut down and power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the top cover as described in the “Removing and Replacing the Server Top Cover” section.

d. Remove the air baffle that covers the CPUs and DIMMs.

Step 2 Replace the chassis mid-brace:

a. Remove the existing PCIe risers. Use the finger holes in each PCIe riser to lift straight up on both ends. Set the risers on an antistatic mat.

b. Remove the existing chassis mid-brace. Compress the finger-latches on each end of the mid-brace toward the center, then lift it straight up (see Figure 3-27).

c. Install the new chassis mid-brace. Set the mid-brace in place while compressing the finger-latches on each end toward the center. Release the latches to lock the mid-brace into position.

Figure 3-27 Removing the Chassis Mid-Brace and PCIe Risers

Step 3 Connect the 8-to-8 pin power cable adapter to the motherboard connector GPU PWR (see Figure 3-28).

Figure 3-28 GPU PWR Motherboard Connector and PCIe Riser Alignment Slots

Step 4 Connect a GPU power cable to the free end of the 8-to-8 pin adapter (see Figure 3-29):

• If you are installing only one GPU, use the straight GPU power cable.
• If you are installing two GPU cards, use the “Y” GPU power cable.

Note Do not connect the power cable to the GPU card at this point.

Figure 3-29 Cabling Diagram, Single- and Dual-GPU Card Examples

Note See Configuration Rules before you install the GPU cards to the risers. Slot population restrictions apply.

Step 5 Install your first GPU card into PCIe slot 5. See Figure 3-30 for the riser and slot locations.

Note The option ROM for the slot in which you are installing the GPU card must be enabled in the BIOS Setup Utility or it will not be recognized by the system.

Figure 3-30 Rear Panel, Showing PCIe Risers and PCIe Slots

a. Open the riser’s hinged card retainer and the long-card retainer (see Figure 3-31).

b. Align the GPU card with the socket on the riser, then gently push the card’s edge connector into the socket. Press evenly on both corners of the card to avoid damaging the connector.

c. Close the hinged card retainer, then the long-card retainer over the end of the card.

Figure 3-31 PCIe Riser (Slot 5 on PCIe Riser 2 Shown)

Step 6 If you are installing a second GPU card, repeat the actions in Step 5 for PCIe slot 2 on the second riser.

Note When a GPU card is in slot 2, slot 1 is blocked and unavailable for use.

Step 7 Install the new PCIe risers:

a. Install any other PCIe cards that you want to install into the new risers.

Note If you are installing a Cisco UCS Virtual Interface Card (VIC), see the slot restrictions in Special Considerations for Cisco UCS Virtual Interface Cards.

b. Install riser 2 first. Position the PCIe riser over its socket on the motherboard and over its alignment slots in the chassis (see Figure 3-28).

Note It is easier to view and position the riser 2 connector over its socket with riser 1 removed.

c. Carefully push down on both ends of the PCIe riser to fully engage its circuit board connector with the socket on the motherboard. The riser top surfaces should sit flat on the chassis and the mid-brace.

d. Install riser 1.

Step 8 Connect the power cable to the GPU cards. See Figure 3-29.

• If you installed one GPU card, connect the free end of the straight cable to the GPU card power connector that is underneath the chassis mid-brace.
• If you installed two GPU cards, connect the shorter branch of the Y cable to the GPU card in slot 5. Connect the longer branch to the GPU card in slot 2.

Note The power connector on the NVIDIA GRID K1 GPU card has only 6 pins, but the power cable has an 8-pin connector. Plug the 8-pin cable connector into the 6-pin connector on the card, aligning the clip and keying features for the correct fit. Pins 4 and 8 on the cable connector do not engage with the connector on the card.

Note The NVIDIA GRID K2 GPU card has a 6-pin power connector and an 8-pin connector. When you connect the power cable, use the 8-pin connector on the card.

Note There are cable clips on the underside of the chassis mid-brace through which you can route the power cable.

Step 9 Install the air baffle.

Step 10 Replace the top cover.

Step 11 Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Step 12 Continue with Installing Drivers to Support the NVIDIA GPU Cards.

Installing the GPU Card in Version 02 and Later Servers

Use the following procedure to install a GPU card into a version 02 or later server, for which the GPU upgrade kit is not required. For these newer servers, the new risers and mid-brace are already installed and the power cable adapter is not required because the motherboard connector is already updated.

Step 1 Prepare the server:

a. Shut down and power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the air baffle that covers the CPUs and DIMMs.

Step 2 Remove the PCI risers:

a. Use the finger holes in each PCIe riser to lift straight up on both ends. Set the risers on an antistatic mat.

b. Remove any existing cards from the PCIe slots that you will use for GPUs (slot 5 or slot 2).

Note See Configuration Rules before you install the GPU cards to the risers. Slot population restrictions apply.

Step 3 Install your first GPU card into PCIe slot 5. See Configuration Rules when planning which slots to use for your card. See Figure 3-32 for the riser and slot locations.

Note The option ROM must be enabled in the BIOS Setup Utility for the slot in which you are installing the GPU card or it will not be recognized by the system.

Figure 3-32 Rear Panel, Showing PCIe Risers and PCIe Slots

a. Open the riser’s hinged card retainer and the long-card retainer (see Figure 3-33).

b. Align the GPU card with the socket on the riser, then gently push the card’s edge connector into the socket. Press evenly on both corners of the card to avoid damaging the connector.

c. Close the hinged card retainer, then the long-card retainer over the end of the card.

Figure 3-33 PCIe Riser (Slot 5 on PCIe Riser 2 Shown)

Step 4 If you are installing a second GPU card, repeat the actions in Step 3 for PCIe slot 2 on the second riser.

Note When a GPU card is in slot 2, slot 1 is blocked and unavailable for use.

Step 5 Install the new PCIe risers:

a. Install any other PCIe cards that you want to install into the new risers.

Note If you are installing a Cisco UCS Virtual Interface Card (VIC), see the slot restrictions in Special Considerations for Cisco UCS Virtual Interface Cards.

b. Install riser 2 first. Position the PCIe riser over its socket on the motherboard and over its alignment slots in the chassis (see Figure 3-34).

Note It is easier to view and position the riser 2 connector over its socket with riser 1 removed.

c. Carefully push down on both ends of the PCIe riser to fully engage its circuit board connector with the socket on the motherboard. The riser top surfaces should sit flat on the chassis and the mid-brace.

d. Install riser 1.

Step 6 Connect the GPU power cable to motherboard connector GPU PWR (see Figure 3-34).

• If you installed one GPU card, connect the white end of the straight cable to GPU PWR.
• If you installed two GPU cards, connect the white end of the Y cable to GPU PWR.

Step 7 Connect the power cable to the GPU cards (see Figure 3-35).

• If you installed one GPU card, connect the free end of the straight cable to the GPU card power connector that is underneath the chassis mid-brace.
• If you installed two GPU cards, connect the shorter branch of the Y cable to the GPU card in slot 5. Connect the longer branch to the GPU card in slot 2.

Note The power connector on the NVIDIA GRID K1 GPU card has only 6 pins, but the power cable has an 8-pin connector. Plug the 8-pin cable connector into the 6-pin connector on the card, aligning the clip and keying features for the correct fit. Pins 4 and 8 on the cable connector do not engage with the connector on the card.

Note The NVIDIA GRID K2 card has a 6-pin power connector and an 8-pin connector. When you connect the power cable, use the 8-pin connector on the card.

Note There are cable clips on the underside of the chassis mid-brace through which you can route the power cable.

Figure 3-34 GPU Motherboard Connector and PCIe Riser Alignment Slots

Figure 3-35 Cabling Diagram, Single- and Dual-GPU Card Examples

Step 8 Replace the air baffle.

Step 9 Replace the top cover.

Step 10 Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Step 11 Continue with Installing Drivers to Support the NVIDIA GPU Cards.

Installing Drivers to Support the NVIDIA GPU Cards

After you install the hardware, you must update to the correct level of server BIOS and then install NVIDIA drivers and other software.

• 1. Update the C240 M3 Server BIOS
• 2. Update the NVIDIA Drivers

1. Update the C240 M3 Server BIOS

Table 3-13 lists the minimum server firmware versions for the GPU cards.

Install the latest Cisco UCS C240 server BIOS by using the Host Upgrade Utility for the Cisco UCS C240 M3 server.

Step 1 Navigate to the following URL: http://www.cisco.com/cisco/software/navigator.html .

Step 2 Click Servers–Unified Computing in the middle column.

Step 3 Click Cisco UCS C-Series Rack-Mount Standalone Server Software in the right-hand column.

Step 4 Click the name of your model of server in the right-hand column.

Step 5 Click Unified Computing System (UCS) Server Firmware.

Step 6 Click the release number.

Step 7 Click Download Now to download the ucs-server platform-huu-version_number.iso file.

Step 8 Verify the information on the next page, then click Proceed With Download.

Step 9 Continue through the subsequent screens to accept the license agreement and browse to a location where you want to save the file.

Step 10 Use the Host Upgrade Utility to update the server BIOS.

The user guides for the Host Upgrade Utility are at: Utility User Guides .

2. Update the NVIDIA Drivers

After you update the server BIOS, you can install NVIDIA drivers to your hypervisor virtual machine.

Step 1 Install your hypervisor software on a computer. Refer to your hypervisor documentation for the installation instructions.

Step 2 Create a virtual machine in your hypervisor. Refer to your hypervisor documentation for instructions.

Step 3 Install the NVIDIA drivers to the virtual machine. Download the drivers from http://www.nvidia.com/Download/index.aspx .

Step 4 Restart the server.

Step 5 Check that the virtual machine is able to recognize the NVIDIA card. In Windows, use the Device Manager and look under Display Adapters.

Resynching the Hypervisor Partition

When one member slot of the SD card pair is replaced, use this option to synchronize the Hypervisor data across two members of the RAID-1 virtual disk. You can initiate this synchronization only if two cards are detected and RAID-1 is determined as unhealthy (one member is corrupt).

When one member slot of the SD card pair is replaced, use this option to synchronize the Hypervisor data across two members of the RAID-1 virtual disk. You can initiate this synchronization only if two cards are detected and RAID-1 is determined as unhealthy (one member is corrupt).

Step 1 Log into the Cisco IMC management interface for the server. Use a browser and enter the IP address that has been assigned to the server.

Step 2 In the Cisco IMC, navigate to Server Inventory > Storage > Controller Info .

Step 3 In the Cisco IMC, click Synchronize Card Configuration .

Step 4 In the Cisco IMC, navigate to Server Inventory > Storage > Physical Drive Info and verify that the replaced card shows Health status Unhealthy and RAID Role as Secondary Active .

Step 5 Launch the Cisco UCS Server Configuration Utility (SCU).

For instructions on downloading the utility ISO file and launching the utility, see the user guides at Cisco UCS Server Configuration Utility User Guides .

Step 6 Use the SCU to synchronize the hypervisor partition.

a. In SCU, click the Hypervisor Sync icon on the toolbar.

A dialog box prompts you to confirm that you want to synchronize the hypervisor RAID.

b. Click Yes .

When the synchronization is complete, a dialog box indicating the completion of the process is displayed.

c. Click OK .

The Hypervisor Sync icon on the toolbar is greyed out.

Step 7 Unmount SCU and reboot the server:

a. In SCU, click the Reboot icon on the toolbar.

The Reboot dialog appears.

b. Click Yes to reboot.

c. In the Cisco IMC KVM window, unmount the SCU ISO.

d. Allow the server to reboot normally.

Replacing the SuperCap Power Module

This server supports installation of two RAID SuperCap Power Modules (SCPMs). The units mount to clips on the removable air baffle (see Figure 3-37).

The SCPM provides approximately 3 years of backup for the disk write-back cache DRAM in the case of sudden power loss by offloading the cache to the NAND flash.

Warning There is danger of explosion if the battery is replaced incorrectly. Replace the battery only with the same or equivalent type recommended by the manufacturer. Dispose of used batteries according to the manufacturer’s instructions.
Statement 1015

To replace the RAID controller backup unit, follow these steps:

Step 1 Remove a backup unit (see Figure 3-37).

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the top cover as described in the “Removing and Replacing the Server Top Cover” section.

d. Disconnect the cable that is attached to the existing backup unit.

e. Slide the backup unit free of the clips on the air baffle mounting point (see Figure 3-37).

Step 2 Install a new backup unit:

a. Slide the new backup unit into the clips on the air baffle mounting point.

b. Connect the cable from the RAID controller to the new backup unit.

c. Replace the top cover.

d. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Figure 3-37 Replacing a BBU or SCPM RAID Backup Unit

Installing a Trusted Platform Module

The trusted platform module (TPM) is a small circuit board that attaches to a motherboard socket. The socket location is on the motherboard, underneath PCIe riser 1 (see Figure 3-38).

Note For security purposes, the TPM is installed with a one-way screw. It cannot be removed with a standard screwdriver.

To install a trusted platform module (TPM), follow these steps:

Step 1 Prepare the server for component installation.

a. Power off the server as described in the “Shutting Down and Powering Off the Server” section.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the top cover as described in the “Removing and Replacing the Server Top Cover” section.

d. Is there a card installed in PCIe riser 1? See Figure 3-38.

• If there is no card installed in PCIe riser 1, you can access the TPM socket. Go to Step 2.
• If there is a card installed in PCIe riser 1, you must remove the riser. Use the finger holes on the top of PCIe riser 1 to lift straight up on both ends and disengage its circuit board from the socket on the motherboard. Set the riser aside on an antistatic mat.

Step 2 Install a TPM (see Figure 3-38):

a. Locate the TPM socket on the motherboard, as shown in Figure 3-38.

b. Align the connector that is on the bottom of the TPM circuit board with the motherboard TPM socket. Align the screw hole and standoff on the TPM board with the screw hole adjacent to the TPM socket.

c. Push down evenly on the TPM to seat it in the motherboard socket.

d. Install the single one-way screw that secures the TPM to the motherboard.

e. If you removed PCIe riser 1, replace it now.

f. Replace the top cover.

g. Replace the server in the rack, replace cables, and then power on the server by pressing the Power button.

Step 3 Enable the TPM:

Note You must set a BIOS Administrator password before performing this procedure. To set this password, press the F2 key when prompted during system boot to enter the BIOS Setup utility. Then navigate to Security > Set Administrator Password and enter the new password twice as prompted.

a. Watch during bootup for the F2 prompt, and then press F2 to enter BIOS setup.

b. Log into the BIOS Setup utility with your BIOS Administrator password.

c. On the BIOS Setup utility screen, select the Advanced tab.

d. Select Trusted Computing to open the TPM Security Device Configuration screen.

e. Change TPM SUPPORT to Enabled .

f. Press F10 to save your settings and reboot the server.

Step 4 Verify that the TPM is now enabled.

a. Watch during bootup for the F2 prompt, and then press F2 to enter BIOS setup.

b. Log into the BIOS Setup utility with your BIOS Administrator password.

c. Select the Advanced tab.

d. Select Trusted Computing to open the TPM Security Device Configuration screen.

e. Verify that TPM SUPPORT is Enabled.

Note If you want to use the Intel Trusted Execution Technology (TXT) feature, it must be enabled in the server BIOS as described in Enabling the Intel Trusted Execution Technology (TXT) Feature For the TPM.

Figure 3-38 TPM Socket Location on Motherboard

Enabling the Intel Trusted Execution Technology (TXT) Feature For the TPM

Intel TXT provides greater protection for information that is used and stored on the business server. A key aspect of that protection is the provision of an isolated execution environment and associated sections of memory where operations can be conducted on sensitive data, invisibly to the rest of the system. Likewise, Intel TXT provides for a sealed portion of storage where sensitive data such as encryption keys can be kept, helping to shield them from being compromised during an attack by malicious code.

Note You must set a BIOS Administrator password before performing this procedure. To set this password, press the F2 key when prompted during system boot to enter the BIOS Setup utility. Then navigate to Advanced > Security > Set Administrator Password and enter the new password twice as prompted.

To enable the TXT feature, follow these steps:

Step 1 Verify that a TPM is now installed and enabled in the server:

a. Either attach a VGA monitor and USB keyboard to the server, or log in remotely to the CIMC interface of the server and open a virtual KVM console window.

b. Reboot the server.

c. Watch during bootup for the F2 prompt, and then press F2 to enter BIOS setup.

d. Log in to the BIOS Setup utility with your BIOS Administrator password.

Note You must be logged in as the BIOS administrator to perform this procedure. If you have not done so already, set a BIOS administrator password on the Security tab of the BIOS Setup utility.

e. Select the Advanced tab.

f. On the Advanced tab, select Trusted Computing to open the TPM Security Device Configuration screen.

g. Verify that TPM SUPPORT is Enabled . If it is not, set TPM SUPPORT to Enabled.

h. Press Escape to return to the BIOS Setup utility Advanced tab.

Step 2 Enable the Intel Trusted Execution Technology (TXT) feature:

a. On the Advanced tab, select Intel TXT(LT-SX) Configuration to open the Intel TXT(LT-SX) Hardware Support screen.

Note The Intel Trusted Execution Technology feature can be enabled only when the server has a TPM installed on the TPM header.

b. Set TXT Support to Enabled .

Step 3 On the same screen, verify that the Intel Virtualization Technology (VT) and the Intel VT for Directed I/O (VT-d) features are enabled (the factory default).

a. On the Intel TXT(LT-SX) Hardware Support screen, verify that VT-d Support and VT Support are both listed as Enabled .

– If they are already enabled, skip to Step 4.

– If VT-d Support and VT Support are not enabled, continue with the next steps to enable them.

b. Press Escape to return to the BIOS Setup utility Advanced tab.

c. On the Advanced tab, select Processor Configuration to open the Processor Configuration screen.

d. Set Intel (R) VT and Intel (R) VT-d to Enabled.

Step 4 Press F10 to save your changes and exit the BIOS Setup utility.

Step 5 Verify that the Intel TXT, VT, and VT-d features are enabled:

a. Reboot the server.

b. Watch during bootup for the F2 prompt, and then press F2 to enter BIOS setup.

c. Select the Advanced tab.

d. Select Intel TXT(LT-SX) Configuration and verify that TXT Support, VT-d Support, and VT Support are Enabled.

Replacing a SCU Upgrade ROM Module

To remove and replace a module, use the following procedure.

Step 1 Prepare the server for component installation:

a. Power off the server as described in Shutting Down and Powering Off the Server.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the top cover as described in Removing and Replacing the Server Top Cover.

Step 2 Remove the SCU upgrade ROM module:

a. Locate the module on the motherboard (see Figure C-1).

b. Grasp the printed circuit board of the module and lift it from the header.

Note The module has a small retention feature that must have clearance from the header before the module can be pulled up. Tilt the module back, then pull up as shown in Figure 3-39.

Step 3 Install a new SCU upgrade ROM module:

a. Align the module with the pins in the motherboard header.

b. Gently press down on the module until it is seated and the retention feature locks into the header.

Figure 3-39 SCU Upgrade ROM Module Retention Feature

For more information about using the module and embedded RAID, see Embedded MegaRAID Controller.

Replacing a Software RAID Key Module

To remove and replace a software RAID key module, use the following procedure.

Step 1 Prepare the server for component installation:

a. Power off the server as described in Shutting Down and Powering Off the Server.

b. Slide the server out the front of the rack far enough so that you can remove the top cover. You might have to detach cables from the rear panel to provide clearance.

c. Remove the top cover as described in Removing and Replacing the Server Top Cover.

Step 2 Remove the software RAID key module:

a. Locate the module on the motherboard (see Figure C-1).

b. Grasp the printed circuit board of the module and lift it from the header.

Note The module must have clearance from the retention clip on the header before the module can be pulled up. Open the retention clip, then pull up as shown in Figure 3-40.

Step 3 Install a new software RAID key module:

a. Align the module with the pins in the motherboard header.

b. Gently press down on the module until it is seated and the retention clip locks over the module.

Figure 3-40 Software RAID Key Module Retention Feature

For more information about using the module and embedded RAID, see Embedded MegaRAID Controller.

Replacing Power Supplies

The server can have one or two power supplies. When two power supplies are installed they are redundant as 1+1.

• See Power Specifications for more information about the power supplies.
• See Rear Panel LEDs and Buttons for information about the power supply LEDs.
• See Wiring a DC Power Supply for information about wiring a DC power supply.

To replace or install a power supply, follow these steps:

Note If you have ordered a server with power supply redundancy (two power supplies), you do not have to power off the server to replace power supplies because they are redundant as 1+1.

Note Do not mix power supply types in the server. Both power supplies must be either 650W or 1200W.

Step 1 Remove the power supply that you are replacing or a blank panel from an empty bay (see Figure 3-41):

a. Perform one of the following actions:

– If your server has only one power supply, shut down and power off the server as described in the “Shutting Down and Powering Off the Server” section.

– If your server has two power supplies, you do not have to shut down the server.

b. Remove the power cord from the power supply that you are replacing.

For a DC power supply, release the electrical connector block from the power supply by pushing the orange plastic button on the top of the connector inward toward the power supply. Pull the connector block from the power supply.

c. Grasp the power supply handle while pinching the release lever towards the handle.

d. Pull the power supply out of the bay.

Step 2 Install a new power supply:

a. Grasp the power supply handle and insert the new power supply into the empty bay.

b. Push the power supply into the bay until the release lever locks.

c. Connect the power cord to the new power supply.

For a DC power supply, push the electrical connector block into the power supply.

Note For a DC power supply, see

d. If you shut down the server, press the Power button to return the server to main power mode.

Figure 3-41 Removing and Replacing Power Supplies

Wiring a DC Power Supply

Warning A readily accessible two-poled disconnect device must be incorporated in the fixed wiring. Statement 1022

Warning This product requires short-circuit (overcurrent) protection, to be provided as part of the building installation. Install only in accordance with national and local wiring regulations. Statement 1045

Warning When installing or replacing the unit, the ground connection must always be made first and disconnected last. Statement 1046

Warning Installation of the equipment must comply with local and national electrical codes. Statement 1074

Warning Hazardous voltage or energy may be present on DC power terminals. Always replace cover when terminals are not in service. Be sure uninsulated conductors are not accessible when cover is in place. Statement 1075

Note The recommended wire gauge is 8 AWG. The minimum wire gauge is 10 AWG.

Use the following procedure to connect wires to the DC power supply connector:

Step 1 Turn off the DC power source from your facility’s circuit breaker to avoid electric shock hazard.

Step 2 Remove the DC power connector block from the power supply. (The spare PID for this connector is UCSC-CONN-930WDC=.)

To release the connector block from the power supply, push the orange plastic button on the top of the connector inward toward the power supply, and then pull the connector block out.

Step 3 Strip 15mm (.59 inches) of insulation off the DC wires that you will use.

Step 4 Orient the connector as shown in Figure 3-42, with the orange plastic button toward the top.

Step 5 Use a small screwdriver to depress the spring-loaded wire retainer lever on the lower spring-cage wire connector. Insert your green (ground) wire into the aperture and then release the lever.

Step 6 Use a small screwdriver to depress the wire retainer lever on the middle spring-cage wire connector. Insert your black (DC negative) wire into the aperture and then release the lever.

Step 7 Use a small screwdriver to depress the wire retainer lever on the upper spring-cage wire connector. Insert your red (DC positive) wire into the aperture and then release the lever.

Step 8 Insert the connector block back into the power supply. Make sure that your red (DC positive) wire aligns with the power supply label, “+ DC”.

Figure 3-42 930 W, –48 VDC Power Supply Connector Block

Enabling or Disabling the Internal USB Port

The factory default is for all USB ports on the server to be enabled. However, the internal USB port can be enabled or disabled in the server BIOS. To enable or disable the internal USB port, follow these steps:

Step 1 Enter the BIOS Setup utility by pressing the F2 key when prompted during bootup.

Step 2 Navigate to the Advanced tab.

Step 3 On the Advanced tab, select USB Configuration.

Step 4 On the USB Configuration page, select USB Ports Configuration.

Step 5 Scroll to USB Port: Internal, press Enter, and then select either Enabled or Disabled from the pop-up menu.

Step 6 Press F10 to save and exit the utility.