Cisco UCS C480 ML Rack Server: Power and Thermal Design White Paper
Available Languages
Summary
The thermal design of the Cisco UCS® C480 ML Rack Server involves ultra-dense packing of the several extremely high-power components with an objective of meeting all of the components’ temperatures specifications in the most-energy efficient manner. The chassis is designed to support highest-power and highest-performance Intel® Skylake and Cascade Lake CPUs up to 205W each, and highest-power NVIDIA Tesla SXM2 GPUs up to 300W each. The design objective is to meet the ASHRAE A2 Class ambient temperature of 35°C without any of the components throttling from over-temperature conditions. This paper explains the engineering effort that went into the design and testing of the C480 ML server. Mechanical design, powers, airflows, and component temperatures are detailed.
Introduction
The Cisco UCS C480 ML Rack Server is a 4U chassis designed to fit industry standard 19 in racks. The chassis is purpose built for deep learning. Some of the key components of the C480 ML are:
· 8 NVIDIA SXM2 V100-32GB Tensor Core modules with NVLink interconnect
· Latest Intel Xeon® Scalable processors with up to 28 cores per socket and support for two processor configurations
· 24 DIMM slots for memory
· Modular storage support with up to 24 front-accessible hot-swappable Hard Disk Drives (HDDs) and Solid-State Disks (SSDs)
Power and cooling goals
The C480 ML is designed to support the highest-performance-available key components:
· NVIDIA SXM2 V100-32GB modules: up to 300W each
· Intel Xeon Scalable processors; up to 205W each
· DDR4 Memory (up to 128 GB each) up to 18W each
· HDDs: up to 25W each
The thermal design goal is to engineer the system to meet ASHRAE Thermal Guidelines to the A2 Environmental Class. Then the goal is to design the chassis to an operating ambient temperature of 35°C and an altitude of 950 m without any of the key components throttling under maximum possible load or stress conditions. Surely extended temperatures above 35°C and extended altitudes over 950 m will be supported – but, if necessary, the key components may throttle to support these extended environmental conditions.
Cisco UCS C480 ML mechanical and thermal design
Figures 1(a) to 1(c) provide an overview of the mechanical layout of the chassis. HDDs and CPU + memory modules are in the front of the chassis, followed by 4x system cooling fans; the 8x GPUs are in the rear of the chassis behind the fans. The 4x Power Supply Units (PSUs) and the I/O module are inserted from the chassis rear on the bottom side, as shown in the Figure 1(c). The GPUs are arranged in a 4x2 pattern – four upstream and four downstream in the airflow path. Figures 1(d) and 1(e) show the layout of the GPUs with and without the GPU heatsinks.
Airflow and cooling paths
The chassis airflow is provided by 4x state-of-the-art custom 92x76 mm counter-rotating fans uniquely developed for usage in this chassis. Counter-rotating fans, compared to single rotor fans, offer advantages of higher airflow performance, lower power consumption and higher efficiency, and better redundancy in case of fan failure.
Air inlet is from the front of the chassis and exhaust is at the rear of the chassis. Air enters the chassis through the CPU module and front-drive sections, is pressurized by the 4x system fans, and then splits into three streams: (a) primary flow through the GPUs, (b) secondary flow through the PSUs, and (c) a smaller flow through the I/O module. Figure 2(a) shows an isometric view of the chassis and the front-to-back airflow.
Figures 2(a) and 2(b) show views of the CPU and memory module. There are two CPU sockets and a total of 24 memory sockets. Figure 2(b) shows airflow that cools the CPUs and components inside the compute module. All heatsinks are custom designed by Cisco to meet the chassis cooling goals in the most energy efficient manner possible.
Figure 3(a) shows an isometric view of the front GPU heatsink and Figure 3(b) shows an isometric view of the rear GPU heatsink. One of the design requirements is that both front and rear GPU temperatures be identical. Due to the preheating of the air as it flows from the front to the rear GPU heatsinks, it is critical to maintain the rear GPU temperatures within the specifications without the GPU throttling. The front heatsink base makes thermal contact with the GPU core and the GPU memory. Six heat pipes are used to move the heat to a “secondary” base on the top of the heatsink. This design results in an extremely efficient design to meet the GPU temperature specifications. By design, the front GPU heatsink is of a shorter height than the rear GPU heatsink. Then, for the rear GPU heatsink, a “secondary” heatsink is built upon the top-side of the front GPU heatsink. The usage of this design scheme, and the design optimizations involved, result in nearly identical temperatures for all of the 8x GPUs.
Test data and reliability
Extensive thermal testing for temperature margins and thermal controls was done for the C480 ML chassis by Cisco. Test data demonstrates that cooling objectives for the chassis are met with positive thermal margins. Figure 4(a) shows measured data for the GPU temperatures. Per NVIDIA thermal specs, the GPUs’ temperature throttling is initiated at 80°C; all of the six 300W GPU temperatures are below the 80°C limit even at Tambient = 36°C. The data in Figure 4(a) is generated using an NVIDIA provided test suite, NVQual. The NVQual test comprises two portions, each running for 30 minutes. The first portion is called TDP, to maximize the power consumption on the GPU core; the second portion is call TDP, max, to maximize the power consumption of the FPU memory. Figure 4(b) shows data from a set of measurements where three loops of NVQual are run. GPU power and temperature data are included in this chart. Clearly, all GPU temperatures are below the design goal of 80°C, and the measured GPU powers are about 300W and 250W for the TDP and TDP, max portions. Figure 4(c) shows the measured chassis data for a test where network traffic stress is started first with the host off (the main fans are off), followed by with host on, then Intel’s PTUgen (Power Thermal Utility generator for CPU and memory) stressor is added, and finally NVIDIA NVQual is added. The system is placed in an environmental chamber, and the chamber temperature is varied from 15 to 35 to 15°C. This test represents extreme stress conditions, and Cisco does not foresee any real customer system or application coming close to these extreme test conditions. As is seen from the measured data, the GPU temperatures are within NVIDIA limits. The maximum measured system inlet AC power usage reached about 4000W during the test.
It is worth mentioning that Cisco uses best-in-class fan speed control algorithms to maximize energy efficiencies. Further, our algorithms react with very low latencies to prevent any throttling of components, and hence there is no performance loss that might result from oscillating loads.
Further information/references
ASHRAE TC 9.9 2011 Thermal Guidelines for Data Processing Environments – Expanded Data Center Classes and Usage Guidance.
Cisco UCS C480 ML Rack Server Data Sheet:
Cisco UCS C480 ML Rack Server Spec Sheet:
NVIDIA SXM2 Data Sheet:
Feedback