UCS C Integration Troubleshooting and Best Practices

Introduction

The purpose of this document is to describe integration, troubleshooting, and best practices for the Cisco Unified Computing System (UCS) C-Series product.

UCS C-Series Integration Basic Connectivity Overview

• Unified Computing System Manager (UCSM) integration was introduced in Version 1.4 of UCSM for small scale deployments with separate infrastructure for management.
• Starting with Version 2.0(2) Scalable Mainstream support for integrated servers begins.
• Integration and connectivity evolved with the release of new hardware and software in UCSM versions 2.1 and 2.2.
• Today there are 3 main connectivity modes supported for UCSM integrated C-Series rack Servers.
1. Dual Wire Management
2. Single Connect (a.k.a. Single Wire Management)
3. Direct Connect to the Fabric Interconnect

• Multiple PCIe Slots in the C-Series servers provide a variety of supported connectivity options

Prerequisite

Understanding of UCS B and C-series servers 

http://www.cisco.com/c/en/us/products/servers-unified-computing/index.html

Dual Wire Management with FEX

• Introduced in Version 2.0(2) and replaced previous topology

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c-series_integration/ucsm2-0-2/b_UCSM_202_C-Integration/b_UCSM_202_C-Integration_chapter_01.html#reference_D893866B8BDF451DB568FA67912E3317

Single Connect with FEX (a.k.a. Single Wire Management)

• Introduced in UCSM Version 2.1

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c-series_integration/ucsm2-1/b_UCSM2-1_C-Integration/b_UCSM2-1_C-Integration_chapter_010.html

Direct Connect to Fabric Interconnect without FEX

• Introduced in UCSM Version 2.2

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c-series_integration/ucsm2-2/b_C-Series-Integration_UCSM2-2/b_C-Series-Integration_UCSM2-2_chapter_0110.html

For Cisco UCS C220 M4 and Cisco UCS C240 M4 servers you can use either Modular Lan On Motherboard (MLOM) or Virtual Interface Card (VIC) to integrate with Cisco UCS Manager in SingleConnect or direct connect mode. The adapter for connectivity is selected as explained below:

• MLOM only: When only MLOM is present in the server, the connectivity adapter is by default MLOM,
• MLOM and VIC adapters: MLOM is the default connectivity adapter.
• No MLOM:

• No MLOM and VIC: No SingleConnect or direct connect support.

Redundant PCIe Adapters Support

• Redundant adapters MUST be cabled so that each card is connected to both sides of the fabric in a cluster

Multiple connectivity types are supported within a single UCS Domain provided you are at the appropriate UCSM software level.

Requirements

Basics of Discovery

• By default a factory new UCS C-Series Server will discover automatically when plugged into a UCS Fabric using one of the supported connectivity models.
• You must meet the minimum requirements, but in general C-Series servers with Cisco Integrated Management Controller (CIMC) version 1.4(4) or higher can be discovered.
• Plug the server in the way that you want it to be discovered (e.g. Dual Wire or Single Wire).

• Verify that your server is at the minimum levels of required FW to support integration.

• By default management connectivity and server discovery are automatic, you can control this with Global Policies.

• Server and FEX discovery is a RACE condition. First one connected and powered on will be first discovered.

• FEX numbers are the same ID schema as Chassis. If you have 2 chassis then add 2 FEXs the numbers will be 3 and 4 respectively

Discovery Policies

• Global Discovery Policies are located in the Equipment Tab

• 3 Policies apply for C-Series Integration Discovery.

1: FEX Discovery Policy - # of links, Port-Channel vs. Pinned

2: Rack Discovery Policy - Immediate discovery and Inventory or wait for user action

3: Rack Management Connectivity Policy - Automatic discovery of Dual or Single Wire or require User input

Troubleshooting Methodology

User Acknowledged Server Discover Policy

• Server shows up in UCSM, but discovery (inventory and availability) does not occur.

• Overall Status shows as Inoperable until user Re-acknowledges server.

• Server number assigned even though inventory not completed.

User Acknowledged Management Connectivity Policy

• Management connection not established until user provides input.

• Server->Inventory->CIMC

• Select Connection type, only available connection types will show.

• Once management connection type is chosen it can not be changed without a rediscovery (Decom/Recom)

Hybrid Connection Display

• For UCS C-Series servers you can view the connection path for management and data through the Equipment -> Server->Hybrid Display tab.
• This can be useful for troubleshooting connectivity problems and verifying management connection types.

• Red links indicate that there is not an active VIF on the link. This is normal for Dual-Wire servers that are not active.

• If you reattach an integrated server to different ports than it was discovered on you must decom/recom the server for proper operations.

• Hovering the mouse over the link will provide additional detail.

Troubleshooting Discovery Problems

• Main Causes for Sever Discovering Properly
• CIMC Settings not default. (Most typical problem for servers not fresh out of the box)
• Discovery Policy or Management connection policy set set to User-Acknowledged
• Server Not in UCSM Catalog
• CIMC or VIC Firmware not at appropriate levels (CIMC Minimum for Discover 1.4(4).
• Improper Connectivity

Improper Cabling

• LOM and Fabric Adapter on different FEX

• LOM connected to FEX with VIC directly connected to Fabric.

Initial Firmware Policies

• Servers can be discovered as long as they are running an appropriate level of Firmware
• Version 1.4 and 2.0(1) would only run and discover with CIMC FW version 1.2(2l)
• Starting with Version 2.0(2) any CIMC FW after the minimum 1.4(4) would allow discovery.

• Only the FW packaged with an appropriate version of UCS Software bundles are supported with UCSM integration.

• After Discovery all servers should be flashed (upgraded/downgraded) to the supported bundle.

• Each version release note describes mixed version support.

Examining UCSM Firmware Bundles

vCon Placement policies

• An advantage of UCS C-Series servers is multiple PCIe Adapters.

• By Default UCS will automatically place Virtual Network Interface Card (vNIC) or Virtual Host Bus Adapter (vHBA) adapters onto physical adapters.

• A network Placement policy determines which physical adapter a specific vNIC and/or vHBA will be assigned to.

• When using multiple cards and card types it is important to manage the vNICs/vHBAs to achieve the desired network connectivity.

• VCons are abstractions of physical adapters and provide a method for consistent PCIe mappings for a service profile regardless of how many physical cards are installed.

• Setting the desired order for vNICs and vHBAs allows the system to automatically place them onto vCONs.

• When creating a service profile you can explicitly place the vNICs to provide deterministic connectivity.

• This is accomplished through vNIC/vHBA placement

There are 2 main reasons to manually specify vNIC/vHBA placement for C-Series integrated servers.

• Preventing automatic mapping to adapters that are unused by UCSM (e.g. C460 LOMs or Externally connected adapters)

• Traffic engineering (e.g. FCoE on one adapter and Ethernet on the other).

UCSM Local Disk Policies

Aside from PCIe slots the other major form factor decision for choosing a C-Series servers is the quantity and flexibility of drive configurations.

• Blade servers support a maximum of 4 drives where as C-Series rack mount servers support up to 24 drives today.

• UCSM provides management of local drive configurations through local disk policies which are part of the service profiles.

• The current implementation of local disk policies is limited in scope and does not yet meet some of the requirements for customers. To compensate users can continue to use tools outside of UCSM o customize drive configurations.

• The key to configuring local drives for a C-Series server is an understanding of local disk policies.

• Firmware management of controllers and drives is also achieved through service profiles.

• A local disk policy describes how a logical (virtual) drive will be configured on the raid controller using the resources available.

• Depending on the controller capabilities the options for local disk configuration include.
• RAID 0, 1, 5, 6, 10, 50, 60
• NO RAID
• Any Configuration
• No Local Storage

• Protect configuration prevents a service profile with a local disk policy that DOES NOT match the actual configuration on the controller from being applied.

• This will protect the data on an existing volume during Service Profile Mobility.

Best Practices

Best Practices with Discovery

• Use Auto-Acknowledge for Rack Server Discovery Policy even if you have a 160 server environment.
• If the server is Single Connect capable, wire for desired connectivity and use Auto-Acknowledge management policy.
• Note if you wire LOMs and SFPs, Shared-LOM will be the system default mode.
• If you remove the LOMs after shared-LOM is discovered you will lose management connectivity.
• When using more than 1 VIC in Single Connect mode the primary card slot will be used for management. If that card has a catastrophic failure, data can continue on the second card, but management through UCSM would be lost.
• Decom/Recom is Required to establish a different management path between shared-LOM and sideband, or in the event of a VIC failure.

• Enabling both User Acknowledge policies requires 2 re-acknowledgements
• Discover FEXs first and renumber them prior to discovering servers because you will have to take them offline and this could disrupt server discovery.

vCon Placement Policy Configuration – Best Practices

• Placement policies are designed to provide consistent PCIe numbering when moving profiles between servers with a different number of adapters.

• Automatic placement will be sufficient unless you have a setup that requires strict control of vNIC/vHBA Placement. Examples Include:
• Servers with onboard adapters manageable through UCSM that you do not wish to use (e.g. C460).
• Using UCSM discoverable adapters to connect to external networks (e.g. Broadcom, Intel, etc.)
• Strict separation of FCoE traffic and Ethernet Traffic.

Multi-Volume Drive Management Best Practices

• ALWAYS use protect configuration when moving service profiles between servers unless your intent is to reconfigure the logical drives on the controller thereby destroying all data on the drives.

• If you need any configuration other than a single volume or all the drives to be individual RAID 0 volumes use a local disk policy of any configuration.

• If you do only need a single volume and think you may grow your existing volume or create an additional volume in the future use a local disk policy of any configuration.

• If you have an existing stand-alone C-Series server with logical drives and data on those drives you can still integrate without loosing the data if you associate a service provide that has a local disk policy of Any configuration.

Related Information

C-Series Servers UCSM Integration Configuration Example

http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-infrastructure-ucs-manager-software/116741-troubleshoot-ucsm-00.html

http://www.cisco.com/c/en/us/support/servers-unified-computing/ucs-c-series-rack-mount-ucs-managed-server-software/products-installation-and-configuration-guides-list.html

http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/ucs-virtual-interface-card-1240/data_sheet_c78-699459.html

https://www.ciscolive.com/online/connect/sessionDetail.ww?SESSION_ID=77758&backBtn=true

Known issue: 

CSCup69532 UCSM 'remove' server functionality does not work