Troubleshoot UC on UCS TRC, UC on UCS Specs-based, and 3rd-party Specs-based Deployments

Introduction

This document clarifies some support aspects of Cisco Unified Communications (UC) applications, VMware vSphere virtualization software and server hardware (Cisco or 3rd-party) when deployed following the support policy at www.cisco.com/go/virtualized-collaboration. Of particular interest is the supported hardware content.

This document is applicable to all virtualization options, which include:

• Business Edition 6000 and 7000 appliances

• UC on Unified Communications System (UCS) Tested Reference Configuration (TRC)

• UC on UCS Specs-based

• 3rd-party Specs-based

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics (see the Related Information at the end of this document for web page links):

• UC on UCS solution (Cisco Unified Communications on Cisco Unified Computing System)

• UCS Tested Reference Configuration (TRC) hardware configurations

• Specs-based hardware configurations (UCS or 3rd-party server vendor)

• Virtualization of Cisco Collaboration applications

• VMware vSphere software

• Cisco Unified Computing System hardware

Components Used

The information in this document is based on these software and hardware versions:

• Cisco Collaboration applications that support virtualization (see At a Glance on www.cisco.com/go/virtualized-collaboration).

• Support policy for Virtualization of Cisco UC/Collaboration applications (see Supporting Documentation on www.cisco.com/go/virtualized-collaboration).

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

What Does “Supported” Mean?

In general, there are always four dimensions of “support” to consider. They are listed below in the form of questions, with answers specific to virtualization of Cisco UC/Collaboration applications:

• “Does it ‘work’?” While this sounds trite, in virtualization there are many items that appear to “work”, but might not be stable or perform adequately for real-time applications.  While "works" is necessary, it is not sufficient on its own to be “allowed” or supported by Cisco, and might not have been “validated” by VMware or Cisco.  

• “If it works, is it allowed by the vendor’s support policy rules?” Cisco defines what is supported vs. what is allowed on www.cisco.com/go/virtualized-collaboration. For Cisco Collaboration, an item that is “not allowed even if it ‘works’” is usually due to one of these reasons:

  • It creates an application problem that can only be fixed with software enhancements or re-architecture; for example, certain kinds of snapshots that hang or crash Cisco Unified Communications Manager.

  • It can negatively impact application stability or predictable capacity/performance, and required Cisco validation has not yet occurred; for example, vMotion with Cisco Unified Communications Manager before March 2011.

  • A valid usage scenario does not exist for Cisco Collaboration applications. For example, vSphere Dynamic Resource Scheduler for applications that do not support CPU Reservations.

• “If it is allowed, did the vendor validate it?” For example, formal testing and assurance, which is particularly important for UC/Collaboration deployments of real-time voice and video, customer contact centers, and other mission-critical communications. Some “allowed” items are not “validated”, either because they are outside Cisco’s responsibility demarcation (such as customer-provided 3rd-party virtualized servers or storage arrays) or because they are outside the scope of what Cisco explicitly tested (such as the UC application performance “guarantee” with UCS C-Series Tested Reference Configuration (TRC) direct attached storage (DAS) hardware vs. “guidance-only” with Specs-based hardware). Part of the value of infrastructure solutions such as Vblock or FlexPod is that is provides “validation” at the system level for a multi-product, multi-vendor deployment.

• “Does the vendor provide technical assistance for ‘how-to’ or ‘break-fix’?” For example, assistance with configuration, or troubleshooting to establish root cause and fix for a problem. Cisco Technical Assistance Center (TAC) supports products purchased from Cisco with a valid, paid-up maintenance contract.

Here are some real-world “support” examples that illustrate these concepts:

• VMware Boot from SAN: In 2010, this feature “worked” as an experimental VMware feature in vSphere 4.0, but was not officially “supported” by VMware until vSphere 4.1, which affected when Cisco could consider supporting it for its customers.

• Fibre Channel SAN with virtualized UC applications: Cisco’s support policy “allows” UC apps to connect to 3rd-party storage arrays via SAN networks from either Cisco or 3rd-parties, provided they meet the requirements at www.cisco.com/go/virtualized-collaboration. However, Cisco does not validate 3rd-party SAN switches or 3rd-party storage arrays, and Cisco TAC does not provide assistance on 3rd-party switches or arrays.

• Virtualizing UC applications on desktop-class CPUs (for example, Core-i3): This might or might not “work” in the sense that the application can successfully install and boot up, but it is unlikely that it “works” in the sense of providing production-class stability, capacity or performance. These CPUs are not allowed, validated, or supported by Cisco Collaboration applications, even if they appear to “work”.

It is impossible for Cisco to test every aspect and combination of hardware, VMware and application for assurance, particularly for 3rd-party hardware and software. Therefore, Cisco defines various hardware support policies that represent tradeoffs between “assurance” and “flexibility”, based on how much of the solution the customer wants Cisco to “own”, while it ensures minimum requirements for production application operation are met.

Note: Customers who do not follow Cisco’s published support policy will be asked to reproduce a problem in a supported configuration before Cisco TAC can effectively provide support.

Support Clarifications for Virtualized Hardware Options

For all options, it is a requirement that the host (physical hardware + VMware vSphere) is supported by all co-resident applications on that host. Refer to these links for application support:

• At a Glance on www.cisco.com/go/virtualized-collaboration

• Collaboration Virtualization Hardware

UC on UCS TRC

UCS TRC hardware configurations that meet the requirements at Collaboration Virtualization Hardware are “allowed”, specifically designed for and “validated” with UC applications by Cisco, and fully “supported” by Cisco TAC within Cisco’s support demarcation. For example, Cisco owns all the hardware on a UCS C-Series TRC with DAS storage. However, for a UCS B-Series TRC, Cisco does not validate or support 3rd-party storage switches or storage arrays, and Cisco TAC does not assist with these 3rd-party components.

Performance of Cisco UC app VMs is committed when installed on a UCS TRC meeting all the requirements at Collaboration Virtualization Hardware (including storage performance requirements for SAN), and when all conditions in the co-residency policy at Collaboration Virtualization Sizing are followed.  For UCM and IMP using CPU Reservations, there are additional considerations described here.

UC on UCS TRCs also specify a hardware bill of materials, which is useful to those wanting Cisco to own the hardware design as with older MCS 7800 appliance offers.

UC on UCS Specs-based

Specs-based UCS hardware meeting the requirements of Collaboration Virtualization Hardware and all application-specific requirements is “allowed” and fully “supported” by Cisco TAC within Cisco’s support demarcation just like UCS TRC.

The difference is that Specs-based UCS hardware configurations are not explicitly validated with Collaboration applications. Therefore, no prediction or assurance of UC application VM performance is made when installed on UCS Specs-based hardware. Only guidance is provided, and ownership of assuring that the pre-sales hardware design provides the performance required by UC applications shifts from Cisco to the customer. Otherwise, if all rules atwww.cisco.com/go/virtualized-collaboration are followed, Cisco TAC will assist with troubleshooting UCS Specs-based hardware, which include UC application performance issues. Keep in mind the points listed in “Key Support Considerations When Deploying on Specs-based Hardware”. These points help clarify what Cisco TAC can require in order to provide effective support and how far TAC will take a problem.

UCS TRCs can be thought of as “design reference points” for UCS Specs-based. The “risk” that a UCS Specs-based hardware design will not provide sufficient performance to a set of UC application VMs is proportional to the amount of “deviation” from UCS TRCs. More specifically:

• UCS server model not in any TRC: Normally not an issue unless the firmware or drivers used on that model are substantially different from models validated as part of a TRC.

• CPU model not in any TRC: A different CPU model not validated as part of a TRC is normally not an issue as long as it is an allowed CPU architecture with required core speed, and UC virtual-to-physical sizing rules for required core count are followed (refer to supported processors ). For example, UC application VMs did not experience much difference in performance between the Intel Xeon E5640 vs. X5650 (same architecture, similar performance characteristics, same core speed, just different core counts that enable different VM counts). However, due to interactions of CPU models with server model firmware and other system components, UC application VM performance can only be committed for CPU models validated in a TRC (which was only the E5640).

• Memory: A different memory configuration than what TRCs use is rarely an issue as long as it follows Cisco memory population guidelines for optimum performance on the server model, plus Cisco UC application virtual-to-physical sizing rules for required capacity at Collaboration Virtualization Hardware. Note that UCS TRC memory is intentionally sized for any possible mix of UC app VMs that can “fit” on the host, which results in total RAM that can be higher than what your particular deployment needs.

• Adapters: LAN utilization for UC application VMs is usually low for signaling, but can be high for deployments that are media-intensive (for example, lots of voicemail audio streams or conferencing video streams vs. signaling traffic) or using NAS/SAN storage (in which case the adapters are part of the Storage solution below). UCS C-Series TRCs are configured with enough Ethernet ports to handle the typical needs of the types of UC application VM mixes that they can host. Part of the design process is to ensure these ports are sufficient for your specific deployment.

• Storage: This is where most of the complexity and “risk” lies, due to the IO-intensive nature of most Cisco UC applications. There are several calculators available for theoretical DAS IO capacity, but it is very difficult to accurately predict actual DAS capacity without formal testing. NAS and SAN attached storage arrays provide more robust design assurance tools, but Cisco does not validate 3rd-party storage arrays or storage switches (UC on Vblock can be used to provide this assurance). UCS C-Series TRCs have DAS configurations tested vs. the latency tolerance of and IOPS generated by the types of UC app VM mixes that the TRC can host.

Specs-based uncertainty can be further reduced by pre-deployment testing, baselining, following general principles of virtualization, and following the rules of Cisco UC virtualization (at Cisco Collaboration Virtualization). However, Cisco cannot guarantee that VMs will never be starved for resources and never have performance problems outside of a UCS TRC. “Headroom” remains a design best practice, either in the form of leaving some unused capacity on a host, or provisioning additional hosts.

UC on UCS Specs-based does not specify a hardware bill of materials (BOM), since by definition Specs-based is for deployments where the customer requires different specs/BOM than what was validated in a TRC. Customers should use the TRC BOMs as guidance, and leverage their partner and Cisco teams for assistance on server BOM generation.

3rd-Party Specs-based

Specs-based 3rd-party server hardware meeting the requirements at Collaboration Virtualization Hardware is “allowed” by Cisco, but Cisco does not perform any testing or validation on the 3rd-party hardware.

No prediction or assurance of UC application VM performance is made when installed on 3rd-party Specs-based hardware. Only guidance is provided, and ownership of assuring that the pre-sales hardware design provides the performance required by UC applications shifts from Cisco to the customer. Otherwise, if all rules at Cisco Collaboration Virtualization are followed, Cisco TAC will assist with troubleshooting in order to rule out application issues as the root cause. Customer owns driving resolution of non-Cisco hardware/software issues, or non-Cisco hardware/software root causes of application issues (which includes customer-provided VMware software as described in Support Clarifications for Virtualization Software later in this document). Customer might need to engage 3rd-party vendors in order to investigate the non-Cisco components.

Also, keep in mind the points listed in Key Support Considerations When Deploying on Specs-based Hardware. These points help clarify what Cisco TAC might require to provide effective support and how far TAC will take a problem.

Note that Cisco does not support virtualization on legacy OEM HP/IBM servers (7800 Series Media Convergence Servers, or “MCS 7800”).

UCS TRCs can be used as “design reference points” for 3rd-party Specs-based as with UCS Specs-based described earlier in this document. Similar considerations for CPU, Memory, Adapters and Storage exist. Note that there are no TRCs based on 3rd-party server models.

Specs-based uncertainty can be further reduced by pre-deployment testing, baselining, following general principles of virtualization, and following the rules of Cisco UC virtualization (at Cisco Collaboration Virtualization ). However, Cisco cannot guarantee that VMs will never be starved for resources and never have performance problems outside of a UCS TRC.

Cisco does not specify a hardware bill of materials (BOM) for 3rd-party Specs-based servers, since by definition these are customer-provided, 3rd-party, non-OEM servers. Customers can use the UCS TRC BOMs for guidance, and leverage their 3rd-party server vendor and internal server IT teams for assistance on 3rd-party hardware BOM generation.

Key Support Considerations When Deploying on Specs-based Hardware

• In order to enable Cisco TAC to effectively provide support when you run Cisco UC VMs on Specs-based hardware configurations, Cisco requires VMware vCenter for UCS Specs-based and 3rd-party Specs-based. For additional details refer to Collaboration Virtualization Hardware and Virtualization Software Requirements. Customers must provide VMware vCenter data when required by Cisco TAC that demonstrates compliance with UC virtualization requirements such as storage performance.

• In order to enable Cisco TAC to effectively provide support when you run Cisco UC VMs on Specs-based hardware configurations, Cisco can require these activities from the customer for problem diagnosis or resolution: Changes to either the software workload or the physical hardware, in order to troubleshoot or resolve application performance problems. Examples of when these changes might be required are UC VM receiving insufficient CPU, Memory, network, disk capacity or storage IOPS from the hardware.

• Examples of what these changes look like in an actual deployment are listed here:

  • Software: temporary power-down of non-critical VMs in order to facilitate performance troubleshooting

  • Software: move critical VMs and/or non-critical VMs to alternate virtualization host/physical server as temporary or permanent solution.

    • Temporarily reduce the number of virtual machines running on a host if Cisco deems necessary for troubleshooting purposes.

    • Permanently reduce the number of virtual machines that run on a host if Cisco determines the host is overloaded.

    • Splitting a dense UC app VM into multiple less-dense VMs, then moving those less-dense VMs to alternate host; for example, splitting a CUCM 10K user OVA into multiple CUCM 7.5K user OVAs, then relocating some of those CUCM 7.5K user OVAs.

  • These approaches allow a reduction in the software workload on an overloaded virtualization host/physical server, so that the workload is no longer starved for hardware resources.

• Hardware: additions/upgrades to "fix" an overloaded host as an alternative to powering-down VMs or changing VM placement or density.

  • For example, addition of more physical disks to increase storage capacity and/or provide IOPS

  • For example, addition of more physical memory or more physical CPU cores

  • For example, addition of physical NIC interfaces to address LAN congestion

  • These approaches allow "upgrading" the overloaded hardware in order to accommodate the resource-starved software workload.

  • “How-to” support can be provided by Cisco only for UCS servers. For 3rd-party servers, the customer needs to engage 3rd-party support resources.

• If these requirements are unacceptable, it is recommended to deploy on a UCS C-Series TRC with DAS storage.

Cisco's provision of support is contingent on customer maintaining a current and fully paid support contract with Cisco.

Support Clarifications for Virtualization Software

Customers have these sourcing options for virtualization software that Cisco Collaboration applications can be deployed on:

1. Cisco UC Virtualization Hypervisor or Hypervisor Plus (supported only with Cisco Business Edition 6000)

2. Cisco UC Virtualization Foundation (supported only with UC applications deployed either as UC on UCS solution or as part of Cisco Business Edition 6000/7000)

3. VMware vSphere Standard, Enterprise or Enterprise Plus Editions purchased from Cisco

4. VMware vSphere Standard, Enterprise or Enterprise Plus Editions purchased direct from VMware

For options 1, 2 and 3, Cisco TAC is available to assist. For option 4, Cisco TAC does not assist with the virtualization software, and the customer should engage their 3rd-party vendor.

Cisco's provision of support is contingent on the customer maintaining a current and fully paid support contract with Cisco.

Related Information

• Cisco Unified Communications on Cisco Unified Computing System
• Cisco Collaboration Virtualization
• Collaboration Virtualization Hardware
• Virtualization Software Requirements
• Servers - Unified Computing
• DC Partner - VMware
• Technical Support & Documentation - Cisco Systems