This document provides design guidance and best practices in deploying the Cisco Nexus™ 1000V Series Switches-software switches embedded within the VMware vSphere hypervisor-on top of the Cisco Unified Computing System™ with respect to deployment options using the various adapter types that the Cisco Unified Computing System supports. For detailed Cisco Nexus 1000V Series configuration documentation, please refer to the respective Cisco® and VMware product configuration guides. Links to the product configuration guides can be found in the "For More Information" section of this document.
Audience
This document is intended for Cisco systems engineers supporting customer server administrators, network administrators, and infrastructure architects (both customers and partners) interested in understanding and deploying Cisco Nexus 1000V Series Switches on the Cisco Unified Computing System. It assumes a basic functional knowledge of the Cisco Nexus 1000V Series, VMware vNetwork Distributed Switching, and Ethernet switching within a VMware environment in general. It also assumes a general understanding of the Cisco Unified Computing System, server I/O, virtualization, and networking concepts.
Terminology
This document refers to the virtual machine network interface cards (NICs) that are presented by the VMware ESX hypervisor as virtual NICs (vNICs), and the service profile-defined interfaces that are created and managed by Cisco UCS Manager as virtual interfaces (VIFs). The Cisco Nexus 1000V Series is the Cisco implementation of what the IEEE 802 working groups call a virtual Ethernet bridge (VEB). For more information about this work, please refer to http://www.ieee802.org/1/pages/dcbridges.html.
Introduction
Cisco Nexus 1000V Series Switches are virtual machine access switches that are an intelligent software switch implementation for VMware vSphere environments running the Cisco NX-OS Software operating system. Operating inside the VMware ESX hypervisor, the Cisco Nexus 1000V Series supports Cisco VN-Link server virtualization technology to provide:
• Policy-based virtual machine connectivity
• Mobile virtual machine security and network policy
• Nondisruptive operational model for your server virtualization and networking teams
When server virtualization is deployed in the data center, virtual servers typically are not managed the same way as physical servers. Server virtualization is treated as a special deployment, leading to longer deployment time, with a greater degree of coordination required among server, network, storage, and security administrators. With the Cisco Nexus 1000V Series, you can have a consistent networking feature set and infrastructure. Virtual servers can now use the same network configuration, security policy, diagnostic tools, and operational models as their physical server counterparts attached to dedicated physical network ports. Server administrators can access predefined network policy that follows mobile virtual machines to help ensure proper connectivity, saving valuable time for focusing on virtual machine administration. This comprehensive set of capabilities helps you deploy server virtualization faster and achieve its benefits sooner (Figure 1).
Figure 1. The Cisco Nexus 1000V Series Allows a Virtualized Server Operational Model Similar to That of a Physical Server
The Cisco UCS integrates low-latency unified network fabric with enterprise-class, x86-based servers, creating an integrated, scalable, multichassis platform in which all resources participate in a unified management domain. A single system scales to up to 40 chassis, 320 compute nodes, and thousands of virtual machines.
The Cisco Unified Computing System enables more dynamic and agile data centers, in which server identity (MAC addresses, worldwide names [WWNs], firmware and BIOS revisions, network and storage connectivity profiles and policies, etc.) can be dynamically provisioned or migrated to any physical server within the system. The Cisco Unified Computing System creates a highly dynamic and cohesive integration with network and storage resources to meet the rapidly changing needs in today's data centers. New computing resources can be deployed in a just-in-time approach. Traditional physical and virtual workloads can be easily migrated between servers through remote management, regardless of physical connectivity. The Cisco Unified Computing System improves availability, security, agility, and performance through an integrated architecture (Figure 2).
Some functions in the Cisco Unified Computing System are similar to those offered by the Cisco Nexus 1000V Series Switches, but with a different set of applications and design scenarios. The Cisco Unified Computing System offers the capability to present adapters to physical and (in the near future) virtual machines directly. When presented to the virtual machines, VN-Link technology can provide many of the same benefits as those described for the Cisco Nexus 1000V Series.
Figure 2. The Cisco UCS Integrates Network, Compute, and Virtualization Resources into a Single Cohesive System
Cisco UCS Common Configurations
This section highlights some of the areas of special interest in the Cisco Unified Computing System that pertain to the configuration and use of the Cisco Nexus 1000V Series. The configurations discussed here apply regardless of adapter type.
Cisco UCS Switching Functions
The Cisco UCS uses three adapter types, with four specific models: the Cisco UCS 82598KR-CI 10 Gigabit Ethernet Adapter, UCS M71KR-Q QLogic Converged Network Adapter, UCS M71KR-E Emulex Converged Network Adapter, and UCS M81KR Virtual Interface Card. Each of these cards has a pair of 10 Gigabit Ethernet connections to the Cisco Unified Computing System backplane that support the IEEE 802.1 Data Center Bridging function (formerly called Cisco Data Center Ethernet) to facilitate I/O unification within these adapters. On each adapter type, one of these backplane ports is connected through 10GBASE-KR to the A-side I/O module; then that connection goes to the A-side fabric interconnect. 10GBASE-KR is a copper midplane technology for interfacing adapters and switching elements through these midplanes. The other connection is 10GBASE-KR to the B-side I/O module; that connection then goes to the B-side fabric interconnect. Figure 3 later in this document shows this connectivity.
Within the Cisco UCS M71KR-E, M71KR-Q, and M81KR adapter types, the Cisco Unified Computing System can enable a fabric failover capability in which loss of connectivity on a path in use will cause remapping of traffic through a redundant path within the Cisco Unified Computing System. In determining the adapter type to use to support the Cisco Nexus 1000V Series, the interface count available to the VMware ESX host is a key factor. Here are some high-level trade-offs:
• With the Cisco UCS M81KR adapter, multiple virtual interfaces (VIFs) can be identified to match those used by the service console, VMware VMkernel, virtual machine data, Cisco Nexus 1000V Series VLANs, etc. as required. Each of these adapters can be defined within the Cisco Unified Computing System as connected to an individual fabric interconnect and, optionally, enabling a failover to the other.
• With the Cisco UCS M71KR-E and M71KR-Q adapters, a maximum of two adapters are presented to the VMware ESX hypervisor running on the blade. Each of these adapters can be defined within the Cisco Unified Computing System as connected to an individual fabric interconnect and, optionally, enabling a failover to the other. This fabric failover enables a model in which the virtual machine data can use one path within the Cisco Unified Computing System by default, and all other connections can go on the other path.
• With the Cisco UCS 82598KR-CI adapters, a maximum of two adapters are presented to the VMware ESX hypervisor running on the blade. This interface count does not support a fabric failover, and the service console must be migrated to the Cisco Nexus 1000V Series Switch along with all these other adapter types if any high-availability requirement exists. The actual migration of this interface during VMware ESX deployment on the blade is discussed in the specific adapter section later in this document, but for more detailed information about how to migrate a user's service console, see the Cisco Nexus 1000V Series documentation.
Note: The user should assign the VMware service console and VMkernel interface VLANs in the Cisco Nexus 1000V Series Switches as system VLANs along with the management, control, and packet VLANs.
Inside the Cisco UCS 6100 Series Fabric Interconnects, local Layer 2 switching is performed between the server adapters that connect to them, and any traffic that needs other communication will exit on an uplink to the data center switches. Examples of this traffic are Layer 3 communications to the outside network, to other Cisco Unified Computing Systems, and to other VMware nodes, as well as traffic if some servers in a given Cisco Unified Computing System are active on the A side and some are active on the B side (the A-B data communications path is through the upstream switch). The administrator can define which uplink or uplink channel a given server adapter or VIF will use as a point of uplink to get control of the traffic flows.
The Cisco UCS 6100 Series operates in two discrete modes with respect to flows in the Cisco Unified Computing System. The first is assumed to be more common and is called end-host mode; the other is the switched mode, in which the fabric interconnect acts as a normal Ethernet bridge device. Discussion of the differences between these modes is beyond the scope of this document; however, the Cisco Nexus 1000V Series Switches on the server blades will operate regardless of the mode of the fabric interconnects. With respect to a VMware environment running the Cisco Nexus 1000V Series, the preferred solution is end-host mode to help ensure predictable traffic flows.
End-Host Mode Configuration
Figure 3. End-Host Mode Configuration with Cisco Unified Computing System
With the end-host mode configuration, when the Layer 2 communication flows within the Cisco Nexus 1000V Series infrastructure occur, these flows may be either local to a given Cisco UCS 6100 Series Fabric Interconnect or through the upstream data center switch with more hops. Applying a quality-of-service (QoS) policy here to help ensure a minimum bandwidth is recommended. The recommendation is to assign CoS = 6 to these VMware service console and VMkernel flows and honor these QoS markings on the data center switch to which the Cisco UCS 6100 Series Fabric Interconnect connects. This marking of QoS values can be performed on the Cisco Nexus 1000V Series Switch in all cases, or can be performed on a per-VIF basis on the Cisco UCS M81KR adapter within the Cisco Unified Computing System with or without the Cisco Nexus 1000V Series Switch.
It is also recommended in this scenario that you connect the Cisco UCS 6100 Series Fabric Interconnects with redundant links to a pair of data center devices that are either separate switches or joined through some method for multichassis EtherChannel (MCEC) such as Cisco Virtual Switching System (VSS) or Cisco Virtual PortChannel (vPC). With these redundant uplinks to the data center edge, when the link failure indications are passed to the Cisco Nexus 1000V Series Switch, it will fail traffic over to itself. Do not create static pin groups on the system uplinks, which will by default allow the Cisco UCS 6100 Series Fabric Interconnects to dynamically pin the traffic to the uplinks. Again, this configuration reduces the requirements on the Cisco Nexus 1000V Series and increases availability.
Another recommendation is that the customer consolidate the Cisco UCS 6100 Series Fabric Interconnect links from the A- and B-side links on the same line cards within the aggregation layer so that the user can take advantage of local switching performance on that device to offer better scale because the flows need not traverse the fabric. Note that the channel bundling is performed through Link Aggregation Control Protocol (LACP), and the Cisco UCS 6100 Series Fabric Interconnects will appear to the network to be running in LACP active mode.
In addition, the Cisco Nexus 1000V Series has management and data VLANs, along with a pair of special VLANs called the control and packet VLANs. These VLANs must be allowed between the data center aggregation switches to which the Cisco UCS 6100 Series Fabric Interconnects connect (along with any other hosts of interest outside the Cisco Unified Computing System).
Switch Mode Configuration
When using switch mode (Figure 4), the Layer 2 communication flows within the Cisco Nexus 1000V Series infrastructure could be local to a given Cisco UCS 6100 Series Fabric Interconnect or through inter-fabric interconnect links within the given Cisco Unified Computing System. All the QoS recommendations in the preceding design scenario still apply in this environment; however, this design is not recommended for the following reasons:
• If the upstream device is MCEC capable, a spanning-tree block will form between the Cisco UCS 6100 Series Fabric Interconnects.
• If there is a Layer 2 link between the aggregation switches, a spanning-tree block also will form.
• If more than one Cisco Unified Computing System will be connected to this U design, a loop will be formed, and the fabric interconnect-to-fabric interconnect links will be blocked on the other Cisco Unified Computing System.
Figure 4. End-Host Mode Configuration with Cisco Unified Computing System
The switch mode design is often referred to as the U design for Ethernet switching, and the First-Hop Routing Protocol (FHRP) is used to give the default gateway visibility to the servers. With these restrictions, the end-host mode of operation is the general recommended best practice.
Service Profile Templates and Service Profiles
For the Cisco Nexus 1000V Series, some general best practices exist for the definitions of the service profile templates and service profiles within Cisco UCS Manager. When the adapters are defined and used within a service profile, they become available according to the availability of the fabric to which they are assigned. The definitions available to the VMware ESX hypervisor depend on the order in which the adapter is defined within the service profile (which is interface 0, interface 1, etc.). The general goal is a correlation between the adapter-creation function that the Cisco Unified Computing System administrator performs and the VMware ESX hypervisor network assignment. This function is similar to the physical server function of determining which RJ-45 port on the server plugs into which network and then identifying this adapter on the server. For this reason, the recommended practice is to invoke all service profiles from a common template to help prevent adapters and VIFs from being improperly assigned. This practice has far more importance for the Cisco UCS M81KR adapter, but it is also worth noting here.
In this same way, the administrator can help ensure a common baseline of adapter firmware to the common servers, along with QoS, VLAN settings, etc.
Cisco UCS 82598KR-CI Adapter Use Cases for the Cisco Nexus 1000V Series
High Availability with Cisco UCS 82598KR-CI
The Cisco UCS 82598KR-CI adapter runs a classic Ethernet connection from the adapter to the I/O module, where the system QoS policies are enforced and the virtual network tag (VNTag) field is added. There is no Cisco microcode being executed on this adapter itself, and no registration process to the fabric interconnect from the adapter. Therefore, this adapter has no fabric failover capability, so the use case is a single-use scenario: a pair of virtual network interface cards (vNICs), one each to the A- and B-side fabric interconnects. Another important point is that Fibre Channel over Ethernet (FCoE) is not supported from the Cisco Unified Computing System point of view (the software FCoE is at the OS level), so no SAN boot capabilities are available, so the system needs to rely on the local disk. This factor affects the stateless use cases within the Cisco Unified Computing System, although customers can move the physical disks along with the service profiles.
To achieve the goal of high availability to the virtual machines within the VMware ESX host, use of vPC host mode (vPC-HM) on the Cisco Nexus 1000V Series Switch is recommended. For the Cisco Nexus 1000V Series Switch to receive the Cisco Discovery Protocol packets to automatically build the vPC-HM on this adapter, the configuration on the service profile needs to include the definition of a default VLAN (security best practices recommend a VLAN other than VLAN 1 here). This default VLAN is where the Cisco Discovery Protocol packets are sent from the Cisco UCS Cisco UCS 6100 Series Fabric Interconnects to the host and require the administrator to enter only the following under the Cisco Nexus 1000V Series uplink port profile:
channel-group auto mode on sub-group cdp
If the Cisco Unified Computing System service profile definition of the adapters does not include the default VLAN, no Cisco Discovery Protocol information will be received on the host, and the user must configure vPC-HM manually. This manual configuration is mandatory in the other adapter types and is covered in the sections that cover them.
The Cisco Nexus 1000V Series uplink port profile that is to be assigned to the Cisco UCS 82598KR-CI adapter should be given a descriptive name, because VMware vCenter will display these names, and the administrator will map the Cisco Nexus 1000V Series Ethernet ports to the appropriate uplink port groups. The interesting parts of the Cisco Nexus 1000V Series configurations are shown here. The VLAN mappings used here were:
• VLAN 2: Management and console
• VLAN 3: VMware VMkernel
• VLAN 4: Cisco Nexus 1000V Series control
• VLAN 5: Cisco Nexus 1000V Series packet
• VLAN 6: Virtual machine data network 1
These mappings are used throughout this document. The relevant portion of the Cisco Nexus 1000V Series configuration is shown here:
<config-snippet>
port-profile 82598KR-CI-uplink
capability uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1-6
channel-group auto mode on sub-group cdp
no shutdown
system vlan 2-5
state enabled
interface Ethernet3/1
inherit port-profile 82598KR-CI-uplink
interface Ethernet3/2
inherit port-profile 82598KR-CI-uplink
</config-snippet>
Service Profile Template Configuration
The following rules apply in the creation of a Cisco UCS 82598KR-CI service profile for hosting a VMware ESX instance that will be included in a Cisco Nexus 1000V Series system:
• The network administrator must define the VLANs that will initially be passed to the adapter and thus to the Cisco Nexus 1000V Series virtual Ethernet module (VEM).
• The network administrator should define a default (or native) VLAN for Cisco Discovery Protocol flow and automatic vPC-HM.
• The network administrator should define a default QoS profile for any traffic that the Cisco Nexus 1000V Series Switch does not mark.
• The server administrator should then use the appropriate network policies when building a service profile.
Figure 5 shows an example of a service profile with the pertinent service profile networking configuration.
Figure 5. Cisco UCS 82598KR-CI Service Profile: vNIC Definitions for the Cisco Nexus 1000V Series
Operations
As discussed earlier, the design requires trade-offs. Since multiple adapters cannot be mapped to the host, the user will need to rely on the QoS settings from the Cisco Nexus 1000V Series Switch to help ensure smooth operation. This solution does not currently allow the user to define a primary and a backup path for the console, kernel, production, or other interfaces into the fabric.
The Cisco UCS 82598KR-CI uses Intel Virtual Machine Device Queue (VMDq) technology to assist in CPU operations to switch packets from the adapter to the virtual machines and to define multiple transmit queues to reduce any head-of-line blocking on transmission of data. This document does not explore these details; for more information, see the Intel document at http://www.intel.com/network/connectivity/vtc_vmdq.htm.
Performance and Failover Test
To see the basic numbers and failover times that can be expected, an iPerf test was performed from a virtual machine running behind the Cisco Nexus 1000V Series Switch to a host external to the Cisco Unified Computing System. Figure 6 shows the testing topology.
Figure 6. Cisco Nexus 1000V Series Switch on Cisco UCS 82598KR-CI
Not shown in Figure 6 is a traffic target server (Dell 2950) with a Gigabit Ethernet connection to Cisco Nexus 5010 Switch A.
• Baseline observations:
– A 90-second iPerf flow test was performed.
– Throughput on a single virtual machine to a single external Dell 2950 Gigabit Ethernet port was 365 Mbps.
– All MAC addresses were visible on the Cisco Nexus 1000V Series, Cisco Nexus 5010, and Cisco UCS 6120XP.
– The B fabric was the side that was actively passing traffic by default.
– One thread on the Cisco UCS blade (out of the 16 in the pair of E5520 CPUs) reached 100 percent for the duration of the test.
• Observations on the failure of the link from Cisco UCS 6120XP B to Cisco Nexus 5010 A (active path for flow):
– A 90-second iPerf flow test was performed.
– On failure, the MAC address still remained on the B fabric.
– Cisco Unified Computing System repinning time for this flow was 1.23 seconds.
– The Cisco Nexus 1000V Series Switch did not detect a link outage.
• Observations on the failure of the link from Cisco UCS 6120XP B to Cisco Nexus 5010 B (current active path for flow):
– A 90-second iPerf flow test was performed.
– On failover, the MAC address showed up on the A fabric.
– Failover time for this flow was 1.73 seconds.
– On link restoration (to restore the test bed), the Cisco Nexus 1000V Series Switch did not experience an outage.
– Restoration of the second uplink did not cause a Cisco Unified Computing System repinning and there was no outage.
– The Cisco Nexus 1000V Series vPC-HM provided high availability.
Cisco UCS M71KR-Q and M71KR-E Adapter Use Cases for the Cisco Nexus 1000V Series
High Availability with Cisco M71KR
From the perspective of high availability in the Cisco Unified Computing System, there is no difference between the Cisco UCS M71KR-Q and M71KR-E adapters. These adapters do function differently within the Cisco Unified Computing System than in the PCI Express (PCIe) form-factor cards available today. The Cisco UCS M71KR adapters (on the Cisco UCS M71KR application-specific integrated circuit [ASIC]) run a firmware that is maintained by Cisco, as opposed to the initial versions of the PCIe cards from QLogic and Emulex (which maintain the Cisco UCS M71KR firmware themselves). As a result, some functions are different:
• IEEE 802.1 Data Center Bridging mode
• Failover options of the adapters
The Cisco UCS M71KR adapters in the Cisco Unified Computing System run a VNTag-based communication flow to the Cisco UCS Cisco UCS 6100 Series Fabric Interconnects rather than a mode called I/O consolidation (IOC) on the PCIe adapters. This VNTag mode enables an additional fabric failover mode (in which the adapter registers with both the Cisco UCS 6100 Series Fabric Interconnects), using one of the paths as an active path and the other as a backup path.
Note: VNTag is not discussed in detail in this document; please refer to other documents for details on the technology.
As stated earlier, Cisco microcode is executed on this adapter, and the adapter performs a registration process to the fabric interconnects at server startup. This adapter type is its own communications endpoint, and since the VNTag mechanism uniquely identifies the adapter, the registration process registers a wildcard MAC address with the fabric interconnects. This behavior is useful for hosts that need to run promiscuous-mode operations: software sniffers, software switches, etc. that need access to traffic with more than a single MAC address. When source traffic is seen on the fabric interconnects with a source MAC address behind the Cisco UCS M71KR adapter, the MAC address table on the active link's fabric interconnect will be dynamically updated. Currently, this information is not replicated to the backup fabric interconnect, prompting the recommendations here for high availability when using the Cisco Nexus 1000V Series.
To achieve the goal of high availability to the virtual machines on the VMware ESX host using Cisco UCS M71KR, the use of vPC-HM on the Cisco Nexus 1000V Series Switch is recommended. Since these adapters are virtual behind the Cisco UCS M71KR ASIC, currently the hosts do not receive Cisco Discovery Protocol packets, and the user must configure vPC-HM manually. This manual configuration consists of adding the following commands for each interface (there will be two on a nonfabric failover host equipped with Cisco UCS M71KR) on the Cisco Nexus 1000V Series module that represents the host:
sub-group-id 0 (or 1 for the peer port)
Next, the user will set up the PortChannel to the module by adding the following command under the appropriate uplink port profile:
channel-group auto mode on (case of manual vPC-HM)
The Cisco Nexus 1000V Series uplink port profile that is to be assigned to the Cisco UCS M71KR adapters should have a descriptive name, because VMware vCenter will display these names, and the administrator will map the Cisco Nexus 1000V Series Ethernet ports to the appropriate uplink port groups.
The relevant portion of the Cisco Nexus 1000V Series configuration is shown here:
<config-snippet>
port-profile M71KR-no-fabric-ha-uplink
capability uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1-6
channel-group auto mode on
no shutdown
system vlan 2-5
state enabled
interface Ethernet5/1
inherit port-profile M71KR-no-fabric-ha-uplink
sub-group-id 0
interface Ethernet5/2
inherit port-profile M71KR-no-fabric-ha-uplink
sub-group-id 1
</config-snippet>
Service Profile Template Configuration
The following rules apply in the creation of a Cisco UCS M71KR service profile for hosting a VMware ESX instance that will be included in a Cisco Nexus 1000V Series system:
• The administrator must define the VLANs that will initially be passed to the adapter and thus to the Cisco Nexus 1000V Series VEM.
• The network administrator should define a default QoS profile for any traffic that the Cisco Nexus 1000V Series Switch does not mark.
Figure 7 shows an example of a service profile with the pertinent service profile networking configuration.
Figure 7. Cisco UCS M71KR Nonfabric High-Availability Service Profile: vNIC Definitions for the Cisco Nexus 1000V Series
Operations
As discussed earlier, the design requires some trade-offs. Since multiple adapters cannot be mapped to the host, the user will need to rely on the QoS settings from the Cisco Nexus 1000V Series Switch to help ensure smooth operation. This solution does not currently allow the user to define a primary and a backup path for the console, kernel, production, or other interfaces into the fabric. The Cisco UCS M71KR adapter also contains an Intel 82598 10 Gigabit Ethernet adapter and uses Intel VMDq technology as discussed in the Cisco UCS 82598KR-CI section earlier in this document.
Performance and Failover Test
To see the basic numbers and failover times that can be expected, an iPerf test was performed from a virtual machine running behind the Cisco Nexus 1000V Series Switch to a host external to the Cisco Unified Computing System. Figure 8 shows the testing topology.
Figure 8. Cisco Nexus 1000V Series Switch on Cisco UCS M71KR Without Fabric Failover
Not shown in Figure 8 is a traffic target server (Dell 2950) with a Gigabit Ethernet connection to Cisco Nexus 5010 Switch A.
• Baseline observations:
– A 90-second iPerf flow test was performed.
– Throughput on a single virtual machine to a single external Dell 2950 Gigabit Ethernet port was 368 Mbps.
– All MAC addresses were visible on the Cisco Nexus 1000V Series, Cisco Nexus 5010, and Cisco UCS 6120XP.
– The A fabric was the side that was actively passing traffic by default.
– One thread on the Cisco UCS blade (out of the 16 in the pair of E5520 CPUs) reached 100 percent for the duration of the test.
• Observations on failure of the link from Cisco 6120XP A to Cisco Nexus 5010 A (active path for flow):
– A 90-second iPerf flow test was performed.
– On failure, the MAC address still remained on the A fabric.
– Cisco Unified Computing System repinning time for this flow was 1.23 seconds.
– The Cisco Nexus 1000V Series Switch did not detect a link outage.
• Observations on failure of the link from Cisco 6120XP A to Cisco Nexus 5010 B (current active path for flow):
– A 90-second iPerf flow test was performed.
– On failover, the MAC address showed up on the B fabric.
– Failover time for this flow was 4.05 seconds.
– On link restoration (to restore the test bed), the Cisco Nexus 1000V Series Switch did not experience an outage.
– Restoration of the second uplink did not cause a Cisco Unified Computing System repinning, and there was no outage.
– The Cisco Nexus 1000V Series vPC-HM provided high availability.
Cisco UCS M81KR Adapter Use Cases for the Cisco Nexus 1000V Series
High Availability with Cisco UCS M81KR
The Cisco UCS M81KR adapter functions in a unique way in that it can be fully virtualized and presented not only to the CPU and BIOS as multiple adapters, but it can also be defined on the Cisco UCS 6100 Series Fabric Interconnects as different adapters. The management and control of these definitions are within the capability of the Cisco UCS Manager service profile. This adapter fully utilizes the VNTag technology and an additional protocol called the Virtual Interface Control (VIC) Protocol for adapter operations. This VNTag mode enables a fabric failover mode (in which the virtual adapters register with both the Cisco UCS 6100 Series Fabric Interconnects), using one of the paths as an active path and the other as a backup path. The host will see multiple 10 Gigabit Ethernet Cisco adapters that are mapped to either the A side or the B side in the Cisco Unified Computing System. The Cisco Unified Computing System administrator has the option of running these adapters in a fabric failover mode; however, the high availability is maintained within the fabric and not on the host.
As with the Cisco UCS M71KR ASIC, Cisco microcode is executed on this adapter, and the adapter performs a registration process to the fabric interconnects at server startup. Note that the Cisco UCS M81KR adapter runs in a pair of discrete modes as mentioned earlier. With the static VIF mode, the idea is that the administrator will create and maintain all virtual interfaces. In this mode, the administratively defined adapter is its own endpoint, and since the VNTag mechanism uniquely identifies the adapter, the registration process will register a unicast MAC address for the adapter and an additional wildcard MAC address with the fabric interconnects if a promiscuous mode is requested. This behavior is useful for hosts that need to run promiscuous-mode operations: software sniffers, software switches, etc. that need access to traffic with more than a single MAC address. When source traffic is seen on the fabric interconnects with a source MAC address behind the Cisco UCS M71KR adapter, the MAC address table on the active link's fabric interconnect will be dynamically updated. Currently, this information is not replicated to the backup fabric interconnect, prompting the recommendations here for high availability when using the Cisco Nexus 1000V Series.
The Cisco UCS M81KR dynamic VIF mode provides an alternative operating model in which each virtual interface is created, deleted, and maintained by the VMware vCenter communications path during virtual machine operations within the cluster. The goal is for these adapters to be coordinated and presented directly to the virtual machines, with no additional management overhead needed to help ensure port-group consistency as these machines are migrated within a cluster. This mode runs on the Cisco UCS M81KR card exclusively, and the host cannot be a blade within the Cisco Nexus 1000V Series. Since this document focuses on the Cisco Nexus 1000V Series interaction with the Cisco Unified Computing System, these modes are not discussed further here.
To achieve the goal of high availability to the virtual machines on the VMware ESX host using Cisco UCS M81KR, the use of vPC-HM is recommended. Since these adapters are virtual behind the Cisco UCS M81KR ASIC, currently the hosts do not receive Cisco Discovery Protocol packets, and the user must configure vPC-HM manually. This configuration is identical to that for the Cisco UCS M71KR discussed earlier; refer to that section of this document for configuration details.
The Cisco Nexus 1000V Series uplink port profile that is to be assigned to the Cisco UCS M81KR adapters should be given a descriptive name because VMware VCenter will display these names, and the administrator will map the Cisco Nexus 1000V Series Ethernet ports to the appropriate uplink port groups.
The relevant portion of the Cisco Nexus 1000V Series configuration is shown here:
<config-snippet>
port-profile M81KR-no-ha-AIPC-uplink
capability uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 4-5
channel-group auto mode on
no shutdown
system vlan 4-5
state enabled
port-profile M81KR-no-ha-VmkSc-uplink
capability uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 2-3
channel-group auto mode on
no shutdown
system vlan 2-3
state enabled
port-profile M81KR-no-ha-VmData-uplink
capability uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1,6
channel-group auto mode on
no shutdown
system vlan 1,6
state enabled
interface Ethernet4/1
inherit port-profile M81KR-no-ha-AIPC-uplink
sub-group-id 0
interface Ethernet4/2
inherit port-profile M81KR-no-ha-AIPC-uplink
sub-group-id 1
interface Ethernet4/3
inherit port-profile M81KR-no-ha-VmkSc-uplink
sub-group-id 0
interface Ethernet4/4
inherit port-profile M81KR-no-ha-VmkSc-uplink
sub-group-id 1
interface Ethernet4/5
inherit port-profile M81KR-no-ha-VmData-uplink
sub-group-id 0
interface Ethernet4/6
inherit port-profile M81KR-no-ha-VmData-uplink
sub-group-id 1
</config-snippet>
One advantage of the Cisco Nexus 1000V Series Switch on the Cisco UCS M81KR adapter should be apparent: the capability to define unique interfaces for uplink to specific areas of the network, such as to constrain and pass VMware VMkernel traffic on alternate links to the location where the virtual machine production traffic normally passes.
Service Profile Template Configuration
The following rules apply in the creation of a Cisco UCS M81KR service profile for hosting a VMware ESX instance that will be included in a Cisco Nexus 1000V Series system:
• The network administrator must define the VLANs that will initially be passed to the adapter and thus to the Cisco Nexus 1000V Series VEM.
• The network administrator should define a default QoS profile for any virtual interface to which the Cisco Nexus 1000V Series Switch will uplink. An important difference with this adapter is the Cisco UCS M81KR firmware does not honor Cisco Nexus 1000V Series markings, but instead imposes them on traffic entering the Cisco Unified Computing System from the host.
Figure 9 shows an example of a service profile with the pertinent service profile networking configuration.
Figure 9. Cisco UCS M81KR Nonfabric High-Availability Service Profile: Three vNIC Definitions for the Cisco Nexus 1000V Series
Operations
As discussed earlier, the design requires some trade-offs. Since the CoS markings from the Cisco Nexus 1000V Series are not currently honored, multiple adapters need to be defined on the host, and the user will need to define the system QoS settings in the Cisco Unified Computing System to help ensure smooth operation. This solution allows the user to define a primary and a backup path for the console, kernel, production, and other interfaces into the fabric.
Note, too, that the Cisco UCS M81KR can support enough interfaces so that migrating the service console to the VEM may be unnecessary, because these additional adapters can also be used by VMware's integrated software switch if that customer use case is required.
Performance and Failover Test
To see the basic numbers and failover times that can be expected, an iPerf test was performed from a virtual machine running behind the Cisco Nexus 1000V Series Switch to a host external to the Cisco Unified Computing System. Figure 10 shows the testing topology.
Figure 10. Cisco Nexus 1000V Series Switch on Cisco UCS M81KR Without Fabric Failover
Not shown in Figure 10 is a traffic target server (Dell 2950) with a Gigabit Ethernet connection to Cisco Nexus 5010 Switch A.
• Baseline observations:
– A 90-second iPerf flow test was performed.
– Throughput on a single virtual machine to a single external Dell 2950 Gigabit Ethernet port was 368 Mbps.
– All MAC addresses were visible on the Cisco Nexus 1000V Series, Cisco Nexus 5010, and Cisco UCS 6120XP.
– The A fabric was the side that was actively passing traffic by default.
– One thread on the Cisco UCS blade (out of the 16 in the pair of E5540 CPUs) briefly reached 100 percent in the test (due to a higher-performance processor).
• Observations on failure of the link from Cisco UCS 6120SP A to Cisco Nexus 5010 A (active path for flow):
– A 90-second iPerf flow test was performed.
– On failure, the MAC address still remained on the A fabric.
– Cisco Unified Computing System repinning time for this flow was 1.23 seconds.
– The Cisco Nexus 1000V Series Switch did not detect a link outage.
• Observations on failure of the link from Cisco 6120XP A to Cisco Nexus 5010 B (current active path for flow):
– A 90-second iPerf flow test was performed.
– On failover, the MAC address showed up on the B fabric.
– Failover time for this flow was 3.63 seconds.
– On link restoration (to restore the test bed), the Cisco Nexus 1000V Series Switch did not experience an outage.
– Restoration of the second uplink did not cause a Cisco Unified Computing System repinning, and there was no outage.
– The Cisco Nexus 1000V Series vPC-HM provided high availability.
Cisco Nexus 1000V Series Virtual Switch Module Placement in the Cisco Unified Computing System
This section provides recommendations for the placement of the virtual switch module (VSM) relative to the Cisco Unified Computing System and the physical Ethernet interface counts. The options are listed in order of recommended customer use in a Cisco Unified Computing System environment.
Option 1: VSM Inside the Cisco Unified Computing System on the VEM
In this scenario, fabric failover on the Cisco UCS M71KR is recommended because traffic on the links will keep the dynamic tables current.
Option 2: VSM Outside the Cisco Unified Computing System on the Cisco Nexus 1000V Series VEM
This model allows simple migration to appliance-based solutions when they are supported.
Option 3: VSM Inside the Cisco Unified Computing System on the VMware vSwitch
This model also was stable in the test deployments. It provides isolation of the managed devices, and it migrates well to the appliance model when that is supported.
A possible concern here is the management and operational model of the networking links between the VSM and the VEM devices.
Option 4: VSM Inside the Cisco Unified Computing System on the VMware vSwitch
A possible concern here is the management and operational model of the networking links between the VSM and the VEM devices.
Cisco Unified Computing System and Cisco Nexus 1000V Series Versions Used in This Document
The Cisco Unified Computing System software for this test was Version 4.0(1a)N2(1.0.1e), with a BIOS of 36-191. The Cisco Nexus 1000V Series software used in this testing was Version 1000v.4.0.4.SV1.1.
Conclusion
When using the Cisco Nexus 1000V Series at the first availability of the Cisco Unified Computing System, always do so without any adapter fabric failover for the reasons discussed earlier, and leave the hashing to the Cisco Nexus 1000V Series vPC-HM function. This recommendation also applies in a scenario with Cisco UCS M81KR without dynamic VIF policies in the pass-through switching (PTS).
The recommended placement of the VSM is outside the chassis, on a switch it does not manage itself (as a general best practice, a manager should not be placed on a device it manages).
When promiscuous mode and sharing of MAC information between the Cisco UCS fabric interconnects is supported, new possibilities for segmenting traffic within the Cisco Unified Computing System are present. This document will be updated with a major revision number to reflect those changes.
Appendix A: Troubleshooting VMware ESX Server in the Cisco Unified Computing System
You may need to troubleshoot when using the Cisco Nexus 1000V Series in a Cisco Unified Computing System environment if the system loses communication with the VMware ESX server, preventing further operations. The first step in resolving this situation is to troubleshoot the connection to the console port directly within the blade running VMware ESX. In the Cisco Unified Computing System, when a service profile is instantiated on a blade, an option will be presented to open a keyboard, video, and mouse (KVM) console session to that particular blade. This option will open a Java plug-in to perform the remote KVM console operations and to share media directly from the administrator's machine. A screen similar to the one in Figure 11 will appear.
From the KVM console, you can press ALT-F1 to open a login screen which you can use to log in to the VMware ESX directly.
Figure 11. KVM Console to a Cisco UCS Blade Running VMware ESX
From the output of the following command, note the distributed virtual port IDs of the management uplink and VMware VMkernel NICs (vmknics) that you want to revert.
# esxcfg-vswitch -l | more
At the same time, note whether the vSwitch configuration contains a port group called "Service Console."
Use the esxcfg-vswitch command to remove the uplink from the VMware vNetwork Distributed Switch (vDS). If you want to migrate more than one adapter, do so one at a time. In the example here, vmnic0 has distributed virtual port ID 160 in the VMware vDS named n1000v:
# esxcfg-vswitch -Q vmnic0 -V 160 n1000v
Use the esxcfg-vswitch command to add these uplinks, one at a time, to the vSwitch. For example:
# esxcfg-vswitch -L vmnic0 vSwitch0
If the vSwitch does not already have a port group called "Service Console," enter the following:
# esxcfg-vswitch -A "Service Console" vSwitch0
Now enter the following command for the virtual interfaces (vswifs):
# esxcfg-vswif -p "Service Console" vswif0
Now you should be able to start pinging other devices on your network.
