This document describes a problem encountered with the number of maximum ports the VMware Distributed Virtual Switch (DVS) available in its interactions with Cisco Nexus 1000V Series switch (N1KV), and offers a solution to the problem.
Cisco recommends that you have knowledge of these topics:
The information in this document is based on N1KV and VMware DVS.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
One of the most common issues with the N1KV and its interaction with the VMware DVS infrastructure is the limit on the maximum number of ports for the DVS. It is important to be able to create a port profile on the N1KV with enough available ports in order to fulfill your present and future needs. Therefore, most people increase the maximum ports number in the virtual Ethernet (vEth) port profile to a larger number.
From the N1KV side, this increase is fine; from the VMware DVS side, the increase can be a problem. When you create a vEth port profile and enable it, the ports are immediately reserved on the VMware DVS. This can be a problem in the cloud and some enterprise environments. This document examines some limits with the maximum number of ports in order to understand the problem.
Maximum Number of Ports
Here is a list of the maximum number of ports (max ports) per device:
- VMware vCenter Version 4.0 max ports per DVS = 8,192
- VMware vCenter Version 4.1 max ports per DVS = 20,000
- VMware vCenter Version 5.0 max ports per DVS = 30,000
- VMware vCenter Version 5.5 max ports per DVS = 60,000
- N1KV maximum vEth ports = 2,048
Here are some important points about port consumers:
- Every VMware ESX/ESXi host that is added to the N1KV always consumes 32 ports. There is no way to change port consumption for the vEth type of port profiles.
- Every vEth port profile consumes its max ports setting when it is enabled.
- Some ports are internally-consumed by the DVS and N1KV.
You encounter a problem when you configure a N1KV with eight vEth port profiles and 32 ESX hosts. Here are the details:
- N1KV Version SV(1.4) with vSphere 4.0 has a DVS maximum of 8,192 ports.
- A setup with 32 ESX hosts requires 1,024 DVS ports (32 ports x 32 hosts).
- With 8,192 max ports, minus the 1,024-port requirement, only 7,168 max ports are available.
- If there are eight vEth port profiles, each with the max ports set to 1,024, then they use 8,192 ports. But, there are only 7,168 max ports left. Therefore, the N1KV is in a deficit. (8,192 - 7,168 = 1,024)
- Another 1,024 available ports are needed on the DVS in order to make the configuration work.
Until Version SV(1.4a), the solution was to set the max ports number lower, upgrade to a newer version of vSphere, or set the DVS available ports higher with the help of VMware or the Increasing the maximum number of vNetwork Distributed Switch (vDS) ports in vSphere 4.x (1038193) Knowledge Base (KB) article. All of the previous options were compromises or difficult to configure, as in the case of the KB article.
N1KV Version SV(1.4a) and Later Solutions
Two new options were introduced in Version 1.4a that improve interactions with the DVS available ports:
- SVS (Software Virtual Switch) connection max port
- vEth port-binding auto
SVS Connection Max Port
This is a new option under the SVS connection where you can actually increase the number of available DVS ports through the SVS connection between the N1KV and the vCenter Server. You set it with this command:
n1kv-14a(config)# svs connection VC
n1kv-14a(config-svs-conn)# max-ports ?
<0-50000> Number of max-ports for this connection
n1kv-14a(config-svs-conn)# max-ports 9000
In this command, the available ports on a vCenter 4.0 device were changed to 9,000. The downside is that vCenter does not show the extra ports in the vSphere client. However, the next image shows that DVS ports past 8,192 were created:
You must work with your VMware administrator when you make this change. VMware publishes maximum settings for DVS ports for a reason, so you should contact the administrator before you make any drastic changes to this setting.
vEth Port-Binding Auto
The vEth port-binding auto is a great feature. It allocates ports as it needs them, instead of all at once. The auto feature works with static and dynamic port-bindings. Both static and dynamic port-bindings preallocate DVS ports, while the ephemeral does not. In order to turn it on, use this command:
n1kv-14a(config)# port-profile type vethernet profile2
n1kv-14a(config-port-prof)# vmware port-group
n1kv-14a(config-port-prof)# switchport mode access
n1kv-14a(config-port-prof)# switchport access vlan 158
n1kv-14a(config-port-prof)# no shutdown
n1kv-14a(config-port-prof)# max-ports 1024
n1kv-14a(config-port-prof)# port-binding static auto
n1kv-14a(config-port-prof)# state enabled
The port profile is created with 17 ports allocated from the DVS. You can allocate in chunks of 16 up to the max ports for the port profile. When you get within three ports of the current allocation, you add another 16 ports. Here is a screen shot of the previous port profile in vCenter 4.0:
Here is a screen shot after the addition of the 14th Virtual Machine (VM) to port profile profile2:
If you want to preallocate more than 17 ports to the port profile, then use the new min-ports command under port profile. Min-ports preallocates more ports on the DVS so that you have more overhead if you must create large numbers of VMs on the port profile.
n1kv-14a(config)# port-profile profile8
n1kv-14a(config-port-prof)# min-ports 40