This chapter contains the following sections:
Information About VXLANs
Overview of VXLANs
Extensible LAN (VXLAN) technology enables you to create virtual domains by
running a Layer 2 overlay network on top of Layer 3 with MAC-in-UDP
encapsulation and a 24-bit VXLAN ID. The original Layer 2 frame from a Virtual
Machine (VM) is encapsulated from within the Virtual Ethernet Module (VEM).
Each VEM is assigned at least one IP address that is used as the source IP
address when the encapsulated MAC frames are sent to other VEMs over the
network. See the following figure.
The IP addresses,
which are known as VXLAN Tunnel End Point (VTEP) IP addresses, are assigned to
selected vmknics on the corresponding VEM. The encapsulation carries the VXLAN
ID to scope the MAC address of the payload frame. The VM's VXLAN ID is
indicated within the port profile configuration of the vNIC and is applied when
the VM connects to the network.
supports three different modes for flood traffic.
- Multicast mode—A VXLAN uses
an IP multicast network to send broadcast, multicast, and unknown unicast flood
frames. Each multicast mode VXLAN has an assigned multicast group IP address.
When a new VM joins a host in a multicast mode VXLAN, a VEM joins the assigned
multicast group IP address by sending IGMP join messages. Flood traffic
(broadcast, multicast and unknown unicast) from the VM is encapsulated and is
sent using the assigned multicast group IP address as the destination IP
address. Packets sent to known unicast MAC addresses are encapsulated and sent
directly to the destination server VTEP IP addresses.
mode—A VXLAN uses each VEM's single unicast IP address as the destination IP
address to send broadcast, multicast, and unknown unicast flood frames of the
designated VTEP on each VEM that has at least one VM in the corresponding
VXLAN. When a new VM joins the host in a unicast-mode VXLAN, a designated VTEP
is selected for receiving flood traffic on that host. This designated VTEP is
communicated to all other hosts through the Virtual Supervisor Module (VSM).
Flood traffic (broadcast, multicast, and unknown unicast) is replicated on each
VEM's designated VTEP in that VXLAN by encapsulating it with a VXLAN header.
Packets are sent only to VEMs with a VM in that VXLAN. Packets that have a
unicast MAC address are encapsulated and sent directly to the destination
server's VTEP IP address.
mode (supported only in unicast mode)—In this mode, unknown unicast flooding in
the network is eliminated. The VSM learns all the MAC addresses from the VEMs
in all the VXLANs and distributes those MAC addresses with VTEP IP mappings to
other VEMs. Therefore, there is no unknown unicast MAC address in the network
when the VMs on the VEMs are communicating and controlled by the same VM.
works only for static MAC addresses. If dynamic MAC addresses are found on
ports that use VXLANs that operate in MAC distribution mode, syslogs are
generated to indicate that MAC distribution does not work with dynamic MAC
You can configure
the above modes globally and override them for each bridge domain.
default, if you upgrade the VSM from an earlier version of the Cisco Nexus
1000V to the current version with the segmentation feature enabled, all the
VXLANs continue to operate in multicast mode. If you enable the feature when
the VSM is running the current version of the Cisco Nexus 1000V, by default,
the bridge domains change to unicast-only mode. You must explicitly enable MAC
distribution mode because it is disabled by default.
completing the software upgrade, you need to explicitly configure the segment
mode to multicast mode.
During an upgrade,
you cannot enable unicast-only mode unless you upgrade all VEMs and the VEM
Protocol Control Plane Feature
The Border Gateway
Protocol (BGP) control plane feature enables the
Cisco Nexus 1000V to exchange the VXLAN
information collected on the VSM-VTEP flood list across VSMs. The
Cisco Nexus 1000V supports BGP peering between 16 VSMs
to allow VXLAN segments to reach across servers. BGP runs on the VSM and can
exchange VXLAN information with the BGP on any other
Cisco Nexus 1000V.
The Cisco Nexus 1000V can also be used as a route reflector
to exchange VTEP list between VSMs.
The BGP control plane
feature extends the unicast-only mode to a multi-VSM environment using a L2VPN
EVPN address family. The VTEP information is not exchanged with the VSMs that are
running the old version. They will continue to work in multicast mode (VXLAN 1.0)
or unicast-only mode in a single
Cisco Nexus 1000V (VXLAN 1.5).
Each VEM requires at
least one IP/MAC address pair to terminate VXLAN packets. This IP/MAC address
pair is known as the VXLAN Tunnel End Point (VTEP) IP/MAC addresses. The VEM
supports IPv4 addressing for this purpose. The IP/MAC address that the VTEP
uses is configured when you enter the
vxlan command. You can have a maximum of four VTEPs in a single
One VTEP per VXLAN
segment is designated to receive all broadcast, multicast, and unknown unicast
flood traffic for the VEM.
traffic is destined to a VEM that is connected to a different subnet, the VEM
does not use the
VMware host routing table.
it can use one of the following:
Proxy Address Resolution Protocol (ARP): To use Proxy ARP,
you must configure the upstream router for Proxy ARP. The VTEPs initiate the
ARP for remote VEM VTEP IP addresses. If the VTEPs in the different VEMs are in
different subnets, the upstream router can respond using Proxy ARP.
Default Gateway: To use a default gateway, you must configure
the VTEP with the
transport ip address external command to
specify the netmask and gateway IP address for the VTEP to use. For example,
from the interface command mode, enter
transport ip address external netmask 255.255.255.0
VMs brought up
behind VEMs cannot use the transport VLAN of the VTEP, because VLANs used on
VTEPs are isolated and reserved for VXLAN traffic only.
VXLAN termination (encapsulation and decapsulation) is supported on virtual switches. As a result, the only endpoints that can connect into VXLANs are VMs that are connected to a virtual switch. Physical servers cannot be in VXLANs and routers or services that have traditional VLAN interfaces cannot be used by VXLAN networks. The only way that VXLANs can currently interconnect with traditional VLANs is through VM-based software routers.
The supported gateways are as follows:
VMware vShield Edge
Cisco VXLAN Gateway
The configuration for such VLAN-VXLAN translation/mappings for
the VXLAN gateway must be configured from the VSM and must always be a 1:1 mapping for each Layer 2 domain. Each VXLAN gateway can support multiple VLAN-VXLAN mappings.
A VXLAN trunk allows you to trunk multiple VXLANs on a single virtual Ethernet interface.
In order to achieve this configuration, you must encapsulate a VLAN-VXLAN mapping
VLAN-VXLAN mappings are
applied on a virtual Ethernet interface using a port profile. A single port profile
can support multiple VLAN-VXLAN mappings.
You must use the multi-MAC capability feature to mark a
interface as capable of sourcing packets from
multiple MAC addresses.
For example, you can use this feature if you have a virtual Ethernet port and you have enabled VXLAN trunking on it and the VM that is connected to the port bridges packets that are sourced from multiple MAC addresses.
By using this feature, you can easily identify such multi-MAC capable
ports and handle live migration scenarios correctly for those
The VXLAN encapsulation overhead is 50 bytes. In order to prevent performance degradation due to fragmentation, the entire interconnection infrastructure between all VEMs that exchange VXLAN packets must be configured to carry 50 bytes more than what the VM VNICs are configured to send. For example, if the default VNIC configuration is 1500 bytes, the VEM uplink port profile, upstream physical switch port, and interswitch links, and any routers if present, must be configured to carry a maximum transmission unit (MTU) of at least 1550 bytes. If that is not possible, we recommend that you configure the MTU within the guest VMs to be smaller by 50 bytes.
If you do not configure a smaller MTU, the VEM attempts to notify the VM if it performs Path MTU (PMTU) Discovery. If the VM does not send packets with a smaller MTU, the VM fragments the IP packets. Fragmentation occurs only at the IP layer. If the VM sends a frame that is too large, the frame will be dropped after VXLAN encapsulation and if the frame does not contain an IP packet.
Maximum Number of
The Cisco Nexus 1000V
supports up to 6000 VLANs and 6000 VXLANs with a combined maximum of 12000.
Jumbo frames are supported by the Cisco Nexus 1000V if there is space on the frame to accommodate the VXLAN encapsulation overhead of at least 50 bytes, and the physical switch/router infrastructure has the capability to transport these jumbo-sized IP packets.
VXLAN Feature Disabled
As a safety precaution, do not use the no feature segmentation command if there are any ports associated with a VXLAN port profile. You must remove all associations before you can disable this feature. You can use the no feature segmentation command to remove all the VXLAN bridge domain configurations on the Cisco Nexus 1000V.
The Cisco Nexus 1000V
supports offloading VXLAN checksum and TSO computations of inner
packets for VXLAN-encapsulated packets. The VXLAN offload feature
is supported only if an adapter supports the
offload feature and VMware supports the offload feature on that
adapter. To verify if the Cisco Nexus 1000V supports the
VXLAN offload feature on an adapter, use the vemcmd show pd-port command on
the host. The V flag in the Flags column indicates that
the VXLAN offload feature is supported. The TSO computations are
automatically offloaded when the VXLAN offload feature is