Configuring VXLANs
Download this chapter
Configuring VXLANsConfiguring VXLANs
 Feedback

Configuring VXLANs

This chapter contains the following sections:

Prerequisites for VXLANs

VXLANs have the following prerequisites:

  • The Cisco Nexus 1000V uplink port profiles and all interconnecting switches and routers between the KVM hosts must have their supported maximum transmission unit (MTU) set to at least 50 bytes larger than the MTU of the Virtual Machines (VMs). For example, the VMs default to using a 1500 byte MTU (same as the uplinks and physical devices), so you must set them to at least 1550 bytes. If this configuration is not possible, you should lower all VM vNICs MTU to 50 bytes smaller than what the physical network supports, such as 1450 bytes. For more information, see the Cisco Nexus 1000V Port Profile Configuration Guide.
  • If the Cisco Nexus 1000V is using a port channel for its uplinks, you should set the load distribution algorithm to a 5-tuple hash (IP/Layer 4/Layer 4 ports). Use the same setting for any port channels on the physical switches. For more information, see the Cisco Nexus 1000V Interface Configuration Guide.
  • By default, VXLAN uses MAC in IP (UDP) with a destination port of 8472. However, you can change this setting to the IETF approved value of 4789. Whichever port you use, you must allow it through any intermediate firewall.
  • If you are using the VXLAN multicast mode, you must configure an IGMP querier in the VXLAN transport VLANs.

Guidelines and Limitations for VXLANs

VXLAN has the following configuration guidelines and limitations:

  • When encapsulated traffic is destined to a VEM that is connected to a different subnet, the VEM does not use the Linux host routing table. Instead, it can use either Proxy Address Resolution Protocol (ARP) or a default gateway.
    • To use Proxy ARP, you must configure the upstream router for Proxy ARP. With ARP configured, if the remote VTEP is in the same subnet as the VXLAN Gateway, the VEM uses ARP to obtain the IP address of the remote VTEP. If the remote VTEP is in a different subnet than the VXLAN Gateway, the VEM uses ARP to obtain the IP address of the VXLAN Gateway.
    • To use a default gateway, you must configure the VTEP with the transport ip address external command to specify the netmask and gateway IP address for the VTEP to use. For example, from the interface command mode, enter transport ip address external netmask 255.255.255.0 gateway 1.2.3.4.
  • If you configure load-balancing with a VPC-HM where multiple VTEPS exist in the same subnet on the KVM platform, you might experience a Linux kernel issue where ARP responses from the Linux kernel for the VTEPs might have the wrong MAC address. This situation could adversely affect the flow of VXLAN traffic. To work around this issue, see Configuring the Linux System for Multiple VTEPS in vPC-HM.
  • VXLANs in unicast-only mode are supported only between VTEPs that are managed by a single VSM. A VXLAN in unicast-only mode cannot be shared across two different distributed virtual switches.
  • When a VXLAN is configured in the unicast-only mode with MAC distribution enabled, the VXLAN gateway does not register any MAC addresses that it learns on the VLAN side. If these MAC addresses have not been learned yet, the traffic to these MAC addresses is delivered by replicating of unknown unicast packets to the VXLAN gateway. This is the only scenario where unknown unicast packets are replicated in the MAC distribution mode.
  • Microsoft Network Load Balancing (NLB) servers in unicast mode require unknown unicast packets to be delivered to all the server ports, because the shared MAC address of the NLB servers is never discovered. This solution will break the unknown unicast semantics of unicast-only mode with MAC distribution. We recommend that you use either multicast mode or unicast-only mode without MAC distribution.
  • You cannot enable the MAC distribution mode and the multi-MAC capability feature together. You must use either the MAC distribution or the muti-MAC capability feature.

VXLAN has the following configuration guidelines and limitations for changing the VXLAN configuration:

  • Use the segment mode unicast-only command to change the global configuration mode from multicast to unicast. This command affects all bridge domains with no overrides.
  • You can use multicast or unicast mode if you override the global configuration for the bridge domain by entering the segment mode unicast-only or no segment mode unicast-only commands.
  • You can enable the segment distribution MAC command only after entering the segment mode unicast-only command.
  • You can disable the segment distribution MAC address configuration globally by entering the no segment distribution mac command.
  • You cannot use the no segment mode unicast-only command if you already entered the segment distribution MAC command.
  • You must configure a multicast IP address that is required for a VXLAN that is in the multicast mode.
  • If you remove the multicast IP address while VXLAN is in the multicast mode, the ports that use that VXLAN go to the inactive state.

Note

Ports become inactive if you change the mode from unicast to multicast if a multicast IP address is not configured or a segment ID is removed.

Default Settings for VXLANs

The following table lists the default settings for VXLAN parameters.

Table 1 Default VXLAN Parameters

Parameter

Default

Feature Segmentation

Disabled

Configuring VXLANs

Steps to Configure VXLANs

To configure VXLANs, follow these steps:

Before You Begin
Procedure
     Command or ActionPurpose
    Step 1Enable the Segmentation feature, which allows you to configure VXLANs. 

    See Enabling the VXLAN Segmentation Feature.

     
    Step 2Configure VTEPs for VXLAN encapsulation.  

    See Configuring VTEPs for VXLAN Encapsulation.

     
    Step 3Create a bridge domain. 

    See Configuring a VXLAN Bridge Domain.

     
    Step 4Create a Port Profile for a VXLAN. 

    See Configuring a vEthernet Port Profile for a VXLAN

     
    Step 5If you plan to configure multiple VTEPs in Virtual port channel host mode (vPC-HM) for load balancing in the same subnet, you need to make changes to the sysctl file in the Linux system.  (Optional)

    See Configuring the Linux System for Multiple VTEPS in vPC-HM.

     

    Enabling the VXLAN Segmentation Feature

    Procedure
       Command or ActionPurpose
      Step 1switch# configure terminal 

      Enters global configuration mode.

       
      Step 2switch(config)# feature segmentation 

      Enables the VXLAN segmentation feature.

       
      Step 3switch(config)# show feature | grep segmentation   (Optional)

      Displays whether the VXLAN feature is enabled.

       
      Step 4switch(config)# copy running-config startup-config  (Optional)

      Saves the change persistently through reboots and restarts by copying the running configuration to the startup configuration.

       

      This example shows how to enable the VXLAN segmentation feature:

      switch# configure terminal 
switch(config)# feature segmentation 
switch(config)# show feature | grep segmentation 
network-segmentation 1 disabled 
segmentation         1 enabled 
switch(config)# copy running-config startup-config

      Configuring VTEPs for VXLAN Encapsulation

      Before You Begin
      • Identify a VLAN to be used for transporting VXLAN-encapsulated traffic.
      • Ensure that it is configured on the uplink port profile for all VEMs on which the VXLAN can be configured.
      Procedure
         Command or ActionPurpose
        Step 1switch# configure terminal 

        Enters global configuration mode.

         
        Step 2switch(config)# port-profile type veth profilename  

        Enters port profile configuration mode for the named port profile. If the port profile does not already exist, it is created using the following characteristics:

        • profilename—The port profile name can be up to 80 characters and must be unique for each port profile on the Cisco Nexus 1000V.
        Note   

        If a port profile is configured as an Ethernet type, it cannot be used to configure VTEPs.

         
        Step 3switch(config-port-prof)# switchport mode access 

        Designates the interfaces as switch access ports (the default).

         
        Step 4switch(config-port-prof)# switchport access vlan id  

        Assigns a VLAN ID to this port profile.

        Note   

        A VLAN ID must be created and should be in the active state.

         
        Step 5switch(config-port-prof)# capability vxlan  

        Assigns the VXLAN capability to the port profile to ensure that the interfaces that inherit this port profile are used as sources for VXLAN-encapsulated traffic.

         
        Step 6switch(config-port-prof)# no shutdown  

        Administratively enables all ports in the profile.

         
        Step 7switch(config-port-prof)# state enabled  

        Sets the operational state of a port profile.

         
        Step 8switch(config-port-prof)# publish port-profile 

        Pushes the port profile to the OpenStack controller.

         
        Step 9switch(config-port-prof)# show port-profile name profilename  

        Displays the port profile configuration.

         
        Step 10switch(config-port-prof)# copy running-config startup-config  (Optional)

        Saves the change persistently through reboots and restarts by copying the running configuration to the startup configuration.

         

        This example shows how to configure an interface for VXLAN encapsulation:

        switch# configure terminal
switch(config)# port-profile type veth vxlan-pp
switch(config-port-prof)# switchport mode access
switch(config-port-prof)# switchport access vlan 100 
switch(config-port-prof)# capability vxlan
switch(config-port-prof)# no shutdown
switch(config-port-prof)# state enabled
switch(config-port-prof)# publish port-profile
switch(config-port-prof)# show port-profile name vxlan-pp
port-profile vxlan-pp
type: Vethernet
description:
status: enabled
max-ports: 32
min-ports: 1
inherit:
config attributes:
switchport mode access
switchport access vlan 100
capability vxlan
no shutdown
evaluated config attributes:
switchport mode access
switchport access vlan 100
capability vxlan
no shutdown
assigned interfaces:
port-group: vmknic-pp
system vlans: none
capability l3control: no
capability iscsi-multipath: no
capability vxlan: yes
capability l3-vservice: no
port-profile role: none
port-binding: static

switch(config-port-prof)# 
switch(config-port-prof)# copy running-config startup-config

        Configuring a VXLAN Bridge Domain

        You are limited to creating a maximum of 2048 VXLAN bridge domains.

        Procedure
           Command or ActionPurpose
          Step 1switch# configure terminal 

          Enters global configuration mode.

           
          Step 2switch(config)# bridge-domain name-string 

          Creates a VXLAN and associates an identifying name to it.

           
          Step 3switch(config-bd)# segment id [number]  

          Specifies the VXLAN segment ID. Only one bridge domain can use a particular segment ID value.

          Valid values are from 4096 to 16000000. (1 to 4095 are reserved for VLANs.)

           
          Step 4switch(config-bd)# group ipaddr   (Optional)

          Associates the multicast group for broadcasts and floods.

          Note   

          Reserved multicast addresses are not allowed.

           
          Step 5switch (config-bd)# [no] segment mode unicast-only | default segment mode  (Optional)

          Enables the segment mode as unicast only.

          The mode can be configured globally or for a specific bridge domain. When configured under a specific bridge domain, the mode is treated as an override to the global configuration for that specific bridge domain. Any change in the global configuration affects all the bridge domains that do not have overrides. The mode configuration on a specific bridge domain overwrites the global bridge domain. The overrides configured on the bridge domain can be removed by using the default segment mode.

          Note   

          Use the no segment mode unicast-only command to override the configuration under a bridge domain. If you have unicast enabled globally, the bridge domain can use the multicast mode. To override, use the default segment mode command.

           
          Step 6switch (config-bd)# [no] segment distribution mac | default segment distribution mac  (Optional)

          Enables MAC distribution for the bridge domain.

          Note   

          To configure an override under a bridge domain, you must enter the segment mode unicast-only command as an override first.

           
          Step 7switch(config-bd)# show bridge-domain name-string   (Optional)

          Displays bridge domain information.

           
          Step 8switch(config-bd)# copy running-config startup-config  (Optional)

          Saves the change persistently through reboots and restarts by copying the running configuration to the startup configuration.

           

          This example shows how to create a VXLAN for multicast mode.:

          switch# configure terminal
switch(config)# bridge-domain tenant-red
switch(config-bd)# segment id 4096
switch(config-bd)# group 239.1.1.1
switch(config-bd)# no segment mode unicast-only
switch(config-bd)# show bridge-domain tenant-red 
Bridge-domain tenant-red (0 ports in all)
Segment ID: NULL
Mode: Unicast-only (default)
MAC Distribution: Disable (default)
Group IP: 239.1.1.1
State: UP Mac learning: Enabled
switch(config-bd)#
switch(config-bd)# copy running-config startup-config

          Configuring a vEthernet Port Profile for a VXLAN

          Alternatively, you can associate ports with a bridge domain by modifying the configuration of an existing virtual Ethernet port profile to use VXLANs instead of VLANs. To do so, enter the switchport access bridge-domain name command on a profile with switchport mode access configured.

          Procedure
             Command or ActionPurpose
            Step 1switch# configure terminal 

            Enters global configuration mode.

             
            Step 2switch(config)# port-profile [type {vethernet}] name  

            Enters port profile configuration mode for the named port profile. If the port profile does not already exist, it is created using the following characteristics:

            • name—The port profile name can be up to 80 characters and must be unique for each port profile on the Cisco Nexus 1000V.
            • type—The port profile type is virtual Ethernet. Once configured, the type cannot be changed. The default is the virtual Ethernet type.
             
            Step 3switch(config-port-prof)# switchport mode access  

            Designates that the interfaces are to be used as trunking ports.

            A trunk port transmits untagged packets for the native VLAN and transmits encapsulated, tagged packets for all other VLANs.

             
            Step 4switch(config-port-prof)# switchport access bridge-domain <bridge-domain name>  

            Assigns a VXLAN bridge domain to this port profile.

            You must configure the bridge domain with its segment ID for the port to be active. You should configure a multicast IP address if you prefer multicast mode. The multicast mode is displayed in the running configuration as no segment mode unicast-only.

             
            Step 5switch(config-port-prof)# no shutdown  

            Administratively enables all ports in the profile.

             
            Step 6switch(config-port-prof)# state enabled  

            Sets the operational state of a port profile.

             
            Step 7switch(config-port-prof)# publish port-profile  

            Pushes the port profile to the VEM.

             
            Step 8switch(config-port-prof)# show port-profile [brief | expand-interface | usage] [name profile-name]   (Optional)

            Displays the configuration for verification.

             
            Step 9switch(config-port-prof)# show running-config bridge-domain   (Optional)

            Displays the segmentation configuration.

             
            Step 10switch(config-port-prof)# copy running-config startup-config  (Optional)

            Saves the change persistently through reboots and restarts by copying the running configuration to the startup configuration.

             

            This example shows how to create a port profile configured to use a VXLAN: 

            switch# configure terminal
switch(config)# port-profile tenant-profile 
switch(config-port-prof)# switchport mode access
switch(config-port-prof)# switchport access bridge-domain tenant-red
switch(config-port-prof)# no shutdown 
switch(config-port-prof)# state enabled 
switch(config-port-prof)# publish port-profile
switch(config-port-prof)# show port-profile name tenant-profile
port-profile tenant-profile
type: Vethernet
description:
status: enabled
max-ports: 32
min-ports: 1
inherit:
config attributes:
switchport mode access
switchport access bridge-domain tenant-red
no shutdown
evaluated config attributes:
switchport mode access
switchport access bridge-domain tenant-red
no shutdown
assigned interfaces:
port-group: tenant-profile
system vlans: none
capability l3control: no
capability iscsi-multipath: no
capability vxlan: no
capability l3-vservice: no
port-profile role: none
port-binding: static

switch(config-port-prof)# 
switch(config-port-prof)# show running-config bridge-domain
switch(config-port-prof)# copy running-config startup-config

            Configuring the Linux System for Multiple VTEPS in vPC-HM

            You can configure multiple VTEPs in Virtual port channel host mode (vPC-HM) for load balancing. If these VTEPs are in the same subnet, you must modify the sysctl settings in the Linux system and save the changes persistently through reboots and restarts.


            Note

            If you created the VTEPs before you made changes to the sysctl settings, you must reboot the Linux system for the settings to take effect.

            Procedure
              Step 1   Apply the following sysctl settings to the Linux system before you create the VTEPS. 
              sysctl -w net.ipv4.default.vxlannic0.rp_filter=2
sysctl -w net.ipv4.all.vxlannic1.rp_filter=2
sysctl -w net.ipv4.conf.default.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2
sysctl -w net.ipv4.conf.default.arp_announce=2
              Step 2   Save the changes persistently through reboots and restarts by adding the following configuration to the /etc/sysctl.conf file. 
              net.ipv4.conf.default.rp_filter=2
net.ipv4.conf.all.rp_filter=2
net.ipv4.conf.default.arp_ignore=1
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.default.arp_announce=2
              Step 3   If you created the VTEPs before you made changes to the sysctl settings, you must reboot the Linux system for the settings to take effect.

              Removing Ports from a VXLAN

              By performing this procedure, you move the ports to the default VLAN.

              Procedure
                 Command or ActionPurpose
                Step 1switch# configure terminal 

                Enters global configuration mode.

                 
                Step 2switch(config)# port-profile [type {vethernet}] name 

                Enters port profile configuration mode for the named port profile. If the port profile does not already exist, it is created using the following characteristics:

                • name—The port profile name can be up to 80 characters and must be unique for each port profile on the Cisco Nexus 1000V.
                • type—The port profile type is vEthernet. Once configured, the type cannot be changed. The default is the vEthernet type. Defining a port profile type as Ethernet allows the port profile to be used for physical (Ethernet) ports. In vCenter Server, the corresponding port group can be selected and assigned to physical ports (PNICs).
                  Note   

                  If a port profile is configured as an Ethernet type, it cannot be used to configure VMware virtual ports.

                 
                Step 3switch(config-port-prof)# no switchport access bridge-domain  

                Removes the VXLAN bridge domain from this port profile.

                 
                Step 4switch(config-port-prof)# show port-profile usage  (Optional)

                Displays a list of interfaces that inherited a port profile.

                 
                Step 5switch(config-port-prof)# show bridge-domain   (Optional)

                Displays all bridge domains.

                 
                Step 6switch(config-port-prof)# copy running-config startup-config  (Optional)

                Saves the change persistently through reboots and restarts by copying the running configuration to the startup configuration.

                 

                This example shows how to remove ports from a VXLAN:

                switch# configure terminal
switch(config)# port-profile tenant-profile 
switch(config-port-prof)# no switchport access bridge-domain tenant-red 
switch(config-port-prof)# show port-profile usage
switch(config-port-prof)# show bridge-domain 
switch(config-port-prof)# copy running-config startup-config

                Deleting a VXLAN Bridge Domain

                When you delete an existing bridge domain with ports on it, all the ports are moved to a down state and traffic stops flowing.

                Procedure
                   Command or ActionPurpose
                  Step 1switch# configure terminal 

                  Enters global configuration mode.

                   
                  Step 2switch(config)# no bridge-domain name 

                  Deletes a VXLAN.

                   
                  Step 3switch(config-bd)# show bridge-domain  (Optional)

                  Displays all bridge domains.

                   
                  Step 4switch(config-bd)# copy running-config startup-config  (Optional)

                  Copies the running configuration to the startup configuration.

                   

                  This example shows how to delete a VXLAN:

                  switch# configure terminal
switch(config)# no bridge-domain group-red
switch(config)# show bridge-domain
switch(config)# copy running-config startup-config

                  Disabling the VXLAN Segmentation Feature

                  Procedure
                     Command or ActionPurpose
                    Step 1switch# configure terminal 

                    Enters global configuration mode.

                     
                    Step 2switch(config)# show bridge-domain 

                    Displays all bridge domains.

                    Note   

                    You must identify all bridge domains with nonzero port counts.

                     
                    Step 3switch(config)# show running port-profile  (Optional)

                    Displays the running configuration for all port profiles.

                    Note   

                    You must use this command to identify which port profiles have bridge domains identified in Step 2 configured.

                     
                    Step 4switch(config)# port-profile name 

                    Names the port profile and enters port profile configuration mode. If the port profile does not already exist, it is created using the following characteristics:

                    name—The port profile name can be up to 80 characters and must be unique for each port profile on the Cisco Nexus 1000V.

                    Note   

                    If a port profile is configured as an Ethernet type, it cannot be used to configure VMware virtual ports.

                     
                    Step 5switch(config-port-prof)# no switchport access bridge-domain name-string  

                    Removes the VXLAN bridge domain from this port profile and moves the ports to VLAN1.

                     
                    Step 6switch(config-port-prof)# show port-profile usage  (Optional)

                    Displays a list of interfaces that inherited a port profile.

                     
                    Step 7switch(config-port-prof)# show bridge-domain   (Optional)

                    Displays all bridge domains.

                     
                    Step 8switch(config-port-prof)# no feature segmentation  

                    Removes the segmentation feature.

                     
                    Step 9switch(config-port-prof)# show feature | grep segmentation   (Optional)

                    Displays if the segmentation feature is running or not running.

                     
                    Step 10switch(config-port-prof)# copy running-config startup-config  (Optional)

                    Saves the change persistently through reboots and restarts by copying the running configuration to the startup configuration.

                     

                    This example shows how to disable segmentation:

                    switch# configure terminal
switch(config)# show bridge-domain

Global Configuration:
Mode: Unicast-only
MAC Distribution: Disable

Bridge-domain tenant-red (4 ports in all)
Segment ID: 4096 (Manual/Active)
Mode: Unicast-only
MAC Distribution: Disable
Group IP: NULL
State: UP Mac learning: Enabled
Veth1, Veth2, Veth4, Veth11

switch(config)# show running-config port-profile
port-profile default max-ports 32
port-profile default port-binding static
port-profile type ethernet Unused_Or_Quarantine_Uplink
vmware port-group
shutdown
description Port-group created for Nexus1000V internal usage. Do not use.
state enabled
port-profile type vethernet Unused_Or_Quarantine_Veth
vmware port-group
shutdown
description Port-group created for Nexus1000V internal usage. Do not use.
state enabled
port-profile type vethernet tenant-profile
vmware port-group
switchport mode access
switchport access bridge-domain tenant-red
no shutdown
state enabled

switch(config)#
switch(config-port-prof)# show port-profile usage

port-profile Unused_Or_Quarantine_Uplink

port-profile Unused_Or_Quarantine_Veth

port-profile tenant-profile
Vethernet1
Vethernet2
Vethernet4
Vethernet11

switch(config-port-prof)# show bridge-domain

Global Configuration:
Mode: Unicast-only
MAC Distribution: Disable

Bridge-domain tenant-red (0 ports in all)
Segment ID: 4096 (Manual/Active)
Mode: Unicast-only
MAC Distribution: Disable
Group IP: NULL
State: UP Mac learning: Enabled

switch(config-port-prof)#
switch(config-port-prof)# no feature segmentation
switch(config-port-prof)# 2013 May 23 05:34:42 switch-cy %SEG_BD-2-SEG_BD_DISABLED: Feature Segmentation disabled

switch(config-port-prof)# show feature | grep seg_bd
- NR - 1 - seg_bd

                    Verifying the VXLAN Configuration

                    To display the VXLAN configuration information, perform one of the following tasks:

                    Command

                    Purpose

                    show feature | grep segmentation

                    Displays if the segmentation feature is running.

                    show bridge-domain

                    Displays all bridge domains with the mode.

                    show bridge-domain vteps

                    Displays the bridge domain-to-VTEP mappings that are maintained by the VSM and are pushed to all VEMs.

                    show bridge-domain mac bd-name

                    Displays all the MAC addresses that are learned by the VSMs on VXLANs that are configured with the MAC distribution feature.

                    show run bridge-domain

                    Displays the running bridge domain.

                    show bridge-domain bd-name

                    Displays the specified bridge domain.

                    show bridge-domain bd-name vteps

                    Displays the specific bridge domain-to-VTEP mappings that are maintained by the VSM and are pushed to all VEMs.

                    show interface brief

                    Displays a short version of the interface configuration.

                    show interface switchport

                    Displays information about switchport interfaces.

                    show module vteps

                    Displays the IP addresses available on each module that can be used for VXLAN Tunnel Endpoints.

                    Feature History for VXLAN

                    Feature Name

                    Releases

                    Feature Information

                    VXLAN

                    Release 5.2(1)SK1(1.1)

                    Introduced the Virtual Extensible Local Area Network (VXLAN) feature, including the enhanced VXLAN commands.