• A loopback address is required when using the source-interface config command. The loopback address represents the local VTEP IP.

• For the Network Virtualization Endpoint (NVE) source loop back address, secondary address should be same on both the vPC peers and the primary address should be different.

• To establish IP multicast routing in the core, IP multicast configuration, PIM configuration, and RP configuration is required.

• VTEP to VTEP unicast reachability can be configured through any IGP protocol.

• Bind NVE to a loopback address that is separate from other loopback addresses that are required by Layer 3 protocols. A best practice is to use a dedicated loopback address for VXLAN.

• The loopback address used by NVE needs to be configured to have a primary IP address and a secondary IP address.

• The secondary IP address is used for all VXLAN traffic that includes multicast and unicast encapsulated traffic.

• vPC peers must have identical configurations as listed below:

  • Consistent Bridge-domain to VNI mapping.

  • Consistent NVE binding to the same loopback interface.

  • The same secondary IP address must be confgured on both the switches. However, each vPC peer switch should have a unique primary IP address assigned to it.

  • Consistent NVE interface to VNI mapping.

  • Consistent VNI to group mapping.

• For multicast, the vPC node that receives the (S, G) join from the RP (rendezvous point) becomes the DF (designated forwarder). On the DF node, encap routes are installed for multicast.

  Decap routes are installed based on the election of a decapper from between the VPC primary node and the VPC secondary node. The winner of the decap election is the node with the least cost to the RP or the source. However, if the cost to the RP is the same for both nodes, the VPC primary node is elected.

  The winner of the decap election has the decap mroute installed. The other node does not have a decap route installed.

• On a VPC device, BUM traffic (broadcast, unknown-unicast, and multicast traffic) from hosts is replicated on the vPC peer-link. A copy is made of every native packet and each native packet is sent across the vPC peer-link to service orphan-ports connected to the peer VPC switch.

  To prevent traffic loops in VXLAN networks, native packets ingressing the vPC peer-link cannot be sent to an uplink. However, if the peer switch is the encapper, the copied packet traverses the vPC peer-link and is sent to the uplink.

  In a VXLAN vPC deployment with peer switch, encapsulation profile, and bridge domain configurations, the vPC secondary peer switch does not generate or process BPDUs for bridge domains.

• When vPC peer-link is shut, the loopback primary address is used as the source IP address for encapsulation by both vPC switches.

  Note	Orphans connected to the VPC secondary will experience loss of traffic for the period that the vPC peer-link is shut. This is similar to Layer 2 orphans in a VPC secondary of a traditional VPC setup.

• When vPC peer-link is no-shut, the NVE loopback secondary address is used.

• For VPC, the loopback interface has 2 IP addresses: the primary IP address and the secondary IP address.

  The primary IP address is unique and is used by Layer 3 protocols.

  The secondary IP address on loopback is necessary because the interface NVE uses it for the VTEP IP address. The secondary IP address must be same on both vPC peers.

• The VPC peer-gateway feature must be enabled on both peers. As a best practice, use peer-switch, peer gateway, ip arp sync, ipv6 nd sync configurations for improved convergence in VPC topologies.

  The following is an example (best practice) of a VPC configuration:

  
  switch# sh ru vpc
  
  version 6.1(2)I3(1)
  feature vpc
  vpc domain 2
    peer-switch
    peer-keepalive destination 172.29.206.65 source 172.29.206.64
    peer-gateway
    ipv6 nd synchronize
    ip arp synchronize
  
  

• On a VPC pair, shutting down NVE or NVE loopback on one of the VPC nodes is not a supported configuration. This means that traffic fail over on one-side NVE shut or one-side loopback shut is not supported.
• Redundant anycast RPs configured in the network for multicast load-balancing and RP redundancy are supported on VPC VTEP topologies.

• The following are the examples of SVI with PIM enabled:

  
  switch# show run interface BDI3
  
  interface BDI3
    description special_svi_over_mct
    no shutdown
    ip address 30.2.1.1/30
    ip pim sparse-mode
  
  
  switch# show run interface BDI3
  
  interface BDI3
   description special_svi_over_vPC peer-link no shutdown
   ipv6 address FE80::290:27FF:FE8C:B709
   ip pim sparse-mode
  

  Note	The SVI must be configured on both VPC peers and requires PIM to be enabled.

• As a best practice when changing the secondary IP address of an anycast VPC VTEP, the NVE interfaces on both the VPC primary and the VPC secondary should be shut before the IP changes are made.

• For a VXLAN vPC deployment, you should configure the switchport trunk native vlan tag exclude control command on the interface port channel configured as the vPC peer-link.

• MTU Size in the Transport Network

  Due to the MAC-to-UDP encapsulation, VXLAN introduces 50-byte overhead to the original frames. Therefore, the maximum transmission unit (MTU) in the transport network needs to be increased by 50 bytes. If the overlays use a 1500-byte MTU, the transport network needs to be configured to accommodate 1550-byte packets at a minimum. Jumbo-frame support in the transport network is required if the overlay applications tend to use larger frame sizes than 1500 bytes.

• ECMP and LACP Hashing Algorithms in the Transport Network

  As described in a previous section, Cisco Nexus 7000 Series Switches introduce a level of entropy in the source UDP port for ECMP and LACP hashing in the transport network. As a way to augment this implementation, the transport network uses an ECMP or LACP hashing algorithm that takes the UDP source port as an input for hashing, which achieves the best load-sharing results for VXLAN encapsulated traffic.

• Multicast Group Scaling

  The VXLAN implementation on Cisco Nexus 7000 Series Switches uses multicast tunnels for broadcast, unknown unicast, and multicast traffic forwarding. Ideally, one VXLAN segment mapping to one IP multicast group is the way to provide the optimal multicast forwarding. It is possible, however, to have multiple VXLAN segments share a single IP multicast group in the core network. VXLAN can support up to 16 million logical Layer 2 segments, using the 24-bit VNID field in the header. With one-to-one mapping between VXLAN segments and IP multicast groups, an increase in the number of VXLAN segments causes a parallel increase in the required multicast address space and the amount of forwarding states on the core network devices. At some point, multicast scalability in the transport network can become a concern. In this case, mapping multiple VXLAN segments to a single multicast group can help conserve multicast control plane resources on the core devices and achieve the desired VXLAN scalability. However, this mapping comes at the cost of suboptimal multicast forwarding. Packets forwarded to the multicast group for one tenant are now sent to the VTEPs of other tenants that are sharing the same multicast group. This causes inefficient utilization of multicast data plane resources. Therefore, this solution is a trade-off between control plane scalability and data plane efficiency.

  Despite the suboptimal multicast replication and forwarding, having multiple-tenant VXLAN networks to share a multicast group does not bring any implications to the Layer 2 isolation between the tenant networks. After receiving an encapsulated packet from the multicast group, a VTEP checks and validates the VNID in the VXLAN header of the packet. The VTEP discards the packet if the VNID is unknown to it. Only when the VNID matches one of the VTEP’s local VXLAN VNIDs, does it forward the packet to that VXLAN segment. Other tenant networks will not receive the packet. Thus, the segregation between VXLAN segments is not compromised.

• Spanning Tree Protocol (STP) domain

  Ensure that the root of an STP domain local to the VXLAN fabric is a VTEP, or placed within the fabric. The STP root should not be outside the VXLAN fabric (below the VTEPs) since it will lead to Layer 2 loops.

The following are considerations for the configuration of the core VRF:

• On the VTEP device:

  • Enable and configure IP multicast.

  • Create and configure a loopback interface with a /32 IP address.

  • Enable IP multicast on the loopback interface.

  • Advertise the loopback interface/32 addresses through the routing protocol that runs in the transport network.

  • Enable IP multicast on the uplink outgoing physical interface.

• Throughout the transport network:

  • Enable and configure IP multicast.