Configuring vPC Fabric Peering

This chapter contains these sections:

vPC fabric peerings

A vPC fabric peering is a switch fabric feature that

  • enables enhanced dual-homing access without using physical ports for a vPC peer link,

  • retains all core characteristics of traditional vPCs, and

  • uses VXLAN encapsulation for communication between vPC member switches.

The following features and behaviors characterize vPC fabric peering:

  • Port-channels use virtual members (tunnels) rather than physical peer links.

  • Up/down events are triggered by route updates and changes in the fabric.

  • Uplink tracking with state dependency and up/down signalization for vPCs.

  • Positive uplink state tracking drives vPC primary role election.

  • Communication is routed through the network fabric, such as the spine, instead of dedicated peer links.

  • The control plane operates with increased resiliency over TCP/IP (CFSoIP).

  • Data plane traffic travels over VXLAN tunnels.

  • Infra-VLANs are not required for fabric peering.

  • Enhances forwarding to orphan hosts by extending the VIP/PIP feature to Type-2 routes.


Note

The vPC fabric peering counts as three VTEPs, unlike a normal vPC which counts as one VTEP.

Supported platforms and limitations for vPC fabric peering

This topic outlines supported platforms, software releases, key feature limitations, configuration recommendations, and unsupported scenarios for vPC fabric peering on Cisco Nexus switches.

Configuration recommendations

  • vPC Fabric Peering requires TCAM carving of the region ing-flow-redirect. TCAM carving requires saving the configuration and reloading the switch prior to using the feature.


    Note

    This requirement only applies to Cisco Nexus 9300-EX, 9300-FX, 9300-FX2, 9300-FX3, and 9364C platform switches.

  • Before reconfiguring the vPC Fabric Peering source and destination IP, shut down the vPC domain. After adjusting the IPs, reenable the domain no shutdown .

  • Only class A, B, and C IP addresses are supported for the virtual peer-link destination ; class D and E are not supported.

  • VLAN inconsistency within a vPC environment only suspends the affected VLANs instead of bringing down the entire vPC leg on the secondary switch.

  • After converting from fabric peering to a physical peer link, both peers must:

    1. Globally configure a TCAM region using the hardware access-list tcam region ing-flow-redirect 0 command.

    2. Optionally, allocate the free space to other classes. For more information, see Understand How to Carve Nexus 9000 TCAM Space.

    3. Save the running configuration using the copy running-config startup-config command.

    4. Reload the switch.

  • Layer 3 Tenant Routed Multicast (TRM) is supported, but Layer 2/Layer 3 TRM (Mixed Mode) is not supported.

  • If Type-5 routes are used with this feature, the advertise-pip command is required.

Supported feature, release and platforms

  • Cisco Nexus 9332C, 9364C, and 9300-EX/FX/FXP/FX2/FX3/GX/GX2/H2R/H1 platform switches support vPC Fabric Peering. Cisco Nexus 9200 and 9500 platform switches do not support vPC Fabric Peering.


    Note

    For Cisco Nexus 9300-EX switches, mixed-mode multicast and ingress replication are not supported. VNIs must be configured with either multicast or IR underlay, but not both.

  • Beginning with Cisco NX-OS Release 10.3(2)F, the vPC Fabric Peering is supported for IPv6 underlay on Cisco Nexus 9300-EX/FX/FXP/FX2/FX3/GX/GX2 ToR switches.

  • Beginning with Cisco NX-OS Release 10.4(1)F, the vPC Fabric Peering is supported for IPv6 underlay on Cisco Nexus 9332D-H2R switches.

  • Beginning with Cisco NX-OS Release 10.4(2)F, the vPC Fabric Peering is supported for IPv6 underlay on Cisco Nexus 93400LD-H1 switches.

  • Beginning with Cisco NX-OS Release 10.4(3)F, vPC Fabric Peering is supported for IPv6 underlay on Cisco Nexus 9364C-H1 switches.

Unsupported features

  • The vPC Fabric Peering domain cannot act as a Multi-Site vPC BGW.

  • VTEPs behind vPC ports are not supported. This means that virtual peer-link peers cannot act as a transit node for the VTEPs behind the vPC ports.

  • SVI and sub-interface uplinks are not supported.

  • Routing over SVIs for vPC fabric peering pairs is not supported.

ISSU

  • Beginning with Cisco NX-OS Release 10.2(3)F, ND-ISSU and LXC-ISSU are supported with vMCT for IPv4 underlay on Cisco Nexus 9300-EX/FX/FXP/FX2/FX3/GX/GX2 ToR switches.

  • Beginning with Cisco NX-OS Release 10.3(2)F, ND-ISSU and LXC-ISSU are supported with vMCT for IPv6 underlay on Cisco Nexus 9300-EX/FX/FXP/FX2/FX3/GX/GX2 ToR switches.

QoS configuration for vPC fabric peering

  • The vPC Fabric Peering peer-link is established over the transport (spine) network. As communication between vPC peers occurs in this manner, control plane information CFS messages used to synchronize port state information, VLAN information, VLAN-to-VNI mapping, host MAC addresses are transmitted over the fabric. CFS messages are marked with the appropriate DSCP value, which should be protected in the transport network. The following example shows a sample QoS configuration on the spine layer of Cisco Nexus 9000 Series switches.

    Classify traffic by matching the DSCP value (DSCP 56 is the default value):

    class-map type qos match-all CFS
                        match dscp 56

    Set traffic to the qos-group that corresponds with the strict priority queue for the appropriate spine switch. In this example, the switch sends traffic to qos-group 7, which corresponds to the strict priority queue (Queue 7). Note that different Cisco Nexus platforms might have a different queuing structure. 

    policy-map type qos CFS
                        class CFS
                        Set qos-group 7

    Assign a classification service policy to all interfaces toward the VTEP (the leaf layer of the network):

    interface Ethernet 1/1
                        service-policy type qos input CFS

Orphon port support

  • Enhance forwarding to orphan hosts by extending the VIP/PIP feature to Type-2 routes.

  • An orphan Type-2 host is advertised using PIP. A vPC Type-2 host is advertised using VIP. This is the default behavior for a Type-2 host.

    To advertise an orphan Type-5 route using PIP, you need to advertise PIP under BGP.

  • For orphan ports, it is highly recommended to configure vpc orphan-port suspend command on both vPC nodes, to avoid traffic disruption during NVE failure scenarios.

  • Traffic from remote VTEP to orphan hosts would land on the actual node which has the orphans. Bouncing of the traffic is avoided.


    Note

    When the vPC leg is down, vPC hosts are still advertised with the VIP IP.

  • ARP behavior differs between vPC fabric peering and physical peer links for orphan ports. vPC fabric peering does not sync ARP entries as it does with physical peer links. An orphan Type-2 host is advertised using the PIP in vPC Fabric peering.

FEX support

  • Beginning with Cisco NX-OS Release 10.1(1), FEX Support is provided with vMCT for IPv4 underlay on Cisco Nexus 9300-EX/FX/FX2/FX3 platform switches.

  • Beginning with Cisco NX-OS Release 10.2(2)F, FEX Support is provided with vMCT for IPv4 underlay on Cisco Nexus 9300-GX platform switches.

  • Beginning with Cisco NX-OS Release 10.1(1), vPC Fabric Peering supports FEX in Straight Through and Active-Active (dual home) modes in N9K-C9336C-FX2-E, N9K-C93108TC-EX, N9K-C93108TC-FX,N9K-C93180YC-EX, N9K-C93180YC-FX, N9K-C93216TC-FX2, N9K-C93240YC-FX2, N9K-C93360YC-FX2, N9K-C9336C-FX2, N9K-C93180YC-FX3, N9K-C93180YC-FX3S platform switches.

    Refer to Cisco Nexus 2000 Series NX-OS Fabric Extender Configuration Guide for Cisco Nexus 9000 Series Switches for details on FEX (Straight Through and Active-Active modes).

  • vMCT for IPv6 underlay does not support attaching FEX to it.

Configure vPC fabric peering connections

Use this task to configure vPC Fabric Peering, including underlay and overlay protocols, TCAM carving, and vPC domain settings for both IPv4 and IPv6 environments.

Follow these steps to configure vPC fabric peering connections:

Before you begin

  • Verify that all ports connected to spines are configured as port-type fabric.

  • Do not use SVI or sub-interfaces for underlay connectivity—only L3 physical interfaces or L3 port channels are supported.

  • Ensure the vPC fabric peering DSCP value is consistent on both vPC switches and that the corresponding QoS policy matches the DSCP marking. Use show vpc virtual-peerlink vlan consistency command.

  • Confirm proper VLAN-to-VXLAN (vn-segment) mapping for all VLANs that require communication across the vPC fabric peering.

  • Configure peer-keepalive for vPC fabric peering using one of the supported interfaces.

    • Management interface

    • Dedicated Layer 3 link in default or non-default VRF

    • Loopback interface reachable using the spine.

  • All VLANs that require communication traversing the vPC fabric peering must have a VXLAN enabled (vn-segment), including the native VLAN.


    Note

    For MSTP, VLAN 1 must be extended across vPC Fabric Peering if the peer-link and vPC legs have the default native VLAN configuration. This behavior can be achieved by extending VLAN 1 over VXLAN (vn-segment). If the peer-link and vPC legs have non-default native VLANs, those VLANs must be extended across vPC Fabric Peering by associating the VLANs with VXLAN (vn-segment).

Procedure

Step 1

Enable required features and configure underlay routing protocol.

Example:

configure terminal
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature vpc

feature nv overlay

Step 2

Configure TCAM carving as required for your platform.

Example:

hardware access-list tcam region ing-racl 0
hardware access-list tcam region ing-sup 768
hardware access-list tcam region ing-flow-redirect 512

Minimum size for the Ingress-Flow-redirect TCAM region is 512, and should always be in multiples of 512.

  • TCAM carving for ing-flow-redirect region is only required on Cisco Nexus 9300-EX, 9300-FX, 9300-FX2, 9300-FX3, and 9364C platform switches.

  • Switch reload is required for the TCAM carving to take effect.

Step 3

Configure the vPC domain and peer connections.

  1. For IPv4:

    Example:

    vpc domain 100
peer-keepalive destination 192.0.2.1 
virtual peer-link destination 192.0.2.100 source 192.0.2.20/32 [dscp <dscp-value>] 
Warning: Appropriate TCAM carving must be configured for virtual peer-link vPC
peer-switch
peer-gateway
ip arp synchronize
ipv6 nd synchronize
exit

  2. For IPv6

    Example:

    vpc domain 100
peer-keepalive destination 192:0:2::1 
virtual peer-link destination 192:0:2::100 source 192:0:2::20/32 [dscp <dscp-value>] 
Warning: Appropriate TCAM carving must be configured for virtual peer-link vPC
peer-switch
peer-gateway
ipv6 arp synchronize
ipv6 nd synchronize
exit

Note

 

The dscp keyword is optional. Range is 1 to 63. The default value is 56.

Step 4

Configure the vPC fabric peering port channel.

No need to configure members for the following port channel.

Example:

interface port-channel 10
switchport
switchport mode trunk
vpc peer-link

Step 5

Configure loopback interfaces for peer-link and underlay routing.

Example:

interface loopback0

Note

 

This loopback is not the NVE source-interface loopback (interface used for the VTEP IP address).

  1. Configure loopback 0 for IPv4 underlay.

    Example:

    interface loopback 0
ip address 192.0.2.20/32
ip router ospf 1 area 0.0.0.0

  2. Configure loopback 0 for IPv6 underlay.

    Example:

    interface loopback 0
ipv6 address 192:0:2::20/32
ipv6 router ospfv3 1 area 0.0.0.0

    Note

     

    You can use the loopback for BGP peering or a dedicated loopback. This loopback must be different than the loopback for peer keep alive.

Step 6

Configure the underlay interfaces for IPv4 and IPv6.

Both L3 physical and L3 port channels are supported. SVI and sub-interfaces are not supported.

  1. For IPv4

    Example:

    router ospf 1
interface Ethernet1/16
port-type fabric 
ip address 192.0.2.2/24
ip router ospf 1 area 0.0.0.0
no shutdown
interface Ethernet1/17
port-type fabric 
ip address 192.0.2.3/24
ip router ospf 1 area 0.0.0.0
no shutdown
interface Ethernet1/40
port-type fabric 
ip address 192.0.2.4/24
ip router ospf 1 area 0.0.0.0
no shutdown
interface Ethernet1/41
port-type fabric 
ip address 192.0.2.5/24
ip router ospf 1 area 0.0.0.0
no shutdown

  2. For IPv6

    Example:

    router ospfv3 1
interface Ethernet1/16
port-type fabric
ipv6 address 192:0:2::2/24
ipv6 router ospfv3 1 area 0.0.0.0
no shutdown
interface Ethernet1/17
port-type fabric 
ipv6 address 192:0:2::3/24
ipv6 router ospfv3 1 area 0.0.0.0
no shutdown
interface Ethernet1/40
port-type fabric 
ipv6 address 192:0:2::4/24
ipv6 router ospfv3 1 area 0.0.0.0
no shutdown
interface Ethernet1/41
port-type fabric 
ipv6 address 192:0:2::5/24
ipv6 router ospfv3 1 area 0.0.0.0
no shutdown

    Note

     

    All ports connected to spines must be port-type fabric.

Step 7

Configure NVE and VXLAN settings.

Note

 

Configuring advertise virtual-rmac (NVE) and advertise-pip (BGP) are required steps. For more information, see the Configure vPC Multi-Homing chapter.

Step 8

Configure VLANs and SVI.

Example:

vlan 10
vn-segment 10010
vlan 101
vn-segment 10101

interface Vlan101
no shutdown
mtu 9216
vrf member vxlan-10101
no ip redirects
ip forward
ipv6 address use-link-local-only
no ipv6 redirects
interface vlan10
no shutdown
mtu 9216
vrf member vxlan-10101
no ip redirects
ip address 192.0.2.102/24
ipv6 address 2001:DB8:0:1::1/64
no ipv6 redirects
fabric forwarding mode anycast-gateway

Step 9

Configure virtual port channel member interfaces and port-channels.

Example:

interface Ethernet1/3
switchport
switchport mode trunk
channel-group 100
no shutdown
exit
interface Ethernet1/39
switchport
switchport mode trunk
channel-group 101
no shutdown
interface Ethernet1/46
switchport
switchport mode trunk
channel-group 102
no shutdown
interface port-channel100 
vpc 100
interface port-channel101
vpc 101 
interface port-channel102
vpc 102
exit

vPC fabric peering is successfully configured with all necessary underlay and overlay features, proper TCAM allocation, and functional BGP/OSPF routing. VLANs are correctly mapped to VXLAN segments, with all interfaces operational for vPC.

Migrate from vPC to vPC fabric peering

Use this procedure when you need to move from regular vPC to vPC fabric peering. This migration is disruptive and should be performed during a planned maintenance window.

Follow these steps to migrate from vPC to vPC fabric peering:

Before you begin

  • Shut all physical Layer 2 links between the vPC peers.

  • Map VLANs with vn-segment, either before or after the migration.

  • Ensure any direct Layer 3 link between vPC peers is used only for peer-keepalive, not for vPC fabric peering loopback advertisements.

Procedure

Step 1

Enter global configuration mode.

Example:

switch# configure terminal

Step 2

Verify existing vPC and port-channel members.

  1. Determine the number of members in the port channel.

    Example:

    switch(config)# show vpc

  2. Determine the number of members.

    Example:

    switch(config)# show port-channel summary

Step 3

For every interface in the vPC peer link port-channel remove the channel group.

  1. Enter interface configuration mode.

    Example:

    switch(config)# interface ethernet 1/4

    Specifies the interface you are configuring.

    Note

     

    This is the peer link port channel.

  2. Remove the interface from the channel group

    Example:

    switch(config-if)# no channel-group

    Remove vPC peer-link port-channel members.

    Note

     

    Disruption occurs following this step.

Step 4

Verify the vPC domain configuration mode.

Example:

switch(config-if)# show running-config vpc

Step 5

Enter vPC domain configuration mode.

Example:

switch(config-if)# vpc domain 100

Step 6

Specify the destination and source IP addresses for vPC fabric peering.

Example:

Example:

For IPv4
switch(config-vpc-domain)# virtual peer-link destination 192.0.2.1 source 192.0.2.100

OR

switch(config-vpc-domain)# virtual peer-link destination 192.0.2.1 source 192.0.2.100 dhcp 56

Example:

For IPv6
switch(config-vpc-domain)# virtual peer-link destination 6001:aaa::11 source 6001:aaa::22

OR

switch(config-vpc-domain)# virtual peer-link destination 6001:aaa::11 source 6001:aaa::22 dhcp 56

If using DHCP, append dhcp <value>. Use IPv4/IPv6 addresses as appropriate based on your underlay.

Step 7

Configure Layer 3 underlay interfaces as fabric ports.

Example:

switch(config-if)# interface Ethernet1/17
switch(config-if)# port-type fabric

Configures port-type fabric for underlay interface.

Note

 

All ports connected to spines must be port-type fabric.

  1. Verify the fabric ports connected to spine.

    Example:

    switch# show vpc fabric-ports

Step 8

Reserve TCAM resources for flow redirection.

Example:

switch(config-vpc-domain)# hardware access-list tcam region ing-flow-redirect 512

The minimum size for Ingress-Flow-redirect TCAM region size is 512. Also ensure it is configured in multiples of 512.

Step 9

Save configuration and reload the switch.

Example:

switch(config-vpc-domain)# copy running-config startup-config
switch(config-vpc-domain)# reload
The switches operate using vPC fabric peering. All relevant interfaces and TCAM resources are configured for the new topology.

Verifying vPC fabric peering configuration

This reference provides commands to verify the vPC fabric peering configuration and their output examples.

Table 1. vPC fabric peering verification Commands

Command

Purpose

show vpc fabric-ports

Displays the fabric ports state.

show vpc

Displays information about vPC fabric peering mode.

show vpc virtual-peerlink vlan consistency

Displays the VLANs which are not associated with vn-segment.

Example of the show vpc fabric-ports command

switch# show vpc fabric-ports 
Number of Fabric port : 9
Number of Fabric port active : 9

Fabric Ports State
-------------------------------------
Ethernet1/9 UP 
Ethernet1/19/1 ( port-channel151 ) UP 
Ethernet1/19/2 ( port-channel151 ) UP 
Ethernet1/19/3 UP 
Ethernet1/19/4 UP 
Ethernet1/20/1 UP 
Ethernet1/20/2 ( port-channel152 ) UP 
Ethernet1/20/3 ( port-channel152 ) UP 
Ethernet1/20/4 ( port-channel152 ) UP

Example of the show vpc command

switch# show vpc
Legend:
                 (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 3
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : primary
Number of vPCs configured         : 1
Peer Gateway                      : Enabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Enabled, timer is off.(timeout = 240s)
Delay-restore status              : Timer is off.(timeout = 30s)
Delay-restore SVI status          : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router    : Disabled
Virtual-peerlink mode             : Enabled

vPC Peer-link status
---------------------------------------------------------------------
id    Port   Status Active vlans
--    ----   ------ -------------------------------------------------
1     Po100  up     1,56,98-600,1001-3401,3500-3525

vPC status
----------------------------------------------------------------------------
Id    Port          Status Consistency Reason                Active vlans
--    ------------  ------ ----------- ------                ---------------
101   Po101         up     success     success               98-99,1001-280
                                                             0

Please check "show vpc consistency-parameters vpc <vpc-num>" for the
consistency reason of down vpc and for type-2 consistency reasons for
any vpc.

ToR_B1#

Example of the show vpc virtual-peerlink vlan consistency command

switch# show vpc virtual-peerlink vlan consistency
Following vlans are inconsistent
23
switch#