This document explains Virtual PortChannel (vPC) Role election process on Nexus Series Switches.
Contributed by Nikolay Kartashev, Jun Wang, Ken Zheng, Cisco TAC Engineers.
Cisco recommends that you have knowledge of these topics:
vPC on Nexus Series Switches
Spanning Tree Protocol (STP)
The information in this document is based on the Nexus 9000 Series Switch platform.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Virtual PortChannel Technology
Virtual PortChannels (vPCs) allow links that are physically connected to two different Ciscoswitches to appear as a single PortChannel to a third device. The third device can be a switch, a server, or any other networking device that supports IEEE 802.3ad PortChannels. vPC also allows the creation of Layer 2 PortChannels that span two switches. At this time being, vPC is implemented on Cisco Nexus 9000, 7000, 5000 and 3000 Series platforms (with or without Cisco Nexus 2000 Series Fabric Extenders).
Cisco NX-OS Software vPCs and Cisco CatalystVirtual Switching Systems (VSS) are of similar technologies. For Cisco EtherChannel technology, the term “Multi-chassis EtherChannel (MCEC)" interchangeably refers to either technology.
Although both vPC switches appear as a single switch to a downstream device, among themselves two vPC switches have clearly defined vPC roles: vPC Primary and vPC Secondary.
vPC roles are non-preemptive, which means a device can be configured as vPC primary, but operate as vPC secondary peer device. This can happen in the following scenario:
When the original primary device fails, the secondary vPC device becomes the new primary device
When the system recovers itself, the previously primary device is now the secondary device and vice versa.
vPC role defines which of the two vPC peer devices processes Bridge Protocol Data Units (BPDUs) and responds to Address Resolution Protocol (ARP) requests. vPC role also defines a set of actions to be taken by vPC primary and vPC secondary in response to vPC peer-link down situation.
vPC Role Priority
You can also use "role priority" in vPC domain mode command to influence vPC Election process. The range of values is from 1 to 65636, and the default value is 32667. A lower value means that this switch has a better chance of being the primary vPC.
Changing the priority of the vPC peer devices can cause the interfaces in your network to go up and down. If you want to configure the role priority again to make one vPC device the primary device, configure the role priority on both the primary vPC device with a lower priority value and the secondary vPC device with the higher value. Then, shut down the vPC peer link on both devices by entering the shutdown command, and finally re-enable the port channel on both devices by entering the no shutdown command.
Hitless vPC Role Change
The vPC hitless role change feature provides a framework to switch vPC roles between vPC peers without impacting traffic flows. The vPC role swapping is done based on the role priority value of the device under the vPC domain. A vPC peer device with lower role priority is selected as the primary vPC device when the"vpc role preempt"command is executed.
vPC Systems Behavior When a vPC Peer-Link Goes Down
When vPC peer-link fails down and vPC peer-keepalive link is still up, the vPC secondary peer device performs the following operations:
Suspends its vPC member ports
Shuts down the SVI associated to the vPC VLAN
This protective behaviour from vPC redirects all south-to-north traffic to the vPC primary device.
Please note that when vPC peer-link is down, both vPC peer devices cannot synchronize each other anymore so designed protection mechanism leads to isolate one of the peer device (in occurrence the secondary peer device) from the data path.
vPC Master Sticky Bit
vPC Master Sticky bit is a Programmed Protection Mechanism introduced to avoid unnecessary role change (which would potentially be causing disruption on the network) when the Primary Switch gets reloaded unexpectedly. vPC Master Sticky Bit allows the alive switch “sticks” to its PRIMARY role when a dead switch is coming back alive or when an isolated switch is being integrated back into the VPC domain.
Toggling vPC Master Sticky Bit:
vPC Master Sticky Bit value will be set to True in the following scenarios:
vPC Master Sticky Bit value will be set to FALSE when a vPC-enabled switch is rebooted (Note: Sticky Bit is set to FALSE by default)
When current vPC Primary reboots and vPC switch switches its role from vPC Secondary to vPC Primary.
When "delay-restore" feature is configured.
vPC Sticky master bit is reported under vPC Manager software component structure and can be checked with this NX-OS exec mode command
Campus_N7K2-VPC# show system internal vpcm info global | include ignore-case sticky
Sticky Master: TRUE
vPC Delay Restore
After a vPC peer device reloads and comes back up, the routing protocol needs time to reconverge. The recovering vPCs leg may black-hole routed traffic from access to aggregation/core until uplink Layer 3 connectivity is reestablished.
vPC Delay Restore feature delays vPCs leg bring-up on the recovering vPC peer device. vPC Delay Restore allows for Layer 3 routing protocols to converge before allowing any traffic on vPC leg. This results in a more graceful restoration and zero packet loss during the recovery phase (traffic still get diverted on the alive vPC peer device). This feature is enabled by default with a vPC restoration default timer of 30 seconds. The timer can be tuned according to a specific Layer 3 convergence baseline from 1 to 3600 seconds.
vPC Delay Restore Interface Vlan
To delay the VLAN interfaces on the restored vPC peer device from coming up, use the "interfaces-vlan" option of the "delay restore" command. This feature is enabled by default with a vPC restoration default timer of 10 seconds.
vPC Election Process
In a vPC system, one vPC peer device is defined as vPC primary and one is defined as vPC secondary, based on these parameters and in this order
vPC master sticky-bit set to 0 or 1
User-defined vPC role priority (Cisco NX-OS software uses the lowest numeric value to elect the primary device)
System mac address value (Cisco NX-OS software uses the lowest MAC address to elect the primary device)
This flowchart (Figure 1) summarizes steps that both vPC peer devices go through during vPC primary switch election process.
The first checked parameter between two devices during vPC primary election process is vPC Master Sticky Bit. If vPC peer device wins this comparison, it will become vPC primary regardless of the configured vPC role priority value or system MAC addresses both peers have.
If both vPC peer switches have the same Sticky Bit value, the election process will proceed to the next step to compare the user-defined vPC role priority.
If both vPC roles are configured to the same value, the election process will proceed to compare the system MAC addresses.
As shown in Table 1, when vPC switch has vPC master Sticky Bit set to 1 (TRUE condition) and its peer with the Sticky Bit set to 0 (FALSE condition), the TRUE side will win the election and assume the role of vPC Primary.
vPC Peer 1 Sticky Bit set to 1
vPC Peer 2 Sticky Bit set to 1
vPC Peer 1
vPC Peer 2
vPC Recovery Scenario
Importance of understanding vPC Election process cannot be underestimated, especially in vPC recovery scenarios.
Figure 2 shows a typical VPC set up, Nexus-01 is the VPC Primary and Nexus-02 is the VPC Secondary. Both of them has their Sticky Bits re-set to FALSE by default.
As shown in Figure 3, Nexus-01 now has a power outage and has been isolated from the network. Nexus-02 promoted itself to vPC Primary and set vPC Sticky Bit to TRUE.
And Nexus-02 now becomes Operational Primary, and the sticky bit is now set to TRUE.
As shown in Figure 4, when Nexus-01 comes back online after the power outage has been restored, Nexus-02 will retain the Operational PRIMARY role regardless of its role priority (because it has a TRUE sticky bit) and Nexus-02 will take the SECONDARY role when it comes online. Only Nexus-01 will begin the VPC initialization process whereas N7K-02 will remain as Primary and be forwarding traffic as usual. Therefore,no network outage will be seen.
There are two timers associated with the vPC initialization process on Nexus-01, which is now the vPC operational Secondary device:
delay restore SVI (10 seconds by default)
delay restore (30 seconds by default)
As a result, you can expect a 40-second recovery time on Nexus-01 after Nexus-01 is re-introduced back into the network as a vPC Secondary device. However, since Nexus-02 takes the Primary role, all traffic now is passing through Nexus-01 as mentioned above, no network outage will be seen.
An Example of Network Outage Relevant to Wrongly Set Sticky Bit
Network outage is caused by an INCORRECTLY set sticky bit when an isolated switch (Nexus-02) is being introduced back to the VPC domain
As shown in Figure 5, Nexus-01 is configured with a higher VPC role priority than Nexus-02, and Nexus-02has its Sticky Bit set to TRUE. Link E1/1 and E1/2 of Nexus-01 are in forwarding state whereas E1/1 and E1/2 in shut down state.
When the PKA and Peer Link are restored, Nexus-02 will take the PRIMARY role regardless of its role priority (because it has a TRUE sticky bit) and force Nexus-01 to become SECONDARY and the VPC initialization process will begin on Nexus-01. Therefore, link E1/1 and E1/2 of Nexus-01 will be suspended by VPC and will come online after the relay restore timers (40 seconds by default) expire. In this case,a 40-second network outage will be seen after the PKA and Peer Link are restored, as shown in Figure 6.
When re-introducing a Nexus back to the vPC domain, we must ensure that there will be no vPC role change in the active vPC device.
To avoid a vPC role change when the sticky bits of both switches are set to the same value, the active vPC device has to have a higher role priority for it to retain its PRIMARY role. Please refer to Figure 1 in this article for more information on VPC role election process.