Cisco Nexus 5000 Series NX-OS Interfaces Operations Guide, Release 5.0(3)N2(1)

Type 1 and Type 2 Consistency Check Parameters

Before a Cisco Nexus 5000 Series switch brings up a vPC, the two Cisco Nexus 5000 Series switches in the same vPC domain exchange configuration information to verify if both switches have compatible configurations for a vPC topology. Depending on the severity of the impact of possible mismatched configurations, some configuration parameters are considered as Type 1 consistency check parameters while others are considered as Type 2.

When a mismatch in Type 1 parameters occur, the following applies:

• If a graceful consistency check is enabled (default), the primary switch keeps the vPC up while the secondary switch brings it down
• If a graceful consistency check is disabled, both peer switches suspend VLANs on the vPC ports.

Note The graceful consistency check is a new feature introduced in Cisco NX-OS Release 5.0(2)N2(1) and is enabled by default. For more details, see the “Graceful Consistency Check” section.

When Type 2 parameters exist, a configuration mismatch generates a warning syslog message. The vPC on the Cisco Nexus 5000 Series switch remains up and running. The global configuration, such as Spanning Tree Protocol (STP), and the interface-level configurations are included in the consistency check.

The show vpc consistency-parameters global command lists all global consistency check parameters. Beginning with Cisco NX-OS Release 5.0(2)N1(1), QoS parameters have been downgraded from Type 1 to Type 2.

This example shows how to display all global consistency check parameters:

Use the show vpc consistency-parameters interface port-channel number command to display the interface-level consistency parameters.

This example shows how to display the interface-level consistency parameters:

The Cisco Nexus 5000 Series switch conducts vPC consistency checks when it attempts to bring up a vPC or when you make a configuration change.

In the interface consistency parameters shown in the above output, all configurations except the Allowed VLANs are considered as Type 1 consistency check parameters. The Allowed VLAN (under the trunk interface) is considered as a Type 2 consistency check parameter. If the Allowed VLAN ranges are different on both VLANs that means that only common VLANs are active and trunked for the vPC while the remaining VLANs are suspended for this port channel.

Graceful Consistency Check

Beginning with Cisco NX-OS Release 5.0(2)N2(1) and later releases, when a Type 1 mismatch occurs, by default, the primary vPC links are not suspended. Instead, the vPC remains up on the primary switch and the Cisco Nexus 5000 Series switch performs Type 1 configurations without completely disrupting the traffic flow. The secondary switch brings down its vPC until the inconsistency is cleared.

However, in Cisco NX-OS Release 5.0(2)N2(1) and earlier releases, this feature is not enabled for dual-homed FEX ports. When Type-1 mismatches occur in this topology, the VLANs are suspended on both switches. The traffic is disrupted on these ports for the duration of the inconsistency.

To minimize disruption, we recommend that you use the configuration synchronization feature for making configuration changes on these ports.

To enable a graceful consistency check, use the graceful consistency-check command. Use the no form of this command to disable the feature. The graceful consistency check feature is enabled by default.

This example shows how to enable a graceful consistency check:

This example shows that the vPC ports are down on a secondary switch when an STP mode mismatch occurs:

This example shows that the vPC ports and the VLANs remain up on the primary switch when an STP mode mismatch occurs:

This example shows that the vPC ports are down on a secondary switch when an interface-level Type 1 inconsistency occurs:

This example shows that the vPC ports and the VLANs remain up on the primary switch when an interface-level Type 1 inconsistency occurs:

Configuring Per-VLAN Consistency Checks

Beginning with Cisco NX-OS Release 5.0(2)N2(1), the Cisco Nexus 5000 Series switch performs Type-1 consistency checks on a per-VLAN basis when you enable or disable STP on a VLAN. VLANs that do not pass this consistency check are brought down on the primary and secondary switches while other VLANs are not affected.

When you enter the no spanning-tree vlan number command on one peer switch, only the specified VLAN is suspended on both peer switches; the other VLANs remain up.

Note Per-VLAN consistency checks are not dependent on whether graceful consistency checks are enabled.

This example shows the active VLANs before suspending a specified VLAN:

This example shows that VLAN 5 is suspended but the remaining VLANs are up:

Identifying Inconsistent vPC Configurations

The show vpc command displays the vPC status and the vPC consistency check result for the global consistency check and the interface-specific consistency check.

This example shows the global vPC consistency check failed because of the mismatched Network QoS configuration:

You can use the show vpc consistency-parameters global command to identify the configuration difference between two vPC peer switches.

This example shows the global consistency check failed because the STP mode was configured differently on the two vPC switches:

You can use the show vpc command also shows the vPC consistency check result for each vPC and the reason for the consistency check failure.

This example shows how to display the vPC consistency check status:

If the consistency check fails, the consistency check is not performed on vPC member ports that are down.

If the consistency check has succeeded and the port is brought down, the consistency check shows that it was successful.

You can use the show vpc consistency-parameters interface ethernet slot/port command to identify the configuration difference that leads to a consistency check failure for a specified interface or port channel.

This example shows how to display configuration differences that lead to consistency check failures.

Bypassing a vPC Consistency Check When a Peer Link is Lost

The vPC consistency check message is sent by the vPC peer link. The vPC consistency check cannot be performed when the peer link is lost. When the vPC peer link is lost, the operational secondary switch suspends all of its vPC member ports while the vPC member ports remain on the operational primary switch. If the vPC member ports on the primary switch flaps afterwards (for example, when the switch or server that connects to the vPC primary switch is reloaded), the ports remain down due to the vPC consistency check and you cannot add or bring up more vPCs.

Beginning with Cisco NX-OS Release 5.0(2)N2(1), the auto-recovery feature brings up the vPC links when one peer is down. This feature performs two operations:

• If both switches reload, and only one switch boots up, auto-recovery allows that switch to assume the role of the primary switch. The vPC links come up after a configurable period of time if the vPC peer-link and the peer-keepalive fail to become operational within that time. If the peer-link comes up but the peer-keepalive does not come up, both peer switches keep the vPC links down. This feature is similar to the reload restore feature in Cisco NX-OS Release 5.0(2)N1(1) and earlier releases. The reload delay period can range from 240 to 3600 seconds.
• When you disable vPCs on a secondary vPC switch because of a peer-link failure and then the primary vPC switch fails, the secondary switch reenables the vPCs. In this scenario, the vPC waits for three consecutive keepalive failures before recovering the vPC links.

Note The auto-recovery feature in Cisco NX-OS Release 5.0(2)N2(1) and later releases replaces the reload restore feature in Cisco NX-OS Release 5.0(2)N1(1) and earlier releases.

The auto-recovery feature is disabled by default. To enable auto-recovery, enter the auto-recovery command in the vPC domain mode.

This example shows how to enable the auto-recovery feature and to set the reload delay period:

This example shows how to display the status of the auto-recovery feature:

Replacing a Cisco Nexus 5000 Series Switch

When you replace a Cisco Nexus 5000 Series switch, you must perform the following procedure on the replacement switch to synchronize the configuration with the existing Cisco Nexus 5000 Series switch. The procedure can be done in a hybrid single/dual-homed Fabric Extender vPC topology.

Note Do not connect a peer-link, vPC, or single/dual homed Fabric Extender topology fabric port to the replacement switch.

Note For a vPC+ topology, ensure that you wait for twenty minutes before you replace a vPC+ switch. Otherwise, vPC legs in the primary switch will get suspended due to switch-id conflict.

Before You Begin

• Power up replacement switch with no cables other than mgmt0 and console cable connected to the switch.
• Copy the required Cisco NX-OS kickstart/system files into the switch bootflash.
• If you have a backup of the switch configuration, copy it to new switch bootflash.
• Enable the FEX pre-provisioning feature on the switch in the vPC topology.
• Enable the configuration synchronization feature on the switch and apply all the switch profile configurations except for the sync peer destination IP address.

To replace a Cisco Nexus 5000 Series switch in a vPC topology, follow these steps:

Step 1 Boot the replacement switch.

The new switch comes up without a configuration. Ensure the software version is upgraded to match the existing switch.

Step 2 Enable FEX pre-provisioning for all single or dual homed Fabric Extender modules on the replacement switch.

Note Ensure that you unconfigure the system default switchport shutdown command on the replacement switch. Otherwise, when Fabric Extender Modules are coming online on the replacement switch, dual-homed FEX ports on the primary switch will flap causing traffic disruption.

Step 3 Configure the replacement switch as follows:

Note Before you configure the replacement switch using any of the following method, disable the vPC auto-recovery feature on both the vPC peers using the no auto-recovery command under the vPC domain. This is to ensure that there is no vPC role change because of the sticky bit feature, when the replacement switch is brought up. vPC auto-recovery feature is enabled by default in Cisco NX-OS release 7.x and later.

• If the running configuration was saved offline, go to Step 4 to Step 10 to apply the configuration.
• If the running configuration was not saved offline, you can obtain it from the peer switch if the configuration synchronization feature is enabled. (Create a switch profile and then go to Step 11).
• If neither condition is met, manually add the configuration and then go to Step 11.

Step 4 Edit the configuration file to remove the sync-peer command if using the configuration synchronization feature.

Step 5 Configure the mgmt0 port IP address and download the configuration file.

Step 6 Copy the saved configuration file to the running configuration.

Step 7 Edit the saved configuration file and delete all commands between the configure sync command and the commit command, including these two commands.

Step 8 Copy the new, edited configuration file to the running configuration again.

Step 9 Verify that the configuration is correct by entering the show running-config command and the show provision failed-config slot command.

Step 10 If switch profile configuration changes were made on the peer switch while the replacement switch was out of service, apply those configurations in the switch profile and then enter the commit command.

Step 11 Shut down all single-homed Fabric Extender vPC host ports.

Step 12 Connect the single-homed Fabric Extender topology fabric ports.

Step 13 Wait for single-homed Fabric Extenders to come online.

Step 14 Ensure the vPC role priority of the existing switch is better than the replacement switch.

Step 15 Connect the vPC peer keepalive link to the peer switch. Ensure that vPC peer keepalive link is operational by entering the show vpc command.

Warning If the auto-recovery feature was not disabled in Step 3, ensure that either the vPC peer-keepalive or the vPC peer-link comes up before the auto-recovery timer expires (default 240 seconds). If this does not happen, the replacement switch will assume the vPC primary role (dual active). If the vPC peer-link is restored in this state, there will be a vPC role change causing vPCs on the peer switch to go down as the switch transitions to vPC secondary role. If required, with just peer-keepalive link operational, reload the replacement switch one more time with all the other interfaces still in the shutdown state.

Step 16 Ensure that the vPC role field is none established . Use the show vpc or show vpc role command to view the vPC role. If the vPC role field displays Primary, then do not proceed with the replacement procedure. Reload the switch to get the vPC role field to none established.

Step 17 Connect the vPC peer-link ports to the peer switch. Ensure that the vPC peer link is operational by entering the show vpc command.

Step 18 Connect the dual-homed Fabric Extender topology fabric ports.

Step 19 Connect the switch vPC ports.

Step 20 Enter the no shutdown command on all single-homed Fabric Extender vPC ports.

Step 21 Verify that the replaced vPC switch and the Fabric Extenders on the replacement switch are online and there is no traffic disruption.

Step 22 If you are using the configuration synchronization feature, add the sync-peer configuration to the switch profile if this wasn’t enabled in Step 3.

Step 23 If you are using the configuration synchronization feature, enter the show switch-profile name status command to ensure both switches are synchronized.

Step 24 If vPC auto recovery was disabled, enable auto recovery using the auto-recovery command under vPC domain on both switches.

Replacing a Cisco Nexus 2000 Series Fabric Extender

This section describes how to replace a Cisco Nexus 2000 Series Fabric Extender with minimal disruption. This section includes the following topics:

• Replacing a Fabric Extender in a Dual-Homed Fabric Extender vPC Topology
• Replacing a Fabric Extender in a Single-Homed Fabric Extender vPC Topology
• Installing a New Cisco Nexus 2000 Series Fabric Extender

Replacing a Fabric Extender in a Dual-Homed Fabric Extender vPC Topology

Because the hosts behind a Fabric Extender in a dual-homed Fabric Extender vPC topology are by definition singly-connected, traffic disruption will occur for those hosts.

If the replacement Fabric Extender is a different model, the Cisco Nexus 5000 Series switch does not allow you to pre-provision a new type until you disconnect the old Fabric Extender.

To retain the configuration on both Cisco Nexus 5000 Series peer switches in the vPC topology, follow these steps.

Step 1 Save the configuration for the Fabric Extender interfaces to a file.

Step 2 Disconnect the Fabric Extender fabric ports and wait until the Fabric Extender is offline.

Step 3 Pre-provision the slot with the new Fabric Extender model.

Step 4 Modify the configuration file if necessary for the new Fabric Extender if the configurations are incompatible.

Note For vPC ports, this step might affect consistency.

Step 5 Copy the file to the running configuration.

Step 6 Connect the Fabric Extender fabric and host ports and then wait for the Fabric Extender to come online.

Step 7 Verify that all ports are up with the correct configuration.

Replacing a Fabric Extender in a Single-Homed Fabric Extender vPC Topology

If the replacement Fabric Extender is the same model as the original Fabric Extender, then there is no disruption; the configuration on the Fabric Extender interfaces remain unchanged.

If the replacement Fabric Extender is a different model, the Cisco Nexus 5000 Series switch does not allow you to pre-provision a new type until you disconnect the old Fabric Extender.

To replace a Fabric Extender in a single homed Fabric Extender vPC topology, follow the procedure described in “Replacing a Fabric Extender in a Dual-Homed Fabric Extender vPC Topology” section.

Installing a New Cisco Nexus 2000 Series Fabric Extender

With pre-provisioning, you can fully configure the new Fabric Extender before the Fabric Extender is connected to a Cisco Nexus 5000 Series switch.

To install a new Cisco Nexus 2000 Series Fabric Extender, follow these steps:

Step 1 Pre-provision the slot with the Fabric Extender model.

Step 2 Configure the interfaces as though the Fabric Extender is connected.

Step 3 Connect the Fabric Extender and wait for it to come online.

Step 4 Verify that all configurations are applied correctly

Note The switch applies all configurations serially in a best-effort fashion when the Fabric Extender comes online.

vPC Member Port Failure

Figure 1-1 shows the traffic flow when one vPC member port fails. Once the host MAC_A detects a link failure on one of the port-channel members, it redistributes the affected flows to the remaining port channel members. The return flow from MAC_C to MAC_A could take the path of the left- or the right-side Cisco Nexus 5000 Series switch, depending on the port-channel hash algorithm of the top switch. For those flows that traverse the right-side Cisco Nexus 5000 Series switch (the red line), the Cisco Nexus 5000 Series switch passes the traffic to the left-side Cisco Nexus 5000 Series switch, because it no longer has the local connection to host MAC_A. This is one of the scenarios where a vPC peer link is used to carry data traffic.

We recommend that you provision enough bandwidth for peer links to accommodate the bandwidth needed for link failure scenarios.

Figure 1-1 vPC Response to a Member Port Failure

vPC Peer Link Failure

Figure 1-2 shows the vPC response to a peer link failure. In a vPC topology, one vPC peer switch is elected as the vPC primary switch and the other switch is elected as the vPC secondary switch, based on the configured role priority for the switch. In the unlikely scenario where the vPC peer link goes down, the vPC secondary switch shuts down all of its vPC member ports if it can still receive keepalive messages from the vPC primary switch (which indicates that the vPC primary switch is still alive). The vPC primary switch keeps all of its interfaces up. As a result, the hosts or switches that are connected to the Cisco Nexus 5000 Series switch or Cisco Nexus 2000 Series Fabric Extender vPC pair redistributes all the flows to the vPC member ports that are connected to the vPC primary switch.

As a best practice, we recommend that you configure a physical port channel that has at least two 10 Gigabit-Ethernet ports as the vPC peer link.

Figure 1-2 vPC Response to a Peer Link Failure

A vPC consistency check cannot be done when a vPC peer-link is down either due to a link failure or when the peer switch is completely down. In either case, any newly configured vPC does not come up because the vPC consistency check cannot proceed, or the existing vPC remains disabled after the link flaps.

Use the reload restore feature that was introduced in Cisco NX-OS Release 5.0(2)N1(1) to fix this problem. The reload restore feature allows a switch to bypass the vPC consistency check and bring up vPC ports when the peer-link or peer switch fails. The reload restore feature has been replaced with the auto-recovery feature in Cisco NX-OS Release 5.0(2)N2(1).

vPC Peer Keepalive Link Failure

The vPC keepalive link carries the heartbeat message between two vPC peer switches. The failure of the vPC keepalive link alone does not impact the vPC operation or data forwarding. Although it has no impact on data forwarding, we recommend that you fix the keepalive as soon as possible to avoid a double failure scenario that could impact the data traffic.

When both switches come up together (such as after power gets restored following a power outage) and only the mgmt/keepalive link fails, the peers are unreachable. However, all other links, including vPC peer links, are up. In this scenario, reaching the vpc-peers through keepalives are achieved through keepalive links while the primary and secondary role election is established through the vpc-peer link. You must establish the first keepalive for the role election to occur in the case when a switch comes up and the vPC-peer link is up.

When keepalives fail to reach the peer switches, role election does not proceed and the primary or secondary role is not established on either vPC peer switch and all vPC interfaces are kept down on both switches.

Note If this scenario occurs again or if the keepalive link goes down after vPC peers are established, the roles do not change and all vPCs remain up.

vPC Peer Switch Failure

When one peer switch fails, half of the network bandwidth is lost and the remaining vPC switch maintains the network connectivity. If the failure occurs on a primary switch, the secondary switch becomes the primary switch.

When one peer switch fails, the remaining peer switch maintains network connectivity for the vPC until it is reloaded. This situation could happen if both vPC peer switches are reloaded and only one switch comes up or both switches loose power and then the power is restored only on one switch. In either case, since the vPC primary election cannot proceed, the Cisco Nexus 5000 Series switch keeps the vPC ports in suspend mode.

To fix these problems, use the reload restore feature and the auto recovery feature as follows:

In NX-OS Release 5.0(2)N1(1), enter the reload restore command:

In NX-OS Release 5.0(2)N2(1), enter the auto-recovery reload-delay command:

These commands allow the vPC peer switch to bypass the vPC consistency check and bring up vPC ports after the delay timer expires.

vPC Peer Link Failure Followed by a Peer Keepalive Link Failure

If a peer link failure occurs, the vPC secondary switch checks if the primary switch is alive. The secondary switch suspends its vPC member ports after it confirms that the primary switch is up.

If the vPC primary switch goes down, the vPC secondary switch stops receiving Keepalive messages on the vPC Peer Keepalive link. After three consecutive Keepalive message timeouts, the vPC secondary switch changes its role to be the vPC primary switch and brings up its vPC member ports.

In Cisco NX-OS Release 5.0(2)N2(1), if you enable the auto-recovery feature and if the vPC primary switch goes down, the vPC secondary switch does not receive messages on the vPC peer keepalive link. Then, after three consecutive keepalive timeouts, the vPC secondary switch changes its role to primary and brings up the vPC member ports.

vPC Keepalive Link Failure Followed by a Peer Link Failure

If the vPC keepalive link fails first and then a peer link fails, the vPC secondary switch assumes the primary switch role and keeps its vPC member ports up.

If the peer link and keepalive link fails, there could be a chance that both vPC switches are healthy and the failure occurs because of a connectivity issue between the switches. In this situation, both vPC switches claim the primary switch role and keep the vPC member ports up. This situation is known as a split-brain scenario. Because the peer link is no longer available, the two vPC switches cannot synchronize the unicast MAC address and the IGMP group and therefore they cannot maintain the complete unicast and multicast forwarding table. This situation is rare.

We recommend that you have a well-planned network design that includes spreading peer links and keepalive links to multiple ASICs or multiple modules and different cabling routes for keepalive and peer links to avoid a double failure.