Understand Source MAC Address Field in Spanning-Tree PDUs on Nexus Series Switches

Introduction

This document explains how Source MAC address field in Spanning Tree Protocol (STP) control packets is populated on Nexus Series Switches.

Contributed by Nikolay Kartashev, Jun Wang, Cisco TAC Engineers.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Virtual port channels (vPC) on Nexus Series Switches
• STP

Components Used

The information in this document is based on the Nexus 7000 Series Switch platform.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Problem Description

vPC allows links that are physically connected to two different Cisco Nexus 7000 Series devices to appear as a single port channel by a third device. The third device can be a switch, server, or any other network device that supports link aggregation technology.

Similar to Cisco Catalyst Series Switches, Cisco Nexus Series Switches use STP to build a logical loop-free topology for Ethernet networks. 

Since vPC belongs to Multichassis EtherChannel (MCEC) family of technology, Source MAC address field of STP control packets, also known as Bridge Protocold Data Units (BPDU) requires special guide to properly represent vPC domain as a single switch.

Here is a reminder of typical BPDU stucture, where Source Address field is the focus of this document's discussion as shown in the image

Cisco Nexus Series Switches use virtual MAC address in Source MAC address field of BPDUs sent out virtual port channel interfaces. This MAC address is the same for both vPC peers. This ensures consistent and seamless behaviour in vPC failover scenarios.

When you troubleshoot STP in vPC network environment there is often a confusion caused by the fact that Nexus Series Switches might use other vendors MAC addresses in the Source MAC address field of some originated BPDUs. These sections explain the reason behind this, and compare this behaviour among different Nexus Series Platforms.

Network Diagram

Consider an example where a pair of Nexus 7000 Series Switches form vPC domain and have connections to a couple of access switches. One access switch is connected to vPC domain via vPC orphan port and another access switch is connected via virtual port channel interface. Both vPC orphan port and virtual port channel are configured as layer 2 trunk interfaces as shown in the image

In this example, while vPC interface carries vPC-enabled vlans only, vPC orphan port trunks both vPC-enabled and non-vPC-enabled vlans.

Configurations

Here is the configuration of vPC interface on first Nexus 7000 Series Switch. Second Nexus 7000 Series Switch has identical configuration.

Nexus7000-1# show running-config interface port-channel 60

!Command: show running-config interface port-channel60
!Time: Fri Jul 14 02:56:21 2017

version 7.2(2)D1(2)

interface port-channel60
  switchport
  switchport trunk allowed vlan 1-199
  switchport mode trunk
  vpc 60

Nexus7000-1#

vPC orphan port configuration on the first Nexus 7000 Series Switch is as follows:

Nexus7000-1# show running-config interface ethernet 3/13

!Command: show running-config interface Ethernet3/13
!Time: Sun Jul 16 04:49:43 2017

version 7.2(2)D1(2)

interface Ethernet3/13
  switchport
  switchport mode trunk
  no shutdown

Nexus7000-1#

STP on vPC Orphan Ports

Packet capture on vPC orphan port off of first Nexus 7000 Series Switch shows Source MAC address of outgoing BPDUs is based on port MAC address, both for vPC and non-vPC vlans.

#Nexus7000-1# show interface ethernet 3/13
Ethernet3/13 is up
admin state is up, Dedicated Interface
 Hardware: 10000 Ethernet, address: 503d.e5b8.7298 (bia 503d.e5b8.7298)
...

Nexus7000-1# ethanalyzer local interface inband-out display-filter stp limit-captured-frames 1000 | include b8:72:98
Capturing on inband
2017-07-16 04:47:17.383777 Cisco_b8:72:98 -> Spanning-tree-(for-bridges)_00 STP 60 RST. Root = 0/1/a4:4c:11:6a:24:41  Cost = 2  Port = 0x818d
2017-07-16 04:47:17.383876 Cisco_b8:72:98 -> PVST+        STP 64 RST. Root = 0/1/a4:4c:11:6a:24:41  Cost = 2  Port = 0x818d
2017-07-16 04:47:17.384182 Cisco_b8:72:98 -> PVST+        STP 64 RST. Root = 4096/2/00:23:04:ee:be:01  Cost = 0  Port = 0x818d
2017-07-16 04:47:17.384483 Cisco_b8:72:98 -> PVST+        STP 64 RST. Root = 4096/3/00:23:04:ee:be:01  Cost = 0  Port = 0x818d
2017-07-16 04:47:17.384876 Cisco_b8:72:98 -> PVST+        STP 64 RST. Root = 4096/4/00:23:04:ee:be:01  Cost = 0  Port = 0x818d
2017-07-16 04:47:17.385189 Cisco_b8:72:98 -> PVST+        STP 64 RST. Root = 4096/5/00:23:04:ee:be:01  Cost = 0  Port = 0x818d
2017-07-16 04:47:17.385504 Cisco_b8:72:98 -> PVST+        STP 64 RST. Root = 4096/6/00:23:04:ee:be:01  Cost = 0  Port = 0x818d
...
2017-07-16 04:47:17.399802 Cisco_b8:72:98 -> PVST+        STP 64 RST. Root = 4096/c8/00:23:04:ee:be:01  Cost = 0  Port = 0x818d

Note: Use of port MAC address as the Source MAC address field in outgoing BPDUs is the default behaviour on Cisco Nexus Series Switch and Cisco Catalyst Series Switch platforms.

STP on vPC

Source MAC address of BPDUs sent out virtual port channels by Nexus Series Switches is constructed in this way:

vPC BPDU Source MAC address = 0026.fxxx.0000

where xxx is vPC port channel number.

For example, this packet capture shows value 0x03c in vPC port channel number position, which translates to decimal value of 60. This is the number of virtual port channel configured on Nexus 7000 Series Switches.

2017-07-13 02:54:12.710581 00:26:f0:3c:00:00 -> 01:00:0c:cc:cc:cd STP 100 RST. Root = 4096/43/00:23:04:ee:be:01 Cost = 0 Port = 0x903b
2017-07-13 02:54:12.710599 00:26:f0:3c:00:00 -> 01:00:0c:cc:cc:cd STP 100 RST. Root = 4096/44/00:23:04:ee:be:01 Cost = 0 Port = 0x903b
2017-07-13 02:54:12.710601 00:26:f0:3c:00:00 -> 01:00:0c:cc:cc:cd STP 100 RST. Root = 4096/45/00:23:04:ee:be:01 Cost = 0 Port = 0x903b
2017-07-13 02:54:12.710603 00:26:f0:3c:00:00 -> 01:00:0c:cc:cc:cd STP 100 RST. Root = 4096/46/00:23:04:ee:be:01 Cost = 0 Port = 0x903b

However, checks for Organizationally Unique Identifier (OUI) of Source MAC address 00:26:f0:3c:00:00 show that this MAC address is part of the range allocated to cTrixs International GmbH organization.

Note: In order to find the allocation of MAC address blocks, you can use Wireshark OUI Lookup available at this link https://www.wireshark.org/tools/oui-lookup.html, or any similar tool.

Same output with MAC address resolution in place is shown in the capture.  As a network operator, this is what you can see when you troubleshoot STP in vPC network environments.

Nexus7000-1# ethanalyzer local interface inband-out display-filter stp limit-captured-frames 1000 | include CtrixIn
Capturing on inband
2017-07-17 04:34:32.324661 CtrixsIn_3c:00:00 -> PVST+        STP 64 RST. Root = 4096/5/00:23:04:ee:be:01  Cost = 0  Port = 0x903b
2017-07-17 04:34:32.324864 CtrixsIn_3c:00:00 -> PVST+        STP 64 RST. Root = 4096/6/00:23:04:ee:be:01  Cost = 0  Port = 0x903b
2017-07-17 04:34:32.325075 CtrixsIn_3c:00:00 -> PVST+        STP 64 RST. Root = 4096/7/00:23:04:ee:be:01  Cost = 0  Port = 0x903b
2017-07-17 04:34:32.325265 CtrixsIn_3c:00:00 -> PVST+        STP 64 RST. Root = 4096/8/00:23:04:ee:be:01  Cost = 0  Port = 0x903b
2017-07-17 04:34:32.325466 CtrixsIn_3c:00:00 -> PVST+        STP 64 RST. Root = 4096/9/00:23:04:ee:be:01  Cost = 0  Port = 0x903b

Note: Non-vPC Port Channel interfaces get MAC address from the first operational interface. Source MAC address field of outgoing BPDUs uses Port Channel interface MAC address.

Change of Behaviour

Starting from 5.2(1)N1(9), 7.1(4)N1(1) for Nexus 5000 Series Switches, Cisco allocates a range of MAC addresses from 0026.0bf1.f000 to 0026.0bf2.2ffff to be used by NX-OS for Source MAC address in BPDUs sent on Virtual Port-Channel interfaces.

vPC BPDU Source MAC address = 0026.0bf1.fxxx 

where xxx is vPC port channel number.

With introduced changes, Source MAC address of originated BPDUs sent out virtual port channel 60 on Nexus 5000 Series Switches would be 00:26:0b:f1:f0:3c, which has OUI of Cisco Systems, Inc.

 14 2017-07-13 04:38:16.781559 00:26:0b:f1:f0:3c -> 01:00:0c:cc:cc:cd STP 100 RST. Root = 4096/18/00:23:04:ee:be:01  Cost = 0  Port = 0x903b
 15 2017-07-13 04:38:16.781561 00:26:0b:f1:f0:3c -> 01:00:0c:cc:cc:cd STP 100 RST. Root = 4096/19/00:23:04:ee:be:01  Cost = 0  Port = 0x903b
 16 2017-07-13 04:38:16.782222 00:26:0b:f1:f0:3c -> 01:00:0c:cc:cc:cd STP 100 RST. Root = 4096/20/00:23:04:ee:be:01  Cost = 0  Port = 0x903b
 17 2017-07-13 04:38:16.782229 00:26:0b:f1:f0:3c -> 01:00:0c:cc:cc:cd STP 100 RST. Root = 4096/21/00:23:04:ee:be:01  Cost = 0  Port = 0x903b

Default behaviour is not changed for Nexus 7000 and Nexus 9000 Series Switches. However, starting from 6.1(3) for Nexus 7000 and 7.0(3)I6(2), 7.0(3)I7(2) for Nexus 9000, you can use this command in vPC domain configuration mode to perform this change.

Nexus7000-1(config-vpc-domain)# mac-address bpdu source version 2

 This warning message is displayed to inform you of impact this configuration command has.

Warning: This command will trigger STP to use new Cisco MAC address (00:26:0b:xx:xx:xx) as the sources address of BPDU generated on vPC ports. It is important both vPC peer devices have identical configuration of this parameter. You may also disable Ether channel guard on the edge devices prior to issuing this comm- and to minimize traffic disruption due STP inconsistencies. It is recommended to re-enable the Ether channel guard after updating the related configuration on both peers.
Continue? (yes/no) [no]

Verify

There is currently no verification procedure available for this configuration.

Troubleshoot

There is currently no specific troubleshooting information available for this configuration.

Summary

STP PDUs are not used by MAC address learning mechanism on Cisco switches, hence use of non-Cisco Source MAC address does not have a negative impact in day to day Layer 2 network operations. However, to comply with standards, self-originated BPDUs should have Source MAC address field populated from allocated range of MAC addresses. Cisco Nexus Series Switches provide such compliance in Cisco NX-OS Software with the change of default setting for Nexus 5000 Series Switches and Nexus 9000 Series Switches, and with the provision of the command line configuration option on Nexus 7000 Series Switches.  

Related Information

• Technical Support & Documentation - Cisco Systems