Change of EtherChannel Member Interface Causes Disruption

Introduction

This document describes the root cause analysis of downtime caused by any change in the EtherChannel member interface.

Prerequisites

EtherChannel is part of Layer 2 Loop/Star/Mesh/Redundant Topology running over Spanning Tree Protocol (STP).

Requirements

Cisco recommends that you have knowledge of these topics:

Basic understanding of

• Cisco Switching and Catalyst Operations
• STP
• EtherChannel Load Balancing

Components Used

The information in this document is based on these software and hardware exhibit-based versions but not limited to:

• Cisco C9000 switches
• Virtual Switch Link (VSL) Setup
• EtherChannel

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The root cause of changes like addition/deletion/removal of the link or movement of the member interface link from one port to another in the EtherChannel member Interface is included in this document. Additionally, it outlines a mitigation plan in order to avoid such unexpected downtime.

Problem Description

In general, the addition of a member interface link to an existing EtherChannel must not cause any downtime or disruption to the traffic forwarding. Similarly, after the removal of the member interface link, if the remaining bandwidth of an EtherChannel is greater than its overall utilization before the link removal, then, there must not be any traffic disruption. But the earlier mentioned scenarios cause downtime of a few seconds in peculiar Layer 2 loop topologies.

Topology

Explanation

The IP address of Laptop 1 is 10.10.10.50/24 and that of Laptop 2 is 10.10.10.51/24. Both are in the same VLAN.

9600 Core switch is connected to 9500 Dist 1 with po1 and 9500 Dist 2 with po2.

9300 Access Switch connected to 9500 Dist 1 through interface GI 1/0/14 and to 9500 Dist 2 through GI 1/0/46.

The holistic packet path between Laptop 1 and Laptop 2 is:

Laptop 1 > (GI 1/0/1) Cat 9300 (GI 1/0/46) > (Te 1/0/12) Cat 9500 Dist 2 (Port-channel 1) > (Port-channel 2) Cat 9600 (Port-channel 4) > (Port-channel 1) Cat 9500 Dist 4 (Te 1/0/1) > Laptop 2

However, there are two packet path options between Laptop 1 and Laptop 2, depending on the load-balancing algorithm.

Option 1: Laptop 1 > (GI 1/0/1) Cat 9300 (GI 1/0/46) > (Te 1/0/12) Cat 9500 Dist 2 (Te1/0/4) > (Twe 1/1/0/24) Cat 9600 - Active (Twe 1/1/0/8) > (Twe 1/0/5) Cat 9500 Dist 4 (Te 1/0/1) > Laptop 2

Option 2: Laptop 1 > (GI 1/0/1) Cat 9300 (GI 1/0/46) > (Te 1/0/12) Cat 9500 Dist 2 (Te1/0/5) > (Twe 2/1/0/10) Cat 9600 - Standby (Twe 2/1/0/34) > (Twe 1/0/8) Cat 9500 Dist 4 (Te 1/0/1) > Laptop 2

Initially, since two interfaces are connected to Cat 9300 in order to reach the root bridge 9600, one of the interfaces GI 1/0/14 is put in an alternate blocking state and GI 1/0/46 is in root forwarding state depending on the STP path selection criteria.

In order to reach the root from Cat 9300 either from interface GI 1/0/14 or from GI 1/0/46 is the same and equal to 30000.

Cat 9300#show spanning-tree

VLAN0001
Spanning tree enabled protocol rstp
Root ID    Priority    24577
Address    549f.c666.c580
Cost       30000
Port       46 (GigabitEthernet1/0/46)
Hello Time  2 sec  Max Age 20 sec  Forward Delay 15 sec

Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
Address     2416.9d7a.2480
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------
GI 1/0/14            Altn BLK 20000     128.14   P2p 
GI 1/0/46            Root FWD 20000     128.46   P2p 

Cat 9600#show spanning-tree 
VLAN0001
Spanning tree enabled protocol rstp
Root ID    Priority    24577
Address     549f.c666.c580
This bridge is the root
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Bridge ID  Priority    24577  (priority 24576 sys-id-ext 1)
Address     549f.c666.c580
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Po1                 Desg FWD 10000     128.3433 P2p 
Po2                 Desg FWD 10000     128.3434 P2p  
Po3                 Desg FWD 10000     128.3435 P2p 
Po4                 Desg FWD 10000     128.3436 P2p 

Current STP state:

Cat 9600#show spanning-tree  detail | include is exec|changes|from

VLAN0001 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 8 last change occurred 00:10:28 ago
from Port-channel1

The activity of shutting down one of the member interfaces (Twe 2/1/0/10) from Port-channel 2 of 9600 results in:

1. Increment of the STP path cost of the port channel between Cat 9600 and Cat 9500 Dist 2 from 10000 to 20000.

Cat 9600#show spanning-tree 
VLAN0001
Spanning tree enabled protocol rstp
Root ID    Priority    24577
Address     549f.c666.c580
This bridge is the root
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Bridge ID  Priority    24577  (priority 24576 sys-id-ext 1)
Address     549f.c666.c580
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
Aging Time  300 sec

Interface   Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------
Po1         Desg FWD 10000     128.3433 P2p 
Po2         Desg FWD 20000     128.343  P2p <<<pathcost updated in po2
Po3         Desg FWD 10000     128.3435 P2p 
Po4         Desg FWD 10000     128.3436 P2p 

2. Increment of the STP path cost towards the root on Cat 9300 from 30000 to 40000 of interface GI 1/0/46. Hence, 9300 chooses the interface GI 1/0/14 as the root port having less STP path cost.

Cat 9300#show spanning-tree 

VLAN0001
Spanning tree enabled protocol rstp
Root ID    Priority    24577
Address     549f.c666.c580
Cost        30000
Port        14 (GigabitEthernet1/0/14)
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
Address     2416.9d7a.2480

Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
GI 1/0/14            Root FWD 20000     128.14   P2p  <<<<< GI 1/0/14 is coming to forward state 
GI 1/0/46            Altn BLK 20000     128.46   P2p 

3. This change results in a Topology Change Notification (TCN) generation in the entire spanning tree domain.

Cat 9600#show spann det | inc is exec|changes|from

VLAN0001 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 9 last change occurred 00:03:21 ago
from Port-channel1   <<< TCN is received over port-channel 1

As a result of the Spanning Tree reconvergence, a disruption can be observed in the network.

The time interval of the disruption can vary; based on the other factors and protocols.

The real-time impact of the EtherChannel change in the network is demonstrated by initiating continuous ping from Laptop 1 to Laptop 2.

In the same way, the addition of a member interface on Port-channel 1 between Cat 9500 Dist 1 and Cat 9600 Core Switch has a decrement in STP cost and also has the impact of topology change.

Root Cause

The addition or removal of the member interface of EtherChannel changes its STP port cost. This can cause STP reconvergence and TCN generation.

Mitigation

Option 1: Hard code the STP cost of the EtherChannel during the change using these steps:

Step 1. Find out the existing STP cost of the EtherChannel where the change (addition or removal of the member interface) has been planned.

Cat 9600#show spanning-tree interface port-channel 1

Vlan                Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
VLAN0001            Desg FWD 10000     128.3433 P2p

In this case, the cost is 10000 for VLAN 1.

Step 2. Hard code the same STP Cost.

Switch#

Switch#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Switch(config)#interface port-channel 1
Switch(config-if)#spanning-tree cost 10000
Switch(config-if)#end

Switch#

In the same way, the cost change configuration must be done on the peer device-connected interface too.

Step 3. Perform the change.
This includes:

• Addition of member interface to existing EtherChannel

or

• Removal of member interface from existing EtherChannel

or

• Shutting/no shutting of any of the member interfaces

or

• Movement of the member interface from one port to another port

The addition or deletion of the member interface does not have any impact on STP topology as the STP cost of the link is hardcoded.

Note:

The mitigation plan has an impact on STP calculations and has its side effects. However, this is useful in any activity of EtherChannel where a temporary change in the number of member interfaces is required during the activity.

Consider a scenario where it is needed to shut down a member interface of an EtherChannel during a certain network activity and restore the member interface at the end. In this case, the STP cost of the EtherChannel can be hard-coded to its default value at the start of the activity.

Shut down the respective member interface, complete the activity, and restore the interface; later, default the STP cost again. The STP reconvergence can be avoided using this method.

It is suggested to restore the spanning tree cost to its default settings at the end of the respective change with caution.

Option 2: For a few designs there can be alternate ways to avoid disruption, like the Spanning Tree enhancement feature, for example, uplink fast, root guard, and so on.

Related Information

• https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4000/8-2glx/configuration/guide/spantree.html#wp1193602
• Cisco Technical Support & Downloads