Troubleshoot polarization in port-channel load balancing

Introduction

This document explains the scenarios under which polarization in port-channel load balancing could occur and provides suggestion on how to prevent them.

Background

Polarization is an issue where the hash algorithm selects certain paths in the network and leaves redundant paths unused

Prerequisites

It is recommended to to have knowledge on following topics.

Link aggregation control protocol

Cisco Nexus Platforms

Topology

Configuration

N7K1 and N7K2 connected in VPC and Po100, Po200, Po300 and Po301 are in VPC port-channel.

N7K1 and N7K2 act as a pure L2 switch with no routing happening on these switches.

All switches are running same port-channel load-balancing algorithm

The polarisation issue is seen on traffic going out of N7K1 and N7K2 irrespective of whether the traffic from source to the destination was in the same vlan (no routing) or if they were in different vlan with the routing happening on N7K3 or N7k4.

Traffic flow

The source sends multiple streams to the destination (with multiple source and destination IP addresses, and the L4 port information also vary from packet to packet). A good mix of traffic is used to ensure that in an ideal situation, the traffic would be evenly distributed among the port-channel member interfaces.

The traffic from source land on N7k3/N7k4 and then go via N7K1/N7K2 to the destination.

One link among member links of Po100 and Po200 on each of N7K1 and N7K2 is sending out almost 99% of the traffic and the other link remains idle. ( ie, on each switch N7K1 and N7K2, one link among 4/2 and 4/3 carries 99% unicast traffic and the other link carries less than 1%, similarly one link among 9/2 and 9/3 carries 99% traffic and the other link carries less than 1%. output in the troubleshooting section shows traffic on po100 and po200 member interfaces on N7K1, similar output can be seen on N7K2).

Irrespective of the type of port-channel load-balancing algorithm used, the issue can be seen as long as same port-channel load-balancing algorithm is used on N7K1/N7K2 pair and N7K3/N7K4 pair. The command to check port-channel load-balancing algorithm is given below.

N7K1# show port-channel load-balance
Warning: Per Packet Load balance configuration has higher precedence
System config:
  Non-IP: src-dst mac
  IP: src-dst ip-l4port-vlan rotate 0 
Port Channel Load-Balancing Configuration for all modules:
Module 1:
  Non-IP: src-dst mac
  IP: src-dst ip rotate 0 
Module 2:
  Non-IP: src-dst mac
  IP: src-dst ip rotate 0 
Module 3:
  Non-IP: src-dst mac
  IP: src-dst ip rotate 0 
Module 4:
  Non-IP: src-dst mac
  IP: src-dst ip-l4port-vlan rotate 0 
Module 7:
  Non-IP: src-dst mac
  IP: src-dst ip-l4port-vlan rotate 0 
Module 8:
  Non-IP: src-dst mac
  IP: src-dst ip-l4port-vlan rotate 0 
Module 9:
  Non-IP: src-dst mac
  IP: src-dst ip-l4port-vlan rotate 0

Troubleshooting

If uneven load balancing is seen on a port-channel, it can be because of polarization.

When traffic reaches N7K3 and N7K4 switches, they are forwarded to the N7K1/N7K2 switches via Po301 of N7K4 and Po300 of N7K3. Here, load balancing algorithm kicks in and some flows are forwarded to N7K1 and other flows forwarded to N7K2.

Initially, all the traffic comes into switches N7K3/N7K4 on eth1/1 and based on src-dest ip and l4 port information, certain flows are hashed on the link going towards N7K1 and other flows hashed on the link going towards N7K2. The hashing is done based on the rbh value which is calculated by the switch. For simplicity, let us assume that based on load-balance algorithm used, the switch segregates the incoming traffic into two flows (flow X and flow Y). Flow X sent out of one port-channel member link and flow Y sent out of the other port-channel member link.

Now, when the traffic is landing on the N7K1/N7K2 pair, there can be two possibilities. (Considering X and Y to be interchangeable)

Case1:

N7K3 sent flow X to N7K1 and flow Y to N7K2

and

N7K4 sent flow Y to N7K1 and flow X to N7K2

Case2:

N7K3 sent flow X to N7K1 and flow Y to N7K2

and

N7K4 sent flow X to N7K1 and flow Y to N7K2

In Case 1, N7K1 and N7K2 receive both type of flows (flow X and flow Y) and even after using same port-channel load balancing algorithm as that used by N7K3/N7K4, no polarisation would be seen as the flows egress out of Po100 and Po200 on different links and hence, we see a better traffic distribution among port-channel member interfaces.

In Case 2, N7K1 receives only flow X and N7K2 receives only flow Y and this could create polarization if the port-channel load-balancing algorithm used on the these switches are same as the one used in N7K3/N7K4 pair. As N7K1 and N7K2 are using the same port-channel load balancing algorithm, N7K1 sends flow X on only one member link of Po100/Po200 and the other member link will not forward any traffic. Similarly, N7K2 sends flow Y on only one member link of Po100/Po200 and the other member link will not forward any traffic.

Since the traffic that switches N7K1 and N7K2 are receiving is already classified to begin with, only one port-channel member link will be used to send all incoming traffic out of switch N7K1/N7K2 and nothing would be sent out of the other member link. In case if incoming traffic rate exceeds the bandwidth of the single port-channel link, the additional traffic can be dropped as the other port-channel member link would not forward this traffic.

Similar issue can be seen when more than two links are used in the port-channel. For example, if four links are used in a port-channel, then depending on  the hashing happening, either no polarisation would happen or we will see partial polarisation where only two of the four port-channel member links would be be used to forward all the incoming traffic and the other two links would not forward anything

The polarisation is caused because of the design and hence it is important to analyse the design to make sure no polarisation happens. Output indicating polarization happening Po100 and Po200 on N7k1 is given below (similar output can be seen on N7K2 as well). 

N7K1# show port-channel summary | i 200
200   Po200(SU)   Eth      LACP      Eth9/2(P)    Eth9/3(P)

N7K1# show port-channel traffic interface port-channel 200
NOTE: Clear the port-channel member counters to get accurate statistics
ChanId      Port Rx-Ucst Tx-Ucst Rx-Mcst Tx-Mcst Rx-Bcst Tx-Bcst
------ --------- ------- ------- ------- ------- ------- -------
   200    Eth9/2    0.0%  99.99%  44.44%   4.00%    0.0% 100.00%
   200    Eth9/3    0.0%   0.00%  55.55%  96.00%    0.0%    0.0%

N7K1# show port-channel summary | i 100
100   Po100(SU)   Eth      LACP      Eth4/2(P)    Eth4/3(P)

N7K1# show port-channel traffic interface port-channel 100
NOTE: Clear the port-channel member counters to get accurate statistics
ChanId      Port Rx-Ucst Tx-Ucst Rx-Mcst Tx-Mcst Rx-Bcst Tx-Bcst
------ --------- ------- ------- ------- ------- ------- -------
   100    Eth4/2    0.0%  99.99%  40.55%   7.00%    0.0% 100.00%
   100    Eth4/3    0.0%   0.00%  54.44%  93.00%    0.0%    0.0%

CSCvq26885 was filed for external documentation.

Workaround

Following some of the workarounds that can be used to make sure that polarization does not happen.

1. Proper Design: Since the main cause for polarization is improper design, best to make sure that we change the network design to make sure that there is no room for polarization in the topology

If no changes to the design is possible, we can do the following.

2. Use different port-channel load balancing algorithms at each level of switches (one algorithm on N7K1/N7k2 pair and a different algorithm on N7K3//N7k4 pair). When the load-balancing algorithm is changed, the N7k1/N7k2 switches now hash the incoming traffic based on some other information than the ones used by N7k3/N7k4 switches and hence the outgoing traffic use all the port-channel member links. (The decision on what algorithm to choose depends on the type of traffic received by the switch)

3. If the customer wants to use same load-balancing algorithm, use different rotate values at each level of switches. Rotate command introduces randomness in hashing algorithm by offsetting the hash-input by user configured bytes and helps to avoid polarization. (Use one rotate value for N7k1/N7k2 pair and a different rotate value for N7k3/N7k4 pair)