Troubleshoot TX Pauses on Nexus 2232
Available Languages
Contents
Introduction
This document describes the information in order to help troubleshoot Transmit (TX) pauses on Nexus 2232 Host Interface (HIF) ports. It focusses on traffic in the Host to Network (H2N) direction (traffic that comes in from the servers towards network, south to north). It does not cover scenarios related to Network to Host (N2H) traffic flows.
This document is primarily authored for Nexus 2232 Fabric Extender (FEX) but the concept applies for B22 and 2248UPQ FEX.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics
- Cisco Nexus 2000 Series configuration
- Cisco Nexus 6000 Series configuration
Components Used
The information in this document is based on these software and hardware versions:
- Cisco Nexus N2K-C2232PP-10GE
- Cisco Nexus 6001
- 7.1(1)N1(1)
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Ingress Buffers
Nexus 2232 has 32 1/10G Host Facing (HIF) ports and 8 10G Network Facing (NIF) ports.
Before you deep dive into the TX pause issue, you need to understand the available buffers on the FEX interfaces. The buffers assigned to the interface/qos-group can be checked via this command on the parent switch:
esc-6001# show queuing interface ethernet 147/1/1
if_slot 79, ifidx 0x1f920000
Ethernet147/1/1 queuing information:
Input buffer allocation:
Qos-group: 0
frh: 8
drop-type: drop
cos: 0 1 2 3 4 5 6
xon xoff buffer-size
---------+---------+-----------
0 126720 151040
<snip>
As seen, with default Quality of Service (QoS), for drop class traffic (qos-group 0), the FEX HIF has 151040 bytes in order to buffer H2N traffic and XOFF threshold is 126720 bytes.
Flow-Control Configuration
Nexus 2232 is oversubscribed to 8:1. In order to avoid packet drops in H2N direction due to oversubscription and buffer overruns, Nexus 2232 has HIF flow-control send on by default:
esc-6001# show run int ethernet 147/1/1 all | inc flow
priority-flow-control mode auto
flowcontrol receive off
flowcontrol send on
esc-6001# show interface ethernet 147/1/1 flowcontrol
--------------------------------------------------------------------------------
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
--------------------------------------------------------------------------------
Eth147/1/1 on on off off 0 0
Causes for TX Pause on Nexus 2232
When the XOFF threshold of 126720 bytes is hit, the Nexus 2232 sends a TX Pause towards the host on the HIF. Common causes for this are:
1. H2N Traffic that comes into the FEX is very bursty, that causes the ingress buffers to be full fill up and hit the XOFF threshold.
2. Most FEX deployments use port-channels to aggregate multiple NIFs. TX Pause is also seen due to ingress buffers, which gets full due to etherchannel hash collision on FEX. This happens when multiple HIF ports try to egress out of single NIF due to etherchannel results.
Drops in H2N direction
In order to avoid packet drops, TX Pause is sent once the XOFF threshold is hit. However, H2N traffic drops can be seen if:
1. the servers do not honor pause or,
2. have a delay to honor the pause that causes the 151KB drop threshold to be hit.
Lab Test Cases
Network Diagram
For this test in the lab, there are four 10G spirent ports which acts as hosts, two are on the FEX and two are on the parent Nexus 6001. All ports are in VLAN 50. There are no other port active on the FEX or the parent:
esc-6001# show port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
147 Po147(SU) Eth NONE Eth1/41(P) Eth1/42(P) Eth1/43(P)
Eth1/44(P) Eth1/45(P) Eth1/46(P)
Eth1/47(P) Eth1/48(P)
esc-6001# show fex 147 detail | exc Down
FEX: 147 Description: FEX0147 state: Online
FEX version: 7.1(1)N1(1) [Switch version: 7.1(1)N1(1)]
FEX Interim version: 7.1(1)N1(1)
Switch Interim version: 7.1(1)N1(1)
Extender Serial: FOT1635R003
Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05
Card Id: 82, Mac Addr: 20:3a:07:34:5b:02, Num Macs: 64
Module Sw Gen: 21 [Switch Sw Gen: 21]
post level: complete
Pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/47
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Po147 - Interface Up. State: Active
Eth1/41 - Interface Up. State: Active
Eth1/42 - Interface Up. State: Active
Eth1/43 - Interface Up. State: Active
Eth1/44 - Interface Up. State: Active
Eth1/45 - Interface Up. State: Active
Eth1/46 - Interface Up. State: Active
Eth1/47 - Interface Up. State: Active
Eth1/48 - Interface Up. State: Active
Fex Port State Fabric Port
Eth147/1/1 Up Po147
Eth147/1/9 Up Po147
Logs:
04/21/2015 21:58:30.162193: Module register received
04/21/2015 21:58:30.164611: Registration response sent
04/21/2015 21:58:30.196708: create module inserted event.
04/21/2015 21:58:30.197425: Module Online Sequence
04/21/2015 21:58:35.051474: Module Online
Test 1. Bursty Traffic with Flow Control Not Enabled on Host
When you send 100K 1500 byte line rate unicast bursts from host on Eth147/1/1(to Eth1/16) and Eth147/1/9(do Eth1/32). Each stream is a single flow. Flow control is disabled on host (Spirent).
Results: Receiver ports reported about 563 packets dropped for each flow. Since Flow control is disabled on the host, you can see a lot more TX Pause and also high latency (about 100 micro seconds):
esc-6001# clear counters ; clear qos stat
esc-6001#
esc-6001# show interface ethernet 147/1/1, ethernet 147/1/9 | egrep Ethernet147|pause|unicast
Ethernet147/1/1 is up
100000 unicast packets 0 multicast packets 0 broadcast packets
0 Rx pause
0 unicast packets 578269 multicast packets 0 broadcast packets
578267 Tx pause
Ethernet147/1/9 is up
100000 unicast packets 0 multicast packets 0 broadcast packets
0 Rx pause
0 unicast packets 578275 multicast packets 0 broadcast packets
578273 Tx pause
esc-6001# show interface ethernet 147/1/1, eth147/1/9 flowcontrol
--------------------------------------------------------------------------------
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
--------------------------------------------------------------------------------
Eth147/1/1 on on off off 0 578267
Eth147/1/9 on on off off 0 578273
The drops reported by the receiver is actually dropped on the FEX itself. There are internal hardware commands that can show the drops, but it requires you to completely understand the FEX internal architecture which is outside the scope of this document. If you need to check these counters, engage TAC for this aspect of the troubleshooting.
Test 2. Bursty Traffic with Flow Control Enabled on Host
When you send single flow 100K 1500 byte line rate unicast bursts from host on Eth147/1/1(to Eth1/16) and Eth147/1/9(do Eth1/32). Each stream is a single flow. Flow control is enabled on host (Spirent).
Results:
Receiver ports report no loss. Minimal TX Pause and average latency is about 19 microseconds:
esc-6001# clear counters ; clear qos stat
esc-6001# show interface ethernet 147/1/1, ethernet 147/1/9 | egrep Ethernet147|pause|unicast
Ethernet147/1/1 is up
100000 unicast packets 0 multicast packets 0 broadcast packets
0 Rx pause
0 unicast packets 4743 multicast packets 0 broadcast packets
4739 Tx pause
Ethernet147/1/9 is up
100000 unicast packets 0 multicast packets 0 broadcast packets
0 Rx pause
0 unicast packets 4703 multicast packets 0 broadcast packets
4700 Tx pause
esc-6001# show interface ethernet 147/1/1, eth147/1/9 flowcontrol
--------------------------------------------------------------------------------
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
--------------------------------------------------------------------------------
Eth147/1/1 on on off off 0 4739
Eth147/1/9 on on off off 0 4700
Drops:
There are no drops since host honors the flow control sent from the FEX.
Test 3. Ether-Channel Hash Collision
The uplink between FEX and parent is a port-channel. While it depends on which member in the port-channel is picked and how busy it is, TX Pause can be seen on the FEX HIFs. In the lab, there are only two ports active on the FEX and all 8 uplinks used in the port-channel.
But for this test, with default hashing, traffic from host on Ethernet 147/1/1 and Ethernet 147/1/9 gets hashed to NIF0 which connects to Eth1/41 on the 6001. If you send 98% line rate traffic from the hosts, TX Pause is sent on both HIFs.
For this test, flow control is disabled on the hosts:
esc-6001# show interface ethernet 147/1/1, ethernet 147/1/9 | inc Ethernet14|rate|pause
Ethernet147/1/1 is up
30 seconds input rate 9836009128 bits/sec, 819667 packets/sec
30 seconds output rate 2516922296 bits/sec, 4915863 packets/sec
input rate 9.84 Gbps, 819.67 Kpps; output rate 2.52 Gbps, 4.91 Mpps
0 Rx pause
98376923 Tx pause
Ethernet147/1/9 is up
30 seconds input rate 9836252112 bits/sec, 819687 packets/sec
30 seconds output rate 2516980960 bits/sec, 4915978 packets/sec
input rate 9.84 Gbps, 819.69 Kpps; output rate 2.52 Gbps, 4.91 Mpps
0 Rx pause
98376916 Tx pause
esc-6001# show port-channel traffic interface port-channel 147
ChanId Port Rx-Ucst Tx-Ucst Rx-Mcst Tx-Mcst Rx-Bcst Tx-Bcst
------ --------- ------- ------- ------- ------- ------- -------
147 Eth1/41 99.99% 0.0% 12.50% 12.50% 0.0% 0.0%
147 Eth1/42 0.0% 0.0% 12.50% 12.50% 0.0% 0.0%
147 Eth1/43 0.0% 0.0% 12.50% 12.50% 0.0% 0.0%
147 Eth1/44 0.0% 0.0% 12.50% 12.50% 0.0% 0.0%
147 Eth1/45 0.0% 0.0% 12.50% 12.50% 0.0% 0.0%
147 Eth1/46 0.0% 0.0% 12.50% 12.50% 0.0% 0.0%
147 Eth1/47 0.00% 99.00% 12.50% 12.50% 0.0% 0.0%
147 Eth1/48 0.0% 1.00% 12.50% 12.50% 0.0% 0.0%
esc-6001# attach fex 147
Attaching to FEX 147 ...
To exit type 'exit', to abort type '$.'
fex-147# dbgexec w
woo> rate
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+
| Port || Tx Packets | Tx Rate | Tx Bit || Rx Packets | Rx Rate | Rx Bit |Avg Pkt|Avg Pkt| |
| || | (pkts/s) | Rate || | (pkts/s) | Rate | (Tx) | (Rx) |Err|
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+
| 0-NI8 || 24 | 4 | 11.23Kbps || 22 | 4 | 16.49Kbps | 272 | 448 | |
| 0-NI7 || 15 | 3 | 4.17Kbps || 17 | 3 | 3.81Kbps | 154 | 120 | |
| 0-NI6 || 1 | 0 | 1.76Kbps || 1 | 0 | 1.08Kbps | 1080 | 656 | |
| 0-NI5 || 1 | 0 | 1.76Kbps || 1 | 0 | 1.08Kbps | 1080 | 656 | |
| 0-NI4 || 1 | 0 | 1.76Kbps || 1 | 0 | 1.08Kbps | 1080 | 656 | |
| 0-NI3 || 1 | 0 | 1.76Kbps || 1 | 0 | 1.08Kbps | 1080 | 656 | |
| 0-NI2 || 1 | 0 | 1.76Kbps || 1 | 0 | 1.08Kbps | 1080 | 656 | |
| 0-NI1 || 1 | 0 | 1.76Kbps || 1 | 0 | 1.08Kbps | 1080 | 656 | |
| 0-NI0 || 4108297 | 821659 | 10.05Gbps || 1 | 0 | 1.08Kbps | 1509 | 656 | |
| 0-HI31 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI30 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI29 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI28 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI27 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI26 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI25 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI24 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI23 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI22 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI21 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI20 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI19 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI18 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI17 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI16 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI14 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI13 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI12 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI11 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI10 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI9 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI8 || 24556087 | 4911217 | 3.30Gbps || 4094470 | 818894 | 9.95Gbps | 64 | 1500 | |
| 0-HI6 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI5 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI4 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI3 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI2 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI1 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI0 || 24560241 | 4912048 | 3.30Gbps || 4095156 | 819031 | 9.95Gbps | 64 | 1500 | |
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+
Drops:
There are drops since host is not configured for flow control.
With flow-control enabled on the hosts, the hosts honor pause and throttle backs:
esc-6001# clear counters ; clear qos stat
esc-6001#
esc-6001# show interface ethernet 147/1/1, ethernet 147/1/9 | inc Ethernet14|rate|pause
Ethernet147/1/1 is up
30 seconds input rate 4926871976 bits/sec, 410572 packets/sec
30 seconds output rate 1288637816 bits/sec, 2516870 packets/sec
input rate 4.93 Gbps, 410.57 Kpps; output rate 1.29 Gbps, 2.52 Mpps
0 Rx pause
88129183 Tx pause
Ethernet147/1/9 is up
30 seconds input rate 4924820632 bits/sec, 410401 packets/sec
30 seconds output rate 1287225224 bits/sec, 2514111 packets/sec
input rate 4.92 Gbps, 410.40 Kpps; output rate 1.29 Gbps, 2.51 Mpps
0 Rx pause
88069874 Tx pause
esc-6001# show port-channel traffic interface port-channel 147
ChanId Port Rx-Ucst Tx-Ucst Rx-Mcst Tx-Mcst Rx-Bcst Tx-Bcst
------ --------- ------- ------- ------- ------- ------- -------
147 Eth1/41 99.99% 0.0% 12.50% 12.50% 0.0% 0.0%
147 Eth1/42 0.0% 0.0% 12.50% 12.50% 0.0% 0.0%
147 Eth1/43 0.0% 0.0% 12.50% 12.50% 0.0% 0.0%
147 Eth1/44 0.0% 0.0% 12.50% 12.50% 0.0% 0.0%
147 Eth1/45 0.0% 0.0% 12.50% 12.50% 0.0% 0.0%
147 Eth1/46 0.0% 0.0% 12.50% 12.50% 0.0% 0.0%
147 Eth1/47 0.00% 99.00% 12.50% 12.50% 0.0% 0.0%
147 Eth1/48 0.0% 1.00% 12.50% 12.50% 0.0% 0.0%
esc-6001# attach fex 147
Attaching to FEX 147 ...
To exit type 'exit', to abort type '$.'
fex-147# dbgexec w
woo> rate
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+
| Port || Tx Packets | Tx Rate | Tx Bit || Rx Packets | Rx Rate | Rx Bit |Avg Pkt|Avg Pkt| |
| || | (pkts/s) | Rate || | (pkts/s) | Rate | (Tx) | (Rx) |Err|
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+
| 0-NI8 || 32 | 6 | 19.76Kbps || 19 | 3 | 16.01Kbps | 366 | 506 | |
| 0-NI7 || 13 | 2 | 3.85Kbps || 20 | 4 | 5.14Kbps | 165 | 140 | |
| 0-NI6 || 1 | 0 | 1.76Kbps || 2 | 0 | 2.16Kbps | 1080 | 656 | |
| 0-NI5 || 1 | 0 | 1.76Kbps || 2 | 0 | 2.16Kbps | 1080 | 656 | |
| 0-NI4 || 1 | 0 | 1.76Kbps || 2 | 0 | 2.16Kbps | 1080 | 656 | |
| 0-NI3 || 1 | 0 | 1.76Kbps || 2 | 0 | 2.16Kbps | 1080 | 656 | |
| 0-NI2 || 1 | 0 | 1.76Kbps || 2 | 0 | 2.16Kbps | 1080 | 656 | |
| 0-NI1 || 1 | 0 | 1.76Kbps || 2 | 0 | 2.16Kbps | 1080 | 656 | |
| 0-NI0 || 4105292 | 821058 | 10.04Gbps || 2 | 0 | 2.16Kbps | 1509 | 656 | |
| 0-HI31 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI30 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI29 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI28 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI27 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI26 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI25 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI24 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI23 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI22 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI21 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI20 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI19 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI18 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI17 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI16 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI14 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI13 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI12 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI11 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI10 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI9 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI8 || 12556848 | 2511369 | 1.68Gbps || 2049754 | 409950 | 4.98Gbps | 63 | 1500 | |
| 0-HI6 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI5 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI4 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI3 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI2 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI1 || 1 | 0 | 2.28Kbps || 1 | 0 | 2.28Kbps | 1412 | 1412 | |
| 0-HI0 || 12573036 | 2514607 | 1.68Gbps || 2051092 | 410218 | 4.98Gbps | 64 | 1499 | |
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+
Remediation
By default, for IP traffic, FEX load-balance is based on source-destination MAC/IP. For problems like this, change the hashing algorithm in order to get a better distribution of traffic over the fabric port-channel. Use this method if you see unequal load balancing. This option is not an absolute solution:
esc-6001# show port-channel load-balance
Port Channel Load-Balancing Configuration:
System: source-dest-ip
Port Channel Load-Balancing Addresses Used Per-Protocol:
Non-IP: source-dest-mac
IP: source-dest-ip source-dest-mac
Which hashing algorithm to choose depends on traffic profile. Here are the options available.
esc-6001# conf t
Enter configuration commands, one per line. End with CNTL/Z.
esc-6001(config)# port-channel load-balance ethernet ?
destination-ip Destination IP address
destination-mac Destination MAC address
destination-port Destination TCP/UDP port
source-dest-ip Source & Destination IP address (includes l2)
source-dest-ip-only Source & Destination IP addresses only
source-dest-mac Source & Destination MAC address
source-dest-port Source & Destination TCP/UDP port (includes l2 and l3)
source-dest-port-only Source & Destination TCP/UDP port only
source-ip Source IP address
source-mac Source MAC address
source-port Source TCP/UDP port
Conclusions and Best Practices
1. TX Pause is normal operational mechanism in order to avoid packet drops in 2232/2248UPQ/B22 FEX.
2. Maximise number of uplinks between 2232/2248UPQ/B22 FEX and parent. In order to be able to have more paths towards the network and also it helps to have max buffers for N2H traffic.
3. If uplinks between FEX and parent and not evenly used, the change of port-channel hashing can help.
4. Since there is no local switching on FEX, avoid have east-west traffic flow profiles on hosts on FEX.
5. Avoid bursty appliances such as NAS devices, blade chassis on FEXes. These need to be on the parent.
6. Newer 2348UPQ FEX with 32M shared buffer, has 1MB shared buffer per HIF for H2N traffic for better burst absorption. Also, with 40G NIF uplinks, chances of a hash collisions/uplink congestion are minimized greatly.
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)