Check the physical interface statistics in order to determine if the traffic is dropped in the egress direction. Determine if the "output discard" counter in the TX direction increments and/or is non-zero.
Nexus3548# show interfce Eth1/7 Ethernet1/7 is up Dedicated Interface Hardware: 100/1000/10000 Ethernet, address: a44c.116a.913c (bia a44c.116a.91ee) Description: Unicast Only Internet Address is 220.127.116.11/30 MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec reliability 255/255, txload 35/255, rxload 1/255 Encapsulation ARPA full-duplex, 1000 Mb/s, media type is 1G Beacon is turned off Input flow-control is off, output flow-control is off Rate mode is dedicated Switchport monitor is off EtherType is 0x8100 Last link flapped 00:03:48 Last clearing of "show interface" counters 00:03:55 1 interface resets 30 seconds input rate 200 bits/sec, 0 packets/sec 30 seconds output rate 0 bits/sec, 0 packets/sec Load-Interval #2: 5 minute (300 seconds) input rate 40 bps, 0 pps; output rate 139.46 Mbps, 136.16 Kpps RX 1 unicast packets 118 multicast packets 0 broadcast packets 119 input packets 9830 bytes 0 jumbo packets 0 storm suppression bytes 0 runts 0 giants 0 CRC 0 no buffer 0 input error 0 short frame 0 overrun 0 underrun 0 ignored 0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop 0 input with dribble 0 input discard 0 Rx pause TX 23605277 unicast packets 0 multicast packets 0 broadcast packets 23605277 output packets 3038908385 bytes 0 jumbo packets 0 output errors 0 collision 0 deferred 0 late collision 0 lost carrier 0 no carrier 0 babble 11712542 output discard 0 Tx pause
Determine if the Drops are Unicast or Multicast
Once it is determined that the interface drops the traffic, enter the show queuing interface <x/y> command in order to find out if the traffic dropped is multicast or unicast. In releases earlier than 6.0(2)A3(1), the output looks like:
Note: If the multicast slow receiver is configured for the port, see for feature information, drops are not tracked with the show queuing interface Eth<x/y> command due to a hardware limitation. See Cisco bug ID CSCuj21006.
Determine Which Output Buffer is Used
In Nexus 3500, there are three buffer pools used in the egress direction. The output of the show hardware internal mtc-usd info port-mapping command provides the mapping information.
If the output discards actively increment, enable Active Buffer Monitoring (ABM) with this command. Note that the command allows you to monitor unicast or multicast, but not both. Also, it lets you configure the sampling interval and threshold values.
Information in each row is logged at a second interval. Each column represents the buffer usage. As mentioned in the command results, if there is a non-zero value reported for column "384", it means that buffer usage was between 0-384 KBytes when the ABM polled the OB usage. The non-zero number is the number of times the usage was reported.
These results indicate that OB1 averaged 5.376 MB of usage between 249 - 253 times for each second in the last 10 seconds for Eth1/7. It takes 4298 microseconds (us) in order to clear the buffer of this traffic.
Generate a Log When a Threshold Is Crossed
If the drop counter and buffer usage increments periodically, then it is possible to set a threshold and generate a log message when the threshold is crossed.
The command is set to monitor unicast traffic at a 10 nanoseconds interval and when it goes above 75% of the buffer it generates a log.
You can also create a scheduler in order to collect ABM statistics and interface counter output every hour and append it to bootflash files. This example is for multicast traffic:
hardware profile buffer monitor multicast
feature scheduler scheduler job name ABM show hardware profile buffer monitor detail >> ABMDetail.txt show clock >> ABMBrief.txt show hardware profile buffer monitor brief >> ABMBrief.txt show clock >> InterfaceCounters.txt show interface counters errors >> InterfaceCounters.txt scheduler schedule name ABM time start now repeat 1:0 job name ABM
Notable Cisco Bug IDs
Cisco bug ID CSCum21350: Rapid port flaps cause all ports in same QoS buffer to drop all TX Multicast/Broadcast traffic. This is fixed in Release 6.0(2)A1(1d) and later.
Cisco bug ID CSCuq96923: The multicast buffer block is stuck, which results in egress multicast/broadcast drops. This issue is still under investigation.
Cisco bug ID CSCva20344: Nexus 3500 Buffer block / lockup - no TX multicast or broadcast. Unreproducible issue, potentially got fixed in Releases 6.0(2)U6(7), 6.0(2)A6(8), and 6.0(2)A8(3).
Cisco bug ID CSCvi93997: Cisco Nexus 3500 switches output buffer block stuck. This is fixed in Releases 7.0(3)I7(8) and 9.3(3).
Frequently Asked Questions
Does ABM impact performance or latency?
No, this feature does not impact latency or performance of the device.
What is the impact of the lower ABM hardware polling interval?
By default, the hardware polling interval is 4 milliseconds. You can configure this value as low as 10 nanoseconds. There is no performance or latency impact because of the lower hardware polling interval. The default hardware polling of 4 milliseconds is selected in order to make sure you do not overflow the histogram counters before the software polls every one second. If you lower the hardware polling interval then it might saturate the hardware counters at 255 samples. The device cannot handle a software polling lower than one second, in order to match the lower hardware polling due to CPU and memory restrictions. The whitepaper has the example of the lower hardware polling interval and its use case.
Appendix - Feature Information
18 MB packet buffer shared by three OB blocks:
~4 MB Reserved: Size based on configured Maximum Transmission Unit (MTU) (Per port sum of 2 x MTU Size x # of enabled QoS groups)
~14 MB Shared: Remainder of total buffer
~767 KB of OB: 0 for CPU destined packets
6 MB for each OB is shared by a set of 16 ports (show hardware internal mtc-usd info port-mapping command)
Unicast and multicast
Traffic classes of the same scheduling scheme
Traffic classes across the scheme
In this diagram:
Sustained congestion is introduced on 1 G Eth1/40.
Other multicast receivers (Eth1/1 - 3) on the buffer block are impacted due to the multicast scheduling behavior. Receivers on other buffer blocks remain unaffected.
"Multicast slow-receiver" can be applied to e1/40 in order to avoid traffic loss on non-congested ports.
"Multicast slow-receiver" drains the multicast at a 10 G rate on Eth1/40. Drops are still expected to occur on the congested port.
Configured with the hardware profile multicast slow-receiver port <x> command.
ASIC has 18 buckets and each bucket corresponds to a range of buffer utilization (for example, 0-384KB, 385-768KB, and so on).
ASIC polls the buffer utilization for all ports every 4 milliseconds (default). This ASIC polling interval is configurable as low as 10 nanoseconds.
Based on the buffer utilization for each hardware polling interval, the bucket counter for the corresponding range is incremented. That is, if port 25 consumes 500 KB of buffer, the bucket #2 (385-768KB) counter is incremented.
This buffer utilization counter is maintained for each interface in histogram format.
Each bucket is represented with 8 bits, so the counter maxes out at 255 and it is reset once the software reads the data.
Every one second, the software polls ASIC in order to download and clear all histogram counters.
These histogram counters are maintained in the memory for 60 minutes with one second granularity.
The software also makes sure it copies the buffer histogram to the bootflash every hour, which can be copied to the analyzer for further analysis.
Effectively, this maintains two hours worth of buffer histogram data for all ports, the latest one hour in the memory and second hour in the bootflash.