This document describes high CPU utilization on various Catalyst platforms due to flooding of IPV6 Multicast Listener Discovery packets and ways to mitigate this problem.
There are no prerequisites.
There are no specific requirements for this document.
The information in this document is based on Cisco Catalyst 6500 Series Switches, Catalyst 4500 Series Switches and Catalyst 3850 Series Switches.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration.
High CPU Utilization may be seen on some Cisco Catalyst platforms due to IPv6 Multicast traffic with MAC address in the range 3333.xxxx.xxxx being punted to CPU.
As per RFC7042, all MAC-48 multicast identifiers prefixed "33-33" (that is, the 2**32 multicast MAC identifiers in the range from 33-33-00-00-00-00 to 33-33-FF-FF-FF-FF) are used as specified in [RFC2464] for IPv6 multicast. An IPv6 packet with a multicast destination address DST, consisting of the sixteen octets DST through DST, is transmitted to the Ethernet multicast address whose first two octets are the value 3333 hexadecimal and whose last four octets are the last four octets of DST as shown in Figure 1.
|0 0 1 1 0 0 1 1|0 0 1 1 0 0 1 1|
| DST | DST |
| DST | DST |
(Figure 1: Address Mapping - Multicast)
It has been seen on some occasions that when hosts devices using a certain NIC card go to sleep mode, they flood IPv6 multicast traffic. This issue is not limited to a particular host vendor, though certain chipsets have been seen to exhibit this behavior more often than others.
Troubleshooting & Solution
You can use following procedures to find out if your Catalyst switch seeing high CPU utilization is affected by this problem, and implement respective solutions.
Catalyst 3850 Series Switches
On Catalyst 3850 switches, NGWC L2M Process uses CPU to process IPv6 packets. When Multicast Listener Discovery (MLD) snooping is disabled on the switch, MLD join/leave packet are flooded to all the member ports. And, if there are many incoming MLD join/leave packets, this process will consume more CPU cycles to send out the packets on all the member ports. It has been seen that when certain host machines go to sleep mode, they may send several thousand packets/sec of IGMPv6 MLD traffic.
3850#show processes cpu detailed process iosd sorted | exc 0.0
Core 0: CPU utilization for five seconds: 43%; one minute: 35%; five minutes: 33%
Core 1: CPU utilization for five seconds: 54%; one minute: 46%; five minutes: 46%
Core 2: CPU utilization for five seconds: 75%; one minute: 63%; five minutes: 58%
Core 3: CPU utilization for five seconds: 48%; one minute: 49%; five minutes: 57%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
12577 L 2766882 2422952 291 23.52 23.67 23.69 34816 iosd
12577 L 3 12577 1911782 1970561 0 23.34 23.29 23.29 34818 iosd
12577 L 0 14135 694490 3264088 0 0.28 0.34 0.36 0 iosd.fastpath
162 I 2832830 6643 0 93.11 92.55 92.33 0 NGWC L2M
Configure ipv6 mld snooping on the affected switches to globally enable ipv6 mld snooping. This should lower down the CPU utilization.
Enter configuration commands, one per line. End with CNTL/Z.
3850(config)#ipv6 mld snooping
When MLD snooping is enabled, a per-VLAN IPv6 multicast address table is constructed in software and hardware. Switch then performs IPv6 multicast-address based bridging in hardware, which prevents these packets to be processed by software.
Click on link for more information on Configuring MLD Snooping
On earlier versions of IOS XE, it was found that CPU queue could get stuck due to this problem which would stop all control packets in that queue from going to the CPU. This was fixed through CSCuo14829 in IOS versions 3.3.3 and 3.6.0 and later. Please refer this bug for details.
Catalyst 4500 Series Switches
Catalyst 4500 series switches support hardware forwarding of IPv6 Multicast traffic using Ternary Content Addressable Memory (TCAM). This is explained in Multicast on Cisco Catalyst 4500E and 4500X Series Switches
When it comes to IPv6 Multicast Listener Discovery traffic, switch needs to perform software forwarding (using CPU resources). As explained in Configuring IPv6 MLD Snooping on Catalyst 4500 Switches MLD snooping can be enabled or disabled globally or per VLAN. When MLD snooping is enabled, a per-VLAN IPv6 multicast MAC address table is constructed in software and a per-VLAN IPv6 multicast address table is constructed in software and hardware. The switch then performs IPv6 multicast-address based bridging in hardware. This is the expected behavior on Catalyst 4500 series switches.
In order to check type of packet being punted to CPU we can run “debug platform packet all buffer” followed by “show platform cpu packet buffered” command.
4500#debug platform packet all buffer
platform packet debugging is on
Cat4500#sh platform cpu packet buffered
Total Received Packets Buffered: 1024
33 days 11:42:21:833532 - RxVlan: 214, RxPort: Te1/15
Priority: Normal, Tag: Dot1Q Tag, Event: L2 Router, Flags: 0x40, Size: 90
Eth: Src 44:39:C4:39:5A:4A Dst 33:33:FF:7F:EB:DB Type/Len 0x86DD
0: 0x60 0x0 0x0 0x0 0x0 0x20 0x0 0x1 0xFE 0x80
10: 0x0 0x0 0x0 0x0 0x0 0x0 0x46 0x39 0xC4 0xFF
20: 0xFE 0x39 0x5A 0x4A 0xFF 0x2 0x0 0x0 0x0 0x0
30: 0x0 0x0 0x0 0x0 0x0 0x1 0xFF 0x7F 0xEB 0xDB
40: 0x3A 0x0 0x5 0x2 0x0 0x0 0x1 0x0 0x83 0x0
This packet arrived on interface Tengigabitethernet1/15 on vlan 214 from source mac address 44:39:C4:39:5A:4A. Protocol 0x86DD is IPv6 and Dst MAC 33:33:FF:7F:EB:DB is being used for Multicast IPv6 MLD nodes in this case.
We have two options to fix high CPU utilization due to this traffic.
- Disable generation of IPv6 Multicast Listener Discovery traffic on end host. This can done by upgrading NIC drivers or disabling the feature on the BIOS of hosts sending IPv6 packets. You can contact your client machine's vendor who can help to disable feature on BIOS or upgrade NIC drivers.
- Enable Control Plane Policing (CoPP) in order to drop the excessive amount of IPv6 Multicast Listener Discovery traffic which is being punted to the CPU. And, these packets are hop limit of one link local, thus it is expected behavior that these packets will be punted to CPU.
ipv6 access-list IPv6-Block
permit ipv6 any any
match access-group name IPv6-Block
police 32000 conform-action drop exceed-action drop
service-policy input ipv6
In above example, we are limiting the amount of IPv6 traffic which is handled by the CPU to 32000 packets per second.
Catalyst 6500 Series Switches
Catalyst 6500 Switches make forwarding decisions in hardware using TCAM which does not normally need CPU assistance as long as TCAM has the forwarding entry.
Supervisor Enginet 720 on Catalyst 6500 Switches have two CPUs. One CPU is the Network Management Processor (NMP) or the Switch Processor (SP). The other CPU is the Layer 3 CPU, which is called the Route Processor (RP).
Process and Interrupt CPU utilization are listed in show process cpu command. As shown below, High CPU caused by interrupts are mostly traffic based. Interrupt switched traffic, is traffic that does not match a specific process, but still needs to be forwarded. Following example shows a Catalyst 6500 switch having high CPU utilization on RP due to interrupts.
6500#show process cpu
CPU utilization for five seconds: 98%/92%;
one minute: 99%; five minutes: 99% PID Runtime(ms) Invoked
Check if any interface or Layer 3 Vlan is dropping high amount of traffic. (Input Queue drops). If so, traffic may be getting punted to RP from that vlan.
Vlan19 is up, line protocol is up
Input queue: 0/75/6303532/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
5 minute input rate 19932000 bits/sec, 26424 packets/sec
5 minute output rate 2662000 bits/sec, 1168 packets/sec
Following command can be used to find all packets in input queue buffer for interface vlan 19.
6500#show buffer input-interface vlan 19 packet
Alternatively, you can use NetDR capture to capture traffic going to CPU on a Catalyst 6500 switch. This document explains how to interpret packets captured using NetDR capture.
------- dump of incoming inband packet -------
interface Vl16, routine mistral_process_rx_packet_inlin, timestamp 03:17:56.380
dbus info: src_vlan 0x10(16), src_indx 0x1001(4097), len 0x5A(90)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x4010(16400)
E8820000 00100000 10010000 5A080000 0C000418 01000008 00000008 4010417E
mistral hdr: req_token 0x0(0), src_index 0x1001(4097), rx_offset 0x76(118)
requeue 0, obl_pkt 0, vlan 0x10(16)
destmac 33.33.FF.4A.C3.FD, srcmac C8.CB.B18.104.22.168, protocol 86DD
protocol ipv6: version 6, flow 1610612736, payload 32, nexthdr 0, hoplt 1
class 0, src FE80::CACB:B8FF:FE29:3362, dst FF02::1:FF4A:C3FD
Use one or more of below solutions.
- Drop IPv6 Multicast packets by using following configuration.
6500(config)#mac-address-table static 3333.FF4A.C3FD vlan <vlan #> drop
- Redirect IPv6 Multicast traffic to an unused or admin shutdown interface (Gi1/22 in this example).
6500(config)#mac-address-table 3333.FF4A.C3FD vlan 19 interface Gi1/22
- Use Vlan Access Control List (VACL) to drop IPv6 Multicast traffic.
6500(config)#mac access-li extended Multicast_MAC
6500(config-ext-macl)#permit any host 3333.FF4A.C3FD
6500(config)#vlan access-map block-ipv6 10
6500(config-access-map)#match mac address Multicast_MAC
6500(config-access-map)#vlan access-map block-ipv6 20
6500(config)#vlan filter block-ipv6 vlan-list <vlan #>
- Disable IPv6 MLD snooping.
6500(config)#no ipv6 mld snoopin
- Drop IPv6 Multicast traffic using Control Plane Policing (CoPP)
6500(config)#ipv6 access-list test
6500(config-ipv6-acl)#permit ipv6 any any
6500(config-cmap)#match access-group name test
6500(config-pmap-c)#police 320000 conform-action drop exceed-action drop
6500(config-cp)#service-policy in ipv6
- Use storm-control on ingress interfaces. storm-control monitors incoming traffic levels over a 1 second interval and during this interval it compares traffic level with configured traffic storm-control level. Traffic storm-control level is a percentage of the total available bandwidth of the port. Each port has a single traffic storm-control level that is used for all types of traffic (broadcast, multicast, and unicast).
6500(config-if)#storm-control multicast level 10
7. In case if CPU is High on SP (Switch Processor), apply workaround below.
6500(config)#mls rate-limit ipv6 mld 10 1
If you are unable to determine reason based on information provided in this document, please open a TAC service request to investigate further.