This document describes high CPU utilization on various Catalyst platforms due to flooding of IPV6 Multicast Listener Discovery packets and ways to mitigate this problem.
There are no prerequisites.
There are no specific requirements for this document.
The information in this document is based on Cisco Catalyst 6500 Series Switches, Catalyst 4500 Series Switches and Catalyst 3850 Series Switches.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration.
High CPU Utilization may be seen on some Cisco Catalyst platforms due to IPv6 Multicast traffic with MAC address in the range 3333.xxxx.xxxx being punted to CPU.
As per RFC7042, all MAC-48 multicast identifiers prefixed "33-33" (that is, the 2**32 multicast MAC identifiers in the range from 33-33-00-00-00-00 to 33-33-FF-FF-FF-FF) are used as specified in [RFC2464] for IPv6 multicast. An IPv6 packet with a multicast destination address DST, consisting of the sixteen octets DST through DST, is transmitted to the Ethernet multicast address whose first two octets are the value 3333 hexadecimal and whose last four octets are the last four octets of DST as shown in Figure 1.
It has been seen on some occasions that when hosts devices using a certain NIC card go to sleep mode, they flood IPv6 multicast traffic. This issue is not limited to a particular host vendor, though certain chipsets have been seen to exhibit this behavior more often than others.
Troubleshooting & Solution
You can use following procedures to find out if your Catalyst switch seeing high CPU utilization is affected by this problem, and implement respective solutions.
Catalyst 3850 Series Switches
On Catalyst 3850 switches, NGWC L2M Process uses CPU to process IPv6 packets. When Multicast Listener Discovery (MLD) snooping is disabled on the switch, MLD join/leave packet are flooded to all the member ports. And, if there are many incoming MLD join/leave packets, this process will consume more CPU cycles to send out the packets on all the member ports. It has been seen that when certain host machines go to sleep mode, they may send several thousand packets/sec of IGMPv6 MLD traffic.
3850#show processes cpu detailed process iosd sorted | exc 0.0 Core 0: CPU utilization for five seconds: 43%; one minute: 35%; five minutes: 33% Core 1: CPU utilization for five seconds: 54%; one minute: 46%; five minutes: 46% Core 2: CPU utilization for five seconds: 75%; one minute: 63%; five minutes: 58% Core 3: CPU utilization for five seconds: 48%; one minute: 49%; five minutes: 57% PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 12577 L 2766882 2422952 291 23.52 23.67 23.69 34816 iosd 12577 L 3 12577 1911782 1970561 0 23.34 23.29 23.29 34818 iosd 12577 L 0 14135 694490 3264088 0 0.28 0.34 0.36 0 iosd.fastpath 162 I 2832830 6643 0 93.11 92.55 92.33 0 NGWC L2M
Configure ipv6 mld snooping on the affected switches to globally enable ipv6 mld snooping. This should lower down the CPU utilization.
3850#conf t Enter configuration commands, one per line. End with CNTL/Z. 3850(config)#ipv6 mld snooping 3850(config)#end
When MLD snooping is enabled, a per-VLAN IPv6 multicast address table is constructed in software and hardware. Switch then performs IPv6 multicast-address based bridging in hardware, which prevents these packets to be processed by software.
On earlier versions of IOS XE, it was found that CPU queue could get stuck due to this problem which would stop all control packets in that queue from going to the CPU. This was fixed through CSCuo14829 in IOS versions 3.3.3 and 3.6.0 and later. Please refer this bug for details.
When it comes to IPv6 Multicast Listener Discovery traffic, switch needs to perform software forwarding (using CPU resources). As explained in Configuring IPv6 MLD Snooping on Catalyst 4500 Switches MLD snooping can be enabled or disabled globally or per VLAN. When MLD snooping is enabled, a per-VLAN IPv6 multicast MAC address table is constructed in software and a per-VLAN IPv6 multicast address table is constructed in software and hardware. The switch then performs IPv6 multicast-address based bridging in hardware. This is the expected behavior on Catalyst 4500 series switches.
In order to check type of packet being punted to CPU we can run “debug platform packet all buffer” followed by “show platform cpu packet buffered” command.
This packet arrived on interface Tengigabitethernet1/15 on vlan 214 from source mac address 44:39:C4:39:5A:4A. Protocol 0x86DD is IPv6 and Dst MAC 33:33:FF:7F:EB:DB is being used for Multicast IPv6 MLD nodes in this case.
We have two options to fix high CPU utilization due to this traffic.
Disable generation of IPv6 Multicast Listener Discovery traffic on end host. This can done by upgrading NIC drivers or disabling the feature on the BIOS of hosts sending IPv6 packets. You can contact your client machine's vendor who can help to disable feature on BIOS or upgrade NIC drivers.
Enable Control Plane Policing (CoPP) in order to drop the excessive amount of IPv6 Multicast Listener Discovery traffic which is being punted to the CPU. And, these packets are hop limit of one link local, thus it is expected behavior that these packets will be punted to CPU.
ipv6 access-list IPv6-Block permit ipv6 any any ! class-map TEST match access-group name IPv6-Block ! policy-map ipv6 class TEST police 32000 conform-action drop exceed-action drop ! control-plane service-policy input ipv6
In above example, we are limiting the amount of IPv6 traffic which is handled by the CPU to 32000 packets per second.
Catalyst 6500 Series Switches
Catalyst 6500 Switches make forwarding decisions in hardware using TCAM which does not normally need CPU assistance as long as TCAM has the forwarding entry.
Supervisor Enginet 720 on Catalyst 6500 Switches have two CPUs. One CPU is the Network Management Processor (NMP) or the Switch Processor (SP). The other CPU is the Layer 3 CPU, which is called the Route Processor (RP).
Process and Interrupt CPU utilization are listed in show process cpu command. As shown below, High CPU caused by interrupts are mostly traffic based. Interrupt switched traffic, is traffic that does not match a specific process, but still needs to be forwarded. Following example shows a Catalyst 6500 switch having high CPU utilization on RP due to interrupts.
6500#show process cpu CPU utilization for five seconds: 98%/92%; one minute: 99%; five minutes: 99% PID Runtime(ms) Invoked
Check if any interface or Layer 3 Vlan is dropping high amount of traffic. (Input Queue drops). If so, traffic may be getting punted to RP from that vlan.
Vlan19 is up, line protocol is up Input queue: 0/75/6303532/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo 5 minute input rate 19932000 bits/sec, 26424 packets/sec 5 minute output rate 2662000 bits/sec, 1168 packets/sec
Following command can be used to find all packets in input queue buffer for interface vlan 19.
6500#show buffer input-interface vlan 19 packet
Alternatively, you can use NetDR capture to capture traffic going to CPU on a Catalyst 6500 switch. This document explains how to interpret packets captured using NetDR capture.
Drop IPv6 Multicast traffic using Control Plane Policing (CoPP)
6500(config)#ipv6 access-list test 6500(config-ipv6-acl)#permit ipv6 any any 6500(config-ipv6-acl)#exit
6500(config)#class-map TEST 6500(config-cmap)#match access-group name test 6500(config-cmap)#exit
6500(config)#policy-map ipv6 6500(config-pmap)#class TEST 6500(config-pmap-c)#police 320000 conform-action drop exceed-action drop 6500(config-pmap-c)#exit
6500(config)#control-plane 6500(config-cp)#service-policy in ipv6 6500(config-cp)#exit
Use storm-control on ingress interfaces. storm-control monitors incoming traffic levels over a 1 second interval and during this interval it compares traffic level with configured traffic storm-control level. Traffic storm-control level is a percentage of the total available bandwidth of the port. Each port has a single traffic storm-control level that is used for all types of traffic (broadcast, multicast, and unicast).