This document describes how the hardware ip glean throttle feature works with examples and the intention of this feature.
Cisco recommends that you have basic knowledge of Nexus 7000 Series Switches configuration.
The information in this document is based on these software and hardware versions:
Nexus 7000 with Release 6.2.x and later
F2e series line card
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
When you forward an incoming IP packet in a line card, if the Address Resolution Protocol (ARP) request for the next hop is not resolved, the line card forwards the packets to the supervisor in order to generate an ARP request. Once the ARP request responds to the supervisor, it resolves the MAC address for the next hop and programs the hardware.
If the supervisor cannot resolve the ARP entry, then the linecard sends all packets destined to that address to the supervisor. The supervisor generates ARP requests indefinitely until the ARP entry is resolved. There is a hardware rate limiter called glean placed in order to protect the supervisor's processor (CPU) from excessive traffic.
An issue that can arise is a single destination IP drops off the network due to maintenance or a hardware problem, and suddenly all traffic destined to it is being sent to the CPU. Since the rate limiter is in place, the CPU does not go high but this single destination IP can consume the entire rate limiter and not give other legitimate IP's access to the CPU. It is for this scenario that hardware ip glean throttle was created.
With the hardware ip glean throttle configuration, routed traffic for each unknown destination IP reaches the CPU post Hardware Rate Limiter (HWRL) action for ARP resolution. Unreachable destination will result in a /32 drop adjacency to be created in hardware. This prevents additional packets to the same next-hop IP address to be forwarded to the supervisor. While this drop adjacency is added, subsequent packets are dropped yet the supervisor continues to generate ARP requests until the next-hop is resolved. The drop adjacency is installed for a short period of time, which is configurable. Once the timer expires, one packet is again sent to the CPU and the process repeats. The number of entries that is installed in this fashion is limited to 1000 by default, but is configurable to a larger number of desired. This is to limit the impact on the Routing Information Base (RIB) table size.
In this case, you have a server, 172.28.191.200, which is down due to a hardware failure, and is currently unavailable to service traffic.
Note: There is no ARP entry for the host and no adjacency is created.
N7K# show ip route vrf VRF_ABC 172.28.191.200 IP Route Table for VRF "VRF_ABC" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%<string>' in via output denotes VRF <string>
172.28.191.192/28, ubest/mbest: 1/0, attached >>> There is no /32 entry *via 172.28.191.195, Vlan1601, [0/0], 02:01:17, direct
Traffic is sent to the supervisor in order to generate an ARP request:
The glean rate limiter for the specific module throttles the traffic to 100 packets per second, per module. You can see that some of the packets gets dropped.
N7K# show hardware rate-limiter Units for Config: packets per second Allowed, Dropped & Total: aggregated since last clear counters rl-1: STP and Fabricpath-ISIS rl-2: L3-ISIS and OTV-ISIS rl-3: UDLD, LACP, CDP and LLDP rl-4: Q-in-Q and ARP request rl-5: IGMP, NTP, DHCP-Snoop, Port-Security, Mgmt and Copy traffic
When the hardware ip glean throttle command is configured:
N7K(config)#hardware ip glean throttle
An adjacency is installed in the RIB:
N7K# show ip route 172.28.191.200 vrf VRF-ABC IP Route Table for VRF "VRF-ABC" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%<string>' in via output denotes VRF <string>
172.28.191.200/32, ubest/mbest: 1/0, attached *via 172.28.191.200, Vlan1601, [250/0], 00:01:37, am
When you look at the hardware programming, a drop index is installed:
N7K# show system internal forwarding vrf VRF_ABC ipv4 route 172.28.191.200 detail
You can now see that the hardware rate-limiter does not see any drops.
N7K# show hardware rate-limiter
Units for Config: packets per second Allowed, Dropped & Total: aggregated since last clear counters rl-1: STP and Fabricpath-ISIS rl-2: L3-ISIS and OTV-ISIS rl-3: UDLD, LACP, CDP and LLDP rl-4: Q-in-Q and ARP request rl-5: IGMP, NTP, DHCP-Snoop, Port-Security, Mgmt and Copy traffic