Guest

Cisco 12000 Series Routers

Troubleshooting Input Drops on the Cisco 12000 Series Internet Router

Cisco - Troubleshooting Input Drops on the Cisco 12000 Series Internet Router

Document ID: 18004

Updated: Jul 07, 2005

   Print

Introduction

This document explains how to troubleshoot an increase in the number of input drops that appears in the output of the show interface command on a Cisco 12000 Series Internet Router.

Prerequisites

Requirements

Readers of this document should have knowledge of these topics:

  • Cisco 12000 Series Internet Router architecture

Components Used

The information in this document is based on these software and hardware versions:

  • Any Cisco IOS® Software Release that supports the Cisco 12000 Series Internet Router. For example, Cisco IOS Software Releases12.0S and 12.0ST.

  • All the Cisco 12000 platforms, which include the 12008, 12012, 12016, 12404, 12410, and 12416.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

Symptoms

The most common symptom is an increase in the number of input drops. You can see the number of input drops in the output of the show interfaces command on the Cisco 12000 Series Internet Router. Here is a sample output of the show interfaces command:

Router#show interface Gig2/0
GigabitEthernet2/0 is up, line protocol is up
  Hardware is GigMac 3 Port GigabitEthernet, address is 0003.fd1a.9040
(bia 0003.fd1a.9040)
  Internet address is 203.177.3.21/24
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, rely 255/255, load 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex mode, link type is force-up, media type is SX
  output flow-control is unsupported, input flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:55:39
  Queueing strategy: fifo
  Output queue 0/40, 0 drops; input queue 27/75, 954 drops  
  
!--- Here are the input drops.

  5 minute input rate 3000 bits/sec, 5 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     7167 packets input, 601879 bytes, 0 no buffer
     Received 2877 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 3638 multicast, 0 pause input
     992 packets output, 104698 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     1 lost carrier, 21992 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Execute the show interfaces command every 10 seconds to check whether the drop counter increases for the input queue.

When a packet enters the router, the router attempts to forward the packet at interrupt level. If the router cannot find a match in an appropriate cache table, the router queues the packet in the input queue of the incoming interface to process the packet later. The router always processes some packets. However, the rate of processed packets never congests the input queue in stable networks with the appropriate configuration. If the input queue is full, the router drops the packet.

In the sample output, you cannot identify exactly which packets the router drops. In order to troubleshoot input queue drops, you need to find out which packets fill the input queue. The sample output indicates that 27 packets await in the input queue of interface GigabitEthernet2/0. The queue depth is 75 packets, and there have been 954 drops after you last cleared the interface counters.

Troubleshoot

In a network that clears a large number of routes, input queue drops can cause:

  • Layer 2 keepalive failures

  • Hot Standby Routing Protocol/Virtual Router Redundancy Protocol (HSRP/VRRP)

  • Interface flaps

Default values are inadequate for systems that support a large number of interfaces or routes, especially in larger Service Provider networks. A single clearing of Border Gateway Protocol (BGP) can often result in thousands of input queue drops on the same interface. Large input drops can severely hamper convergence times.

Complete these steps in order to avoid such a situation:

  1. Use the spd headroom 1000 global command to increase Selective Packet Discard (SPD) headroom.

    The default value for SPD headroom is 100. The spd headroom command specifies how many high-precedence packets you can enqueue over the normal input hold queue limit. High-precedence packets include routing protocol updates and other important control traffic, for example, Layer 2 keepalives and IS-IS hello. When you specify this value, you reserve room for incoming high precedence packets. In Cisco IOS Software Release 12.0(22)S and later, the default value for SPD headroom is 1000 for Cisco 12000 Series Internet Router. Use the show ip spd command to check the value.

  2. Use hold-queue 1500 for each interface to increase the interface hold queue value. The default value is 75.

As mentioned earlier in the document, only packets destined to the router reach the input queue. The Gigabit Route Processor (GRP) must determine how to handle the packets. All the packets are process-switched. Therefore, the packets take the slow path. Usually, all packets that the Cisco 12000 router switches use Distributed Cisco Express Forwarding (dCEF) through the line cards. This platform supports only dCEF as the switching method.

Sometimes drops occur during Border Gateway Protocol (BGP) convergence if the router has a large number of peers. However, there are plenty of valid reasons why the GRP has to look at some packets. Some of the reasons are listed here:

  • The GRP receives routing updates.

  • The GRP handles Internet Control Message Protocol (ICMP) packets.

  • The GRP establishes and keeps BGP peer sessions.

Use the show interfaces stat command to check whether there are any process-switched packets.

If the Cisco 12000 router is not yet in production, you can enable some debug commands. Debug commands enable you to capture more information about the type of packets that the GRP receives. The debug ip packet output is very useful. However, be very cautious with this command, because this command can affect the behavior of the router through a hang, crash, or similar issues. Disable console logs to avoid a burst of messages to the console port. Enable the log buffer to redirect the output of the debug command to a buffer that you can consult later. Use the show logging command to view the buffer. You can also specify an access-list to narrow down the debug output. In order to specify an access-list, use this configuration:

no logging console
logging buffer 128000
debug ip packet <ACL #> 

!--- Warning: 
!--- Be aware that this configuration on a production router can damage the box.

undebug all (after 5-10 seconds)

This debug command enables you to see all the process-switched packets that the GRP receives. Alternatively, you can use the show buffers input-interface [interface type] [interface number] header command to identify the type of packets that fill up the input queue.

Note: This command is useful only when the input queue contains a lot of packets.

      Router#show buffers input-interface serial 0/0
       Buffer information for Small buffer at 0x612EAF3C
         data_area 0x7896E84, refcount 1, next 0x0, flags 0x0
         linktype 7 (IP), enctype 0 (None), encsize 46, rxtype 0
         if_input 0x6159D340 (FastEthernet3/2), if_output 0x0 (None)
         inputtime 0x0, outputtime 0x0, oqnumber 65535
         datagramstart 0x7896ED8, datagramsize 728, maximum size 65436
         mac_start 0x7896ED8, addr_start 0x7896ED8, info_start 0x0
         network_start 0x7896ED8, transport_start 0x0
         source: 212.176.72.138, destination: 212.111.64.174, id: 0xAAB8, 
         ttl: 118, prot: 1
       Buffer information for Small buffer at 0x612EB1D8
         data_area 0x78A6E64, refcount 1, next 0x0, flags 0x0
         linktype 7 (IP), enctype 0 (None), encsize 46, rxtype 0
         if_input 0x6159D340 (FastEthernet3/2), if_output 0x0 (None)
         inputtime 0x0, outputtime 0x0, oqnumber 65535
         datagramstart 0x78A6EB8, datagramsize 728, maximum size 65436
         mac_start 0x78A6EB8, addr_start 0x78A6EB8, info_start 0x0
         network_start 0x78A6EB8, transport_start 0x0
         source: 212.176.72.138, destination: 212.111.64.174, id: 0xA5B8, 
         ttl: 118, prot: 1

Often, the same type of packet is present in large quantities. For example, the sample output indicates a large number of ICMP packets (IP protocol 1).

Note: If you are unable to identify a pattern in the outputs of the debug or show buffers input-interface commands, the problem is most likely an incorrect router configuration.

Note: For more information, refer to Troubleshooting Input Queue Drops and Output Queue Drops.

Perform the appropriate actions based on the output of the debug ip packet detail command, or as outlined in Troubleshooting Input Queue Drops and Output Queue Drops. For a detailed example, see the Case Study section.

Case Study

Sometimes, when you check the interface of your Cisco 12000 router, you notice that the interface drops incoming packets. As a result, the input drops counter value increases regularly. For example, consider this sample output:

Router#show interface Gig2/0
GigabitEthernet2/0 is up, line protocol is up 
Hardware is GigMac 3 Port GigabitEthernet, address is 0003.fd1a.9040
(bia 0003.fd1a.9040)
  Internet address is 203.177.3.21/24
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, rely 255/255, load 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex mode, link type is force-up, media type is SX
  output flow-control is unsupported, input flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:55:39
  Queueing strategy: fifo
  Output queue 0/40, 0 drops; input queue 27/75, 954 drops  

  !--- This is the input drops counter value.

  5 minute input rate 3000 bits/sec, 5 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     7167 packets input, 601879 bytes, 0 no buffer
     Received 2877 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 3638 multicast, 0 pause input
     992 packets output, 104698 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     1 lost carrier, 21992 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Some input drops appear in the show interfaces command output. If you issue this command every 10 seconds, you can check whether the drop counter increases for the input queue.

Use the show interface stat command to check for the presence of process-switched packets:

Router#show interfaces stat
.....
GIG2/0
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out           
               Processor      45354    1088496          0          0
           !--- Here are the packets that are process-switched (sent to the GRP)
             Route cache          0          0          0          0
         Distributed cef          0          0       8575     207958
                   Total      45354    1088496       8575     207958
....

If the Cisco 12000 router is not yet in production, you can enable some debug commands to capture more information on the type of packets that the GRP receives. The output of the debug ip packet command is interesting. With this debug command, you can see all the process-switched packets that the GRP receives. Issue the show logging command after some time:

Router#show log
 Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns)
 Console logging: disabled
 Monitor logging: level debugging, 1110 messages logged
 Logging to: vty2(572) vty3(538)
 Buffer logging: level debugging, 107 messages logged
 Trap logging: level informational, 162 message lines logged
 Log Buffer (10000 bytes):
 *Jan 13 08:03:51.550: %SYS-5-CONFIG_I: Configured from console by vty2 (144.254.2.215)
 1w5d: IP: s=203.177.3.21 (local), d=144.254.2.215 (GigabitEthernet2/0), len 79,
 sending
 1w5d: IP: s=203.177.3.62 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=0.0.0.0 (GigabitEthernet2/0), d=255.255.255.255, len 328, rcvd 2
 1w5d: IP: s=203.177.3.15 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=144.254.2.215 (GigabitEthernet2/0), d=203.177.3.21 (GigabitEthernet2/0), 
  len 40, rcvd 3
 1w5d: IP: s=203.177.3.1 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.2 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.10 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.6 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.8 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.62 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.1 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.15 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.8 (GigabitEthernet2/0), d=224.0.0.10, len 69, unroutable
 1w5d: IP: s=203.177.3.2 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.10 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.8 (GigabitEthernet2/0), d=224.0.0.10, len 89, unroutable
 1w5d: IP: s=203.177.3.6 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.8 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.62 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.15 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.1 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=144.254.2.215 (GigabitEthernet2/0), d=203.177.3.21 (GigabitEthernet2/0), 
  len 41, rcvd 3
 1w5d: IP: s=203.177.3.21 (local), d=144.254.2.215 (GigabitEthernet2/0), len 41,
 sending
 1w5d: IP: s=203.177.3.2 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.10 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=144.254.2.215 (GigabitEthernet2/0), d=203.177.3.21 (GigabitEthernet2/0), 
  len 41, rcvd 3
 1w5d: IP: s=203.177.3.21 (local), d=144.254.2.215 (GigabitEthernet2/0), len 41,
 sending
 1w5d: IP: s=203.177.3.8 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=203.177.3.6 (GigabitEthernet2/0), d=224.0.0.10, len 60, unroutable
 1w5d: IP: s=144.254.2.215 (GigabitEthernet2/0), d=203.177.3.21 (GigabitEthernet2/0), 
  len 43, rcvd 3
 1w5d: IP: s=203.177.3.21 (local), d=144.254.2.215 (GigabitEthernet2/0), len 41,
 sending
 1w5d: IP: s=203.177.3.21 (local), d=144.254.2.215 (GigabitEthernet2/0), len 41,
 sending

In this example, the GigabitEthernet2/0 interface receives a lot of Enhanced Interior Gateway Routing Protocol (EIGRP) packets. EIGRP uses the multicast address 224.0.0.10, but you have not configured the router to handle such packets. Therefore, the router sends these packets to the GRP. The GRP takes a decision to drop the packets, because the GRP cannot handle these packets fast enough.

In order to ensure that the GRP does not receive these EIGRP packets, you can perform one of these actions:

  • Specify the interface as passive on the other routers.

  • Specify different neighbor routers.

Cisco IOS Software Bugs

Sometimes, the number of input drops increases because of a Cisco IOS software defect. For example, in Cisco IOS Software Release 12.0(11)S, the Cisco 12000 Series Internet Router incorrectly increments the input drops counter because of an accounting issue. The output does not correctly reflect the number of dropped packets during congestion. All interfaces can indicate this issue, but the issue does not affect the service or functionality of the interfaces. There is no known workaround.

Ensure that you run the latest available Cisco IOS software release in your train to eliminate the bugs that are fixed. If you still see drops afterwards, open a service request through the .

Related Information

Updated: Jul 07, 2005
Document ID: 18004