Guest

IP Switching

Troubleshooting Incomplete Adjacencies with CEF

Cisco - Troubleshooting Incomplete Adjacencies with CEF

Document ID: 17812

Updated: Jun 07, 2006

   Print

Introduction

Network nodes in the network are considered adjacent if they can reach each other with a single hop across a link layer. This document provides tips on how to troubleshoot incomplete adjacencies, as the output of the show ip cef adjacency command shows when Cisco Express Forwarding (CEF) is enabled on an interface.

Router#show ip cef adjacency serial 4/0/1 10.10.78.69 detail 
IP Distributed CEF with switching (Table Version 2707655) 
   130703 routes, 0 reresolve, 0 unresolved (0 old, 0 new), peak 39517 
   130703 leaves, 9081 nodes, 26227536 bytes, 2685255 inserts, 2554552 invalidations 
   949 load sharing elements, 318864 bytes, 71787 references 
   universal per-destination load sharing algorithm, id 9E3B1A95 
   2 CEF resets, 23810 revisions of existing leaves  
   Resolution Timer: Exponential (currently 1s, peak 16s)
   22322 in-place/0 aborted modifications 
   refcounts:  2175265 leaf, 1972988 node 

   Table epoch: 0 (17 entries at this epoch)

Adjacency Table has 112 adjacencies 
   4 IPv4 incomplete adjacencies

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

Components Used

The information in this document is based on the Cisco IOS® Software Release 12.3(3).

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

What Is an Adjacency?

CEF describes a very high speed switching mechanism that a router uses to forward packets from the inbound to the outbound interface. CEF uses two sets of data structures or tables, which it stores in router memory:

  • Forwarding Information Base (FIB) leavingcisco.com—Taken from the common International Organization for Standardization (ISO) usage, an FIB describes a database of information used to make forwarding decisions. It is conceptually similar to a routing table or route-cache, although it is very different from a routing table in implementation.

  • Adjacency table —Two nodes in the network are considered adjacent if they can reach each other using a single hop across a link layer. For example, when a packet arrives at one of the router's interfaces, the router strips off the data-link layer framing and passes the enclosed packet to the network layer. At the network layer, the destination address of the packet is examined. If the destination address is not an address of the router's interface or the all hosts broadcast address, then the packet must be routed.

    At a minimum, each route entry in the database must contain two items:

    • Destination address—This is the address of the network the router can reach. The router may have more than one route to the same address.

    • Pointer to the destination—This pointer indicates that the destination network is directly connected to the router, or it indicates the address of another router on a directly-connected network towards the destination. That router, which is one hop closer to the destination, is the next-hop router. An adjacency represents the pointer to the destination.

This example uses an Ethernet interface of a router (for example R1) configured with an IP address of 172.16.81.98 and a simple default static route that points all destinations to the Ethernet interface of a neighboring router R2, with an IP address of 172.16.81.1 as the next hop. In general, CEF needs to be enabled on the incoming interface for packets to be CEF switched. Since CEF makes the forwarding decision on input, use the no ip route-cache cef command on the ingress interface to disable CEF.

Note: In fast-switching, Cisco IOS builds a fast-switching cache entry after it switches a packet. For example, a packet that comes on a process-switched interface and is sent out through a fast-switched interface is fast switched. Issue the no ip route-cache command on the egress interface to disable fast switching. This is in contrast to CEF.

  1. Use the show ip route command to view the contents of the IP routing table.

    R1#show ip route
    Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP 
           D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
           N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 
           E1 - OSPF external type 1, E2 - OSPF external type 2 
           i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 
           ia - IS-IS inter area,  * - candidate default, U - per-user static route
           o - ODR, P - periodic downloaded static route 
    Gateway of last resort is 172.16.81.1 to network 0.0.0.0 
    
         172.16.0.0/24 is subnetted, 1 subnets 
    C       172.16.81.0 is directly connected, Ethernet0/0
    S*   0.0.0.0/0 [1/0] via 172.16.81.1 
    
    !--- A simple default static route points all destinations to 
    !--- a next-hop address of 172.16.81.1.
    
    
  2. Use the show ip arp or the show arp command to display the Address Resolution Protocol (ARP) table.

    Note: The "Hardware Addr" field in the ARP table displays entries for the local interface and the next-hop interface.

    R1#show ip arp 
    Protocol  Address       Age (min)  Hardware Addr   Type   Interface 
    Internet  172.16.81.98         -   0030.71d3.1000  ARPA   Ethernet0/0
    Internet  172.16.81.1          0   0060.471e.91d8  ARPA   Ethernet0/0
  3. Use the show adjacency ethernet 0/0 detail and the show adjacency ethernet 0/0 internal commands to view the contents of the adjacency table entry.

    R1#show adjacency ethernet 0/0 detail
    Protocol Interface                 Address 
    IP       Ethernet0/0               172.16.81.1(7) 
                                       0 packets, 0 bytes 
                                       0060471E91D8003071D310000800  
                                       ARP        03:57:08 
                                       Epoch: 1					
    R1#show adjacency ethernet 0/0 internal
    Protocol Interface     Address 
    IP       Ethernet0/0   172.16.81.1(7) 
                           0 packets, 0 bytes 
                           0060471E91D8003071D310000800  
                           ARP        03:57:00
                           Epoch: 1
                           Fast adjacency enabled 
                           IP redirect enabled 
                           IP mtu 1500 (0x48000082) 
                           Fixup disabled 
                           Adjacency pointer 0x62515AC0, refCount 7 
                           Connection Id 0x0 
                           Bucket 236
    

    This output illustrates that in CEF, an adjacency refers to a control structure that holds Layer 2 information for an IP address on a particular interface. It contains the rewrite string that varies with the encapsulation protocol of the outbound interface. An adjacency is CEF's equivalent of an ARP entry.

This table describes key fields in the show adjacency [interface-type interface-number] internal command.

Field Description
172.16.81.1(7)
IP address of the next-hop interface. The value in parenthesis refers to the "refCount" or the number of times that this adjacency is pointed to by FIB entries. The same value appears later in the entry.
0 packets, 0 bytes
Use the ip cef accounting command to enable packet and byte counters.
0060471E91D8003071D310000800
The first twelve characters are the MAC address of the destination next-hop interface. The next twelve characters represent the MAC address of the source interface of the packet. (In other words, the outbound interface of the local router). The last four characters represent the well-known Ethertype value 0x0800 for IP (with Advanced Research Projects Agency (ARPA) encapsulation).
003071D310000800
MAC address and well-known Ethertype value 0x0800 for IP (with ARPA encapsulation) of the source interface of the packet. (In other words, the outbound interface of the local router).
ARP    03:57:00
ARP indicates how the entry is discovered. The timestamp indicates how long to go before the entry times out.
Epoch: 1
CEF adjacency table Epoch information. Use the show ip cef epoch command to display the epoch information for the adjacency table and all FIB tables.
Fast adjacency enabled
An FIB entry caches an adjacency for a next-hop interface when not doing load-sharing over multiple active paths. A fast adjacency facilitates faster switching of packets.
Adjacency pointer 0x62515AC0 
 
refCount 7
The number of references to the adjacency that are currently stored in the router's memory. There is one for each corresponding entry in the CEF table, plus a few others for a variety of reasons (such as one for the code that performs the show adjacency command).
Connection Id 0x0
 
Bucket 236
 

Types of Adjacency

Adjacency Type Adjacency Processing
Null adjacency Packets destined for a Null0 interface are dropped. This can be used as an effective form of access filtering.
Glean adjacency When a router is connected directly to several hosts, the FIB table on the router maintains a prefix for the subnet rather than for the individual host prefixes. The subnet prefix points to a glean adjacency. When packets need to be forwarded to a specific host, the adjacency database is gleaned for the specific prefix.
Punt adjacency Features that require special handling or features that are not yet supported in conjunction with CEF switching paths are forwarded to the next switching layer for handling. Features that are not supported are forwarded to the next higher switching level.
Discard adjacency Packets are discarded.
Drop adjacency Packets are dropped, but the prefix is checked.
Cached Adjacency Cached Adjacency is the Acknowledgement update received for the adjacency packet sent.

Adjacency Discovery

Adjacencies are added to the table either through indirect manual configuration or dynamically, when discovered through a mechanism like ARP or using a routing protocol, such as BGP and OSPF, which forms neighbor relationships. If an adjacency is created by the FIB and is not discovered dynamically, then the Layer 2 addressing information is not known and the adjacency is considered incomplete. Once the Layer 2 information is known, the packet is forwarded to the route processor, and the adjacency is determined through ARP.

ATM and Frame Relay interfaces can be configured as point-to-point or as a multipoint. The number of the type of adjacencies varies with the configuration:

  • Point-to-point interface—Uses a single adjacency for the interface.

  • Multipoint interface—Uses a unique adjacency or Layer 2 rewrite structure for each host IP address. The information to complete the adjacency comes from IP ARP, static ATM, or Frame Relay map statements, and inverse ARP on ATM and Frame Relay.

    Router#show adjacency serial 0 detail
    Protocol Interface                 Address 
    IP       Serial0                   140.108.1.1(25) 
                                       0 packets, 0 bytes 
                                       18410800 
                                       FR-MAP     never
                                       Epoch: 1
    IP       Serial0                   140.108.1.2(5) 
                                       0 packets, 0 bytes 
                                       18510800 
                                       FR-MAP     never
                                       Epoch: 1

When an ATM interface supports more than one permanent virtual circuit (PVC) on an interface, the "incomplete" error indication can appear for up to one minute, but it should not persist.

Note: In addition to regular adjacencies, CEF also supports five adjacency types that require special handling. These types are described in the Adjacency Types That Require Special Handling section of the Cisco Express Forwarding Overview and are outside the scope of this document.

Reasons for Incomplete Adjacencies

There are two known reasons for an incomplete adjacency:

  • The router cannot use ARP successfully for the next-hop interface.

  • After a clear ip arp or a clear adjacency command, the router marks the adjacency as incomplete. Then it fails to clear the entry.

  • In an MPLS environment, IP CEF should be enabeled for Label Switching. Interface level command ip route-cache cef

The symptoms of an incomplete adjacency include random packet drops during a ping test. Output drops result from throttling the rate at which CEF punts the arriving packets to the CPU. Use the debug ip cef command to view CEF drops due to an incomplete adjacency.

Router# 
 *Oct 11 17:08:03.275: CEF-Drop: 
Stalled adjacency for 192.168.10.2 on Serial0/1/3 for 
destination 192.168.11.1 
 *Oct 11 17:08:03.275: CEF-Drop: 
Packet for 192.168.11.1 -- encapsulation 
 *Oct 11 17:08:05.307: CEF-Drop: 
Stalled adjacency for 192.168.10.2 on Serial0/1/3 for 
destination 192.168.11.1 
 *Oct 11 17:08:05.307: CEF-Drop: 
Packet for 192.168.11.1 -- encapsulation 

In addition, use the show cef drop command several times and look for an incrementing value for the 'Encap_fail' counter. Refer to the show cef commands for more information.

No ARP Entry

When CEF cannot locate a valid adjacency for a destination prefix, it punts the packets to the CPU for ARP resolution and, in turn, for completion of the adjacency. In rare cases, the adjacency persists in an incomplete state. For example, if the ARP table already lists a particular host, then punting it to the process level does not trigger an ARP.

Determine whether an ARP entry exists in order to troubleshoot this problem. Use these commands and specify a specific IP address:

Use the debug arp command to confirm that the router sends an ARP request.

Router#ping 10.12.241.4 
Type escape sequence to abort. 
Sending 5, 100-byte ICMP Echos to 10.12.241.4, timeout is 2 seconds: 
..... 
Success rate is 0 percent (0/5) 
Router# 
.Aug 21 18:59:07.175 PDT: 
IP ARP: 
creating incomplete entry for IP address:10.12.241.4 interface FastEthernet0/1 
.Aug 21 18:59:07.177 PDT: IP ARP: sent req src 10.12.241.252 0006.529c.9801, 
					dst 10.12.241.4 0000.0000.0000 FastEthernet0/1 
.Aug 21 18:59:07.180 PDT: IP ARP throttled out the ARP Request for 10.12.241.4 
.Aug 21 18:59:09.182 PDT: IP ARP: sent req src 10.12.241.252 0006.529c.9801, 
					dst 10.12.241.4 0000.0000.0000 FastEthernet0/1 
.Aug 21 18:59:09.183 PDT: 
IP ARP throttled out the ARP Request for 10.12.241.4

When the ping process tries to send the first packet and does not see an ARP entry, it initiates an ARP request. It continues to try to send the packet, and then drops the packet after a defined wait period. When an ARP response is received and the ARP entry is completed using a background process, the ping success rate is 100 percent.

Not Deleted After Marked Incomplete

When adjacency information needs to be changed, the adjacency aging logic removes an entry in two stages:

  • First it changes the status of the entry from complete to incomplete.

    Router#show adjacency
    Protocol  Interface     Address
    IP        Serial0       10.10.10.2(2) (incomplete)
    IP        Serial0       10.10.10.3(7)
    IP        Ethernet0     172.16.81.1(7)
  • Then, at the next one-minute interval, the adjacency walker process "wakes up" and completes the deletion.

    Router#show adjacency
    Protocol  Interface     Address
    IP        Serial0       10.10.10.3(7)
    IP        Ethernet0     172.16.81.1(7)

    In distributed CEF mode, the process on the RP informs the line cards to complete the deletion. This sequence illustrates that a window of up to 60 seconds exists in order for a transient incomplete adjacency to exist.

Known Issues

On a Frame Relay interface, configuring a static map statement prompts CEF to add a host prefix entry to the CEF table. Originally, CEF did not consider whether the PVC was in an "ACTIVE" status before creating the entry. This issue is resolved in Cisco bug ID CSCdr71258 (registered customers only) .

In addition, after attaching to and then removing an interface from a Multiprotocol Label Switching (MPLS) Virtual Private Network (VPN) route forwarding (VRF) instance, CEF sets the adjacency to incomplete. However, the Frame Relay dynamic map entry is not cleared. When the IP address is reapplied, the dynamic mapping still exists. This prevents the adjacency from ever being completed. Issue the clear frame-relay-inarp command when the IP address is removed (for example when the VRF is applied) to avoid this problem. The IP address can then be reapplied, and the adjacency is completed as soon as the dynamic map is recreated.

Related Information

Updated: Jun 07, 2006
Document ID: 17812