This document describes how to adjust/approach the Adjacency allocations on the Policy Feature Card (PFC)-3A, 3B, 3BXL, 3C, and 3CXL modules for the Cisco 7600 Series routers. This feature is included in Image 15.2(2)S and later.
Nodes in a network are said to be adjacent if they are reachable with a single hop across a link layer. Two main components of Cisco Express Forwarding (CEF) operation are the Forwarding Information Base (FIB) and the adjacency table. In addition to the FIB, CEF uses adjacency tables in order to prepend Layer 2 addressing information. The adjacency table maintains Layer 2 next-hop addresses for all FIB entries. The adjacency table is formed with the help of hardware Ternary Content Addressable Memory (TCAM) called as Adjacency CAM. Each time an adjacency entry is created (such as through Address Resolution Protocol (ARP), and Switched Virtual Circuit (SVC) in case of ATM), a link-layer header for that adjacent node is precomputed and stored in the adjacency table. Once a route is determined, it points to a next hop and corresponding adjacency entry. It is subsequently used for encapsulation during CEF switching of packets.
Adjacency Allocation Scheme
Both unicast and multicast adjacency share the same space on an adjacency table, but the adjacency entry allocation scheme is different for unicast and multicast streams. There are two regions in CAM table called the stat and non-stat region. 7600 Series routers support up to one million adjacencies, in which the first 512,000 support statistics reports and the rest are non-statistics support. When the 512,000 has gone to the stats region, new adjacencies are created from the non-stats region.
Adjacency Entry Allocation for Unicast
The adjacency driver code searches all of the regions in stats/non-stats for adjacency entry allocation for unicast streams. The adjacency driver code returns any available adjacency entry in the entire adjacency table. The search typically progresses from low regions (stats region) to higher address (non-stats) regions.
Adjacency Entry Allocation for Multicast
The multicast application selects a region based on multicast (S,G) hashing. There are a total of 32 regions. The adjacency driver code searches for a free adjacency entry in the selected region and returns one if available. If there are no free adjacency entries in that particular region, the driver code does not return any adjacency entry to the multicast application from other regions even if adjacency entries are available in other regions.
Here is a diagram of the software architecture.
Issues with Current Design
This section lists some issues with the current design.
In a typical router boot up, first unicast entries are allocated. Once unicast routes are established, multicast routes are built and entries allocated.
When a multicast application requests an adjacency entry from a particular region based on S,G hashing, sometimes the adjacency driver code cannot allocate an entry if that region is already full due to unicast entries. The driver code does not allocate adjacency entries from other regions even if adjacency entries are available in other regions.
Customers face this issue when unicast routes comes up first, which usually is the case, and end up completely occupying the first few regions. Later when multicast comes up and adjacency entries are required from the initial regions based on S,G hashing, entries might not be available for multicast streams.
A common problem encountered is that the adjacency driver code is not able to allocate an entry if that region is already full due to unicast or multicast entries. The router prints this error:
%MMLS-SP-STDBY-6-ADJ_ALLOC_FAILURE: Failed to allocate Adjacency for the multicast flow (S, G) in region 31
%CONST_MFIB_LC-SP-STDBY-6-FIB_MCAST_INSTALL_FAILURE: Failed to install FIB multicast entry (table:21, S:x, G:y)
Router#show mls cef adjacency usage
Adjacency Table Size: 1048576 ACL region usage: 3 Non-stats region usage: 101110 Stats region usage: 495778 <<<<< adjacencies entries are being exhausted Total adjacency usage: 596891
When the unicast application requests the adjacency entry in STATS PREFERRED mode, the adjacency driver code allocates an adjacency entry from the stats region if available. The multicast application must allocate from the STATS REQUIRED region because the multicast entries are dependent on the liveliness of the flow
Users may observe that the adjacency entries were not available for some of the multicast streams in a high-scale network even if enough adjacency entries are available in the stats regions (the first 32 regions).
If users scale up unicast routes, they see the issue of the adjacency entry being unavailable for most of the multicast streams.
Solution - New Adjacency Entry Allocation Scheme for 7600 Series
An enhancement supplies a fix that facilitates the ability to reserve the space for multicast adjacencies.
In order to enable the enhancement fix, enter this command:
Router(config)#mls cef adjacency-mcast ?
<1-16> number of 1K entries region Configure per region adjacency entries for mcast
As per above command we can reserve the space for multicast adjacencies in two ways.
A: Reserve common space across all stat regions for multicast adjacencies:
Router(config)#mls cef adjacency-mcast <1K - 16K>
Unicast adjacencies cannot use these reserved space even in absence of multicast adjacencies.
B: Reserve space in particular region for multicast adjacencies:
Router(config)#mls cef adjacency-mcast region <region#1 - region#31> <1K - 16K>
If multicast consumes the reserved space in particular region, it can still use non-reserved space also, if availability is there.
Note: a) Region # 0 is left out of this reservation scheme. b) For region # 31 maximum reservation possible is only of 13K. For all other feasible regions, possible limit is 16K.