ATM and Layer 3 Troubleshooting Guide, 12.0(13)W5(19)
Troubleshooting Layer 3 Network Connections

Table of Contents

Troubleshooting Layer 3 Network Connections
Overview of Layer 3 Switching
System Architecture
Troubleshooting Half- or Full-Duplex Negotiation
Troubleshooting IP Layer 3 Connections
Troubleshooting IPX Layer 3 Routing
Troubleshooting Layer 3 IP Multicast Switching
Troubleshooting IP and IPX Load Balancing
Troubleshooting Route Processor Route Table and Utilization Problems
Troubleshooting SDM Problems

Troubleshooting Layer 3 Network Connections


This chapter provides troubleshooting information about connectivity and performance problems in the Layer 3 network connections of the switch router.

The chapter includes the following sections:

Overview of Layer 3 Switching

This section provides an overview of Layer 3 switching using the switch router. It shows how a switch router fits into the network, the architecture of the switch router, and the course of a Layer 2 and Layer 3 packet through the switch router. Also included is a list of Layer 3 switching software features with brief descriptions of selected features.

Defining Layer 3 Switching

Layer 3 switching refers to a class of high-performance switch routers optimized for the campus LAN or intranet, providing both wirespeed Ethernet routing and switching services.

A Layer 3 switch router performs the following three major functions:

  • Packet switching
  • Route processing
  • Intelligent network services

Compared to other routers, Layer 3 switch routers process more packets faster by using application-specific integrated circuit (ASIC) hardware instead of microprocessor-based engines. Layer 3 switch routers also improve network performance with two software functions—route processing and intelligent network services.

To simplify forwarding of the IP packets, route processing is usually executed during the initial call or session setup. At that point, the Layer 3 enabled ATM switch router determines the appropriate route, and forwards to the interfaces information describing the path to be used. In fact, data exchanged between the communicating source and destination end nodes may never need to flow to or through a conventional router.

Frame forwarding on subsequent packets in the same flow is performed using the Layer 3 switch functions at the line card. Once the route has been determined, all subsequent frames in the flow are simply switched or forwarded across the chosen path. This takes advantage of the high throughput and low latency characteristics of switching by enabling the traffic to bypass the route processor once a path calculation has been performed.

Understanding Packet Flow

Figure 11-1 shows and describes, in Steps 1 through 4, how the initial packet travels through the switch Layer 3 route processor to set up the network route.


Note   When making Layer 3 switching decisions, the route processor does not reference the switch fabric, (that is, the PVC configuration). The interface map (where the switch maps an egress interface to a Broute VC) is programmed when the switch is booted up. At that time, the PVCs are automatically configured.


Figure 11-1   Phase 1 — Layer 3 Packet Flow


Figure 11-2 shows and describes, in Steps 5 through 7, how the route processor sends the ARP and propagates the updated routing tables to the interfaces.


Note   In Figure 11-2, the ARP requests are described only for illustration purposes. In most cases, if you are running a dynamic protocol, the switches will have already sent and received ARP packets, and built the route tables.


Figure 11-2   Phase 2 — Layer 3 Packet Flow


Figure 11-3 shows and describes, in Steps 8 and 9, how subsequent packets sent by Host A, to Host B, are switched without the help of the route processor.


Figure 11-3   Phase 3 — Layer 3 Packet Flow


Layer 3 Forwarding

By using CEF, each of the line cards maintains a Forwarding Information Base (FIB) table downloaded from the switch processor. Any changes made to the route processor routing table, caused by additions or deletions of routes or route flaps, are updated in the central FIB, which in turn updates the line card FIBs. This means that, at all times, all line cards have a correct map of the network topology.

Packet switching in the Layer 3 enabled ATM switch router takes place as follows:


Step 1   A packet is received at the physical interface. The CEFA ASIC provides the MAC-layer functions, and the packet is stored in internal memory.

Step 2   As soon as the first 64 bytes of the frame are read, the microcode running on the microcontroller reads the source and destination IP addresses, or IPX network information. If the destination MAC address belongs to the switch router, the packet is routed. If not, it is bridged.

Step 3   The destination IP address information is used by the search engine to begin a lookup, in the CAM table, for the longest match entry.

Step 4   The destination network is matched within 64 clocks (or approximately 2.5 microseconds). The match is returned to the microcontroller, which in turn moves the frame from the internal memory to the Fabric Interface frame FIFO buffer. At the same time, the search engine returns relevant information such as quality of service (QoS) classifications, and MAC header rewrite information, to the Control FIFO buffer.

Step 5   Packet rewrite and QoS classifications take place at the input Ethernet processor interface or Cisco Express Forwarding ASIC (CEFA).

Step 6   The VPI and VCI are attached at the beginning of the packet. The VPI and VCI used corresponds to the particular QoS being requested. The packet then goes through the SAR (Segmentation and Reassembly), which segments the packet into 48-byte payloads. The previously retrieved VPI and VCI-value is written into the cell header to complete the 53-byte ATM Cell.

Step 7   As soon as the entire frame is received into the Frame FIFO buffer, the frame moves into the shared fabric and is stored with a pointer to the output port.

Step 8   If that output interface is currently busy transmitting a frame, the scheduler uses WRR to determine which packet should be sent next.

Step 9   The destination port is signaled, by the switching-fabric ASIC, to take the frame out of a known memory location. The destination port knows that it is receiving the correct frame because of the internal routing tag corresponding to a particular, internal, port-to-port circuit.

Step 10   The frame is sent out to the network.





Layer 2 Bridging

When a port or group of ports are running in bridging mode, the search engine initiates a lookup, in the CAM table, based on the Layer 2 MAC address. Because the Layer 3 enabled ATM switch router is a distributed switching system, each port (or in this case, CEFA) maintains a list of addresses and ports of exit that are of local significance. For example, if Address A is a destination learned on interface FastEthernet 0/0/1, the remaining interfaces on the switch do not have to have that address stored in their CAM tables unless they have a packet to send to Address A.

If the destination MAC address is a broadcast address (FFFF.FFFF.FFFF), the packet is tagged with the destination set as all ports in that bridge group, and it is sent out to the switching fabric. The fabric ASIC creates a pointer from that point in the memory to all ports in that bridge group. For example, if there were eight ports in a bridge group, all eight ports would receive that broadcast.

How MAC Addresses are Learned by the Switch

The following steps describe the MAC address learning process used by the switch router.


Step 1   When a port receives a packet with an unknown source and destination MAC address, it stores the source address as "locally learned" and forwards the packet, as an "unknown unicast," to all ports in the bridge group (similar to a broadcast).

Step 2   The receiving port also sends a LightStream InterProcess Communication (LSIPC) message to the route processor to allow it to update the bridging table on the route processor.


Note   This bridging table in the route processor is only used to allow you check the learned MAC addresses using the show bridge command.

Step 3   All ports in the bridge group receive a copy of the "unknown unicast" and forward the packet.

Step 4   The receiving ports learn the new source address of the packet as a "remote entry."

Step 5   These receiving ports determine which interface sent the packet, based on the VPI and VCI header that points to a P2MP leaf, and the port already knows the corresponding P2MP root.

Step 6   Now all ports in the bridge port have learned the new source MAC address.

Step 7   The destination station for that frame responds.

Step 8   The port that receives the response learns the MAC address of the destination station (now the source address in the response). It has already learned the destination address, allowing it to forward the packet to the correct port.

Step 9   Only that egress port will then learn the new source address.

Step 10   The route processor is also notified of the new destination station source MAC address.

Step 11   Layer 2 switching then occurs between the two ports.






Note   After 5 minutes of inactivity, MAC addresses are deleted from the CAM. The port sends another message to the route processor to remove the MAC from the bridging table.

After both the source and the destination MAC address have been learned, the following procedure occurs during Layer 2 frame switching:


Step 1   A packet is received at the physical interface. The CEFA ASIC provides the MAC-layer functions, and the packet is stored in internal memory.

Step 2   As soon as the first 64 bytes of the frame are read, the microcode running on the microcontroller reads the MAC source and destination addresses. If the destination MAC address is not that of the interface, Layer 2 switching is required. This information can now be used by the search engine.

Step 3   Because the packet has been received on a particular VLAN, the search engine begins a search for the MAC address and its corresponding port of exit.

Step 4   The destination MAC address is found. The microcontroller moves the frame from the internal memory to the switching fabric. At the same time, the search engine returns relevant information such as QoS classifications or ISL information to the switching fabric.

Step 5   The VPI and VCI are attached at the beginning of the packet. The VPI and VCI that are used correspond to the particular quality of service being requested, the appropriate port of exit. The packet then goes through the SAR (Segmentation and Reassembly), which segments the packet into 48-byte payloads. The previously retrieved VPI and VCI values are written into the cell header to complete the 53-byte ATM Cell.

Step 6   The frame moves into the shared fabric and is stored sequentially.

Step 7   The destination port is signaled by the switching-fabric ASIC to take the frame out of memory. The destination port knows that it is receiving the correct frame because of the internal routing tag.

Step 8   The frame is re-encapsulated via ISL, if necessary, and sent out to the network.





System Architecture

The best way to understand the architecture of the Layer 3 enabled ATM switch router is to divide the switch into the following three distinct, functional segments:

  • Switch route processor
  • Switch fabric
  • Line cards

The switch route processor engine, show in Figure 11-4, is responsible for all address and route learning and distribution. Because the Layer 3 enabled ATM switch router is designed as a distributed switching system, the route processor (CPU) needs to ensure that all Layer 3 routes and Layer 2 MAC addresses are maintained and the line cards are updated as needed. The route processor is also responsible for handling all system management, including SNMP and remote monitoring (RMON) statistics.


Figure 11-4   High-Level Layer 3 Enabled ATM Switch Router Architecture


The switching fabric or shared memory fabric, show in Figure 11-4, differs for the two Catalyst 8500 CSR switches. The Catalyst 8540 includes 12-MB shared memory while the Catalyst 8510 includes 3-MB of shared-memory. This shared memory is dynamic, meaning that a packet stored in memory takes only as much memory as it needs. Access into and out of the shared memory is dynamically allocated by the direct memory access (DMA) ASIC. Because the switch fabric is nonblocking, it does not require per-port buffers; the fabric speed is faster than the combined speed of all the ports. Congestion, therefore, only occurs when an individual output port is congested.

The line cards, show in Figure 11-4, are designed to carry considerable intelligence for the switching system. Each line card contains ASICs designed to provide input and output into the fabric as well as to maintain a Layer 3 FIB or a Layer 2 MAC address table. These tables allow the Layer 3 enabled ATM switch router to make switching decisions very quickly prior to transmission across the switching fabric. The line cards, therefore, must work closely with the route processor to ensure that all address tables and routing information is current. The line cards are also responsible for presenting a uniform frame to the switching fabric for effective buffering, QoS policy enforcement, and packet switching.

Each of the three main components of the Catalyst 8540 CSR are described in detail in the following sections.

Route Processor

The system route processor is the first element of the Layer 3 enabled ATM switch router architecture and resides at the core of the switch. The route processor resides on the switch route processor (SRP) module, along with the shared memory fabric, described in the "Switching Fabric and Arbitration" section. The route processor for the Catalyst 8510 CSR is a 64-bit 100Mhz R4600 RISC processor. Its architecture is very similar to that of the Cisco 7500 Route Switch Processor (RSP). The route processor for the Catalyst 8540 CSR is a 200Mhz R5000 RISC processor, very similar to the RSP-4 engine. The Layer 3 enabled ATM switch router SRP runs the Cisco IOS Release 12.0 or later software.

Routing Protocols

The route processor is responsible for running all of the routing protocols shown in Table 11-1<Xref_Color> on the Layer 3 enabled ATM switch router. Other protocols, such as AppleTalk, DECNet, and VINES are bridged in the switch.

s

Table 11-1   Supported Routing Protocols

IP Networks IPX Networks AppleTalk Networks

RIP

RIP-2

OSPF

IGRP

EIGRP

BGP

IPX RIP

EIGRP

RTMP

EIGRP

AURP


Note   The Catalyst 8540 CSR is designed to support multiprotocol routing.

Most importantly, the route processor is responsible for maintaining the routing table. By using Cisco Express Forwarding, the route processor creates a FIB, which contains a subset of the routing table. The FIB is based on a topology map of the network, allowing routing to take place via the network topology at high speed. The FIB is then downloaded to the line cards, allowing the line cards to make Layer 3 routing decisions without having to interrupt the route processor. This capability allows the Layer 3 enabled ATM switch router to forward all frames at wire speed for all ports. The FIB and Cisco Express Forwarding are also described in the "Line Card Architecture" section.

The route processor is also responsible for maintaining state information regarding multicast routing. The Layer 3 enabled ATM switch router supports PIM (sparse mode and dense mode) as well as Distance Vector Multicast Routing Protocol (DVMRP) interoperability. The route processor is responsible for responding to and forwarding joins and leaves as well as responding to pruning messages sent by PIM. Multicast forwarding takes place at the line card level.

Layer 2 VLAN and Switching

Although the switching decisions are made at the line cards, the route processor is still responsible for maintaining Layer 2 information. The route processor is responsible for bridge group configuration and spanning tree calculation.

Bridge groups are configured on the Layer 3 enabled ATM switch router in the same way they are in other Cisco routers. Instead of routing traffic to an outgoing interface, the traffic is bridged via its Layer 2 address. Integrated Routing and Bridging (IRB) is also supported in the Layer 3 enabled ATM switch router in order to support both bridging and routing at the same time.

Spanning tree information within the switch is maintained by the route processor. This includes calculation of the root bridge, optimum path determination to the root, and determining the forwarding and blocking links.

Cisco Express Forwarding

Cisco Express Forwarding (CEF) evolved to best accommodate the changing network dynamics and traffic characteristics resulting from increasing numbers of short-duration flows typically associated with Web-based applications and interactive multimedia sessions. Other Layer 3 switching paradigms use a route-cache model to maintain a fast lookup table for destination network prefixes (see Figure 11-5). The route-cache entries are traffic driven, in that the first packet to a new destination is routed via routing table information, and as part of that forwarding operation, a route-cache entry for that destination is added. This process allows subsequent packet flows to that same destination network to be switched based on a route-cache match. These entries are periodically aged out to keep the route cache current and can be immediately invalidated if the network topology changes.


Figure 11-5   Route-Cache and Distributed Routing Comparison


This "demand-caching" scheme used by other Layer 3 switches is optimized for networks where the majority of traffic flows are associated with a subset of destinations. Since the traffic profiles at the core of the Internet (and potentially within some large enterprise networks) no longer resemble this model, CEF was introduced. CEF eliminates the increasing cache maintenance problem resulting from growing numbers of topologically dispersed destinations and dynamic network changes.

CEF avoids the potential overhead of continuous cache churn by using a FIB on the line card for the destination switching decision. The FIB mirrors the entire contents of the IP and IPX routing table. This means that there is a one-to-one correspondence between FIB table entries and routing table prefixes; therefore, a route cache does not need to be maintained.


Note   Although CEF has been specified for IP, it also applies to IPX as well.

CEF Operation

CEF provides features comparable to fast switching, including load sharing, recursive route resolution, and access lists. CEF uses two tables maintained in the SRP and downloaded to the line cards: the FIB and adjacency table. The FIB table is used for making forwarding decisions. The adjacency table maintains the adjacent nodes, and the link-layer information (such as packet rewrite information) necessary to reach that adjacent node. Every entry in the FIB table has a pointer to a corresponding entry in the adjacency table shown in Figure 11-6.


Figure 11-6   FIB and Adjacency Table


The FIB table is populated by callbacks (inputs) from the routing table. After a route is resolved, it points to a next hop, which should be an adjacency. This step is done at the SRP and then downloaded to the line cards, allowing the line cards to maintain a current topology of the network, which enables rapid switching decisions (within 10 ms) as well as fast convergence in the event of a routing topology change. The FIB is modified when a route is added, removed, or changed in the routing table. This information is immediately downloaded to the line cards.

The adjacency table is also populated by callbacks from the routing protocols, which include information such as next-hop information and (source, group [S,G]) interfaces for multicast groups. Adjacencies are added when a protocol detects that there is an adjacent node via the routing protocol. When a packet arrives at the ingress port, the CEF ASIC performs a FIB lookup based on the destination IP address. The matching FIB entry points to an adjacency entry, which in turn provides the valid link layer rewrite and outgoing interface. The packet is forwarded based on this information. Figure 11-6 shows the relation of the FIB to the adjacency table.

Switching Fabric and Arbitration

The Catalyst 8540 and Catalyst 8510 CSRs have different shared-memory architecture and system bandwidth. The Catalyst 8540 is based on a 12-MB shared-memory architecture with a total system bandwidth of 40 Gbps. The Catalyst 8510 is based on a 3-MB shared-memory architecture with a total system bandwidth of 10 Gbps. Both systems shared memory is completely nonblocking, meaning that all input ports have equal and full access into the shared memory for packet switching. The Layer 3 enabled ATM switch router also provides four queues per port, allowing the Frame Scheduler to make intelligent QoS decisions based on the priority of each queue.

In the Catalyst 8540, each line card has 5-Gbps access into the shared memory fabric as shown in Figure 11-7. This bandwidth is also divided into 2.5 Gbps transmit and 2.5 Gbps receive paths into the fabric. This allows for nonblocking switching capacity within the switching system by ensuring that each line card is given more bandwidth than all of the ports on the line card can generate. Each of the line cards in the Catalyst 8510 is allotted 2.5 Gbps of capacity into the fabric. The 2.5-Gbps bandwidth is divided into transmit and receive paths, each of 1.25 Gbps, to ensure that both reads and writes to the shared memory can be accomplished simultaneously.


Figure 11-7   Switching Bandwidth per Slot on Catalyst 8540 CSR


Because the Layer 3 enabled ATM switch router includes nonblocking memory, every port in the switch has full access to every other port. Each packet entering the switch fabric is tagged with an internal routing tag. This routing tag provides the switching fabric with the appropriate port of exit information, the QoS priority queue the packet is to be stored in, and the drop priority, shown in Figure 11-8.


Figure 11-8   Internal Routing Label Format


The 4 byte routing tag contains a 20-bit label value, a 3-bit QoS value, a 1-bit stack indicator, and an 8-bit TTL value.

The Fabric-Switching ASIC (FSA) then queues each packet into memory and creates a pointer, based on the internal routing tag, to the appropriate destination port. The Frame Scheduler is then responsible for scheduling the frame out of memory based on the queue where the packet is being stored.

Each port transmitting through the fabric is, by default, placed in the lowest-priority queue. This places all traffic at a "best-effort" QoS level. When you configure a policy, that traffic is transmitted in the queue corresponding to the specified IP precedence. That queue is granted more service, thereby reducing latency and the possibility that traffic on that queue will be dropped.


Note   All management and control plane traffic, such as BDPU information, routing protocol updates, and management frames are placed in the highest-priority queue for transmission to the route processor.

The Frame Scheduler

The Frame Scheduler has two main responsibilities within the Layer 3 enabled ATM switch router: first, to schedule frames into the switching fabric based on the priority queue being requested, and second, to schedule frames out of the switching fabric based on the Weighted Round Robin (WRR) scheduling algorithm.

At the input to the switching fabric, the CEF ASIC posts a request to the Frame Scheduler for access to the fabric. The Frame Scheduler handles each request in a time-division multiplexing (TDM) fashion, meaning that each CEF ASIC will have the opportunity to clock an entire frame into the fabric when access has been granted. Because each CEF ASIC handles four ports, the Frame Scheduler allows the CEF ASIC to clock in a maximum of four packets into memory (see the "CEFA" section).

Each packet in memory has an internal routing tag added to the beginning which, as mentioned earlier, contains the port of exit, queuing priority, and drop priority. Based on the routing tag, the input Frame Scheduler places the packet in the correct queue (see Figure 11-9).


Figure 11-9   Input Scheduling and Queue Allocation


The "HH," "HL," "LH," and "LL" designations refer to the IP precedence fields used by the Layer 3 enabled ATM switch router to determine the appropriate queue.


Note   Although not shown, a fifth, critical high-priority routing tag is added to the beginning of all management and control plane packets for immediate delivery to the route processor.

On the output side, the Frame Scheduler is responsible for servicing each queue based on the WRR priority scheme. WRR allows the network manager to configure how much service each queue receives. In a situation where there is no congestion, WRR and the weights provided do not play a real part on how packets are switched out of the fabric, because there is plenty of bandwidth available. However, if a link is congested, WRR services each queue per port based on the priority set by the network manager. For example, look at the weights assigned by a network manager in Table 11-1<Xref_Color>.

Table 11-1 Sample WRR Priority Weights

Quality of Service Priority Weight Given by Network Manager Bandwidth Assignment Calculation Bandwidth Assigned

QoS-0

8

=(8/(8+4+2+1)) x 100

53 Mbps

QoS-1

4

=(4/(8+4+2+1)) x 100

27 Mbps

QoS-2

2

=(2/(8+4+2+1)) x 100

13 Mbps

QoS-3

1

=(1/(8+4+2+1)) x 100

7 Mbps

Based on the priorities and weights provided, the Frame Scheduler services QoS-0 more often, granting queue 53 Mbps out of the 100 Mbps possible on the output link. The second queue, QoS-1, receives 27 Mbps of the bandwidth, and so forth. These commands are set globally on the switch router and function the same for all ports on the switch.

The switch router also allows you to override the global QoS settings by allowing port-to-port communications to have a different level of priority. You have the option of configuring bandwidth based on a source-destination, destination, or source basis and provide weights based on certain IP addresses having more bandwidth then others.


Note   This feature is available with the hardware access list daughter card installed on an Ethernet interface module installed in the Catalyst 8510 CSR.


Figure 11-10   WRR Scheduling and Bandwidth Allocation


Line Card Architecture

The last major component of the Layer 3 enabled ATM switch router architecture is the line cards. Because the switch uses a distributed architecture, the line cards must be intelligent enough to make both Layer 3 and Layer 2 forwarding decisions at wire speed for all media types, as well as enforce QoS policies. Figure 11-11 details the architecture of the Layer 3 enabled ATM switch router line cards. In Figure 11-11, notice that the Catalyst 8540 uses four CEFAs per line card.

The Layer 3 enabled ATM switch router line cards are based on the Cisco Express Forwarding ASIC (CEFA). The CEF ASIC is based on the MMC Ethernet processor interface ASIC. It is called the CEF ASIC since the Cisco Express Forwarding mechanism is programmed into the ASICs. This ASIC is responsible for the Ethernet MAC layer functions, address or network lookup in the content-addressable memory (CAM) table, and forwarding of the packet with its correct rewrite information to the Fabric Interface. The Fabric Interface is also resident on the line card and is responsible for the packet rewrite, QoS classification, and signalling to the Frame Scheduler.


Figure 11-11   Catalyst 8540 CSR Line Card Architecture


CEFA

The CEFA is at the heart of the line card architecture. This ASIC has several key components that will be discussed in detail. Each CEFA services four ports on the line card. In order to service eight ports, two CEFAs are used per line card. On the Catalyst 8540, four CEFAs are used in order to service 16 ports. Although not shown in Figure 11-11, the CEFA is responsible for all MAC layer functions. The MAC is 10/100 autosensing and autonegotiating, if so configured. The MAC can also be run in either full or half duplex mode.

Packets entering the switch port and having passed though MAC functions are stored in an internal block of SRAM. This memory is 8 kilobytes in size, with 2K reserved for command instructions. This memory is used to hold the packet while the appropriate lookups take place.

The CEFA microcontroller is a mini-route processor that is local to four ports on the Layer 3 enabled ATM switch router line module. The microcontroller is designed to handle the traffic on each of the ports in a fair manner. This means the CEFA must ensure that all packets have equal access into internal memory and that lookups via the search engine are done fairly by arbitrating service between the four ports. This is handled in a round-robin manner, meaning that the microcontroller cycles between each port, processing requests as needed.

The microprocessor also has the critically important task of forwarding system messages such as spanning tree BPDUs, routing advertisements, Cisco Discovery Protocol (CDP) packets, Address Resolution Protocol (ARP) frames, and other control-type messages back to the route processor. Those messages are forwarded by the CEFA to the route processor.

CEFA Search Engine

The search engine in the CEFA performs the address lookup or network output interface lookup. It performs its lookup in the CAM table, which can hold either 16,000 or an optional 64,000 entries. The search engine can make two types of switching decisions: Layer 2 based or Layer 3 based. With the hardware-based access list feature card, the search engine can also perform lookups based on Layer 4 information. The search engine is therefore responsible for maintaining the Layer 2 MAC address table and the Layer 3 FIB.

An incoming packet is placed into the internal memory. As soon as the first 64 bytes of the frame are read into memory, the microcode signals the search engine with the relevant source or destination MAC address, destination network, or Layer 4 port information. The search engine can then conduct a lookup in the CAM table for the corresponding entry. Using a binary tree lookup method, the search engine can hit a MAC address or perform a longest match on the destination network address very quickly. The corresponding rewrite information, which is stored in the CAM table, is then delivered to the control FIFO buffer of the Fabric Interface.

Fabric Interface

The final stage in packet switching within the Layer 3 enabled ATM switch router can now occur. The switching CEFA now knows the port-of-exit for the packet based either on its MAC address or on the Layer 3 IP or IPX network numbers. The packet must now be transferred across the switching fabric to the destination. The Fabric Interface is responsible for preparing the packet for its journey across the switching fabric.

The Fabric Interface consists of two main components: the frame FIFO buffer and the control FIFO buffer. Figure 11-11 shows the internal memory of the CEFA, its direct connection into the frame FIFO buffer, and the direct connection from the search engine into the control FIFO buffer. When the search engine completes the lookup, the packet moves from internal memory into the frame FIFO buffer. In parallel, the search engine returns to the control FIFO buffer all of the relevant rewrite and QoS information.

The Fabric Interface then rewrites the packet with the appropriate information and calculates the checksum. At the same time, the Fabric Interface adds to the beginning of the packet the internal routing tag containing port of exit, the QoS priority, and drop priority (see Figure 11-8). Once completed, the Frame Scheduler is signaled to place the frame into the fabric.

At the output port, the Fabric Interface forwards the packet to its output MAC. Since all rewrite and error checking has been done at the ingress port, no additional work needs to be performed on that frame.

Private, Shared, and Dual CAMs

Private CAM describes where each interface has its own CAM. The CAM space is used to store direct lookup tables, Layer 2 and Layer 3 forwarding tables that assist in the ASIC hardware forwarding. See Figure 11-12.

The various CAM types are described as follows:

  • Private CAM
    • Each FastEthernet interface has its own CAM space
    • 1-to-1 ratio between hardware interface and CAM
  • Shared CAM
    • One Ethernet processor interface (4 Ports) share CAM Space
    • 1-to-many ratio between hardware interface and CAM
  • Dual CAM found on current Gigabit Ethernet module
    • One CAM per Ethernet processor interface (2 Ethernet processor interfaces per Gigabit Ethernet processor interface)
    • Many-to-1 ratio between hardware interface and CAM

Figure 11-12   Private CAM


The shared CAM allows one single CAM space per Ethernet processor interface, and this CAM space is physically shared among all four ports within this interface. See Figure 11-13. Shared CAM space has implications in the way the direct lookup table and Layer 3 database are maintained in the CAM.


Note   A shared CAM board and non-shared CAM board can co-exist in the same switch router.


Figure 11-13   Shared CAM


There are always five P2MP VCs in the switch router:

  • One VC for all Gigabit processor interfaces, with two leaves for each Gigabit processor interface.
  • Four P2MP VCs for each Ethernet processor interface, one corresponding to each channel.

With shared CAM, Gigabit processor interface P2MP remains the same. However, for Ethernet processor interfaces with shared CAM, only channel-0 leaf is created. Other channel leaves are not created. This allows a mix of private, shared, and dual CAM interfaces in the switch router.

To determine what type of CAM is installed on your interface use the show hardware detail command as shown in the following example:

Switch# show hardware detail
C8540 named Switch, Date: 10:41:12 UTC Thu Dec 7 2000
Slot Ctrlr-Type Part No. Rev Ser No Mfg Date RMA No. Hw Vrs Tst EEP
---- ------------ ---------- -- -------- --------- -------- ------- --- ---
0/* Super Cam 73-2739-03 D0 03170TAL May 03 99 0 3.1
0/0 8T1 IMA PAM 73-3367-02 B2 03100061 Mar 15 99 00-00-00 2.0 0 0
0/1 8E1 IMA PAM 73-3378-02 B2 03120056 Mar 25 99 00-00-00 2.0 0 2
2/* ARM PAM 73-4208-01 05 03150016 Apr 18 99 1.0
3/* ETHERNET PAM 73-3754-06 B0 03282WBF Jul 13 99 0 5.1
9/* OC48c PAM 73-3745-02 12 03190UXC Jun 28 99 2.1
10/* OCM Board 73-4165-01 04 03230ZZ2 Jun 28 99 10.1
10/0 QUAD 622 Gen 73-2851-05 A0 03160RVS Jun 16 99 5.0
11/* OC48c PAM 73-3745-02 12 03100015 Jun 28 99 2.1
12/* OCM Board 73-4165-01 04 03190UJV Jun 28 99 10.1
12/0 QUAD 622 Gen 73-2851-05 A0 03160S9J Jun 16 99 0 5.0
.
(Information Deleted)
.
slot: 2/* Controller-Type : ARM PAM
Part Number: 73-4208-01 Revision: 05
Serial Number: SCA03150016 Mfg Date: Apr 18 99
RMA Number: H/W Version: 1.0
FPGA Version: 2.3
EPIF Version: 1704 CAM size: 64 KB EPIF Version: 1704 CAM size: 64 KB
Ucode Version: 0.0 CAM Type: Dual
Port Phy Setup
Port 0: DONE GBIC Vendor: No vendor info.
Port 1: DONE GBIC Vendor: No vendor info.
slot: 3/* Controller-Type : ETHERNET PAM
Part Number: 73-3754-06 Revision: B0
Serial Number: CAB03282WBF Mfg Date: Jul 13 99
RMA Number: 0 H/W Version: 5.1
FPGA Version: 3.2
Chip 0 Reset Count: 0
Chip 1 Reset Count: 0
Chip 2 Reset Count: 0
Chip 3 Reset Count: 0
EPIF Version: 1704 CAM size: 16 KB
Ucode Version: 1.0 CAM Type: Private
--More--

In the previous example, the CAM Type field lists the CAM type for the ARM module in slot 2/* as Dual and the CAM type for the Ethernet module in slot 3/* as Private.

Comparing Data Plane and Control Plane Traffic

Data plane traffic is traffic between two endpoints (for example, a host on subnet A communicating with a host on subnet B). This data plane traffic will be typically switched by the Ethernet processor interface or Gigabit processor interface. Control Plane traffic is traffic which is handled by the route processor, typically Layer 2 and Layer 3 protocol updates.

The following is a list of traffic considered to be Control Plane traffic and handled by the route processor:

IP Packet Traffic on the Control Plane

IP packets are sent to the route processor in the following situations:

  • Packets matching the switch router IP address
  • No route found on the line card with "ICMP unreachable" is enabled
  • Packets with TTL=0 after TTL decrement
  • Packets with options set in IP header
  • Packets in or out on the same interface and with ICMP redirect enabled
  • ARP and Reverse ARP packets
  • Certain multicast and broadcast packets (for example, OSFP/EIGRP route updates).
  • RIP broadcasts
  • HSRP hellos
  • DHCP helper
  • Invalid next hop

IPX Packet Traffic on the Control Plane

IPX packets are sent to the route processor in the following situations:

  • Packets matching the switch router IPX address
  • GNS packets
  • Certain broadcast packets (for example, RIP/EIGRP/SAP route updates).
  • Destination node broadcast
  • Invalid next hop

Miscellaneous Packet Traffic on the Control Plane

The following packets are sent to the route processor on the control plane:

  • SNMP Queries
  • BPDUs
  • Layer 2 Learning
  • CAM Entry Overflows

Troubleshooting Half- or Full-Duplex Negotiation

Autonegotiation converges to using the minimum capability of the local interface and the peer interface. For example, if the local interface is capable of full-duplex transmission and the peer interface is only capable of half-duplex transmission, after the local interface performs autonegotiation the interface changes to operate in half-duplex mode.

If the peer interface does not have transmission mode autonegotiation capability, but the local interface has transmission mode autonegotiation capability and the local interface receives no response to its negotiation requests, the local interface changes to operate in half-duplex mode.

To Support half-duplex and full-duplex autonegotiation, your interface must confirm to the following:

  • media type must be UTP
  • Ethernet processor interface version must be C1 (Slicer Register EVER 0x1704)

Other interfaces (10/100Mbps Ethernet processor interface versions less than C1) have a default speed of 100Mbps, full duplex, and are not capable of autonegotiation.

To determine the installed interface version, use the show controllers {fastethernet | gigabitethernet} slot/subslot/port command and find the EVER field under the Slicer registers. The Ever field should be EVER 0x1704 (C1); or if it is not, your interface is not capable of autonegotiation.

Switch# show controllers fastEthernet 3/0/0
IF Name: FastEthernet3/0/0
Port Status UP
Loopback Reg [3-0]|[7-4]: 0x8|0x8
Duplex/Speed Reg [3-0]|[7-4]: 0xFFF7|0x0
FPGA Rev : 3.8
Internal Reset Trigger Count: 0
Slicer registers
SMDR 0x0060 (Tx En,Rx En)
SSTR 0x1000
EVER 0x1704 (C1)
SSMR 0x4000 SIMR 0x0000 MBXW 0x0000 MBXR 0x0000
SPER 0xF000 GMUX VER 0xF000 MARKER 0x0000
.
(Information Deleted)
.

Half- and Full-Duplex Troubleshooting Commands

To troubleshoot half- and full-duplex negotiation problem, use the following commands:

Command Purpose

show interfaces {fastethernet | gigabitethernet} slot/subslot/port

Displays interface configuration, status, and statistics.

show controllers {fastethernet | gigabitethernet} slot/subslot/port

Displays controller status for the specified interface.

Follow these steps to troubleshoot the half- and full-duplex negotiation problem on an interface:


Step 1   Use the show interfaces fastEthernet card/subcard/port command to check the half-duplex and full-duplex autonegotiation configuration.

Switch# show interfaces fastEthernet 3/0/0
FastEthernet3/0/0 is up, line protocol is up
Hardware is epif_port, address is 0090.2156.d837 (bia 0090.2156.d837)
Internet address is 172.20.52.36/27
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
  Auto-duplex, Auto Speed, 100BaseTX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output never, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 1000 bits/sec, 2 packets/sec
33684 packets input, 11817561 bytes
Received 9 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 abort
0 watchdog, 33546 multicast
0 input packets with dribble condition detected
61232 packets output, 13584791 bytes, 0 underruns(0/0/0)
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Switch#

Step 2   Check the Auto-duplex, Auto Speed, 100BaseTX fields. They should have the following default configuration:

  • Auto-duplex—Auto duplex negotiation
  • Auto-Speed—Auto speed negotiation
  • 100BASE-TX—100-Mbps BASE-TX

If they do not, check the peer interface and determine whether it is capable of this configuration.

Step 3   Use the show controllers fastEthernet card/subcard/port command to check the half-duplex and full-duplex autonegotiation configuration.


Note   The show controllers command for a specific interface has different information depending on the IOS software version running on your Layer 3 enabled ATM switch router.

Step 4   Use the show controllers fastEthernet card/subcard/port command to check the configuration. The following example uses the Cisco IOS Release 12.0(5)W5(13b) and later display:

Switch# show controllers fastEthernet 3/0/0
IF Name: FastEthernet3/0/0
.
(Information Deleted)
.
MII registers:
Control Register (0x0): 0x1000 (Auto negotiation enabled)
Status Register (0x1): 0x782D (Auto negotiation complete)
PHY Identification Register 1 (0x2): 0x7810
PHY Identification Register 2 (0x3): 0x43
Auto Neg. Advertisement Reg (0x4): 0x1E1 (Speed 100 ,Duplex Full )
Auto Neg. Partner Ability Reg (0x5): 0x81 (Speed 100 ,Duplex Half )
Auto Neg. Expansion Register (0x6): 0x0
Mirror Register (0x10): 0x630
Interrupt Enable Register (0x11): 0x0
Interrupt Status Register (0x12): 0x4000
Configuration Register (0x13): 0x0 (UTP, Tx Enabled)
Chip Status Register (0x14): 0x28C9 (Link Up,a-Half,a-100 )
Link Status Register [3-0]|[7-4]: 0x1|0x0
Counters :
.
(Information Deleted)
.

Use the show controllers fastEthernet card/subcard/port command to check the configuration. The following example uses the Cisco IOS Release 12.0(5)W5(13) and earlier display:

Switch# show controller fastEthernet 1/0/0
IF Name: FastEthernet1/0/0
.
(Information Deleted)
.
MII registers:
Control Register               (0x0): 0x1000
Status Register                (0x1): 0x782D
PHY Identification Register 1  (0x2): 0x7810
PHY Identification Register 2  (0x3): 0x43
Auto Neg. Advertisement Reg    (0x4): 0x1E1
Auto Neg. Partner Ability Reg  (0x5): 0x1E1
Auto Neg. Expansion Register   (0x6): 0x1
Mirror Register               (0x10): 0x30
Interrupt Enable Register     (0x11): 0x0
Interrupt Status Register     (0x12): 0x4000
Configuration Register        (0x13): 0x0
Chip Status Register          (0x14): 0x38C8
Link Status Register     [3-0]|[7-4]: 0x1|0x0
.
(Information Deleted)
.

Step 5   Check the Auto Neg. Advertisement Register (Reg 0x4). If it is set to 1, the following are the capabilities:

  • Bit 8 - 100 BASE-TX Full Duplex
  • Bit 7 - 100 BASE-TX
  • Bit 6 - 10 BASE-T Full Duplex
  • Bit 5 - 10 BASE-T

Step 6   Check the Auto Neg. Partner Ability Reg (Reg 0x5). If it is set to 1, the following are the status and capabilities:

  • Bit 14 - Link Partner has received the Link code word from the local
  • Bit 13 - Remote Fault
  • Bit 8 - 100 BASE-TX Full Duplex