Guest

Cisco 12000 Series Routers

Cisco 12000 Series Internet Router Architecture: Line Card Design

Cisco - Cisco 12000 Series Internet Router Architecture: Line Card Design

Document ID: 47242

Updated: Jul 07, 2005

   Print

Introduction

This document provides an overview of the Cisco 12000 Series Internet Router line card design.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on the following hardware:

  • Cisco 12000 Series Internet Router

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Fundamental Major Operations

The Cisco 12000 Series Internet Router has a truly distributed architecture in that all line cards (LCs) run a copy of the Cisco IOS® software image, and all switching is done on the LCs. Cisco Express Forwarding switching is the ONLY switching path. There is no fast switching, optimum switching, and so on as found on other platforms such as the 7500 series. For an overview of the non-distributed switching paths available in the different platforms, see How to Choose the Best Router Switching Path for Your Network.

The packet forwarding functions are performed by each line card. A copy of the forwarding tables computed by the Gigabit Route Processor (GRP) is distributed to each line card in the system. Each line card performs independent lookup of a destination address for each datagram received on a local copy of the forwarding table, and the datagram is switched across a crossbar switch fabric to the destination line card. The basic functions of the LCs are IP/Multiprotocol Label Switching (MPLS) forwarding, ping response, and packet fragmentation.

The line card takes care of:

Before going any further with line card architecture, it is important to understand the specific Cisco 12000 operations. They can be divided into the following categories:

  • Path determination

  • Cisco Express Forwarding

  • Quality of Service (QoS), such as Congestion Management

Path Determination

The path determination process for the Cisco 12000 involves the following activities:

  • Processing internal routing protocols such as Enhanced Interior Gateway Routing Protocol (EIGRP), Intermediate System-to-Intermediate System (IS-IS), Open Shortest Path First (OSPF)

  • Processing external gateway protocol, such as Border Gateway Protocol (BGP)

  • Issuing and responding to routing updates

  • Creating and maintaining the routing table

  • Resolving recursive routes

  • Sending updates to the forwarding tables

Before the 12000 can forward any IP datagrams, the GRP must build a local routing table. This routing table contains the next hop information for the incoming IP packet.

The GRP builds and maintains the routing table by processing the interior routing protocols such as Enhanced Interior Gateway Routing Protocol (EIGRP), Intermediate System-to-Intermediate System (ISIS), Open Shortest Path First (OSPF), and Border Gateway Protocol (BGP).

This table contains all the route entries and metrics (for example, path length) necessary to forward an IP packet. Additionally, the GRP calculates all of the recursive routes that occur when support is provided for both an interior protocol and an external gateway protocol such as BGP. The GRP and the line cards use a new distributed switching method called distributed Cisco Express Forwarding (dCEF). With this distributed switching method, packet forwarding, including the precalculated recursive route information, is sent to each line card.

Cisco Express Forwarding

To learn more about Cisco Express Forwarding, see Understanding Cisco Express Forwarding on the Cisco 12000 Series Internet Router.

Line Card Architecture

There are different kinds of line card architecture based on the Engine type. The figure below shows a generic common diagram for all LCs:

Line Card Diagram

lcdesign1.gif

Each LC can be divided into three major sections:

  • Physical Layer Interface Module (PLIM) - This is the hardware module that terminates the physical connection (media dependent; therefore, Asynchronous Transfer Mode (ATM), Packet-over-SONET (POS), and Fast Ethernet)

  • L3 Switching Engine - This forwarding engine actually prepares packets for transmission across the switching fabric to the destination LC. It handles L3 lookups, rewrites, buffering, congestion control, and all L3, QoS features. Five types of packet forwarding engines exist, namely, engines 0, 1, 2, 3 and 4. Line cards as of this writing are classified by the packet forwarding engine type described in the table below.

  • Fabric Interface - The Fabric Interface ASIC (FIA) prepares the packets for transmission across the switching fabric to the destination LC. It takes care of fabric grant requests, fabric queuing, per-slot multicast replication, and so on.

The Cisco 12000 Series offers an extensive portfolio of line cards, including Core, Edge, Channelized Edge, Asynchronous Transfer Mode (ATM), Ethernet, Dynamic Packet Transport (DPT), and End-of-Sale line cards. These line cards deliver high performance, guaranteed priority packet delivery and service, and transparent online insertion and removal (OIR) through the Cisco 12000 Series distributed system architecture. The following tables list the released line cards as of December 2001 with the corresponding Engine type:

Core Line Cards

Line Card Name Engine Chassis Supported IOS Release Resources
1-Port OC-48 POS ISE One-Port OC-48c/STM -16c POS/SDH ISE Line Card Engine 3 (ISE) 10G Chassis 2.5G Chassis 12.0(21)S 12.0(21)ST Datasheet
1-Port OC-48 POS One-Port OC-48c/STM-16c POS/SDH Line Card Engine 2 10G Chassis 2.5G Chassis 12.0(10)S 12.0(11)ST Datasheet
4-Port OC-48 POS Four-Port OC-48c/STM-16c POS/SDH Line Card Engine 4 10G Chassis only 12.0(15)S 12.0(17)ST Datasheet
1-Port OC-192 POS One-Port OC-192c/STM-64c POS/SDH Line Card Engine 4 10G Chassis only 12.0(15)S 12.0(17)ST Datasheet

Edge Line Cards

Line Card Name Engine Chassis Supported IOS Release Resources
6-Port DS3 Six-Port DS3 Line Card Engine 0 10G Chassis 2.5G Chassis 12.0(10)S 12.0(11)ST Datasheet
12-Port DS3 Twelve-Port DS3 Line Card Engine 0 10G Chassis 2.5G Chassis 12.0(10)S 12.0(11)ST Datasheet
6-Port E3 Six-Port E3 Line Card Engine 0 10G Chassis 2.5G Chassis 12.0(15)S 12.0(16)ST Datasheet (pdf version)
12-Port E3 Twelve-Port E3 Line Card Engine 0 10G Chassis 2.5G Chassis 12.0(15)S 12.0(16)ST Datasheet (pdf version)
4-Port OC-3 POS Four-Port OC-3c/STM-1c POS/SDH Line Card Engine 0 10G Chassis 2.5G Chassis 12.0(05)S 12.0(11)ST
8-Port OC-3 POS Eight-Port OC-3c/STM-1c POS/SDH Line Card Engine 2 10G Chassis 2.5G Chassis 12.0(10)S 12.0(11)ST Datasheet
16-Port OC-3 POS Sixteen-Port OC-3c/STM-1c POS/SDH Line Card Engine 2 10G Chassis 2.5G Chassis 12.0(10)S 12.0(11)ST Datasheet
16-Port OC-3 POS ISE Sixteen-Port OC-3c/STM-1c POS/SDH ISE Engine 3 (ISE) 10G Chassis 2.5G Chassis 12.0(21)S 12.0(21)ST Datasheet
1-Port OC-12 POS One-Port OC-12c/STM-4c POS/SDH Line Card Engine 0 10G Chassis 2.5G Chassis 12.0(10)S 12.0(11)ST Datasheet
4-Port OC-12 POS Four-Port OC-12c/STM-4c POS/SDH Line Card Engine 2 10G Chassis 2.5G Chassis 12.0(10)S 12.0(11)ST Datasheet
4-Port OC-12 POS ISE Four-Port OC-12c/STM-4c POS/SDH ISE Line Card Engine 3 (ISE) 10G Chassis 2.5G Chassis 12.0(21)S 12.0(21)ST Datasheet
1-Port OC-48 POS ISE One-Port OC-48c/STM -16c POS/SDH ISE Line Card Engine 3 (ISE) 10G Chassis 2.5G Chassis 12.0(21)S 12.0(21)ST Datasheet

Channelized Edge Line Cards

Line Card Name Engine Chassis Supported IOS Release Resources
2-Port CHOC-3, DS1/E1 Two-Port Channelized OC-3/STM-1(DS1/E1) Line Card Engine 0 10G Chassis 2.5G Chassis 12.0(17)S 12.0(17)ST Datasheet
1-Port CHOC-12, DS3 One-Port Channelized OC-12 (DS3) Line Card Engine 0 10G Chassis 2.5G Chassis 12.0(05)S 12.0(11)ST Datasheet
1-Port CHOC-12, OC-3 One-Port Channelized OC-12/STM-4 (OC-3/STM-1) Line Card Engine 0 10G Chassis 2.5G Chassis 12.0(05)S 12.0(11)ST Datasheet
4-Port CHOC-12 ISE Four-Port Channelized OC-12/STM-4 (DS3/E3, OC-3c/STM-1c) POS/SDH ISE Engine 3 (ISE) 10G Chassis 2.5G Chassis 12.0(21)S 12.0(21)ST Datasheet
1-Port CHOC-48 ISE One-Port Channelized OC-48/STM-16 (DS3/E3, OC-3c/STM-1c, OC-12c/STM-4c) POS/SDH ISE Line Card Engine 3 (ISE) 10G Chassis 2.5G Chassis 12.0(21)S 12.0(21)ST Datasheet
6-Port Ch T3 Six-Port Channelized T3 (T1) Line Card Engine 0 10G Chassis 2.5G Chassis 12.0(14)S 12.0(14)ST

Asynchronous Transfer Mode (ATM) Line Cards

Line Card Name Engine Chassis Supported IOS Release Resources
4-Port OC-3 ATM Four-Port OC-3c/STM-1c ATM Engine 0 10G Chassis 2.5G Chassis 12.0(5)S 12.0(11)ST Datasheet
1-Port OC-12 ATM One-Port OC-12c/STM-4c ATM Engine 0 10G Chassis 2.5G Chassis 12.0(7)S 12.0(11)ST Datasheet
4-Port OC-12 ATM Four-Port OC-12c/STM-4c ATM Line Card Engine 2 10G Chassis 2.5G Chassis 12.0(13)S 12.0(14)ST Datasheet

Ethernet Line Cards

Line Card Name Engine Chassis Supported IOS Release Resources
8-Port FE w/ ECC Eight-Port Fast Ethernet Line Card Engine 1 10G Chassis 2.5G Chassis 12.0(10)S 12.0(16)ST Datasheet
1-Port GE w/ ECC One-Port Gigabit Ethernet Line Card Engine 1 10G Chassis 2.5G Chassis 12.0(10)S 12.0(16)ST Datasheet
3-Port GE Three-Port Gigabit Ethernet Line Card Engine 2 10G Chassis 2.5G Chassis 12.0(11)S 12.0(16)ST Datasheet
10-Port GE Ten-Port Gigabit Ethernet Engine 4 w/RX/TX+ /density 10G Chassis 2.5G Chassis 12.0(22)S 12.0(22)ST Datasheet

Dynamic Packet Transport (DPT) Line Cards

Line Card Name Engine Chassis Supported IOS Release Resources
2-Port OC-12 DPT Two-Port OC-12c/STM-4c DPT Engine 1 10G Chassis 2.5G Chassis 12.0(10)S 12.0(11)ST Datasheet Announcement
1-Port OC-48 DPT One-Port OC-48c/STM-16c DPT Engine 2 10G Chassis 2.5G Chassis 12.0(15)S 12.0(16)ST Datasheet Announcement

End of Sale (EOS) Line Cards

The following line cards are no longer sold. They are listed here for your reference only.

Line Card Name Engine Chassis Supported IOS Release
1-Port OC-192c/ STM- 64c Enabler Card One-Port OC-192c/STM-64c POS/Enabler Card Engine 2 10G Chassis 2.5G Chassis 12.0(10)S 12.0(11)ST

You can get all the available datasheets from the Product Literature page.

Note: Engine 3 line cards are capable of performing edge features at line rate.The higher the Layer 3 engine, the more packets get switched in hardware.

The only things that really differentiate one line card from another are the physical layer interface module (PLIM) and the Layer 3 Forwarding Engine. Line cards vary by PLIMs only within the same L3 Forwarding Engine. PLIMs have media-dependent components (for example, the Asynchronous Transfer Mode (ATM) PLIM has a segmentation and reassembly (SAR), and the GigE PLIM has a Media Access Control Application-Specific Integrated Circuit - MAC ASIC), but the theory of the packet path across all PLIMs is very similar. This document concentrates on the Packet Over SONET (POS) PLIM, but useful differences are noted when applicable.

In order to determine the Layer 3 Engine type of a line card, Cisco IOS Software Release 12.0(9)S has added the "L3 Engine" type to the output of the show diag command, as illustrated below:

SLOT 1 (RP/LC 1 ): 1 Port Packet Over SONET OC-12c/STM-4c Single Mode
    MAIN: type 34, 800-2529-02 rev C0 dev 16777215
          HW config: 0x00 SW key: FF-FF-FF
    PCA:  73-2184-04 rev D0 ver 3
          HW version 1.1 S/N CAB0242ADZM
    MBUS: MBUS Agent (1) 73-2146-07 rev B0 dev 0
          HW version 1.2 S/N CAB0236A4LE
          Test hist: 0xFF RMA#: FF-FF-FF RMA hist: 0xFF
    DIAG: Test count: 0xFFFFFFFF Test results: 0xFFFFFFFF
    L3 Engine: 0 - OC12 (622 Mbps) 
    
!--- Engine 0 card.

    MBUS Agent Software version 01.40 (RAM) (ROM version is 02.02)
    Using CAN Bus A
    ROM Monitor version 10.00
    Fabric Downloader version used 13.01 (ROM version is 13.01)
    Primary clock is CSC 1
    Board is analyzed
    Board State is Line Card Enabled (IOS RUN )
    Insertion time: 00:00:11 (2w1d ago)
    DRAM size: 268435456 bytes
    FrFab SDRAM size: 67108864 bytes
    ToFab SDRAM size: 67108864 bytes
    0 crashes since restart

There is a shortcut command that you can use to get the same result, but with only the useful information:

Router#show diag | i (SLOT | Engine) 

... 
SLOT 1  (RP/LC 1 ): 1 port ATM Over SONET OC12c/STM-4c Multi Mode 
  L3 Engine: 0 - OC12 (622 Mbps) 
SLOT 3  (RP/LC 3 ): 3 Port Gigabit Ethernet 
  L3 Engine: 2 - Backbone OC48 (2.5 Gbps) 
...

Cisco now offers five types of L3 engines:

  • Engine 0 - OC12/BMA: The IP/MPLS lookup is done in software by an R5K CPU. This engine uses the legacy Buffer Management ASIC (BMA) which manages packet buffers and segments and reassembles packets for transmission across the switch fabric. The receiving BMA is responsible for receiving packets from the PLIM, segmenting packets into fixed-size cells, and presenting them to the Fabric Interface ASIC (FIA) for transmission across the switching fabric. The transmitting BMA, with help from the FIA, performs the reassembly of the cells arriving from the switch fabric into packets, and hands packets to the PLIM for transmission from the box. Most of the features on this line card are implemented in software.

  • Engine 1 - Salsa/BMA48 (TTM48): This second engine has been improved. First, a new ASIC has been developed in order to perform the IP lookup in hardware. This new ASIC is called Salsa. Only the Media Access Control (MAC) rewrite is done in the software on this engine. The BMA has also been upgraded to get more bandwidth. It is now called the BMA48. There is no MDRR or WRED support for this engine.

    Engine 0 and Engine 1 forwarding engines are shown with their key components in the figure below:

    Engine 0 and Engine 1 Packet Forwarding Engine

    lcdesign2.gif

  • Engine 2 - PSA/TBM/RBM (Perf48): A new ASIC is present on these LCs to improve the way the IP/MPLS lookup is done. The Packet Switching ASIC (PSA) performs a hardware lookup and rewrite for Tag and IP packets. For this purpose, the PSA uses a distilled local copy of the FIB table (show ip psa a.b.c.d). All packet switching on an Engine 2 LC is done in hardware by the PSA. The CPU on the LC is interrupted for a packet forwarding decision only if a feature is configured on the line card that is not supported by the PSA. This PSA table is stored in external memory which is only present on Engine 2 LCs.

    Router#exec slot 11 show controller psa mem
     ========= Line Card (Slot 11) =======
    PLU SDRAM: Size 0x4000000,  Banks 4
    TLU SDRAM: Size 0x4000000,  Banks 4
    PSA SSRAM: Size 0x100000
    

    The packet memory has been increased by default to 256 MB and can reach 512 MB.

    There are also new Rx and Tx Buffer manager ASICs (called RBM and TBM, respectively) which are a key for hardware-based support for Class of Service (CoS) features on this LC: WRED and MDRR are performed in hardware. CAR is not available, but a subset of CAR known as Per-Interface Rate Control (PIRC) can be configured instead. As of Cisco IOS software release 12.0(14)S, Sampled NetFlow is supported on Engine 2 Packet-over-SONET (POS) line cards. The Sampled NetFlow feature allows you to sample one out of "x" IP packets being forwarded to routers, by allowing the user to define the "x" interval with a value between a minimum and maximum. Sampling packets are accounted for in the NetFlow Flow Cache of the router. These sampling packets substantially decrease the CPU utilization needed to account for NetFlow packets by allowing the majority of the packets to be switched faster because they do not need to go through additional NetFlow processing. See Sampled NetFlow for further information.

    As of Cisco IOS software release 12.0(16)S, Sampled NetFlow is supported on 3-Port Gigabit Ethernet line cards.

    As of Cisco IOS software release 12.0(18)S, Sampled NetFlow and 128 access control lists (ACLs) on the PSA can now be configured at the same time on Engine 2 Packet-over-SONET (POS) line cards.

    Everything is switched through the PSA, with the exception of some features that have to go to the local CPU of the LC: the output CAR, packets with Access Lists applied if they don't fit into the PSA restrictions, options/non-transit traffic, multicast packets, IPv6 packets, and so on. Output CAR has been replaced by distributed traffic shaping (DTS) as of Cisco IOS software release 12.0(16)S. More information is available at Distributed Traffic Shaping for Line Cards in the Cisco 12000 Series Internet Router.

    Support for ACLs has been changed in hardware on Engine 2 cards. If you do not want to configure them, you should add the line no access-list hard psa to your configuration.

    Below is a diagram of an Engine 2 forwarding engine and its key components:

    Engine 2 Packet Forwarding Engine

    lcdesign3.gif

  • Engine 3 - Edge Engine: This engine is completely new architecture Layer 3 Engine. It also has OC48 bandwidth, but it integrates some new ASICs in order to improve the forwarding speed with any QoS and ACL features. Engine 3 line cards are capable of performing edge features at line rate.

  • Engine 4 - Backbone OC192: These latest LCs are not supported in the 12008 and 12012 series routers. They support the OC192 line rate.

  • Engine 4+ - Same as Engine 4, except that they support many more features at line rate.

Line Card Installation

Below are links related to the installation and configuration of LCs and LC support for different chassis:

For more information about the types of memory on the line cards, see Memory Present on the Line Cards.

Related Information

Updated: Jul 07, 2005
Document ID: 47242