Guest

Cisco 12000 Series Routers

Understanding the Booting Process on the Cisco 12000 Series Internet Router

Document ID: 40060

Updated: Aug 31, 2006

   Print

Introduction

This document explains the Route Processor (RP) and line card bootup process on the Cisco 12000 Series Internet Router.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on the software and hardware versions below.

  • Cisco 12000 Series Internet Router

  • All versions of Cisco IOS® software that run on this platform

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

The Bootup Process

This is the step-by-step process that explains the RP and line card bootup:

  1. Power on or reload.

    If this is a clean power on, the Maintenance Bus (MBUS) is initialized, and the power supplies provide a 5V line to all the MBUS modules and a 48V line to the RP card. If this is a reload, the 5 VDC line is already applied to the MBUS modules.

    MBUS modules provide an interface to the active RP over the MBUS, and are located physically on these cards:

    • Route Processor (RP)

    • Line cards (LCs)

    • Switch fabric cards (SFCs)

    • Clock scheduler cards (CSCs)

    • Blowers/fans

    • Power supplies

  2. The RP boots ROMMON.

    The RP accesses the bootstrap image loaded in ROM, decompresses it, and runs it from ROM. The RP examines the configuration register. Refer to Virtual Configuration Register Settings for more information.

    1. If you set the configuration register to 0x0, then the RP boots to ROMMON and boots no further.

    2. Otherwise, the RP uses the boot variables to determine the Cisco IOS software image source.

    3. You can issue the show bootvar command in order to see what the boot variables are set to for the next reload.

  3. The RP might boot the bootloader.

    1. The RP loads the appropriate Cisco IOS software image into the Dynamic RAM (DRAM) of the RP.

      1. If the image is sourced from a Trivial File Transfer Protocol (TFTP) source, then the bootloader is loaded first before it retrieves the Cisco IOS software image.

      2. If you set the configuration register to 0x1, then the RP boots the bootloader and boots no further.

      3. Otherwise, the bootloader is not used.

    2. The RP decompresses and then runs the Cisco IOS software image.

  4. RP self discovery.

    1. The RP card discovers itself and its slot information.

      This is an example:

      RP State: IOS STRT --- 
      EV_RP_MBUS_DISCOVERY_SUCCESS
    2. The RP downloads bundled MBUS agent software into the MBUS RAM and generates an internal report.

      RP State: IOS UP --- 
      EV_RP_LOCAL_AGENT_REPORT
    3. The RPs in the chassis usE the MBUS to arbitrate for mastership.

      • One becomes the Active RP, the other becomes the Standby RP.

      • If there is a Performance Route Processor (PRP) and an RP in the same system, then the PRP becomes the Active RP.

    4. If running in Route Processor Redundancy (RPR) mode:

      • Only the Active RP decompresses the Cisco IOS software image and runs it. The Standby RP only loads the uncompressed Cisco IOS software image in DRAM.

      • Only the Active RP decompresses the configuration file stored in non-volatile RAM (NVRAM).

    5. If running in Route Processor Redundancy Plus (RPR+) mode or non-stop forwarding (NSF)/Stateful Switchover mode:

      • Both the Active RP and the Standby RP decompress and run the Cisco IOS software image.

      • Both the Active RP and the Standby RP decompress the configuration file stored in NVRAM.

  5. The fabric cards initialize.

    1. The Active RP selects the primary CSC and the backup CSC.

      • If there is only one CSC, that CSC becomes the primary.

      • If there are two CSCs, the CSC which is in clock sync with the most line cards becomes the primary CSC.

      • All things being equal, CSC1 becomes the primary.

        Note: If there are two CSCs and one fails when the router is up and running, the defective CSC is kept in Admin Shutdown mode and the hw-mod slot xx shut command is enabled in the command-line interface (CLI). If the defective CSC has been replaced with a new non-defective CSC in the same slot where the defective was fuctioning, and if the router is restarted or freshly booted, the CLI still displays in Admin Shutdown mode. You need to configure the no hw-mod slot xx shut command in the configuration mode in order to bring up the replaced CSC. This enables the redundancy.

    2. The Active RP determines the rest of the fabric configuration: quarter bandwidth or full bandwidth, redundant, or non-redundant.

      RP State: IOS UP --- EV_RP_LOCAL_FAB_READY
  6. The line cards initialize.

    1. MBUS initializes.

      1. From the start, all MBUS modules on the line cards receive 5 V from the power supplies that turn on the MBUS modules. The MBUS agents run in ROM to start and then run from RAM.

      2. The Active RP discovers the existence of the line cards through the MBUS.

        1. The RP sends broadcast requests to all possible slots.

        2. All components with MBUS modules respond with their MBUS RAM version.

      3. You can upgrade the line card MBUS ROM with the upgrade mbus-agent-rom slot <x> command.

      4. The MBUS agent turns on the 48V line to the line card.

    2. ROMMON

      1. The ROMMON executes basic tests and intialization.

      2. You can upgrade line card ROMMON with the upgrade rom-monitor slot <x> command.

      3. After the RP reaches the IOS UP state and generates the MBUS agent report, the RP requests the line cards to get their ROM monitor (also known as ROMMON) version:

        ROMVGET --- EV_AGENT_REPORT_POWERED
      4. Once the line cards are powered, they use the ROM monitor to execute basic tests and intialization.

        ROMIGET --- EV_LC_ROM_MON_RESET
      5. The line card ROM generates a report and waits for the fabric downloader.

        FABIWAIT --- EV_LC_ROM_IMAGES_REPORT
    3. The fabric downloader

      1. The Active RP downloads the fabric downloader (also known as the secondary bootstrap program of the line card) serially over the MBUS to each of the line cards.

      2. The line card starts to receive the fabric downloader.

        FABLDNLD --- 
        EV_FAB_DOWNLOADER_DOWNLOAD_STARTABLE
      3. The line card finishes receiving the fabric downloader and loads the fabric downloader into the DRAM memory of the line card.

        FABLSTRT --- 
        EV_FAB_DOWNLOADER_DOWNLOAD_SUCCESS
      4. The line card launches and runs the fabric downloader. The fabric downloader initializes some of the hardware components on the line card to enable it to download the Cisco IOS software image across the switch fabric.

        FABLRUN --- 
        EV_FAB_DOWNLOADER_LAUNCH_SUCCESS
      5. You can upgrade the line card fabric downloader and program it into the Flash card with the upgrade fabric-downloader slot <x> command.

  7. The line cards download Cisco IOS software.

    1. The line card waits to receives the Cisco IOS software image from the RP across the fabric:

      IOS DNLD --- EV_IOS_DOWNLOAD_WAIT_DL_CONFIRM.
    2. The line card confirms that the checksum on the Cisco IOS software image checks out:

      IOS STRT --- EV_IOS_DOWNLOAD_SUCCESS
    3. The RP sends a launch request to the line card and the line card sends a report back to the RP to tell it that it successfully launched.

      IOS UP --- EV_IOS_REPORT
    4. The line card carves the necessary buffers in DRAM and runs the Cisco IOS software image:

      IOS RUN --- EV_BUFF_CARVE_SUCCESS
  8. Cisco Express Forwarding (CEF) syncs and routing processes converge.

    1. CEF on the line cards sync to the RP. You can verify this with the show cef linecard command:

      Router#show cef linecard
      
       Slot      MsgSent    XDRSent Window   LowQ   MedQ HighQ Flags
      
       2             886       1769   2495      0      0     0  up
      
       4             878       1764   2495      0      0     0  up
      
       5             882       1768   2495      0      0     0  up
      
       6             874       1759   2495      0      0     0  up
      
       
      
      VRF Default, version 1027, 37 routes
      
       Slot   Version    CEF-XDR   I/Fs State    Flags
      
       2         1018         40     12 Active   sync, table-up
      
       4         1018         40      9 Active   sync, table-up
      
       5         1018         40      9 Active   sync, table-up
      
       6         1018         40     10 Active   sync, table-up
    2. Links transition to UP/UP.

      Router#show ip interface brief 
      
      Interface              IP-Address      OK? Method Status               Protocol
      
      POS2/0                 137.40.9.1     YES   NVRAM  up                    up
      
      POS2/1                 137.40.18.1    YES   NVRAM  up                    up 
      
      POS2/2                 137.40.11.1    YES   NVRAM  up                    up
      
      POS2/3                 137.40.12.2    YES   NVRAM  up                    up      
      
      GigabitEthernet4/0     137.40.199.2   YES   NVRAM  up                    up
      
      GigabitEthernet5/0     137.40.42.2    YES   NVRAM  up                    up
      
      ATM6/0                 unassigned     YES   NVRAM  administratively down down
      
      Loopback0              137.39.39.4    YES   NVRAM  up                    up      
      
      Ethernet0              10.11.11.4     YES   NVRAM  up                    up
    3. Interior Gateway Protocol (IGP) and Border Gateway Protocol (BGP) peers are established:

      1. The RP advertises and receives routes.

      2. The RP updates the routing information database (RIB) and builds the CEF table.

      3. The RP uses Interprocess Commications Protocol (IPC) to download the CEF table to all the synchronized line cards in the show cef linecard output.

      4. BGP converges.

States and Events

The previous section describes the normal states that you see when the RP or line card boots up. This section describes additional states that you might come across when you examine the bootup process of your line cards:

service upgrade all

The fabric downloader always needs to be launched in order for the line card to always go through this state:

FABLRUN --- EV_FAB_DOWNLOADER_LAUNCH_SUCCESS

There are different ways to acquire the fabric downloader, such as download it from the RP each time or program it into Flash.

If the service upgrade all command is not configured, then the fabric downloader is not programmed into Flash. The line card needs to download the fabric downloader each time the line card boots and go through these states:

ROMVGET  EV_AGENT_REPORT_POWERED

ROMIGET  EV_LC_ROM_MON_RESET 

FABIWAIT EV_LC_ROM_IMAGES_REPORT              

FABLDNLD EV_FAB_DOWNLOADER_DOWNLOAD_STARTABLE 

FABLSTRT EV_FAB_DOWNLOADER_DOWNLOAD_SUCCESS   

FABLRUN  EV_FAB_DOWNLOADER_LAUNCH_SUCCESS     

IOS DNLD EV_IOS_DOWNLOAD_WAIT_DL_CONFIRM

IOS STRT EV_IOS_DOWNLOAD_SUCCESS         

IOS UP   EV_IOS_REPORT                         

IOS RUN  EV_BUFF_CARVE_SUCCESS

In addition, you see this warning message for your line cards in the output of the show version command:

WARNING: Old Fabric Downloader in slot 2

Use "upgrade fabric-downloader" command to update the image

On the other hand, if the service upgrade all command is configured, then on the first load of a particular Cisco IOS software image, the line card loads the fabric downloader and programs it into Flash:

NOT YET --- EV_FLASH_PROG_DONE     
IN  RSET --- EV_FLASH_PROG_DONE

The line card goes through these states only on the first load:

ROMVGET  EV_AGENT_REPORT_POWERED

ROMIGET  EV_LC_ROM_MON_RESET                   

FABIWAIT EV_LC_ROM_IMAGES_REPORT              

FABLDNLD EV_FAB_DOWNLOADER_DOWNLOAD_STARTABLE 

FABLSTRT EV_FAB_DOWNLOADER_DOWNLOAD_SUCCESS   

FABLRUN  EV_FAB_DOWNLOADER_LAUNCH_SUCCESS     

IOS DNLD EV_IOS_DOWNLOAD_WAIT_DL_CONFIRM

IOS STRT EV_IOS_DOWNLOAD_SUCCESS         

IOS UP   EV_IOS_REPORT                         

IOS RUN  EV_BUFF_CARVE_SUCCESS     

NOT YET  EV_FLASH_PROG_DONE     

IN  RSET EV_FLASH_PROG_DONE     

ROMIGET  EV_LC_ROM_MON_RESET                   

FABLSTRT EV_FAB_DOWNLOADER_DOWNLOAD_SUCCESS   

FABLRUN  EV_FAB_DOWNLOADER_LAUNCH_SUCCESS     

IOS DNLD EV_IOS_DOWNLOAD_WAIT_DL_CONFIRM

IOS STRT EV_IOS_DOWNLOAD_SUCCESS         

IOS UP   EV_IOS_REPORT                         

IOS RUN  EV_BUFF_CARVE_SUCCESS

If the service upgrade all command is configured, and this is a reload after the first reload with this Cisco IOS software image, the bootup looks like this:

ROMVGET  EV_AGENT_REPORT_POWERED

ROMIGET  EV_LC_ROM_MON_RESET                   

FABIWAIT EV_LC_ROM_IMAGES_REPORT              

FABLRUN  EV_FAB_DOWNLOADER_LAUNCH_SUCCESS     

IOS DNLD EV_IOS_DOWNLOAD_WAIT_DL_CONFIRM

IOS STRT EV_IOS_DOWNLOAD_SUCCESS         

IOS UP   EV_IOS_REPORT                         

IOS RUN  EV_BUFF_CARVE_SUCCESS

Even though the first load with the service upgrade all command has a long boot up time, the advantage is that subsequent boots do not waste time to download the fabric downloader.

Online Insertion and Removal (OIR)

A removal of a line card generates this state:

NOT YET --- EV_ENVMON_CARD_REMOVED

Similarly, an insertion generates this state:

NEW INS --- EV_ENVMON_CARD_INSERTED

After the new line card has been inserted, the MBUS must be powered up, followed by the rest of the line card:

MBUSWAIT   EV_AGENT_REPORT_AGENT_IN_ROM
MBUSWAIT   EV_AGENT_REPORT_AGENT_IN_ROM
MBUSDNLD   EV_MBUS_AGENT_DOWNLOAD_STARTABLE
MBUSDONE   EV_MBUS_AGENT_DOWNLOAD_SUCCESS
PWR ON   EV_AGENT_REPORT_UNPOWERED

The normal boot up process then continues from:

ROMIGET --- EV_LC_ROM_MON_RESET

hw-module slot shutdown

You can configure the hw-module slot <x> shutdown command in order to cleanly reset the line card and leave it in a shut (also known as administrative down) state. After you issue this command, the line card boots up until IOS STRT and then stays in ADMNDOWN. When you configure this command, the log shows these state transitions:

NOT YET  EV_ADMIN_SLOT_SHUT 

IN  RSET EV_ADMIN_SLOT_SHUT 

ROMVGET  EV_LC_ROM_TYPE_AFTER_RESET_TIMEOUT

ROMIGET  EV_LC_ROM_MON_RESET

FABLWAIT EV_LC_ROM_IMAGES_REPORT_WAIT_FAB

FABLDNLD EV_FAB_DOWNLOADER_DOWNLOAD_STARTABLE

FABLSTRT EV_FAB_DOWNLOADER_DOWNLOAD_SUCCESS

FABLRUN  EV_FAB_DOWNLOADER_LAUNCH_SUCCESS

IOS DNLD EV_IOS_DOWNLOAD_WAIT_DL_CONFIRM

IOS STRT EV_IOS_DOWNLOAD_SUCCESS

ADMNDOWN EV_IOS_REPORT

The line card stays in this last state until the hw-module slot <x> shutdown configuration is removed. When you choose to bring the line card back up with the no hw-module slot <x> shutdown command, the line card boots again as it did originally and starts off with these events:

NOT YET --- EV_ADMIN_NO_SLOT_SHUT
IN  RSET --- EV_ADMIN_NO_SLOT_SHUT

After this, the normal bootup process continues from:

ROMIGET --- EV_LC_ROM_MON_RESET

microcode reload

A microcode reload simply restarts the bootup process of a line card and starts off with these events:

NOT YET --- EV_ADMIN_LC_RELOAD 
IN  RSET --- EV_ADMIN_LC_RELOAD

Then the normal bootup process continues from:

ROMIGET --- EV_LC_ROM_MON_RESET

Troubleshooting

If the line card status is anything other than IOS RUN, or the RP is neither the active Master/Primary or the Slave/Secondary, this means that there is a problem and the card has not fully loaded correctly. Before you replace the card, Cisco recommends these steps to fix the issue:

  1. Use the Software Advisor tool (registered customers only) to determine if the new card is supported in your current Cisco IOS software version. If the line card is supported, then configure the service upgrade all command, save the configuration with the copy run start command and power-cycle the router. Sometimes a reload is not sufficient, but a power-cycle will fix the problem.

    If the new card is not supported in your current Cisco IOS software version, verify that you have enough route memory installed on the line card before you upgrade the Cisco IOS software version. For Cisco IOS Software Release 12.0(21)S, 256 MB of route memory is required, especially if Border Gateway Protocol (BGP) is configured with many peers and many routes.

    You can also refer to these links for more information:

  2. Verify which stage of booting the line card is stuck. You can issue the show led command in order to see what state the line card is currently in.

    If the output of the show led command shows MEM INIT, then you should re-seat the memory on the line card.

    If the output of the show led command shows MRAM, then the line card is probably not seated properly and you should re-seat it. You also need to check and make sure you have the appropriate number of CSC and SFCs in the chassis in order for the line card to work. Only Engine 0-based line cards work in a quarter bandwidth configuration. All the other line cards need at least four switching fabric cards to run correctly.

    You can always issue the show event-trace slot-state command in order to look at the boot process of the the line card.

These are some tips that can help solve a booting issue on a card:

  • Issue the global configuration microcode reload <slot> command in order to reload the microcode.

  • Issue the hw-module slot <slot> reload command in order to reload the card. This causes the line card to reset and re-download the MBUS and Fabric Downloader software modules before you attempt to re-download the Cisco IOS Line Card software.

  • Issue the upgrade all slot command in order to upgrade the MBUS Agent ROM, the MBUS Agent RAM, and the fabric downloader. Refer to Upgrading Line Card Firmware on a Cisco 12000 Series Internet Router.

  • Reset the line card manually. This can rule out any problems that are caused by a bad connection to the MBUS or switching fabric.

You might see this error message on the Gigabit Route Processor (GRP):

%GRP-3-UCODEFAIL: Download failed to slot 5

This message means that the image that was downloaded to the line card has been rejected. You can issue the microcode reload configuration command in order to reload the microcode. If the error message recurs, issue the upgrade all slot command in order to upgrade the MBUS Agent ROM, the MBUS Agent RAM, and the fabric downloader. Refer to Upgrading Line Card Firmware on a Cisco 12000 Series Internet Router for more information.

Engine 2-based line cards sometimes get stuck in STRTIOS. This might be due to packet memory DIMMs installed in the TLU/PLU socket and vice-versa. Refer to Memory Locations on an Engine 2 Line Card for information on the memory location of this type of card.

There is one command sequence to check the amount of TLU/PLU memory:

Router#attach <slot #>
LC-Slot#show control psa mem

The following symptoms are :
1)"show LED" is in STRTIOS
2)"show diag" may indicate 
Board is disabled analyzed idbs-rem 
Board State is Launching IOS (IOS  STRT):

Router#show led
SLOT 4  : STRTIOS 
SLOT 7  :  RP ACTV
 
Router#show diag 4
 
SLOT 4  (RP/LC 4 ): 3 Port Gigabit Ethernet
  MAIN: type 68,  800-6376-01 rev C0
        Deviation: 0
        HW config: 0x00    SW key: 00-00-00
  PCA:  73-4775-02 rev C0 ver 2
        Design Release 1.0  S/N SDK0433157H
  MBUS: Embedded Agent
        Test hist: 0x00    RMA#: 00-00-00    RMA hist: 0x00
  DIAG: Test count: 0x00000000    Test results: 0x00000000
  FRU:  Linecard/Module: 3GE-GBIC-SC=
  L3 Engine: 2 - Backbone OC48 (2.5 Gbps)
  MBUS Agent Software version 01.51 (RAM) (ROM version is 02.17)
  ROM Monitor version 10.06
  Fabric Downloader version used 08.01 (ROM version is 05.03)
  Primary clock is CSC 1
  Board is disabled analyzed idbs-rem 
  Board State is Launching IOS (IOS  STRT)
  Insertion time: 00:00:06 (00:11:00 ago)

This board cannot boot up to IOS RUN and is stuck at IOS START. The 64 MB SDRAMS were installed on J5 and J8 instead of 128 MB SDRAMs, and the 128 MB SDRAMs were installed on J4 and J6 instead of 64 M SDRAMS. The root cause of this failure was due to mismatch memory, SDRAMS in which the transmitted SDRAMs were 128 MB compared with the received SDRAMS that were 64 MB. After reconfiguring 128 MB SDRAMs on J5 and J8, this board booted up properly.

The wrong size memory put into the wrong slot is only possible for the Engine 2-based line cards because these are the only ones that have PLU/TLU with the same physical look as the RX/TX packet memory.

Refer to Cisco 12000 Series Router Memory Replacement Instructions for information on the memory locations on the Engine 2-based line card.

Troubleshooting Commands

show version

Router#show version
Cisco Internetwork Operating System Software 
IOS (tm) GS Software (GSR-P-M), Version 12.0(22)S, EARLY DEPLOYMENT RELEASE SOFTWARE (fc2)

The Cisco IOS Software version loaded on the RP is 12.0(22)S. The Cisco IOS software image is copied from the location specified by the boot system <source> command. Then, it is decompressed and loaded into the DRAM of the RP.

Note: If you configure the boot system <source> command without specifying the image name, the RP tries to load the first file in that slot/disk. Therefore, make sure the first image is a valid Cisco IOS software image.

Refer to Cisco 12000 Routers May Fail to Boot from an ATA Disk During Upgrades to Cisco IOS Software Release 12.0(22)S if you use an ATA disk.

TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2002 by cisco Systems, Inc.
Compiled Sat 20-Jul-02 04:40 by nmasa
Image text-base: 0x50010968, data-base: 0x5207A000

ROM: System Bootstrap, Version 11.2(20010625:183716) [bfr_112 181], DEVELOPMENT SOFTWARE

Bootstrap Version 181—The version of bootstrap which is also known as ROM Monitor or ROMMON that runs on the RP. The bootstrap image is run by default directly from ROM, or issue the boot bootstrap <source> command in order to specify the source. You can complete these steps for 512MB DRAM support on the RP:

Once you have identified the type of GRP you have and the current ROMMON version, these are the different possibilities:

  • GRP—This one does not support the 512 MB option. You need to replace this card with a GRP-B.

  • GRP-B with ROMMON version 180—First you need to upgrade the Cisco IOS Software Release to 12.0(19)S or later. Then, issue the upgrade rom slot X (where X is the slot number where the GRP is located) command in order to upgrade the ROMMON version manually. Once these steps have been performed, you can physically upgrade the memory as described in Cisco 12000 Series Router Memory Replacement Instructions.

  • GRP-B with ROMMON version 181 or later—You need to check that you are running Cisco IOS Software Release 12.0(19)S or later. Then you can physically upgrade the memory as described in Cisco 12000 Series Router Memory Replacement Instructions.

BOOTLDR: GS Software (GSR-BOOT-M), Version 12.0(8)S, EARLY 
DEPLOYMENT MAINTENANCE INTERIM SOFTWARE

Bootloader Version 12.0(8)S—The version of bootloader that runs on the RP. Issue the boot bootldr <source> command in order to specify the source. The bootloader is required for netbooting (booting a Cisco IOS software image from a TFTP source). You should upgrade the bootloader to the most recent version.

Router uptime is 1 hour, 18 minutes

The uptime is the duration of time since the last reload.

System returned to ROM by reload at 16:02:27 UTC Mon Aug 19 2002
System image file is "slot0:gsr-p-mz.120-22.S"

This shows the source of the Cisco IOS software image. In this case, it is an image stored in slot0:

cisco 12410/GRP (R5000) processor (revision 0x01) with 524288K bytes of memory.
R5000 CPU at 200Mhz, Implementation 35, Rev 2.1, 512KB L2 Cache
Last reset from power-on

1 Route Processor Card
2 Clock Scheduler Cards
5 Switch Fabric Cards
1 Single-port OC12c ATM controller (1 ATM).
1 four-port OC48 POS controller (4 POS).
2 Single Port Gigabit Ethernet/IEEE 802.3z controllers (2 GigabitEthernet).
1 Ethernet/IEEE 802.3 interface(s)
2 GigabitEthernet/IEEE 802.3 interface(s)
1 ATM network interface(s)
4 Packet over SONET network interface(s)
507K bytes of non-volatile configuration memory.

16384K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
8192K bytes of Flash internal SIMM (Sector size 256K).
Configuration register is 0x2002

show led

Router#show led
SLOT 2  : RUN IOS 

The slots that contain line cards display one of several outputs (details later). In this case, the line card in slot 2 is fully booted and in the RUN IOS state.

SLOT 4  : RUN IOS 
SLOT 5  : RUN IOS 
SLOT 6  : RUN IOS 
SLOT 9  : RP ACTV

The slots that contain RPs display one of two outputs: RP ACTV and RP STBY. This depends on which RP is the active and which is the standby. In this case, the RP in slot 9 is fully booted and is the Active RP.

show diags <x>

Router#show diags 2
 

SLOT 2  (RP/LC 2 ): 4 Port Packet Over SONET OC-48c/STM-16 Single Mode/SR SC connector

  MAIN: type 67, 800-5517-03 rev A0

        Deviation:  D026529

        HW config: 0x04   SW key: 00-00-00

  PCA:  73-4203-04 rev B0 ver 3

        Design Release 2.0 S/N CAB0543L3FH

  MBUS: Embedded Agent

        Test hist: 0x00   RMA#: 00-00-00    RMA hist: 0x00

  DIAG: Test count: 0x00000000   Test results: 0x00000000

  FRU:  Line card/Module: 4OC48/POS-SR-SC=

        Route Memory: MEM-LC4-256=

        Packet Memory: MEM-LC4-PKT-512=

  L3 Engine: 4 - Backbone OC192/QOC48 (10 Gbps)

  MBUS Agent Software version 01.50 (RAM) (ROM version is 02.10)

MBUS Agent software versions—RAM information is displayed if the MBUS agent runs from RAM, as it should be.

ROM Monitor version 01.04

  Fabric Downloader version used 05.00 (ROM version is 04.01)

  Primary clock is CSC 1 Board is analyzed 

  Board State is Line Card Enabled (IOS  RUN )

  Insertion time: 00:00:12 (01:17:53 ago)

Insertion time—The time duration for which the line card has been powered on. The first time 00:00:12 (HH:MM:SS) is the time the line card was powered after the RP reloaded. The second time 01:17:53 (HH:MM:SS) is the time duration that the line card has been powered. The first time added to the second time equals the uptime in the show version command output.

DRAM size: 268435456 bytes

  FrFab SDRAM size: 268435456 bytes

  ToFab SDRAM size: 268435456 bytes

  0 crashes since restart

show monitor event-trace slot-state <x>

The show gsr slot <x> command provides the same output and is easier to remember.

Router#show gsr slot 0
SLOT STATE TRACE TABLE -- Slot 0  (Current Time is 4116199.392)

Current time: 4116199.392 seconds is the time duration the RP has been powered on.

+-----------------------------------------------------------------------
|  Timestamp    Pid State    Event                                 Flags
+-----------------------------------------------------------------------
         3.296   2  IOS STRT EV_RP_MBUS_DISCOVERY_SUCCESS
        22.536   2  IOS UP   EV_RP_LOCAL_AGENT_REPORT
        33.184  46  IOS UP   EV_RP_LOCAL_FAB_READY                 an

The output for a line card is similar:

Router#show gsr slot 2 

SLOT STATE TRACE TABLE -- Slot 2  (Current Time is 4776.108)

Current time: 4776.108 seconds is the time duration the line card has been powered on.

+-----------------------------------------------------------------------

|  Timestamp    Pid State    Event                                 Flags

+-----------------------------------------------------------------------

      12.756   3  ROMVGET EV_AGENT_REPORT_POWERED

      15.056  10  ROMIGET EV_LC_ROM_MON_RESET                      an 

      15.448  10  FABIWAIT EV_LC_ROM_IMAGES_REPORT                 an 

      34.048  48  FABLDNLD EV_FAB_DOWNLOADER_DOWNLOAD_STARTABLE    an 

      50.740  10  FABLSTRT EV_FAB_DOWNLOADER_DOWNLOAD_SUCCESS      an

      54.936  10  FABLRUN EV_FAB_DOWNLOADER_LAUNCH_SUCCESS         an 

      77.580  77  IOS DNLD EV_IOS_DOWNLOAD_WAIT_DL_CONFIRM         an

      77.636  10  IOS STRT EV_IOS_DOWNLOAD_SUCCESS                 an 

      92.148  10  IOS UP  EV_IOS_REPORT                            an 

      93.168 288  IOS RUN  EV_BUFF_CARVE_SUCCESS                   an

The rest of the output from the show monitor event-trace slot-state <x> command describes each of the states that the line card went through.

Information to Collect if You Contact Technical Support

If you contact Technical Support, attach this information to your case to troubleshoot a line card status which is anything other than IOS RUN:
  • The show tech-support command output in enable mode, if possible.
  • A complete bootup sequence captured from the console port.
  • The show log command output or console captures, if available.
  • Output from these show commands:
    • show gsr slot <slot>
    • show monitor event-trace mbus
    • show monitor event-trace mbus | incl slot# (where # is the slot number of the stuck line card)
    • show monitor event-trace fab
    • show ipc ports
    • show ipc nodes
    • show ipc stat
    • show controller sca
    • show controller xbar
    • show controller clock
    • show controller csc-fpga
  • A detailed description of the troubleshooting steps you performed.
Refer to the Service Request Tool (registered customers only) to upload and attach information to your case. If you cannot access this tool, you can send the information in an email attachment to attach@cisco.com with your case number in the subject line of your message to attach the relevant information to your case.

Note: Do not manually reload or power-cycle the router before you collect this information unless required to troubleshoot a booting issue on a line card/GRP. This can cause important information to be lost that is needed to determine the root cause of the problem.

Related Information

Updated: Aug 31, 2006
Document ID: 40060