Cisco IOS XR Troubleshooting Guide, Release 3.5
Troubleshooting Booting
Downloads: This chapterpdf (PDF - 416.0KB) The complete bookPDF (PDF - 2.2MB) | Feedback

Troubleshooting Booting

Table Of Contents

Troubleshooting Booting

Booting Tips

Verifying Successful Bootup

Verifying and Troubleshooting the Console Connection

Verifying and Troubleshooting Route Processor and Shelf Controller Cards

Troubleshooting RP and SC Cards Not Booting

Troubleshooting RP and SC Cards Resetting While Booting

Troubleshooting Blocked FCC Shelf Controller or LCC Route Process Minimum Boot Image Requests


Troubleshooting Booting


This chapter describes techniques that you can use to troubleshoot a router running Cisco IOS XR software. It includes the following sections:

Booting Tips

Verifying Successful Bootup

Verifying and Troubleshooting the Console Connection

Verifying and Troubleshooting Route Processor and Shelf Controller Cards

Verifying and Troubleshooting Route Processor and Shelf Controller Cards

Booting Tips

The following booting sequence should be followed. See Cisco IOS XR Getting Started Guide for detailed information on the booting sequence for routers running Cisco IOS XR software.

1. Connect to the console port.

2. Boot the route processor (RP) of the designated shelf controller (DSC).

For Cisco CRS-1 Multishelf Systems, when the RP of the DSC boots up, if the RP or shelf controller (SC) on other chassis are configured to autoboot (for example, they are in a loop sending minimum boot image [MBI] requests), they boot automatically. If they have not been configured to autoboot, enter reset at the ROM Monitor (ROMMON) prompt.

3. Verify that the value of the configuration register (confreg) is 0x102. Enter confreg at the ROMMON prompt to display the configuration register value. If the configuration register value is not set to 0x102, see Cisco IOS XR Getting Started Guide for information on changing the configuration register value.

4. It takes approximately 20 minutes for the cards on all chassis to be displayed in the Cisco IOS XR software show commands. Managed nodes (service processor [SP], line card [LC], and distributed route processor [DRP]) are booted by shelfmgr, mbimgr, and instdir automatically when ROMMON is set to autoboot.

Use the following chassis numbering guidelines when numbering the chassis in a Cisco CRS-1 Multishelf System:


Note It is recommended that the line card chassis containing the active DSC be numbered 0, the second line card chassis be numbered 1, and the fabric card chassis be numbered F0. See Cisco IOS XR Getting Started Guide for more information on recommended chassis numbering.


The line card chassis (LCC) numbering must be in the range of 0 to 127.

The fabric card chassis (FCC) numbering must be in the range of 240 to 255.

In a multishelf configuration, all racks must have a unique number.

Verifying Successful Bootup

For information on verifying the status of the router, see Cisco IOS XR Getting Started Guide. It contains detailed procedures for verifying a standalone router and multishelf system.

The show platform command displays the state transitions when a node is booting:

SP RMON—The service processor (SP) is starting and the basic code is being loaded on the SP (LC only).

ROMMON—Once the SP has started, the LC CPU is on and the LC ROMMON software is started (LC only).

MBI BOOT—The minimum boot image code is loading on the LC CPU from disk or the DSC (LC only).

MBI RUNNING—The required processes on the LC are starting. The normal transition from this state is to start the Cisco IOS XR operating system.

IOS XR—The Cisco IOS XR software is running.

Verifying and Troubleshooting the Console Connection

The default configuration selects the route processor (RP) in the lowest numbered slot. For the Cisco CRS-1 Multishelf System, the RP in the lowest numbered slot in the LCC configured as Rack 0 is selected as the primary RP and the designated shelf controller (DSC).

To determine the primary RP on routers running Cisco IOS XR software:

On Cisco CRS-1 RPs, the primary RP is identified by a lighted Primary LED on the RP front panel. The primary RP is also identified by "ACTV RP" in the alphanumeric display on the card. The console port is located on the primary RP.

On Cisco XR 12000 Series Routers, the alphanumeric LED display on the active PRP displays: PRI RP.

The console connection through the DSC provides a communications path to the routing system. The console port is designed for a serial cable connection to a terminal or computer running a terminal emulation program. See Cisco IOS XR Getting Started Guide for information on connecting to the console port.

To verify and troubleshoot the console connection on the DSC, perform the following procedures.

SUMMARY STEPS

1. Check the cable between the console port on the DSC and the terminal or computer.

2. Open the terminal emulation application, select the correct COM port and verify the settings.

3. Swap cables.

4. Test the cable on another console port.

5. Contact Cisco Technical Support if the problem is not resolved.

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

Check the cable between the console port on the DSC and the terminal or computer.

Ensure that there is a serial cable connection from a terminal or computer running a terminal emulation program to the console port on the active RP of the DSC. See Cisco IOS XR Getting Started Guide for information on connecting to the console port.

Step 2 

Open the terminal emulation application, select the correct COM port and verify the settings.

Verify that the terminal settings are as follows:

Bits per second: 9600/9600

Data bits: 8

Parity: None

Stop bit: 2

Flow control: None

If the correct settings are not applied, the console port outputs garbage characters or no output at all. This makes the router appear as if it is hanging or not responding.

If this fails to solve the problem, proceed to Step 3.

Step 3 

Swap cables.

Swap cables.

Step 4 

Test the cable on another console port.

Test the console cable by attaching the console cable to the console port of another router.

Step 5 

Contact Cisco Technical Support.

If these steps do not resolve the problem, contact Cisco Technical Support. See the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface.

Verifying and Troubleshooting Route Processor and Shelf Controller Cards

The following RP and SC verification and troubleshooting procedures are provided:

Troubleshooting RP and SC Cards Not Booting

Troubleshooting RP and SC Cards Resetting While Booting

Troubleshooting Blocked FCC Shelf Controller or LCC Route Process Minimum Boot Image Requests

Troubleshooting RP and SC Cards Not Booting

To troubleshoot the RP and SC cards in a router running Cisco IOS XR software, perform the following procedure.

Troubleshooting procedures for the following cards is provided in this section:

Cisco CRS-1 4-Slot Line Card Chassis/Cisco CRS-1 8-Slot Line Card Chassis/Cisco CRS-1 16-Slot Line Card Chassis

Primary (DSC) and standby route processors (RPs)

Cisco XR 12000 Series Router

Performance route processor (PRP)

Cisco CRS-1 Multishelf System

DSC in the line card chassis (LCC)

Standby shelf controller (SC) in the LCC

SC in the fabric card chassis (FCC)

Primary and standby route processor (RP) in the LCC

SUMMARY STEPS

1. Place the RP (or SC) in ROMMON mode.

2. set

3. Reset the RP (or SCs) in the affected chassis.

4. Swap RP slots.

5. Verify connection status.

6. Collect console messages generated by the RP.

7. Contact Cisco Technical Support if the problem is not resolved.

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

Place the RP or SC not booting in ROMMON mode.

See Router Recovery with ROM Monitor in Cisco IOS XR ROM Monitor Guide for information on entering ROMMON mode.

Step 2 

set

Example:

rommon B1 > set

Displays the environment variable settings for the card.

Verify that the variables are valid.

The following are valid variables:

rommon B1 > set
PS1=rommon ! > ?=0 
 
        

The following variables are the main variables used for netboot:

IP_SUBNET_MASK=255.255.0.0 
TFTP_SERVER=223.255.254.254 
IP_ADDRESS=12.2.53.41 
TFTP_FILE=muck/jasamson/comp-hfr-full-hfr34-qq.vm 
DEFAULT_GATEWAY=12.2.0.1
 
        

The following variables add verbosity to certain ROMMON functions and to bypass auxiliary authentication in Cisco IOS XR:

TFTP_VERBOSE=2 
AUX_AUTHEN_LEVEL=0 
 

The following variables are used by the Config Manager to immediately apply the configuration after bootup:

IOX_ADMIN_CONFIG_FILE=nvram:startup-config-admin.00 
IOX_CONFIG_FILE=nvram:startup-config-lr.00 
 
        

The following variables above may be specified may be displayed as follows if a path has not been specified:

IOX_CONFIG_FILE=
IOX_ADMIN_CONFIG_FILE=
 
        

The following variable may be displayed before the system has been Turbobooted:

TURBOBOOT=on,format,disk0
 
        

The following variables may be displayed after a system has been Turbobooted. A 'boot' variable is saved to indicate where the minimum boot image (MBI) is stored and the turboboot variable is set to NULL:

BOOT=disk0:hfr-os-mbi-3.3.96/mbihfr-rp.vm,1
TURBOBOOT= 
 
        

The following variable is Reboot/Reload code saved after any kind of shutdown, failover, and so on:

ReloadReason=0
 
        

The following variable is saved to NVRAM.

BSI=0

Step 3 

Remove then reinsert the SC or RP.

See the following documents for information on removing then reinserting the SC (or RP):

Installing the Cisco CRS-1 Carrier Routing System 16-Slot Line Card Chassis

Installing the Cisco CRS-1 Carrier Routing System 8-Slot Line Card Chassis

The documents are located at the following URL:

http://www.cisco.com/en/US/products/ps5763/tsd_products_support_series_home.html

Step 4 

Swap RP (or SC) slots.

Swap the RP (or SC) card in the chassis:

RP slots—RP0 and RP1

SC slots—SC0 and SC1

Step 5 

Verify connection status.

For the single-shelf Cisco CRS-1 and Cisco XR 12000 Series Routers:

Use the show platform command in administration mode to verify that all cards are in the IOS XR RUN state.

Use the show controllers switch 0 statistics location 0/RP0/CPU0 command in administration mode to verify the switch statistics.

For the Cisco CRS-1 Multishelf System:

Use the show controllers switch switch-instance statistics location 0/RP0/CPU0 command in administration mode to verify the switch statistics. Repeat this command for switches on each shelf in the system.

For the SC on the FCC and the primary and standby RP on the LCC, verify the system control plane Ethernet network connection status between the DSC and card that will not boot. See the "Troubleshooting the Multishelf System Router Topology" section of Chapter 3 "Troubleshooting the Control Plane Ethernet Network," for information on verifying connection status between the DSC and the SC or RP.

Step 6 

Collect console messages generated by the RP (or SC).

See Chapter 10 "Collecting System Information" for details on the information required when contacting Cisco Technical Support.

Step 7 

Contact Cisco Technical Support.

For Cisco Technical Support contact information, see the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface.

Troubleshooting RP and SC Cards Resetting While Booting

To troubleshoot resetting RP and SC cards on Cisco CRS-1 systems, perform the following procedures.

SUMMARY STEPS

1. Place both RPs in ROMMON mode.

2. set

3. Verify that all three LEDs on power modules are green.

4. Power cycle the power modules.

5. Check that the DSC is active.

6. Collect console messages generated by the RP.

7. Contact Cisco Technical Support if the problem is not resolved.

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

Place both RPs in ROMMON mode.

See Router Recovery with ROM Monitor in the Cisco IOS XR ROM Monitor Guide for information on entering ROMMON mode.

Step 2 

set

Example:

rommon B1 > set

Displays the environment variable settings for the card.

Verify that the variables are valid.

Step 3 

For each affected chassis, verify that all three LEDs on power modules are green.

See Cisco CRS-1 Carrier Routing System 16-Slot Line Card Chassis Hardware Operations and Troubleshooting Guide for the location of the power modules.

Step 4 

Power cycle the power modules to confirm that the LEDs transition from amber to green.

If the LEDs on the power modules stay amber or go from amber to green then back to amber, there is a problem with the power module.

See the following documents for power cycling information:

Installing the Cisco CRS-1 Carrier Routing System Fabric Card Chassis

Installing the Cisco CRS-1 Carrier Routing System Line Card Chassis

Need to check R3.3 for these docs since FCC is not in R3.2.

The documents can be found at the following URL:

http://www.cisco.com/en/US/products/ps5763/tsd_products_support_series_home.html

Step 5 

Check that the DSC is active.

Verify that the Primary LED on the DSC front panel is lit.

Step 6 

Collect console messages generated by the RP.

See Chapter 10 "Collecting System Information" for details on the information required when contacting Cisco Technical Support.

Step 7 

Contact Cisco Technical Support.

See the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface for Cisco Technical Support contact information.

Troubleshooting Blocked FCC Shelf Controller or LCC Route Process Minimum Boot Image Requests


Note This procedure applies to Cisco CRS-1 Multishelf Systems only.


To troubleshoot blocked fabric card chassis SC or line card chassis RP minimum boot image (MBI) requests, perform the following procedures.

If the shelf controller (SC) or route processor (RP) in the remote line card chassis fails to boot correctly, messages are displayed on the console of either device, indicating that it is waiting for information from the designated shelf controller (DSC).

SUMMARY STEPS

1. Verify the physical system control plane Ethernet network connection between the DSC in the LCC to the Catalyst switch.

2. Connect to the Catalyst switch console port.

3. Verify the physical system control plane Ethernet network connection between the SC in the FCC to the Catalyst switch.

4. show spanning-tree

5. Contact Cisco Technical Support if the problem is not resolved.

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

Verify the physical system control plane Ethernet network connection between the DSC in the LCC and the Catalyst switch.

Verify the physical connectivity between the DSC and Catalyst switch. See Cisco CRS-1 Carrier Routing System Multishelf System Interconnection and Cabling Guide for details on cabling between the DSC and the Catalyst switch.

Step 2 

Connect to the Catalyst switch console port.

Provides connection to the system control plane Ethernet network.

See Cisco CRS-1 Carrier Routing System Multishelf System Interconnection and Cabling Guide for details on connecting to the external switch.

Step 3 

Verify the system control plane Ethernet network.

Verify the system control plane Ethernet network connection status between the DSC and SC. See the "Troubleshooting the Multishelf System Router Topology" section in Chapter 3 "Troubleshooting the Control Plane Ethernet Network" for information on verifying the system control plane Ethernet network connection status between the DSC and SC.

Step 4 

show spanning tree

Example:

Router# show spanning-tree

Displays the spanning tree port states.

Verify that the ports used to connect the DSC, remote LCC RP, and the FCC SC are in the forwarding state. The listed interfaces should include the port to which you have connected.

If the port is not listed, contact Cisco Technical Support. For Cisco Technical Support contact information, see the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface.

Step 5 

Contact Cisco Technical Support.

See the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface for Cisco Technical Support contact information.