Cisco IOS XR Troubleshooting, Release 3.4 - General Troubleshooting [Cisco IOS XR Software (End-of-Sale)]

Table Of Contents

General Troubleshooting

Troubleshooting Techniques and Approaches

Documenting the Network

Verifying and Troubleshooting CLI Access

General CLI Access Information

User Access Privileges

CLI Access Through a Console Port

CLI Access Through a Terminal Server

CLI Access Through the Management Ethernet Interface

Examples

Basic Cisco IOS XR Verification and Troubleshooting Tools

man Command

describe Command

show platform Command

top Command

show context Command

show users Command

Verifying the System

Examples

Validating and Troubleshooting Cisco IOS XR Software Package Installation

Verifying the Software Version

Validating the Installation

install verify Command

show install active Command

show install committed Command

Validating and Troubleshooting Cisco IOS XR Software Configuration

Local and Global Configurations

Collecting Configuration Information

Verifying the Running Configuration

Examples

Using the show configuration failed Command

Startup Failed Configuration

Commit Configuration Failed

ASIC Errors

Trace Commands

Packets

Logging Archive for Harddisk

SNMP Polling Awareness of SystemOwner, LR Owner, MIB Location

Error File Locations and Data Collection Scripts

Error File Locations

harddisk:

disk1:

Sysmgr Collection Scripts

Wdsysmon Collection Scripts

Shutdown Collection Scripts

ASIC error Collection Scripts

Monitoring

monitor interface Command

monitor controller Command

monitor processes Command

monitor threads Command

Gathering Information for Technical Support

General Troubleshooting

This chapter describes general troubleshooting techniques you can use to troubleshoot routers using Cisco IOS XR software. This chapter includes the following sections:

•Troubleshooting Techniques and Approaches

•Verifying and Troubleshooting CLI Access

•Basic Cisco IOS XR Verification and Troubleshooting Tools

•Verifying the System

•Validating and Troubleshooting Cisco IOS XR Software Package Installation

•Validating and Troubleshooting Cisco IOS XR Software Configuration

•ASIC Errors

•Trace Commands

•Packets

•Logging Archive for Harddisk

•Logging Archive for Harddisk

•SNMP Polling Awareness of SystemOwner, LR Owner, MIB Location

•Error File Locations and Data Collection Scripts

•Monitoring

•Gathering Information for Technical Support

For information on troubleshooting hardware, see the hardware documentation listed in the "Related Documents" section on page xiv.

Troubleshooting Techniques and Approaches

The following techniques and approaches are recommended when troubleshooting using Cisco IOS XR software:

•Maintain current documentation about the network. See the "Documenting the Network" section for more information.

•Maintain current documentation about the system, including chassis numbers, serial numbers, installed cards, and location of chassis details.

•Maintain diagrams illustrating the connectivity of the router control plane Ethernet network.

•Capture and save the output of all commands. This information is useful when contacting Cisco Technical Support. For information on contacting Cisco Technical Support, see the "Obtaining Technical Assistance" section in the Preface.

•Have the output of the relevant show tech-support command captured and saved. The output from the show tech-support command provides a traditional dump of the configuration and show command outputs. For more commands used to collect system information, see Chapter 10 "Collecting System Information."

Documenting the Network

To be prepared to troubleshoot the network, you should maintain current documentation about the network, including the following:

•An up-to-date internetwork map that outlines the physical location of all the devices about the network and how they are connected

•A logical map of network addresses, network numbers, subnetworks, and so on

•A list of all network protocols implemented in your network. For each of the protocols implemented, a list of the network numbers, subnetworks, zones, areas, and so on that are associated with them

•For multishelf systems, note the Layer 2 connections used to provide ***router control plane Ethernet network connectivity between racks (including the fabric card chassis [FCC])

•All points of contact to external networks

•The routing protocol for each external network connection

•The established baseline for your network, that is, the normal network behavior and performance at different times of the day so that you can compare any problems with a baseline

•Which device is the spanningtree route bridge for the system control plane Ethernet network

Verifying and Troubleshooting CLI Access

Ensure the system has been booted. If the system has not booted, see Cisco IOS XR Getting Started Guide for information on booting a router running Cisco IOS XR software. The following CLI access troubleshooting information is provided:

•General CLI Access Information

•User Access Privileges

•CLI Access Through a Console Port

•CLI Access Through a Terminal Server

•CLI Access Through the Management Ethernet Interface

General CLI Access Information

The following CLI access information applies to a console port, terminal server, and management Ethernet interface connections.

Once the terminal emulation software is started and you press Enter, a prompt should appear. If a prompt does not appear, verify the physical connection to the console port and press Enter again. If the prompt still does not appear, contact Cisco Technical Support. See the "Obtaining Technical Assistance" section in the Preface for Cisco Technical Support contact information.

If a prompt appears, indicating that the command-line interface (CLI) is accessible, but your login username and password are invalid, you are prevented from accessing the router. Verify that you have the correct username and password. If you have the correct username and password, but are locked out of the router, you may need to perform password recovery to access the system. See Cisco IOS XR Getting Started Guide for password recovery procedures.

User Access Privileges

When logging on to the router, use a username that is associated with a valid user group that has the authorization to execute the required commands.

•If you are troubleshooting all Secure Domain Routers (SDRs), the username must be associated with the root-system user group.

•If you are troubleshooting a single SDR, the username must be associated with the root-lr user group.

See Cisco IOS XR System Security Command Reference and Cisco IOS XR System Security Configuration Guide for information on users, usernames, and user groups.

CLI Access Through a Console Port

The first time a router is started, you must use a direct connection to the Console port to connect to the router and enter the initial configuration. See Cisco IOS XR Getting Started Guide for information on connecting to the router through a console port. When you use a direct connection to the Console port, CLI commands are entered at a terminal or at a computer running terminal emulation software. A direct Console port connection is useful for entering initial configurations and performing some debugging tasks.

CLI Access Through a Terminal Server

A terminal server connection provides a way to access the Console port from a remote location. A terminal server connection is used when you need to perform tasks that require Console port access from a remote location.

Connecting to a router through a terminal server is similar to directly connecting through the Console port. For both connection types, the physical connection takes place through the Console port. The difference is that the terminal server connects directly to the Console port, and you must use a Telnet session to establish communications through the terminal server to the router.

If you are unable to access the CLI through a terminal server, perform the following procedure.

SUMMARY STEPS

1. Disable flow control (XON/XOFF) on the Terminal Server.

2. Disable local echo mode on the Terminal Server.

3. Verify the router name configured using the hostname command.

4. Check whether the port address is configured correctly.

5. Verify whether the address (interface) used for the reverse Telnet is up/up. The output of the show interfaces brief command provides this information. Cisco recommends you to use loopbacks because they are always up.

6. Ensure that you have the correct type of cabling. For example, you must not use a crossover cable to extend the length.

7. Establish a Telnet connection to the IP address port to test direct connectivity. You must Telnet from both an external device and the terminal server. For example, telnet 172.21.1.1 2003.

8. Ensure that you have the transport input telnet command under the line for the target device. The target device is the device that is connected to the terminal server.

9. Use a PC/dumb terminal to connect directly to the console of the target router. The target router is the device connected to the terminal server. This step helps you identify the presence of a port issue.

10. If you are disconnected, check timeouts. You can remove or adjust timeouts.

Note Note: If you encounter authentication failures, remember that the terminal server performs the first authentication (if configured), while the device to which you try to connect performs the second authentication (if configured). Verify whether AAA is configured correctly on both the terminal server and the connecting device.

11. Contact Cisco Technical Support if the problem is not resolved. See the "Obtaining Technical Assistance" section in the Preface for Cisco Technical Support contact information.

CLI Access Through the Management Ethernet Interface

The Management Ethernet interface allows you to manage the router using a network connection. Before you can use the Management Ethernet interface, the interface must be configured. See Cisco IOS XR Getting Started Guide for information on configuring the interface.

Once configured, the network connection takes place between client software on a workstation computer and a server process within the router. The type of client software you use depends on the server process you want to use. See Cisco IOS XR Getting Started Guide for information on the client and server services supported by the Cisco IOS XR software.

If you are unable to access the CLI through a management Ethernet interface, perform the following procedure.

SUMMARY STEPS

1. show interface MgmtEth interface-instance

2. show arp MgmtEth interface-instance

3. ping

4. Contact Cisco Technical Support if the problem is not resolved

DETAILED STEPS

Command or Action

Purpose

Step 1

show interfaces MgmtEth interface-instance
Example:

RP/0/RP0/CPU0:router# show interfaces MgmtEth 0/rp0/cpu0/0

Displays statistics for all interfaces configured on the router.

Check the following:

•MgmtEth interface is up

•Line protocol (state of the Layer 2 line protocol) is up

•Number of input and output errors

If an interface is down, use the no shutdown command to enable the interface.

If the interface is up and the input and output errors are within an acceptable range, proceed to Step 2.

If input or output errors are not within an acceptable range, the management Ethernet interface is not enabled when the no shutdown command is used, or the line protocol is down, see Chapter 5 "Troubleshooting Interfaces," for detailed information on troubleshooting interfaces.

Step 2

show arp MgmtEth interface-instance
Example:

RP/0/RP0/CPU0:router# show arp MgmtEth 0/rp0/cpu0/0

Displays the Address Resolution Protocol (ARP) table for the management Ethernet interface.

Ensure that the expected ARP entries exist for the management Ethernet interface.

If the expected ARP entries do not exist, verify the physical layer Ethernet interface connectivity. Use the show arp trace command to display the ARP entries in the buffer.

See the Chapter 5 "Troubleshooting Interfaces," for more information on troubleshooting interfaces.

If the expected ARP entries exist, proceed to Step 3.

Step 3

ping ip-address
Example:

RP/0/RP0/CPU0:router# ping 172.16.0.1 count 10 source MgmtEth 0/RP0/CPU0

Checks host reachability and network connectivity of the neighbor over the management Ethernet interface.

Try pinging the IP address of the target neighbor to see if it responds.

If the target neighbor does not respond, proceed to Step 4

Step 4

Contact Cisco Technical Support.

If the problem is not resolved, contact Cisco Technical Support. For Cisco Technical Support contact information, see the "Obtaining Technical Assistance" section in the Preface.

Examples

The output from the show interfaces MgmtEth command displays the status of the management Ethernet interface. For example, in the following output the management Ethernet interface is up. MgmtEth0/RP1/CPU0/0 is up indicates that the interface hardware is currently active and line protocol is up indicates that the keep a lives are successful. There are 42 input errors and 0 output errors.
RP/0/RP0/CPU0:router# show interfaces MgmtEth 0/rp1/cpu0/0 
MgmtEth0/RP1/CPU0/0 is up, line protocol is up 
  Hardware is Management Ethernet, address is 0011.93ef.e8fe (bia 0011.93ef.e8fe
)
  Description: Connected to Lan1 LAN
  Internet address is 172.29.52.71/24
  MTU 1514 bytes, BW 100000 Kbit
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA,
  Half-duplex, 100Mb/s, link type is autonegotiation
  loopback not set,
  ARP type ARPA, ARP timeout 04:00:00
  Last clearing of "show interface" counters never
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 1000 bits/sec, 1 packets/sec
     122444 packets input, 7450512 bytes, 45 total input drops
     0 drops for unrecognized upper-level protocol
     Received 98306 broadcast packets, 0 multicast packets
              0 runts, 0 giants, 0 throttles, 0 parity
     42 input errors, 37 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     26741 packets output, 5100214 bytes, 0 total output drops
     Output 48 broadcast packets, 0 multicast packets
     0 output errors, 0 underruns, 0 applique, 0 resets
     0 output buffer failures, 0 output buffers swapped out
     10 carrier transitions
The output from the show arp MgmtEth 0/rp1/cpu0/0 command displays the ARP table for the management Ethernet interface.
RP/0/RP0/CPU0:router# show arp MgmtEth 0/rp1/cpu0/0 
-------------------------------------------------------------------------------
0/RP1/CPU0
-------------------------------------------------------------------------------
Address         Age        Hardware Addr   State      Type  Interface
10.86.154.82    01:32:10   0030.f2f2.1038  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.128   03:55:55   0013.c4cb.a200  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
10.21.82.85     00:09:17   0030.f2f2.1038  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.20.212.227  03:51:56   0030.f2f2.1038  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.18.196.200  02:32:14   0030.f2f2.1038  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.201   00:07:54   0010.7b3c.6847  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.200   00:09:56   0010.7b3c.689f  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.19.16.196   00:11:19   0030.f2f2.1038  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.1     00:10:41   0030.f2f2.1038  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.121   01:41:03   0012.da0b.97ff  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.120   01:41:20   0004.2892.c7ff  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.127   03:55:56   0013.c4cb.a200  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.71    -          0011.93ef.e8fe  Interface  ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.70    00:26:29   0011.93ef.e8ea  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.73    01:42:04   0014.a9bc.6600  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.72    -          0011.93ef.e8fe  Interface  ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.75    00:11:14   0011.93ef.e8e2  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.77    00:11:14   0011.93ef.e8e2  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.76    02:12:54   0011.93ef.e8e6  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.29.52.78    01:42:02   0014.a8de.6700  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.23.105.135  03:15:34   0030.f2f2.1038  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.22.45.18    01:00:50   0030.f2f2.1038  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.22.45.17    01:09:39   0030.f2f2.1038  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.23.93.112   00:22:35   0030.f2f2.1038  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.22.58.32    00:22:32   0030.f2f2.1038  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
172.27.90.107   02:58:54   0030.f2f2.1038  Dynamic    ARPA  MgmtEth0/RP1/CPU0/0
Use the output from the show arp MgmtEth 0/rp1/cpu0/0 command to verify that there are dynamic ARP addresses in the table and that ARP is functioning over the interface. The output shows that ARP is functioning over the management Ethernet interface 0/RP1/CPU0.

The ping command checks to see if the neighbor is reachable.
RP/0/RP0/CPU0:router# ping 172.16.0.1 count 10 source mgmteth0/rp0/cpu0/0
Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 172.16.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 1/2/4 ms
Basic Cisco IOS XR Verification and Troubleshooting Tools

The following commands are used to collect information to aid in verifying the system and troubleshooting problems:

•man Command

•describe Command

•show platform Command

•top Command

•show context Command

•show users Command

man Command

The man command provides online help for standard Cisco IOS XR command-line interface (CLI) commands using manual (man) pages. The command is used to display the manual pages for a specific command on the basis of the command name, a feature, or a keyword. Each man page contains the command name, syntax, command mode, usage, examples, and related commands.

The following example shows typical example output from the man command show users command.
RP/0/RP0/CPU0:router# man command show users 
COMMAND
    show users
DESCRIPTION
  To display information about the active lines on the router, use the show
  users command in EXEC mode.
  show users
SYNTAX DESCRIPTION
  This command has no arguments or keywords.
DEFAULTS
  No default behavior or values
COMMAND MODES
  EXEC
COMMAND HISTORY
Release
          Modification
  Release 2.0
          This command was introduced on the Cisco CRS-1.
  Release 3.0
          No modification.
  Release 3.2
          This command was first supported on the Cisco XR 12000 Series Router.
USAGE GUIDELINES
  To use this command, you must be in a user group associated with a task group
  that includes the proper task IDs. For detailed information about user groups
  and task IDs, see the Configuring AAA Services on Cisco IOS XR Software
  module of the Cisco IOS XR System Security Configuration Guide.
  Use the show users command to display the line number, connection name, idle
  time, hosts, and terminal location. An asterisk (*) indicates the current
  terminal session.
EXAMPLES
  The following is sample output identifying an active vty terminal session:
* * * * * * * * * * * * * * * * START OF LISTING * * * * * * * * * * * * * * * *
RP/0/RP0/CPU0:router# show users
 * * * * * * * * * * * * * * * * END OF LISTING * * * * * * * * * * * * * * * *
Line User Service Conns Idle Location
con0_RP0_CPU0 cisco hardware 0 18:33:48
vty0 cisco telnet 0 00:30:36 10.33.54.132
* vty1 cisco telnet 0 00:00:00 10.33.54.132
 * * * * * * * * * * * * * * * * END OF LISTING * * * * * * * * * * * * * * * *
  Table 89 describes the significant fields shown in the display.
Table^B^`89 show users Field Descriptions^B^`
  Field
          Description
  Line
          All current connections. An asterisk (*) indicates the active
          connection.
  User
          Username of the user logged into the line.
  Service
          Physical or remote login service used.
  Conns
          Number of outgoing connections.
  Idle
          Interval (in hours:minutes:seconds) since last keystroke.
  Location
          IP address of remote login host. For local (physical) terminal
          connections, this field is blank.
RELATED COMMANDS
  Command
          Description
  show line
          Displays the parameters of a terminal line.
describe Command

The describe command provides package, component, and task ID information for a specific command. You must be in the appropriate configuration mode for the specific command. For example, to display the package, component, and task ID information for the router bgp 1 command, you must be in global configuration mode.

The following example shows typical example output from the describe router bgp 1 command.
RP/0/RP0/CPU0:router(config)# describe router bgp 1
Package:
    hfr-rout
        hfr-rout V3.3.0 Routing Package
        Vendor : Cisco Systems
        Desc   : Routing Package
        Build  : Built on Tue Jan 31 10:56:38 UTC 2006
        Source : By edde-bld1 in /files/3.3.0/hfr/workspace fo8
        Card(s): RP, DRP, DRPSC
Component:
    ipv4-bgp V[fwd-33/53]  IPv4 Border Gateway Protocol (BGP)
User needs ALL of the following taskids:
        bgp (READ WRITE) 
show platform Command

The show platform command displays a high level overview of the entire physical system. Use the show platform command in administration mode to display a summary of the nodes in the system, including node type and status.

Note The show platform command in EXEC mode displays a high level overview of the specific secure domain router (SDR).

The following example shows typical example output from the show platform command in administration mode.
RP/0/RP0/CPU0:router(admin)# show platform
Node            Type            PLIM            State           Config State
-----------------------------------------------------------------------------
0/1/SP          MSC(SP)         N/A             IOS XR RUN      PWR,NSHUT,MON
0/1/CPU0        MSC             Jacket Card     IOS XR RUN      PWR,NSHUT,MON
0/1/0           MSC(SPA)        4XOC3-POS       OK              PWR,NSHUT,MON
0/1/5           MSC(SPA)        8X1GE           OK              PWR,NSHUT,MON
0/6/SP          MSC(SP)         N/A             FDIAG RUNNING   PWR,NSHUT,MON
0/6/CPU0        MSC             Jacket Card     FDIAG RUNNING   PWR,NSHUT,MON
0/RP0/CPU0      RP(Active)      N/A             IOS XR RUN      PWR,NSHUT,MON
0/RP1/CPU0      RP(Standby)     N/A             IOS XR RUN      PWR,NSHUT,MON
0/SM0/SP        FC/S(SP)        N/A             IOS XR RUN      PWR,NSHUT,MON
0/SM1/SP        FC/S(SP)        N/A             IOS XR RUN      PWR,NSHUT,MON
0/SM2/SP        FC/S(SP)        N/A             IOS XR RUN      PWR,NSHUT,MON
0/SM3/SP        FC/S(SP)        N/A             IOS XR RUN      PWR,NSHUT,MON
top Command

The top command is used to monitor CPU usage on the system through interactive process statistics.

The following example show the output from the top command.
RP/0/RP0/CPU0:router# top 
Computing times...
224 processes; 803 threads;
CPU states: 93.6% idle, 3.6% user, 2.7% kernel
Memory: 4096M total, 3504M avail, page size 4K
   JID   TID PRI STATE  HH:MM:SS      CPU  COMMAND
 65555     2  10 Rcv     4:59:34     1.51% devb-ata
     1     6  10 Run     0:15:01     1.20% procnto-600-smp-cisco-instr
     1    13  10 Rcv     0:39:58     1.03% procnto-600-smp-cisco-instr
    57     5  10 Rcv     0:27:47     0.53% dllmgr
     1    28  10 Rcv     0:34:59     0.47% procnto-600-smp-cisco-instr
 65756     1  10 Rply    0:00:00     0.20% top
 65555     7  10 Rcv     0:00:00     0.10% devb-ata
    59     7  55 Rcv     0:22:50     0.09% eth_server
    59     9  10 Rcv     0:05:13     0.09% eth_server
   319     5  10 Rcv     0:15:38     0.07% shelfmgr
Press `q' to exit the command.

show context Command

The show context command displays core dump context information for the last ten core dumps. The command output is used for post-analysis in the debugging of processes (determine if any process crashes have occurred).

If there are no crashed processes, the show context command displays no output for each node. The following example shows typical example output of the show context command with no crashed processes.
RP/0/RP1/CPU0:router# show context 
node:      node0_1_CPU0
------------------------------------------------------------------
node:      node0_6_CPU0
------------------------------------------------------------------
node:      node0_RP0_CPU0
------------------------------------------------------------------
node:      node0_RP1_CPU0
------------------------------------------------------------------
The following example shows the output from the show context command where there is a crashed process.

Note Use the dumpcore command to generate a core dump.
RP/0/RP1/CPU0:router# show context 
node:      node0_1_CPU0
------------------------------------------------------------------
Crashed pid = 61524 (pkg/bin/tcam_mgr)
Crashed tid = 1
Crash time: Wed Apr 05, 2006: 18:27:26
Core for process at harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z
    Stack Trace
#0 0xfc1d3fa0
#1 0xfc1c6340
#2 0xfc1c5364
#3 0xfc1c542c
#4 0x48210930
#5 0x482110b8
#6 0x48212ba4
#7 0x48203dd8
#8 0x4820c61c
#9 0xfc1557ec
#10 0xfc15573c
#11 0xfc152fb8
#12 0x4820d140
                Registers info 
            r0       r1       r2       r3
  R0   00000000  481ff7b0  4824a55c  00000000  
            r4       r5       r6       r7
  R4   0000f054  00000001  00000006  00000000  
            r8       r9      r10      r11
  R8   00000000  fc220000  481fffc0  00000000  
           r12      r13      r14      r15
  R12  4823be90  4824a4a0  48230000  00000000  
           r16      r17      r18      r19
  R16  00000048  00000001  00000019  48256520  
           r20      r21      r22      r23
  R20  00000000  00000000  00000003  00000045  
           r24      r25      r26      r27
  R24  00000003  00000000  00000003  4825dc34  
           r28      r29      r30      r31
  R28  00000006  0000f054  48254064  481ff810  
           cnt       lr      msr       pc
  R32  00000000  fc1c6340  0000d932  fc1d3fa0  
           cnd      xer
  R36  28004024  00000008  
                      DLL Info 
DLL path    Text addr.  Text size   Data addr. Data size   Version
/hfr-os-3.3.90/lib/libinfra.dll  0xfc142000  0x00034200  0xfc1343b8  0x00000bbc 
       0
/lib/libc.dll  0xfc1a8000  0x00079dd8  0xfc222000  0x00002000        0
    Crash Package Infomation
Package: hfr-mgbl, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-mcast, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wo
rkspace for c2.95.3-p8
Package: hfr-mpls, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-rout, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-k9sec, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wo
rkspace for c2.95.3-p8
Package: hfr-lc, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/works
pace for c2.95.3-p8
Package: hfr-fwdg, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-admin, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wo
rkspace for c2.95.3-p8
Package: hfr-base, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-os-mbi, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/w
orkspace for c2.95.3-p8
node:      node0_6_CPU0
------------------------------------------------------------------
node:      node0_RP0_CPU0
------------------------------------------------------------------
node:      node0_RP1_CPU0
------------------------------------------------------------------
Use the show context command to locate the core dump file path. For example, the core dump file path shown in the command output is: harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z. The command output shows a crashed on a node. The process is pkg/bin/tcam_mgr.

Collect the following information and sent it to Cisco Technical Support. For Cisco Technical Support contact information, see the "Obtaining Technical Assistance" section in the Preface.

•ppc.Z file—This file contains the binary core dump information. Use the path listed in the command output to copy the contents of the ppc.Z file. The path shown in the command output is: harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z

•ppc.txt file—This file contains content on the core dump similar to the show context command output. Use the path listed in the command output to copy the contents of the ppc.txt file. The path shown in the command output is: harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.txt

•Collect the show version or show install active command output.

show users Command

The show users command displays information on active lines on the router including the line number, user, service, number of connections, idle time, and remote terminal location. An asterisk (*) indicates the current terminal session.

The following example shows the output from the show users command.
RP/0/RP0/CPU0:router# show users
   Line            User                 Service  Conns   Idle        Location
*  vty0            User_A               telnet       0  00:00:00     161.44.1925
   vty1            User-B               telnet       0  00:00:03     161.44.1929
Verifying the System

To verify the general status and state of a router using Cisco IOS XR software, perform the following procedure.

SUMMARY STEPS

1. admin

2. show platform [node-id]

3. show version

4. show running-config

5. show logging

6. show environment

7. show context

8. exit

9. show context

10. show memory summary detail location all

11. show memory heap summary {job-id | all}

12. top processes

13. show running-config

14. show system verify start
show system verify report

15. show {ipv4 | ipv6} interface brief

DETAILED STEPS

Command or Action

Purpose

Step 1

admin
Example:

RP/0/RP0/CPU0:router# admin

Enters administration mode.

Step 2

show platform [node-id]
Example:

RP/0/RP0/CPU0:router(admin)# show platform

Displays information about the status of cards and modules installed in the router.

•Some cards support a CPU module and service processor (SP) module. Other cards support only a single module.

•A card module is also called a node. When all nodes are working properly, the status of each node displayed in the State column is IOS-XR RUN.

•Type the show platform node-id command to display information for a specific node. Replace node-id with a node name from the show platform command Node column.

Step 3

show version
Example:

RP/0/RP0/CPU0:router(admin)# show version

Displays information about the router, including image names, uptime, and other system information.

Verify that the expected software version and images are installed.

Step 4

show running-config
Example:

RP/0/RP0/CPU0:router(admin)# show running-config

Displays hardware module power status, secure domain router (SDR) configuration, and fabric configuration. The output also displays the users defined in administration mode with root-system access.

For Cisco CRS-1 Multishelf Systems, it displays the rack numbers and serial numbers for the nodes in the currently running administration configuration.

Verify that the rack numbers and serial numbers for the nodes in the current running configuration are what is expected. The expected rack numbers and serial numbers should be listed in the current system documentation. See the "Troubleshooting Techniques and Approaches" section.

Also verify that the hardware module power status is as expected and the SDR and fabric configurations are as expected.

Step 5

show logging
Example:

RP/0/RP0/CPU0:router(admin)# show logging

Displays all syslog messages stored in the buffer. The command output displays the device operation history from a system perspective.

Analyze the logged events and their order of happening. Check for anything out of the ordinary such as errors, tracebacks, or crashes. Also check for any Severity 1 or Severity 2 errors.

Step 6

show environment
Example:

RP/0/RP0/CPU0:router(admin)# show environment

Displays display environmental monitor parameters for the system.

Verify that the parameters are as expected.

Step 7

show context
Example:

RP/0/RP0/CPU0:router(admin)# show context

Displays core dump context information on fabric cards, alarm modules, fan controllers, and service processors (system-owned cards). See the "show context Command" section for more information on the show context command output.

Step 8

exit
Example:

RP/0/RP0/CPU0:router(admin)# exit

Exits administration mode.

Step 9

show context
Example:

RP/0/RP0/CPU0:router# show context

Displays core dump context information on CPUs responsible for routing and Cisco Express Forwarding (CEF). See the "show context Command" section for more information on the show context command output.

Step 10

show memory summary detail location all
Example:

RP/0/RP0/CPU0:router# show memory summary detail location all

Displays information about the memory available on the router after the system image decompresses and loads.

Verify that the expected memory is available or installed. Ensure that all memory regions have adequate free space available.

Step 11

show memory heap summary {job-id | all}
Example:

RP/0/RP0/CPU0:router# show memory heap summary all

Displays a summary of the information about the heap space. The output displays each process and the amount of memory allocated for each process.

Verify if there are any processes using a large amount of memory.

Step 12

top processes
Example:

RP/0/RP0/CPU0:router# top processes

To get a live update of process resource consumption, use the top processes command and press `M' to sort by memory usage.

Verify that the resource consumption is as expected.

Step 13

show running-config
Example:

RP/0/RP0/CPU0:router# show running-config

Displays the contents of the currently running configuration.

Verify that the contents of the current running configuration are what is expected.

Step 14

show system verify start
show system verify report
Example:

RP/0/RP0/CPU0:router# show system verify start
RP/0/RP0/CPU0:router# show system verify report

A two-step command that produces system reports.

•show system verify start—Starts the system verify process (creates the initial baseline file)

•show system verify report—Generates a report for the system verification process (report of the current status)

The output of the show system verify report command provides a comparison of the system at the time of the show system verify start snapshot and the show system verify report snapshot. The output provides a sanity check of the system provided the show system verify start system snapshot was taken when the system was healthy or before an event.

Verify that the system parameters are as expected.

Step 15

show (ipv4 | ipv6} interface brief
Example:

RP/0/RP0/CPU0:router# show ipv4 interface brief

Displays the usability status of interfaces.

Verify that all expected interfaces are listed, that they have the correct assigned address, and that they are in the expected states.

Examples

The output from the show platform command indicates that all expected nodes are in the run state. If all nodes in the system are active, the cards should be in the IOS XR RUN and the SPAs should be in the OK state.
RP/0/RP0/CPU0:router(admin)# show platform
Node            Type            PLIM            State           Config State
-----------------------------------------------------------------------------
0/1/SP          MSC(SP)         N/A             IOS XR RUN      PWR,NSHUT,MON
0/1/CPU0        MSC             Jacket Card     IOS XR RUN      PWR,NSHUT,MON
0/1/0           MSC(SPA)        4XOC3-POS       OK              PWR,NSHUT,MON
0/1/5           MSC(SPA)        8X1GE           OK              PWR,NSHUT,MON
0/6/SP          MSC(SP)         N/A             IOS XR RUN      PWR,NSHUT,MON
0/6/CPU0        MSC             Jacket Card     IOS XR RUN      PWR,NSHUT,MON
0/6/0           MSC(SPA)        4XOC3-POS       OK              PWR,NSHUT,MON
0/6/4           MSC(SPA)        8XOC3/OC12-POS  OK              PWR,NSHUT,MON
0/6/5           MSC(SPA)        8X1GE           OK              PWR,NSHUT,MON
0/RP0/CPU0      RP(Active)      N/A             IOS XR RUN      PWR,NSHUT,MON
0/RP1/CPU0      RP(Standby)     N/A             IOS XR RUN      PWR,NSHUT,MON
0/SM0/SP        FC/S(SP)        N/A             IOS XR RUN      PWR,NSHUT,MON
0/SM1/SP        FC/S(SP)        N/A             IOS XR RUN      PWR,NSHUT,MON
0/SM2/SP        FC/S(SP)        N/A             IOS XR RUN      PWR,NSHUT,MON
0/SM3/SP        FC/S(SP)        N/A             IOS XR RUN      PWR,NSHUT,MON
The example output shows that all expected nodes are in the run state.

The output from the show version command indicates the version of software being run on the nodes and from which location (disk or network). Check that the expected software version and images are installed.
RP/0/RP0/CPU0:router(admin)# show version
Cisco IOS XR Software, Version 3.3.0 
Copyright (c) 2006 by cisco Systems, Inc.
ROM: System Bootstrap, Version 1.32(20050525:193559) [CRS-1 ROMMON],  
CRS-8_X1 uptime is 1 week, 6 days, 23 hours, 5 minutes
System image file is "disk0:hfr-os-mbi-3.3.0/mbihfr-rp.vm"
cisco CRS-8/S (7457) processor with 4194304K bytes of memory.
7457 processor at 1197Mhz, Revision 1.2
16 Packet over SONET/SDH network interface(s)
16 SONET/SDH Port controller(s)
2 Ethernet/IEEE 802.3 interface(s)
16 GigabitEthernet/IEEE 802.3 interface(s)
2043k bytes of non-volatile configuration memory.
38079M bytes of hard disk.
1000592k bytes of ATA PCMCIA card at disk 0 (Sector size 512 bytes).
1000640k bytes of ATA PCMCIA card at disk 1 (Sector size 512 bytes).
Package active on node 0/1/SP:
hfr-diags, V 3.3.0, Cisco Systems, at disk0:hfr-diags-3.3.0
    Built on Tue Jan 31 13:32:40 UTC 2006
    By edde-bld1 in /files/3.3.0/workspace for c2.95.3-p8
hfr-admin, V 3.3.0, Cisco Systems, at disk0:hfr-admin-3.3.0
    Built on Tue Jan 31 10:02:02 UTC 2006
    By edde-bld1 in /files/3.3.0/workspace for c2.95.3-p8
.
.
.
The example output shows that the Cisco IOS XR software version is 3.3.0 and that the installed pie versions are also 3.3.0.

The output from the show running-config command displays hardware module power status, secure domain router (SDR) configuration, fabric configuration, and rack numbers and serial numbers. The rack and serial numbers are displayed for Cisco CRS-1 Multishelf Systems only.
RP/0/RP0/CPU0:router(admin)# show running-config
Building configuration...
username user_A
 secret 5 $1$SopS$KK2gkdTQYDTKMbWMILZ5P1
 group root-system
!
dsc serial TBA08260159 rack 0
dsc serial TBA08440024 rack 1
dsc serial TBC0820052000000 rack 240
controllers fabric plane 0 topology single-module location F0/SM0/FM
controllers fabric plane 1 topology single-module location F0/SM1/FM
controllers fabric plane 4 topology single-module location F0/SM4/FM
controllers fabric plane 5 topology single-module location F0/SM5/FM
end
The example output shows the contents of the current running configuration for a Cisco CRS-1 Multishelf System.

The output from the show logging command displays the contents of the logging buffer. The output displays details on syslog historical events. Analyze the logged events and their order of happening. Check for anything out of the ordinary such as errors, tracebacks, or crashes. Also check for any Severity 1 or Severity 2 errors.
RP/0/RP0/CPU0:router(admin)# show logging
Syslog logging: enabled (63 messages dropped, 0 flushes, 0 overruns)
    Console logging: level informational, 16647 messages logged
    Monitor logging: level debugging, 0 messages logged
    Trap logging: level informational, 0 messages logged
    Buffer logging: level debugging, 16695 messages logged
Log Buffer (16384 bytes):
RP/0/RP0/CPU0:Aug 29 06:49:10.642 : exec[65714]: %SECURITY-login-4-AUTHEN_FAILED
 : Failed authentication attempt by user 'user_A' from '161.44.192.231' 
RP/0/RP0/CPU0:Aug 29 08:45:58.249 : config[65771]: %MGBL-SYS-5-CONFIG_I : Config
ured from console by user_A on vty0 (161.44.212.109) 
RP/0/RP0/CPU0:Aug 29 08:57:51.183 : exec[65755]: %MGBL-exec-3-LOGIN_AUTHEN : Log
in Authentication failed. Exiting... 
LC/0/1/CPU0:Aug 30 20:17:34.692 : obflmgr[66]: %OS-OBFLMGR-6-COALESCE_START : Co
alescing started for feature:temperature on device:nvram:(nodeid:0x11). Continuo
us file size:65522 limit:65536. Historical file size:10864 limit:32768. 
LC/0/6/CPU0:Aug 30 21:04:18.913 : obflmgr[66]: %OS-OBFLMGR-6-COALESCE_START : Co
alescing started for feature:temperature on device:nvram:(nodeid:0x61). Continuo
us file size:65512 limit:65536. Historical file size:12976 limit:32768. 
RP/0/RP0/CPU0:Aug 31 06:47:56.740 : config[65716]: %MGBL-SYS-5-CONFIG_I : Config
ured from console by user_B on vty0 (10.86.240.72) 
RP/0/RP0/CPU0:Aug 31 07:44:12.233 : config[65716]: %MGBL-SYS-5-CONFIG_I : Config
ured from console by user_B on vty0 (10.86.240.72) 
RP/0/RP0/CPU0:Aug 31 07:45:31.728 : config[65716]: %MGBL-SYS-5-CONFIG_I : Config
ured from console by user_B on vty0 (10.86.240.72) 
RP/0/RP1/CPU0:Aug 31 10:13:09.415 : obflmgr[68]: %OS-OBFLMGR-6-COALESCE_START : 
Coalescing started for feature:temperature on device:bootflash:(nodeid:0x211). C
ontinuous file size:65513 limit:65536. Historical file size:6444 limit:32768. 
RP/0/RP0/CPU0:Aug 31 11:46:49.356 : config[65771]: %MGBL-SYS-5-CONFIG_I : Config
ured from console by user_B on vty1 (10.86.240.72) 
RP/0/RP0/CPU0:Sep  1 03:10:13.537 : obflmgr[68]: %OS-OBFLMGR-6-COALESCE_START : 
Coalescing started for feature:temperature on device:bootflash:(nodeid:0x201). C
ontinuous file size:65520 limit:65536. Historical file size:4164 limit:32768. 
RP/0/RP0/CPU0:Sep  1 07:39:13.096 : config[65755]: %MGBL-SYS-5-CONFIG_I : Config
ured from console by user_B on vty1 (161.44.192.236) 
SP/0/SM1/SP:Sep  1 11:48:46.765 : obflmgr[66]: %OS-OBFLMGR-6-COALESCE_START : Co
alescing started for feature:temperature on device:configflash:(nodeid:0x810). C
ontinuous file size:65512 limit:65536. Historical file size:0 limit:32768. 
SP/0/SM1/SP:Sep  5 02:53:03.819 : obflmgr[66]: %OS-OBFLMGR-6-COALESCE_START : Co
alescing started for feature:temperature on device:configflash:(nodeid:0x810). C
ontinuous file size:65512 limit:65536. Historical file size:1392 limit:32768. 
SP/0/SM0/SP:Sep  5 02:53:18.837 : obflmgr[66]: %OS-OBFLMGR-6-COALESCE_START : Co
alescing started for feature:temperature on device:configflash:(nodeid:0x800). C
ontinuous file size:65512 limit:65536. Historical file size:1716 limit:32768. 
The example output shows the level of logging. For example, the level of buffer logging is debugging. Ensure that the appropriate logging levels are available for each type of logging (console, monitor, trap, and buffer).

The output from the show environment command displays environmental monitor parameters for the system. Verify that the environment parameters are as expected. Environment parameter anomalies are logged in the syslog, so if an environment parameter displayed in the show environment command output is not as expected, check the syslog using the show logging command. The syslog will provide details on any logged problems.
RP/0/RP0/CPU0:router(admin)# show environment 
Temperature Information
---------------------------------------------
R/S/I   Modules         Inlet           Exhaust         Hotspot
                        Temperature     Temperature     Temperature
                        (deg C)         (deg C)         (deg C)
0/1/*
        host              29, 28         25, 25         32           
        cpu                                             31           
        fabricq0                                        26           
        fabricq1                                        29           
        ingressq                                        33           
        egressq                          30             25           
        ingresspse                                      32           
        egresspse                                       27           
        jacket            23             23             23           
        spa0              18                            25, 30       
        spa5              23                            23           
.
.
.
0/RP0/*
        host              22             23             22, 31, 25,  
                                                            23, 25        
0/RP1/*
        host              22             22             23, 30, 25,  
                                                            23, 25        
.
.
.
0/SM3/*
        host              39, 39                        36, 44       
Threshold Information
---------------------------------------------
R/S/I   Modules Sensor          Minor           Major           Critical
                                (Lo Hi)         (Lo Hi)         (Lo Hi)         
0/1/*
        host    Inlet0          0,57            -10,67          -15,76          
        host    Inlet1          0,64            -10,74          -15,83          
        host    Exhaust0        0,57            -10,67          -15,76          
        host    Exhaust1        0,59            -10,69          -15,78          
        host    Hotspot0        0,63            -10,73          -15,82          
        cpu     Hotspot0        0,62            -10,81          -15,91          
        fabricq Hotspot0        0,59            -10,70          -15,79          
        fabricq Hotspot0        0,62            -10,73          -15,82          
        ingress Hotspot0        0,69            -10,79          -15,88          
        egressq Exhaust0        0,64            -10,76          -15,85          
        egressq Hotspot0        0,59            -10,73          -15,82          
        ingress Hotspot0        0,69            -10,80          -15,89          
        egressp Hotspot0        0,60            -10,74          -15,83          
        jacket  Inlet0          0,52            -10,60          -15,70          
        jacket  Exhaust0        0,60            -10,70          -15,80          
        jacket  Hotspot0        0,65            -10,75          -15,85          
        spa0    Inlet0          0,50            -10,60          -15,70          
        spa0    Hotspot0        0,64            -10,74          -15,84          
        spa0    Hotspot1        0,69            -10,79          -15,88          
        spa5    Inlet0          0,58            -10,71          -15,81          
        spa5    Hotspot0        0,60            -10,73          -15,83          
.
.
.
0/RP0/*
        host    Inlet0          0,46            -10,55          -15,72          
        host    Hotspot0        0,46            -10,57          -15,74          
        host    Hotspot1        0,55            -10,67          -15,84          
        host    Hotspot2        0,50            -10,63          -15,79          
        host    Hotspot3        0,49            -10,57          -15,74          
        host    Hotspot4        0,52            -10,66          -15,80          
        host    Exhaust0        0,47            -10,61          -15,77          
.
.
.
0/SM1/*
        host    Inlet0          0,68            -10,74          -15,82          
        host    Inlet1          0,66            -10,72          -15,81          
        host    Hotspot0        0,59            -10,67          -15,77          
        host    Hotspot1        0,66            -10,72          -15,81          
0/SM1/*
        host    3.3V            2970,3630       2805,3795       2640,3960       
        host    1.8V            1620,1980       1530,2070       1440,2160       
        host    1.8V            1620,1980       1530,2070       1440,2160       
        host    2.5V            2250,2750       2125,2875       2000,3000       
        host    2.5V            2250,2750       2125,2875       2000,3000       
.
.
.
Voltage Information
---------------------------------------------
R/S/I   Modules 1.2V    1.25V   1.5V    1.6V    1.8V    2.5V    3.3V    5V      
                (mv)    (mv)    (mv)    (mv)    (mv)    (mv)    (mv)    (mv)
0/1/*
        host            1254    1494            1790    2548    3250    5000    
                        1240                            2548    3336    5018    
                                                                        5018    
        cpu                             1410    1818    2535    3336    4966    
        fabq0   1261    1238                            2538            5000    
                        1240                                            5018    
        fabq1   1261    1238                            2538            5000    
                        1240                                            4992    
        ingq    1248    1238                            2552            5000    
                        1240                                            4992    
        egrq    1248                                    2538            5000    
        i-pse   1261                                    2538            5000    
        e-pse   1261                                    2538            5000    
        jacket                  1530                    2522    3316            
        spa0                    1510                    2510    3307            
        spa5                    1522                    2535    3307            
                                1526                            3308            
.
.
.
0/RP0/*
        host    1254    1240    1508    1297    1794    2545    3302    5000    
                1254    1226                    1820    2535    3319    4992    
                                                                3285    4992    
0/RP1/*
        host    1254    1240    1508    1297    1807    2545    3328    5000    
                1254    1226                    1807    2535    3336    4992    
                                                                3268    4940    
0/SM0/*
        host                                    1762    2496    3302            
                                                1804    2496                    
0/SM1/*
        host                                    1762    2483    3285            
                                                1804    2496                    
0/SM2/*
        host                                    1762    2496    3285            
                                                1804    2496                    
0/SM3/*
        host                                    1776    2496    3285            
                                                1790    2470                    
LED Information
---------------------------------------------
0/1/*: Module (host) LED status says: OK
0/1/*: Module (jacket) LED status says: OK
0/1/*: Module (spa0) LED status says: OK
0/1/*: Module (spa5) LED status says: OK
0/6/*: Module (host) LED status says: OK
0/6/*: Module (jacket) LED status says: OK
0/6/*: Module (spa0) LED status says: OK
0/6/*: Module (spa4) LED status says: OK
0/6/*: Module (spa5) LED status says: OK
0/RP0/*: Module (host) LED status says: OK
0/RP0/*: Alarm LED status says: NONE
Rack 0: Upper Fan Tray: LED status : OK
Rack 0: Lower Fan Tray: LED status : OK
0/RP1/*: Module (host) LED status says: OK
0/RP1/*: Alarm LED status says: NONE
0/SM0/*: Module (host) LED status says: OK
0/SM1/*: Module (host) LED status says: OK
0/SM2/*: Module (host) LED status says: OK
0/SM3/*: Module (host) LED status says: OK
Fan Information
---------------------------------------------
  Fan speed (rpm):
        FAN1    FAN2    FAN3    FAN4
Rack 0:
 Upper  4842    4882    4842    4882
 Lower  4882    4842    4923    4842
Power Supply Information
---------------------------------------------
                Power-Supply            Voltage         Current
                                (V)             (A)
 Zone 1:        [A], [B]        54.867, 54.377   4.596,  4.387
 Zone 2:        [A], [B]        54.573, 53.985   6.894,  5.745
 Zone 3:        [A], [B]        55.161, 54.279   4.387,  4.491
 Total Current:   30.500 A
 Total Power  : 1677.500 W
The example output shows the system environmental monitor parameter variables.

The output from the show context command displays core dump context information. See the "show context Command" section for more information on the show context command output.
RP/0/RP0/CPU0:router# show context
node:      node0_1_CPU0
------------------------------------------------------------------
Crashed pid = 61524 (pkg/bin/tcam_mgr)
Crashed tid = 1
Crash time: Wed Apr 05, 2006: 18:27:26
Core for process at harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z
    Stack Trace
#0 0xfc1d3fa0
#1 0xfc1c6340
#2 0xfc1c5364
#3 0xfc1c542c
#4 0x48210930
#5 0x482110b8
#6 0x48212ba4
#7 0x48203dd8
#8 0x4820c61c
#9 0xfc1557ec
#10 0xfc15573c
#11 0xfc152fb8
#12 0x4820d140
                Registers info 
            r0       r1       r2       r3
  R0   00000000  481ff7b0  4824a55c  00000000  
            r4       r5       r6       r7
  R4   0000f054  00000001  00000006  00000000  
            r8       r9      r10      r11
  R8   00000000  fc220000  481fffc0  00000000  
           r12      r13      r14      r15
  R12  4823be90  4824a4a0  48230000  00000000  
           r16      r17      r18      r19
  R16  00000048  00000001  00000019  48256520  
           r20      r21      r22      r23
  R20  00000000  00000000  00000003  00000045  
           r24      r25      r26      r27
  R24  00000003  00000000  00000003  4825dc34  
           r28      r29      r30      r31
  R28  00000006  0000f054  48254064  481ff810  
           cnt       lr      msr       pc
  R32  00000000  fc1c6340  0000d932  fc1d3fa0  
           cnd      xer
  R36  28004024  00000008  
                      DLL Info 
DLL path    Text addr.  Text size   Data addr. Data size   Version
/hfr-os-3.3.90/lib/libinfra.dll  0xfc142000  0x00034200  0xfc1343b8  0x00000bbc 
       0
/lib/libc.dll  0xfc1a8000  0x00079dd8  0xfc222000  0x00002000        0
    Crash Package Infomation
Package: hfr-mgbl, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-mcast, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wo
rkspace for c2.95.3-p8
Package: hfr-mpls, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-rout, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-k9sec, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wo
rkspace for c2.95.3-p8
Package: hfr-lc, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/works
pace for c2.95.3-p8
Package: hfr-fwdg, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-admin, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wo
rkspace for c2.95.3-p8
Package: hfr-base, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-os-mbi, Source: By edde-bld1 in /vws/3.3.90.1I/hfr/w
orkspace for c2.95.3-p8
node:      node0_6_CPU0
------------------------------------------------------------------
node:      node0_RP0_CPU0
------------------------------------------------------------------
node:      node0_RP1_CPU0
------------------------------------------------------------------
The example output shows that the pkg/bin/tcam_mgr process crashed.

The output from the show memory command displays information about the memory available on the router after the system image decompresses and loads. Verify that the expected memory is available or installed. Ensure that all memory regions have adequate free space available.
RP/0/RP0/CPU0:router# show memory summary detail location all
Physical Memory: 4.000G total
 Application Memory : 3.857G (3.455G available)
 Image: 17.880M (bootram: 17.880M)
 Reserved: 128.000M, IOMem: 1.980G, flashfsys: 0
 Shared window infra_ital: 323.628K
 Shared window ipv4_fib: 1.003M
 Shared window ifc-mpls: 961.714K
 Shared window ifc-ipv6: 1.189M
 Shared window ifc-ipv4: 1.251M
 Shared window ifc-protomax: 641.714K
 Shared window infra_statsd: 3.714K
 Shared window aib: 203.687K
 Shared window PFI_IFH: 155.652K
 Shared window squid: 2.152M
 Shared window atc_cache: 35.671K
 Total shared window: 7.867M
 Allocated Memory: 170.406M
 Program Text: 21.242M
 Program Data: 1.761M
 Program Stack: 6.878M
The example output shows that there is 3.455 gigabits of application memory available.

The output from the show running-config command displays the current running configuration. Verify that the contents of the current running configuration are what is expected.
RP/0/RP0/CPU0:router# show running-config
Building configuration...
!! Last configuration change at 18:56:31 UTC Tue Feb 28 2006 by user_A
!
hostname CRS-8_X1
line console
 exec-timeout 120 0
 session-timeout 120
!
line default
 exec-timeout 120 0
 session-timeout 120
!
telnet vrf default ipv4 server max-servers no-limit
domain ipv4 host x1 172.16.52.72
domain ipv4 host x2 172.16.52.77
domain ipv4 host xe1 172.16.52.73
domain ipv4 host xe2 172.16.52.78
domain-lookup
vty-pool default 0 25
ipv4 virtual address 172.16.52.72 255.255.255.0
interface Loopback0
 ipv4 address 10.10.20.0 255.255.255.255
!
interface MgmtEth0/RP0/CPU0/0
 description Connected to RTR RTR
 ipv4 address 172.16.52.70 255.255.255.0
!
interface MgmtEth0/RP1/CPU0/0
 description Connected to lanA LAN
 ipv4 address 172.16.52.71 255.255.255.0
!
interface GigabitEthernet0/1/5/0
 description Connected to CRS-8_X2 GE 0/1/5/0
 ipv4 address 10.50.40.0 255.255.255.0
!
interface GigabitEthernet0/1/5/1
 description Connected to C12810_XF GE 5/2
 ipv4 address 10.50.56.0 255.255.255.0
 negotiation auto
!
interface GigabitEthernet0/1/5/2
 shutdown
!
interface GigabitEthernet0/1/5/3
 shutdown
!
interface GigabitEthernet0/1/5/4
 shutdown 
!
interface GigabitEthernet0/1/5/5
 shutdown
!
interface GigabitEthernet0/1/5/6
 shutdown
!
interface GigabitEthernet0/1/5/7
 shutdown
!
interface GigabitEthernet0/6/5/0
 description Connected to C7304_XR1 GE2
 ipv4 address 10.55.12.0 255.255.255.0
!
interface GigabitEthernet0/6/5/1
 description Connected to CRS-8_X2 GE 0/6/5/1
 ipv4 address 10.55.48.0 255.255.255.0
!
.
.
.
!         
controller SONET0/1/0/0
 delay trigger line 100
 clock source internal
!
controller SONET0/1/0/1
 clock source internal
!
controller SONET0/6/0/0
 delay trigger line 100
 clock source internal
!
controller SONET0/6/0/1
 clock source internal
!
controller SONET0/6/4/4
 clock source internal
!
controller SONET0/6/4/5
 clock source internal
!
controller SONET0/6/4/6
 clock source internal
!         
controller SONET0/6/4/7
 clock source internal
!
router static
 address-family ipv4 unicast
  0.0.0.0/0 172.16.52.1
 !
!
ssh server
end
The example output displays the contents of the current running configuration.

The show system verify start command starts the system verification process and the show system verify report generates the output from the system verification process. The output allows you to verify that the system parameters are as expected.
RP/0/RP0/CPU0:router# show system verify start
Storing initial router status ... 
done.
RP/0/RP0/CPU0:router#
RP/0/RP0/CPU0:router# show system verify report
Getting current router status ... 
System Verification Report
==========================
- Verifying Memory Usage
- Verified Memory Usage                                 : [OK]     
- Verifying CPU Usage
- Verified CPU Usage                                    : [OK]     
- Verifying Blocked Processes
- Verified Blocked Processes                            : [OK]     
- Verifying Aborted Processes
- Verified Aborted Processes                            : [OK]     
- Verifying Crashed Processes
- Verified Crashed Processes                            : [OK]     
- Verifying LC Status
- Verified LC Status                                    : [OK]     
- Verifying QNET Status
Unable to get current LC status info
- Verified QNET Status                                  : [FAIL]   
- Verifying GSP Fabric Status
- Verified GSP Fabric Status                            : [OK]     
- Verifying GSP Ethernet Status
- Verified GSP Ethernet Status                          : [OK]     
- Verifying POS interface Status
- Verified POS interface Status                         : [OK]     
- Verifying TenGigE interface Status
- Verified TenGigE interface Status                     : [OK]     
- Verifying TCP statistics
- Verified TCP statistics                               : [OK]     
- Verifying UDP statistics
  tcp_udp_raw WARNING messages for router 
  UDP Packets sent has not increased during this period.
- Verified UDP statistics                               : [WARNING]
- Verifying RAW statistics
- Verified RAW statistics                               : [OK]     
- Verifying RIB Status
- Verified RIB Status                                   : [OK]     
- Verifying CEF Status
- Verified CEF Status                                   : [OK]     
- Verifying CEF Consistency Status
- Verified CEF Consistency Status                       : [OK]     
- Verifying BGP Status
- Verified BGP Status                                   : [OK]     
- Verifying ISIS Status
- Verified ISIS Status                                  : [OK]     
- Verifying OSPF Status
- Verified OSPF Status                                  : [OK]     
- Verifying Syslog Messages
- Verified Syslog Messages                              : [OK]     
System may not be stable. Please look into WARNING messages.
The example output compares the system from the time the show system verify start command took the first snapshot to the snapshot taken of the system when the show system verify report command took the second snapshot and generated the comparison. If there are no changes, [OK] is displayed. If there are changes between the first and second snapshot, the specific change is noted and marked with [WARNING] or [FAIL].

The show interface brief command displays the usability status of the configured interfaces. Verify that all expected interfaces are listed. For an interface to be usable, both the interface hardware (Status) and line protocol must be up. The protocol is Up if the interface can provide two-way communication.
RP/0/RP0/CPU0:router# show ipv4 interface brief 
Interface                      IP-Address      Status                Protocol
Loopback0                      10.10.20.1      Up                    Up      
MgmtEth0/RP0/CPU0/0            172.29.52.70    Up                    Up      
POS0/1/0/0                     10.50.4.1       Up                    Up      
POS0/1/0/1                     10.50.32.1      Up                    Up      
POS0/1/0/2                     unassigned      Shutdown              Down    
POS0/1/0/3                     unassigned      Shutdown              Down    
GigabitEthernet0/1/5/0         10.50.40.1      Up                    Up      
GigabitEthernet0/1/5/1         10.50.56.1      Up                    Up      
GigabitEthernet0/1/5/2         unassigned      Shutdown              Down    
GigabitEthernet0/1/5/3         unassigned      Shutdown              Down    
GigabitEthernet0/1/5/4         unassigned      Shutdown              Down    
GigabitEthernet0/1/5/5         unassigned      Shutdown              Down    
GigabitEthernet0/1/5/6         unassigned      Shutdown              Down    
GigabitEthernet0/1/5/7         unassigned      Shutdown              Down    
POS0/6/0/0                     10.50.8.1       Up                    Up      
POS0/6/0/1                     10.50.36.1      Up                    Up      
POS0/6/0/2                     unassigned      Shutdown              Down    
POS0/6/0/3                     unassigned      Shutdown              Down    
POS0/6/4/0                     unassigned      Shutdown              Down    
POS0/6/4/1                     unassigned      Shutdown              Down    
POS0/6/4/2                     unassigned      Shutdown              Down    
POS0/6/4/3                     unassigned      Shutdown              Down    
POS0/6/4/4                     10.50.52.1      Up                    Up      
POS0/6/4/5                     10.50.28.1      Up                    Up      
POS0/6/4/6                     10.50.104.1     Up                    Up      
POS0/6/4/7                     10.50.44.1      Up                    Up      
GigabitEthernet0/6/5/0         10.50.12.1      Up                    Up      
GigabitEthernet0/6/5/1         10.50.48.1      Up                    Up      
GigabitEthernet0/6/5/2         unassigned      Shutdown              Down    
GigabitEthernet0/6/5/3         unassigned      Shutdown              Down    
GigabitEthernet0/6/5/4         unassigned      Shutdown              Down    
GigabitEthernet0/6/5/5         unassigned      Shutdown              Down    
GigabitEthernet0/6/5/6         unassigned      Shutdown              Down    
GigabitEthernet0/6/5/7         unassigned      Shutdown              Down    
MgmtEth0/RP1/CPU0/0            172.29.52.71    Up                    Up      
The example output displays IP addresses, status, and protocol status for each interface. The output shows that all assigned interfaces (interfaces that are configured with IP addresses) have a interface hardware status and line protocol status of up.

Validating and Troubleshooting Cisco IOS XR Software Package Installation

The Cisco IOS XR software is divided into software packages allowing you to select which features run on your router. Each package contains the components to perform a specific set of router functions, such as routing, security, or Modular Services card support. Bundles are groups of packages that can be downloaded as a set. For example, the Unicast Routing Core Bundle provides six packages for use on every router. See Cisco IOS XR Getting Started Guide for a list of all packages.

This section provides information on how to validate and troubleshoot the Cisco IOS XR software package installation. The following sections are provided:

•Verifying the Software Version

•Validating the Installation

Verifying the Software Version

To verify the Cisco IOS XR software version, perform the following procedure.

SUMMARY STEPS

1. show version

2. show install

DETAILED STEPS

Command or Action

Purpose

Step 1

show version
Example:

RP/0/RP0/CPU0:router# show version

Displays a variety of system information, including hardware and software version, router uptime, boot settings (configuration register), and active software.

Determine if all expected packages are installed and the current software versions are the expected versions.

If the expected packages are not installed or are not the expected version, install the correct package. See Cisco IOS XR Getting Started Guide for information on installing and upgrading Cisco IOS XR software packages.

Step 2

show install
Example:

RP/0/RP0/CPU0:router# show install

Displays a list of all installed and active packages on each node.

Determine if the expected packages are installed on each node.

If the software or active package versions are not as expected for a node, the package is not compatible with the node for which it is being activated, or the package being activated is not compatible with the current active software set, install the correct software or package on the node. See Cisco IOS XR Getting Started Guide for information on installing and upgrading Cisco IOS XR software packages.

The following example shows that the Cisco IOS XR software and active packages are version 3.3.0.
RP/0/RP0/CPU0:router# show version
Cisco IOS XR Software, Version 3.3.0
Copyright (c) 2006 by cisco Systems, Inc.
ROM: System Bootstrap, Version 1.32(20050525:193559) [CRS-1 ROMMON],  
CRS-8_X1 uptime is 2 weeks, 4 days, 23 hours, 27 minutes
System image file is "disk0:hfr-os-mbi-3.3.0/mbihfr-rp.vm"
cisco CRS-8/S (7457) processor with 4194304K bytes of memory.
7457 processor at 1197Mhz, Revision 1.2
16 Packet over SONET/SDH network interface(s)
16 SONET/SDH Port controller(s)
2 Ethernet/IEEE 802.3 interface(s)
16 GigabitEthernet/IEEE 802.3 interface(s)
2043k bytes of non-volatile configuration memory.
38079M bytes of hard disk.
1000592k bytes of ATA PCMCIA card at disk 0 (Sector size 512 bytes).
1000640k bytes of ATA PCMCIA card at disk 1 (Sector size 512 bytes).
Package active on node 0/1/SP:
hfr-diags, V 3.3.0, Cisco Systems, at disk0:hfr-diags-3.3.0
    Built on Tue Jan 31 13:32:40 UTC 2006
    By edde-bld1 in /files/3.3.0/workspace for c2.95.3-p8
hfr-admin, V 3.3.0, Cisco Systems, at disk0:hfr-admin-3.3.0
    Built on Tue Jan 31 10:02:02 UTC 2006
    By edde-bld1 in /files/3.3.0/workspace for c2.95.3-p8
hfr-base, V 3.3.0, Cisco Systems, at disk0:hfr-base-3.3.0
    Built on Tue Jan 31 09:48:20 UTC 2006
    By edde-bld1 in /files/3.3.0/workspace for c2.95.3-p8
.
.
.
The following example shows that the Cisco IOS XR software and active packages are version 3.3.0.
RP/0/RP0/CPU0:router# show install 
  Node 0/1/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Active Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/1/CPU0 [LC] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/lc/mbihfr-lc.vm
    Active Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/6/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Active Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/6/CPU0 [LC] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/lc/mbihfr-lc.vm
    Active Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/RP0/CPU0 [RP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/mbihfr-rp.vm
    Active Packages: 
      disk0:hfr-mgbl-3.3.0
      disk0:hfr-k9sec-3.3.0
      disk0:comp-hfr-mini-3.3.0
  Node 0/RP1/CPU0 [RP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/mbihfr-rp.vm
    Active Packages: 
      disk0:hfr-mgbl-3.3.0
      disk0:hfr-k9sec-3.3.0
      disk0:comp-hfr-mini-3.3.0
  Node 0/SM0/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Active Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/SM1/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Active Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/SM2/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Active Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/SM3/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Active Packages: 
      disk0:comp-hfr-mini-3.3.0
The example output shows that all the active Cisco IOS XR software packages are version 3.3.0. If there is an expected package missing or an active package is not an expected package, install and activate the missing package or upgrade the unexpected package to the appropriate package. See Cisco IOS XR Getting Start Guide for details on installing, activating, and upgrading software packages.

Validating the Installation

Validate the Cisco IOS XR software package installation to ensure the packages were installed correctly. The following commands are used to validate the currently installed software packages:

•install verify Command

•show install active Command

•show install committed Command

install verify Command

Use the install verify command to verify the consistency of a previously installed software set with the package file from which it originated.

This command can be used as a debugging tool to verify the validity of the files that constitute the packages to determine if there are any corrupted files. The command is also used to check that the install infrastructure is up and running and to determine if all files are expected. If there are corrupted files, see Cisco IOS XR Getting Started Guide for information on deactivating and removing software packages and adding and activating software packages.

Note The install verify command can take up to two minutes per package to process.

Note The install verify command ignores secure domain router (SDR) boundaries and performs the operation in global scope.

The following example shows the output of the install verify command. The output is used to verify the consistency of a previously installed software set with the package file from which it originated.
RP/0/RP0/CPU0:router(admin)# install verify 
Install operation 6 'install verify' started by user 'user_a' at 07:25:16 UTC
Tue Mar 07 2006.
The install operation will continue asynchronously.
RP/0/RP1/CPU0:router(admin)#Info:     This operation can take up to 2 minutes.
Info:     Please be patient.
Info:     Verify operation successful, no anomalies found.
Info:     Node 0/1/SP
Info:         [SUCCESS] /bootflash/hfr-diags-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-admin-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-base-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-os-mbi-3.3.0: Verification Successful.
Info:     Node 0/1/CPU0
Info:         [SUCCESS] /bootflash/hfr-diags-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-mcast-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-mpls-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-lc-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-fwdg-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-admin-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-base-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-os-mbi-3.3.0: Verification Successful.
Info:     Node 0/6/SP
Info:         [SUCCESS] /bootflash/hfr-diags-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-admin-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-base-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-os-mbi-3.3.0: Verification Successful.
Info:     Node 0/6/CPU0
Info:         [SUCCESS] /bootflash/hfr-diags-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-mcast-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-mpls-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-lc-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-fwdg-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-admin-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-base-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-os-mbi-3.3.0: Verification Successful.
Info:     Node 0/RP0/CPU0
Info:         [SUCCESS] /disk0/hfr-diags-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-k9sec-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-mgbl-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-rout-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-mcast-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-mpls-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-lc-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-fwdg-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-admin-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-base-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-os-mbi-3.3.0: Verification Successful.
Info:     Node 0/RP1/CPU0
Info:         [SUCCESS] /disk0/hfr-diags-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-k9sec-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-mgbl-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-rout-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-mcast-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-mpls-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-lc-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-fwdg-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-admin-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-base-3.3.0: Verification Successful.
Info:         [SUCCESS] /disk0/hfr-os-mbi-3.3.0: Verification Successful.
Info:     Node 0/SM0/SP
Info:         [SUCCESS] /bootflash/hfr-diags-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-admin-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-base-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-os-mbi-3.3.0: Verification Successful.
Info:     Node 0/SM1/SP
Info:         [SUCCESS] /bootflash/hfr-diags-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-admin-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-base-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-os-mbi-3.3.0: Verification Successful.
Info:     Node 0/SM2/SP
Info:         [SUCCESS] /bootflash/hfr-diags-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-admin-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-base-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-os-mbi-3.3.0: Verification Successful.
Info:     Node 0/SM3/SP
Info:         [SUCCESS] /bootflash/hfr-diags-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-admin-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-base-3.3.0: Verification Successful.
Info:         [SUCCESS] /bootflash/hfr-os-mbi-3.3.0: Verification Successful.
Install operation 6 completed successfully at 07:30:48 UTC Tue Mar 07 2006.
show install active Command

Use the show install active command to display active software packages. Verify that the command output matches the output of the show install committed command. If the output does not match, when you reload the router, the software displayed in the show install committed command output is the software that will be loaded. For example, the following output shows two different software package versions, one is the active version and the other is the committed version, so when the router reloads, The 3.2.6 version will be loaded even though 3.3.0 is the currently active version on 0/RP0/CPU0:
RP/0/RP0/CPU0:router(admin)# show install active location 0/rp0/cpu0   
Node 0/RP0/CPU0 [RP] [SDR: Owner] 
Boot Image: /disk0/hfr-os-mbi-3.3.0/mbihfr-rp.vm 
Active Packages: 
disk0:hfr-infra-test-3.3.0 
disk0:hfr-mgbl-3.3.0 
disk0:hfr-mcast-3.3.0 
disk0:hfr-mpls-3.3.0 
disk0:hfr-k9sec-3.3.0 
disk0:comp-hfr-mini-3.3.0 
RP/0/RP0/CPU0:router(admin)# show install committed location 0/rp0/cpu0    
Node 0/RP0/CPU0 [RP] [SDR: Owner]     
Boot Image: /disk0/hfr-os-mbi-3.2.6/mbihfr-rp.vm     
Committed Packages:        
disk0:hfr-mgbl-3.2.6       
disk0:hfr-mcast-3.2.6      
disk0:hfr-mpls-3.2.6       
disk0:hfr-k9sec-3.2.6       
disk0:comp-hfr-mini-3.2.6 
If the expected active software packages are not displayed, install the packages (if required) and activate the packages. See Cisco IOS XR Getting Started Guide for information on installing and activating Cisco IOS XR software packages. The following example output shows the active packages for all cards in a router:
RP/0/RP0/CPU0:router# show install active
Node 0/1/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Active Packages: 
      disk0:hfr-diags-3.3.0
      disk0:comp-hfr-mini-3.3.0
  Node 0/1/CPU0 [LC] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/lc/mbihfr-lc.vm
    Active Packages: 
      disk0:hfr-diags-3.3.0
      disk0:comp-hfr-mini-3.3.0
  Node 0/6/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Active Packages: 
      disk0:hfr-diags-3.3.0
      disk0:comp-hfr-mini-3.3.0
  Node 0/6/CPU0 [LC] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/lc/mbihfr-lc.vm
    Active Packages: 
      disk0:hfr-diags-3.3.0
      disk0:comp-hfr-mini-3.3.0
  Node 0/RP0/CPU0 [RP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/mbihfr-rp.vm
    Active Packages: 
      disk0:hfr-diags-3.3.0
      disk0:hfr-mgbl-3.3.0
      disk0:hfr-k9sec-3.3.0
      disk0:comp-hfr-mini-3.3.0
  Node 0/RP1/CPU0 [RP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/mbihfr-rp.vm
    Active Packages: 
      disk0:hfr-diags-3.3.0
      disk0:hfr-mgbl-3.3.0
      disk0:hfr-k9sec-3.3.0
      disk0:comp-hfr-mini-3.3.0
  Node 0/SM0/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Active Packages: 
      disk0:hfr-diags-3.3.0
      disk0:comp-hfr-mini-3.3.0
  Node 0/SM1/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Active Packages: 
      disk0:hfr-diags-3.3.0
      disk0:comp-hfr-mini-3.3.0
  Node 0/SM2/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Active Packages: 
      disk0:hfr-diags-3.3.0
      disk0:comp-hfr-mini-3.3.0
  Node 0/SM3/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Active Packages: 
      disk0:hfr-diags-3.3.0
      disk0:comp-hfr-mini-3.3.0
The active packages for each node are on disk0, and for all nodes, the composite package hfr-os-mbi-3.3.0 is active. Additional packages shown are optional packages that have been activated after the initial loading of the Cisco IOS XR Unicast Routing Core Bundle.

show install committed Command

Use the show install committed command to display committed software packages. The committed software packages are the software packages that will be booted on a router reload.

Committed packages are the packages that are persistent across router reloads. If you install and activate a package, it remains active until the next router reload. If you commit a package set, all packages in that set remain active across router reloads until the package set is replaced with another committed package set. The show install committed command is useful to ensure software is installed and committed after a router reload. If the expected software is not installed and committed, see Cisco IOS XR Getting Started Guide for information on installing and committing Cisco IOS XR software packages.

The following command output shows the committed software packages on all card in the router.
RP/0/RP0/CPU0:router# show install committed 
Secure Domain Router: Owner
  Node 0/1/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Committed Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/1/CPU0 [LC] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/lc/mbihfr-lc.vm
    Committed Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/6/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Committed Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/6/CPU0 [LC] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/lc/mbihfr-lc.vm
    Committed Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/RP0/CPU0 [RP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/mbihfr-rp.vm
    Committed Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/RP1/CPU0 [RP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/mbihfr-rp.vm
    Committed Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/SM0/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Committed Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/SM1/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Committed Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/SM2/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Committed Packages: 
      disk0:comp-hfr-mini-3.3.0
  Node 0/SM3/SP [SP] [SDR: Owner]
    Boot Image: /disk0/hfr-os-mbi-3.3.0/sp/mbihfr-sp.vm
    Committed Packages: 
      disk0:comp-hfr-mini-3.3.0
The active packages for each node are on disk0, and for all nodes, the composite package hfr-os-mbi-3.3.0 is active.

Validating and Troubleshooting Cisco IOS XR Software Configuration

Validating the Cisco IOS XR software configuration includes collecting configuration information on the router to determine configuration changes and verifying the current running configuration. When a configuration fails during a commit, the failed configuration can be viewed to help determine why the configuration was not committed.

The following sections are provided:

•Local and Global Configurations

•Collecting Configuration Information

•Verifying the Running Configuration

•Using the show configuration failed Command

Local and Global Configurations

Configuration data is split between global (shared) and local configurations. Configurations are held locally to the appropriate node. For example, the system configuration is distributed to the node it belongs on. The routing protocol configurations that are shared for all nodes are part of the SysDB shared plane running on the dLRSC RP node.

The local plane configurations, such as interface-specific configuration, belong to the local plane SysDB running on each node. Every node has a data store containing the local data for that node (local plane), including configuration and operational data for the local interfaces. There is also a data store containing the shared data (shared plane) which is primarily used by RP and DRP applications, but accessible to all nodes.

Each SysDB item is categorized as either `local' or `shared'. Local data is that primarily of interest to a single node and shared data is everything else. Since almost all line card usage involves only local data, their SysDB clients only use their local server which minimizes remote inter-process communication (IPC).

When troubleshooting configurations, you need to determine whether the problem is local or shared (global). To view the local configuration, use the show running-config interface * command.
RP/0/RP0/CPU0:router# show running-config interface * 
interface Bundle-Ether28
 bundle minimum-active bandwidth 1000000
 bundle minimum-active links 1
 ipv4 address 10.12.28.1 255.255.255.0
 description Connected to P2_CRS-8 Bundle-Ether 28
!
interface Bundle-Ether28.1
 dot1q vlan 29
 ipv4 address 10.12.29.1 255.255.255.0
 description Connected to CRS-8 Bundle-Ether 28.1
.
.
.
interface Bundle-POS24
 bundle minimum-active bandwidth 2488320
 bundle minimum-active links 1
 ipv4 address 10.12.24.1 255.255.255.0
 description Connected to CRS-8 Bundle-POS 24
!
interface Loopback0
 ipv4 address 10.1.1.1 255.255.255.255
!
interface MgmtEth0/RP0/CPU0/0
 description Connected to lanA LAN
 ipv4 address 172.29.52.70 255.255.255.0
!
interface MgmtEth0/RP1/CPU0/0
 description Connected to lanA LAN
 ipv4 address 172.29.52.71 255.255.255.0
!
interface GigabitEthernet0/1/5/0
 description Connected to CRS-8 GE 0/1/5/0
 ipv4 address 10.12.16.1 255.255.255.0
!
interface GigabitEthernet0/1/5/1
 description Connected to C12810 GE 5/2
 ipv4 address 10.14.8.1 255.255.255.0
!
interface GigabitEthernet0/1/5/2
 description Connected to C12406 GE 0/4/0/1
 ipv4 address 10.16.4.1 255.255.255.0
!
interface GigabitEthernet0/1/5/3
 shutdown
.
.
.
interface POS0/1/4/1
 description Connected to CRS-8 POS 0/1/4/1
 bundle id 24 mode active
!
interface POS0/1/4/2
 description Connected to CRS-8 POS 0/1/4/2
 ipv4 address 10.12.32.1 255.255.255.0
!
interface POS0/1/4/3
 description Connected to CRS-8 POS 0/1/4/3
 ipv4 address 10.12.36.1 255.255.255.0
.
.
.
controller SONET0/6/4/6
 clock source internal
!
The output displays all the configured interfaces on the node.

Use the show sysdb trace commands to display the contents of the SysDB after a configuration change. The trace information includes a history of any changes to the running configuration. You can specify either a local node or the shared plane.

The following example output shows the contents of the SysDB local plane:
RP/0/RP0/CPU0:router# show sysdb trace verification location 0/5/cpu0 reverse 
Timestamp             jid       tid   reg handle  connid     action                           
             path 
656 wrapping entries (4096 possible, 3440 filtered, 6460 total) 
Aug 29 06:14:38.443      116       1     20       38446      apply reply                      
'--' 
Aug 29 06:14:38.442      116       1     20       1139       Apply/abort called               
'cfg/if/act/POS0_5_0_0/keepalive' 
Aug 29 06:14:38.441      116       1     20       1139       verify reply: accept             
'--' 
Aug 29 06:14:38.438      116       1     20       1139       Verify called                    
'cfg/if/act/POS0_5_0_0/keepalive' 
The following example output shows the contents of the SysDB shared plane:
RP/0/RP0/CPU0:router# show sysdb trace verification shared-plane reverse 
Timestamp             jid       tid   reg handle  connid     action                           
             path 
4 wrapping entries (4096 possible, 4092 filtered, 904284 total) 
Aug 29 06:16:53.244      526       1     880      12043      apply reply                      
'--' 
Aug 29 06:16:53.229      526       1     880      1111       Apply/abort called               
'cfg/gl/aaa/tacacs/source-interface' 
Aug 29 06:16:53.225      526       1     880      1111       verify reply: accept             
'--' 
Aug 29 06:16:53.214      526       1     880      1111       Verify called                    
'cfg/gl/aaa/tacacs/source-interface' 
The show processes location node-id | include sysdb command displays all active SysDB processes for a specified node.

Collecting Configuration Information

Collecting configuration information allows you to determine if changes to the system have occurred. It also allows you to determine if these changes could impact the system. The following commands allow you to determine if there was an unknown commit, if there was a commit that overwrote a previous configuration, or there are configuration changes that should be removed from the running configuration.

•show config commit history—the command output displays information about the last (up to) 1000 commits, of which only the last (up to) 100 commits are available for rollback operations.

•show configuration commit changes {[since] commit-id | last number-of-commits} [diff]—the command output displays changes made to the running configuration by previous configuration commits.
RP/0/RP0/CPU0:router# show config commit changes since 1000000319 
Wed May 17 09:30:27.877 UTC 
Building configuration...
no logging console
no domain ipv4 host ce1
no domain ipv4 host ce2
domain ipv4 host ce6 172.29.52.73
domain ipv4 host ce7 172.29.52.78
no domain ipv4 host pe1
no domain ipv4 host pe2
domain ipv4 host pe6 172.29.52.128
domain ipv4 host pe7 172.29.52.182
interface GigabitEthernet0/1/5/1
 no negotiation
!
end
•show configuration commit list [number-of-commits] [detail]—the command output displays a list of the commit IDs (up to 100) available for rollback.
RP/0/RP0/CPU0:router# show configuration commit list 
Wed May 17 09:31:21.727 UTC 
SNo. Label/ID    User      Line        Client      Time Stamp
~~~~ ~~~~~~~~    ~~~~      ~~~~        ~~~~~~      ~~~~~~~~~~
1    1000000324  userA     vty0        CLI         16:50:33 UTC Wed May 10 2006
2    1000000323  userA     vty0        CLI         16:49:51 UTC Wed May 10 2006
3    1000000322  userB     vty0        CLI         16:48:05 UTC Wed May 10 2006
4    1000000321  userC     vty2        CLI         19:11:26 UTC Wed May 03 2006
5    1000000320  userA     vty2        CLI         19:10:45 UTC Wed May 03 2006
6    1000000319  userB     vty2        CLI         18:03:01 UTC Wed May 03 2006
7    1000000318  userB     vty2        CLI         18:02:43 UTC Wed May 03 2006
8    1000000317  userB     vty2        CLI         18:02:38 UTC Wed May 03 2006
9    1000000316  userC     vty2        CLI         17:59:16 UTC Wed May 03 2006
10   1000000315  userC     vty2        CLI         17:46:38 UTC Wed May 03 2006
11   1000000314  userA     vty2        CLI         15:40:04 UTC Wed May 03 2006
12   1000000313  userA     vty2        CLI         13:05:09 UTC Wed May 03 2006
13   1000000312  userD     con0_RP0_C  CLI         13:49:31 UTC Mon May 01 2006
•commit confirmed minutes—the command commits the configuration on a trial basis for a minimum of 30 seconds and a maximum of 300 seconds (5 minutes). During the trial configuration period, enter commit to confirm the configuration. If commit is not entered, then the system will revert to the previous configuration when the trial time period expires.

Verifying the Running Configuration

To verify the running configuration, perform the following procedure.

SUMMARY STEPS

1. configure

1. show running-config

2. describe hostname hostname

3. end

4. show sysdb trace verification shared-plane | include path

5. show sysdb trace verification location node-id

6. show cfgmgr trace

7. show config commit history

8. show config commit changes

9. show config failed startup

10. cfs check

DETAILED STEPS
Command or Action

Purpose

Step 1

configure
Example:

RP/0/RP0/CPU0:router# configure

Enters global configuration mode.

Step 1

show running-config
Example:

RP/0/RP0/CPU0:router(config)# show running-config

Displays the contents of the running configuration.

Verify that the running configuration is as expected.

Step 2

describe hostname hostname
Example:

RP/0/RP0/CPU0:router(config)# describe hostname router_A

Determines the path.
Step 3

end
Example:

RP/0/RP0/CPU0:router(config)# end
Saves configuration changes.

•When you issue the end command, the system prompts you to commit changes:
Uncommitted changes found, commit them before 
exiting(yes/no/cancel)? 
[cancel]:
–Entering yes saves configuration changes to the running configuration file, exits the configuration session, and returns the router to EXEC mode.

–Entering no exits the configuration session and returns the router to EXEC mode without committing the configuration changes.

–Entering cancel leaves the router in the current configuration session without exiting or committing the configuration changes.
Step 4

show sysdb trace verification shared-plane | include path
Example:

RP/0/RP0/CPU0:router# show sysdb trace verification shared-plane | include gl/a/hostname

Displays details of recent verification sysDB transactions and changes on the shared plane allowing you to verify whether the configuration was verified correctly.

Specifying the path filters the data to display only the sysDB path for the router.

Verify that changes to the SysDB were verified and accepted.

Step 5

show sysdb trace verification location node-id
Example:

RP/0/RP0/CPU0:router# show sysdb trace verification location 0/3/CPU0

Displays details of recent verification sysDB transactions and changes on local plane configurations.

Verify that changes to the SysDB were verified and accepted.

Step 6

show cfgmgr trace
Example:

RP/0/RP0/CPU0:router# show cfgmgr trace

Displays cfgmgr trace information.

Step 7

show configuration history commit
Example:

RP/0/RP0/CPU0:router# show configuration history commit

Displays a list of historical changes to the configuration.

Verify that the timeline of changes is as expected.

Step 8

show configuration commit changes
Example:

RP/0/RP0/CPU0:router# show configuration commit changes

Displays detailed committed configuration history information.

Verify that the history information is as expected.

Step 9

show configuration failed startup
Example:

RP/0/RP0/CPU0:router# show configuration failed startup

Displays information on any configurations that failed during startup.

Step 10

cfs check
Example:

RP/0/RP0/CPU0:router# cfs check

Checks the current configuration to see if there are any missing configurations.
Examples

The following example shows the output of the show running-config command:
RP/0/RP0/CPU0:router# show running-config  
Thu May 18 13:13:05.187 UTC 
Building configuration...
!! Last configuration change at 12:51:32 UTC Wed May 17 2006 by user_A
!
snmp-server traps fabric plane
hostname router_A
line console
 exec-timeout 600 0
 session-timeout 600
!
line default
 exec-timeout 600 0
 session-timeout 600
!
telnet vrf default ipv4 server max-servers no-limit
domain ipv4 host p1 192.0.2.72
domain ipv4 host p2 192.0.2.77
domain ipv4 host ce6 192.0.2.73
domain ipv4 host ce7 192.0.2.78
domain ipv4 host pe6 192.0.2.128
domain ipv4 host pe7 192.0.2.182
domain-lookup
vty-pool default 0 25
ipv4 virtual address 192.0.2.72 255.255.255.0
class-map match-any 11g
 match mpls experimental topmost 5
 match precedence critical
!
class-map match-any default
 match any
!
class-map match-any business
 match mpls experimental topmost 5
 match precedence flash
!
policy-map fabric-qos
 class 11g
  priority
 !
 class business
  bandwidth remaining percent 65
 !
 class default
  bandwidth remaining percent 35
 !
!
interface Loopback0
 ipv4 address 10.1.1.1 255.255.255.255
!         
interface MgmtEth0/RP0/CPU0/0
 description Connected to router LAN
 ipv4 address 192.0.2.70 255.255.255.0
!
.
.
.
!
interface POS0/6/0/3
 shutdown
!
interface POS0/6/4/0
 shutdown
!
interface POS0/6/4/1
The output is used to determine if the configuration is as expected.

In the following example, the path to SysDB where the configuration is stored in the database is displayed.
RP/0/RP0/CPU0:router(config)# describe hostname router 
The command is defined in shellutil.parser
Node 0/RP0/CPU0 has file shellutil.parser for boot package /disk0/hfr-os-mbi-3.3
.0/mbihfr-rp.vm from hfr-base
Package:
    hfr-base
        hfr-base V3.3.0[2I]  Base Package
        Vendor : Cisco Systems
        Desc   : Base Package
        Build  : Built on Mon May  1 06:27:09 UTC 2006
        Source : By edde-bld1 in /vws/3.3.0./file for 
c2.95.3-p8
        Card(s): RP, DRP, DRPSC, OC3-POS-4, OC12-POS, GE-3, OC12-POS-4, OC48-POS
, E3-OC48-POS, E3-OC12-POS-4, E3-OC3-POS-16, E3-OC3-POS-8, E3-OC3-POS-4, E3-OC48
-CH, E3-OC12-CH-4, E3-OC3-CH-16, E3-GE-4, E3-OC3-ATM-4, E3-OC12-ATM-4, E5-CEC, L
C, SP
        Restart information:
          Default:
            parallel impacted processes restart
Component:
    shellutil V[r33x/1]  Common shell utility applications
File: shellutil.parser
User needs ALL of the following taskids:
        root-lr (READ WRITE) 
It will take the following actions:
  Create/Set the configuration item:
         Path: gl/a/hostname
        Value: p1_CRS-8
The output shows that the path is gl/a/hostname.
RP/0/RP0/CPU0:router(config)# end 
In the following example, the verification details for the specified hostname is displayed.
RP/0/RP0/CPU0:router# show sysdb trace verification shared-plane | include gl/a/hostname 
May 18 19:16:17.143      340       3     210      962        Apply/abort called 
              'cfg/gl/a/hostname'
May 18 19:16:17.132      340       3     210      962        Verify called      
              'cfg/gl/a/hostname'
May 18 19:16:17.126      340       3     210      962        Apply/abort called 
              'cfg/gl/a/hostname'
May 18 19:16:17.109      340       3     210      962        Verify called      
              'cfg/gl/a/hostname'
May 18 18:43:16.065      340       3     210      962        register           
              'cfg/gl/a/hostname'
May 18 18:41:41.048      340       3     16       362        register           
              'cfg/gl/a/hostname'
The output shows that changes to the SysDB shared plane were verified and accepted.

In the following example, the verification details for the specified location is displayed.
RP/0/RP0/CPU0:router# show sysdb trace verification location 0/3/CPU0 
Timestamp             jid       tid   reg handle  connid     action             
              path
323 wrapping entries (4096 possible, 299 filtered, 622 total)
Jul  7 20:10:36.212      260       1     90       8782       apply reply        
              '--'
Jul  7 20:10:35.476      260       1     90       4912       Apply/abort called 
              'cfg/if/act/GigabitEthernet0_3_4_0.1/a/sub_vlan/0x2/________/Gigab
itEthernet0_3_4_0/________'
Jul  7 20:10:35.475      260       1     90       4912       verify reply: accep
t             '--'
Jul  7 20:10:35.471      260       1     90       4912       Verify called      
              'cfg/if/act/GigabitEthernet0_3_4_0.1/a/sub_vlan/0x2/________/Gigab
itEthernet0_3_4_0/________'
Jul  7 20:10:35.471      144       1     4        8782       apply reply        
              '--'
Jul  7 20:10:35.471      144       1     4        8782       apply reply        
              '--'
Jul  7 20:10:35.471      144       1     4        8782       apply reply        
              '--'
Jul  7 20:10:35.471      144       1     4        8782       apply reply        
              '--'
Jul  7 20:10:35.471      144       1     4        8782       apply reply        
              '--'
Jul  7 20:10:35.471      144       1     4        8782       apply reply        
              '--'
Jul  7 20:10:35.471      144       1     4        8782       apply reply        
              '--'
Jul  7 20:10:35.471      144       1     4        8782       apply reply        
              '--'
Jul  7 20:10:35.470      144       1     4        474        Apply/abort batch e
nded          ''
Jul  7 20:10:35.470      144       1     4        474        Apply/abort called 
              'cfg/if/act/GigabitEthernet0_3_4_0/ord_x/im/shutdown'
Jul  7 20:10:35.470      144       1     4        474        Apply/abort called 
              'cfg/if/act/GigabitEthernet0_3_4_1/ord_x/im/shutdown'
Jul  7 20:10:35.470      144       1     4        474        Apply/abort called 
              'cfg/if/act/GigabitEthernet0_3_4_2/ord_x/im/shutdown'
Jul  7 20:10:35.470      144       1     4        474        Apply/abort called 
              'cfg/if/act/GigabitEthernet0_3_4_3/ord_x/im/shutdown'
Jul  7 20:10:35.470      144       1     4        474        Apply/abort called 
              'cfg/if/act/GigabitEthernet0_3_4_4/ord_x/im/shutdown'
Jul  7 20:10:35.469      144       1     4        474        Apply/abort called 
              'cfg/if/act/GigabitEthernet0_3_4_5/ord_x/im/shutdown'
Jul  7 20:10:35.469      144       1     4        474        Apply/abort called 
              'cfg/if/act/GigabitEthernet0_3_4_6/ord_x/im/shutdown'
Jul  7 20:10:35.469      144       1     4        474        Apply/abort called 
              'cfg/if/act/GigabitEthernet0_3_4_7/ord_x/im/shutdown'
Jul  7 20:10:35.469      144       1     4        474        Apply/abort batch s
tarted        ''
Jul  7 20:10:35.469      144       1     4        474        verify reply: accep
t             '--'
Jul  7 20:10:35.469      144       1     4        474        verify reply: accep
t             '--'
Jul  7 20:10:35.469      144       1     4        474        verify reply: accep
t             '--'
!
!
!
The output shows that changes to the SysDB local plane were verified and accepted.

In the following example, the cfgmgr trace details are displayed.
RP/0/RP0/CPU0:router# show cfgmgr trace 
69 wrapping entries (2048 possible, 0 filtered, 69 total)
Jul  5 14:47:17.967 cfgmgr/common 0/RP0/CPU0 t3  Config media returned from disk
util: '/disk0:/'.
Jul  5 14:47:46.994 cfgmgr/common 0/RP0/CPU0 t1  Received a state change event. 
 State is 'active'
Jul  5 14:47:47.218 cfgmgr/common 0/RP0/CPU0 t1  Config media returned from disk
util: '/disk0:/'.
Jul  5 14:47:56.502 cfgmgr/common 0/RP0/CPU0 t6  Received a state change event. 
 State is 'active'
Jul  5 14:47:56.512 cfgmgr/common 0/RP0/CPU0 t4  State of the request queue is '
PROCESSABLE'
Jul  5 14:47:56.520 cfgmgr/common 0/RP0/CPU0 t4  Startup config apply requested 
with option '0x2'
Jul  5 14:47:57.471 cfgmgr/common 0/RP0/CPU0 t4  Attempting to apply ascii admin
 startup config from file '/qsm/cfsroot/admin/admin.cfg'.
Jul  5 14:48:09.156 cfgmgr/common 0/RP0/CPU0 t4  Clean all admin files since adm
in commit is empty.
Jul  5 14:48:28.044 cfgmgr/common 0/RP0/CPU0 t6  Infra band COMPLETE 0
Jul  5 14:49:11.832 cfgmgr/common 0/RP0/CPU0 t4  Suspend flag value 1
Jul  5 14:49:11.832 cfgmgr/common 0/RP0/CPU0 t4  State of the request queue is '
NOT-PROCESSABLE'
Jul  5 14:51:22.738 cfgmgr/common 0/RP0/CPU0 t4  Suspend flag value 1
Jul  5 14:51:22.738 cfgmgr/common 0/RP0/CPU0 t4  State of the request queue is '
NOT-PROCESSABLE'
Jul  5 14:51:22.738 cfgmgr/common 0/RP0/CPU0 t4  State of the request queue is '
PROCESSABLE'
Jul  5 14:51:22.738 cfgmgr/common 0/RP0/CPU0 t4  Startup config apply requested 
with option '0x1'
Jul  5 14:51:22.793 cfgmgr/common 0/RP0/CPU0 t4  Turboboot flag = '0x0', Passwor
d recovery flag = '0x0'
Jul  5 14:51:26.114 cfgmgr/common 0/RP0/CPU0 t4  Attempting to apply binary LR s
tartup config.
Jul  5 14:51:26.128 cfgmgr/common 0/RP0/CPU0 t4  commitdb_purge_entries called w
ith option '0x0'
Jul  5 14:51:26.135 cfgmgr/common 0/RP0/CPU0 t4  commitdb_check_status returns s
tatus - '0x0' with error: 'No error'
Jul  5 14:51:26.245 cfgmgr/common 0/RP0/CPU0 t4  commitdb_load_changes returns e
rror: 'Invalid argument'
Jul  5 14:51:26.288 cfgmgr/common 0/RP0/CPU0 t4  commitdb_create_delta returns e
rror: 'Invalid argument'
Jul  5 14:51:26.296 cfgmgr/common 0/RP0/CPU0 t4  commitdb_save_running_from_comm
itdb returns error: 'Invalid argument'
!
!
!
The output shows that the configuration files are stored on disk0 and the state of the node is active (State is 'active'). The administration configuration is applied during startup for the designated shelf controller (DSC) (Attempting to apply ascii admin startup config from file '/qsm/cfsroot/admin/admin.cfg'.). The secure domain router (SDR)-specific configuration is applied from the saved binary check points (Attempting to apply binary LR s tartup config.). Several invalid argument errors were returned when attempting to restore the startup configuration from the binary checkpoints. If there are invalid argument errors, contact Cisco Technical Support. For Cisco Technical Support contact information, see the "Obtaining Technical Assistance" section in the Preface.

The output also shows the state of queue as PROCESSABLE or NOT-PROCESSABLE. A NOT-PROCESSABLE state can indicate that the interface is still in the preconfiguration state. If the queue is not processable, then interfaces cannot be restored. This queue state information can be looked at along with the startup config information, as it is one of the gating factors for allowing the queue to be processable.

In the following example, a list of historical changes to the configuration is displayed.
RP/0/RP0/CPU0:router# show configuration history commit
Sno.  Event      Info                           Time Stamp
~~~~  ~~~~~      ~~~~                           ~~~~~~~~~~
1     commit     id 1000000001                  Mon Aug 14 17:09:58 2006
2     commit     id 1000000002                  Mon Aug 14 17:41:34 2006
3     commit     id 1000000003                  Mon Aug 21 08:05:42 2006
4     commit     id 1000000004                  Mon Aug 21 12:39:31 2006
5     commit     id 1000000005                  Mon Aug 21 12:43:29 2006
6     commit     id 1000000006                  Mon Aug 21 12:45:48 2006
7     commit     id 1000000007                  Mon Aug 21 13:23:08 2006
8     commit     id 1000000008                  Mon Aug 21 13:24:56 2006
9     commit     id 1000000009                  Mon Aug 21 13:26:05 2006
10    commit     id 1000000010                  Mon Aug 21 13:55:08 2006
11    commit     id 1000000011                  Tue Aug 22 13:52:09 2006
12    commit     id 1000000012                  Tue Aug 22 14:15:34 2006
13    commit     id 1000000013                  Wed Aug 23 07:26:38 2006
14    commit     id 1000000014                  Wed Aug 23 10:44:20 2006
15    commit     id 1000000015                  Wed Aug 23 10:44:44 2006
16    commit     id 1000000016                  Wed Aug 23 10:44:58 2006
17    commit     id 1000000017                  Wed Aug 23 17:47:45 2006
18    commit     id 1000000018                  Thu Aug 24 07:55:57 2006
19    commit     id 1000000019                  Thu Aug 24 07:58:05 2006
20    commit     id 1000000020                  Thu Aug 24 08:01:00 2006
21    commit     id 1000000021                  Thu Aug 24 08:01:17 2006
22    commit     id 1000000022                  Thu Aug 24 08:01:47 2006
23    commit     id 1000000023                  Thu Aug 24 08:02:21 2006
24    commit     id 1000000024                  Thu Aug 24 08:03:35 2006
25    commit     id 1000000025                  Mon Aug 28 05:34:58 2006
26    commit     id 1000000026                  Mon Aug 28 05:50:10 2006
27    commit     id 1000000027                  Mon Aug 28 06:07:13 2006
28    commit     id 1000000028                  Mon Aug 28 06:09:15 2006
29    commit     id 1000000029                  Mon Aug 28 06:44:35 2006
30    commit     id 1000000030                  Mon Aug 28 08:36:33 2006
31    commit     id 1000000031                  Mon Aug 28 09:40:48 2006
32    commit     id 1000000032                  Mon Aug 28 11:29:35 2006
33    commit     id 1000000033                  Mon Aug 28 11:54:54 2006
34    commit     id 1000000034                  Mon Aug 28 12:29:37 2006
35    commit     id 1000000001                  Mon Aug 28 20:28:44 2006
36    commit     id 1000000002                  Mon Aug 28 21:07:05 2006
In the following example, detailed information on the last committed configuration is displayed.
RP/0/RP0/CPU0:router# show configuration commit changes last 1
Building configuration...
interface Bundle-Ether28
 description Connected to CRS-8 Bundle-Ether 28
 ipv4 address 10.12.28.1 255.255.255.0
 bundle minimum-active links 1
 bundle minimum-active bandwidth 1000000
!
interface Bundle-Ether28.1
 description Connected to CRS-8 Bundle-Ether 28.1
 ipv4 address 10.12.29.1 255.255.255.0
 dot1q vlan 29
!
interface Bundle-Ether28.2
 description Connected to CRS-8 Bundle-Ether 28.2
 ipv4 address 10.12.30.1 255.255.255.0
 dot1q vlan 30
!
interface Bundle-Ether28.3
 description Connected to CRS-8 Bundle-Ether 28.3
 ipv4 address 10.12.31.1 255.255.255.0
 dot1q vlan 31
!
interface Bundle-POS24
 description Connected to CRS-8 Bundle-POS 24
 ipv4 address 10.12.24.1 255.255.255.0
 bundle minimum-active links 1
 bundle minimum-active bandwidth 2488320
!
no interface Loopback0
interface Loopback0
 ipv4 address 10.1.1.1 255.255.255.255
!
interface MgmtEth0/RP0/CPU0/0
 no description
 description Connected to lanA LAN
 no ipv4 address 172.29.52.70 255.255.255.0
 ipv4 address 172.29.52.70 255.255.255.0
.
.
.
router ospf 100
 router-id 10.1.1.1
 router-id Loopback0
 area 0
  interface Loopback0
   passive enable
  !
  interface GigabitEthernet0/1/5/2
  !
  interface POS0/1/0/1
  !
 !
!         
mpls ldp
 router-id Loopback0
 log
  neighbor
  graceful-restart
 !
 interface POS0/1/0/1
 !
 interface GigabitEthernet0/1/5/2
 !
!
mpls oam
!
ssh server
xml agent tty
xml agent corba
http server
end
In the following example, information on any configurations that failed during startup is displayed.
RP/0/RP0/CPU0:router# show configuration failed startup
!!20:16:32 UTC Mon Aug 28 2006
!! CONFIGURATION FAILED DUE TO SYNTAX/AUTHORIZATION ERRORS 
domain-lookup
router ospf 100
 area 0
  interface POS0/1/0/1
mpls ldp
 router-id Loopback0
 log
  neighbor
  graceful-restart
 interface POS0/1/0/1
 interface GigabitEthernet0/1/5/2
mpls oam
router igmp
 version 1
ssh server
xml agent tty
xml agent corba
http server
In the following example, a check of the current configuration for any missing configurations is run and the results are displayed.
RP/0/RP0/CPU0:router# cfs check
Creating any missing directories in Configuration File system...OK
Initializing Configuration Version Manager...OK
Syncing commit database with running configuration...OK
Re-initializing cache files...OK
Updating Commit Database.  Please wait...[OK]
Using the show configuration failed Command

Use the show configuration failed command to browse a failed configuration. The configuration can be classified as failed during startup or during a configuration commit.

•Startup Failed Configuration

•Commit Configuration Failed

Startup Failed Configuration

A configuration can be classified as failed during startup for three reasons:

•Syntax errors—syntax errors are generated by the parser and usually indicate that there is an incompatibility with the command-line interface (CLI) commands. Correct the syntax errors and reapply the configuration. A syntax error can be an invalid CLI entry or a CLI syntax change. See the "Obtaining Documentation" section in the Preface for information on obtaining Cisco IOS XR software CLI documentation.

•Semantic errors—semantic errors are generated by the backend components when the configuration is being restored by the configuration manager during startup of the router. Semantic errors include logical problems (invalid logic).

•Apply errors—apply errors are generated when a configuration has been successfully verified and accepted as part of running configuration but the backend component is not able to update its operational state. The configuration shows both as the running configuration (since it was correctly verified) and as a failed configuration because of the backend operational error. To find the component apply owner, use the describe on the CLI that failed to be applied.

Note You may browse startup failed configurations for up to four previous router reloads.

Use the show configuration failed startup command and the load configuration failed startup command to browse and reapply any failed configuration. The load configuration failed startup command can be used in configuration mode to load the failed startup configuration into the target configuration session, then the configuration can be modified and committed. See Cisco IOS XR Getting Started Guide for information on committing a configuration.
RP/0/RP0/CPU0:router# show configuration failed startup 
!! CONFIGURATION FAILED DUE TO SYNTAX/AUTHORIZATION ERRORS 
telnet vrf default ipv4 
server max-servers 5 interface POS0/7/0/3 router static 
address-family ipv4 unicast 
  0.0.0.0/0 172.18.189.1 
!! CONFIGURATION FAILED DUE TO SEMANTIC ERRORS 
router bgp 217 
!!% Process did not respond to sysmgr ! 
RP/0/RP0/CPU0:router# 
RP/0/RP0/CPU0:router# config 
RP/0/RP0/CPU0:router(config)# load config failed startup noerror 
Loading. 263 bytes parsed in 1 sec (259)bytes/sec 
RP/0/RP0/CPU0:router(config-bgp)#show configuration 
Building configuration... 
telnet vrf default ipv4 server max-servers 5 router static 
address-family ipv4 unicast 
  0.0.0.0/0 172.18.189.1 
  ! 
!
router bgp 217 
! 
end 
The failed configuration is loaded into the target configuration, minus the errors that caused the startup configuration to fail.
RP/0/RP0/CPU0:router(config-bgp)# commit 
Use the show configuration failed command to display failed items in the last configuration commit, including reasons for the error.

In any mode, the configuration failures from the most recent commit operation are displayed.

The show configuration failed command can be used in EXEC mode and configuration mode. The command is used in EXEC mode when the configuration does not load during startup. The command is used in configuration mode to display information when a commit fails.

Note The show configuration failed command in configuration mode only exists as long as the configuration session is active. Once you exit configuration mode, the command cannot be used to display the failed configuration.

Commit Configuration Failed

The following example shows an invalid task ID configuration that fails to commit. The show configuration failed command provides information on why the configuration failed.
RP/0/RP0/CPU0:router(config)# taskgroup isis  
RP/0/RP0/CPU0:router(config-tg)# commit
% Failed to commit one or more configuration items during an atomic operation, s
RP/0/RP0/CPU0:router(config-tg)# show configuration failed
!! CONFIGURATION FAILED DUE TO SEMANTIC ERRORS
taskgroup isis
!!% Usergroup/Taskgroup names cannot be taskid names
!
If a configuration commit fails, do not exit configuration mode (return to EXEC mode) as you will not be able to view the failed configuration.
RP/0/RP0/CPU0:router# configure
RP/0/RP0/CPU0:router(config)# taskgroup bgp 
RP/0/RP0/CPU0:router(config-tg)# end
Uncommitted changes found, commit them before exiting(yes/no/cancel)? [cancel]:y
% Failed to commit one or more configuration items during an atomic operation, s
RP/0/RP0/CPU0:router(config)# exit
Uncommitted changes found, commit them before exiting(yes/no/cancel)? [cancel]:n
RP/0/RP0/CPU0:router# show configuration failed 
ASIC Errors

The following ASIC error types are supported:

•FIA (Fabric Interface ASIC)

•PSE (packet switching engine)

•Cpuctrl

•Egressq

•Fabricq

•Discovery

•Plim-asic

The following ASIC error classifications are supported:

•Single Bit Errors (SBE)—Correctable ECC protected single bit errors in external or internal memory

Not reported to PM on each occurrence and reported to the platform manager (PM) as Minor when software threshold rate is exceeded. Report alarm using Alarm Logging, and Debugging Event Management System (ALDEMS).

Error data:

–Address—Address that encountered the SBE

–Syndrome—Syndrome if available

•Multiple Bit Errors—Uncorrectable multiple bit error in memory.

Reported to PM as Major and ALDEMS for each occurrence.

Error data:

–Address—Address that encountered the SBE

–Data—Actual error data

•PARITY Errors—Parity error in all applicable memory

Reported to PM as Major.

•Cyclic redundancy check (CRC) Errors—CRC errors in EIO other links.

Not reported for each occurrence. When the threshold is reached it is reported as Major to the PM.

•GENERIC Errors—Errors that do not fall under any of the other classifications.

Threshold and alarm reporting is done.

•RESET Errors—Logged for each reset instance of the ASIC.

Reported to PM when threshold is exceeded.

Error data:

–Interrupt status—Interrupt status bits due to ASIC reset.

–Halt status—Halt status bits.

–Reset node key—Key for the error node that causes the reset.

–Time—Reset time.

The following ASIC error fault severities are supported:

•Critical—Affected component is unusable or card is reset if no redundant card exists

•Major—Partially service affecting fault or if redundant card, do a failover otherwise the card runs in degraded mode

•Minor—Non-service affecting fault

•OK—No fault

Use the show asic-errors command to view if any ASIC errors have occurred on nodes.

The following example shows how to display ASIC errors. The ASIC-ERRORs folder is created after the first node reset. A folder is created for each node that has reloaded because of an ASIC error. If the ASIC-ERROR folder does not exist, there have not been any node resets on the system.
RP/0/RP0/CPU0:router# dir harddisk:
Directory of harddisk:
5           drwx  4096        Fri Jun 10 10:27:32 2005  LOST.DIR
6           drwx  4096        Fri Jun 10 10:27:32 2005  usr
7           drwx  4096        Fri Jun 10 10:27:32 2005  var
131328      -rwx  173056      Tue Apr 18 17:18:50 2006  instdb_backup.tar
19          drwx  4096        Tue Apr 18 17:32:46 2006  dumper
1880        drwx  4096        Fri Oct 14 13:52:22 2005  ASIC-ERROR
The following example shows how to display node-specific ASIC errors.
RP/0/RP0/CPU0:router# dir harddisk:/ASIC-ERROR 
Directory of harddisk:/ASIC-ERROR
1881        drwx  4096        Thu Jun 16 08:32:14 2005  node0_3_CPU0
2141        drwx  4096        Fri Oct 14 13:52:22 2005  node0_5_CPU0
The output shows that there were two line card reloads caused by ASIC errors (June 16 and October 14).

The following example lists the PSE bfiles for a specific node reload. The PSE files contain the actual ASIC error data that triggered the reload.
RP/0/RP0/CPU0:router# dir harddisk:/ASIC-ERROR/node0_3_CPU0 
Directory of harddisk:/ASIC-ERROR/node0_3_CPU0
123273312   -rwx  4823        Sat Aug 13 20:04:06 2005  pse_00.err
123273376   -rwx  4794        Sat Aug 13 20:04:06 2005  pse_01.err
The following example shows how to display the contents of a specific PSE file.
RP/0/RP0/CPU0:router# more harddisk:/ASIC-ERROR/node0_3_CPU0/pse_00.err 
Next file write offset = 4823
^@
################# Start of data pse_00_061605_083214.err #################
************************************************************
*                   ASIC Errors Summary                    *
************************************************************
Number of nodes     : 2
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
The following example shows how to display a summary for each ASIC. If an error is displayed, dump the individual asic instance number to obtain details on the ASIC error.
RP/0/RP0/CPU0:router# show asic-errors all location 0/6/cpu0 
************************************************************
*                  Fia ASIC Error Summary                  *
************************************************************
Instance            : 0
Number of nodes     : 130
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
Instance            : 1
Number of nodes     : 130
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
************************************************************
*                  Pse ASIC Error Summary                  *
************************************************************
Instance            : 0
Number of nodes     : 2
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
Instance            : 1
Number of nodes     : 2
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
************************************************************
*                Cpuctrl ASIC Error Summary                *
************************************************************
Instance            : 0
Number of nodes     : 0
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
************************************************************
*                Egressq ASIC Error Summary                *
************************************************************
Instance            : 0
Number of nodes     : 1
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
************************************************************
*                Fabricq ASIC Error Summary                *
************************************************************
Instance            : 0
Number of nodes     : 3
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
Instance            : 1
Number of nodes     : 2
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
************************************************************
*               Ingressq ASIC Error Summary                *
************************************************************
Instance            : 0
Number of nodes     : 1
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
************************************************************
*               Discovery ASIC Error Summary               *
************************************************************
Instance            : 0
Number of nodes     : 0
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
************************************************************
*               Plim-asic ASIC Error Summary               *
************************************************************
Instance            : 0
Number of nodes     : 2
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 2
Reset error count   : 0
--------------------
Instance            : 1
Number of nodes     : 3
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 22448
Reset error count   : 0
--------------------
Trace Commands

Trace commands provide an `always on' debug feature. Many major functions in Cisco IOS XR software have "trace" functionality to show the last actions it conducted allowing you to analyze function events. Use the show trace commands to display the trace data for a specific feature or process. Use the ? in the CLI to determine if a command has the trace keyword. The following example shows that the show arp command has the trace keyword.
RP/0/RP0/CPU0:router# show arp ? 
A.B.C.D          IP address or hostname of ARP entry
  Bundle-Ether     Aggregated Ethernet interface(s)
  GigabitEthernet  GigabitEthernet/IEEE 802.3 interface(s)
  H.H.H            48-bit hardware address of ARP entry
  MgmtEth          Ethernet/IEEE 802.3 interface(s)
  idb              Show the internal ARP interface data block
  location         specify a node name
  trace            Show trace data for the ARP component
  traffic          ARP traffic statistics
  vrf              Specify a VRF
  |                Output Modifiers
  <cr> 
The following example shows the last 20 events in the address resolution protocol (ARP) table.
RP/0/RP0/CPU0:router# show arp trace tailf last 20 
1349 wrapping entries (2048 possible, 0 filtered, 1349 total)
Apr 19 09:52:29.857 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: creating incomplete entry for 
address: 172.18.105.255
Apr 19 09:52:34.501 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: address resolution failed for 
172.18.105.255
Apr 19 09:52:41.856 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: received address resolution 
request for 172.18.105.255
Apr 19 09:52:46.324 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: address resolution failed for 
172.18.105.255
Apr 19 09:52:59.979 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: entry 172.18.105.255: deleted 
from table
Apr 19 09:59:37.463 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: received address resolution 
request for 172.18.105.255
Apr 19 09:59:37.463 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: creating incomplete entry for 
address: 172.18.105.255
Apr 19 09:59:39.515 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: received address resolution 
request for 172.18.105.255
Apr 19 09:59:42.082 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: address resolution failed for 
172.18.105.255
Apr 19 09:59:45.007 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: entry 172.18.105.255: deleted 
from table
Apr 19 09:59:50.101 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: received address resolution 
request for 172.18.105.255
Apr 19 09:59:50.101 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: creating incomplete entry for 
address: 172.18.105.255
Apr 19 09:59:54.820 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: address resolution failed for 
172.18.105.255
Apr 19 10:00:00.008 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: entry 172.18.105.255: deleted 
from table
Apr 19 10:04:11.675 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: received address resolution 
request for 172.18.105.255
Apr 19 10:04:11.675 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: creating incomplete entry for 
address: 172.18.105.255
Apr 19 10:04:16.272 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: address resolution failed for 
172.18.105.255
Apr 19 10:04:30.028 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: entry 172.18.105.255: deleted 
from table
Apr 19 10:04:44.097 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: received address resolution 
request for 172.18.105.255
Apr 19 10:04:44.097 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: creating incomplete entry for 
address: 172.18.105.255
Apr 19 10:04:48.810 ipv4_arp/arp 0/RP0/CPU0 t1  ARP-TABLE: address resolution failed for 
172.18.105.255
Packets

By default, if packet capture is not enabled on an interface, the show packet command displays only punt packets and software switch packets (router generated packets or anything that impacts the line card CPU). You have the option of retrieving information on packets that are switched and punted in the software using the show captured packets command. You have to turn on packet capture using the capture software packets command, then use the show captured packets command to display the packet capture content.

The following example shows that interface does not have packet capture enabled using the capture software packets command:
RP/0/RP0/CPU0:router# show captured packets ingress interface pos 0/1/0/0 location 
0/RP0/CPU0 
please enable packet capture on interface to see pkts
RP/0/RP0/CPU0:router# 
Note This feature is supported on the Cisco CRS-1.

To turn on packet capture and view capture packet output, perform the following procedure.

SUMMARY STEPS

1. configure

2. interface type instance

3. capture software packets

4. end
or
commit

5. show captured packets {ingress | egress} [interface type instance] [hexdump] [last] [single-line] location node-id

DETAILED STEPS
Command or Action

Purpose

Step 1

configure
Example:

RP/0/RP0/CPU0:router# configure

Enters global configuration mode.

Step 1

interface type instance
Example:

RP/0/RP0/CPU0:router(config)# interface pos 0/1/0/0

Enters interface configuration mode.

Step 2

capture software packets
Example:

RP/0/RP0/CPU0:router(config-if)# capture software packets

Turns on software packet capture for the POS 01/0/0 interface.
Step 3

end
or

commit
Example:

RP/0/RP0/CPU0:router(config-if)# end
or

RP/0/RP0/CPU0:router(config-if)# commit
Saves configuration changes.

•When you issue the end command, the system prompts you to commit changes:
Uncommitted changes found, commit them before 
exiting(yes/no/cancel)? 
[cancel]:
–Entering yes saves configuration changes to the running configuration file, exits the configuration session, and returns the router to EXEC mode.

–Entering no exits the configuration session and returns the router to EXEC mode without committing the configuration changes.

–Entering cancel leaves the router in the current configuration session without exiting or committing the configuration changes.

•Use the commit command to save the configuration changes to the running configuration file and remain within the configuration session.
Step 4

show captured packets {ingress | egress} [hexdump] [interface type instance] [last] [single-line] location node-id
Example:

RP/0/RP0/CPU0:router# show captured packets ingress location cpu 0/1/cpu0

Displays information on packets that are switched and punted in the software.
The following example shows how to turn on packet capture for POS 0/1/0/0.
RP/0/RP0/CPU0:router(config)# interface pos 0/1/0/0
RP/0/RP0/CPU0:router(config-if)# capture software packets
RP/0/RP0/CPU0:router(config-if)# end
Uncommitted changes found, commit them before exiting(yes/no/cancel)? [cancel]: yes
Logging Archive for Harddisk

Use the logging archive command to configure attributes for archiving syslogs. Configuring the logging archive is recommended as sometimes syslog does not make it over the network and the archive can be used for post problem analysis help.

The following example shows how to configure a syslog logging archive that uses the harddisk, is for all severities (0 through 7), and collects logs daily.
RP/0/RP0/CPU0:router# configure 
RP/0/RP0/CPU0:router(config)# logging archive 
RP/0/RP0/CPU0:router(config-logging-arch)# device harddisk 
RP/0/RP0/CPU0:router(config-logging-arch)# severity debugging 
RP/0/RP0/CPU0:router(config-logging-arch)# frequency daily 
SNMP Polling Awareness of SystemOwner, LR Owner, MIB Location

By default, if you configure SNMP in the Secure Domain Router (SDR), you only see what is in the logical router (LR plane) and you do not have snmp access to fan, power, and fabric card information (admin plane). If you add systemowner on the community string using the snmp-server community command, you will have access to the entire system allowing you to poll information such as fabric information and status.

In order to view entire MIB table, the community string needs to have `systemowner'. This allows the user to view admin plane objects as well as LR plane.

Error File Locations and Data Collection Scripts

Errors are logged and stored on the system.

Data collection scripts are used to collect relevant information for troubleshooting the system. Scripts are stored locally The following script types are supported:

•Sysmgr Collection Scripts

•Wdsysmon Collection Scripts

•Shutdown Collection Scripts

•ASIC error Collection Scripts

Error File Locations

Error files are located in the following folders:

•harddisk:

•disk1:

Use the dir command to display a list of files on a file system or in a specific directory. The following examples show how to display the contents of the harddisk: directory, and the files in the var and shutdown directories.
RP/0/RP0/CPU0:router# dir harddisk:
Directory of harddisk:
5           drwx  4096        Tue Oct  4 09:55:32 2005  LOST.DIR
6           drwx  4096        Tue Oct  4 09:55:34 2005  usr
7           drwx  4096        Tue Oct  4 09:55:18 2005  var
131328      -rwx  31744       Fri Apr 28 17:38:34 2006  instdb_backup.tar
15          drwx  4096        Wed Apr  5 18:27:48 2006  dumper
3678        drwx  4096        Wed Mar 15 17:03:48 2006  shutdown
4008        drwx  4096        Thu Mar  2 09:24:10 2006  malloc_dump
39929724928 bytes total (39908044800 bytes free)
RP/0/RP0/CPU0:router# dir harddisk:/var 
Directory of harddisk:/var
8           drwx  4096        Tue Oct  4 09:55:18 2005  log
9           drwx  4096        Tue Oct  4 09:55:18 2005  tmp
39929724928 bytes total (39908044800 bytes free)
Use the cd command to change the current working directory.

The following example shows how to change from the default working directory to the harddisk: directory. The pwd command displays the present working directory.
RP/0/RP0/CPU0:router# cd
RP/0/RP0/CPU0:router# pwd
disk0:/usr
RP/0/RP0/CPU0:router# cd harddisk:
RP/0/RP0/CPU0:router# pwd
harddisk: 
Use the more command to view the contents of a file.

harddisk:

The following folders are located on the harddisk:

•shutdown—Contains shutdown scripts collected for nodes

•var/log—Contains any syslog archive data to harddisk if enabled

•ASIC-ERROR—Contains ASIC error data that resulted in a node reload

•asic_snapshot—Contains ASIC data collected in the ASIC error trigger node reload event

•dumper—Contains all process core files in event process crashes

•malloc_dump—Used for memory compare tool

The following example shows the contents of the harddisk: folder:
RP/0/RP0/CPU0:router# dir harddisk:
Directory of harddisk:
5           drwx  4096        Fri Jun 10 10:27:32 2005  LOST.DIR
6           drwx  4096        Fri Jun 10 10:27:32 2005  usr
7           drwx  4096        Fri Jun 10 10:27:32 2005  var
19          drwx  4096        Tue Apr 18 17:32:46 2006  dumper
1880        drwx  4096        Fri Oct 14 13:52:22 2005  ASIC-ERROR
1884        drwx  4096        Mon Apr 10 07:36:04 2006  shutdown
1901        drwx  4096        Sat Aug 13 20:04:58 2005  asic_snapshots
6623        drwx  4096        Thu Jul  7 07:12:06 2005  malloc_dump
4460        drwx  4096        Wed Jan 25 09:24:00 2006  pm
disk1:

The following folders are located on disk1:

•wdsysmon_debug—Contains wdsysmon debug data collected in event script is run

The following example shows the contents of the disk1: folder. See "Wdsysmon Collection Scripts" section for information on the contents of the wdsysmon_debug folder.
RP/0/RP0/CPU0:router# dir disk1:
Directory of disk1:
2           drwx  16384       Mon Nov 21 13:28:56 2005  LOST.DIR
5           dr-x  16384       Tue Mar  7 17:54:24 2006  bcm-prev
13          dr-x  16384       Tue Mar  7 18:10:19 2006  bcm-cur
3           drwx  16384       Wed Dec 21 21:57:14 2005  wdsysmon_debug
65888       -rwx  54104355    Wed Nov 23 17:27:08 2005  comp-hfr-mini.pie-3.2.2
65984       -rwx  1360942     Tue Mar  7 18:18:15 2006  hfr-k9sec-p.pie-3.3.86.I
66080       -rwx  9215019     Tue Mar  7 18:19:06 2006  hfr-mgbl-p.pie-3.3.86.1I
66176       -rwx  82022548    Tue Mar  7 18:26:20 2006  comp-hfr-mini.vm-3.3.86I
66496       -rwx  31232       Tue Mar  7 20:56:20 2006  instdb_backup.tar
66688       -rwx  3769071     Wed Nov 23 17:28:22 2005  hfr-diags-p.pie-3.2.2
66784       -rwx  1357245     Wed Nov 23 17:29:30 2005  hfr-k9sec-p.pie-3.2.2
66880       -rwx  2440530     Wed Nov 23 17:30:28 2005  hfr-mcast-p.pie-3.2.2
66976       -rwx  7648034     Wed Nov 23 17:31:34 2005  hfr-mgbl-p.pie-3.2.2
67072       -rwx  2414096     Wed Nov 23 17:32:34 2005  hfr-mpls-p.pie-3.2.2
1024655360 bytes total (859799552 bytes free)
Sysmgr Collection Scripts

The sysmgr is responsible for starting, monitoring, stopping, and if necessary, restarting most processes on the system.

Significant sysmgr events are stored in /tmp/sysmgr.log. The log wraps so it is recommended that you save a snapshot to disk at the start of the session. The debug script contains commands that provide a snapshot of the box and also provides details on the specific process having problems.

Wdsysmon Collection Scripts

Wdsysmon is the WatchDog and SYStem MONitor. Wdsysmon monitors memory and CPU resources, watches for IP communications and mutual exclusion object (mutex) deadlocks, issues notifications or alarms when resource thresholds are exceeded, and logs historical process data.

The wdsysmon collection scripts contain wdsysmon debug data. The wdsysmon monitors CPU utilization, diskspace, and memory, and generates an alarm when a threshold is crossed. The output contains trace information for process resource utilization.
RP/0/RP0/CPU0:router# dir disk1:/wdsysmon_debug 
Directory of disk1:/wdsysmon_debug
196736      -rwx  37151       Wed Dec 21 17:47:40 2005  debug_evm.364641
196832      -rwx  39422       Wed Dec 21 21:42:10 2005  debug_evm.364640
196928      -rwx  39577       Wed Dec 21 21:57:14 2005  debug_evm.307296
1024655360 bytes total (927186944 bytes free)
RP/0/RP0/CPU0:router# more debug_evm.364641 
/pkg/bin/wdsysmon_debug_evm_script invoked by pid 45103 (wdsysmon) for pid 8200.
Called by wd_heartbeat_timeout_hndlr at line 390 at 17:47:40.029 UTC Wed Dec 21 
2005.
----------------------------------------------------------------
Output from pidin:
     pid tid name               prio STATE       Blocked                 
       1   1 procnto              0f READY                               
       1   2 procnto             63r RECEIVE     1                       
       1   3 procnto             63r RECEIVE     1                       
       1   5 procnto             63r RECEIVE     1                       
       1   6 procnto             63r RECEIVE     1                       
       1   7 procnto             63r RECEIVE     1                       
       1   8 procnto             63r RECEIVE     1                       
       1   9 procnto             63r RECEIVE     1                       
       1  10 procnto             63r RECEIVE     1                       
       1  11 procnto             63r RECEIVE     1                       
       1  12 procnto             10r RECEIVE     1                       
       1  13 procnto             10r RECEIVE     1                       
       1  14 procnto             10r RECEIVE     1                       
       1  15 procnto             10r RECEIVE     1                       
       1  16 procnto             10r RECEIVE     1                       
       1  17 procnto             10r RECEIVE     1                       
       1  18 procnto             10r RECEIVE     1                       
       1  19 procnto             10r RECEIVE     1                       
       1  20 procnto             11r RECEIVE     1                       
.
.
.
86112   3 pkg/bin/instdir     10r RECEIVE     1                       
   86112   4 pkg/bin/instdir     10r CONDVAR     482f9fd8                
  364641   1 pkg/bin/ksh         10r SIGSUSPEND                          
  381026   1 pkg/bin/ksh         10r SIGSUSPEND                          
  389219   1 pkg/bin/pidin       10r REPLY       1                       
----------------------------------------------------------------
Output from attach_process -A -p 8200 -i 1
Attaching to process pid = 8200 (pkg/bin/devc-conaux)
No tid specified, following all threads
DLL Loaded by this process 
-------------------------------
DLL path                 Text addr. Text size  Data addr. Data size  Version
/pkg/lib/libsysmgr.dll   0xfc122000 0x0000df88 0xfc0c2b14 0x000004ac        0
/pkg/lib/libcerrno.dll   0xfc130000 0x00002f24 0xfc133000 0x00000128        0
/pkg/lib/libcerr_dll_tbl.dll 0xfc134000 0x00004964 0xfc133128 0x00000148        
0
/pkg/lib/libltrace.dll   0xfc139000 0x00007adc 0xfc133270 0x00000148        0
/pkg/lib/libinfra.dll    0xfc141000 0x000341a4 0xfc1333b8 0x00000bbc        0
/pkg/lib/cerrno/libinfra_error.dll 0xfc1121dc 0x00000cd8 0xfc176000 0x000000a8  
      0
/pkg/lib/libios.dll      0xfc177000 0x0002dc38 0xfc1a5000 0x00002000        0
/pkg/lib/cerrno/libevent_manager_error.dll 0xfc1a7000 0x00000e88 0xfc133f74 0x00
000088        0
/pkg/lib/libc.dll        0xfc1a8000 0x00079d70 0xfc222000 0x00002000        0
.
.
.
/pkg/lib/cerrno/libsysdb_error_callback.dll 0xfc4f3000 0x0000168c 0xfc47ece8 0x0
0000088        0
/pkg/lib/cerrno/libsysdb_error_distrib.dll 0xfc4f5000 0x00001780 0xfc47ed70 0x00
000088        0
Iteration 1 of 1
-----REPLY (node node0_RP0_CPU0, pid 81994)
-------------------------
Current process = "pkg/bin/devc-conaux", PID = 8200 TID = 1 
trace_back: #0 0xfc1642b8 [MsgSendv]
trace_back: #1 0xfc14e358 [msg_sendv]
trace_back: #2 0xfc49d870 [sysdb_lib_send_opt_v]
trace_back: #3 0xfc4b86fc [sysdb_lib_notification_send_reg]
trace_back: #4 0xfc4b8a7c [sysdb_notification_register_internal]
trace_back: #5 0xfc4b8e74 [_sysdb_register_notification]
trace_back: #6 0xfc274eec [tty_sysdb_cached_item_notify]
trace_back: #7 0xfc275278 [tty_sysdb_cached_items_open]
trace_back: #8 0xfc156b90 [event_conn_evm_handler]
trace_back: #9 0xfc1563ec [event_conn_timeout]
trace_back: #10 0xfc152908 [evm_timeout]
trace_back: #11 0xfc153954 [_event_pulse_handler]
trace_back: #12 0xfc151e94 [event_dispatch]
trace_back: #13 0xfc26c5d8 [tty_io_devctl]
trace_back: #14 0xfc26d7ec [tty_server_main]
trace_back: #15 0x482000b0 [<N/A>]
ENDOFSTACKTRACE
Current process = "pkg/bin/devc-conaux", PID = 8200 TID = 2 
trace_back: #0 0xfc1d4048 [SignalWaitinfo]
trace_back: #1 0xfc1b7d40 [sigwaitinfo]
trace_back: #2 0xfc155594 [event_signal_thread]
ENDOFSTACKTRACE
.
.
.
----------------------------------------------------------------
Output from top_procs 
Computing times...Unable to enter cbreak mode.: Inappropriate I/O control operat
ion
Error entering control break mode
node0_RP0_CPU0: 97 procs, 1 cpu, 1.04 delta, 00:03:56 uptime
Memory: 4096 MB total, 3.630 GB free, sample time: Wed Dec 21 17:47:41 2005
cpu 0 idle: 93.22%, kernel:  0.29%
      pid   mem MB   user cpu kernel cpu   delta  % ker  % tot name    
    28691    0.371      0.108      0.003   0.064   0.00   6.37 devb-ata
   405603    0.109      0.008      0.010   0.002   0.19   0.19 top_procs
    41001    0.531      0.087      0.061   0.001   0.09   0.09 dsc
    45103    2.132      0.166      0.098   0.001   0.00   0.09 wdsysmon
    28694   36.304      0.230      0.188   0.000   0.00   0.00 eth_server
    86101    0.792      0.111      0.034   0.000   0.00   0.00 shelfmgr
    77896    1.054      0.175      0.063   0.000   0.00   0.00 gsp
    32792    0.484      0.100      0.012   0.000   0.00   0.00 bcm_process
    32794    0.097      0.020      0.011   0.000   0.00   0.00 stp_process
    32802    0.234      0.353      0.120   0.000   0.00   0.00 sysmgr
Output from top_procs 
----------------------------------------------------------------
Exiting at  at 17:47:41.487 UTC Wed Dec 21 2005.
Shutdown Collection Scripts

Shutdown scripts are generated when a failure occurs and contains information on why the failure occurred. The system attempts to gather as much information as possible upon failure.
RP/0/RP0/CPU0:router# dir harddisk:/shutdown 
Directory of harddisk:/shutdown
3683        drwx  4096        Wed Dec 21 17:49:34 2005  node0_RP0_CPU0
3672        drwx  4096        Sun Mar 12 15:48:20 2006  node0_1_CPU0
241041792   -rwx  12334       Wed Dec 28 16:00:36 2005  node0_1_CPU0.log.first.gz
241041888   -rwx  11181       Sun Mar 12 15:48:08 2006  node0_1_CPU0.log.next.gz
39929724928 bytes total (39908044800 bytes free).
The shutdown scripts are saved as a zipped file (.gz). The following example shows how to view the zipped files using Ksh commands.
RP/0/RP0/CPU0:router# run 
# gunzip node0_1_CPU0.log.first.gz 
# cat node0_1_CPU0.log.first.gz 
ASIC error Collection Scripts

ASICs are monitored for errors. There are preset thresholds for each ASIC which allows a specific number of errors before an action is taken.

ASIC error scripts are generated when major thresholds are exceeded and the node is reloaded. The scripts contain a snapshot of error types and number of errors for the ASIC on which they occurred.
RP/0/RP0/CPU0:router# dir harddisk:/ASIC-ERROR
Directory of harddisk:/ASIC-ERROR
1881        drwx  4096        Thu Jun 16 08:32:14 2005  node0_3_CPU0
2141        drwx  4096        Fri Oct 14 13:52:22 2005  node0_5_CPU0
See the "ASIC Errors" section for information on asic errors.

Monitoring

Monitoring allows you to view interface, controller fabric, or controller SONET counters, and auto-updating statistics on processes and threads in real-time. The following commands are used for monitoring:

•monitor interface Command

•monitor controller Command

•monitor processes Command

•monitor threads Command

monitor interface Command

Use the monitor interface command to monitor interface counters in real-time. The following example shows the output of the monitor interface command.

Note The Cisco IOS XR Manageability Package is required to use the monitor interface command.

The following example shows the output of the monitor interface command with a specified interface.
RP/0/RP0/CPU0:router# monitor interface tenGigE 0/3/0/7 
CRS-A_IOX            Monitor Time: 00:00:08          SysUptime: 118:58:20  
TenGigE0/3/0/7 is up, line protocol is up 
Encapsulation ARPA  
Traffic Stats:(2 second rates)                                     Delta   
Input  Packets:                   1466462                            4  
Input  pps:                             1   
Input  Bytes:                   379782697                         1730  
Input  Kbps (rate):                     6                       (  0%)   
Output Packets:                   2403444                            2  
Output pps:                             0   
Output Bytes:                   269350468                          140   
Output Kbps (rate):                     0                       (  0%)  
Errors Stats:  
Input  Total:                           2                            0  
Input  CRC:                             0                            0   
Input  Frame:                           0                            0   
Input  Overrun:                         0                            0   
Output Total:                           0                            0   
Output Underrun:                        0                            0  
Quit='q', Freeze='f', Thaw='t', Clear='c', Interface='i', 
Next='n', Prev='p'  
Brief='b', Detail='d', Protocol(IPv4/IPv6)='r' 
The following example shows the output of the monitor interface command.
RP/0/RP0/CPU0:router# monitor interface 
CRS-A_IOX            Monitor Time: 00:00:08          SysUptime: 118:58:50   
Interface             In(bps)      Out(bps)     InBytes/Delta  OutBytes/Delta 
MgmtEth0/RP0/CPU0/0   45014/  0%    52156/  0%     3.9G/14686    842.2M/17016  
Bundle-POS100             0/ --%        0/ --%        0/0             0/0      
Bundle-POS101             0/ --%        0/ --%        0/0             0/0      
MgmtEth0/RP1/CPU0/0       0/  0%        0/  0%    27.8M/0          1.8M/0      
FINT0/RP1/CPU0            0/  0%        0/  0%   121.8M/0          3.5M/0      
TenGigE0/3/0/0            0/  0%        0/  0%        0/0             0/0      
TenGigE0/3/0/1          443/  0%      683/  0%   532.1M/144      744.4M/222   
TenGigE0/3/0/2            0/  0%        0/  0%        0/0             0/0      
TenGigE0/3/0/3            0/  0%        0/  0%        0/0             0/0      
TenGigE0/3/0/4            0/  0%        0/  0%        0/0             0/0      
TenGigE0/3/0/5            0/  0%     5342/  0%     2.4M/0        166.8M/1743   
TenGigE0/3/0/6            0/  0%        0/  0%    11.5T/0         11.5T/0     
TenGigE0/3/0/7          852/  0%      963/  0%   379.7M/276      269.3M/312    
TenGigE0/3/0/6.10         0/  0%        0/  0%     1.8T/0          1.8T/0      
TenGigE0/3/0/6.20         0/  0%        0/  0%     1.8T/0          1.8T/0      
TenGigE0/3/0/6.30         0/  0%        0/  0%     1.8T/0          1.8T/0      
TenGigE0/3/0/6.40         0/  0%        0/  0%     1.8T/0          1.8T/0      
TenGigE0/3/0/6.50         0/  0%        0/  0%     1.8T/0          1.8T/0      
TenGigE0/3/0/6.60         0/  0%        0/  0%     1.8T/0          1.8T/0      
TenGigE0/3/0/1.99         0/  0%        0/  0%     5040/0          4978/0      
POS0/5/0/0                0/  0%        0/  0%        0/0             0/0      
POS0/5/0/1            14658/  0%      796/  0%   611.8M/4764     657.5M/259    
POS0/5/0/2                0/  0%        0/  0%        0/0          2.2M/0     
POS0/5/0/3                0/  0%        0/  0%        0/0         70.0M/0      
POS0/5/0/4            20346/  0%    20033/  0%   992.7M/6638     990.7M/6536   
POS0/5/0/5              306/  0%      306/  0%     4.0M/100        4.0M/100    
POS0/5/0/6                0/  0%        0/  0%        0/0             0/0      
POS0/5/0/8                0/  0%        0/  0%        0/0          2.2M/0      
POS0/5/0/10             729/  0%      649/  0%    97.8M/238      121.5M/212    
POS0/5/0/11             723/  0%    14187/  0%    99.5M/238      648.7M/4664  
POS0/5/0/12               0/  0%        0/  0%        0/0             0/0       
Quit='q',     Clear='c',    Freeze='f', Thaw='t', 
Next set='n', Prev set='p', Bytes='y',  Packets='k' 
monitor controller Command

Use the monitor controller command to monitor controller fabric or SONET counters in real-time. The counters are refreshed every two seconds. The following example shows the output of the monitor controller command.
RP/0/RP0/CPU0:router# monitor controller sonet 0/1/0/0
CRS-8_X1             Monitor Time: 00:00:00          SysUptime: 19:02:19
CRS-8_X1             Monitor Time: 00:00:02          SysUptime: 19:02:21
CRS-8_X1             Monitor Time: 00:00:04          SysUptime: 19:02:23
CRS-8_X1             Monitor Time: 00:00:06          SysUptime: 19:02:25
CRS-8_X1             Monitor Time: 00:00:08          SysUptime: 19:02:27
CRS-8_X1             Monitor Time: 00:00:10          SysUptime: 19:02:29
CRS-8_X1             Monitor Time: 00:00:12          SysUptime: 19:02:31
CRS-8_X1             Monitor Time: 00:00:14          SysUptime: 19:02:33
CRS-8_X1             Monitor Time: 00:00:16          SysUptime: 19:02:35
CRS-8_X1             Monitor Time: 00:00:18          SysUptime: 19:02:38
CRS-8_X1             Monitor Time: 00:00:20          SysUptime: 19:02:40
Controller for SONET0/1/0/0                 0 ( 0 per-se                   Delt
  Path LOP                                  0 ( 0 per-sec)                    0
Controller Stats:                           0 ( 0 per-se                   Delt
  Path LOP                                  0 ( 0 per-sec)                    0
  Path AIS                                  0 ( 0 per-sec)                    0
  Path RDI                                  0 ( 0 per-sec)                    0
  Path BIP                                  0 ( 0 per-sec)                    0
  Path FEBE                                 0 ( 0 per-sec)                    0
  Path NEWPTR                               0 ( 0 per-sec)                    0
  Path PSE                                  0 ( 0 per-sec)                    0
  Path NSE                                  0 ( 0 per-sec)                    0
  Line AIS                                  0 ( 0 per-sec)                    0
  Line RDI                                  0 ( 0 per-sec)                    0
  Line BIP                                  0 ( 0 per-sec)                    0
  Line FEBE                                 0 ( 0 per-sec)                    0
  Section LOS                               1 ( 0 per-sec)                    0
  Section LOF                               0 ( 0 per-sec)                    0
  Section BIP                               0 ( 0 per-sec)                    0
Quit='q', Freeze='f', Thaw='t', Clear='c'
The output allows you to verify if the SONET interfaces have any path errors (Layer 2). Information about the operational status of SONET layers on a particular SONET port is displayed. The output is the same as the show controllers sonet command except the display refreshes every 2 seconds.

monitor processes Command

Use the monitor processes command to display the top ten processes based on CPU usage in real time. The display refreshes every 10 seconds.
RP/0/RP0/CPU0:router# monitor processes 
Computing times...
235 processes; 822 threads; 4468 channels, 5805 fds
CPU states: 98.0% idle, 0.3% user, 1.5% kernel
Memory: 4096M total, 3492M avail, page size 4K
      JID TIDS Chans   FDs Tmrs   MEM   HH:MM:SS   CPU  NAME
        1   28  238    15    1      0   15:50:01  1.58% procnto-600-smp-cisco-in
str
       57    5  238   833    0     4M    0:00:10  0.13% dllmgr
       75   12  230     9    3     1M    0:02:05  0.03% qnet
      145    4   29    39    5   408K    0:00:01  0.03% devc-vty
       53    1    1     7    0   108K    0:00:01  0.03% bcm_logger
       52    5   15     9    4   708K    0:00:07  0.01% bcm_process
      249    3   52    37    9     1M    0:00:00  0.01% lpts_pa
      109    5    5    13    3   756K    0:00:00  0.01% bcdl_agent
    65554    7   16     3    3     7M    0:02:33  0.01% devb-ata
      291   19   22    67    5   828K    0:00:01  0.01% raw_ip
See Chapter 7, "Process Monitoring and Troubleshooting," for more information on processes and process monitoring.

monitor threads Command

Use the monitor threads command to display the top ten threads based on CPU usage in real time. The display refreshes every 10 seconds.
RP/0/RP0/CPU0:router# monitor threads
Computing times...
235 processes; 822 threads;
CPU states: 96.7% idle, 0.9% user, 2.2% kernel
Memory: 4096M total, 3492M avail, page size 4K
   JID   TID PRI STATE  HH:MM:SS      CPU  COMMAND
     1    25  10 Run     0:00:16     2.24% procnto-600-smp-cisco-instr
 65754     1  10 Rply    0:00:00     0.53% top
    59     7  55 Rcv     0:01:41     0.04% eth_server
    59     1  10 Rcv     0:00:47     0.04% eth_server
   308     5  10 Rcv     0:00:21     0.04% shelfmgr
    59     3  50 Sem     0:00:40     0.04% eth_server
   308     1  10 Rcv     0:00:15     0.04% shelfmgr
   341    18  10 Rcv     0:00:02     0.04% udp
   261     9  10 Rcv     0:00:17     0.04% netio
   261     4  10 Rcv     0:00:10     0.04% netio
Gathering Information for Technical Support

Before contacting Cisco Technical Support, please review the information provided at the following URL:

http://www.cisco.com/public/news_training/tac_overview.html

For more information on capturing system information, see Chapter 10 "Collecting System Information".