Troubleshooting the System


Troubleshooting the System
This chapter provides information and instructions for using the system command line interface (CLI) for troubleshooting any issues that may arise during system operation.
The following topics are included:
 
Detecting Faulty Hardware Using Component LEDs
Upon applying power to the chassis, power is provided to the upper and lower fan trays, and every application and line card that is installed.
Each PFU, application, and line card installed in the system has light emitting diodes (LEDs) that indicate it's status. This section provides information and instructions pertaining to using LEDs to verify for verifying that all of the installed components are functioning properly.
Important: As the system progresses through its boot process, some cards may have no immediate LED activity. It is recommended that several minutes elapse prior to checking the LEDs on the various cards to verify the installation.
 
Using the CLI to View Component LEDs
The status of application and line card LEDs can be viewed through the CLI by entering the following command:
show leds all
The following displays a sample of this command’s output.
Slot 01: Run/Fail: Green | Active: Off | Standby: Green
Slot 08: Run/Fail: Green | Active: Green | Standby: Off
Status: Green | Service: Off |
Slot 09: Run/Fail: Green | Active: Off | Standby: Green
Status: Green | Service: Off |
Slot 12: Run/Fail: Green | Active: Green | Standby: Off
Slot 14: Run/Fail: Green | Active: Green | Standby: Off
Slot 17: Run/Fail: Green | Active: Green | Standby: Off
Slot 24: Run/Fail: Green | Active: Green | Standby: Off
Slot 25: Run/Fail: Green | Active: Off | Standby: Green
Slot 30: Run/Fail: Green | Active: Green | Standby: Off
Slot 33: Run/Fail: Green | Active: Off | Standby: Off
Slot 40: Run/Fail: Green | Active: Green | Standby: Off
The status of the chassis’ two Power Filter Units (PFUs) can be viewed by entering the following command:
show power chassis
 
Checking the LED on the PFU(s)
Each PFU has a single status LED labeled POWER.
This LED should be green for normal operating conditions.
PFU LED Location
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information below to diagnose the problem.
PFU POWER LED States
 
 
Checking the LEDs on the SMC(s)
Each SMC is equipped with the following LEDs as shown in the following figure:
 
 
SMC LEDs
The possible states for all SMC LEDs are described in the sections that follow.
 
SMC RUN/FAIL LED States
The SMC RUN/FAIL LED indicates the overall status of the card. This LED should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SMC RUN/FAIL LED States
The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
SNMP MIB Reference for information on associated status and alarm conditions.
Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LEDs on the SPC(s) section in this chapter for troubleshooting information.
 
 
SMC Active LED States
The Active LED on the SMC indicates that the software is loaded on the card and it is ready for operation. For the SMC installed in chassis slot 8, this LED should be green for normal operation. For the SMC installed in slot 9, this LED should be off for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SMC Active LED States
Verify that the Standby LED on the redundant SMC is also blinking green. If so, there is an issue with the active SMC. Refer to one or more of the following to help analyze this problem:
The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
SNMP MIB Reference for information on associated status and alarm conditions.
Card is not receiving powerORCard in Standby Mode
Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, please refer to the SMC RUN/FAIL LED States section of this chapter for troubleshooting information.
Check the state of the Standby LED. If it is green, the card is in standby mode. If needed, refer to the Configuring PSC and Line Card Availability section of the Configuring System Settings chapter in this reference for information on making the card active.
 
 
SMC Standby LED States
The Standby LED on the SMC indicates that software is loaded on the card, but it is serving as a redundant component. For the SMC installed in slot 9, this LED should be green for normal operation. For the SMC installed in slot 8, this LED should be off for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SMC Standby LED States
Verify that the Active LED on the redundant SMC is also blinking green. If so, there is an issue with the active SMC. Refer to one or more of the following to help analyze this problem:
The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
SNMP MIB Reference for information on associated status and alarm conditions.
Card is not receiving powerORCard in Active Mode
Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, please refer to the SMC RUN/FAIL LED States section of this chapter for troubleshooting information on.
Check the state of the Active LED. If it is green, the card is in active mode. If needed, refer to the Manually Initiating an SMC Switchover section in this chapter for information on configuring the card to serve as a redundant component.
 
 
SMC Status LED States
The Status LEDs on the SMC indicate the status of system level hardware such as installed cards, fans, and PFUs. This LED is green during normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information also provided to diagnose the problem.
SMC Status LED States
Check the RUN/FAIL LEDs for all installed application cards, and line cards. If any are red or off, refer to the troubleshooting information in this chapter pertaining to that device.
The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
SNMP MIB Reference for information on associated status and alarm conditions.
Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the SMC RUN/FAIL LED States section of this chapter for troubleshooting information.
 
 
SMC Service LED States
The Service LEDs on the SMCs indicate that the system requires maintenance or service (e.g. the system could not locate a a valid software image at boot-up, or a high temperature condition exists).
This LED is off during normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SMC Service LED States
The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
SNMP MIB Reference for information on associated status and alarm conditions.
 
 
SMC Busy LED States
The Busy LEDs on the SMCs indicate that there is activity on one of their memory devices. Activity is displayed for the following memory devices:
 
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SMC Busy LED States
Important: If removing the SMC from the chassis, it is recommended that you wait until this LED is off to ensure the integrity of all data being transferred to or from the memory device.
 
 
Checking the LEDs on the PSC(s)
Each PSC is equipped with status LEDs as listed below:
 
 
PSC LEDs
The possible states for all PSC LEDs are described below:
 
PSC RUN/FAIL LED States
The PSC RUN/FAIL LED indicates the overall status of the card. This LED should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
PSC RUN/FAIL LED States
The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
SNMP MIB Reference for information on associated status and alarm conditions.
Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU(s) section in this chapter for troubleshooting information.
 
 
PSC Active LED States
The Active LED on the PSC indicates that the software is loaded on the card and that the card is ready for operation.When the system first boots up, all installed PSCs are booted into standby mode. The system must then be configured as to which PSCs should serve as redundant components (i.e. remain in standby mode) and which should function as active components.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
PSC Active LED States
Verify that the Standby LED on a redundant PSC is also blinking green. If so, there is an issue with the PSC that was active and is transferring its processes.
Refer to the Monitoring the System chapter of this reference for information on determining the status of the PSC and system software processes and functionality.
Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, please refer to the PSC RUN/FAIL LED States section of this chapter for troubleshooting information.
Check the state of the Standby LED. If it is green, the card is in standby mode. This is normal operation for the initial power-up. If needed, refer to the Configuring PSC and Line Card Availability section of the Configuring System Settings chapter in this reference for information on making the card active.
 
 
PSC Standby LED States
The Standby LED on the PSC indicates that software is loaded on the card, but the card is serving as a redundant component. When the system first boots up, all installed PSCs are booted into standby mode. The system must then be configured as to which PSCs should serve as redundant components (i.e. remain in standby mode) and which should function as active components.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
PSC Standby LED States
Verify that the Active LED on the redundant PSC is also blinking green.
The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
SNMP MIB Reference for information on associated status and alarm conditions.
Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, please refer to the PSC RUN/FAIL LED States section of this chapter for information on troubleshooting.
Check the state of the Active LED. If it is green, the card is in active mode. If needed, refer to the Manually Initiating a PSC Migration section in this chapter for information on configuring the card to serve as a redundant component.
 
 
Checking the LEDs on the SPIO(s)
Each SPIO is equipped with status LEDs as listed below:
In addition to the LEDs listed above, each interface to the management network (both RJ-45 and SFP) are equipped with the following LEDs:
 
SPIO LED Locations
The possible states for all of the SPIO's LEDs are described in the sections that follow.
SPIO RUN/FAIL LED States
The SPIO's RUN/FAIL LED indicates the overall status of the card. This LED should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SPIO RUN/FAIL LED States
Refer to the Monitoring the System chapter of this reference for information on determining the status of system hardware components.
Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU(s) section in this chapter for troubleshooting information.
 
 
SPIO Active LED States
The Active LED on the SPIO indicates that the software is loaded on the card and that the card is ready for operation. For the SPIO installed in chassis slot 24, this LED should be green for normal operation. For the SPIO installed in slot 25, this LED should be off for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SPIO Active LED States
Card is not receiving power OR Card in Standby Mode
Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the SPIO RUN/FAIL LED States section of this chapter for troubleshooting information.
Check the state of the Standby LED. If it is green, the card is in standby mode. This is normal for the SPIO in slot 25 since the chassis automatically places the card into standby mode at boot up.
 
 
SPIO Standby LED States
The Standby LED on the SPIO indicates that software is loaded on the card, but it is serving as a redundant component. For the SPIO installed in slot 25, this LED should be green for normal operation. For the SPIO installed in slot 24, this LED should be off for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SPIO Standby LED States
The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
SNMP MIB Reference for information on associated status and alarm conditions.
Card is not receiving power OR Card in Active Mode
Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the SPIO RUN/FAIL LED States section of this chapter for information on troubleshooting.
Check the state of the Active LED. If it is green, the card is in active mode. This is normal for the SPIO in slot 24 since the chassis automatically make the card active at boot up.
 
 
SPIO Interface Link LED States
The Link LED, associated with a particular SPIO interface indicates the status of the network link. This LED should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SPIO Interface Link LED States
Important: This LED will not indicate the presence of a network link until the interface parameters are set during the software configuration process.
No power to card OR Link is down
Verify that the RUN/FAIL LED is green. If so, the card is receiving power. If it is off, refer to the SPIO RUN/FAIL LED States section of this chapter for troubleshooting information.
 
 
SPIO Interface Activity LED States
The Activity LED associated with a particular SPIO interface indicates the presence of traffic on the network link. This LED should be green when data is being transmitted or received over the interface.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SPIO Interface Activity LED States
 
 
Checking the LEDs on the Ethernet 10/100 and Ethernet 1000/Quad Gig-E Line Card(s) (QGLC)
Each Ethernet 10/100, Ethernet 1000 Line Card and QGLC is equipped with status LEDs as listed below:
 
In addition to the LEDs listed above, each network interface is equipped with the following LEDs:
 
Ethernet 10/100 and GigE LEDs
 
QGLC LEDs
The possible states for all LEDs on the Ethernet 10/100 and Ethernet 1000/QGLC cards are as follows:
 
Ethernet 10/100 and Ethernet 1000/QGLC RUN/FAIL LED States
The RUN/FAIL LEDs on the Ethernet 10/100 and Ethernet 1000/QGLC Line Cards indicate the overall status of the cards. These LEDs should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
Ethernet 10/100 and Ethernet 1000 RUN/FAIL LED States
Refer to the Monitoring the System chapter of this reference for information on determining the status of system hardware components.
Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU(s) section in this chapter for troubleshooting information.
 
 
Ethernet 10/100 and Ethernet 1000/QGLC Active LED States
The Active LEDs on the Ethernet 10/100 and Ethernet 1000/QGLC Line Cards indicate that the operating software is loaded on the card and that the card is ready for operation.
Important: Quad Gigabit Ethernet line cards (QGLC) only work in an ASR 5000 behind a PSC.
The line cards installed will remain in a ready mode until their corresponding PSC is made active via configuration. While in ready mode the Active LED should be off. After the PSC is made active, the line card installed in the upper-rear chassis slot behind the PSC will also be made active. The line card installed in the lower-rear chassis slot behind the PSC will enter the standby mode.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
Ethernet 10/100 and Ethernet 1000/QGLC Active LED States
Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the Ethernet 10/100 and Ethernet 1000/QGLC RUN/FAIL LED States section of this chapter for troubleshooting information.
Check the state of the Standby LED. If it is green, the card is in standby mode. This is normal operation for the initial power-up. If needed, refer to the Configuring PSC and Line Card Availability section of the Configuring System Settings chapter in this reference for information on making the card active.
 
 
Ethernet 10/100 and Ethernet 1000/QGLC Standby LED States
The Standby LEDs on the Ethernet 10/100 and Ethernet 1000/QGLC Line Cards indicate that software is loaded on the cards, but are serving as redundant components.
Important: Quad Gigabit Ethernet line cards (QGLC) only work in an ASR 5000 behind a PSC.
The line cards installed will remain in a ready mode until their corresponding PSC is made active via configuration. While in ready mode, the Active LED should be off. After the PSC is made active, the line card installed in the upper-rear chassis slot behind the PSC will also be made active. The line card installed in the lower-rear chassis slot behind the PSC will also enter the standby mode.
The possible states for this LED are described below. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
Ethernet 10/100 and Ethernet 1000/QGLC Standby LED States
If green for line cards installed in slots 17 through 23 and 26 through 32, refer to the Monitoring the System chapter of this reference for information on determining the status of the line card and system software processes and functionality.
Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the Ethernet 10/100 and Ethernet 1000/QGLC RUN/FAIL LED States section of this chapter for troubleshooting information.
Check the state of the Active LED. If it is green, the card is in standby mode. If needed, refer to the Manually Initiating a Line Card Switch section in this chapter for information on configuring the card to serve as a redundant component.
 
 
Ethernet 10/100 and Ethernet 1000/QGLC Interface Link LED States
The Link LEDs, associated with a particular network interface on the Ethernet 10/100 and Ethernet 1000/QGLC Line Cards, indicate the status of the network link. These LEDs should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
Ethernet 10/100 and Ethernet 1000 Interface Link LED States
Important: This LED will not indicate the presence of a network link until the interface parameters are set during the software configuration process.
No power to cardORLink is down
Verify that the RUN/FAIL LED is green. If so, the card is receiving power. If it is off, refer to Ethernet 10/100 and Ethernet 1000/QGLC RUN/FAIL LED States section for troubleshooting information.
 
 
Ethernet 10/100 and Ethernet 1000/QGLC Interface Activity LED States
The Activity LEDs, associated with a particular network interface on the Ethernet 10/100 and Ethernet 1000/QGLC Line Cards, indicate the presence of traffic on the network link. These LEDs should be green when data is being transmitted or received over the interface.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
Ethernet 10/100 and Ethernet 1000 Interface Activity LED States
 
 
Checking the LEDs on the RCC(s)
Each RCC is equipped with status LEDs as listed below:
 
 
RCC LED Locations
The possible states for all of the SPIO's LEDs are described in the sections that follow.
 
RCC RUN/FAIL LED States
The RCC's RUN/FAIL LED indicates the overall status of the card. This LED should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
RCC RUN/FAIL LED States
Refer to the Monitoring the System chapter of this reference for information on determining the status of system hardware components.
Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU(s) section in this chapter for troubleshooting information.
 
 
RCC Active LED States
The Active LED on the RCC indicates that the card is being used. For normal operation, this LED should be off on both RCCs.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
RCC Active LED States
Refer to either the Checking the LEDs on the PAC(s) or Checking the LEDs on the PSC(s) section of this chapter to determine which PSC has failed.
Card is not receiving power OR Card in Standby Mode
Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the RCC RUN/FAIL LED States section of this chapter for troubleshooting information.
Check the state of the Standby LED. If it is green, the card is in standby mode. This is the normal operating mode.
 
 
RCC Standby LED States
The Standby LED on the RCC indicates that software is loaded on the card and is ready to provide a path for data or signalling traffic from a line card to a redundant PSC. This LED should be on for normal operation for both RCCs installed.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
RCC Standby LED States
Card is not receiving power ORCard in Active Mode
Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the RCC RUN/FAIL LED States section of this chapter for troubleshooting information.
Check the state of the Active LED. If it is green, the card is in active mode and the RCC is actively routing traffic from a line card installed behind a PSC that has failed.
Refer to either the Checking the LEDs on the PAC(s) or Checking the LEDs on the PSC(s) section of this chapter to determine which PSC has failed. Information on determining the cause of the failure can be found in the Monitoring the System chapter of this reference.
 
 
Testing System Alarm Outputs
The system provides the following two physical alarm mechanisms:
 
System Audible Alarm: Located on the SMC, the speaker is used to provide an audible indicator that a minor, major, or critical alarm has occurred.
CO Alarms Interface: Located on the SPIO, this interface provides a 10-pin connector that enables three normally-closed dry-contact relays for the triggering of external audio and/or visual indicators. These indicators can be used to alert that either a minor, major, or critical alarm has occurred.
The operation of these alarms can be tested by issuing the following command:
test alarm { audible | central-office [ critical | major | minor ] }
critical: Specifies that the critical CO Alarms output is to be tested.
major: Specifies that the major CO Alarms output is to be tested.
minor: Specifies that the minor CO Alarms output is to be tested.
 
When this command is executed, the specified alarm is activated for a period of 10 seconds. After this time, the alarm will return to its previous condition.
 
Taking Corrective Action
In the event that an issue was discovered with an installed application or line card, depending on the severity, it may be necessary to take corrective action.
The system provides several redundancy and fail-over mechanisms to address issues with application and line cards in order to minimize system downtime and data loss. These mechanisms are described in the sections that follow.
 
 
Manually Initiating a Management Card Switchover
When the system boots up, the SMC installed in chassis slot eight will boot into the “active” mode and begin booting other system components. The SMC installed in chassis slot nine will automatically be booted into “standby” mode dictating that it will serve as a redundant component. However, the active SMC will frequently synchronize currently running tasks or processes with the redundant SMC.
In the event of a critical failure on the SMC in slot eight, system control will be switched to the redundant SMC in slot nine. This is a relatively seamless transition because the two are synchronized. The formerly active SMC will then enter the standby mode allowing it to be safely replaced or restored.
In the event that an issue arises that is not severe enough for the system to perform an automatic switchover, a manual switch over can be invoked by executing the following instructions from the Exec mode prompt:
 
[local]host_name#
Step 1
For SMC:
 
card smc switchover
You will receive the following prompt:
 
Are You Sure? [Yes|No]:
Step 2
Press Y to start the switchover.
Step 3
 
show card table
Check the entry in the Oper State column next to the SMC just switched. Its state should be Standby.
 
Manually Initiating a Packet Processing Card Migration
When the system boots up, all installed PSCs enter the “standby” mode. The standby mode indicates that the card is available for use but is not configured for operation. Installed components can be made active through the software configuration process. Cards that are not configured to enter the “active” mode will remain in standby mode for use as redundant components.
In the event of PSC critical failure, tasks will be automatically be migrated from the active card to a redundant card in standby mode. The line card installed behind the PSC that was formerly active will still be used to maintain the interfaces to external network equipment. Installed Redundancy Crossbar Cards (RCCs) will provide a path for signalling and data traffic between the line card and the now active PSC. Therefore, redundant PSCs do not require that line cards be installed behind them.
In the event that an issue arises that is not severe enough for the system to perform an automatic migration, a manual migration can be invoked. Follow the instructions below to manually initiate a PSC migration. These instructions assume you are at the root prompt for the Exec mode:
 
[local]host_name#
Step 1
For PSC:
 
card psc migration from <original_slot#> to <final_slot#>
 
 
You will receive the following prompt:
 
Are You Sure? [Yes|No]:
Step 2
Press Y to start the migration.
Step 3
 
show card table
Check the entry in the Oper State column next to the PSC that was just migrated from. Its state should be Standby. The state of the PSC migrated to should be Active.
 
Manually Initiating a Line Card Switch
Line cards are installed in the half-height slots at the rear of the chassis. This design allows for two line cards (both Ethernet 10/100 both Ethernet 1000 (or both QGLCs on an ASR 5000)) to be installed behind every application card. When two line cards are installed, as the application card that they are installed behind is booted, the card in the upper-rear chassis slot will automatically be made active while the card in the lower-rear chassis slot will automatically be placed in standby mode. In the event that the active card experiences a failure, the system will automatically switch traffic to the standby card in the lower slot.
In the event that a SPIO experiences a failure, the system will automatically switch traffic to the redundant SPIO installed behind the redundant SMC.
In the event that an issue arises that is not severe enough for the system to perform an automatic switch, a manual switch can be performed. Follow the instructions below to manually initiate a line card or SPIO switch. These instructions assume you are at the root prompt for the Exec mode:
 
[local]host_name#
Step 1
 
card [ lc | spio ] switch [ to <slot#> ]
slot# can be any of the following integer values: 17 through 23, 26 through 39, or 42 through 48.
Important: This keyword is only used if the lc keyword is used. The line card being migrated to must be in the standby mode.
 
You will receive the following prompt:
 
Are You Sure? [Yes|No]:
Step 2
Press Y to start the switch.
Step 3
 
show card table
Check the entry in the Oper State column next to the line card or SPIO that was just switched from. Its state should be Standby. The state of the line card or SPIO switched to should be Active.
Halting Cards
PSCs or line cards that are in either the active or standby modes can be halted. Halting these cards places them into the “offline” mode. When in this mode, the card will become unusable for session processing as either an active or redundant component.
If a card in the active mode is halted, its tasks, processes, or network connections will be migrated or switched to a redundant component prior to entering the offline mode.
This section provides information and instructions for initiating a card halt and restoring halted components.
 
 
Initiate a Card Halt
Follow the instructions below to manually initiate a card halt. These instructions assume you are at the root prompt for the Exec mode:
 
[local]host_name#
Step 1
 
card halt <slot#>
slot# is the chassis slot number in which the card to be halted is installed. It can be any integer value between 1 and 7, 10 through 48.
You will receive the following prompt:
 
Are You Sure? [Yes|No]:
Step 2
Press Y to start the halt of the card.
Step 3
 
show card table
Check the entry in the Oper State column next to the line card that was just halted. Its state should be Offline. If the card was in active mode prior to the execution of this command, the state of the redundant component associated with it should now be Active.
 
Restoring a Previously Halted Card
Follow the instructions below to restore a card that was previously halted. These instructions assume you are at the root prompt for the Exec mode:
 
[local]host_name#
Step 1
 
card reboot <slot#> -force
You will receive the following prompt:
 
Are You Sure? [Yes|No]:
Step 2
Press Y to start the reboot of the card.
Step 3
 
show card table
Check the entry in the Oper State column next to the line card that was just restored. Its state should be the state of that it was in before it was halted.
Verifying Network Connectivity
There are multiple commands supported by the system to verify and/or troubleshoot network connectivity. Note that network connectivity can only be tested once system interfaces and ports have been configured and bound.
The commands specified in this section should be issued on a context-by-context basis. This is due to the fact that contexts act like virtual private networks (VPNs) in that they operate independently of the others. Therefore, ports, interfaces, and routes configured in one context can not be tested from another without additional configuration.
To switch between contexts you must enter the following command at the root prompt for the Exec mode:
context <context_name>
context_name is the name of the context that you wish to switch to. The following prompt appears:
[context_name]host_name#
 
Using the Ping Command
Using the ping command verifies the system’s ability to communicate with a remote node in the network by passing data packets between and measuring the response. This command is useful in verifying network routing and if a remote node is able to respond at the IP layer. The command has the following syntax:
ping <host_ip_address> [ count <num_packets> ] [ pattern <packet_pattern> ] [ size <octet_count> ] [ src { <src_host_name> | <src_host_ip_address> } ]
<host_ip_address>
host_ip_address specifies the remote node using the node’s assigned IP address specified using the standard IPv4.
count <num_packets>
num_packets must be within the range 1 through 10000. The default is 5.
pattern <packet_pattern>
packet_pattern must be specified in hexadecimal format with a value in the range hexadecimal 0x0000 through 0xFFFF.
packet_pattern must begin with a ‘0x’ followed by up to 4 hexadecimal digits.
size <octet_count>
octet_count must be a value in the range 40 through 18432. The default is 56.
src { <src_host_name> | <src_host_ip_address> }
src_host_name: specifies the source node using the node’s logical host name which must be resolved via DNS lookup.
src_host_ip_address: specifies the source node using the node’s assigned IP address specified using the standard IPv4.
 
The following displays a sample of a successful response.
PING 192.168.250.1 (192.168.250.1): 56 data bytes
64 bytes from 192.168.250.1: icmp_seq=0 ttl=255 time=0.4 ms
64 bytes from 192.168.250.1: icmp_seq=1 ttl=255 time=0.2 ms
64 bytes from 192.168.250.1: icmp_seq=2 ttl=255 time=0.2 ms
64 bytes from 192.168.250.1: icmp_seq=3 ttl=255 time=0.2 ms
64 bytes from 192.168.250.1: icmp_seq=4 ttl=255 time=0.2 ms
--- 192.168.250.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.2/0.2/0.4 ms
If no response is received from the target follow these troubleshooting procedures:
 
Using the Traceroute Command
The traceroute command collects information on the route data will take to a specified host. This is a useful troubleshooting command that can be used to identify the source of significant packet delays or packet loss on the network. This command can also be used to identify bottle necks in the routing of data over the network.
The command has the following syntax:
traceroute { <host_name> | <host_ip_address> } [ count <packets> ] [ df ] [ maxttl <max_ttl> ] [ minttl <min_ttl> ] [ port <port_number> ] [ size <octet_count> ] [ src { <src_host_name> | <src_host_ip_address> } ] [ timeout <seconds> ]
<host_name>
host_name specifies the remote node using the node’s logical host name which must be resolved via DNS lookup.
<host_ip_address>
host_ip_address specifies the remote node using the node’s assigned IP address specified using the standard IPv4.
maxttl <max_ttl>
max_ttl must be specified as a value in the range of 1 through 255. It is an error if max_ttl is less than min_ttl, whether min_ttl is specified or defaulted.
minttl <min_ttl>
min_ttl must be specified as a value in the range of 1 through 255. It is an error if min_ttl is greater than max_ttl, whether max_ttl is specified or defaulted.
port <port_number>
port_number must be a value in the range 1 through 65535. The default port is 33434.
octet_count must be a value in the range 40 through 32768. The default is 40.
src { <src_host_name> | <src_host_ip_address> }
src_host_name: specifies the remote node using the node’s logical host name which must be resolved via DNS lookup.
src_host_ip_address: specifies the remote node using the node’s assigned IP address specified using the standard IPv4.
timeout <seconds>
seconds must be a value in the range 2 through 100. The default is 5.
 
The following displays a sample output.
traceroute to 192.168.250.1 (192.168.250.1), 30 hops max, 40 byte packets
1 192.168.250.1 (192.168.250.1) 0.446 ms 0.235 ms 0.178 ms
 
Viewing IP Routes
The system provides a mechanism for viewing route information to a specific node or for an entire context. This information can be used to verify network connectivity and to ensure the efficiency of the network connection. The command has the following syntax:
show ip route [ <route_ip_address> [ <route_gw_address> ] ]
<route_ip_address>
<route_gw_address>
 
If no keywords are specified, all IP routes within the context’s routing table are displayed.
The following displays a sample of this command’s output showing a context routing table.
"*" indicates the Best or Used route.
Destination Nexthop Protocol Prec Cost Interface
*0.0.0.0/0 10.0.4.1 static 0 0 SPIO1
*10.0.4.0/24 0.0.0.0 kernel 0 0 SPIO1
*10.0.4.0/32 0.0.0.0 kernel 0 0 SPIO1
*10.0.4.3/32 0.0.0.0 kernel 0 0 SPIO1
*10.0.4.255/32 0.0.0.0 kernel 0 0 SPIO1
 
Viewing the Address Resolution Protocol Table
The system provides a mechanism for viewing Address Resolution Protocol (ARP) table information to a specific node or for an entire context. This information can be used to verify that when the system sends an ARP packet, it receives valid responses from other network nodes. The command has the following syntax:
show ip arp [ <arp_ip_address> ]
arp_ip_address specifies a specific network node for which to display ARP information. If this keyword is not specified, all entries within the context’s ARP table are displayed.
Important: When the VPN Manager restarts, it removes all interfaces from the kernel and thus the kernel removes all ARP entries. When this happens, the NPU still holds all of the ARP entries so that there is no traffic disruption. When this happens, from a user point of view, show ip arp is broken since this command gathers information from the kernel and not the NPU.
The following displays a sample of this command’s output showing a context’s ARP table.
Flags codes:
C - Completed, M - Permanent, P - Published, ! - Not answered
T - has requested trailers
Address Link Type Link Address Flags Mask Interface
10.0.4.240 ether 00:05:47:02:20:20 C SPIO1
10.0.4.7 ether 00:05:47:02:03:36 C SPIO1
10.0.4.1 ether 00:01:30:F2:7F:00 C SPIO1
 
Using the System Diagnostic Utilities
The system provides protocol monitor and test utilities that can are useful when troubleshooting or verifying configurations. The information generated by these utilities can in many cases either identify the root cause of a software or network configuration issue or, at the very least, greatly reduce the number of possibilities.
This section contains information and instructions for using these utilities.
 
Using the Monitor Utility
For troubleshooting purposes, the system provides a powerful protocol monitoring utility. This tool can be used to display protocol information for a particular subscriber session or for every session being processed.
Caution: The monitor tool is intrusive in that it may cause session processing delays and/or data loss. Therefore, it should be used only when troubleshooting .
 
 
Using the Protocol Monitor
The system’s protocol monitor displays information for every session that is currently being being processed. Depending on the number of protocols monitored, and the number of sessions in progress, a significant amount of data is generated. It is highly recommended that logging be enabled on your terminal client in order to capture all of the information that is generated.
Follow the instructions in this section to invoke and configure the protocol monitoring tool.
Step 1
 
monitor protocol
The following output is displayed.
 
MONITOR GLOBAL PROTOCOLS:
11 - SNMP 21 - L2TP (Admin only)
12 - RADIUS Authentication (Admin only) 22 - L2TPMGR (Admin only)
13 - RADIUS Accounting (Admin only) 23 - L2TP Data(Admin only)
14 - A11 (R-P Interface) (Admin only) 24 - GTP (Admin only)
15 - Mobile IPv4 (Admin only) 25 - GTPCMGR (Admin only)
16 - A11MGR (Admin only) 26 - GTPU (Admin only)
17 - PPP (Admin only) 27 - GTPP (Admin only)
18 - A10 (Admin only) 28 - DHCP (Admin only)
19 - User L3 (Admin only) 29 - CDR (Admin only)
31 - RADIUS COA (Admin only) 30 - DHCPV6 (Admin only)
51 - SCTP (Admin only)
32 - MIP Tunnel (Admin only) 52 - M3UA (Admin only)
33 - L3 Tunnel (Admin only) 53 - SCCP (Admin only)
34 - CSS Data (Admin only) 54 - TCAP (Admin only)
35 - CSS Signaling (Admin only) 55 - MAP (Admin only)
36 - EC Diameter (Admin only) 56 - RANAP (Admin only)
37 - SIP (IMS) (Admin only) 57 - GMM (Admin only)
38 - IPSec IKE Only (Admin only) 58 - GPRS-NS (Admin only)
59 - BSSGP (Admin only)
40 - IPSec IKEv2 Only (Admin only)
41 - IPSG RADIUS Signal (Admin only) 61 - SSCOP (Admin only)
42 - ROHC (Admin only) 62 - SSCFNNI (Admin only)
43 - WiMAX R6 (Admin only) 63 - MTP3 (Admin only)
44 - WiMAX Data (Admin only) 64 - LLC (Admin only)
45 - SRP (Admin only) 65 - SNDCP (Admin only)
46 - BCMCS SERV AUTH (Admin only) 66 - BSSAP+ (Admin only)
47 - RSVP (Admin only) 67 - SMS (Admin only)
48 - Mobile IPv6 (Admin only) 68 - MTP2 (Admin only)
49 - ASNGWMGR (Admin only)
50 - STUN                   (Admin only)
70 - DNS Client (Admin only)
75 - App Specific Diameter (Admin only)
  81 - S1AP                   (Admin only)   82 - NAS (Admin only)
(B)egin Protocol Decoding, (Q)uit, <ESC> Prev Menu
Select:
Step 2
Step 3
Repeat step 2 as needed to choose multiple protocols.
Step 4
Press B to begin the protocol monitor. If you selected any protocol other than 11 (SNMP), the following message is displayed:
 
WARNING!!! You have selected options that can DISRUPT USER SERVICE
Existing CALLS MAY BE DROPPED and/or new CALLS MAY FAIL!!!
(Under heavy call load, some debugging output may not be displayed)
Proceed? - Select (Y)es or (N)o
Step 5
Enter Y to proceed with the monitor or N to go back to the previous menu.
 
C - Control Events (ON )
D - Data Events (ON )
E - EventID Info (ON )
I - Inbound Events (ON )
O - Outbound Events (ON )
S - Sender Info (OFF)
T - Timestamps (ON )
X - PDU Hexdump (OFF)
A - PDU Hex/Ascii (OFF)
+/- Verbosity Level ( 1)
L - Limit Context (OFF)
M - Match Newcalls (ON )
R - RADIUS Dict (no-override)
G - GTPP Dict (no-override)
Q)uit, <ENTER> Display Options, <ESC> Prev Menu, <SPACE> Pause
Step 6
The current state, ON (enabled) or OFF (disabled), is shown to the right of each option.
Step 7
Press the Enter key to refresh the screen and begin monitoring.
The monitor remains active until disabled. To quit the protocol monitor and return to the prompt, press q.
 
Using the Protocol Monitor for a Specific Subscriber
The system’s protocol monitor can be used to display information for a specific subscriber session that is currently being processed. Depending on the number of protocols monitored, and the number of sessions in progress, a significant amount of data is generated. It is highly recommended that logging be enabled on your terminal client in order to capture all of the information that is generated.
Follow the instructions in this section to invoke and configure the protocol monitoring tool for a specific subscriber session.
Step 1
 
monitor subscriber
The following screen is displayed, followed by a list of all available monitoring methods:
 
MONITOR SUBSCRIBER:
1) By MSID/IMSI
2) By Username
3) By Callid
4) By IP Address
5) By IPv6 Address
6) Next-Call
7) Next-BCMCS-Call
8) Next-BCMCS-Service-Request
9) By IMEI
a) By MSISDN
b) Next-1xRTT Call
c) Next-ASNGW Call
d) Next-CLOSEDRP Call
e) Next-EVDO-Rev0 Call
f) Next-EVDO-RevA Call
g) Next-GGSN Call
h) Next-HA Call
i) Next-IPSG Call
j) Next-LNS Call
k) Next-PDIF Call
l) By ASN Peer IP Address
m) By PDIF Peer IP Address
n) By Peer LAC IP Address
o) By SGSN IP Address
p) By PCF IP Address
r) By Peer FA IP Address
s) By IPSG Peer IP Address
t) Next-ASNPC Call
u) Next-OpenRP Call
v) Next-Call By APN
 
Q) Quit
 
<ESC> Return to Previous Menu
 
Step 2
Step 3
If no session matching the specified criteria was being processed when the monitor was invoked, the following output is displayed:
 
NO MATCHING CALL - waiting for a matching call to connect...
C - Control Events (ON ) 11 - PPP (ON ) 21 - L2TP (ON )
D - Data Events (ON ) 12 - A11 (ON ) 22 - L2TPMGR (OFF)
E - EventID Info (ON ) 13 - RADIUS Auth (ON ) 23 - L2TP Data (OFF)
I - Inbound Events (ON ) 14 - RADIUS Acct (ON ) 24 - GTPC (ON )
O - Outbound Events (ON ) 15 - Mobile IPv4 (ON ) 25 - GTPCMGR (OFF)
S - Sender Info (OFF) 16 - A11MGR (OFF) 26 - GTPU (OFF)
T - Timestamps (ON ) 17 - SESSMGR (ON ) 27 - GTPP (ON )
X - PDU Hexdump (OFF) 18 - A10 (OFF) 28 - DHCP (ON )
A - PDU Hex/Ascii (OFF) 19 - User L3 (OFF) 29 - CDR (ON )
+/- Verbosity Level ( 1) 31 - Radius COA (ON ) 30 - DHCPV6 (ON )
L - Limit Context (OFF) 32 - MIP Tunnel (ON ) 53 - SCCP (OFF)
M - Match Newcalls (ON ) 33 - L3 Tunnel (OFF) 54 - TCAP (OFF)
R - RADIUS Dict: (no-override) 34 - CSS Data (OFF) 55 - MAP (ON )
G - GTPP Dict: (no-override) 35 - CSS Signal (OFF) 56 - RANAP (OFF)
Y - Multi-Call Trace (OFF) 36 - EC Diameter (ON ) 57 - GMM (ON )
37 - SIP (IMS) (OFF) 58 - GPRS-NS (OFF)
40 - IPSec IKEv2 (OFF) 59 - BSSGP (OFF)
41 - IPSG RADIUS (ON ) 64 - LLC (OFF)
42 - ROHC (OFF) 65 - SNDCP (OFF)
43 - WiMAX R6 (ON ) 66 - BSSAP+ (OFF)
44 - WiMAX Data (OFF) 67 - SMS (OFF)
45 - SRP (OFF)
46 - BCMCS SERV AUTH (OFF)
47 - RSVP (ON )
48 - Mobile IPv6 (ON )
49 - ASNGWMGR (OFF)
50 - STUN (IMS) (OFF)
75 - App Specific Diameter OFF
 
(Q)uit, <ESC> Prev Menu, <SPACE> Pause, <ENTER> Re-Display Options
 
 
Step 4
The current state, ON (enabled) or OFF (disabled), is shown to the right of each option.
Important: Option Y for performing multi-call traces is only supported for use with the GGSN.
Step 5
Repeat step 6 as needed to enable or disable multiple protocols.
Step 6
Press the Enter key to refresh the screen and begin monitoring.
The following displays a portion of a sample of the monitor’s output for a subscriber named user2@aaa. The default protocols were monitored.
 
---------------------------------------------------------------------------
Incoming Call:
---------------------------------------------------------------------------
MSID: 0000012345 Callid: 002dc6c2
Username: user2@aaa SessionType: unknown
Status: Active Service Name: xxx1
Src Context: source Dest Context:
---------------------------------------------------------------------------
 
<<<<OUTBOUND 10:02:35:415 Eventid:25001(0)
PPP Tx PDU (9)
PAP 9: Auth-Ack(1), Msg=
 
<<<<OUTBOUND 10:02:35:416 Eventid:25001(0)
PPP Tx PDU (14)
IPCP 14: Conf-Req(1), IP-Addr=192.168.250.70
 
<<<<OUTBOUND 10:02:35:416 Eventid:25001(0)
PPP Tx PDU (27)
CCP 27: Conf-Req(1), MPPC, Stac-LZS, Deflate, MVRCA
 
INBOUND>>>>> 10:02:35:517 Eventid:25000(0)
PPP Rx PDU (30)
IPCP 30: Conf-Req(1), IP-Comp VJ-Comp, IP-Addr=0.0.0.0, Pri-DNS=0.0.0.0,
Sec-DNS=0.0.0.0
 
<<<<OUTBOUND 10:02:35:517 Eventid:25001(0)
PPP Tx PDU (26)
IPCP 26: Conf-Rej(1), IP-Comp VJ-Comp, Pri-DNS=0.0.0.0, Sec-DNS=0.0.0.0
 
INBOUND>>>>> 10:02:35:517 Eventid:25000(0)
PPP Rx PDU (12)
IPCP 12: Conf-Ack(1), IP-Addr=192.168.250.70
 
INBOUND>>>>> 10:02:35:518 Eventid:25000(0)
PPP Rx PDU (31)
LCP 31: Prot-Rej(1), Rejected-Protocol=CCP (0x80fd)
 
INBOUND>>>>> 10:02:35:518 Eventid:25000(0)
PPP Rx PDU (12)
IPCP 12: Conf-Req(2), IP-Addr=0.0.0.0
 
<<<<OUTBOUND 10:02:35:518 Eventid:25001(0)
PPP Tx PDU (14)
IPCP 14: Conf-Nak(2), IP-Addr=192.168.250.87
 
INBOUND>>>>> 10:02:35:519 Eventid:25000(0)
PPP Rx PDU (12)
IPCP 12: Conf-Req(3), IP-Addr=192.168.250.87
 
The monitor remains active until disabled. To quit the protocol monitor and return to the prompt, press q.
Using the DHCP Testing Tool
The CLI provides a mechanism for testing network connectivity with and configuration of DHCP servers. This functionality can be extremely useful in determining the accuracy of the system’s DHCP configuration and troubleshooting the server’s response time.
This tool provides a mechanism from getting an IP address for one or more DHCP servers that the system is configured to communicate with. In addition, the tool provides a mechanism for releasing the address back to the DHCP server.
Important: This tool must be executed from the context in which the DHCP server(s) are configured.
To execute the DHCP test tool enter the following command:
dhcp test dhcp-service { <service_name> } [ all | <server <ip_addr>|
Sample dhcp test dhcp-service Command Output
 
 
 
 

Cisco Systems Inc.
Tel: 408-526-4000
Fax: 408-527-0883