> Troubleshooting the System
> This chapter provides information and instructions for using the system command line interface (CLI) for troubleshooting any issues that may arise during system operation.
> The following topics are included:
> Detecting Faulty Hardware Using Component LEDs
> Upon applying power to the chassis, power is provided to the upper and lower fan trays, and every application and line card that is installed.
> Each PFU, application, and line card installed in the system has light emitting diodes (LEDs) that indicate it's status. This section provides information and instructions pertaining to using LEDs to verify for verifying that all of the installed components are functioning properly.
> Important: As the system progresses through its boot process, some cards may have no immediate LED activity. It is recommended that several minutes elapse prior to checking the LEDs on the various cards to verify the installation.
> Using the CLI to View Component LEDs
> The status of application and line card LEDs can be viewed through the CLI by entering the following command:
> show leds all
> The following displays a sample of this command’s output.
> Slot 01: Run/Fail: Green | Active: Off | Standby: Green
> Slot 08: Run/Fail: Green | Active: Green | Standby: Off
> Status: Green | Service: Off |
> Slot 09: Run/Fail: Green | Active: Off | Standby: Green
> Status: Green | Service: Off |
> Slot 12: Run/Fail: Green | Active: Green | Standby: Off
> Slot 14: Run/Fail: Green | Active: Green | Standby: Off
> Slot 17: Run/Fail: Green | Active: Green | Standby: Off
> Slot 24: Run/Fail: Green | Active: Green | Standby: Off
> Slot 25: Run/Fail: Green | Active: Off | Standby: Green
> Slot 30: Run/Fail: Green | Active: Green | Standby: Off
> Slot 33: Run/Fail: Green | Active: Off | Standby: Off
> Slot 40: Run/Fail: Green | Active: Green | Standby: Off
> The status of the chassis’ two Power Filter Units (PFUs) can be viewed by entering the following command:
> show power chassis
> Checking the LED on the PFU(s)
> Each PFU has a single status LED labeled POWER.
> This LED should be green for normal operating conditions.
> PFU LED Location
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information below to diagnose the problem.
> PFU POWER LED States

> Color | Description | Troubleshooting
> Green | PFU powered with no errors detected | None needed.
> None | PFU is not receiving power |
> • | Verify that the power switch is in the ON position.
> • | Verify that the RTN and -VDC lugs are attached properly according to the instructions provided in this document.
> • | Verify that the ground lug is attached properly.Verify that the power source is on and is supplying the correct voltage and sufficient current.
> • | Check the cables from the power source to the rack for continuity.
> • | If a fuse panel is installed between the power distribution frame (PDF) and the chassis, verify that the fuses are intact.
> • | If a fuse panel is installed between the PDF and the chassis, check the cables from the fuse panel to the chassis for continuity.
> • | If all of the above suggestions have been verified, then it is likely that the PFU is not functional. Please contact your service representative.

> ‌
> Checking the LEDs on the SPC(s)
> Each SPC is equipped with the following LEDs :

> • | RUN/FAIL

> • | Active

> • | Standby

> • | Status

> • | Service

> SPC LED Locations
> The possible states for all SPC LEDs are described in the sections that follow.
> SPC RUN/FAIL LED States
> The SPC's RUN/FAIL LED indicates the overall status of the card. This LED should be green for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SPC RUN/FAIL LED States

> Color | Description | Troubleshooting
> Green | Card powered with no errors detected | None needed.
> Blinking Green | Card is initializing and/or loading software | This is normal operation during boot-up.
> Red | Card powered with error(s) detected | Errors were detected during the POSTs. It is likely that the errors were logged to the system's command line interface during boot.
> None | Card is not receiving power |
> • | Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU(s) section in this chapter for troubleshooting information.
> • | Verify that the power source is supplying ample voltage and current to the chassis.
> • | Verify that the card is properly installed per the instructions in the Hardware Installation and Administration Guide.
> • | If all of the above suggestions have been verified, it is possible that the SPC is not functional. Please contact your service representative.

> ‌
> SPC Active LED States
> The Active LED on the SPC indicates that the software is loaded on the card and it is ready for operation. For the SPC installed in chassis slot 8, this LED should be green for normal operation. For the SPC installed in slot 9, this LED should be off for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SPC Active LED States

> Color | Description | Troubleshooting
> Green | Card is active | None needed for the SPC in slot 8. If green for the SPC in slot 9, verify that the SPC in slot 8 is installed properly according to the instructions in this document.
> Blinking Green | Tasks or processes being migrated from the active SPC to the redundant/secondary SPC | Verify that the Standby LED on the redundant SPC is also blinking green. If so, there is an issue with the active SPC. Refer to the Monitoring the System chapter of this reference for information on determining the status of the SPC and/or for information determining the status of system software processes and functionality.
> None | Card is not receiving power ORCard in Standby Mode |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the SPC RUN/FAIL LED States section of this chapter for troubleshooting information.
> • | Check the state of the Standby LED. If it is green, the card is in standby mode. This is normal for the SPC in slot 9 since it is automatically placed into standby mode at boot up.

> ‌
> SPC Standby LED States
> The Standby LED on the SPC indicates that software is loaded on the card, but it is serving as a redundant component. For the SPC installed in slot 9, this LED should be green for normal operation. For the SPC installed in slot 8, this LED should be off for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SPC Standby LED States

> Color | Description | Troubleshooting
> Green | Card is in redundant mode | None needed for the SPC in slot 9. If green for the SPC in slot 8, then verify it is installed properly according to the instructions in this document.
> Blinking Green | Tasks or processes being migrated from the active SPC to the redundant/secondary SPC | Verify that the Active LED on the redundant SPC is also blinking green. If so, there is an issue with the active SPC. Refer to the Monitoring the System chapter of this reference for information on determining the status of the SPC and system software processes and functionality.
> None | Card is not receiving power ORCard in Active Mode |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the SPC RUN/FAIL LED States section of this chapter for troubleshooting information on.
> • | Check the state of the Active LED. If it is green, the card is in active mode. This is normal for the SPC in slot 8 since it is automatically made active at boot up.

> ‌
> SPC Status LED States
> The Status LEDs on the SPC indicate the status of system level hardware such as installed cards, fans, and PFUs. This LED is green during normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information also provided to diagnose the problem.
> SPC Status LED States

> Color | Description | Troubleshooting
> Green | No system errors detected | None needed.
> Red | Failures Detected |
> • | Check the RUN/FAIL LEDs for all installed application cards, and line cards. If any are red or off, refer to the troubleshooting information in this chapter pertaining to that device.
> • | Refer to the Monitoring the System chapter of this reference for information on determining the status of system hardware components.
> None | Card is not receiving power | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the SPC RUN/FAIL LED States section of this chapter for troubleshooting information.

> ‌
> SPC Service LED States
> The Service LEDs on the SPCs indicate that the system requires maintenance or service (e.g. the system could not locate a a valid software image at boot-up, or a high temperature condition exists).
> This LED is off during normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SPC Service LED States

> Color | Description | Troubleshooting
> Yellow | System requires maintenance (fan filter, temperature warning, PFU outage, etc.) | Refer to the Monitoring the System chapter of this reference for information on determining the status of system hardware components and system software processes and functionality.
> None | Card is not receiving power | No maintenance needed.

> ‌
> Checking the LEDs on the PAC(s)
> Each PAC is equipped with status LEDs as listed below:

> • | RUN/FAIL

> • | Active

> • | Standby

> • | Status

> • | Service

> PAC LED Locations
> The possible states for all of the PAC's LEDs are described in the sections that follow.
> PAC RUN/FAIL LED States
> The PAC's RUN/FAIL LED indicates the overall status of the card. This LED should be green for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> PAC RUN/FAIL LED States

> Color | Description | Troubleshooting
> Green | Card powered with no errors detected | None needed.
> Blinking Green | Card is initializing and/or loading software | None needed.
> Red | Card powered with error(s) detected | Errors were detected during the POSTs. It is likely that the errors were logged to the system's command line interface during the boot process. Refer to the Monitoring the System chapter of this reference for information on determining the status of system hardware components.
> None | Card is not receiving power |
> • | Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU(s) section in this chapter for troubleshooting information.
> • | Verify that the power source is supplying ample voltage and current to the chassis.
> • | Verify that the card is properly installed per the instructions in the Hardware Installation and Administration Guide.
> • | If all of the above suggestions have been verified, it is possible that the PAC is not functional. Please contact your service representative.

> ‌
> PAC Active LED States
> The Active LED on the PAC indicates that the software is loaded on the card and that the card is ready for operation.When the system first boots up, all installed PACs are booted into standby mode. The system must then be configured as to which PACs should serve as redundant components (i.e., remain in standby mode) and which should function as active components.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> PAC Active LED States

> Color | Description | Troubleshooting
> Green | Card is active | The first time power is applied to the system, all of the PACs should be booted into the standby mode. Therefore, this LED should be off.
> Blinking Green | Tasks or processes being migrated from an active PAC to a redundant/secondary PAC | Verify that the Standby LED on a redundant PAC is also blinking green. If so, there is an issue with the PAC that was active and is transferring its processes. Refer to the Monitoring the System chapter of this reference for information on determining the status of the PAC and system software processes and functionality.
> None | Card is not receiving powerORCard in Standby Mode |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the PAC RUN/FAIL LED States section of this chapter for troubleshooting information.
> • | Check the state of the Standby LED. If it is green, the card is in standby mode. This is normal operation for the initial power-up. If needed, refer to the Configuring PAC/PSC and Line Card Availability section of the Configuring System Settings chapter of this reference for information on making the card active.

> ‌
> PAC Standby LED States
> The Standby LED on the PAC indicates that software is loaded on the card, but the card is serving as a redundant component. When the system first boots up, all installed PACs are booted into standby mode. The system must then be configured as to which PACs should serve as redundant components (i.e. remain in standby mode) and which should function as active components.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> PAC Standby LED States

> Color | Description | Troubleshooting
> Green | Card is in redundant mode | The first time power is applied to the system, all of the PACs should be booted into the standby mode. Therefore, this is normal operation.
> Blinking Green | Tasks or processes being migrated from the active PAC to the redundant/secondary PAC | Verify that the Active LED on the redundant PAC is also blinking green. If so, there is an issue with the active PAC and the system is transferring its processes. Refer to the Monitoring the System chapter of this reference for information on determining the status of the PAC and system software processes and functionality.
> None | Card is not receiving powerORCard in Active Mode |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the PAC RUN/FAIL LED States section of this chapter for information on troubleshooting.
> • | Check the state of the Active LED. If it is green, the card is in active mode.

> ‌
> Checking the LEDs on the SMC(s)
> Each SMC is equipped with the following LEDs as shown in the following figure:

> • | RUN/FAIL

> • | Active

> • | Standby

> • | Status

> • | Service

> • | Busy

> SMC LEDs
> The possible states for all SMC LEDs are described in the sections that follow.
> SMC RUN/FAIL LED States
> The SMC RUN/FAIL LED indicates the overall status of the card. This LED should be green for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SMC RUN/FAIL LED States

> Color | Description | Troubleshooting
> Green | Card powered with no errors detected | None needed.
> Blinking Green | Card is initializing and/or loading software | This is normal operation during boot-up.
> Red | Card powered with error(s) detected | Errors were detected during the POSTs. It is likely that the errors were logged to the system's command line interface during boot. Refer to one or more of the following to help analyze this problem:
> • | The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
> • | The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
> • | SNMP MIB Reference for information on associated status and alarm conditions.
> None | Card is not receiving power |
> • | Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LEDs on the SPC(s) section in this chapter for troubleshooting information.
> • | Verify that the power source is supplying ample voltage and current to the chassis.
> • | Verify that the card is properly installed per the instructions in the Hardware Installation and Administration Guide.
> • | If all of the above suggestions have been verified, it is possible that the SMC is not functional. Please contact your service representative.

> ‌
> SMC Active LED States
> The Active LED on the SMC indicates that the software is loaded on the card and it is ready for operation. For the SMC installed in chassis slot 8, this LED should be green for normal operation. For the SMC installed in slot 9, this LED should be off for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SMC Active LED States

> Color | Description | Troubleshooting
> Green | Card is active | None needed for the SMC in slot 8. If green for the SMC in slot 9, verify that the SMC in slot 8 is installed properly according to the instructions in this document.
> Blinking Green | Tasks or processes being migrated from the active SMC to the redundant/secondary SMC | Verify that the Standby LED on the redundant SMC is also blinking green. If so, there is an issue with the active SMC. Refer to one or more of the following to help analyze this problem:
> • | The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
> • | The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
> • | SNMP MIB Reference for information on associated status and alarm conditions.
> None | Card is not receiving power ORCard in Standby Mode |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, please refer to the SMC RUN/FAIL LED States section of this chapter for troubleshooting information.
> • | Check the state of the Standby LED. If it is green, the card is in standby mode. If needed, refer to the Configuring PAC/PSC and Line Card Availability section of the Configuring System Settings chapter in this reference for information on making the card active.

> ‌
> SMC Standby LED States
> The Standby LED on the SMC indicates that software is loaded on the card, but it is serving as a redundant component. For the SMC installed in slot 9, this LED should be green for normal operation. For the SMC installed in slot 8, this LED should be off for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SMC Standby LED States

> Color | Description | Troubleshooting
> Green | Card is in redundant mode | None needed for the SMC in slot 9. If green for the SMC in slot 8, then verify it is installed properly according to the instructions in this document.
> Blinking Green | Tasks or processes being migrated from the active SMC to the redundant/secondary SMC | Verify that the Active LED on the redundant SMC is also blinking green. If so, there is an issue with the active SMC. Refer to one or more of the following to help analyze this problem:
> • | The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
> • | The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
> • | SNMP MIB Reference for information on associated status and alarm conditions.
> None | Card is not receiving power ORCard in Active Mode |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, please refer to the SMC RUN/FAIL LED States section of this chapter for troubleshooting information on.
> • | Check the state of the Active LED. If it is green, the card is in active mode. If needed, refer to the Manually Initiating an SPC/SMC Switchover section in this chapter for information on configuring the card to serve as a redundant component.

> ‌
> SMC Status LED States
> The Status LEDs on the SMC indicate the status of system level hardware such as installed cards, fans, and PFUs. This LED is green during normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information also provided to diagnose the problem.
> SMC Status LED States

> Color | Description | Troubleshooting
> Green | No system errors detected | None needed.
> Red | Failures Detected |
> • | Check the RUN/FAIL LEDs for all installed application cards, and line cards. If any are red or off, refer to the troubleshooting information in this chapter pertaining to that device.
> • | Refer to one or more of the following to help analyze this problem:
> • | The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
> • | The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
> • | SNMP MIB Reference for information on associated status and alarm conditions.
> None | Card is not receiving power | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the SMC RUN/FAIL LED States section of this chapter for troubleshooting information.

> ‌
> SMC Service LED States
> The Service LEDs on the SMCs indicate that the system requires maintenance or service (e.g. the system could not locate a a valid software image at boot-up, or a high temperature condition exists).
> This LED is off during normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SMC Service LED States

> Color | Description | Troubleshooting
> Yellow | System requires maintenance (fan filter, temperature warning, PFU outage etc.) | Refer to one or more of the following to help analyze this problem:
> • | The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
> • | The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
> • | SNMP MIB Reference for information on associated status and alarm conditions.
> None | Card is not receiving power | No maintenance needed.

> ‌
> SMC Busy LED States
> The Busy LEDs on the SMCs indicate that there is activity on one of their memory devices. Activity is displayed for the following memory devices:

> • | CompactFlash module

> • | PCMCIA device

> • | Nand Flash (used to store SMC firmware).

> • | Hard Drive

> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SMC Busy LED States

> Color | Description | Troubleshooting
> Green/ Blinking Green | Data is being read from/written to one of the memory devices. | No maintenance needed. Important: If removing the SMC from the chassis, it is recommended that you wait until this LED is off to ensure the integrity of all data being transferred to or from the memory device.
> None | The memory devices are not in use. | No maintenance needed.

> ‌
> Checking the LEDs on the PSC(s)
> Each PSC is equipped with status LEDs as listed below:

> • | RUN/FAIL

> • | Active

> • | Standby

> • | Status

> • | Service

> PSC LEDs
> The possible states for all PSC LEDs are described below:
> PSC RUN/FAIL LED States
> The PSC RUN/FAIL LED indicates the overall status of the card. This LED should be green for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> PSC RUN/FAIL LED States

> Color | Description | Troubleshooting
> Green | Card powered with no errors detected | None needed.
> Blinking Green | Card is initializing and/or loading software | None needed.
> Red | Card powered with error(s) detected | Errors were detected during the POSTs. It is likely that the errors were logged to the system's command line interface during the boot process. Refer to one or more of the following to help analyze this problem:
> • | The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
> • | The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
> • | SNMP MIB Reference for information on associated status and alarm conditions.
> None | Card is not receiving power |
> • | Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU(s) section in this chapter for troubleshooting information.
> • | Verify that the power source is supplying ample voltage and current to the chassis.
> • | Verify that the card is properly installed per the instructions in the Hardware Installation and Administration Guide.
> • | If all of the above suggestions have been verified, it is possible that the PSC is not functional. Please contact your service representative.

> ‌
> PSC Active LED States
> The Active LED on the PSC indicates that the software is loaded on the card and that the card is ready for operation.When the system first boots up, all installed PSCs are booted into standby mode. The system must then be configured as to which PSCs should serve as redundant components (i.e. remain in standby mode) and which should function as active components.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> PSC Active LED States

> Color | Description | Troubleshooting
> Green | Card is active | The first time power is applied to the system, all of the PSCs should be booted into the standby mode. Therefore, this LED should be off.
> Blinking Green | Tasks or processes being migrated from an active PSC to a redundant/secondary PSC | Verify that the Standby LED on a redundant PSC is also blinking green. If so, there is an issue with the PSC that was active and is transferring its processes. Refer to the Monitoring the System chapter of this reference for information on determining the status of the PSC and system software processes and functionality.
> None | Card is not receiving powerORCard in Standby Mode |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, please refer to the PSC RUN/FAIL LED States section of this chapter for troubleshooting information.
> • | Check the state of the Standby LED. If it is green, the card is in standby mode. This is normal operation for the initial power-up. If needed, refer to the Configuring PAC/PSC and Line Card Availability section of the Configuring System Settings chapter in this reference for information on making the card active.

> ‌
> PSC Standby LED States
> The Standby LED on the PSC indicates that software is loaded on the card, but the card is serving as a redundant component. When the system first boots up, all installed PSCs are booted into standby mode. The system must then be configured as to which PSCs should serve as redundant components (i.e. remain in standby mode) and which should function as active components.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> PSC Standby LED States

> Color | Description | Troubleshooting
> Green | Card is in redundant mode | The first time power is applied to the system, all of the PSCs should be booted into the standby mode. Therefore, this is normal operation.
> Blinking Green | Tasks or processes being migrated from the active SMC to the redundant/secondary SMC | Verify that the Active LED on the redundant PSC is also blinking green. If so, there is an issue with the active PSC and the system is transferring its processes. Refer to one or more of the following to help analyze this problem:
> • | The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
> • | The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
> • | SNMP MIB Reference for information on associated status and alarm conditions.
> None | Card is not receiving powerORCard in Active Mode |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, please refer to the PSC RUN/FAIL LED States section of this chapter for information on troubleshooting.
> • | Check the state of the Active LED. If it is green, the card is in active mode. If needed, refer to the Manually Initiating a PAC/PSC Migration section in this chapter for information on configuring the card to serve as a redundant component.

> ‌
> Checking the LEDs on the SPIO(s)
> Each SPIO is equipped with status LEDs as listed below:

> • | RUN/FAIL

> • | Active

> • | Standby

> In addition to the LEDs listed above, each interface to the management network (both RJ-45 and SFP) are equipped with the following LEDs:

> • | Link

> • | Activity

> SPIO LED Locations
> The possible states for all of the SPIO's LEDs are described in the sections that follow.
> SPIO RUN/FAIL LED States
> The SPIO's RUN/FAIL LED indicates the overall status of the card. This LED should be green for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SPIO RUN/FAIL LED States

> Color | Description | Troubleshooting
> Green | Card powered with no errors detected | None needed.
> Red | Card powered with error(s) detected | Errors were detected during the POSTs. It is likely that the errors were logged to the system's command line interface during the boot process. Refer to the Monitoring the System chapter of this reference for information on determining the status of system hardware components.
> None | Card is not receiving power |
> • | Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU(s) section in this chapter for troubleshooting information.
> • | Verify that the power source is supplying ample voltage and current to the chassis.
> • | Verify that the card is properly installed per the instructions in the Hardware Installation and Administration Guide.
> • | If all of the above suggestions have been verified, it is possible that the SPIO is not functional. Please contact your service representative.

> ‌
> SPIO Active LED States
> The Active LED on the SPIO indicates that the software is loaded on the card and that the card is ready for operation. For the SPIO installed in chassis slot 24, this LED should be green for normal operation. For the SPIO installed in slot 25, this LED should be off for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SPIO Active LED States

> Color | Description | Troubleshooting
> Green | Card is active | None needed for SPIO in slot 24. If green for SPIO in slot 25, then verify that SPIO in slot 24 is installed properly according to the instructions in this document.
> None | Card is not receiving power OR Card in Standby Mode |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the SPIO RUN/FAIL LED States section of this chapter for troubleshooting information.
> • | Check the state of the Standby LED. If it is green, the card is in standby mode. This is normal for the SPIO in slot 25 since the chassis automatically places the card into standby mode at boot up.

> ‌
> SPIO Standby LED States
> The Standby LED on the SPIO indicates that software is loaded on the card, but it is serving as a redundant component. For the SPIO installed in slot 25, this LED should be green for normal operation. For the SPIO installed in slot 24, this LED should be off for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SPIO Standby LED States

> Color | Description | Troubleshooting
> Green | Card is in redundant mode | None needed for SPIO in slot 25. If green for SPIO in slot 24, check the status of the SPC/SMC installed in slot 8. If the SPC/SMC in slot 8 is in standby mode, it is possible that there is a problem. Refer to one or more of the following to help analyze this problem:
> • | The Monitoring Hardware Status chapter in this reference for show commands; the outputs of which will assist in further determining the problem.
> • | The Configuring and Viewing System Logs chapter in this reference for information on configuring specific types of logging information and how to view logs.
> • | SNMP MIB Reference for information on associated status and alarm conditions.
> None | Card is not receiving power OR Card in Active Mode |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the SPIO RUN/FAIL LED States section of this chapter for information on troubleshooting.
> • | Check the state of the Active LED. If it is green, the card is in active mode. This is normal for the SPIO in slot 24 since the chassis automatically make the card active at boot up.

> ‌
> SPIO Interface Link LED States
> The Link LED, associated with a particular SPIO interface indicates the status of the network link. This LED should be green for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SPIO Interface Link LED States

> Color | Description | Troubleshooting
> Green | Link is up | None needed. Important: This LED will not indicate the presence of a network link until the interface parameters are set during the software configuration process.
> None | No power to card OR Link is down |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power. If it is off, refer to the SPIO RUN/FAIL LED States section of this chapter for troubleshooting information.
> • | Verify that the interface is cabled properly.
> • | Verify that the device on which the interface is located is cabled and powered properly.

> ‌
> SPIO Interface Activity LED States
> The Activity LED associated with a particular SPIO interface indicates the presence of traffic on the network link. This LED should be green when data is being transmitted or received over the interface.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> SPIO Interface Activity LED States

> Color | Description | Troubleshooting
> Flashing Green | Traffic is present on the link | None needed.
> None | No traffic is present on the link | None needed if there is no activity on the link. Prior to configuration, this is normal operation.

> ‌
> Checking the LEDs on the Ethernet 10/100 and Ethernet 1000/Quad Gig-E Line Card(s) (QGLC)
> Each Ethernet 10/100, Ethernet 1000 Line Card and QGLC is equipped with status LEDs as listed below:

> • | RUN/FAIL

> • | Active

> • | Standby

> In addition to the LEDs listed above, each network interface is equipped with the following LEDs:

> • | Link

> • | Activity

> Ethernet 10/100 and GigE LEDs
> QGLC LEDs
> The possible states for all LEDs on the Ethernet 10/100 and Ethernet 1000/QGLC cards are as follows:
> Ethernet 10/100 and Ethernet 1000/QGLC RUN/FAIL LED States
> The RUN/FAIL LEDs on the Ethernet 10/100 and Ethernet 1000/QGLC Line Cards indicate the overall status of the cards. These LEDs should be green for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> Ethernet 10/100 and Ethernet 1000 RUN/FAIL LED States

> Color | Description | Troubleshooting
> Green | Card powered with no errors detected | None needed.
> Red | Card powered with error(s) detected | Errors were detected during the POSTs. It is likely that the errors were logged to the system's command line interface during the boot process. Refer to the Monitoring the System chapter of this reference for information on determining the status of system hardware components.
> None | Card is not receiving power |
> • | Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU(s) section in this chapter for troubleshooting information.
> • | Verify that the power source is supplying ample voltage and current to the chassis.
> • | Verify that the card is properly installed per the instructions in the Hardware Installation and Administration Guide.
> • | If all of the above suggestions have been verified, it is possible that the line card is not functional. Please contact your service representative.

> ‌
> Ethernet 10/100 and Ethernet 1000/QGLC Active LED States
> The Active LEDs on the Ethernet 10/100 and Ethernet 1000/QGLC Line Cards indicate that the operating software is loaded on the card and that the card is ready for operation.
> Important: Quad Gigabit Ethernet line cards (QGLC) only work in an ASR 5000 behind a PSC.
> The line cards installed will remain in a ready mode until their corresponding PAC/PSC is made active via configuration. While in ready mode the Active LED should be off. After the PAC/PSC is made active, the line card installed in the upper-rear chassis slot behind the PAC/PSC will also be made active. The line card installed in the lower-rear chassis slot behind the PAC/PSC will enter the standby mode.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> Ethernet 10/100 and Ethernet 1000/QGLC Active LED States

> Color | Description | Troubleshooting
> Green | Card is active | None needed for line cards installed in slots 17 through 23 and 26 through 32 after configuration. If green for line cards in slots 33 through 39 and 42 through 48, verify that the corresponding line card installed in the upper-rear chassis slot is installed properly according to the instructions in this document. For example, if this LED is green for a line card in slot 33, verify that the line card in slot 17 is installed properly.
> None | Card in Ready ModeORCard is not receiving power ORCard in Standby Mode |
> • | This is normal prior to configuration. Neither the Active or the Standby LED on the card is on.
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the Ethernet 10/100 and Ethernet 1000/QGLC RUN/FAIL LED States section of this chapter for troubleshooting information.
> • | Check the state of the Standby LED. If it is green, the card is in standby mode. This is normal operation for the initial power-up. If needed, refer to the Configuring PAC/PSC and Line Card Availability section of the Configuring System Settings chapter in this reference for information on making the card active.

> ‌
> Ethernet 10/100 and Ethernet 1000/QGLC Standby LED States
> The Standby LEDs on the Ethernet 10/100 and Ethernet 1000/QGLC Line Cards indicate that software is loaded on the cards, but are serving as redundant components.
> Important: Quad Gigabit Ethernet line cards (QGLC) only work in an ASR 5000 behind a PSC.
> The line cards installed will remain in a ready mode until their corresponding PAC/PSC is made active via configuration. While in ready mode, the Active LED should be off. After the PAC/PSC is made active, the line card installed in the upper-rear chassis slot behind the PAC/PSC will also be made active. The line card installed in the lower-rear chassis slot behind the PAC/PSC will also enter the standby mode.
> The possible states for this LED are described below. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> Ethernet 10/100 and Ethernet 1000/QGLC Standby LED States

> Color | Description | Troubleshooting
> Green | Card is in redundant mode | None needed for line cards installed in slots 33 through 39 and 42 through 48 after configuration. If green for line cards installed in slots 17 through 23 and 26 through 32, refer to the Monitoring the System chapter of this reference for information on determining the status of the line card and system software processes and functionality.
> None | Card in Ready ModeORCard is not receiving power OR Card in Active Mode |
> • | This is normal prior to configuration. Neither the Active nor Standby LEDs on the card is on.
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the Ethernet 10/100 and Ethernet 1000/QGLC RUN/FAIL LED States section of this chapter for troubleshooting information.
> • | Check the state of the Active LED. If it is green, the card is in standby mode. If needed, refer to the Manually Initiating a Line Card Switch section in this chapter for information on configuring the card to serve as a redundant component.

> ‌
> Ethernet 10/100 and Ethernet 1000/QGLC Interface Link LED States
> The Link LEDs, associated with a particular network interface on the Ethernet 10/100 and Ethernet 1000/QGLC Line Cards, indicate the status of the network link. These LEDs should be green for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> Ethernet 10/100 and Ethernet 1000 Interface Link LED States

> Color | Description | Troubleshooting
> Green | Link is up | None needed. Important: This LED will not indicate the presence of a network link until the interface parameters are set during the software configuration process.
> None | No power to card ORLink is down |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power. If it is off, refer to Ethernet 10/100 and Ethernet 1000/QGLC RUN/FAIL LED States section for troubleshooting information.
> • | Verify that the interface is cabled properly.
> • | Verify that the device where the interface is connected to is cabled and powered properly.
> • | Check the cable for continuity.

> ‌
> Ethernet 10/100 and Ethernet 1000/QGLC Interface Activity LED States
> The Activity LEDs, associated with a particular network interface on the Ethernet 10/100 and Ethernet 1000/QGLC Line Cards, indicate the presence of traffic on the network link. These LEDs should be green when data is being transmitted or received over the interface.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> Ethernet 10/100 and Ethernet 1000 Interface Activity LED States

> Color | Description | Troubleshooting
> Flashing Green | Traffic is present on the link | None needed.
> None | No traffic is present on the link | None needed if there is no activity on the link. Prior to configuration, this is normal operation.

> ‌
> Checking the LEDs on the RCC(s)
> Each RCC is equipped with status LEDs as listed below:

> • | RUN/FAIL

> • | Active

> • | Standby

> RCC LED Locations
> The possible states for all of the SPIO's LEDs are described in the sections that follow.
> RCC RUN/FAIL LED States
> The RCC's RUN/FAIL LED indicates the overall status of the card. This LED should be green for normal operation.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> RCC RUN/FAIL LED States

> Color | Description | Troubleshooting
> Green | Card powered with no errors detected | None needed.
> Red | Card powered with error(s) detected | Errors were detected during the POSTs. It is likely that the errors were logged to the system's command line interface during the boot process. Refer to the Monitoring the System chapter of this reference for information on determining the status of system hardware components.
> None | Card powered with error(s) detected |
> • | Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU(s) section in this chapter for troubleshooting information.
> • | Verify that the power source is supplying ample voltage and current to the chassis.
> • | Verify that the card is properly installed per the instructions in the Hardware Installation and Administration Guide.
> • | If all of the above suggestions have been verified, it is possible that the SPIO is not functional. Please contact your service representative.

> ‌
> RCC Active LED States
> The Active LED on the RCC indicates that the card is being used. For normal operation, this LED should be off on both RCCs.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> RCC Active LED States

> Color | Description | Troubleshooting
> Green | Card is active | The RCC is actively routing traffic from a line card installed behind a PAC/PSC that has failed to a redundant PAC/PSC. The RCC installed in chassis slot 40 processes traffic for the line cards in chassis slots 17 through 23 and 26 through 32. The RCC installed in chassis slot 41 processes traffic for the line cards in slots 33 through 39 and 42 through 48. Refer to either the Checking the LEDs on the PAC(s) or Checking the LEDs on the PSC(s) section of this chapter to determine which PAC/PSC has failed.
> None | Card is not receiving power OR Card in Standby Mode |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the RCC RUN/FAIL LED States section of this chapter for troubleshooting information.
> • | Check the state of the Standby LED. If it is green, the card is in standby mode. This is the normal operating mode.

> ‌
> RCC Standby LED States
> The Standby LED on the RCC indicates that software is loaded on the card and is ready to provide a path for data or signalling traffic from a line card to a redundant PAC/PSC. This LED should be on for normal operation for both RCCs installed.
> The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
> RCC Standby LED States

> Color | Description | Troubleshooting
> Green | Card is in standby mode | This is the normal operating mode.
> None | Card is not receiving power ORCard in Active Mode |
> • | Verify that the RUN/FAIL LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the RCC RUN/FAIL LED States section of this chapter for troubleshooting information.
> • | Check the state of the Active LED. If it is green, the card is in active mode and the RCC is actively routing traffic from a line card installed behind a PAC/PSC that has failed.
> • | Refer to either the Checking the LEDs on the PAC(s) or Checking the LEDs on the PSC(s) section of this chapter to determine which PAC/PSC has failed. Information on determining the cause of the failure can be found in the Monitoring the System chapter of this reference.

> ‌
> Testing System Alarm Outputs
> The system provides the following two physical alarm mechanisms:

> • | System Audible Alarm: Located on the SPC/SMC, the speaker is used to provide an audible indicator that a minor, major, or critical alarm has occurred.

> • | CO Alarms Interface: Located on the SPIO, this interface provides a 10-pin connector that enables three normally-closed dry-contact relays for the triggering of external audio and/or visual indicators. These indicators can be used to alert that either a minor, major, or critical alarm has occurred.

> The operation of these alarms can be tested by issuing the following command:
> test alarm { audible | central-office [ critical | major | minor ] }

> Keyword/Variable | Description
> audible | Tests the CO Alarm Speaker on the SPC/SMC to verify operation.
> central-office | Tests the CO Alarm Interface on the SPIO to verify operation. Individual alarms can be tested by using one of the following keywords:
> • | critical: Specifies that the critical CO Alarms output is to be tested.
> • | major: Specifies that the major CO Alarms output is to be tested.
> • | minor: Specifies that the minor CO Alarms output is to be tested.
> If no keyword is specified, all alarms are tested.

> ‌
> When this command is executed, the specified alarm is activated for a period of 10 seconds. After this time, the alarm will return to its previous condition.
> Taking Corrective Action
> In the event that an issue was discovered with an installed application or line card, depending on the severity, it may be necessary to take corrective action.
> The system provides several redundancy and fail-over mechanisms to address issues with application and line cards in order to minimize system downtime and data loss. These mechanisms are described in the sections that follow.
> Manually Initiating a Management Card Switchover
> When the system boots up, the SPC/SMC installed in chassis slot eight will boot into the “active” mode and begin booting other system components. The SPC/SMC installed in chassis slot nine will automatically be booted into “standby” mode dictating that it will serve as a redundant component. However, the active SPC/SMC will frequently synchronize currently running tasks or processes with the redundant SPC/SMC.
> In the event of a critical failure on the SPC/SMC in slot eight, system control will be switched to the redundant SPC/SMC in slot nine. This is a relatively seamless transition because the two are synchronized. The formerly active SPC/SMC will then enter the standby mode allowing it to be safely replaced or restored.
> In the event that an issue arises that is not severe enough for the system to perform an automatic switchover, a manual switch over can be invoked by executing the following instructions from the Exec mode prompt:
> [local]host_name#

> Step 1 | Initiate a manual SPC/SMC switch over by entering one of the following commands:

> For SPC:
> card spc switchover
> For SMC:
> card smc switchover
> You will receive the following prompt:
> Are You Sure? [Yes|No]:

> Step 2 | Press Y to start the switchover.

> Step 3 | Verify that the switchover was successful by entering the following command at the prompt:

> show card table
> Check the entry in the Oper State column next to the SPC/SMC just switched. Its state should be Standby.
> Manually Initiating a Packet Processing Card Migration
> When the system boots up, all installed PACs/PSCs enter the “standby” mode. The standby mode indicates that the card is available for use but is not configured for operation. Installed components can be made active through the software configuration process. Cards that are not configured to enter the “active” mode will remain in standby mode for use as redundant components.
> In the event of PAC/PSC critical failure, tasks will be automatically be migrated from the active card to a redundant card in standby mode. The line card installed behind the PAC/PSC that was formerly active will still be used to maintain the interfaces to external network equipment. Installed Redundancy Crossbar Cards (RCCs) will provide a path for signalling and data traffic between the line card and the now active PAC/PSC. Therefore, redundant PACs/PSCs do not require that line cards be installed behind them.
> In the event that an issue arises that is not severe enough for the system to perform an automatic migration, a manual migration can be invoked. Follow the instructions below to manually initiate a PAC/PSC migration. These instructions assume you are at the root prompt for the Exec mode:
> [local]host_name#

> Step 1 | Initiate a manual PAC/PSC migration by entering one of the following commands:

> For PAC:
> card pac migration from <original_slot#> to <final_slot#>
> For PSC:
> card psc migration from <original_slot#> to <final_slot#>

> Keyword/Variable | Description
> from | Specifies the chassis slot number of the PAC/PSC that is being migrated from. original_slot can be any value between 1 through 7, and 10 through 16.
> to | Specifies the chassis slot number of the PAC/PSC that is being migrated to. Variable final_slot can be any value between 1 through 7, and 10 through 16.

> ‌
> You will receive the following prompt:
> Are You Sure? [Yes|No]:

> Step 2 | Press Y to start the migration.

> Step 3 | Verify that the migration was successful by entering the following command at the prompt:

> show card table
> Check the entry in the Oper State column next to the PAC/PSC that was just migrated from. Its state should be Standby. The state of the PAC/PSC migrated to should be Active.
> Manually Initiating a Line Card Switch
> Line cards are installed in the half-height slots at the rear of the chassis. This design allows for two line cards (both Ethernet 10/100 both Ethernet 1000 (or both QGLCs on an ASR 5000)) to be installed behind every application card. When two line cards are installed, as the application card that they are installed behind is booted, the card in the upper-rear chassis slot will automatically be made active while the card in the lower-rear chassis slot will automatically be placed in standby mode. In the event that the active card experiences a failure, the system will automatically switch traffic to the standby card in the lower slot.
> In the event that a SPIO experiences a failure, the system will automatically switch traffic to the redundant SPIO installed behind the redundant SPC/SMC.
> In the event that an issue arises that is not severe enough for the system to perform an automatic switch, a manual switch can be performed. Follow the instructions below to manually initiate a line card or SPIO switch. These instructions assume you are at the root prompt for the Exec mode:
> [local]host_name#

> Step 1 | Initiate a manual line card or SPIO migration by entering the following command:

> card [ lc | spio ] switch [ to <slot#> ]

> Keyword/Variable | Description
> lc | Specifies that a switch will be done for an Ethernet 10/100 or Ethernet 1000 Line Card (Quad Gig-E (QGLC) on an ASR 5000 only).
> spio | Specifies that a switch will be done for a SPIO.
> to | Specifies the chassis slot number of the redundant Ethernet 10/100 or Ethernet 1000/QGLC Line Card that is being migrated to. Executing this command will switch network connections from the active line card that corresponds to the card being migrated to. slot# can be any of the following integer values: 17 through 23, 26 through 39, or 42 through 48. Important: This keyword is only used if the lc keyword is used. The line card being migrated to must be in the standby mode.

> ‌
> You will receive the following prompt:
> Are You Sure? [Yes|No]:

> Step 2 | Press Y to start the switch.

> Step 3 | Verify that the migration was successful by entering the following command at the prompt:

> show card table
> Check the entry in the Oper State column next to the line card or SPIO that was just switched from. Its state should be Standby. The state of the line card or SPIO switched to should be Active.
> Halting Cards
> PACs/PSCs or line cards that are in either the active or standby modes can be halted. Halting these cards places them into the “offline” mode. When in this mode, the card will become unusable for session processing as either an active or redundant component.
> If a card in the active mode is halted, its tasks, processes, or network connections will be migrated or switched to a redundant component prior to entering the offline mode.
> This section provides information and instructions for initiating a card halt and restoring halted components.
> Initiate a Card Halt
> Follow the instructions below to manually initiate a card halt. These instructions assume you are at the root prompt for the Exec mode:
> [local]host_name#

> Step 1 | Initiate a manual line card or SPIO migration by entering the following command:

> card halt <slot#>
> slot# is the chassis slot number in which the card to be halted is installed. It can be any integer value between 1 and 7, 10 through 48.
> You will receive the following prompt:
> Are You Sure? [Yes|No]:

> Step 2 | Press Y to start the halt of the card.

> Step 3 | Verify that the migration was successful by entering the following command at the prompt:

> show card table
> Check the entry in the Oper State column next to the line card that was just halted. Its state should be Offline. If the card was in active mode prior to the execution of this command, the state of the redundant component associated with it should now be Active.
> Restoring a Previously Halted Card
> Follow the instructions below to restore a card that was previously halted. These instructions assume you are at the root prompt for the Exec mode:
> [local]host_name#

> Step 1 | Reboot the card to be restored by entering the following command:

> card reboot <slot#> -force
> You will receive the following prompt:
> Are You Sure? [Yes|No]:

> Step 2 | Press Y to start the reboot of the card.

> Step 3 | Verify that the migration was successful by entering the following command at the prompt:

> show card table
> Check the entry in the Oper State column next to the line card that was just restored. Its state should be the state of that it was in before it was halted.
> Verifying Network Connectivity
> There are multiple commands supported by the system to verify and/or troubleshoot network connectivity. Note that network connectivity can only be tested once system interfaces and ports have been configured and bound.
> The commands specified in this section should be issued on a context-by-context basis. This is due to the fact that contexts act like virtual private networks (VPNs) in that they operate independently of the others. Therefore, ports, interfaces, and routes configured in one context can not be tested from another without additional configuration.
> To switch between contexts you must enter the following command at the root prompt for the Exec mode:
> context <context_name>
> context_name is the name of the context that you wish to switch to. The following prompt appears:
> [context_name]host_name#
> Using the Ping Command
> Using the ping command verifies the system’s ability to communicate with a remote node in the network by passing data packets between and measuring the response. This command is useful in verifying network routing and if a remote node is able to respond at the IP layer. The command has the following syntax:
> ping <host_ip_address> [ count <num_packets> ] [ pattern <packet_pattern> ] [ size <octet_count> ] [ src { <src_host_name> | <src_host_ip_address> } ]

> Keyword/Variable | Description
> <host_ip_address> | Identifies the remote node to which is the target of the ping command. host_ip_address specifies the remote node using the node’s assigned IP address specified using the standard IPv4.
> count <num_packets> | Specifies the number of packets to send to the remote host for verification. num_packets must be within the range 1 through 10000. The default is 5.
> pattern <packet_pattern> | Specifies a pattern to use to fill the internet control message protocol packets with. packet_pattern must be specified in hexadecimal format with a value in the range hexadecimal 0x0000 through 0xFFFF. packet_pattern must begin with a ‘0x’ followed by up to 4 hexadecimal digits. The default is that each octet of the packet is encoded with the octet number of the packet.
> size <octet_count> | Specifies the number of bytes each IP datagram. octet_count must be a value in the range 40 through 18432. The default is 56.
> src { <src_host_name> | <src_host_ip_address> } | Specifies an IP address to use in the packets as the source node. src_host_name: specifies the source node using the node’s logical host name which must be resolved via DNS lookup. src_host_ip_address: specifies the source node using the node’s assigned IP address specified using the standard IPv4. The default is the IP address of the interface through which the ping was issued.

> ‌
> The following displays a sample of a successful response.
> PING 192.168.250.1 (192.168.250.1): 56 data bytes
> 64 bytes from 192.168.250.1: icmp_seq=0 ttl=255 time=0.4 ms
> 64 bytes from 192.168.250.1: icmp_seq=1 ttl=255 time=0.2 ms
> 64 bytes from 192.168.250.1: icmp_seq=2 ttl=255 time=0.2 ms
> 64 bytes from 192.168.250.1: icmp_seq=3 ttl=255 time=0.2 ms
> 64 bytes from 192.168.250.1: icmp_seq=4 ttl=255 time=0.2 ms
> --- 192.168.250.1 ping statistics ---
> 5 packets transmitted, 5 packets received, 0% packet loss
> round-trip min/avg/max = 0.2/0.2/0.4 ms
> If no response is received from the target follow these troubleshooting procedures:

> • | Verify that the correct IP address was entered.

> • | Attempt to ping a different device on the same network. If the ping was successful then it is likely that your system configuration is correct. Verify that the device you are attempting to ping is powered and functioning properly.

> • | Verify the port is operational.

> • | Verify that your configuration of the ports and interfaces within the context are correct.

> • | If your configuration is correct and you have access to the device that you’re attempting to ping, ping the system from that device.

> • | If there is still no response, it is likely that the packets are getting discarded by a network device. Use the traceroute and show ip static-route commands discussed in this chapter to further troubleshoot the issue.

> Using the Traceroute Command
> The traceroute command collects information on the route data will take to a specified host. This is a useful troubleshooting command that can be used to identify the source of significant packet delays or packet loss on the network. This command can also be used to identify bottle necks in the routing of data over the network.
> The command has the following syntax:
> traceroute { <host_name> | <host_ip_address> } [ count <packets> ] [ df ] [ maxttl <max_ttl> ] [ minttl <min_ttl> ] [ port <port_number> ] [ size <octet_count> ] [ src { <src_host_name> | <src_host_ip_address> } ] [ timeout <seconds> ]

> Keyword/Variable | Description
> <host_name> | Identifies the remote node to trace the route to by the hostname. host_name specifies the remote node using the node’s logical host name which must be resolved via DNS lookup.
> <host_ip_address> | Identifies the remote node to trace the route to by the IP address. host_ip_address specifies the remote node using the node’s assigned IP address specified using the standard IPv4.
> count | Specifies the number of UDP probe packets to send. The default is 3.
> df | Indicates the packets for the tracing of the route should not be fragmented. If a packet would require fragmenting then it is dropped and the ICMP response “Unreachable, Needs Fragmentation” is received.
> maxttl <max_ttl> | Specifies the maximum time to live (TTL), in seconds, for the route tracing packets. max_ttl must be specified as a value in the range of 1 through 255. It is an error if max_ttl is less than min_ttl, whether min_ttl is specified or defaulted. The time to live is the number of hops through the network, i.e., it is not a measure of time. The default maximum TTL is 30 seconds.
> minttl <min_ttl> | Specifies the minimum time to live, in seconds, for the route tracing packets. min_ttl must be specified as a value in the range of 1 through 255. It is an error if min_ttl is greater than max_ttl, whether max_ttl is specified or defaulted. The time to live is the number of hops through the network, i.e., it is not a measure of time. The default minimum TTL is 1 second.
> port <port_number> | Specifies a specific port to connect to where port_number must be a value in the range 1 through 65535. The default port is 33434.
> size | Specifies the number of bytes each packet. octet_count must be a value in the range 40 through 32768. The default is 40.
> src { <src_host_name> | <src_host_ip_address> } | Specifies an IP address to use in the packets as the source node. src_host_name: specifies the remote node using the node’s logical host name which must be resolved via DNS lookup. src_host_ip_address: specifies the remote node using the node’s assigned IP address specified using the standard IPv4. The default is the IP address of the interface through which the ping was issued.
> timeout <seconds> | Specifies the maximum time to wait for a response from each route tracing packet. seconds must be a value in the range 2 through 100. The default is 5.

> ‌
> The following displays a sample output.
> traceroute to 192.168.250.1 (192.168.250.1), 30 hops max, 40 byte packets
> 1 192.168.250.1 (192.168.250.1) 0.446 ms 0.235 ms 0.178 ms
> Viewing IP Routes
> The system provides a mechanism for viewing route information to a specific node or for an entire context. This information can be used to verify network connectivity and to ensure the efficiency of the network connection. The command has the following syntax:
> show ip route [ <route_ip_address> [ <route_gw_address> ] ]

> Keyword/Variable | Description
> <route_ip_address> | Specifies the IP address of a network node for which route information is displayed.
> <route_gw_address> | Specifies the IP address of the gateway router between the system and the network node for which route information is displayed. This is an optional keyword.

> ‌
> If no keywords are specified, all IP routes within the context’s routing table are displayed.
> The following displays a sample of this command’s output showing a context routing table.
> "*" indicates the Best or Used route.
> Destination Nexthop Protocol Prec Cost Interface
> *0.0.0.0/0 10.0.4.1 static 0 0 SPIO1
> *10.0.4.0/24 0.0.0.0 kernel 0 0 SPIO1
> *10.0.4.0/32 0.0.0.0 kernel 0 0 SPIO1
> *10.0.4.3/32 0.0.0.0 kernel 0 0 SPIO1
> *10.0.4.255/32 0.0.0.0 kernel 0 0 SPIO1
> Viewing the Address Resolution Protocol Table
> The system provides a mechanism for viewing Address Resolution Protocol (ARP) table information to a specific node or for an entire context. This information can be used to verify that when the system sends an ARP packet, it receives valid responses from other network nodes. The command has the following syntax:
> show ip arp [ <arp_ip_address> ]
> arp_ip_address specifies a specific network node for which to display ARP information. If this keyword is not specified, all entries within the context’s ARP table are displayed.
> Important: When the VPN Manager restarts, it removes all interfaces from the kernel and thus the kernel removes all ARP entries. When this happens, the NPU still holds all of the ARP entries so that there is no traffic disruption. When this happens, from a user point of view, show ip arp is broken since this command gathers information from the kernel and not the NPU.
> The following displays a sample of this command’s output showing a context’s ARP table.
> Flags codes:
> C - Completed, M - Permanent, P - Published, ! - Not answered
> T - has requested trailers
> Address Link Type Link Address Flags Mask Interface
> 10.0.4.240 ether 00:05:47:02:20:20 C SPIO1
> 10.0.4.7 ether 00:05:47:02:03:36 C SPIO1
> 10.0.4.1 ether 00:01:30:F2:7F:00 C SPIO1
> Using the System Diagnostic Utilities
> The system provides protocol monitor and test utilities that can are useful when troubleshooting or verifying configurations. The information generated by these utilities can in many cases either identify the root cause of a software or network configuration issue or, at the very least, greatly reduce the number of possibilities.
> This section contains information and instructions for using these utilities.
> Using the Monitor Utility
> For troubleshooting purposes, the system provides a powerful protocol monitoring utility. This tool can be used to display protocol information for a particular subscriber session or for every session being processed.
> Caution: The monitor tool is intrusive in that it may cause session processing delays and/or data loss. Therefore, it should be used only when troubleshooting .
> Using the Protocol Monitor
> The system’s protocol monitor displays information for every session that is currently being being processed. Depending on the number of protocols monitored, and the number of sessions in progress, a significant amount of data is generated. It is highly recommended that logging be enabled on your terminal client in order to capture all of the information that is generated.
> Follow the instructions in this section to invoke and configure the protocol monitoring tool.

> Step 1 | Invoke the protocol monitor by entering the following command:

> monitor protocol
> The following output is displayed.
> MONITOR GLOBAL PROTOCOLS:
> 11 - SNMP 21 - L2TP (Admin only)
> 12 - RADIUS Authentication (Admin only) 22 - L2TPMGR (Admin only)
> 13 - RADIUS Accounting (Admin only) 23 - L2TP Data(Admin only)
> 14 - A11 (R-P Interface) (Admin only) 24 - GTP (Admin only)
> 15 - Mobile IPv4 (Admin only) 25 - GTPCMGR (Admin only)
> 16 - A11MGR (Admin only) 26 - GTPU (Admin only)
> 17 - PPP (Admin only) 27 - GTPP (Admin only)
> 18 - A10 (Admin only) 28 - DHCP (Admin only)
> 19 - User L3 (Admin only) 29 - CDR (Admin only)
> 31 - RADIUS COA (Admin only) 30 - DHCPV6 (Admin only)
> 51 - SCTP (Admin only)
> 32 - MIP Tunnel (Admin only) 52 - M3UA (Admin only)
> 33 - L3 Tunnel (Admin only) 53 - SCCP (Admin only)
> 34 - CSS Data (Admin only) 54 - TCAP (Admin only)
> 35 - CSS Signaling (Admin only) 55 - MAP (Admin only)
> 36 - EC Diameter (Admin only) 56 - RANAP (Admin only)
> 37 - SIP (IMS) (Admin only) 57 - GMM (Admin only)
> 38 - IPSec IKE Only (Admin only) 58 - GPRS-NS (Admin only)
> 59 - BSSGP (Admin only)
> 40 - IPSec IKEv2 Only (Admin only)
> 41 - IPSG RADIUS Signal (Admin only) 61 - SSCOP (Admin only)
> 42 - ROHC (Admin only) 62 - SSCFNNI (Admin only)
> 43 - WiMAX R6 (Admin only) 63 - MTP3 (Admin only)
> 44 - WiMAX Data (Admin only) 64 - LLC (Admin only)
> 45 - SRP (Admin only) 65 - SNDCP (Admin only)
> 46 - BCMCS SERV AUTH (Admin only) 66 - BSSAP+ (Admin only)
> 47 - RSVP (Admin only) 67 - SMS (Admin only)
> 48 - Mobile IPv6 (Admin only) 68 - MTP2 (Admin only)
> 49 - ASNGWMGR (Admin only)
> 50 - STUN (Admin only)
> 70 - DNS Client (Admin only)
> 75 - App Specific Diameter (Admin only)
> 81 - S1AP (Admin only) 82 - NAS (Admin only)
> (B)egin Protocol Decoding, (Q)uit, <ESC> Prev Menu
> Select:

> Step 2 | Choose the protocol that you wish to monitor by entering the number associated with the protocol (11 through 19 and 21 through 28) at the Select: prompt. A greater-than sign ( > ) appears next to the protocol you selected.

> Step 3 | Repeat step 2 as needed to choose multiple protocols.

> Step 4 | Press B to begin the protocol monitor. If you selected any protocol other than 11 (SNMP), the following message is displayed:

> WARNING!!! You have selected options that can DISRUPT USER SERVICE
> Existing CALLS MAY BE DROPPED and/or new CALLS MAY FAIL!!!
> (Under heavy call load, some debugging output may not be displayed)
> Proceed? - Select (Y)es or (N)o

> Step 5 | Enter Y to proceed with the monitor or N to go back to the previous menu.

> C - Control Events (ON )
> D - Data Events (ON )
> E - EventID Info (ON )
> I - Inbound Events (ON )
> O - Outbound Events (ON )
> S - Sender Info (OFF)
> T - Timestamps (ON )
> X - PDU Hexdump (OFF)
> A - PDU Hex/Ascii (OFF)
> +/- Verbosity Level ( 1)
> L - Limit Context (OFF)
> M - Match Newcalls (ON )
> R - RADIUS Dict (no-override)
> G - GTPP Dict (no-override)
> Q)uit, <ENTER> Display Options, <ESC> Prev Menu, <SPACE> Pause

> Step 6 | Configure the amount of information that is displayed by the monitor. To enable or disable options, enter the letter associated with that option (C, D, E, etc.). To increase or decrease the verbosity, use the plus ( + ) or minus ( - ) keys.

> The current state, ON (enabled) or OFF (disabled), is shown to the right of each option.

> Step 7 | Press the Enter key to refresh the screen and begin monitoring.

> The monitor remains active until disabled. To quit the protocol monitor and return to the prompt, press q.
> Using the Protocol Monitor for a Specific Subscriber
> The system’s protocol monitor can be used to display information for a specific subscriber session that is currently being processed. Depending on the number of protocols monitored, and the number of sessions in progress, a significant amount of data is generated. It is highly recommended that logging be enabled on your terminal client in order to capture all of the information that is generated.
> Follow the instructions in this section to invoke and configure the protocol monitoring tool for a specific subscriber session.

> Step 1 | To invoke the session-specific protocol monitor enter the following command:

> monitor subscriber
> The following screen is displayed, followed by a list of all available monitoring methods:
> MONITOR SUBSCRIBER:
> 1) By MSID/IMSI
> 2) By Username
> 3) By Callid
> 4) By IP Address
> 5) By IPv6 Address
> 6) Next-Call
> 7) Next-BCMCS-Call
> 8) Next-BCMCS-Service-Request
> 9) By IMEI
> a) By MSISDN
> b) Next-1xRTT Call
> c) Next-ASNGW Call
> d) Next-CLOSEDRP Call
> e) Next-EVDO-Rev0 Call
> f) Next-EVDO-RevA Call
> g) Next-GGSN Call
> h) Next-HA Call
> i) Next-IPSG Call
> j) Next-LNS Call
> k) Next-PDIF Call
> l) By ASN Peer IP Address
> m) By PDIF Peer IP Address
> n) By Peer LAC IP Address
> o) By SGSN IP Address
> p) By PCF IP Address
> r) By Peer FA IP Address
> s) By IPSG Peer IP Address
> t) Next-ASNPC Call
> u) Next-OpenRP Call
> v) Next-Call By APN
> Q) Quit
> <ESC> Return to Previous Menu

> Step 2 | Specify the method the monitor should use to select the user to monitor by entering the number that corresponds with the desired selection in the menu.

> Step 3 | Enter the appropriate information for the menu item selected.

> If no session matching the specified criteria was being processed when the monitor was invoked, the following output is displayed:
> NO MATCHING CALL - waiting for a matching call to connect...
> C - Control Events (ON ) 11 - PPP (ON ) 21 - L2TP (ON )
> D - Data Events (ON ) 12 - A11 (ON ) 22 - L2TPMGR (OFF)
> E - EventID Info (ON ) 13 - RADIUS Auth (ON ) 23 - L2TP Data (OFF)
> I - Inbound Events (ON ) 14 - RADIUS Acct (ON ) 24 - GTPC (ON )
> O - Outbound Events (ON ) 15 - Mobile IPv4 (ON ) 25 - GTPCMGR (OFF)
> S - Sender Info (OFF) 16 - A11MGR (OFF) 26 - GTPU (OFF)
> T - Timestamps (ON ) 17 - SESSMGR (ON ) 27 - GTPP (ON )
> X - PDU Hexdump (OFF) 18 - A10 (OFF) 28 - DHCP (ON )
> A - PDU Hex/Ascii (OFF) 19 - User L3 (OFF) 29 - CDR (ON )
> +/- Verbosity Level ( 1) 31 - Radius COA (ON ) 30 - DHCPV6 (ON )
> L - Limit Context (OFF) 32 - MIP Tunnel (ON ) 53 - SCCP (OFF)
> M - Match Newcalls (ON ) 33 - L3 Tunnel (OFF) 54 - TCAP (OFF)
> R - RADIUS Dict: (no-override) 34 - CSS Data (OFF) 55 - MAP (ON )
> G - GTPP Dict: (no-override) 35 - CSS Signal (OFF) 56 - RANAP (OFF)
> Y - Multi-Call Trace (OFF) 36 - EC Diameter (ON ) 57 - GMM (ON )
> 37 - SIP (IMS) (OFF) 58 - GPRS-NS (OFF)
> 40 - IPSec IKEv2 (OFF) 59 - BSSGP (OFF)
> 41 - IPSG RADIUS (ON ) 64 - LLC (OFF)
> 42 - ROHC (OFF) 65 - SNDCP (OFF)
> 43 - WiMAX R6 (ON ) 66 - BSSAP+ (OFF)
> 44 - WiMAX Data (OFF) 67 - SMS (OFF)
> 45 - SRP (OFF)
> 46 - BCMCS SERV AUTH (OFF)
> 47 - RSVP (ON )
> 48 - Mobile IPv6 (ON )
> 49 - ASNGWMGR (OFF)
> 50 - STUN (IMS) (OFF)
> 75 - App Specific Diameter OFF
> (Q)uit, <ESC> Prev Menu, <SPACE> Pause, <ENTER> Re-Display Options

> Step 4 | Configure the amount of information that is displayed by the monitor. To enable or disable options, enter the letter associated with that option (C, D, E, etc.). To increase or decrease the verbosity, use the plus ( + ) or minus ( - ) keys.

> The current state, ON (enabled) or OFF (disabled), is shown to the right of each option.
> Important: Option Y for performing multi-call traces is only supported for use with the GGSN.

> Step 5 | Repeat step 6 as needed to enable or disable multiple protocols.

> Step 6 | Press the Enter key to refresh the screen and begin monitoring.

> The following displays a portion of a sample of the monitor’s output for a subscriber named user2@aaa. The default protocols were monitored.
> ---------------------------------------------------------------------------
> Incoming Call:
> ---------------------------------------------------------------------------
> MSID: 0000012345 Callid: 002dc6c2
> Username: user2@aaa SessionType: unknown
> Status: Active Service Name: xxx1
> Src Context: source Dest Context:
> ---------------------------------------------------------------------------
> <<<<OUTBOUND 10:02:35:415 Eventid:25001(0)
> PPP Tx PDU (9)
> PAP 9: Auth-Ack(1), Msg=
> <<<<OUTBOUND 10:02:35:416 Eventid:25001(0)
> PPP Tx PDU (14)
> IPCP 14: Conf-Req(1), IP-Addr=192.168.250.70
> <<<<OUTBOUND 10:02:35:416 Eventid:25001(0)
> PPP Tx PDU (27)
> CCP 27: Conf-Req(1), MPPC, Stac-LZS, Deflate, MVRCA
> INBOUND>>>>> 10:02:35:517 Eventid:25000(0)
> PPP Rx PDU (30)
> IPCP 30: Conf-Req(1), IP-Comp VJ-Comp, IP-Addr=0.0.0.0, Pri-DNS=0.0.0.0,
> Sec-DNS=0.0.0.0
> <<<<OUTBOUND 10:02:35:517 Eventid:25001(0)
> PPP Tx PDU (26)
> IPCP 26: Conf-Rej(1), IP-Comp VJ-Comp, Pri-DNS=0.0.0.0, Sec-DNS=0.0.0.0
> INBOUND>>>>> 10:02:35:517 Eventid:25000(0)
> PPP Rx PDU (12)
> IPCP 12: Conf-Ack(1), IP-Addr=192.168.250.70
> INBOUND>>>>> 10:02:35:518 Eventid:25000(0)
> PPP Rx PDU (31)
> LCP 31: Prot-Rej(1), Rejected-Protocol=CCP (0x80fd)
> INBOUND>>>>> 10:02:35:518 Eventid:25000(0)
> PPP Rx PDU (12)
> IPCP 12: Conf-Req(2), IP-Addr=0.0.0.0
> <<<<OUTBOUND 10:02:35:518 Eventid:25001(0)
> PPP Tx PDU (14)
> IPCP 14: Conf-Nak(2), IP-Addr=192.168.250.87
> INBOUND>>>>> 10:02:35:519 Eventid:25000(0)
> PPP Rx PDU (12)
> IPCP 12: Conf-Req(3), IP-Addr=192.168.250.87
> The monitor remains active until disabled. To quit the protocol monitor and return to the prompt, press q.
> Using the DHCP Testing Tool
> The CLI provides a mechanism for testing network connectivity with and configuration of DHCP servers. This functionality can be extremely useful in determining the accuracy of the system’s DHCP configuration and troubleshooting the server’s response time.
> This tool provides a mechanism from getting an IP address for one or more DHCP servers that the system is configured to communicate with. In addition, the tool provides a mechanism for releasing the address back to the DHCP server.
> Important: This tool must be executed from the context in which the DHCP server(s) are configured.
> To execute the DHCP test tool enter the following command:
> dhcp test dhcp-service { <service_name> } [ all | <server <ip_addr>|
> Sample dhcp test dhcp-service Command Output

> Keyword | Description
> <service_name> | Name of DHCP service from which to lease an IP address.
> all | Test all DHCP servers in this DHCP service.
> server | Followed by IP address of server under test.

> ‌

Cisco Systems Inc.
Tel: 408-526-4000
Fax: 408-527-0883