Cisco 7600 Series Ethernet Services Plus (ES+) and Ethernet Services Plus T (ES+T) Line Card Configuration Guide
Troubleshooting the Cisco 7600 Series Ethernet Service + Line Card
Downloads: This chapterpdf (PDF - 238.0KB) | Feedback

Table of Contents

Troubleshooting

General Troubleshooting Information

Interpreting Console Error Messages

Using debug Commands

Using show Commands

Using the Cisco IOS Event Tracer to Troubleshoot Problems

Troubleshooting SFP/XFP Issues

Preparing for Online Insertion and Removal of Cisco 7600 Series ES+ Line Card

Preparing for Online Removal of a Cisco 7600 Series ES+ Line Card

Deactivating a Cisco 7600 Series ES+ Line Card

Reactivating a Cisco 7600 Series ES+ Line Card

Verifying Deactivation and Activation of a Cisco 7600 Series ES+ Line Card

Deactivation and Activation Configuration Examples

Deactivation of a Cisco 7600 Series ES+ Line Card Configuration Example

Activation of a Cisco 7600 Series ES+ Line Card Configuration Example

Line Card Online Diagnostics

Onboard Failure Logging

Troubleshooting ES+ Transport Low Queue

Troubleshooting

This chapter describes techniques that you can use to troubleshoot the operation of your Cisco 7600 Series Ethernet Services Plus (ES+) and Ethernet Services Plus T (ES+T) line cards.

For more information about the commands used in this chapter, see the Cisco IOS Release 12.2 SR Command References at http://www.cisco.com/en/US/products/ps6922/prod_command_reference_list.html .

It includes the following sections:

The first section provides information about basic interface troubleshooting. If you are having a problem with your Small Form-factor Pluggable (SFP) and small form factor pluggable (XFP) modules, use the steps in the “Using the Cisco IOS Event Tracer to Troubleshoot Problems” section to begin your investigation of a possible interface configuration problem.


Note The information provided in this chapter is applicable to both the ES+ and ES+T line cards unless specified otherwise.


General Troubleshooting Information

This section describes general information for troubleshooting the ES+ line card. It includes the following sections:

Interpreting Console Error Messages

To view the explanations and recommended actions for Cisco 7600 series router error messages, including messages related to Cisco 7600 series router ES+ line cards, refer to the Cisco 7600 Series Cisco IOS System Message Guide, 12.2SR at http://www.cisco.com/en/US/docs/ios/system/messages/guide/consol_smg.html

Using debug Commands

Along with the other debug commands supported on the Cisco 7600 series router, you can obtain specific debug information for the ES+ line card on the Cisco 7600 series router using the debug hw-module privileged EXEC command.

The debug hw-module command is intended for use by Cisco technical support personnel.


Caution Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use.

For information about other debug commands supported on the Cisco 7600 series routers, refer to the Cisco IOS Debug Command Reference, Release 12.2 SR at http://www.cisco.com/en/US/docs/ios/12_2/debug/command/reference/122debug.html.

Using show Commands

There are several show commands that you can use to monitor and troubleshoot the ES+ line card on the Cisco 7600 series routers. For more information about show commands, see the Cisco IOS Release 12.2 SR Command References at http://www.cisco.com/en/US/products/ps6922/prod_command_reference_list.html .

Using the Cisco IOS Event Tracer to Troubleshoot Problems


Note The Event Tracer feature is intended for use as a software diagnostic tool and should be configured only under the direction of a Cisco Technical Assistance Center (TAC) representative.


The Event Tracer feature provides a binary trace facility for troubleshooting Cisco IOS software. This feature gives Cisco service representatives additional insight into the operation of the Cisco IOS software and can be useful in helping to diagnose problems in the unlikely event of an operating system malfunction or, in the case of redundant systems, Route Processor switch over.

Event tracing works by reading informational messages from specific Cisco IOS software subsystem components that have been pre-programmed to work with event tracing, and by logging messages from those components into system memory. Trace messages stored in memory can be displayed on the screen or saved to a file for later analysis.

For more information about using the Event Tracer feature, refer to the following URL:

http://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/evnttrcr.html

Troubleshooting SFP/XFP Issues


Note The Cisco 7600-ES+ line card uses a slot, port numbering scheme. The slot refers to whichever slot the line card occupies in the router. The port numbering begins at 1 on all versions of the Cisco 7600-ES+ line card. The upper limit depends on the card type. This physical port numbering is reflected in CLI messages and all references to port numbers that are visible to the user.


Use the following commands when troubleshooting small form-factor pluggable (SFP) issues from the route processor (RP) side:

 

Command
Purpose
Router # show interfaces [interface interface-number] capabilities [module number]

Displays the interface capabilities for a module, an interface, or all interfaces.

Router # show interfaces [interface interface-number] status [err-disabled | module number]

Displays the interface status.

Router # show interfaces [interface interface-number] transceiver [threshold violations] [detail | {module number}]

Displays information about the optical transceivers that have digital optical monitoring (DOM) enabled

Router # show idprom interface

Displays IDPROMs for the line cards.

Use the following commands when troubleshooting small form-factor pluggable (SFP) issues from the ES+ line card side:

 

Command
Purpose

Router# remote command module num

Display the details of transceiver configuration and operation status directly from the line card side.

Preparing for Online Insertion and Removal of Cisco 7600 Series ES+ Line Card

The Cisco 7600 series router supports online insertion and removal (OIR) of the ES+ line card, in addition to each of the small form-factor pluggable (SFP or XFP) optical transceivers.

Therefore, you can remove a ES+ line card with its optical transceivers still intact, or you can remove an optical transceiver independently from the ES+ line card, leaving the ES+ line card installed in the router.

This section includes the following topics on OIR support:

Preparing for Online Removal of a Cisco 7600 Series ES+ Line Card

The Cisco 7600 series router supports OIR of the ES+ line card. To do this, you can power down an ES+ line card (which automatically deactivates any installed optical transceivers) and remove the ES+ line card still intact.

Although graceful deactivation of an ES+ line card is preferred using the no power enable module command, the Cisco 7600 series router does support removal of the ES+ line card without deactivating it first. If you plan to remove an ES+ line card, you can deactivate the ES+ line card first, using the no power enable module global configuration command. When you deactivate an ES+ line card using this command, it automatically deactivates each of the optical transceivers that are installed in that ES+ line card. Therefore, it is not necessary to deactivate each of the optical transceivers prior to deactivating the ES+ line card.

Either a blank filler plate or a functional optical transceiver should reside in every subslot of an ES+ line card during normal operation.

For more information about the recommended procedures for physical removal of the ES+ line card, refer to the Cisco 7600 Series Ethernet Services Plus Line Card Hardware Installation Guide.

Deactivating a Cisco 7600 Series ES+ Line Card

To deactivate an ES+ line card and its installed optical transceivers prior to removal of the line card, use the following command in global configuration mode:

 

Command
Purpose

Router(config)# no power enable module slot

Shuts down any installed interfaces, and deactivates the ES+ line card in the specified slot, where:

  • slot— Specifies the chassis slot number where the line card is installed.

For information on how to specify the physical locations of a ES+ line card on the Cisco 7600 series routers, see the Specifying the Slot Location for a Cisco 7600 Cisco 7600 Series ES+ Line Cards section in the Cisco 7600 Series Ethernet Services Plus Line Card Hardware Installation Guide.

Reactivating a Cisco 7600 Series ES+ Line Card

Once you deactivate a ES+ line card, whether or not you have performed an OIR, you must use the power enable module global configuration command to reactivate the ES+ line card.

If you did not issue a command to deactivate the optical transceivers installed in an ES+ line card, but you did deactivate the ES+ line card using the no power enable module command, then you do not need to reactivate the optical transceivers after an OIR of the ES+ line card. The installed optical transceivers automatically reactivate upon reactivation of the ES+ line card in the router.

For example, consider the case in which you remove an ES+ line card from the router to replace it with another ES+ line card. You reinstall the same optical transceivers into the new ES+ line card. When you enter the power enable module command on the router, the optical transceivers will automatically reactivate with the new ES+ line card.

To activate a ES+ line card and its installed optical transceivers after the ES+ line card has been deactivated, use the following command in global configuration mode:

 

Command
Purpose

Router(config)# power enable module slot

Activates the ES+ line card in the specified slot and its installed optical transceivers, where:

  • slot— Specifies the chassis slot number where the ES+ line card is installed.

For information on how to specify the physical locations of a ES+ line card on the Cisco 7600 series routers, see the Specifying the Slot Location for a Cisco 7600 Cisco 7600 Series ES+ Line Cards section in the Cisco 7600 Series Ethernet Services Plus Line Card Hardware Installation Guide.

Verifying Deactivation and Activation of a Cisco 7600 Series ES+ Line Card

To verify the deactivation of an ES+ line card, enter the show module command in privileged EXEC configuration mode. Observe the Status field associated with the ES+ line card that you want to verify.

The following example shows that the ES+ line card located in slot 10 is deactivated. This is indicated by its “PwrDown” status.

Router# show module 10
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
10 20 7600 ES+ 7600-ES+20G3CXL JAE1151865I
 
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
10 001d.e5e8.2a00 to 001d.e5e8.2a3f 0.301 12.2(33r)SRD 12.2(nightly PwrDown
 
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
10 7600 ES+ DFC XL 7600-ES+3CXL JAE115188YM 0.200 PwrDown
10 7600 ES+ 20xGE SFP 7600-ES+20G JAE1151860R 0.301 PwrDown
 
Mod Online Diag Status
---- -------------------
10 Not Applicable
Router#
 

To verify activation and proper operation of an ES+ line card, enter the show module command and observe “Ok” in the Status field as shown in the following example:

Router# show module 10
 
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
10 20 7600 ES+ 7600-ES+20G3CXL JAE1151865I
 
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
10 001d.e5e8.2a00 to 001d.e5e8.2a3f 0.301 12.2(33r)SRD 12.2(nightly Ok
 
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
10 7600 ES+ DFC XL 7600-ES+3CXL JAE115188YM 0.200 Ok
10 7600 ES+ 20xGE SFP 7600-ES+20G JAE1151860R 0.301 Ok
 
Mod Online Diag Status
---- -------------------
10 Pass
Router#
 

For information on how to specify the physical locations of a ES+ line card on the Cisco 7600 series routers, see the Specifying the Slot Location for a Cisco 7600 Cisco 7600 Series ES+ Line Cards section in the Cisco 7600 Series Ethernet Services Plus Line Card Hardware Installation Guide.

Deactivation and Activation Configuration Examples

This section provides the following examples of deactivating and activating an ES+ line card and optical transceivers:

Deactivation of a Cisco 7600 Series ES+ Line Card Configuration Example

Deactivate an ES+ line card when you want to perform OIR of the ES+ line card. The following example deactivates the ES+ line card that is installed in slot 5 of the router, its optical transceivers, and all of the interfaces. The corresponding console messages are shown:

Router# configure terminal
Router(config)# no power enable module 5
1w4d: %OIR-6-REMCARD: Card removed from slot 5, interfaces disabled
1w4d: %C6KPWR-SP-4-DISABLED: power to module in slot 5 set off (admin request)

Activation of a Cisco 7600 Series ES+ Line Card Configuration Example

Activate an ES+ line card if you have previously deactivated it. If you did not deactivate the optical transceivers, the optical transceivers automatically reactivate with reactivation of the ES+ line card.

The following example activates the ES+ line card that is installed in slot 5 of the router, its optical transceivers, and all of the interfaces (as long as the hw-module subslot shutdown command was not issued to also deactivate the optical transceivers):
 
Router# configure terminal
Router(config)# power enable module 5
 
Notice that there are no corresponding console messages shown with activation. If you re-enter the power enable module command, a message is displayed indicating that the module is already enabled:
 
Router(config)# power enable module 5
% module is already enabled

 

Line Card Online Diagnostics


Note Output from this procedure will vary slightly depending on which line card you are using, but the basic information will be the same.


Line card field diagnostic software is bundled with the main Cisco IOS software to enable you to test whether a suspect line card is faulty. For information on running online diagnostics, see the “Configuring Online Diagnostics” chapter in the Cisco 7600 Series Cisco IOS Software Configuration Guide, 15.0 SR at http://www.cisco.com/en/US/docs/routers/7600/ios/15S/configuration/guide/diags.html.

Onboard Failure Logging

The onboard failure logging (OBFL) feature gathers boot, environmental, and critical hardware failure data for field-replaceable units (FRUs), and stores the information in the nonvolatile memory of the FRU. This information is used for troubleshooting, testing, and diagnosis if a failure or other error occurs.

Because OBFL is on by default, data is collected and stored as soon as the card is installed. If a problem occurs, the data can provide information about historical environmental conditions, uptime, downtime, errors, and other operating conditions.

To use the OBFL feature, execute the attach command on the supervisor to enable OBFL on a ES+ line card.


Caution OBFL is activated by default in all cards and should not be deactivated. OBFL is used to diagnose problems in FRUs and to display a history of FRU data.

For information on configuring OBFL, see Onboard Failure Logging at
http://www.cisco.com/en/US/docs/ios/12_2sx/12_2sxh/feature/guide/sxhobfl.html.

Troubleshooting ES+ Transport Low Queue

On ES+T line cards, you can only configure up to 16 queues per port. An error message is displayed if the queue length exceeds the limit. This section explains how to use the show platform lowq command to check the number of queues per port for ES+ T line cards.

Scenario: Queue length exceeded the maximum queue limit for the specified interface and an error message is displayed.

Identifying the issue: Use the show platform lowq command to check the number of queues per port.

In the following example, service policy installation on the specified interface is failed because the number of queues exceeded the maximum queue limit of 16 per interface. Then show platform lowq command is used to check the number of queues per port.

Router(config-subif)# interface ten10/1.5
Router(config-subif)# service-policy input new1
Router(config-subif)#
%X40G_LOWQ-5-MAX_QUEUE_LIMIT_EXCEEDED: The maximum queue limit for interface 10/1 has been exceeded. Please check the configuration.
Router# show platform lowq
TenGigabitEthernet10/1
Input Queue count:8 Output Queue count:8 Total Queue count:16
 

The output of the show platform lowq command indicates that the number of queues created on the interface is already equal to the maximum number of queues allowed. So, you cannot create more queues on that interface. To resolve the issue, reduce the number of queues and then try the service policy installation again.