Troubleshooting
This chapter describes techniques that you can use to troubleshoot the operation of your Cisco 7600 Series Ethernet Services Plus (ES+) and Ethernet Services Plus T (ES+T) line cards.
For more information about the commands used in this chapter, see the Cisco IOS Release 12.2 SR Command References at http://www.cisco.com/en/US/products/ps6922/prod_command_reference_list.html .
It includes the following sections:
The first section provides information about basic interface troubleshooting. If you are having a problem with your Small Form-factor Pluggable (SFP) and small form factor pluggable (XFP) modules, use the steps in the “Using the Cisco IOS Event Tracer to Troubleshoot Problems” section to begin your investigation of a possible interface configuration problem.
Note
The information provided in this chapter is applicable to both the ES+ and ES+T line cards unless specified otherwise.
General Troubleshooting Information
This section describes general information for troubleshooting the ES+ line card. It includes the following sections:
Using debug Commands
Along with the other debug commands supported on the Cisco 7600 series router, you can obtain specific debug information for the ES+ line card on the Cisco 7600 series router using the debug hw-module privileged EXEC command.
The debug hw-module command is intended for use by Cisco technical support personnel.
Caution
Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use
debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use
debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased
debug command processing overhead will affect system use.
For information about other debug commands supported on the Cisco 7600 series routers, refer to the Cisco IOS Debug Command Reference, Release 12.2 SR at http://www.cisco.com/en/US/docs/ios/12_2/debug/command/reference/122debug.html.
Using the Cisco IOS Event Tracer to Troubleshoot Problems
Note
The Event Tracer feature is intended for use as a software diagnostic tool and should be configured only under the direction of a Cisco Technical Assistance Center (TAC) representative.
The Event Tracer feature provides a binary trace facility for troubleshooting Cisco IOS software. This feature gives Cisco service representatives additional insight into the operation of the Cisco IOS software and can be useful in helping to diagnose problems in the unlikely event of an operating system malfunction or, in the case of redundant systems, Route Processor switch over.
Event tracing works by reading informational messages from specific Cisco IOS software subsystem components that have been pre-programmed to work with event tracing, and by logging messages from those components into system memory. Trace messages stored in memory can be displayed on the screen or saved to a file for later analysis.
For more information about using the Event Tracer feature, refer to the following URL:
http://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/evnttrcr.html
Troubleshooting SFP/XFP Issues
Note
The Cisco 7600-ES+ line card uses a slot, port numbering scheme. The slot refers to whichever slot the line card occupies in the router. The port numbering begins at 1 on all versions of the Cisco 7600-ES+ line card. The upper limit depends on the card type. This physical port numbering is reflected in CLI messages and all references to port numbers that are visible to the user.
Use the following commands when troubleshooting small form-factor pluggable (SFP) issues from the route processor (RP) side:
|
|
Router #
show interfaces [interface interface-number] capabilities [module number]
|
Displays the interface capabilities for a module, an interface, or all interfaces. |
Router #
show interfaces [interface interface-number] status [err-disabled | module number]
|
Displays the interface status. |
Router #
show interfaces [interface interface-number] transceiver [threshold violations] [detail | {module number}]
|
Displays information about the optical transceivers that have digital optical monitoring (DOM) enabled |
Router #
show idprom interface
|
Displays IDPROMs for the line cards. |
Use the following commands when troubleshooting small form-factor pluggable (SFP) issues from the ES+ line card side:
|
|
Router# remote command module num |
Display the details of transceiver configuration and operation status directly from the line card side. |
Preparing for Online Insertion and Removal of Cisco 7600 Series ES+ Line Card
The Cisco 7600 series router supports online insertion and removal (OIR) of the ES+ line card, in addition to each of the small form-factor pluggable (SFP or XFP) optical transceivers.
Therefore, you can remove a ES+ line card with its optical transceivers still intact, or you can remove an optical transceiver independently from the ES+ line card, leaving the ES+ line card installed in the router.
This section includes the following topics on OIR support:
Preparing for Online Removal of a Cisco 7600 Series ES+ Line Card
The Cisco 7600 series router supports OIR of the ES+ line card. To do this, you can power down an ES+ line card (which automatically deactivates any installed optical transceivers) and remove the ES+ line card still intact.
Although graceful deactivation of an ES+ line card is preferred using the no power enable module command, the Cisco 7600 series router does support removal of the ES+ line card without deactivating it first. If you plan to remove an ES+ line card, you can deactivate the ES+ line card first, using the no power enable module global configuration command. When you deactivate an ES+ line card using this command, it automatically deactivates each of the optical transceivers that are installed in that ES+ line card. Therefore, it is not necessary to deactivate each of the optical transceivers prior to deactivating the ES+ line card.
Either a blank filler plate or a functional optical transceiver should reside in every subslot of an ES+ line card during normal operation.
For more information about the recommended procedures for physical removal of the ES+ line card, refer to the Cisco 7600 Series Ethernet Services Plus Line Card Hardware Installation Guide.
Reactivating a Cisco 7600 Series ES+ Line Card
Once you deactivate a ES+ line card, whether or not you have performed an OIR, you must use the power enable module global configuration command to reactivate the ES+ line card.
If you did not issue a command to deactivate the optical transceivers installed in an ES+ line card, but you did deactivate the ES+ line card using the no power enable module command, then you do not need to reactivate the optical transceivers after an OIR of the ES+ line card. The installed optical transceivers automatically reactivate upon reactivation of the ES+ line card in the router.
For example, consider the case in which you remove an ES+ line card from the router to replace it with another ES+ line card. You reinstall the same optical transceivers into the new ES+ line card. When you enter the power enable module command on the router, the optical transceivers will automatically reactivate with the new ES+ line card.
To activate a ES+ line card and its installed optical transceivers after the ES+ line card has been deactivated, use the following command in global configuration mode:
|
|
Router(config)# power enable module slot |
Activates the ES+ line card in the specified slot and its installed optical transceivers, where:
- slot— Specifies the chassis slot number where the ES+ line card is installed.
|
For information on how to specify the physical locations of a ES+ line card on the Cisco 7600 series routers, see the Specifying the Slot Location for a Cisco 7600 Cisco 7600 Series ES+ Line Cards section in the Cisco 7600 Series Ethernet Services Plus Line Card Hardware Installation Guide.
Verifying Deactivation and Activation of a Cisco 7600 Series ES+ Line Card
To verify the deactivation of an ES+ line card, enter the show module command in privileged EXEC configuration mode. Observe the Status field associated with the ES+ line card that you want to verify.
The following example shows that the ES+ line card located in slot 10 is deactivated. This is indicated by its “PwrDown” status.
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
10 20 7600 ES+ 7600-ES+20G3CXL JAE1151865I
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
10 001d.e5e8.2a00 to 001d.e5e8.2a3f 0.301 12.2(33r)SRD 12.2(nightly PwrDown
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
10 7600 ES+ DFC XL 7600-ES+3CXL JAE115188YM 0.200 PwrDown
10 7600 ES+ 20xGE SFP 7600-ES+20G JAE1151860R 0.301 PwrDown
To verify activation and proper operation of an ES+ line card, enter the show module command and observe “Ok” in the Status field as shown in the following example:
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
10 20 7600 ES+ 7600-ES+20G3CXL JAE1151865I
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
10 001d.e5e8.2a00 to 001d.e5e8.2a3f 0.301 12.2(33r)SRD 12.2(nightly Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
10 7600 ES+ DFC XL 7600-ES+3CXL JAE115188YM 0.200 Ok
10 7600 ES+ 20xGE SFP 7600-ES+20G JAE1151860R 0.301 Ok
For information on how to specify the physical locations of a ES+ line card on the Cisco 7600 series routers, see the Specifying the Slot Location for a Cisco 7600 Cisco 7600 Series ES+ Line Cards section in the Cisco 7600 Series Ethernet Services Plus Line Card Hardware Installation Guide.
Deactivation and Activation Configuration Examples
This section provides the following examples of deactivating and activating an ES+ line card and optical transceivers:
Deactivation of a Cisco 7600 Series ES+ Line Card Configuration Example
Deactivate an ES+ line card when you want to perform OIR of the ES+ line card. The following example deactivates the ES+ line card that is installed in slot 5 of the router, its optical transceivers, and all of the interfaces. The corresponding console messages are shown:
Router# configure terminal
Router(config)# no power enable module 5
1w4d: %OIR-6-REMCARD: Card removed from slot 5, interfaces disabled
1w4d: %C6KPWR-SP-4-DISABLED: power to module in slot 5 set off (admin request)
Activation of a Cisco 7600 Series ES+ Line Card Configuration Example
Activate an ES+ line card if you have previously deactivated it. If you did not deactivate the optical transceivers, the optical transceivers automatically reactivate with reactivation of the ES+ line card.
The following example activates the ES+ line card that is installed in slot 5 of the router, its optical transceivers, and all of the interfaces (as long as the hw-module subslot shutdown command was not issued to also deactivate the optical transceivers):
Router# configure terminal
Router(config)# power enable module 5
Notice that there are no corresponding console messages shown with activation. If you re-enter the power enable module command, a message is displayed indicating that the module is already enabled:
Router(config)# power enable module 5
% module is already enabled
Line Card Online Diagnostics
Note
Output from this procedure will vary slightly depending on which line card you are using, but the basic information will be the same.
Line card field diagnostic software is bundled with the main Cisco IOS software to enable you to test whether a suspect line card is faulty. For information on running online diagnostics, see the “Configuring Online Diagnostics” chapter in the Cisco 7600 Series Cisco IOS Software Configuration Guide, 15.0 SR at http://www.cisco.com/en/US/docs/routers/7600/ios/15S/configuration/guide/diags.html.
Onboard Failure Logging
The onboard failure logging (OBFL) feature gathers boot, environmental, and critical hardware failure data for field-replaceable units (FRUs), and stores the information in the nonvolatile memory of the FRU. This information is used for troubleshooting, testing, and diagnosis if a failure or other error occurs.
Because OBFL is on by default, data is collected and stored as soon as the card is installed. If a problem occurs, the data can provide information about historical environmental conditions, uptime, downtime, errors, and other operating conditions.
To use the OBFL feature, execute the attach command on the supervisor to enable OBFL on a ES+ line card.
Caution
OBFL is activated by default in all cards and should not be deactivated. OBFL is used to diagnose problems in FRUs and to display a history of FRU data.
For information on configuring OBFL, see Onboard Failure Logging at
http://www.cisco.com/en/US/docs/ios/12_2sx/12_2sxh/feature/guide/sxhobfl.html.
Troubleshooting ES+ Transport Low Queue
On ES+T line cards, you can only configure up to 16 queues per port. An error message is displayed if the queue length exceeds the limit. This section explains how to use the show platform lowq command to check the number of queues per port for ES+ T line cards.
Scenario: Queue length exceeded the maximum queue limit for the specified interface and an error message is displayed.
Identifying the issue: Use the show platform lowq command to check the number of queues per port.
In the following example, service policy installation on the specified interface is failed because the number of queues exceeded the maximum queue limit of 16 per interface. Then show platform lowq command is used to check the number of queues per port.
Router(config-subif)# interface ten10/1.5
Router(config-subif)# service-policy input new1
%X40G_LOWQ-5-MAX_QUEUE_LIMIT_EXCEEDED: The maximum queue limit for interface 10/1 has been exceeded. Please check the configuration.
Router# show platform lowq
Input Queue count:8 Output Queue count:8 Total Queue count:16
The output of the show platform lowq command indicates that the number of queues created on the interface is already equal to the maximum number of queues allowed. So, you cannot create more queues on that interface. To resolve the issue, reduce the number of queues and then try the service policy installation again.