Feedback
Table Of Contents
Troubleshooting Switch System Issues
Recovering the Administrator Password
Troubleshooting System Restarts
Working with a Firmware Upgrade Failure
Working with Recoverable Restarts
Working with Unrecoverable System Restarts
Troubleshooting a Failed Supervisor
Troubleshooting Switch System Issues
This chapter describes how to identify and resolve problems that might occur when accessing or starting up a single Cisco MDS 9000 Family switch. It includes the following sections:
•
Recovering the Administrator Password
•
•
Recovering the Administrator Password
If you forget the administrator password for accessing a Cisco MDS 9000 Family switch, you can recover the password using a local console connection. For the latest instructions on password recovery, go to http://www.cisco.com/warp/public/474/ and click on "MDS 9000 Series Multilayer Directors and Fabric Switches" under Storage Networking Routers.
Troubleshooting System Restarts
This section describes the different types of system crashes and how to respond to each type. It includes the following topics:
•
•
•
Overview
There are three different types of system restarts:
•
•
•
Most system restarts generate a Call Home event, but the condition causing a restart may become so severe that a Call Home event is not generated. Be sure that you configure the Call Home feature properly, follow up on any initial messages regarding system restarts, and fix the problem before it becomes so severe. For information about configuring Call Home, refer to the Cisco MDS 9000 Family Configuration Guide or the Cisco MDS 9000 Family Fabric Manager User Guide.
Working with a Firmware Upgrade Failure
When you perform a hitless upgrade (either by using the install all command, or by using Cisco Fabric Manager), and the upgrade fails with no error message displayed, please call technical support to determine the cause of the failure. Do not attempt to reboot the switch.
Working with Recoverable Restarts
Every process restart generates a Syslog message and a Call Home event. Even if the event is not service affecting you should identify and resolve the condition immediately because future occurrences could cause service interruption.
To respond to a recoverable system restart, follow these steps:
Step 1
switch# sh log logfile | include errorFor information about the meaning of each message, refer to the Cisco MDS 9000 Family System Messages Guide
The system output looks like the following:
Sep 10 23:31:31 dot-6 % LOG_SYSMGR-3-SERVICE_TERMINATED: Service "sensor" (PID 704) has finished with error code SYSMGR_EXITCODE_SY.switch# show logging logfile | include failJan 27 04:08:42 88 %LOG_DAEMON-3-SYSTEM_MSG: bind() fd 4, family 2, port 123, addr 0.0.0.0, in_classd=0 flags=1 fails: Address already in useJan 27 04:08:42 88 %LOG_DAEMON-3-SYSTEM_MSG: bind() fd 4, family 2, port 123, addr 127.0.0.1, in_classd=0 flags=0 fails: Address already in useJan 27 04:08:42 88 %LOG_DAEMON-3-SYSTEM_MSG: bind() fd 4, family 2, port 123, addr 127.1.1.1, in_classd=0 flags=1 fails: Address already in useJan 27 04:08:42 88 %LOG_DAEMON-3-SYSTEM_MSG: bind() fd 4, family 2, port 123, addr 172.22.93.88, in_classd=0 flags=1 fails: Address already in useJan 27 23:18:59 88 % LOG_PORT-5-IF_DOWN: Interface fc1/13 is down (Link failureor not-connected)Jan 27 23:18:59 88 % LOG_PORT-5-IF_DOWN: Interface fc1/14 is down (Link failureor not-connected)Jan 28 00:55:12 88 % LOG_PORT-5-IF_DOWN: Interface fc1/1 is down (Link failure or not-connected)Jan 28 00:58:06 88 % LOG_ZONE-2-ZS_MERGE_FAILED: Zone merge failure, Isolating port fc1/1 (VSAN 100)Jan 28 00:58:44 88 % LOG_ZONE-2-ZS_MERGE_FAILED: Zone merge failure, Isolating port fc1/1 (VSAN 100)Jan 28 03:26:38 88 % LOG_ZONE-2-ZS_MERGE_FAILED: Zone merge failure, Isolating port fc1/1 (VSAN 100)Jan 29 19:01:34 88 % LOG_PORT-5-IF_DOWN: Interface fc1/1 is down (Link failure or not-connected)switch#Step 2
switch# show processesThe following codes are used in the system output for the State (process state):
•
•
•
•
•
•
•
Note
The system output looks like the following (the output has been abbreviated to be more concise):
PID State PC Start_cnt TTY Process----- ----- -------- ----------- ---- -------------1 S 2ab8e33e 1 - init2 S 0 1 - keventd3 S 0 1 - ksoftirqd_CPU04 S 0 1 - kswapd5 S 0 1 - bdflush6 S 0 1 - kupdated71 S 0 1 - kjournald136 S 0 1 - kjournald140 S 0 1 - kjournald431 S 2abe333e 1 - httpd443 S 2abfd33e 1 - xinetd446 S 2ac1e33e 1 - sysmgr452 S 2abe91a2 1 - httpd453 S 2abe91a2 1 - httpd456 S 2ac73419 1 S0 vsh469 S 2abe91a2 1 - httpd470 S 2abe91a2 1 - httpdStep 3
switch# show process logProcess PID Normal-exit Stack-trace Core Log-create-time---------------- ------ ----------- ----------- ------- ---------------ntp 919 N N N Jan 27 04:08snsm 972 N Y N Jan 24 20:50Step 4
switch# show processes log pid 898The system output looks like the following:
Service: idehsdDescription: ide hotswap handler DaemonStarted at Mon Sep 16 14:56:04 2002 (390923 us)Stopped at Thu Sep 19 14:18:42 2002 (639239 us)Uptime: 2 days 23 hours 22 minutes 22 secondsStart type: SRV_OPTION_RESTART_STATELESS (23)Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGTERM (3)Exit code: signal 15 (no core)CWD: /var/sysmgr/workVirtual Memory:CODE 08048000 - 0804D660DATA 0804E660 - 0804E824BRK 0804E9A0 - 08050000STACK 7FFFFD10Register Set:EBX 00000003 ECX 0804E994 EDX 00000008ESI 00000005 EDI 7FFFFC9C EBP 7FFFFCACEAX 00000008 XDS 0000002B XES 0000002BEAX 00000003 (orig) EIP 2ABF5EF4 XCS 00000023EFL 00000246 ESP 7FFFFC5C XSS 0000002BStack: 128 bytes. ESP 7FFFFC5C, TOP 7FFFFD100x7FFFFC5C: 0804F990 0804C416 00000003 0804E994 ................0x7FFFFC6C: 00000008 0804BF95 2AC451E0 2AAC24A4 .........Q.*.$.*0x7FFFFC7C: 7FFFFD14 2AC2C581 0804E6BC 7FFFFCA8 .......*........0x7FFFFC8C: 7FFFFC94 00000003 00000001 00000003 ................0x7FFFFC9C: 00000001 00000000 00000068 00000000 ........h.......0x7FFFFCAC: 7FFFFCE8 2AB4F819 00000001 7FFFFD14 .......*........0x7FFFFCBC: 7FFFFD1C 0804C470 00000000 7FFFFCE8 ....p...........0x7FFFFCCC: 2AB4F7E9 2AAC1F00 00000001 08048A2C ...*...*....,...PID: 898SAP: 0UUID: 0switch#Step 5
switch# sh sys uptimeStart Time: Fri Sep 13 12:38:39 2002Up Time: 0 days, 1 hours, 16 minutes, 22 secondsTo determine if the restart is repetitive or a one-time occurrence, compare the length of time that the system has been up with the timestamp of each restart.
Step 6
switch# show coresThe system output looks like the following:
Module-num Process-name PID Core-create-time---------- ------------ --- ----------------5 fspf 1524 Jan 9 03:116 fcc 919 Jan 9 03:098 acltcam 285 Jan 9 03:098 fib 283 Jan 9 03:08This output shows all the cores presently available for upload from the active supervisor. The column entitled module-num shows the slot# on which the core was generated. In the example shown above, an fspf core was generated on the active supervisor module in slot 5. An fcc core was generated on the standby supervisory module in slot 6. Core dumps generated on the line card in slot 8 include acltcam and fib.
To copy the FSPF core dump in this example to a TFTP server with the IP address 1.1.1.1, enter the following command:
switch# copy core://5/1524 tftp::/1.1.1.1/abcdThe following command displays the file named zone_server_log.889 in the log directory.
switch# sh pro log pid 1473======================================================Service: ipsDescription: IPS ManagerStarted at Tue Jan 8 17:07:42 1980 (757583 us)Stopped at Thu Jan 10 06:16:45 1980 (83451 us)Uptime: 1 days 13 hours 9 minutes 9 secondsStart type: SRV_OPTION_RESTART_STATELESS (23)Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)Exit code: signal 6 (core dumped)CWD: /var/sysmgr/workVirtual Memory:CODE 08048000 - 080FB060DATA 080FC060 - 080FCBA8BRK 081795C0 - 081EC000STACK 7FFFFCF0TOTAL 20952 KBRegister Set:EBX 000005C1 ECX 00000006 EDX 2AD721E0ESI 2AD701A8 EDI 08109308 EBP 7FFFF2ECEAX 00000000 XDS 0000002B XES 0000002BEAX 00000025 (orig) EIP 2AC8CC71 XCS 00000023EFL 00000207 ESP 7FFFF2C0 XSS 0000002BStack: 2608 bytes. ESP 7FFFF2C0, TOP 7FFFFCF00x7FFFF2C0: 2AC8C944 000005C1 00000006 2AC735E2 D..*.........5.*0x7FFFF2D0: 2AC8C92C 2AD721E0 2AAB76F0 00000000 ,..*.!.*.v.*....0x7FFFF2E0: 7FFFF320 2AC8C920 2AC513F8 7FFFF42C ... ..*...*,...0x7FFFF2F0: 2AC8E0BB 00000006 7FFFF320 00000000 ...*.... .......0x7FFFF300: 2AC8DFF8 2AD721E0 08109308 2AC65AFC ...*.!.*.....Z.*0x7FFFF310: 00000393 2AC6A49C 2AC621CC 2AC513F8 .......*.!.*...*0x7FFFF320: 00000020 00000000 00000000 00000000 ...............0x7FFFF330: 00000000 00000000 00000000 00000000 ................0x7FFFF340: 00000000 00000000 00000000 00000000 ................0x7FFFF350: 00000000 00000000 00000000 00000000 ................0x7FFFF360: 00000000 00000000 00000000 00000000 ................0x7FFFF370: 00000000 00000000 00000000 00000000 ................0x7FFFF380: 00000000 00000000 00000000 00000000 ................0x7FFFF390: 00000000 00000000 00000000 00000000 ................0x7FFFF3A0: 00000002 7FFFF3F4 2AAB752D 2AC5154C .... output abbreviated ...Stack: 128 bytes. ESP 7FFFF830, TOP 7FFFFCD0Step 7
switch(config)# sys cores tftp:[//servername][/path]This command causes the switch to enable the automatic copy of core files to a TFTP server. For example, the following command sends the core files to the TFTP server with the IP address 10.1.1.1.
switch(config)# system cores tftp://10.1.1.1/coresThe following conditions apply:
•
•
copy core//module#/pid# tftp//tftp_ip_address/file_name•
•
Step 8
Working with Unrecoverable System Restarts
An unrecoverable system restart may occur in the following cases:
•
•
•
The effect of a process restart is determined by the policy configured for each process. Unrecoverable restarts may cause loss of functionality, restart of the active supervisor, a supervisor switchover, or restart of the switch.
To respond to an unrecoverable restart, perform the steps listed in the "Working with Recoverable Restarts" section.
Troubleshooting a Failed Supervisor
This section provides a workaround for a failed supervisor under certain conditions. An example situation is used to describe the problem and the workaround.
In this case, the supervisor failed when the standby was reloaded, or when the supervisor was replaced with a new one. It was discovered that the failed supervisor either had its version of code changed, or the running configuration on the active supervisor wasn't saved with the appropriate boot parameters. In either case, the problem was mismatched code on the active and standby supervisors. One clue that indicated the mismatched code was a "heartbeat" error on the active supervisor. Because of this error, the current flash images were unable to be copied from the active supervisor to the standby.
The workaround was to copy the images to compact flash, switch consoles, and load code from compact flash onto the second supervisor. The second supervisor was at a "loader" prompt which is indicative of missing boot statements. When a dir slot0 command was executed, none of the images appeared. This may have been due to mismatched images on supervisors or to not having current images in flash of the supervisor. Performing a copy slot0 bootflash command copied the images anyway. Once the images were loaded on the second supervisor and the boot statements were confirmed and saved on the active supervisor, the supervisor loaded and came up in "standby-ha" mode.
As a best practice, we recommended the following in order to understand how the switch can end up in this situation:
1.
2.
3.
4.
5.
6.
7.
