Introduction
This document describes how to troubleshoot unexpected reloads and what specific information to collect post-event on the Catalyst 9000 switches.
Requirements
There are no specific requirements for this document.
Components Used
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
There can be a variety of reasons an unexpected reload or silent reload event can occur. To troubleshoot and find the root cause, it is imperative to know the date/time of the event and collect all the information possible from the device right away. This guarantees that all the log collection is accurate and complete (not overwritten by the device as time goes on).
Common Reload Reasons
| ReloadReason |
Explanation |
Recommendation |
| PowerOn |
The power supply or the inlet circuit was manually shut off and on. |
If power event was unexpected, confirm health of physical power, battery backups, UPS, and so on. |
| Stack Merge due to Incompatibility |
The Standby switch of the stack lost communication with the Active and triggered a merge to re-elect a new Active switch. |
Check the stacking data cables, make sure they are properly seated. Replace the stacking cables. |
| Reload Command |
The command reload was executed on the CLI of the switch to instruct a manual software reload. |
None. |
| LocalSoft or CPU Catastrophic Err |
An error condition encountered in the software or CPU and the device was forced to be reset. |
Collect data and open a TAC case. |
| Power failure or Unknown |
The power supply of the device or inlet circuit must have suddenly failed, momentarily stopping the supply of power to the switch. |
Check if the power supplies are healthy, color of the LEDs and verify the inlet input circuit. |
| Redundancy force-switchover |
The command redundancy force-switchover was executed, causing the Standby to take the Active role and previous Active to go for a software reset. |
None. |
| Image Install |
The device was reloaded as a result of a new Cisco IOS® XE image installation. |
None. |
Collect Relevant Data
- Exact date/time of the occurrence.
- What was happening prior to the unexpected reload event? Perhaps a configuration change, log in to the device, or the device was untouched.
- Are there any Syslog messages of errors reported prior to the event? If there are any external Syslog servers, check if there are error logs related.
- Collect the
show tech-support command from the device.
- Any other insight or observations that can help understand what preceeded the event (power failures, and so on)
Caution: It is really important to collect the information right after the event occurred. There are no guarantees of providing a proper root cause if log collection happens days or weeks after the reload event took place.
Once the device is up and stable, collect this additional information to help determine the cause of the reload event.
Last Reload Reason
Check what was the last Reload Reason recorded by the system.
Switch#show version | include uptime|returned|reload|Version
Cisco IOS XE Software, Version 16.12.08
Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.8, RELEASE SOFTWARE (fc1)
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
GPL code under the terms of GPL Version 2.0. For more details, see the
BOOTLDR: System Bootstrap, Version 17.10.1r, RELEASE SOFTWARE (P)
Switch uptime is 3 hours, 14 minutes
System returned to ROM by Reload Command
Last reload reason: Reload Command
Next reload AIR license Level: AIR DNA Advantage
Switch Ports Model SW Version SW Image Mode
Configuration register is 0x102 (will be 0x2102 at next reload)
Onboard Logging
The device stores historical data on the latests reload events experienced. This information is important to check if the reloads have been recurrent and what was the reason recorded by the switch. The command used to see this data is show logging onboard switch <id> uptime detail.
Switch#show logging onboard switch active uptime detail
--------------------------------------------------------------------------------
UPTIME SUMMARY INFORMATION
--------------------------------------------------------------------------------
First customer power on : 04/28/2022 01:24:40
Total uptime : 0 years 41 weeks 2 days 9 hours 55 minutes
Total downtime : 0 years 22 weeks 4 days 7 hours 3 minutes
Number of resets : 326
Number of slot changes : 2
Current reset reason : Reload Command
Current reset timestamp : 07/19/2023 15:23:33
Current slot : 1
Chassis type : 215
Current uptime : 0 years 0 weeks 0 days 3 hours 0 minutes
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
UPTIME CONTINUOUS INFORMATION
--------------------------------------------------------------------------------
Time Stamp | Reset | Uptime
MM/DD/YYYY HH:MM:SS | Reason | years weeks days hours minutes
--------------------------------------------------------------------------------
07/13/2023 17:57:49 Image Install 0 0 0 0 0
07/13/2023 18:48:58 Reload Command 0 0 0 0 40
07/14/2023 00:55:30 Reload Command 0 0 0 6 0
07/14/2023 11:15:44 Reload Command 0 0 0 10 0
07/14/2023 20:02:26 Reload Command 0 0 0 8 0
07/17/2023 11:32:29 Reload Command 0 0 2 15 0
07/17/2023 21:00:59 Reload Command 0 0 0 9 0
07/18/2023 08:36:20 Reload Command 0 0 0 11 0
07/18/2023 11:00:28 Reload Command 0 0 0 2 0
07/19/2023 07:02:14 Reload Command 0 0 0 19 0
07/19/2023 15:23:33 Reload Command 0 0 0 8 0
--------------------------------------------------------------------------------
Tech-Support Command
This command gathers general information of the system and its current status. The commands described previously are included in this support package. This output file is requested by TAC when opening a case.
There are different ways to collect it:
- Logged Terminal Session: Use a terminal program and record the session to a text file. Then, from CLI, run the command
show tech-support and wait for the collection to finish. Provide the text file to the TAC case.
- Redirected to a File: From the CLI, run the command
show tech-support | redirect flash: ShowTech.txt. This creates a text file on the flash of the device with the results of the tech-support command. Afterwards, need to extract the file from the device using any transfer protocol available in your network. (TFTP, FTP, SCP, and so on).
Switch#show tech-support | redirect flash:ShowTech.txt
Switch#dir flash: | include .txt
499721 -rw- 16090 Jul 19 2023 15:24:02 +00:00 rdope_out.txt
499728 -rw- 1328023 Jul 19 2023 19:12:57 +00:00 ShowTech.txt
Switch#copy flash:ShowTech.txt tftp:
Address or name of remote host []? 192.168.1.1
Destination filename [ShowTech.txt]?
!!!
Commands to Collect for TAC
The switch stores system logging information that can be used to analyze in depth what was happening prior and after the unexpected reload event occurred. If there was a crash, the switch also stores the crash data into a encrypted file that can be analyzed by TAC experts.
Tracelogs
The tracelogs are internal logs of each process within Cisco IOS® XE. The tracelogs directory is created by default and its content is overwritten periodically. This folder can be found in the bootflash or flash of the Catalyst device.
Generate the Tracelogs
Switch#request platform software trace archive
Creating archive file [flash:Switch_1_RP_0_trace_archive-20230719-192115.tar.gz]
Done with creation of the archive file: [flash:Switch_1_RP_0_trace_archive-20230719-192115.tar.gz]
Switch#dir flash: | include .gz
499729 -rw- 29573907 Jul 19 2023 19:21:22 +00:00 Switch_1_RP_0_trace_archive-20230719-192115.tar.gz
System Reports
A system report is a compressed file that collects most of the information available in the software execution when an unexpected reload occurs. The system report contains tracelogs, crashinfo, and core files. This file is automatically created in the case of an unexpected reload on Cisco IOS® XE switches that was caused by a crash or software error. The file can be found in the crashinfo directory. This file is encrypted and requires TAC support for decoding and analysis.
Switch#dir crashinfo: | include .gz
Directory of crashinfo:/
19 -rwx 655080 Aug 7 2013 16:31:31 +00:00 system-report_1_20130807-163050-UTC.gz
Note: These files ending in .gz extension need to be extracted out of the device using any transfer protocol available in your network. (TFTP, FTP, SCP, and so on).
Cisco Bugs
There are some of the defects that can cause an unexpected reload on Catalyst 9000 Series switches.
Note: Only registered Cisco users can access internal bug information.
Related Information
Feedback