The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes the troubleshooting steps for unexpected reloads of Secure Firewall/Firepower firewalls.
Basic product knowledge.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
The information in this document is based on these software and hardware versions:
In this document, “reboot”, “reload”, and “restart” are used interchangeably. From the user perspective an unexpected reboot can roughly be defined as any reboot except for these cases that are well known, documented or expected:
Depending on the trigger, a reboot can be either graceful or ungraceful:
Common causes of reboots:
In chassis-based platforms, it’s important to consider the scope of the reboot, specifically, what exactly rebooted:
The default behavior includes:
The termination of certain critical processes, whether graceful or ungraceful, can cause reboots. For example:
Go through the troubleshooting workflow that refers to the troubleshooting steps:

Firepower 1000, 2100, Secure Firewall 1200, 3100 and 4200 running FTD in native mode or running ASA
4.1. Check the Analyze FTD messages Files section (FTD only).
4.2. Check the Analyze ssp-pm.log Files section.
4.3. Check the Analyze ssp-shutdown.log Files section.
4.4. Check the Analyze ASA/FTD Console logs section.
4.5. Proceed with the steps in Collect Data for Root Cause Analysis of Software Traceback/Crash in Secure Firewall for the verification of symptoms of software traceback and data collection.
4.6. Check the Analyze Chassis Reset Reasons section.
4.7. Check the Analyze Hardware Errors or Exceptions section.
4.8. Check the Analyze Platform log Files section.
Secure Firewall 3100, 4200 running FTD in container mode
4.9. Check the Analyze ssp-pm.log Files section.
4.10. Check the Analyze ssp-shutdown.log Files section.
4.11. Proceed with the steps in Collect Data for Root Cause Analysis of Software Traceback/Crash in Secure Firewall for the verification of symptoms of software traceback and data collection.
4.12. Check the Analyze Chassis Reset Reasons section.
4.13. Check the Analyze Hardware Errors or Exceptions section.
4.14. Check the Analyze Platform log Files section.
Firepower 4100/9300
4.15. Check the Analyze Chassis Reset Reasons section.
4.16. Proceed with the steps in Collect Data for Root Cause Analysis of Software Traceback/Crash in Secure Firewall for the verification of symptoms of software traceback and data collection.
Virtual platforms running FTD
4.17. Check the Analyze FTD messages Files section.
4.18. Check the Analyze ASA/FTD Console logs section.
4.19. Proceed with the steps in Collect Data for Root Cause Analysis of Software Traceback/Crash in Secure Firewall to verify symptoms of software traceback and for data collection.
Firepower 4100/9300 security module running FTD in native mode or running ASA
7.1. Check the Analyze FTD messages Files section (FTD only).
7.2. Check the Analyze ssp-pm.log Files section.
7.3. Check the Analyze ssp-shutdown.log Files section.
7.4. Check the Analyze ASA/FTD Console logs section.
7.5. Proceed with the steps in Collect Data for Root Cause Analysis of Software Traceback/Crash in Secure Firewall to verify symptoms of software traceback and for data collection.
7.6. Check the Analyze Hardware Errors or Exceptions section.
7.7. Check the Analyze SEL/OBFL files section.
7.8. Check the Analyze Platform log Files section.
Firepower 4100/9300 security module running FTD in container mode
7.9. Check the Analyze ssp-pm.log Files section.
7.10. Check the Analyze ssp-shutdown.log Files section.
7.11. Proceed with the steps in Collect Data for Root Cause Analysis of Software Traceback/Crash in Secure Firewall to verify symptoms of software traceback and for data collection.
7.12. Check the Analyze Hardware Errors or Exceptions section.
7.13. Check the Analyze SEL/OBFL Files section.
7.14. Check the Analyze Platform log Files section.
8.1. Check the Analyze FTD messages Files section.
8.2. Check the Analyze ssp-pm.log Files section.
8.3. Check the Analyze ssp-shutdown.log Files section.
8.4. Check the Analyze ASA/FTD Console logs section.
8.5. Proceed with the steps in Collect Data for Root Cause Analysis of Software Traceback/Crash in Secure Firewall to verify symptoms of software traceback and for data collection.
9. If there are signs of reboots due to software traceback, open a TAC case and provide the collected data.
10. Search software defects with the keywords unexpected, reboot, power, reload, restart, silent:

If a matching defect is found, then:
11. Go through the Data Collection section and open a TAC case.
The goal of this section is to check the uptime of the ASA/FTD application, the security module (Firepower 4100/9300 only), and the chassis.
> show version system
---------------[ firewall-FTD ]---------------
Model : Cisco Firepower 4112 Threat Defense (76) Version 7.6.1 (Build 291)
UUID : e9460e12-674c-11f0-97e5-d5ad3617d287
LSP version : lsp-rel-20241211-1948
VDB version : 408
---------------------------------------------------
Cisco Adaptive Security Appliance Software Version 9.22(1)21
SSP Operating System Version 2.16(0.3007)
Compiled on Thu 29-May-25 01:38 GMT by fpbesprd
System image file is "disk0:/fxos-lfbff-k8.2.16.0.136.SPA"
Config file at boot was "startup-config"
firewall-FTD up 5 days 0 hours
In FTD troubleshoot file, check the file command-outputs/'usr-local-sf-bin-sfcli.pl show_tech_support asa_lina_cli_util.output’:
/command-outputs$ less 'usr-local-sf-bin-sfcli.pl show_tech_support asa_lina_cli_util.output'
…
firewall-FTD up 5 days 0 hours
…
For ASA, check the uptime in the show-tech file.
> expert
admin@firepower:/ngfw/Volume/home/admin$ uptime
06:30:40 up 6 days, 8:01, 1 user, load average: 5.14, 5.45, 5.48
In the FTD troubleshoot file, check the file dir-archives/var-log/top.log:
dir-archives/var-log$ less top.log
===================================================================================================
Tue Jul 22 04:02:11 UTC 2025
top - 04:02:11 up 6 days, 13:57, 4 users, load average: 2.23, 2.42, 2.59
Tasks: 167 total, 2 running, 165 sleeping, 0 stopped, 0 zombie
%Cpu(s): 26.6 us, 7.8 sy, 1.6 ni, 64.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 7629.8 total, 2438.4 free, 4559.6 used, 631.8 buff/cache
MiB Swap: 5369.2 total, 3713.7 free, 1655.5 used. 2849.9 avail Mem
You can check the blade uptime:
> connect fxos
…
firewall# scope ssa
firewall /ssa # show slot detail expand
Slot:
Slot ID: 1
…
Blade Uptime: up 16 hours, 2 min
In chassis troubleshoot file, check the uptime in opt/cisco/platform/logs/sysmgr/sam_logs/topout.log file:
opt/cisco/platform/logs/sysmgr/sam_logs/topout.log $ less topout.log
[Jul 23 13:15:22]
top - 13:15:22 up 16:02, 1 user, load average: 12.24, 11.85, 11.75
Chassis uptime:
firewall /ssa # connect fxos
firewall(fxos)# show system uptime
System start time: Tue Jul 22 21:57:44 2025
System uptime: 6 days, 9 hours, 49 minutes, 50 seconds
Kernel uptime: 6 days, 9 hours, 45 minutes, 58 seconds
Active supervisor uptime: 6 days, 9 hours, 49 minutes, 50 seconds
Security module uptime:
firewall# scope ssa
firewall /ssa # show slot detail expand
Slot:
Slot ID: 1
…
Blade Uptime: up 5 days, 37 min
In chassis troubleshoot file, check the outputs of the show system uptime command in file *_BC1_all/FPRM_A_TechSupport/sw_techsupportinfo, and the show slot expand detail expand command in file *_BC1_all/FPRM_A_TechSupport/sam_techsupportinfo where * is the part of the troubleshoot file name, for example, 20250311123356_ FW_BC1_all.
This section covers the analysis of the FTD messages file that contains logs related to shutdown, reboot, and potential triggers.
FTD running on all platforms
The file contains logs related to shutdown, reboot, and their potential triggers on FTD running on:
The file is accessible in:
To find the trigger of shutdowns or reboots, check messages containing Defense in file messages file within the FTD troubleshoot files. Ensure to check all messages files including messages.1, messages.2, and so on. These messages clearly indicate a graceful shutdown:
root@CSF1220-2:/ngfw/var/log# grep -a Defense /ngfw/var/log/messages
messages:Jul 22 21:07:10 firepower-1220 SF-IMS[36894]: [36894] S11audit-startup-shutdown:start [INFO] Cisco Secure Firewall 1220CX Threat Defense has started
messages:Jul 22 21:12:37 firepower-1220 SF-IMS[62594]: [62594] sensor:CLI Start [INFO] Starting Cisco Secure Firewall 1220CX Threat Defense
messages:Jul 22 21:13:03 firepower-1220 SF-IMS[65410]: [65410] S99sensor:CLI Start [INFO] Starting Cisco Secure Firewall 1220CX Threat Defense
messages:Jul 23 12:56:43 CSF1220-2 SF-IMS[61878]: [61878] K01sensor:CLI Stop [INFO] Stopping Cisco Secure Firewall 1220CX Threat Defense
messages:Jul 23 12:57:03 CSF1220-2 SF-IMS[62749]: [62749] K69audit-startup-shutdown:stop [INFO] Cisco Secure Firewall 1220CX Threat Defense is shutting down
messages:Jul 23 13:00:21 CSF1220-2 SF-IMS[10866]: [10866] S11audit-startup-shutdown:start [INFO] Cisco Secure Firewall 1220CX Threat Defense has started
messages:Jul 23 13:00:36 CSF1220-2 SF-IMS[13443]: [13443] S99sensor:CLI Start [INFO] Starting Cisco Secure Firewall 1220CX Threat Defense
root@1140-2:/ngfw/var/log# grep -a Defense /ngfw/var/log/messages
messages:Jul 22 09:20:42 FPR1140-2 SF-IMS[6012]: [6012] S11audit-startup-shutdown:start [INFO] Cisco Firepower 1140 Threat Defense has started
messages:Jul 22 09:21:35 FPR1140-2 SF-IMS[9050]: [9050] S99sensor:CLI Start [INFO] Starting Cisco Firepower 1140 Threat Defense
messages:Jul 22 11:32:37 FPR1140-2 SF-IMS[15129]: [15129] K01sensor:CLI Stop [INFO] Stopping Cisco Firepower 1140 Threat Defense
messages:Jul 22 11:32:55 FPR1140-2 SF-IMS[15940]: [15940] K69audit-startup-shutdown:stop [INFO] Cisco Firepower 1140 Threat Defense is shutting down
messages:Jul 22 11:36:38 FPR1140-2 SF-IMS[5980]: [5980] S11audit-startup-shutdown:start [INFO] Cisco Firepower 1140 Threat Defense has started
messages:Jul 22 11:36:53 FPR1140-2 SF-IMS[7594]: [7594] S99sensor:CLI Start [INFO] Starting Cisco Firepower 1140 Threat Defense
This section covers the analysis of the security module or chassis messages file that contains logs related to OS kernel.
The file contains logs related to shutdown, reboot, and their potential triggers on FTD running on:
The file is accessible in:
These are the key points about the logs in this file:
For this reason, different logs in the same file can have different timestamps with different time zones.
Consider this example from a Firepower 4100 security module with the UTC+2 time zone:
2025-08-18T14:54:16+02:00 Firepower-module1/2 root: FXOS shutdown log started: pid = 52994 cmdline = /bin/sh/sbin/fxos_log_shutdown ####
2025-08-18T14:54:16+02:00 Firepower-module1/2 root: FXOS shutdown log completed: pid = 52994
2025-08-18T14:54:17+02:00 Firepower-module1/2 syslog-ng[1230]: syslog-ng shutting down; version='3.36.1' <--- syslog-ng shutting down indicates graceful shutdown
2025-08-18T12:57:11+00:00 intel-x86-64/1 syslog-ng[1237]: syslog-ng starting up; version='3.36.1' <--- syslog-ng start timestamp with UTC timezone
2025-08-18T12:57:11+00:00 intel-x86-64/1 kernel: [ 0.000000] Linux version 5.10.200-yocto-standard (oe-user@oe-host) (x86_64-wrs-linux-gcc (GCC) 10.3.0, GNU ld (GNU Binutils) 2.36.1.20210703) #1 SMP Fri Feb 7 01:00:00 UTC 2025
2025-08-18T12:57:11+00:00 intel-x86-64/1 kernel: [ 0.000000] Command line: rw console=ttyS0,38400 loglevel=2 auto kstack=128 reboot=force panic=1 ide_generic.probe_mask=0x1 ide1=noprobe pci=nocrs pci=realloc processor.max_cstate=1 iommu=pt nousb platform=sspxru boot_img=disk0:/fxos-lfbff-k8.2.16.0.136.SPA ciscodmasz=1048576 cisconrsvsz=1572864 hugepagesz=1g hugepages=12 pti=off rom_ver=1.3.1.44 ssp_mode=0
…
2025-08-18T12:58:34+00:00 intel-x86-64/1 csp-console: Adi not started yet, trying to bring up
2025-08-18T12:58:34+00:00 intel-x86-64/1 ssp_tz[12159]: INFO: Lina is started
2025-08-18T12:58:34+00:00 intel-x86-64/1 SMA: INFO ExecProcessorMain [140588852127296]: finished processing of sma_slot1.deletePlatformLicense, removed it from queue
2025-08-18T12:58:34+00:00 intel-x86-64/1 ssp_tz[12167]: INFO: notify TZ to ASA LINA <---- Time zone change was propagated to the firewall
2025-08-18T14:58:35+02:00 intel-x86-64/1 ssp_tz[12192]: INFO: successfully notified TZ to ASA LINA, wait <---- Logs with the UTC+2 timestamp
2025-08-18T14:58:36+02:00 intel-x86-64/1 csp-console: Adi not started yet, trying to bring up
2025-08-18T14:58:39+02:00 intel-x86-64/1 csp-console[1237]: Last message 'Adi not started yet,' repeated 1 times, suppressed by syslog-ng on intel-x86-64
This section covers the analysis of the /opt/cisco/platform/logs/ssp-pm.log files that contain logs related to ASA/FTD application shutdown or reboot.
The file contains shutdown or reboot logs for ASA/FTD applications running on:
The file is accessible only in:
To identify graceful shutdowns or reboots, look for lines like SHUTDOWN WARNING matching the reboot/shutdown timestamp. Ensure to check all ssp-pm.log files including ssp-pm.log.1, ssp-pm.log.2, and so on. These messages clearly indicate a graceful shutdown:
Fri Jul 25 10:28:51 UTC 2025 -(PM_UTILS)- SHUTDOWN WARNING: Beginning System Shutdown request for CSP Apps
Fri Jul 25 10:28:51 UTC 2025 -(PM_UTILS)- SHUTDOWN WARNING:-GetShutdownMaxForceQuitTimer: acquiring max force_quit_timeout from all heimdall confs
Fri Jul 25 10:28:51 UTC 2025 -(PM_UTILS)- SHUTDOWN WARNING: Upgrade process ready for reboot
Fri Jul 25 10:28:51 UTC 2025 -(PM_UTILS)- SHUTDOWN WARNING:-GetShutdownMaxForceQuitTimer: No force_quit_timeout found among conf files. Sendsigs_Sleep = 1
Fri Jul 25 10:28:51 UTC 2025 -(PM_UTILS)- SHUTDOWN WARNING: Continue System Shutdown request for CSP Apps
Fri Jul 25 10:28:51 UTC 2025 -(PM_UTILS)- SHUTDOWN WARNING:-SysShutdown_HeimdallSvcDownAllCspApps: corresponding heimdall conf file folder:
cisco-asa.9.22.2.9__asa_001_JMX2324G1DJ947ECO1.conf
Fri Jul 25 10:28:51 UTC 2025 -(PM_UTILS)- SHUTDOWN WARNING:-SysShutdown_HeimdallSvcDownAllCspApps: corresponding cspCfgXML file folder:
...cspCfg_cisco-asa.9.22.2.9__asa_001_JMX2324G1DJ947ECO1.xml.allocated_cpu_core
…
Fri Jul 25 10:28:51 UTC 2025 -(PM_UTILS)- SHUTDOWN WARNING:-SysShutdown_HeimdallSvcDownAllCspApps: stopping all CSPs according to existing heimdall/etc/confs
Fri Jul 25 10:28:51 UTC 2025 -(PM_UTILS)- SHUTDOWN WARNING:-SysShutdown_HeimdallSvcDownAllCspApps: Stopping App for CONF:cisco-asa.9.22.2.9__asa_001_JMX2324G1DJ947ECO1.conf
This section covers the analysis of the /opt/cisco/platform/logs/ssp-shutdown.log files, which contain logs generated when the chassis/security module shuts down or reboots.
The file contains shutdown or reboot logs for:
The file is accessible only in:
Refer to https://www.cisco.com/c/en/us/support/docs/security/sourcefire-defense-center/117663-technote-SourceFire-00.html on how to generate the FTD, chassis and security module troubleshoot files.
To find the trigger of graceful shutdowns or reboots, go through these steps:
#### Wed Jul 23 12:56:42 UTC 2025: FXOS shutdown started: pid = 61677 cmdline = /bin/sh/sbin/fxos_log_shutdown
root 61074 4593 2 12:56 ? 00:00:00 | \_ /usr/bin/perl /ngfw/usr/local/sf/bin/exec_perl.pl 068f09e6-3825-11ee-a72c-e78d34d303cc 192.0.2.100 068f09e6-3825-11ee-a72c-e78d34d303cc SF::System::
Privileged::RebootSystem
root 61097 61074 0 12:56 ? 00:00:00 | \_ /bin/sh /ngfw/usr/local/sf/bin/reboot.sh
root 61670 61097 0 12:56 ? 00:00:00 | \_ /bin/sh /sbin/shutdown -r now
root 61671 61670 0 12:56 ? 00:00:00 | \_ /bin/sh /sbin/exec_fxos_log_shutdown.sh
root 61676 61671 0 12:56 ? 00:00:00 | \_ /usr/bin/timeout 15 /sbin/fxos_log_shutdown
root 61677 61676 0 12:56 ? 00:00:00 | \_ /bin/sh /sbin/fxos_log_shutdown
root 61685 61677 0 12:56 ? 00:00:00 | \_ /bin/ps -ef –forest
> show managers
Type : Manager
Host : 192.0.2.100
Display name : 192.0.2.100
Version : 7.6.1 (Build 291)
Identifier : 068f09e6-3825-11ee-a72c-e78d34d303cc
Registration : Completed
Management type : Configuration and analytics
Example 2 – reboot due to software upgrade:
root 2432 1 0 03:59 ? 00:00:00 /bin/bash /isan/bin/kp_reboot.sh Reboot requested for Software Upgrade
root 2694 2432 0 03:59 ? 00:00:00 \_ /bin/sh /isan/bin/sam_reboot.sh Reboot requested for Software Upgrade
root 3774 2694 0 04:00 ? 00:00:00 \_ /usr/bin/sudo /sbin/shutdown -r now Reboot requested for Software Upgrade
root 3777 3774 0 04:00 ? 00:00:00 \_ /bin/sh /sbin/shutdown -r now Reboot requested for Software Upgrade
root 3778 3777 0 04:00 ? 00:00:00 \_ /bin/sh /sbin/exec_fxos_log_shutdown.sh
root 3783 3778 0 04:00 ? 00:00:00 \_ /usr/bin/timeout 15 /sbin/fxos_log_shutdown
root 3784 3783 0 04:00 ? 00:00:00 \_ /bin/sh /sbin/fxos_log_shutdown
root 3790 3784 0 04:00 ? 00:00:00 \_ /bin/ps -ef –forest
Example 3 – reboot was requested by the admin user by running the CLISH reboot command:
root 22189 1 0 Jul23 ? 00:00:01 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups
root 20963 22189 0 06:55 ? 00:00:00 \_ sshd: admin [priv]
admin 20970 20963 0 06:55 ? 00:00:00 | \_ sshd: admin@pts/0
admin 20975 20970 0 06:55 pts/0 00:00:00 | \_ /isan/bin/ucssh --ucs-mgmt -p admin -c connect ftd -z /home/admin
admin 21077 20975 0 06:55 pts/0 00:00:00 | \_ /ngfw/usr/bin/clish
admin 36022 21077 0 07:23 pts/0 00:00:00 | \_ /ngfw/usr/bin/clish
admin 36023 21077 0 07:23 pts/0 00:00:00 | \_ /bin/sh /tmp/klish.fifo.21077.1Hwyq1
admin 36024 36023 8 07:23 pts/0 00:00:01 | \_ /usr/bin/perl /usr/local/sf/bin/sfcli.pl reboot system
root 36083 36024 0 07:23 pts/0 00:00:00 | \_ /usr/bin/sudo /ngfw/usr/local/sf/bin/reboot.sh
root 36088 36083 0 07:23 pts/0 00:00:00 | \_ /bin/sh /usr/local/sf/bin/reboot.sh
root 36433 36088 0 07:23 pts/0 00:00:00 | \_ /bin/sh /sbin/shutdown -r now
root 36434 36433 0 07:23 pts/0 00:00:00 | \_ /bin/sh /sbin/exec_fxos_log_shutdown.sh
root 36439 36434 0 07:23 pts/0 00:00:00 | \_ /usr/bin/timeout 15 /sbin/fxos_log_shutdown
root 36440 36439 0 07:23 pts/0 00:00:00 | \_ /bin/sh /sbin/fxos_log_shutdown
root 36448 36440 0 07:23 pts/0 00:00:00 | \_ /bin/ps -ef --forest
Example 4 – shutdown was triggered by clicking the power button. PBTN represents a physical power button press. Pressing the power button on a machine triggers this event is triggered. The script uses this event to initiate a shutdown process, typically because a user intends to turn off the firewall safely:
root 1310 1 0 00:14 ? 00:00:00 /usr/sbin/acpid -c /etc/acpi/events
root 50700 1310 0 00:32 ? 00:00:00 \_ /bin/sh /etc/acpi/acpi_handler.sh button/power PBTN 00000080 00000000
root 50701 50700 0 00:32 ? 00:00:00 \_ /bin/sh /sbin/shutdown -h now
root 50702 50701 0 00:32 ? 00:00:00 \_ /bin/sh /sbin/exec_fxos_log_shutdown.sh
root 50707 50702 0 00:32 ? 00:00:00 \_ /usr/bin/timeout 15 /sbin/fxos_log_shutdown
root 50708 50707 0 00:32 ? 00:00:00 \_ /bin/sh /sbin/fxos_log_shutdown
root 50715 50708 0 00:32 ? 00:00:00 \_ /bin/ps -ef –forest
Example 5 – reboot was requested by the internal npu_accel_mgr process:
root 12649 5280 0 13:59 ? 00:01:06 \_ npu_accel_mgr
root 19030 12649 0 18:22 ? 00:00:00 | \_ /bin/sh /sbin/shutdown -r now
root 19031 19030 0 18:22 ? 00:00:00 | \_ /bin/sh /sbin/exec_fxos_log_shutdown.sh
root 19036 19031 0 18:22 ? 00:00:00 | \_ /usr/bin/ timeout 15 /sbin/fxos_log_shutdown
root 19037 19036 0 18:22 ? 00:00:00 | \_ /bin/sh /sbin/fxos_log_shutdown
root 19043 19037 0 18:22 ? 00:00:00 | \_ /bin/ps -ef –forest
Example 6 – shutdown was requested by the internal poshd process:
root 643 1 0 13:00 ? 00:00:00 /usr/bin/poshd
root 13207 643 0 13:43 ? 00:00:00 \_ /bin/sh /sbin/shutdown -h -P now
root 13208 13207 0 13:43 ? 00:00:00 \_ /bin/sh /sbin/exec_fxos_log_shutdown.sh
root 13213 13208 0 13:43 ? 00:00:00 \_ /usr/bin/timeout 15 /sbin/fxos_log_shutdown
root 13214 13213 0 13:43 ? 00:00:00 \_ /bin/sh /sbin/fxos_log_shutdown
root 13221 13214 0 13:43 ? 00:00:00 \_ /bin/ps -ef –forest
poshd is responsible for the graceful shutdown of the device in case of power issue (either through power button or external power factor).
Example 7 – reboot was triggered by the state changes of the firewall engine Lina, such as:
root 56833 56198 0 00:48 ? 00:00:00 | \_ /ngfw/usr/local/sf/bin/consoled /ngfw/var/run/lina.pid -coredump_filter=0x67 /ngfw/usr/local/asa/bin/lina_monitor -l
root 57003 56833 0 00:48 ? 00:00:00 | \_ /bin/sh /sbin/shutdown -r now
root 59448 57003 0 00:48 ? 00:00:00 | \_ /bin/sh /sbin/exec_fxos_log_shutdown.sh
root 59453 59448 0 00:48 ? 00:00:00 | \_ /usr/bin/timeout 15 /sbin/fxos_log_shutdown
root 59454 59453 0 00:48 ? 00:00:00 | \_ /bin/sh /sbin/fxos_log_shutdown
root 59461 59454 0 00:48 ? 00:00:00 | \_ /bin/ps -ef –forest
This section covers the analysis of the chassis reset reasons. On Firepower 4100/9300 chassis, the reset reason is accessible only in:
# connect fxos
(fxos)# show system reset-reason
On Firepower 1000, 2100, Secure Firewall 1200, 3100, 4200, the reset reason is accessible only in:
Refer to https://www.cisco.com/c/en/us/support/docs/security/sourcefire-defense-center/117663-technote-SourceFire-00.html on how to generate the FTD, chassis and security module troubleshoot files.
To find the reset reason for hardware platforms other than Firepower 4100/9300, go through these steps:
|
Reset reason |
Explanation |
|
PowerOn |
The last reset was caused by a power-on event, (AC power cycle) |
|
LocalSoft |
The last reset was caused by a local software reset (software reset to CPU) |
|
FanFail |
Fan-tray absence or fan failure detection causes the system to power cycle |
|
RP-Reset |
This bit indicates that a power cycle was requested by the software |
|
BootRomUpgrade |
The last reset was caused by an image upgrade flash reset |
|
BootRomUpgradeFail |
Upgrade process failed |
|
Watchdog/PCH |
The last reset was caused by a watchdog timer timeout in the FPGA |
|
Manual |
The last reset was caused by a manual push-button reset |
|
SwitchOff |
The last reset was caused by a manual power switch toggle-off |
|
Unrecoverable Error |
Caused by CPU catastrophic error signal |
|
ResetRequest |
The last reset was caused by the CPU asserting the reset signal This can also be caused by a kernel panic This reason is also shown when FTD reboot is manually triggered from the CLISH, Lina or FMC |
|
PowerCycleRequest |
The last reset/power cycle was caused by the CPU asserting any of the sleep signals |
To find the reset reason for Firepower 4100/9300, go through these steps:
FPR4115-2 # connect fxos
…
FPR4115-2(fxos)# show system reset-reason
----- reset reason for Supervisor-module 1 (from Supervisor in slot 1) ---
1) At 612806449 usecs after Tue Jul 22 23:50:33 2025
Reason: Reset Requested by CLI command reload
Service:
Version: 5.0(3)N2(4.141)
2) No time
Reason: Unknown
Service:
Version: 5.0(3)N2(4.141)
3) No time
Reason: Unknown
Service:
Version: 5.0(3)N2(4.141)
4) At 723697 usecs after Thu Oct 3 17:56:08 2024
Reason: Reset Requested by CLI command reload
Service:
Version: 5.0(3)N2(4.120)
The output of this command is also available in the chassis troubleshoot file *_BC1_all/FPRM_A_TechSupport/sw_techsupportinfo, where * is the part of the troubleshoot file name, for example, 20250311123356_ FW_BC1_all
# pwd
20250521112103_FPR9300_BC1_all/FPRM_A_TechSupport
# less sam_techsupportinfo
Most common reasons:
|
Reset requested by CLI command reload |
|
The last reset was caused by watchdog timeout |
|
Reset requested due to fatal system error |
|
Reset Requested due to fatal module error |
|
Power down due to temperature sensor policy trigger |
|
Reset due to bad voltage |
|
Reset requested due to fatal system error |
|
Reset caused by multiple non correctable ASIC memory errors. |
|
Reset due to kernel panic |
|
Reset triggered due to HA policy of switchover |
|
Kernel reboot request |
|
Reset because of an unknown reason |
2. The Unknown reason with no time at step 1 usually indicates abrupt power loss or power supply unit (PSU) issues. The recommendation is to check incidents for external power loss or power distribution (PDU) problems.
3. Additionally, check the output of the show logging onboard internal reset-reason | no-more command. The output of this command is also available in the chassis troubleshoot file *_BC1_all/FPRM_A_TechSupport/sw_kernel_trace_log, where * is the part of the troubleshoot file name, for example, 20250311123356_ FW_BC1_all:
FPR4115-2 # connect fxos
…
KSEC-FPR4115-2(fxos)# show logging onboard internal reset-reason | no-more
----------------------------
OBFL Data for
Module: 1
----------------------------
Reset Reason for this card:
Image Version : 5.0(3)N2(4.141)
Reset Reason (SW): Unknown (0) at time Fri Jul 4 23:57:21 2025
Service (Additional Info):
Reset Reason (HW): FPGA reset code: 0x0001
FPGA Power-ON Reset at time Fri Jul 4 23:57:21 2025
This section covers the analysis of the files that contain hardware errors or exceptions.
Firepower 1000, 2100, Secure Firewall 1200, 3100, 4200, Firepower 4100/9300 security modules
The relevant files are:
These files can contain hardware errors or exceptions logs for:
The files are accessible only in:
Examples of hardware-related errors:
opt/cisco/config/var/logs/ $ less dmesg.log
[Sun Apr 27 05:17:10 2025] mce: [Hardware Error]: Machine check events logged
[Sun Apr 27 05:17:10 2025] mce: [Hardware Error]: Machine check events logged
/opt/cisco/platform/logs $ less messages
Jul 9 17:39:38 fw2 kernel: [ 628.949572] [Hardware Error]: Corrected error, no action required.
Jul 9 17:39:38 fw2 kernel: [ 629.023798] [Hardware Error]: CPU:2 (19:1:1) MC18_STATUS[Over|CE|MiscV|AddrV|-|-|SyndV|CECC|-|-|-]: 0xdc2040000000011b
Jul 9 17:39:38 fw2 kernel: [ 629.152245] [Hardware Error]: Error Addr: 0x0000000068d06d40
Jul 9 17:39:38 fw2 kernel: [ 629.152248] [Hardware Error]: PPIN: 0x02b0bc114ba60075
Jul 9 17:39:38 fw2 kernel: [ 629.281830] [Hardware Error]: IPID: 0x0000009600550f00, Syndrome: 0x000400040a801200
Jul 9 17:39:38 fw2 kernel: [ 629.374797] [Hardware Error]: Unified Memory Controller Ext. Error Code: 0, DRAM ECC error.
Jul 9 17:39:38 fw2 kernel: [ 629.475078] [Hardware Error]: cache level: L3/GEN, tx: GEN, mem-tx: RD
Jul 9 17:44:40 fw2 kernel: [ 930.967028] [Hardware Error]: Corrected error, no action required.
Jul 9 17:44:40 fw2 kernel: [ 931.041247] [Hardware Error]: CPU:2 (19:1:1) MC18_STATUS[Over|CE|MiscV|AddrV|-|-|SyndV|CECC|-|-|-]: 0xdc2040000000011b
Jul 9 17:44:40 fw2 kernel: [ 931.169689] [Hardware Error]: Error Addr: 0x000000007191d0c0
Jul 9 17:44:40 fw2 kernel: [ 931.237616] [Hardware Error]: PPIN: 0x02b0bc114ba60075
Jul 9 17:44:40 fw2 kernel: [ 931.299275] [Hardware Error]: IPID: 0x0000009600550f00, Syndrome: 0x000040000a801100
/opt/cisco/platform/logs $ less mce.log
Hardware event. This is not a software error.
MCE 0
CPU 0 BANK 7 TSC 7b29f624ae62e
MISC 140129286 ADDR 3402b9a00
TIME 1745747035 Sun Apr 27 09:43:55 2025
MCG status:
MCi status:
Corrected error
MCi_MISC register valid
MCi_ADDR register valid
MCA: MEMORY CONTROLLER RD_CHANNEL0_ERR
Transaction: Memory read error
STATUS 8c00004000010090 MCGSTATUS 0
MCGCAP 1000c16 APICID 0 SOCKETID 0
PPIN afeebaf20487b95
MICROCODE 700000d
CPUID Vendor Intel Family 6 Model 86 Step 3
mcelog: failed to prefill DIMM database from DMI data
Catastrophic Errors (CATERR) on Firepower 4100/9300 security modules
A CATERR is an exception asserted by the processor. A CATERR can indicate a CPU crash or low-level problems that render the blade inaccessible. The chassis supervisor reboots the security module that experienced CATERR. The CATERR logs are located inside the Firepower 4100/9300 troubleshoot file, in CIMC<X>_TechSupport/obfl/ and CIMC<X>_TechSupport/var/log/sel, where <X> stands for module number.
An example from a sample chassis troubleshoot file *_BC1_all/CIMC<X>_TechSupport.tar.gz/CIMC<X>_TechSupport.tar/obfl/obfl-log, where * is the part of the troubleshoot file name, for example, 20250311123356_ FW_BC1_all and <X > is the security module ID:
2022 Mar 25 22:33:13 CET:4.1(30b):selparser:1950: selparser.c:727: # BF 06 00 00 01 02 00 00 19 35 3E 62 20 00 04 07 81 00 00 00 04 01 FF FF # 6bf | 03/25/2022 22:33:13 CET | CIMC | Processor CATERR_N #0x81 | Predictive Failure asserted | Asserted
The current Data Management Engine (DME) logs in *_BC1_all/FPRM_A_TechSupport.tar.gz\FPRM_A_TechSupport.tar\var\sysmgr\sam_logs\svc_sam_dme.log* or the logs before reboot in *_BC1_all/FPRM_A_TechSupport.tar.gz/FPRM_A_TechSupport.tar/opt/sam_logs.tgz/sam_logs.tar/sam_logs/svc_sam_dme.log* contains the lines with power cycle blade X due to CATERR, where X is the blade ID.
In the case of CATERR, a crash dump can be generated. The crash dump is in *_BC1_all/CIMC<X>_TechSupport.tar.gz/CIMC<X>_TechSupport.tar/techsupport_pidxxxx/nv/etc/log/eng-repo/caterrlog.first.
Example content:
CRASH DUMP START TIME = Mon Dec 19 00:42:40 2022
SERVER: presidio
GetCpuID: CPUID = 0x00050654
PresidioCatErrHandler: CPU TYPE : Skylake
handleCaterr_Purley: ---
PECI BUS : LOCKED
CPU Socket 1 : Populated
CPU Socket 2 : Populated
Stage1Dump: ---
MC0: IFU - Instruction Fetch Unit and Instruction Cache
CPU1 (0x30): ThreadID 0 : IA32_MC0_CTL (0x400) : 0x0000000000000fff
CPU1 (0x30): ThreadID 0 : IA32_MC0_STATUS (0x401) : 0x0000000000000000
CPU1 (0x30): ThreadID 0 : IA32_MC0_ADDR (0x402) : 0x0000000000000000
This section covers the analysis of the files that contain platform logs. Depending on the hardware, some of these files can contain logs related to reboot triggered by specific components.
Firepower 1000, 2100, Secure Firewall 1200, 3100, 4200, Firepower 4100/9300 security modules
The files contain chassis logs for:
The files are accessible only in:
To find potential logs related to reboot, search for lines that contain keywords such as reboot, restart, power, shut, reload. Note this is a best effort approach, since that the result of the search can return large number of matching lines. The user must analyze the matching lines.
Example 1:
admin@firewall:/opt/cisco/platform/logs$ grep -Ei "power|reboot|reload|restart|shut" *
...
npu_accel_fatal_err_stats.log:nic_top_inb_q_restart_drop_cnt = 0
npu_accel_fatal_err_stats.log:nic_top_norm_q_restart_drop_cnt = 0
npu_accel_fatal_err_stats.log:nic_top_inb_q_restart_drop_cnt = 0
npu_accel_mgr.log:2025 Oct 03 14:29:16 NPU ACCEL MGR: FATAL - NPU Accelerator FAILED - REBOOTING the device !!
npu_accel_mgr.log:2025 Oct 09 18:13:09 NPU ACCEL MGR: FATAL - NPU Accelerator FAILED - REBOOTING the device !!
npu_accel_mgr.log:2025 Oct 17 17:11:23 NPU ACCEL MGR: FATAL - NPU Accelerator FAILED - REBOOTING the device !!
npu_accel_mgr.log:2025 Oct 17 17:31:55 NPU ACCEL MGR: FATAL - NPU Accelerator FAILED - REBOOTING the device !!
npu_accel_mgr.log:2025 Oct 17 19:29:29 NPU ACCEL MGR: FATAL - NPU Accelerator FAILED - REBOOTING the device !!
ntp.log:27 Sep 02:21:16 ntpd[1580]: 0.0.0.0 c016 06 restart
ntp.log:27 Sep 02:23:27 ntpd[10193]: 0.0.0.0 c016 06 restart
...
In this case, the reboot was triggered due to the failure of the Network Processing Unit (NPU) Accelerator of the Secure Firewall 4200.
This section covers the analysis of the files with the security event logs (SEL), and the onboard failure logs (OBFL) on Firepower 4100/9300 security modules.
These files contain the module hardware events and power status changes and are located inside the Firepower 4100/9300 troubleshoot file, in CIMC<X>_TechSupport/obfl/ and CIMC<X>_TechSupport/var/log/sel, where <X> stands for module number.
Look for lines with case-insensitive stop, shutdown, power off, reset, critical.
Example 1:
2023 Sep 21 13:53:13:4.1(30b):cipmi:1088: [[xxxCVxxx]]:oem_command.c:457:IPMI Request Message --> Chan:15, Netfn:0x04, Cmd:0x02, Data: 0x41 0x03 0x20 0x46 0x6f 0xa1 0x61 0x74, CC:0x00
2023 Sep 21 13:53:13:4.1(30b):selparser:1203: [[xxxCVxxx]]:selparser.c:727: # A9 04 00 00 01 02 00 00 49 DA 0B 65 41 F0 04 20 46 00 00 00 6F A1 61 74 # 4a9 | 09/21/2023 13:53:13 AWST | System Mgmt Software | OS stop/shutdown #0x46 | Run-time critical stop | | Asserted
2023 Sep 21 13:53:14:4.1(30b):cipmi:1088: peci.c:278:Error: Failed to read local PCI config after 3 retries! Completion Code error [PECI Dev=0x30,Resp=0x00,CC=0x80] Request Details: [pci_bus=0x02,pci_dev=0x0a,pci_fxn=0x03,pci_reg=0x104,length=0x04]
2023 Sep 21 13:53:15:4.1(30b):kernel:-:<5>[platform_reset_cb_handler]:75:Platform Reset ISR -> ResetState: 1
2023 Sep 21 13:53:15:4.1(30b):cipmi:1088: ocmds_intel_me.c:251:Intel ME Operating State:[M0 without UMA](5)
2023 Sep 21 13:53:15:4.1(30b):cipmi:1088: ocmds_intel_me.c:261:Intel ME is initializing.
Example 2 – OS level shutdown:
2025 Jul 30 23:31:02 UTC:4.1(30b):cipmi:1052: [[xxxCVxxx]]:oem_command.c:457:IPMI Request Message --> Chan:15, Netfn:0x04, Cmd:0x02, Data: 0x41 0x03 0x20 0x46 0x6f 0xa1 0x61 0x74, CC:0x00
2025 Jul 30 23:31:02 UTC:4.1(30b):selparser:1169: [[xxxCVxxx]]:selparser.c:727: # 48 03 00 00 01 02 00 00 36 AB 8A 68 41 F0 04 20 46 00 00 00 6F A1 61 74 # 348 | 07/30/2025 23:31:02 UTC | System Mgmt Software | OS stop/shutdown #0x46 | Run-time critical stop | | Asserted
2025 Jul 30 23:31:04 UTC:4.1(30b):kernel:-:<5>[platform_reset_cb_handler]:75:Platform Reset ISR -> ResetState: 1
2025 Jul 30 23:31:05 UTC:4.1(30b):kernel:-:<4>[peci_ioctl]:293:non-responsive controller, resetting peci. process = qpi_logger.
2025 Jul 30 23:31:05 UTC:4.1(30b):cipmi:1052: ocmds_intel_me.c:251:Intel ME Operating State:[M0 without UMA](5)
2025 Jul 30 23:31:05 UTC:4.1(30b):cipmi:1052: ocmds_intel_me.c:261:Intel ME is initializing.
2025 Jul 30 23:31:05 UTC:4.1(30b):information_logger:589: uptime.c:1219:Host Transition Ocurred. Event Count: 146. New state is "Power Off".
Example 3 – the Baseboard Management Controller reset due to watchdog:
2025 Jun 14 07:44:50 CDT:4.1(30b):kernel:-:<3>[watchdog_init]:487:BMC Watchdog resetted BMC.
2025 Jun 14 07:44:50 CDT:4.1(30b):kernel:-:<3>[watchdog_init]:489:BMC Watchdog System Bus Debug Status Registers: 0x0 and0x0
2025 Jun 14 07:44:50 CDT:4.1(30b):kernel:-:<3>[watchdog_init]:523:BMC Watchdog resetted BMC due to OOM.
This section covers the analysis of the ASAConsole.log files that contain logs from the Lina engine and can be checked for signs of software traceback or reboots.
The file contains shutdown or reboot logs for ASA/FTD running on virtual platforms, Firepower 1000, 2100, and Secure Firewall 1200, 3100, 4200, and Firepower 4100 and 9300 security modules (not the chassis)
The file is accessible only in:
Symptoms of traceback and core file generation:
root@KSEC-CSF1210-6:/ngfw/var/log# less ASAconsole.log
2025-08-09 01:06:11 /bin/grep: (standard input): Resource temporarily unavailable
2025-08-09 01:06:11 /bin/grep: (standard input): Resource temporarily unavailable
2025-08-09 01:06:11 /bin/grep: (standard input): Resource temporarily unavailable
2025-08-09 01:06:35 !! !! First Crash in tid: 14834 signo: 11
2025-08-09 01:06:35
2025-08-09 01:06:35 Writing live core file to flash. Please do not reload.
2025-08-09 01:06:35
2025-08-09 01:06:35 Coredump starting....
2025-08-09 01:06:35 Corehelper: /opt/cisco/csp/cores/core.lina.11.14550.1754701595
2025-08-09 01:06:35 Waiting for Corehelper to finish....
2025-08-09 01:06:35 Livecore: generating coredump of 14550
2025-08-09 01:06:35 [New LWP 14795]
2025-08-09 01:06:35 [New LWP 14796]
2025-08-09 01:06:35 [New LWP 14834]
2025-08-09 01:06:35 [New LWP 14835]
2025-08-09 01:06:35 [New LWP 14836]
2025-08-09 01:06:35 [New LWP 14869]
2025-08-09 01:06:35 [New LWP 14934]
2025-08-09 01:06:35 [New LWP 14939]
2025-08-09 01:06:35 [New LWP 14940]
2025-08-09 01:06:35 [New LWP 14941]
2025-08-09 01:06:35 [New LWP 14942]
2025-08-09 01:06:35 [New LWP 14943]
2025-08-09 01:06:35 [New LWP 14945]
2025-08-09 01:06:35 [New LWP 14947]
2025-08-09 01:06:35 [New LWP 14948]
Symptoms of Lina engine boot, notice the gap in the log timestamps:
root@KSEC-CSF1210-6:/ngfw/var/log# less ASAconsole.log
2024-11-13 22:43:09 INFO: SW-DRBG health test passed.
2024-11-13 22:43:09 M_MMAP_THRESHOLD 65536, M_MMAP_MAX 82155
2024-11-13 22:43:10 /bin/grep: (standard input): Resource temporarily unavailable
2024-11-13 22:43:10 /bin/grep: (standard input): Resource temporarily unavailable
2024-11-13 22:43:10 /bin/grep: (standard input): Resource temporarily unavailable
2024-11-13 22:43:10 /bin/grep: (standard input): Resource temporarily unavailable
2024-11-13 22:43:10 User enable_1 logged in to firepower
2024-11-13 22:43:10 Logins over the last 1 days: 1.
2024-11-13 22:43:10 Failed logins since the last login: 0.
2024-11-13 22:43:10 Type help or '?' for a list of available commands.
2024-11-13 22:43:10 ^Mfirepower>
2025-04-04 09:11:46 System Cores 8 Nodes 1 Max Cores 24
2025-04-04 09:11:46 Number of Cores 3
2025-04-04 09:11:46 IO Memory Nodes: 1
2025-04-04 09:11:46 IO Memory Per Node: 1073741824 bytes num_pages = 262144 page_size = 4096
2025-04-04 09:11:46
2025-04-04 09:11:46 Global Reserve Memory Per Node: 2147483648 bytes Nodes=1
2025-04-04 09:11:46
2025-04-04 09:11:46 LCMB: got DMA 1073741824 bytes on numa-id=0, phys=0x00000001c0000000, virt=0x0000400040000000
2025-04-04 09:11:46 LCMB: HEAP-CACHE POOL got 2147483648 bytes on numa-id=0, virt=0x0000400080000000
2025-04-04 09:11:46
2025-04-04 09:11:46 total_reserved_mem = 1073741824
2025-04-04 09:11:46
2025-04-04 09:11:46 total_heapcache_mem = 2147483648
2025-04-04 09:11:46 total mem 5384115842 system 16318316544 kernel 21847377 image 85732792
2025-04-04 09:11:46 new 5384115842 old 1159474616 reserve 3221225472 priv new 2184737747 priv old 0
2025-04-04 09:11:46 Processor memory: 5384115842
2025-04-04 09:11:46 POST started...
2025-04-04 09:11:46 POST finished, result is 0 (hint: 1 means it failed)
2025-04-04 09:11:46
2025-04-04 09:11:46 Cisco Adaptive Security Appliance Software Version 9.22(1)1
Proceed with this section if as part of the Analyze chassis reset reasons section reset reasons like these are found:
No time
Reason: Unknown
Service:
Version:
These reasons can indicate potential problems with one or more of these:
Proceed with these steps:
scope chassis 1
show psu detail expand
scope psu 1
show stats history psu-stats detail expand
show stats psu-stats detail expand
exit
scope psu 2
show stats history psu-stats detail expand
show stats psu-stats detail expand
| Revision | Publish Date | Comments |
|---|---|---|
5.0 |
25-Nov-2025
|
Removed an internal box again. |
4.0 |
25-Nov-2025
|
Removed an internal box. |
3.0 |
12-Nov-2025
|
Added additional steps for troubleshooting unexpected reboot. |
2.0 |
24-May-2024
|
Updated introduction section to meet Cisco style guide length requirements. Unbolded unncessary words. |
1.0 |
31-Jan-2022
|
Initial Release |
Feedback