Troubleshoot ASA or FTD Unexpected Reloads

Introduction

This document describes how to troubleshoot scenarios where a Firepower Threat Defense (FTD) or Adaptive Security Appliance (ASA) device reloads without an obvious reason.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Understand the basics of Firepower and ASA hardware platforms
• Understand Logical devices on Firepower platforms

Components Used

The information in this document is based on these software and hardware versions:

• ASA 5500-X with ASA Software Version 9. x
• ASA 5500-X with FTD Software Version 6.2.3 and later
• Firepower 1000, 1100, 2100, 4100, and 9300 series with ASA Software Version 9. x
• Firepower 1000, 1100, 2100, 4100, and 9300 series with FTD Software Version 6.2.3 and later

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

In this document, the device refers to ASA or Firepower Next-Generation Firewalls (NGFW), rebranded as Cisco Secure Firewalls, that run either an ASA or FTD image on it as a logical device.

Cisco Secure Firewalls include various hardware and software versions. The ASA family includes 5500-X series firewalls and the Firepower family includes FPR 1000, 2100, 4100, and 9300 series devices. This document discusses the approach to start with in order to identify the level that the device or software has crashed across all the mentioned platforms and if the crash was real or not. It also lists all the artifacts to collect, where to find them, and how to use them in order to find the root cause of the crash.

Common Things to Check Across All Platforms and Logical Devices

Confirm that the Device (Logical or Chassis) has Rebooted or Crashed

For ASA, use the command from configuration mode in order to check the device uptime: # show version | in Up

On Firepower hardware, use these commands in order to check the device uptime and the chassis uptime (FXOS level):

FP4100-3# connect fxos
FP4100-3(fxos)# show system uptime

System start time:          Thu Oct 31 22:50:09 2019

System uptime:              391 days, 19 hours, 30 minutes, 45 seconds

Kernel uptime:              391 days, 19 hours, 34 minutes, 34 seconds

Active supervisor uptime:   391 days, 19 hours, 30 minutes, 45 seconds

Note: If you observe the device is up just from the time of issue, this confirms the device has rebooted.

Check and confirm if there are any power-related issues that can lead to sudden device reboots.

If the uptime does not relate to the timestamp of downtime in the network (or failover or unit leaving cluster), this means the issue did not occur due to device reload and diagnosis must navigate in a different direction altogether.

Check for Crashinfo in the Case of ASA SoftwareLina (on FTD) Crash

A system crash is a situation where the system has detected an unrecoverable error and has restarted itself. When a firewall crashes it creates a special text format file known as a crashinfo  file. This file provides diagnostic information and logs that help determine the root cause analysis of a crash. For an ASA, the crashinfo file is plain text stored in Flash: and contains the memory register contents with a long list of other information - software version, collected data, and so on.

Enter the show crashinfo command in the ASA CLI under privilege exec mode. You can look at the output in any text editor or even on the ASA console itself.

show flash | in crash

Share this output with the Cisco Technical Assistance Center (TAC) in a Service Request and they can decode it with internal tools. This output gives useful information about the processes and threads, which helps developers to review and correlate the crash with other events inside the device.

Note: Generally, when you collect show tech-support output from the ASA or Lina (on FTD), show crashinfo is ideally present in that output. However, many times the output is different or incomplete compared to directly running the show crashinfo command. Therefore, it is recommended to always enter the show crashinfo command directly on the ASA or Lina CLI.

In addition to the common details to check, there are more information and artifacts to collect that depend on the various level of crashes that can occur. On ASA platforms, there can be only a single level of crash. However, Firepower platforms can have either a logical device (FTD or ASA software) level crash or a chassis level (FXOS) crash.

After the uptime confirms that the device has crashed, a coredump file is generated which is necessary for further review by Cisco TAC. The coredump file can be of different types based on what component of the software has crashed. The coredump files also get saved into different directories/parts of the disk, based on what component has crashed.

Things to Check on ASA Platforms

The ASA platforms have only one component which can either be ASA or FTD.

All ASA Platforms that Run an ASA Image

The corefiles related to the crash are stored under disk0 of the internal flash drive. In order to check the corefiles, enter the dir disk0:/coredumpfsys command:

ciscoasa# dir disk0:/coredumpfsys
Directory of disk0:/coredumpfsys/

1071057 drwx 4096 23:14:58 Aug 30 2021 sysdebug
12 -rw- 87580218 04:49:23 Jun 04 2021 core_lina.1227726922.258.11.gz
11 drwx 16384 23:13:37 Aug 30 2021 lost+found

1 file(s) total size: 87580218 bytes
16106127360 bytes total (15749222400 bytes free/97% free)

Enter the show coredump filesystem  command in order to display any files on the coredump filesystem, which also shows disk space. It is recommended to archive the coredump files when convenient, as it is possible that a subsequent coredump can remove the previous coredump(s) in order to fit the current core.

ciscoasa# show coredump filesystem

Coredump Filesystem Size is 100 MB

Filesystem type is FAT for disk0

Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/loop0              102182     75240     26942  74% /mnt/disk0/coredumpfsys

Directory of disk0:/coredumpfsys/

246    -rwx  20205386    19:16:44 Nov 26 2021  core_lina.1227726922.258.11.gz
247    -rwx  36707919    19:21:56 Nov 26 2021  core_lina.1227727222.258.6.gz
248    -rwx  20130838    19:26:36 Nov 26 2021  core_lina.1227727518.258.11.gz

If you do not see a coredump file in disk0, there is a high chance that the coredump is not enabled which means the review can not be completed for this occurrence. In order to enable coredump for future occurrences, enter this command:

ciscoasa(config)#coredump enable

WARNING: Enabling coredump on an ASA5505 platform will delay the reload of the system in the   
event of software forced reload. The exact time depends on the size of the coredump generated.

Proceed with coredump filesystem allocation of 60 MB 
on 'disk0:' (Note this may take a while) ? [confirm]
 
Making coredump file system image!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Coredump file system image created & mounted successfully

/dev/loop0 on /mnt/disk0/coredumpfsys type vfat 
(rw,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1)

ASA Platforms Which Support Running FTD Image

The ASA platforms 5506-X, ASA 5508-X, ASA 5512-X, ASA 5515-X, ASA 5516-X, ASA 5525-X, ASA 5545-X, and ASA 5555-X support running FTD image as well as make it a next-gen firewall.

On all of these supported ASA platforms that run the FTD image, corefiles are located under /var/data/cores or /ngfw/var/data/cores via expert mode. They are also mirrored under the disk0:/coredumpfsys directory of Lina flash.

root@firepower:/var/data/cores# ls -l
total 59660
-rw-r--r-- 1 root root 4815651 Mar 14 17:07 core.SFDataCorrelato.2035.1552608478.gz
-rw-r--r-- 1 root root 56198339 Mar 14 16:47 core.lina.2113.1552607243.gz
root@firepower:/var/data/cores#


firepower# dir disk0:/coredumpfsys
Directory of disk0:/coredumpfsys/

2498562 -rw- 56198339 23:47:26 Mar 14 2019 core.lina.2113.1552607243.gz
2498563 -rw- 4815651 00:07:58 Mar 15 2019 core.SFDataCorrelato.2035.1552608478.gz

2 file(s) total size: 61013990 bytes
42949672960 bytes total (39523602432 bytes free/92% free)

Things to Check on Firepower Platforms

The Firepower platforms come with two software components. The first is the FXOS, which is the chassis operating system, and the second is the app instance, also known as the logical device, which can either be ASA or FTD. Therefore, it is important to identify which part crashed in order to determine in which location to download the corefiles

If the app instance crashes on Firepower 1000/2000/4100 and 9300, the crash info and corefiles are always generated by default. However, the core dump can be disabled in some cases.

In order to check if the core dump is enabled on 4100/9300, enter these commands:

connect module 1 console
Firepower-module1>show platform coredumps

Enable or Disable Firepower Module Core Dumps:

Enable core dumps on a Firepower module in order to help troubleshoot in the event of a system crash, or to send to Cisco TAC if requested.

Firepower# connect module 1 console
show coredump detail

The command output shows the current core dump status information and includes whether core dump compression is enabled.

Firepower-module1>show coredump detail
Configured status: ENABLED.
ASA Coredump: ENABLED.
Bootup status: ENABLED.
Compress during crash: DISABLED.

Use the config coredump command in order to enable or disable core dumps, and to enable or disable core dump compression during a crash.

• Enter the config coredump enable command in order to enable the creation of a core dump during a crash.
• Enter the config coredump disable command in order to disable core dump creation during a crash.
• Enter the config coredump compress enable command in order to enable compression of core dumps.
• Enter the config coredump compress disable command in order to disable core dump compression.

This example shows how to enable the core dump:

Firepower-module1>config coredump enable
Coredump enabled successfully.
ASA coredump enabled, do 'config coredump disableAsa' to disable
Firepower-module1>config coredump compress enable
WARNING: Enabling compression delays system reboot for several minutes after a system failure. Are you sure? (y/n):y
Firepower-module1>

Note: Core dump files consume disk space, and if space runs low and compression is not enabled, a core dump file is not saved even if core dumps are enabled.

Both crash and core files must be uploaded for a complete analysis because it is possible that the crash file does not contain all the data.

FP9300/FP4100 FXOS

On FP9300/FP4100, the FXOS corefiles are located under the local-mgmt cores directory.

firepower-4110# connect local-mgmt
firepower-4110(local-mgmt)# dir cores

1 9337521 Apr 30 11:28:15 2016 1462040896_0x101_snm_log.5289.tar.gz
1 1067736 Oct 09 10:38:49 2017 1507570679_firepower-4110_BC01_MEZZ0101_mcp_log.122.tar.gz
1 798663 Oct 10 18:05:54 2017 1507683913_firepower-4110_BC01_MEZZ0101_mcp_log.122.tar.gz
1 348160 Feb 11 23:53:25 2019 core.21845

Usage for workspace://
3999125504 bytes total
64200704 bytes used
3730071552 bytes free
firepower-4110(local-mgmt)#

In order to copy the core file from FXOS to your local computer, enter this command:

firepower-4110(local-mgmt)# copy workspace:/cores:/<file>.tar.gz scp://username@x.x.x.x

On FP9300/FP4100 Running FTD

On FP9300/FP4100 running FTD, corefiles are located under /var/data/cores or /ngfw/var/data/cores via expert mode. They are also mirrored under the disk0:/coredumpfsys directory of Lina flash.

root@firepower:/var/data/cores# ls -l
total 59660
-rw-r--r-- 1 root root 4815651 Mar 14 17:07 core.SFDataCorrelato.2035.1552608478.gz
-rw-r--r-- 1 root root 56198339 Mar 14 16:47 core.lina.2113.1552607243.gz
root@firepower:/var/data/cores#


firepower# dir disk0:/coredumpfsys
Directory of disk0:/coredumpfsys/

2498562 -rw- 56198339 23:47:26 Mar 14 2019 core.lina.2113.1552607243.gz
2498563 -rw- 4815651 00:07:58 Mar 15 2019 core.SFDataCorrelato.2035.1552608478.gz

2 file(s) total size: 61013990 bytes
42949672960 bytes total (39523602432 bytes free/92% free)

On FP9300/FP4100 Running ASA

On FP9300/FP4100 running ASA, corefiles are located under the disk0:/coredumpfsys directory.

asa# dir disk0:/coredumpfsys

Directory of disk0:/coredumpfsys/

11 drwx 16384 17:34:50 Sep 10 2018 lost+found
12 -rw- 317600388 16:43:40 Mar 14 2019 core.lina.6320.1552607012.gz

1 file(s) total size: 317600388 bytes
21476089856 bytes total (21255872512 bytes free/98% free)

On FP2100 FXOS/ASA/FTD

On FP2100 FXOS/ASA/FTD, corefiles are located under the local-mgmt cores directory whether you use ASA or FTD. On FTD, they are also mirrored under /ngfw/var/data/cores (or /var/data/cores) and /ngfw/var/common/ via expert mode. However, note that FP2100 platforms do not have the disk0:/coredumpfsys directory.

Note: Cisco bug ID CSCvh01912 was submitted in order to make FP2100 consistent with the FP9300/4100 platform. Until that is resolved, use the described location in order to find the corefiles.

Location of Firepower Core Files when the FTD is in Firepower 2100, 1000, ASA Appliance, and ISA 3000 Appliance:

For all of these platforms, use this procedure in order to locate the core files related to all the Firepower processes.

Under /ngfw/var/common/:

1. Connect to the CLI of the appliance via SSH or console.

2. Enter this as the expert mode:

> expert
admin@firepower:~$

3. Become a root user.

admin@firepower:~$ sudo su
Password:
root@firepower:/home/admin#

4. Navigate to the /ngfw/var/common/ folder, where the core files are located.

root@firepower:/home/admin# cd /ngfw/var/common/

5. Check the folder for the file.

root@firepower:/ngfw/var/common# ls -l | grep -i core
total 21616
-rw-r--r-- 1 root root 22130788 Nov  6  2020 process.core.tar.gz

FTD on FP2100: Under /ngfw/var/data/cores:

> expert
admin@firepower:~$ sudo su
[cut]
root@firepower:/home/admin# ls -l /ngfw/var/data/cores
total 133740
-rw-r--r-- 1 root root 4761622 Jun 4 05:13 core.SFDataCorrelato.28634.1622783636.gz
-rw-r--r-- 1 root root 132014190 Jun 4 05:17 core.lina.11.1378.1622783800.gz
drwx------ 2 root root 16384 Nov 5 2019 lost+found
drwxr-xr-x 3 root root 4096 Nov 5 2019 sysdebug


> connect fxos
[cut]
firepower# connect local-mgmt
firepower(local-mgmt)# dir cores

1 4761622 Jun 04 05:13:56 2021 core.SFDataCorrelato.28634.1622783636.gz
1 132014190 Jun 04 05:17:25 2021 core.lina.11.1378.1622783800.gz
2 16384 Nov 05 22:35:15 2019 lost+found/
3 4096 Nov 05 22:36:05 2019 sysdebug/

Usage for workspace://
85963259904 bytes total
15324155904 bytes used
70639104000 bytes free
firepower(local-mgmt)#

ASA on FP2100:

firepower-2110(local-mgmt)# dir cores

1 167408075 Jul 04 00:43:25 2018 core.lina.6.2025.1530657764.gz
2     16384 Mar 28 16:17:56 2018 lost+found/
3      4096 Mar 28 16:18:43 2018 sysdebug/

Note: The FXOS corefiles are stored under the same cores directory from connect local-mgmt.

On FP1000 FXOS/ASA/FTD

On FP1000 FXOS/ASA/FTD, this process is similar to the FP2100. In addition, the disk0:/coredumpfsys directory is available under the Lina side.

FTD on FP1000:

> system support diagnostic-cli
Attaching to Diagnostic CLI ... Press 'Ctrl+a then d' to detach.
Type help or '?' for a list of available commands.

FP1010> ena
Password:
FP1010# dir disk0:/coredumpfsys
Directory of disk0:/coredumpfsys/

13 -rw- 86493184 19:59:39 Jun 03 2021 core.lina.18707.1622750370.gz
1071057 drwx 4096 23:14:58 Aug 30 2019 sysdebug
14 -rw- 4770749 20:19:24 Jun 03 2021 core.SFDataCorrelato.7098.1622751564.gz
12 -rw- 197689 23:01:08 May 19 2021 core.top.6163.1621465268.gz
16 -rw- 4752067 20:28:03 Jun 03 2021 core.SFDataCorrelato.28195.1622752083.gz
11 drwx 16384 23:13:37 Aug 30 2019 lost+found
15 -rw- 5048839 20:20:32 Jun 03 2021 core.SFDataCorrelato.18952.1622751632.gz

5 file(s) total size: 101262528 bytes
123418959872 bytes total (110302621696 bytes free/89% free)


> connect fxos
[cut]

FP1010# connect local-mgmt
FP1010(local-mgmt)# dir cores

1 5048839 Jun 03 20:20:32 2021 core.SFDataCorrelato.18952.1622751632.gz
1 4752067 Jun 03 20:28:03 2021 core.SFDataCorrelato.28195.1622752083.gz
1 4770749 Jun 03 20:19:24 2021 core.SFDataCorrelato.7098.1622751564.gz
1 86493184 Jun 03 19:59:39 2021 core.lina.18707.1622750370.gz
1 197689 May 19 23:01:08 2021 core.top.6163.1621465268.gz
2 16384 Aug 30 23:13:37 2019 lost+found/
3 4096 Aug 30 23:14:58 2019 sysdebug/

Usage for workspace://
159926181888 bytes total
17475063808 bytes used
142451118080 bytes free


> expert
admin@FP1010:~$ sudo su
Password:
root@FP1010:/home/admin# ls -l /var/data/cores
total 99048
-rw-r--r-- 1 root root 5048839 Jun 3 20:20 core.SFDataCorrelato.18952.1622751632.gz
-rw-r--r-- 1 root root 4752067 Jun 3 20:28 core.SFDataCorrelato.28195.1622752083.gz
-rw-r--r-- 1 root root 4770749 Jun 3 20:19 core.SFDataCorrelato.7098.1622751564.gz
-rw-r--r-- 1 root root 86493184 Jun 3 19:59 core.lina.18707.1622750370.gz
-rw-r--r-- 1 root root 197689 May 19 23:01 core.top.6163.1621465268.gz
drwx------ 2 root root 16384 Aug 30 2019 lost+found
drwxr-xr-x 3 root root 4096 Aug 30 2019 sysdebug

ASA on FP1000:

ciscoasa# dir disk0:/coredumpfsys
Directory of disk0:/coredumpfsys/

1071057 drwx 4096 23:14:58 Aug 30 2019 sysdebug
12 -rw- 87580218 04:49:23 Jun 04 2021 core.lina.27515.1622782155.gz
11 drwx 16384 23:13:37 Aug 30 2019 lost+found

1 file(s) total size: 87580218 bytes
16106127360 bytes total (15749222400 bytes free/97% free)


ciscoasa# connect fxos
[cut]
FP1010# connect local-mgmt
FP1010(local-mgmt)# dir cores

1 87580218 Jun 04 04:49:23 2021 core.lina.27515.1622782155.gz
2 16384 Aug 30 23:13:37 2019 lost+found/
3 4096 Aug 30 23:14:58 2019 sysdebug/

Usage for workspace://
159926181888 bytes total
5209071616 bytes used
154717110272 bytes free

Note: FXOS corefiles are stored under the same cores directory from connecting local-mgmt.

Download Corefiles

There is a copy  command under connect local-mgmt and Lina/ASA CLI. For FTD expert mode, use the scp  command.

Other Things to Check (Specific to Firepower 4100 and 9300 Platforms)

Check the output of the show pmon state command under local-mgmt on FXOS. This example shows the desired output when none of the processes crashed. This output captures not just device-level crashes, but interface module/DME crashes and so on as well.

fp1120-v-1(local-mgmt)# show pmon state

SERVICE NAME             STATE     RETRY(MAX)    EXITCODE    SIGNAL    CORE
------------             -----     ----------    --------    ------    ----
svc_sam_dme            running           0(4)           0         0      no 
svc_sam_dcosAG         running           0(4)           0         0      no 
svc_sam_portAG         running           0(4)           0         0      no 
svc_sam_statsAG        running           0(4)           0         0      no 
httpd.sh               running           0(4)           0         0      no 
svc_sam_sessionmgrAG   running           0(4)           0         0      no 
sam_core_mon           running           0(4)           0         0      no 
svc_sam_svcmonAG       running           0(4)           0         0      no 
svc_sam_serviceOrchAG   running           0(4)           0         0      no 
svc_sam_appAG          running           0(4)           0         0      no 
svc_sam_envAG          running           0(4)           0         0      no

If you do not find any core files in the related FTD/ASA directories, the core files can be present at the bootCLI on 4100/9300.

View Corefiles Inside the Module

Enter this command in order to connect to the module console:

/ssa # connect module 1 console
Telnet escape character is '~'.
Trying 127.5.1.1...
Connected to 127.5.1.1.
Escape character is '~'.
CISCO Serial Over LAN:
Close Network Connection to Exit

Firepower-module1>support filelist
============================
Directory: /
Downloads_Directory
CSP_Downloaded_Files
Archive_Files
Crashinfo_and_Core_Files
Boot_Files
ApplicationLogs
Transient_Core_Files
Type a sub-dir name to list its contents, or [x] to Exit: Transient_Core_Files
-----------files------------
[No files]
([b] to go back)
Type a sub-dir name to list its contents: b
============================
Directory: /
Downloads_Directory
CSP_Downloaded_Files
Archive_Files
Crashinfo_and_Core_Files
Boot_Files
ApplicationLogs
Transient_Core_Files
Type a sub-dir name to list its contents, or [x] to Exit: Crashinfo_and_Core_Files
----------sub-dirs----------
lost+found
-----------files------------
2017-03-20 20:45:06 | 40639151 | core.lina.48857.1490042695.gz
2017-03-20 20:48:47 | 40638054 | core.lina.18113.1490042915.gz
2017-03-20 20:52:28 | 40638186 | core.lina.18112.1490043137.gz
2017-03-20 20:56:10 | 40638466 | core.lina.18123.1490043359.gz
2017-03-20 20:59:53 | 40638345 | core.lina.18262.1490043582.gz
2017-03-20 21:03:35 | 40638120 | core.lina.18476.1490043803.gz
2017-03-20 21:07:22 | 40638335 | core.lina.18529.1490044031.gz ([b] to go back)
Type a sub-dir name to list its contents: b ============================
Directory: /
Downloads_Directory
CSP_Downloaded_Files
Archive_Files
Crashinfo_and_Core_Files
Boot_Files
ApplicationLogs
Transient_Core_Files Type a sub-dir name to list its contents, or [x] to Exit: x
Firepower-module1>

If there are no core files at bootCLI, you can check for logs at the FXOS level:

connect fxos
1(fxos)# show logging onboard obfl-logs
2-(fxos)# show logging onboard stack-trace
3-(fxos)# show logging onboard kernel-trace
4-(fxos)# show logging onboard exception-log
5-(fxos)# show logging onboard internal kernel
6-(fxos)# show logging onboard internal platform
7-(fxos)#show logging onboard internal kernel | no-more
8-(fxos)#show logging onboard internal kernel-big | no-more
9-(fxos)#show logging onboard internal platform | no-more
10-(fxos)#show logging onboard internal reset-reason | no-more

If logging at fxos level is enabled, you can check the logs on fxos. 
It contains the syslog buffer and OBFL logs stored in NVRAM 

Connect fxos
show logging log --------------------This is a non-persistent syslog buffer
show logging onboard oblf-logs ------Non-volatile storage for history of boot up and reset occurrences. Look here when software crashes or reboots, etc are reported.
show logging nvram ------------------Non-volatile storage for critical logs.Important for historical issues.

On FXOS CLI, at the top-level scope use following command.

show fault detail or show fault

If you want to view faults for a specific object, scope to that object and then enter the show fault command.

You can check for audit-logs which is a persistent store of user operations. 

This moreover stores the sequence of user operations done.

firepower# scope security 
firepower# /security # show audit-logs 

Sometimes, the device crashes silently and does not generate any crash or core files. In this case, you can check for the logs:

At FTD instance or device level:
###############################

# Navigate to the /ngfw/var/log or /var/log and open the messages log file. Check all the logs generated before the device rebooted.
 You can search for following  messages (in /ngfw/var/log or /var/log) to confirm if device rebooted without generating crash and core files:

firepower shutdown[2313]: shutting down for system reboot
Stopping Cisco Firepower 2130 Threat Defense
pm:process [INFO] Begin Process Shutdown

# Check for syslogs messsages (specific to device up and down )generated when the  device rebooted. 
You can check for syslogs messages  generated 15-30 min before  and after the device reboot to know if there are some related messages which could have caused the device reboot/crash.

Known Bugs Related to the System Crash

Refer to these pages for additional information on the system crash:

• Cisco bug ID CSCvu84127 - FTD silent crash without generating core or crash file
• Cisco bug ID CSCwa35845 - ASA 5516 reloaded generating core files
• Cisco bug ID CSCvw99444 - FTD crashed with crashinfo/corefile
• Cisco bug ID CSCvv86926 - FTD crashed generating crashfile
• Cisco bug ID CSCvp16482 - ASA crashed generating a core file
• Cisco bug ID CSCvm53545 - ASA can traceback and reload without generating a crashinfo file