Cisco ASA Services Module CLI Configuration Guide, 8.5
Troubleshooting
Downloads: This chapterpdf (PDF - 225.0KB) The complete bookPDF (PDF - 12.85MB) | Feedback

Troubleshooting

Table Of Contents

Troubleshooting

Testing Your Configuration

Enabling ICMP Debugging Messages and Syslog Messages

Pinging ASASM Interfaces

Passing Traffic Through the ASASM

Disabling the Test Configuration

Determining Packet Routing with Traceroute

Tracing Packets with Packet Tracer

Handling TCP Packet Loss

Reloading the ASASM

Performing Password Recovery

Recovering Passwords for the ASASM

Disabling Password Recovery

Resetting the Password on the SSM Hardware Module

Using the ROM Monitor to Load a Software Image

Erasing the Flash File System

Other Troubleshooting Tools

Viewing Debugging Messages

Capturing Packets

Viewing the Crash Dump

Coredump

Monitoring Per-Process CPU Usage

Common Problems


Troubleshooting


This chapter describes how to troubleshoot the ASASM and includes the following sections:

Testing Your Configuration

Reloading the ASASM

Performing Password Recovery

Using the ROM Monitor to Load a Software Image

Erasing the Flash File System

Other Troubleshooting Tools

Common Problems

Testing Your Configuration

This section describes how to test connectivity for the single mode ASASM or for each security context, how to ping the ASASM interfaces, and how to allow hosts on one interface to ping through to hosts on another interface.

We recommend that you only enable pinging and debugging messages during troubleshooting. When you are done testing the ASASM, follow the steps in the "Disabling the Test Configuration" section.

This section includes the following topics:

Enabling ICMP Debugging Messages and Syslog Messages

Pinging ASASM Interfaces

Passing Traffic Through the ASASM

Disabling the Test Configuration

Determining Packet Routing with Traceroute

Tracing Packets with Packet Tracer

Handling TCP Packet Loss

Enabling ICMP Debugging Messages and Syslog Messages

Debugging messages and syslog messages can help you troubleshoot why your pings are not successful. The ASASM only shows ICMP debugging messages for pings to the ASASM interfaces, and not for pings through the ASASM to other hosts. To enable debugging and syslog messages, perform the following steps:

 
Command
Purpose

Step 1 

debug icmp trace
Example:
hostname(config)# debug icmp trace

Shows ICMP packet information for pings to the ASASM interfaces.

Step 2 

logging monitor debug
Example:
hostname(config)# logging monitor debug

Sets syslog messages to be sent to Telnet or SSH sessions.


Note You can alternately use the logging buffer debug command to send log messages to a buffer, and then view them later using the show logging command.


Step 3 

terminal monitor
Example:
hostname(config)# terminal monitor

Sends the syslog messages to a Telnet or SSH session.

Step 4 

logging on
Example:
hostname(config)# logging on

Enables syslog message generation.

To enable ICMP inspection to the default global policy, perform the following steps:

 
Command
Purpose

Step 1 

policy-map name
Example:

hostname(config)# policy-map global_policy

Configures the policy map and attach the action to the class of traffic.

Step 2 

class classmap_name
Example:

hostname(config-pmap)# class inspection_default

Assigns a class map to the policy map so that you can assign actions to the class map traffic.

Step 3 

inspect icmp
Example:
hostname(config)# inspect icmp

Enables ICMP inspection.

Examples

The following example shows a successful ping from an external host (209.165.201.2) to the ASASM outside interface (209.165.201.1):

hostname(config)# debug icmp trace
Inbound ICMP echo reply (len 32 id 1 seq 256) 209.165.201.1 > 209.165.201.2
Outbound ICMP echo request (len 32 id 1 seq 512) 209.165.201.2 > 209.165.201.1
Inbound ICMP echo reply (len 32 id 1 seq 512) 209.165.201.1 > 209.165.201.2
Outbound ICMP echo request (len 32 id 1 seq 768) 209.165.201.2 > 209.165.201.1
Inbound ICMP echo reply (len 32 id 1 seq 768) 209.165.201.1 > 209.165.201.2
Outbound ICMP echo request (len 32 id 1 seq 1024) 209.165.201.2 > 209.165.201.1
Inbound ICMP echo reply (len 32 id 1 seq 1024) 209.165.201.1 > 209.165.201.2
 
   

The output shows the ICMP packet length (32 bytes), the ICMP packet identifier (1), and the ICMP sequence number (the ICMP sequence number starts at 0, and is incremented each time that a request is sent).

Pinging ASASM Interfaces

To test whether the ASASM interfaces are up and running and that the ASASM and connected routers are operating correctly, you can ping the ASASM interfaces. To ping the ASASM interfaces, perform the following steps:


Step 1 Draw a diagram of your single-mode ASASM or security context that shows the interface names, security levels, and IP addresses.


Note Although this procedure uses IP addresses, the ping command also supports DNS names and names that are assigned to a local IP address with the name command.


The diagram should also include any directly connected routers and a host on the other side of the router from which you will ping the ASASM. You will use this information in this procedure and in the procedure in the "Passing Traffic Through the ASASM" section. (See Figure 57-1.)

Figure 57-1 Network Diagram with Interfaces, Routers, and Hosts

Step 2 Ping each ASASM interface from the directly connected routers. For transparent mode, ping the management IP address. This test ensures that the ASASM interfaces are active and that the interface configuration is correct.

A ping might fail if the ASASM interface is not active, the interface configuration is incorrect, or if a switch between the ASASM and a router is down (see Figure 57-2). In this case, no debug messages or syslog messages appear, because the packet never reaches the ASASM.

Figure 57-2 Ping Failure at the ASASM Interface

If the ping reaches the ASASM, and it responds, debugging messages similar to the following appear:

ICMP echo reply (len 32 id 1 seq 256) 209.165.201.1 > 209.165.201.2
ICMP echo request (len 32 id 1 seq 512) 209.165.201.2 > 209.165.201.1
 
   

If the ping reply does not return to the router, then a switch loop or redundant IP addresses may exist (see Figure 57-3).

Figure 57-3 Ping Failure Because of IP Addressing Problems

Step 3 Ping each ASASM interface from a remote host. For transparent mode, ping the management IP address. This test checks whether the directly connected router can route the packet between the host and the ASASM, and whether the ASASM can correctly route the packet back to the host.

A ping might fail if the ASASM does not have a return route to the host through the intermediate router (see Figure 57-4). In this case, the debugging messages show that the ping was successful, but syslog message 110001 appears, indicating a routing failure.

Figure 57-4 Ping Failure Because the Security Appliance has No Return Route


Passing Traffic Through the ASASM

After you successfully ping the ASASM interfaces, make sure that traffic can pass successfully through the ASASM. For routed mode, this test shows that NAT is operating correctly, if configured. For transparent mode, which does not use NAT, this test confirms that the ASASM is operating correctly. If the ping fails in transparent mode, contact the Cisco TAC.

To ping between hosts on different interfaces, perform the following steps:

 
Command
Purpose

Step 1 

access-list ICMPACL extended permit icmp 
any any
Example:
hostname(config)# access-list ICMPACL 
extended permit icmp any any

Adds an access list to allow ICMP traffic from any source host.


Note By default, when hosts access a lower security interface, all traffic is allowed through. However, to access a higher security interface, you need the preceding access list.


Step 2 

access-group ICMPACL in interface 
interface_name
Example:
hostname(config)# access-group ICMPACL in 
interface inside

Assigns the access list to each source interface. Repeat this command for each source interface.

Step 3 

class-map ICMP-CLASS
match access-list ICMPACL
policy-map ICMP-POLICY
class ICMP-CLASS
inspect icmp
service-policy ICMP-POLICY global
Example:
hostname(config)# class-map ICMP-CLASS
hostname(config-cmap)# match access-list 
ICMPACL
hostname(config)# policy-map ICMP-POLICY
hostname(config-pmap)# class ICMP-CLASS
hostname(config-pmap)# inspect icmp
hostname(config)# service-policy 
ICMP-POLICY global

Enables the ICMP inspection engine and ensures that ICMP responses may return to the source host.

For a host to access a lower security interface, you must enable ICMP inspection. However, to access a higher security interface, you must enable ICMP inspection and the preceding access list.


Note Alternatively, you can also apply the ICMP access list to the destination interface to allow ICMP traffic back through the ASASM.


Step 4 

logging on
Example:
hostname(config)# logging on

Enables syslog message generation.

If the ping succeeds, a syslog message appears to confirm the address translation for routed mode (305009 or 305011) and that an ICMP connection was established (302020). You can also enter either the show xlate or show conns command to view this information.

If the ping fails for transparent mode, contact Cisco TAC.

For routed mode, the ping might fail because NAT is not configured correctly (see Figure 57-5). In this case, a syslog message appears, showing that the NAT failed (305005 or 305006). If the ping is from an outside host to an inside host, and you do not have a static translation, the following syslog message appears:

%ASA-3-106010: deny inbound icmp.

Note The ASASM only shows ICMP debugging messages for pings to the ASASM interfaces, and not for pings through the ASASM to other hosts.

Figure 57-5 Ping Failure Because the ASASM is Not Translating Addresses

Disabling the Test Configuration

After you complete your testing, disable the test configuration that allows ICMP to and through the ASASM and that prints debugging messages. If you leave this configuration in place, it can pose a serious security risk. Debugging messages also slow the ASASM performance.

To disable the test configuration, perform the following steps:

 
Command
Purpose

Step 1 

no debug icmp trace
Example:

hostname (config)# no debug icmp trace

Disables ICMP debugging messages.

Step 2 

no logging on
Example:
hostname (config)# no 
logging on

Disables logging.

Step 3 

no access-list ICMPACL
Example:
hostname (config)# no 
access-list ICMPACL

Removes the ICMPACL access list, and deletes the related access-group commands.

Step 4 

no service-policy 
ICMP-POLICY
Example:
hostname (config)# no 
service-policy ICMP-POLICY

(Optional) Disables the ICMP inspection engine.

Determining Packet Routing with Traceroute

You can trace the route of a packet using the traceroute feature, which is accessed with the traceroute command. A traceroute works by sending UDP packets to a destination on an invalid port. Because the port is not valid, the routers along the way to the destination respond with an ICMP Time Exceeded Message, and report that error to the ASASM.

Tracing Packets with Packet Tracer

The packet tracer tool provides packet tracing for packet sniffing and network fault isolation, as well as detailed information about the packets and how they are processed by the ASASM. If a configuration command did not cause the packet to drop, the packet tracer tool provides information about the cause in an easily readable manner.

In addition, you can trace the lifespan of a packet through the ASASM to see whether the packet is operating correctly with the packet tracer tool. This tool enables you to do the following:

Debug all packet drops in a production network.

Verify the configuration is working as intended.

Show all rules applicable to a packet, along with the CLI commands that caused the rule addition.

Show a time line of packet changes in a data path.

Inject tracer packets into the data path.

Search for an IPv4 or IPv6 address based on the user identity and the FQDN.

To trace packets, enter the following command:

Command
Purpose
packet-tracer input [ifc_name] [icmp [sip | user 
username | fqdn fqdn-string] type code ident [dip | 
fqdn fqdn-string]] | [tcp [sip | user username | 
fqdn fqdn-string] sport [dip | fqdn fqdn-string] 
dport] | [udp [sip | user username | fqdn fqdn- 
string] sport [dip | fqdn fqdn-string] dport] | 
[rawip [sip | user username | fqdn fqdn-string] [dip 
| fqdn fqdn-string]] [detailed] [xml]
Example:

hostname# packet-tracer input inside tcp 10.2.25.3 www 209.165.202.158 aol detailed

Provides detailed information about the packets and how they are processed by the ASASM. The example shows how to enable packet tracing from inside host 10.2.25.3 to external host 209.165.202.158, including detailed information.

Handling TCP Packet Loss

To troubleshoot TCP packet loss, see the "Customizing the TCP Normalizer with a TCP Map" section for more information.

Reloading the ASASM

To reload the ASASM, enter the following command:

Command
Purpose
reload
Example:
hostname (config)# reload

Restarts the ASASM.

Note In multiple mode, you can only reload from the system execution space.


Performing Password Recovery

This section describes how to recover passwords if you have forgotten them or you are locked out because of AAA settings, and how to disable password recovery for extra security. This section includes the following topics:

Recovering Passwords for the ASASM

Disabling Password Recovery

Resetting the Password on the SSM Hardware Module

Recovering Passwords for the ASASM

To recover passwords for the ASASM, perform the following steps:


Step 1 Connect to the ASASM console port according to the instructions in "Accessing the ASA Services Module Command-Line Interface" section.

Step 2 Power off the ASASM, and then power it on.

Step 3 After startup, press the Escape key when you are prompted to enter ROMMON mode.

Step 4 To update the configuration register value, enter the following command:

rommon #1> confreg 0x41
Update Config Register (0x41) in NVRAM...
 
   

Step 5 To set the ASASM to ignore the startup configuration, enter the following command:

rommon #1> confreg
 
   

The ASASM displays the current configuration register value, and asks whether you want to change it:

Current Configuration Register: 0x00000041
Configuration Summary: 
  boot default image from Flash
  ignore system configuration
 
   
Do you wish to change this configuration? y/n [n]: y
 
   

Step 6 Record the current configuration register value, so you can restore it later.

Step 7 At the prompt, enter Y to change the value.

The ASASM prompts you for new values.

Step 8 Accept the default values for all settings. At the prompt, enter Y.

Step 9 Reload the ASASM by entering the following command:

rommon #2> boot
Launching BootLoader...
Boot configuration file contains 1 entry.
 
   
Loading disk0:/asa800-226-k8.bin... Booting...Loading...
 
   

The ASASM loads the default configuration instead of the startup configuration.

Step 10 Access the privileged EXEC mode by entering the following command:

hostname# enable
 
   

Step 11 When prompted for the password, press Enter.

The password is blank.

Step 12 Load the startup configuration by entering the following command:

hostname# copy startup-config running-config
 
   

Step 13 Access the global configuration mode by entering the following command:

hostname# configure terminal
 
   

Step 14 Change the passwords, as required, in the default configuration by entering the following commands:

hostname(config)# password password
hostname(config)# enable password password
hostname(config)# username name password password
 
   

Step 15 Load the default configuration by entering the following command:

hostname(config)# no config-register 
 
   

The default configuration register value is 0x1. For more information about the configuration register, see the command reference.

Step 16 Save the new passwords to the startup configuration by entering the following command:

hostname(config)# copy running-config startup-config
 
   

Disabling Password Recovery

To disable password recovery to ensure that unauthorized users cannot use the password recovery mechanism to compromise the ASASM, enter the following command:

Command
Purpose
no service password-recovery
Example:

hostname (config)# no service password-recovery

Disables password recovery.


On the ASASM, the no service password-recovery command prevents you from entering ROMMON mode with the configuration intact. When you enter ROMMON mode, the ASASM prompts you to erase all Flash file systems. You cannot enter ROMMON mode without first performing this erasure. If you choose not to erase the Flash file system, the ASASM reloads. Because password recovery depends on using ROMMON mode and maintaining the existing configuration, this erasure prevents you from recovering a password. However, disabling password recovery prevents unauthorized users from viewing the configuration or inserting different passwords. In this case, to restore the system to an operating state, load a new image and a backup configuration file, if available.

The service password-recovery command appears in the configuration file for information only. When you enter the command at the CLI prompt, the setting is saved in NVRAM. The only way to change the setting is to enter the command at the CLI prompt. Loading a new configuration with a different version of the command does not change the setting. If you disable password recovery when the ASASM is configured to ignore the startup configuration at startup (in preparation for password recovery), then the ASASM changes the setting to load the startup configuration as usual. If you use failover, and the standby unit is configured to ignore the startup configuration, then the same change is made to the configuration register when the no service password recovery command replicates to the standby unit.

Resetting the Password on the SSM Hardware Module

To reset the password to the default of "cisco" on the SSM hardware module, enter the following command:


Note Make sure that the SSM hardware module is in the Up state and supports password reset.


Command
Purpose
hw-module module 1 
password-reset
Reset the password on 
module in slot 1? [confirm] 
y
hostname# y

Where 1 is the specified slot number on the SSM hardware module.


Note On the AIP SSM, entering this command reboots the hardware module. The module is offline until the rebooting is finished. Enter the show module command to monitor the module status. The AIP SSM supports this command in version 6.0 and later.

On the CSC SSM, entering this command resets web services on the hardware module after the password has been reset. You may lose connection to ASDM or be logged out of the hardware module. The CSC SSM supports this command in the most recent version of 6.3, dated January 2010, and later releases.


Using the ROM Monitor to Load a Software Image

To load a software image to an ASASM from the ROM monitor mode using TFTP, perform the following steps:


Step 1 Connect to the ASASM console port according to the instructions in the "Accessing the ASA Services Module Command-Line Interface" section.

Step 2 Make sure that you reload the ASASM image.

Step 3 During startup, press the Escape key when you are prompted to enter ROMMON mode.

Step 4 In ROMMOM mode, define the interface settings to the ASASM, including the IP address, TFTP server address, gateway address, software image file, port, and VLAN, as follows:

rommon #1> ADDRESS=172.16.145.149
rommon #2> SERVER=172.16.171.125
rommon #3> GATEWAY=172.16.145.129
rommon #4> IMAGE=f1/asa851-smp-k8.bin
rommon #5> PORT=Data0
rommon #6> VLAN=1
Data0
Link is UP
MAC Address: 0012.d949.15b8

Note Be sure that the connection to the network already exists.


Step 5 To validate your settings, enter the set command.

rommon #7> set
ROMMON Variable Settings:
  ADDRESS=172.16.145.149
  SERVER=172.16.171.125
  GATEWAY=172.16.145.129
  PORT=Data0
  VLAN=1
  IMAGE=f1/asa851-smp-k8.bin
  CONFIG=
  LINKTIMEOUT=20
  PKTTIMEOUT=2
  RETRY=20
 
   

Step 6 Ping the TFTP server by entering the ping server command.

rommon #8> ping server
Sending 20, 100-byte ICMP Echoes to server 172.16.171.125, timeout is 2 seconds:
 
   
Success rate is 100 percent (20/20)
 
   

Step 7 Load the software image by entering the tftp command.

rommon #9> tftp
Clearing EOBC receive queue ...
cmostime_set = 1
ROMMON Variable Settings:
  ADDRESS=172.16.145.149
  SERVER=172.16.171.125
  GATEWAY=172.16.145.129
  PORT=Data0
  VLAN=1
  IMAGE=f1/asa851-smp-k8.bin
  CONFIG=
  LINKTIMEOUT=20
  PKTTIMEOUT=2
  RETRY=20
 
   
tftp f1/asa851-smp-k8.bin@172.16.171.125 via 172.16.145.129
Starting download. Press ESC to abort.
 
   

After the software image is successfully loaded, the ASASM automatically exits ROMMOM mode.


Note You must download the image to the system flash separately after ROMMON boot is complete; booting the module into ROMMON mode does not preserve the system image across reloads.


Step 8 To verify that the correct software image has been loaded into the ASASM, check the version by entering the following command:

hostname# show version
 
   

Erasing the Flash File System

To erase the flash file system, perform the following steps:


Step 1 Connect to the ASASM console port according to the instructions in "Accessing the ASA Services Module Command-Line Interface" section.

Step 2 Power off the ASASM, then power it on.

Step 3 During startup, press the Escape key when you are prompted to enter ROMMON mode.

Step 4 Enter the erase command, which overwrites all files and erases the file system, including hidden system files.

rommon #1> erase [ disk0: | disk1: | flash:]

Other Troubleshooting Tools

The ASASM provides other troubleshooting tools that you can use. This section includes the following topics:

Viewing Debugging Messages

Capturing Packets

Viewing the Crash Dump

Coredump

Monitoring Per-Process CPU Usage

Viewing Debugging Messages

Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco TAC. Moreover, it is best to use debug commands during periods of less network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use. To enable debugging messages, see the debug commands in the command reference.

Capturing Packets

Capturing packets is sometimes useful when troubleshooting connectivity problems or monitoring suspicious activity. We recommend that you contact the Cisco TAC if you want to use the packet capture feature. See the capture command in the command reference.

Viewing the Crash Dump

If the ASASM crashes, you can view the crash dump information. We recommend contacting Cisco TAC if you want to interpret the crash dump. See the show crashdump command in the command reference.

Coredump

A coredump is a snapshot of the running program when the program has terminated abnormally, or crashed. Coredumps are used to diagnose or debug errors and save a crash for future off-site analysis. Cisco TAC may request that users enable the coredump feature to troubleshoot application or system crashes on the ASASM. See the coredump command in the command reference.

Monitoring Per-Process CPU Usage

You can monitor the processes that run on the CPU. You can obtain information about the percentage of CPU that is used by a certain process. CPU usage statistics are sorted in descending order to display the highest consumer at the top. Also included is information about the load on the CPU per process, at 5 seconds, 1 minute, and 5 minutes before the log time. This information is updated automatically every 5 seconds to provide real-time statistics.You can use the show process cpu-usage sorted command to find a breakdown of the process-related load-to-CPU that is consumed by any configured contexts.

Common Problems

This section describes common problems with the ASASM, and how you might resolve them.

Symptom    The context configuration was not saved, and was lost when you reloaded.

Possible Cause    You did not save each context within the context execution space. If you are configuring contexts at the command line, you did not save the current context before you changed to the next context.

Recommended Action    Save each context within the context execution space using the copy start run command. Load the startup configuration as your active configuration. Then change the password and then enter the copy run start command. You cannot save contexts from the system execution space.

Symptom    You cannot make a Telnet or SSH connection to the ASASM interface.

Possible Cause    You did not enable Telnet or SSH to the ASASM.

Recommended Action    Enable Telnet or SSH to the ASASM according to the instructions in the "Configuring ASA Access for ASDM, Telnet, or SSH" section.

Symptom    You cannot ping the ASASM interface.

Possible Cause    You disabled ICMP to the ASASM.

Recommended Action    Enable ICMP to the ASASM for your IP address using the icmp command.

Symptom    You cannot ping through the ASASM, although the access list allows it.

Possible Cause    You did not enable the ICMP inspection engine or apply access lists on both the ingress and egress interfaces.

Recommended Action    Because ICMP is a connectionless protocol, the ASASM does not automatically allow returning traffic through. In addition to an access list on the ingress interface, you either need to apply an access list to the egress interface to allow replying traffic, or enable the ICMP inspection engine, which treats ICMP connections as stateful connections.

Symptom    Traffic does not pass between two interfaces on the same security level.

Possible Cause    You did not enable the feature that allows traffic to pass between interfaces at the same security level.

Recommended Action    Enable this feature according to the instructions in the "Allowing Same Security Level Communication" section.

Symptom    IPsec tunnels do not duplicate during a failover to the standby device.

Possible Cause    The switch port that the ASASM is plugged into is set to 10/100 instead of 1000.

Recommended Action    Set the switch port that the ASASM is plugged into to 1000.