Cisco Security Appliance Command Line Configuration Guide, Version 7.0
Monitoring and Troubleshooting
Downloads: This chapterpdf (PDF - 176.0KB) The complete bookPDF (PDF - 15.34MB) | Feedback

Monitoring and Troubleshooting

Table Of Contents

Monitoring and Troubleshooting

Monitoring the Security Appliance

Using System Log Messages 

Using SNMP

SNMP Overview

Enabling SNMP

Troubleshooting the Security Appliance

Testing Your Configuration

Enabling ICMP Debug Messages and System Messages

Pinging Security Appliance Interfaces

Pinging Through the Security Appliance

Disabling the Test Configuration

Reloading the Security Appliance

Performing Password Recovery

Performing Password Recovery for the ASA 5500 Series Adaptive Security Appliance

Password Recovery for the PIX 500 Series Security Appliance

Disabling Password Recovery

Other Troubleshooting Tools

Viewing Debug Messages

Capturing Packets

Viewing the Crash Dump

Common Problems


Monitoring and Troubleshooting


This chapter describes how to monitor and troubleshoot the security appliance, and includes the following sections:

Monitoring the Security Appliance

Troubleshooting the Security Appliance

Monitoring the Security Appliance

This section describes how to monitor the security appliance, and includes the following topics:

Using System Log Messages

Using SNMP

Using System Log Messages 

The security appliance provides extensive system log messages. See the Cisco Security Appliance Logging Configuration and System Log Messages to configure logging and to view system log message descriptions.

Using SNMP

This section describes how to use SNMP and includes the following topics:

SNMP Overview

Enabling SNMP

SNMP Overview

The security appliance provides support for network monitoring using SNMP V1 and V2c. The security appliance supports traps and SNMP read access, but does not support SNMP write access.

You can configure the security appliance to send traps (event notifications) to a network management station (NMS), or you can use the NMS to browse the MIBs on the security appliance. MIBs are a collection of definitions, and the security appliance maintains a database of values for each definition. Browsing a MIB entails issuing an SNMP get request from the NMS. Use CiscoWorks for Windows or any other SNMP V1, MIB-II compliant browser to receive SNMP traps and browse a MIB.

Table 33-1 lists supported MIBs and traps for the security appliance and, in multiple mode, for each context. You can download Cisco MIBs from the following website.

http://www.cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml

After you download the MIBs, compile them for your NMS.

Table 33-1 SNMP MIB and Trap Support 

MIB or Trap Support
Description

SNMP core traps

The security appliance sends the following core SNMP traps:

authentication—An SNMP request fails because the NMS did not authenticate with the correct community string.

linkup—An interface has transitioned to the "up" state.

linkdown—An interface is down, for example, if you removed the nameif command.

coldstart—The security appliance is running after a reload.

MIB-II

The security appliance supports browsing of the following groups and tables:

system

IF-MIB

The security appliance supports browsing of the following tables:

ifTable

ifXTable

RFC1213-MIB

The security appliance supports browsing of the following table:

ip.ipAddrTable

SNMPv2-MIB

The security appliance supports browsing the following:

snmp

ENTITY-MIB

The security appliance supports browsing of the following groups and tables:

entPhysicalTable

entLogicalTable

The security appliance supports browsing of the following traps:

snmp-server enable traps entity {config-change|fru-insert|fru-remove}

CISCO-IPSEC-FLOW-MONITOR-MIB

The security appliance supports browsing of the MIB.

The security appliance supports browsing of the following traps:

snmp-server enable traps ipsec {start|stop}

CISCO-REMOTE-ACCESS-MONITOR-MIB

The security appliance supports browsing of the MIB.

The security appliance supports browsing of the following traps:

snmp-server enable traps remote-access {session-threshold-exceeded}

CISCO-CRYPTO-ACCELERATOR-MIB

The security appliance supports browsing of the MIB.

ALTIGA-GLOBAL-REG

The security appliance supports browsing of the MIB.

Cisco Firewall MIB

The security appliance supports browsing of the following groups:

cfwSystem

The information is cfwSystem.cfwStatus, which relates to failover status, pertains to the entire device and not just a single context.

Cisco Memory Pool MIB

The security appliance supports browsing of the following table:

ciscoMemoryPoolTable—The memory usage described in this table applies only to the security appliance general-purpose processor, and not to the network processors.

Cisco Process MIB

The security appliance supports browsing of the following table:

cpmCPUTotalTable

Cisco Syslog MIB

The security appliance supports the following trap:

clogMessageGenerated

You cannot browse this MIB.


Enabling SNMP

The SNMP agent that runs on the security appliance performs two functions:

Replies to SNMP requests from NMSs.

Sends traps (event notifications) to NMSs.

To enable the SNMP agent and identify an NMS that can connect to the security appliance, follow these steps:


Step 1 To identify the IP address of the NMS that can connect to the security appliance, enter the following command:

hostname(config)# snmp-server host interface_name ip_address [trap | poll] [community 
text] [version 1 | 2c] [udp-port port]

Specify trap or poll if you want to limit the NMS to receiving traps only or browsing (polling) only. By default, the NMS can use both functions.

SNMP traps are sent on UDP port 162 by default. You can change the port number using the udp-port keyword.

Step 2 To specify the community string, enter the following command:

hostname(config)# snmp-server community key

The SNMP community string is a shared secret between the security appliance and the NMS. The key is a case-sensitive value up to 32 characters in length. Spaces are not permitted.

Step 3 (Optional) To set the SNMP server location or contact information, enter the following command:

hostname(config)# snmp-server {contact | location} text

Step 4 To enable the security appliance to send traps to the NMS, enter the following command:

hostname(config)# snmp-server enable [traps [all | feature [trap1] [trap2]] [...]]

By default, SNMP core traps are enabled (snmp). If you do not enter a trap type in the command, syslog is the default. To enable or disable all traps, enter the all option. For snmp, you can identify each trap type separately. See Table 33-1 for a list of traps.

Step 5 To enable system messages to be sent as traps to the NMS, enter the following command:

hostname(config)# logging history level

You must also enable syslog traps using the preceding snmp-server enable traps command.

Step 6 To enable logging, so system messages are generated and can then be sent to an NMS, enter the following command:

hostname(config)# logging on


The following example sets the security appliance to receive requests from host 192.168.3.2 on the inside interface.

hostname(config)# snmp-server host 192.168.3.2
hostname(config)# snmp-server location building 42
hostname(config)# snmp-server contact Pat lee
hostname(config)# snmp-server community ohwhatakeyisthee

Troubleshooting the Security Appliance

This section describes how to troubleshoot the security appliance, and includes the following topics:

Testing Your Configuration

Reloading the Security Appliance

Performing Password Recovery

Other Troubleshooting Tools

Common Problems

Testing Your Configuration

This section describes how to test connectivity for the single mode security appliance or for each security context. The following steps describe how to ping the security appliance interfaces, and how to allow hosts on one interface to ping through to hosts on another interface.

We recommend that you only enable pinging and debug messages during troubleshooting. When you are done testing the security appliance, follow the steps in the "Disabling the Test Configuration" section.

This section includes:

Enabling ICMP Debug Messages and System Messages

Pinging Security Appliance Interfaces

Pinging Through the Security Appliance

Disabling the Test Configuration

Enabling ICMP Debug Messages and System Messages

Debug messages and system messages can help you troubleshoot why your pings are not successful. The security appliance only shows ICMP debug messages for pings to the security appliance interfaces, and not for pings through the security appliance to other hosts. To enable debugging and system messages, perform the following steps:


Step 1 To show ICMP packet information for pings to the security appliance interfaces, enter the following command:

hostname(config)# debug icmp trace

Step 2 To set system messages to be sent to Telnet or SSH sessions, enter the following command:

hostname(config)# logging monitor debug

You can alternately use logging buffer debug to send messages to a buffer, and then view them later using the show logging command.

Step 3 To send the system messages to your Telnet or SSH session, enter the following command:

hostname(config)# terminal monitor

Step 4 To enable system messages, enter the following command:

hostname(config)# logging on


The following example shows a successful ping from an external host (209.165.201.2) to the security appliance outside interface (209.165.201.1):

hostname(config)# debug icmp trace
Inbound ICMP echo reply (len 32 id 1 seq 256) 209.165.201.1 > 209.165.201.2
Outbound ICMP echo request (len 32 id 1 seq 512) 209.165.201.2 > 209.165.201.1
Inbound ICMP echo reply (len 32 id 1 seq 512) 209.165.201.1 > 209.165.201.2
Outbound ICMP echo request (len 32 id 1 seq 768) 209.165.201.2 > 209.165.201.1
Inbound ICMP echo reply (len 32 id 1 seq 768) 209.165.201.1 > 209.165.201.2
Outbound ICMP echo request (len 32 id 1 seq 1024) 209.165.201.2 > 209.165.201.1
Inbound ICMP echo reply (len 32 id 1 seq 1024) 209.165.201.1 > 209.165.201.2

The preceding example shows the ICMP packet length (32 bytes), the ICMP packet identifier (1), and the ICMP sequence number (the ICMP sequence number starts at 0 and is incremented each time a request is sent).

Pinging Security Appliance Interfaces

To test that the security appliance interfaces are up and running and that the security appliance and connected routers are routing correctly, you can ping the security appliance interfaces.


Note For security purposes the security appliance does not support far-end interface ping, that is pinging the IP address of the outside interface from the inside network.


To ping the security appliance interfaces, perform the following steps:


Step 1 Create a sketch of your single mode security appliance or security context showing the interface names, security levels, and IP addresses. The sketch should also include any directly connected routers, and a host on the other side of the router from which you will ping the security appliance. You will use this information for this procedure as well as the procedure in the "Pinging Through the Security Appliance" section. For example:

Figure 33-1 Network Sketch with Interfaces, Routers, and Hosts

Step 2 Ping each security appliance interface from the directly connected routers. For transparent mode, ping the management IP address.

This test ensures that the security appliance interfaces are active and that the interface configuration is correct.

A ping might fail if the security appliance interface is not active, the interface configuration is incorrect, or if a switch between the security appliance and router is down (see Figure 33-2). In this case, no debug messages or system messages appear on the security appliance, because the packet never reaches it.

Figure 33-2 Ping Failure at Security Appliance Interface

If the ping reaches the security appliance, and the security appliance responds, you see debug messages like the following:

ICMP echo reply (len 32 id 1 seq 256) 209.165.201.1 > 209.165.201.2
ICMP echo request (len 32 id 1 seq 512) 209.165.201.2 > 209.165.201.1

If the ping reply does not return to the router, then you might have a switch loop or redundant IP addresses (see Figure 33-3).

Figure 33-3 Ping Failure Because of IP Addressing Problems

Step 3 Ping each security appliance interface from a remote host. For transparent mode, ping the management IP address.

This test checks that the directly connected router can route the packet between the host and the security appliance, and that the security appliance can correctly route the packet back to the host.

A ping might fail if the security appliance does not have a route back to the host through the intermediate router (see Figure 33-4). In this case, the debug messages show that the ping was successful, but you see system message 110001 indicating a routing failure.

Figure 33-4 Ping Failure Because the Security Appliance has no Route


Pinging Through the Security Appliance

After you successfully ping the security appliance interfaces, you should make sure traffic can pass successfully through the security appliance. For routed mode, this test shows that NAT is working correctly, if configured. For transparent mode, which does not use NAT, this test confirms that the security appliance is operating correctly; if the ping fails in transparent mode, contact Cisco TAC.

To ping between hosts on different interfaces, perform the following steps:


Step 1 To add an access list allowing ICMP from any source host, enter the following command:

hostname(config)# access-list ICMPACL extended permit icmp any any

By default, when hosts access a lower security interface, all traffic is allowed through. However, to access a higher security interface, you need the preceding access list.

Step 2 To assign the access list to each source interface, enter the following command:

hostname(config)# access-group ICMPACL in interface interface_name

Repeat this command for each source interface.

Step 3 To enable the ICMP inspection engine, so ICMP responses are allowed back to the source host, enter the following commands:

hostname(config)# class-map ICMP-CLASS
hostname(config-cmap)# match access-list ICMPACL
hostname(config-cmap)# policy-map ICMP-POLICY
hostname(config-pmap)# class ICMP-CLASS
hostname(config-pmap-c)# inspect icmp
hostname(config-pmap-c)# service-policy ICMP-POLICY global

Alternatively, you can also apply the ICMPACL access list to the destination interface to allow ICMP traffic back through the security appliance.

Step 4 Ping from the host or router through the source interface to another host or router on another interface.

Repeat this step for as many interface pairs as you want to check.

If the ping succeeds, you see a system message confirming the address translation for routed mode (305009 or 305011) and that an ICMP connection was established (302020). You can also enter the show xlate and show conns commands to view this information.

If the ping fails for transparent mode, contact Cisco TAC.

For routed mode, the ping might fail because NAT is not configured correctly (see Figure 33-5). This is more likely if you enable NAT control. In this case, you see a system message showing that the NAT translation failed (305005 or 305006). If the ping is from an outside host to an inside host, and you do not have a static translation (which is required with NAT control), you see message 106010: deny inbound icmp.


Note The security appliance only shows ICMP debug messages for pings to the security appliance interfaces, and not for pings through the security appliance to other hosts.


Figure 33-5 Ping Failure Because the Security Appliance is not Translating Addresses


Disabling the Test Configuration

After you complete your testing, disable the test configuration that allows ICMP to and through the security appliance and that prints debug messages. If you leave this configuration in place, it can pose a serious security risk. Debug messages also slow the security appliance performance.

To disable the test configuration, perform the following steps:


Step 1 To disable ICMP debug messages, enter the following command:

hostname(config)# no debug icmp trace

Step 2 To disable logging, if desired, enter the following command:

hostname(config)# no logging on

Step 3 To remove the ICMPACL access list, and also delete the related access-group commands, enter the following command:

hostname(config)# no access-list ICMPACL

Step 4 (Optional) To disable the ICMP inspection engine, enter the following command:

hostname(config)# no service-policy ICMP-POLICY


Reloading the Security Appliance

In multiple mode, you can only reload from the system execution space. To reload the security appliance, enter the following command:

hostname# reload

Performing Password Recovery

This section describes how to recover if you forget passwords, or you create a lockout situation because of AAA settings. You can also disable password recovery for extra security. This section includes the following topics:

Performing Password Recovery for the ASA 5500 Series Adaptive Security Appliance

Password Recovery for the PIX 500 Series Security Appliance

Disabling Password Recovery

Performing Password Recovery for the ASA 5500 Series Adaptive Security Appliance

To recover from the loss of passwords, perform the following steps:


Step 1 Connect to the security appliance console port according to the "Accessing the Command-Line Interface" section on page 2-1.

Step 2 Power off the security appliance, and then power it on.

Step 3 During the startup messages, press the Escape key when prompted to enter ROMMON.

Step 4 To set the security appliance to ignore the startup configuration at reload, enter the following command:

rommon #1> confreg

The security appliance displays the current configuration register value, and asks if you want to change the value:

Current Configuration Register: 0x00000011
Configuration Summary:
  boot TFTP image, boot default image from Flash on netboot failure
Do you wish to change this configuration? y/n [n]:

Step 5 Record your current configuration register value, so you can restore it later.

Step 6 At the prompt, enter Y to change the value.

The security appliance prompts you for new values.

Step 7 Accept the default values for all settings, except for the "disable system configuration?" value; at that prompt, enter Y.

Step 8 Reload the security appliance by entering the following command:

rommon #2> boot

The security appliance loads a default configuration instead of the startup configuration.

Step 9 Enter privileged EXEC mode by entering the following command:

hostname> enable

Step 10 When prompted for the password, press Return.

The password is blank.

Step 11 Load the startup configuration by entering the following command:

hostname# copy startup-config running-config

Step 12 Enter global configuration mode by entering the following command:

hostname# configure terminal

Step 13 Change the passwords in the configuration by entering the following commands, as necessary:

hostname(config)# password password
hostname(config)# enable password password
hostname(config)# username name password password

Step 14 Change the configuration register to load the startup configuration at the next reload by entering the following command:

hostname(config)# config-register value

Where value is the configuration register value you noted in Step 5. 0x1 is the default configuration register. For more information about the configuration register, see the Cisco Security Appliance Command Reference.

Step 15 Save the new passwords to the startup configuration by entering the following command:

hostname(config)# copy running-config startup-config


Password Recovery for the PIX 500 Series Security Appliance

Performing password recovery on the security appliance erases the login password, enable password, and aaa authentication console commands. To erase these commands so you can log in with the default passwords, perform the following steps:


Step 1 Download the PIX password tool from Cisco.com to a TFTP server accessible from the security appliance. See the link in the "Password Recovery Procedure for the PIX" document at the following URL:


Step 2 Connect to the security appliance console port according to the "Accessing the Command-Line Interface" section on page 2-1.

Step 3 Power off the security appliance, and then power it on.

Step 4 Immediately after the startup messages appear, press the Escape key to enter monitor mode.

Step 5 Configure the network settings for the interface that accesses the TFTP server by entering the following commands:

monitor> interface interface_id
monitor> address interface_ip
monitor> server tftp_ip
monitor> file pw_tool_name
monitor> gateway gateway_ip

Step 6 Download the PIX password tool from the TFTP server by entering the following command:

monitor> tftp

If you have trouble reaching the server, you can enter the ping address command to test the connection.

Step 7 At the "Do you wish to erase the passwords?" prompt, enter Y.

You can now log in with the default login password of "cisco" and the blank enable password.


The following example shows the PIX password recovery with the TFTP server on the outside interface:

monitor> interface 0
0: i8255X @ PCI(bus:0 dev:13 irq:10)
1: i8255X @ PCI(bus:0 dev:14 irq:7 )
 
Using 0: i82559 @ PCI(bus:0 dev:13 irq:10), MAC: 0050.54ff.82b9
monitor> address 10.21.1.99
address 10.21.1.99
monitor> server 172.18.125.3
server 172.18.125.3
monitor> file np70.bin
file np52.bin
monitor> gateway 10.21.1.1
gateway 10.21.1.1
monitor> ping 172.18.125.3
Sending 5, 100-byte 0xf8d3 ICMP Echoes to 172.18.125.3, timeout is 4 seconds:
!!!!!
Success rate is 100 percent (5/5)
monitor> tftp
tftp np52.bin@172.18.125.3 via 10.21.1.1...................................
Received 73728 bytes
 
Cisco PIX password tool (4.0) #0: Tue Aug 22 23:22:19 PDT 2005
Flash=i28F640J5 @ 0x300
BIOS Flash=AT29C257 @ 0xd8000
 
Do you wish to erase the passwords? [yn] y
Passwords have been erased.
 
Rebooting....

Disabling Password Recovery

You might want to disable password recovery to ensure that unauthorized users cannot use the password recovery mechanism to compromise the security appliance. To disable password recovery, enter the following command:

hostname(config)# no service password-recovery

On the ASA 5500 series adaptive security appliance, the no service password-recovery command prevents a user from entering ROMMON with the configuration intact. When a user enters ROMMON, the security appliance prompts the user to erase all Flash file systems. The user cannot enter ROMMON without first performing this erasure. If a user chooses not to erase the Flash file system, the security appliance reloads. Because password recovery depends on using ROMMON and maintaining the existing configuration, this erasure prevents you from recovering a password. However, disabling password recovery prevents unauthorized users from viewing the configuration or inserting different passwords. In this case, to recover the system to an operating state, load a new image and a backup configuration file, if available. The service password-recovery command appears in the configuration file for informational purposes only; when you enter the command at the CLI prompt, the setting is saved in NVRAM. The only way to change the setting is to enter the command at the CLI prompt. Loading a new configuration with a different version of the command does not change the setting. If you disable password recovery when the security appliance is configured to ignore the startup configuration at startup (in preparation for password recovery), then the security appliance changes the setting to boot the startup configuration as usual. If you use failover, and the standby unit is configured to ignore the startup configuration, then the same change is made to the configuration register when the no service password recovery command replicates to the standby unit.

On the PIX 500 series security appliance, the no service password-recovery command forces the PIX password tool to prompt the user to erase all Flash file systems. The user cannot use the PIX password tool without first performing this erasure. If a user chooses not to erase the Flash file system, the security appliance reloads. Because password recovery depends on maintaining the existing configuration, this erasure prevents you from recovering a password. However, disabling password recovery prevents unauthorized users from viewing the configuration or inserting different passwords. In this case, to recover the system to an operating state, load a new image and a backup configuration file, if available.

Other Troubleshooting Tools

The security appliance provides other troubleshooting tools to be used in conjunction with Cisco TAC:

Viewing Debug Messages

Capturing Packets

Viewing the Crash Dump

Viewing Debug Messages

Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco TAC. Moreover, it is best to use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use. To enable debug messages, see the debug commands in the Cisco Security Appliance Command Reference.

Capturing Packets

Capturing packets is sometimes useful when troubleshooting connectivity problems or monitoring suspicious activity. We recommend contacting Cisco TAC if you want to use the packet capture feature. See the capture command in the Cisco Security Appliance Command Reference.

Viewing the Crash Dump

If the security appliance crashes, you can view the crash dump information. We recommend contacting Cisco TAC if you want to interpret the crash dump. See the show crashdump command in the Cisco Security Appliance Command Reference.

Common Problems

This section describes common problems with the security appliance, and how you might resolve them.

Symptom    The context configuration was not saved, and was lost when you reloaded.

Possible Cause    You did not save each context within the context execution space. If you are configuring contexts at the command line, you did not save the context before you changed to the next context.

Recommended Action    Save each context within the context execution space using the copy run start command. You cannot save contexts from the system execution space.

Symptom    You cannot make a Telnet connection or SSH to the security appliance interface.

Possible Cause    You did not enable Telnet or SSH to the security appliance.

Recommended Action    Enable Telnet or SSH to the security appliance according to the "Allowing Telnet Access" section on page 31-1 or the "Allowing SSH Access" section on page 31-2.

Symptom    You cannot ping the security appliance interface.

Possible Cause    You disabled ICMP to the security appliance.

Recommended Action    Enable ICMP to the security appliance for your IP address using the icmp command.

Symptom    You cannot ping through the security appliance, even though the access list allows it.

Possible Cause    You did not enable the ICMP inspection engine or apply access lists on both the ingress and egress interfaces.

Recommended Action    Because ICMP is a connectionless protocol, the security appliance does not automatically allow returning traffic through. In addition to an access list on the ingress interface, you either need to apply an access list to egress interface to allow replying traffic, or enable the ICMP inspection engine, which treats ICMP connections as stateful connections.

Symptom    Traffic does not pass between two interfaces on the same security level.

Possible Cause    You did not enable the feature that allows traffic to pass between interfaces on the same security level.

Recommended Action    Enable this feature according to the "Allowing Communication Between Interfaces on the Same Security Level" section on page 6-5.