Identifying Incidents Using Firewall and Cisco IOS Router Syslog Events

Introduction

Syslog messages from transit network devices can provide insight into and context for security events that may not be available from other sources. This insight aids in determining the validity and extent of an incident. Within the context of a security incident, administrators can use syslog messages to understand communication relationships, timing, and, in some cases, the attacker's motives and/or tools. These events should be considered complementary and should be used in conjunction with other forms of network monitoring that may already be in place.

The focus of this paper is the analysis of data contained in the memory buffer of the device, using tools native to the device itself, after a trigger event has occurred. This kind of analysis may require an administrator to obtain additional information about the event. This paper does not provide recommendations for logging configurations or tools that provide automated analysis. In most cases, historical analysis will be performed on logging-specific servers, where tools and syntax may be similar. It should be noted that if logging servers exist, using these techniques on such servers would be preferable. However, because of the varied nature of implementations, these examples will focus on using native Cisco IOS Software capabilities and locally available syslog information. References are also provided to aid in automation or syntactical subtleties.

Log Messages of Interest

For the purpose of this guide, Cisco Adaptive Security Appliance (ASA) software version 7.2 will be used for firewall examples and Cisco IOS Software version 12.3 will be the primary IOS version used for router examples, although the ACL Syslog Correlation feature requires Cisco IOS Software 12.4(22)T or later. Syslog messages can be one of eight predefined severity levels. The following table briefly summarizes the different severity logging levels:

Level System Description
Emergency 0 System unusable messages
Alert 1 Immediate action required messages
Critical 2 Critical condition messages
Error 3 Error condition messages
Warning 4 Warning condition messages
Notification 5 Normal but significant messages
Information 6 Informational messages
Debugging 7 Debugging messages

 

Cisco ASA Firewall

Although all log messages can be of use in certain circumstances, in most cases a small subset of log messages will initially provide the most benefit. After these events have been examined, administrators can expand the scope of their analysis by searching for additional details. The following table summarizes the most common messages and the associated severity level. Fortunately, most of these messages come from fairly contiguous mnemonic identifiers, which aid in identification when using command-line tools.

Mnemonic Severity Description
4000nn ("nn" indicates multiple messages currently 400000 - 400050)4IPS:number string from IP_address to IP_address on interface interface_name
1060012Inbound TCP connection denied from IP_address/port to IP_address/port flags tcp_flags on interface interface_name
1060022protocol Connection denied by outbound list acl_ID src inside_address dest outside_address
1060062Deny inbound UDP from outside_address/outside_port to inside_address/inside_port on interface interface_name
1060072Deny inbound UDP from outside_address/outside_port to inside_address/inside_port due to DNS {Response|Query}
1060103Deny inbound protocol src interface_name:dest_address/dest_port dst
1060123Deny IP from IP_address to IP_address, IP options hex
1060133Dropping echo request from IP_address to PAT address IP_address
1060143Deny inbound icmp src interface_name: IP_address dst interface_name: IP_address (type dec, code dec)
1060156Deny TCP (no connection) from IP_address/port to IP_address/port flags tcp_flags on interface interface_name
1060162Deny IP spoof from (IP_address) to IP_address on interface interface_name.
1060172Deny IP due to Land Attack from IP_address to IP_address
1060182ICMP packet type ICMP_type denied by outbound list acl_ID src inside_address dest outside_address
1060202Deny IP teardrop fragment (size = number, offset = number) from IP_address to IP_address
1060211Deny protocol reverse path check from source_address to dest_address on interface interface_name
1060221Deny protocol connection spoof from source_address to dest_address on interface interface_name
1060234Deny protocol src [interface_name:source_address/source_port] dst interface_name:dest_address/dest_port [type {string}, code {code}] by access_group acl_ID
1061004access-list acl_ID {permitted | denied | est-allowed} protocol interface_name/source_address(source_port) -> interface_name/dest_address(dest_port) hit-cnt number ({first hit | number-second interval})
7100033{TCP|UDP} access denied by ACL from source_IP/source_port to interface_name:dest_IP/service

 

Note: Basic IPS support needs to be enabled using the ip audit feature in order for log messages 400000 - 400050 (4000nn) to be generated. These messages do not require the use of the AIP SSM IPS features. Information on configuring the ip audit feature is available in the Cisco Security Appliance Command Reference, Version 7.2.

In addition to the messages in the preceding table, several other connection-related messages of severity levels 6 (informational) and 7 (debug) are commonly used during analysis. Logging at severity levels 6 and 7 will have a performance impact.

Cisco Router

As with the Cisco ASA, a large number of log messages may be useful on Cisco IOS Software. However, in most cases a small subset that will need to be examined more often. Router log messages do not contain numerical identifiers that assist in identifying the messages. The following is a list of router log messages that are most likely to be useful when analyzing security-related incidents. Because many organizations do not make extensive use of logging on routers and because router logging is somewhat limited, NetFlow is often a more effective means of analysis.

Mnemonic Severity Description
%SEC-6-IPACCESSLOGDP6A packet matching the log criteria for the given access list has been detected.
%SEC-6-IPACCESSLOGNP6A packet matching the log criteria for the given access list has been detected.
%SEC-6-IPACCESSLOGP6A packet matching the log criteria for the given access list has been detected (TCP or UDP)
%SEC-6-IPACCESSLOGRL6Some packet-matching logs were missed because the access list log messages were rate limited, or no access list log buffers were available.
%SEC-6-IPACCESSLOGRP6A packet matching the log criteria for the given access list has been detected.
%SEC-6-IPACCESSLOGS6A packet matching the log criteria for the given access list was detected.
%SEC-4-TOOMANY4The system was not able to process the packet because there was not enough room for all of the desired IP header options. The packet has been discarded.
%IPV6-6-ACCESSLOGP6A packet matching the log criteria for the given access list was detected.
%IPV6-6-ACCESSLOGDP6A packet matching the log criteria for the given access list was detected.
%IPV6-6-ACCESSLOGNP6A packet matching the log criteria for the given access list was detected.

 

Command-Line Syntax

Cisco ASA

Administrators can use the show logging command with several different keywords to find relevant log messages. In most cases, the grep command followed by a regular expression will yield the most flexibility and best results. It should be noted that often several different regular expressions may be used to accomplish the same task. These examples are for messages stored in the logging buffer locally on the device itself. The size of the buffer can be modified using the logging buffer size size in bytes command.

More information on the specific command syntax and regular expressions is available in Using the Command Line Interface. To find messages of a specific severity (e.g. ASA-4), administrators can use grep as follows:

Firewall# show logging | grep ASA-4
Aug 24 2007 08:54:31: %ASA-4-500004: Invalid transport field for protocol=TCP, from 192.168.208.63/46855 to 192.168.150.77/0
Aug 24 2007 08:54:31: %ASA-4-500004: Invalid transport field for protocol=TCP, from 192.168.208.63/46856 to 192.168.150.77/0
Aug 24 2007 08:54:48: %ASA-4-106023: Deny tcp src outside:192.168.208.63/46857 dst inside:192.168.150.77/443 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 08:54:48: %ASA-4-106023: Deny tcp src outside:192.168.208.63/46863 dst inside:192.168.150.77/256 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 08:54:48: %ASA-4-106023: Deny tcp src outside:192.168.208.63/46867 dst inside:192.168.150.77/389 by access-group "OUTSIDE" [0x5063b82f, 0x0]

To find a specific logging message (for example, 106023), grep can be used as follows:

Firewall# show logging | grep 106023
Aug 24 2007 08:54:49: %ASA-4-106023: Deny tcp src outside:192.168.208.63/47511 dst inside:192.168.150.77/352 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 08:54:49: %ASA-4-106023: Deny tcp src outside:192.168.208.63/47513 dst inside:192.168.150.77/167 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 08:54:49: %ASA-4-106023: Deny tcp src outside:192.168.208.63/47517 dst inside:192.168.150.77/210 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 08:54:49: %ASA-4-106023: Deny tcp src outside:192.168.208.63/47522 dst inside:192.168.150.77/281 by access-group "OUTSIDE" [0x5063b82f, 0x0]

To find more than one specific logging message, administrators can use regular expressions with the grep command as follows:

Firewall# show logging | grep (106023|106100)
Aug 24 2007 09:02:37: %ASA-4-106023: Deny tcp src outside:192.168.208.63/51584 dst inside:192.168.150.77/239 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 09:02:37: %ASA-4-106023: Deny tcp src outside:192.168.208.63/51585 dst inside:192.168.150.77/288 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 09:02:37: %ASA-4-106023: Deny tcp src outside:192.168.208.63/51586 dst inside:192.168.150.77/147 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 09:02:37: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(51587) -> inside/192.168.150.77(104) hit-cnt 1 first hit [0x22e8ac21, 0x0]

The grep command and regular expressions can be extended to find a group of log messages. In the following example, the syslog messages ranging from 1060nn to 1062nn will be displayed using grep and the respective regular expressions:

Firewall# show logging | grep 106[0-2][0-9][0-9]:
Aug 24 2007 09:14:54: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(38807) -> inside/192.168.150.77(80) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 09:16:14: %ASA-6-106015: Deny TCP (no connection) from 192.168.150.65/2278 to 64.101.128.83/80 flags RST  on interface inside
Aug 24 2007 09:16:41: %ASA-4-106023: Deny icmp src outside:192.168.208.63 dst inside:192.168.150.77 (type 8, code 0) by access-group "OUTSIDE" [0xd3f63b90, 0x0]
Aug 24 2007 09:16:41: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(38664) -> inside/192.168.150.77(80) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 09:16:43: %ASA-4-106023: Deny icmp src outside:192.168.208.63 dst inside:192.168.150.77 (type 8, code 0) by access-group "OUTSIDE" [0xd3f63b90, 0x0]
Aug 24 2007 09:16:43: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(38665) -> inside/192.168.150.77(80) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 09:17:32: %ASA-1-106021: Deny ICMP reverse path check from 192.168.150.60 to 192.168.2.1 on interface outside

Cisco Router

As with the Cisco ASA, the show logging command combined with the use of certain keywords and search-specific filters is the most effective means of finding relevant information. It should be reiterated that the logging capabilities of IOS routers are much more limited than those of the Cisco ASA. The grep keyword does not exist in this context, so instead administrators must use the include keyword. In addition, either the log or log-input keyword must be present on each access control entry (ACE) that should generate logging messages. If neither log nor log-input is present on a given ACE, no log messages will be generated for it.

The following example (access list 185) will demonstrate how those log messages will be displayed. In this example, TCP ports 53 through 123 will not be logged, TCP ports 137 through 445 will use the log keyword, and TCP ports 500 through 1024 will use the log-input keyword. For this example, the access list counter and logging buffer will be cleared and connections will then be created to each port group. The router response demonstrates the different results that are obtained when using no keyword, the log keyword, or the log-input keyword. The first part of the example shows the access list counters, whereas the second part of the example shows the available logging messages.

For more information on the use of the log and log-input keywords, refer to the white paper Understanding Access Control List Logging.

Access List Counters

Router#show access-lists 185
Extended IP access list 185
    10 deny tcp 172.16.1.0 0.0.0.255 host 192.168.2.1 range domain 123 (284 matches)
    20 deny tcp 172.16.1.0 0.0.0.255 host 192.168.2.1 range 137 445 log (1236 matches)
    30 deny tcp 172.16.1.0 0.0.0.255 host 192.168.2.1 range 500 1024 log-input (1574 matches)
    40 permit ip any any (1550 matches)
Router#

Available Logging Messages

For the logging messages that follow, note that there is no logging for ACE sequence ID 10 of access list 185 because a logging keyword was not used in this ACE. Note also the different syntax of the logs for which the log keyword was used (messages 2092 through 2094), versus those for which the log-input keyword was used (messages 2095 through 2097).

Router#show logging | include 185
002092: Mar 30 2010 11:41:48.681 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.92(59078) -> 192.168.2.1(417), 1 packet
002093: Mar 30 2010 11:41:49.681 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.95(14897) -> 192.168.2.1(427), 1 packet
002094: Mar 30 2010 11:41:50.681 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.182(16737) -> 192.168.2.1(437), 1 packet
002095: Mar 30 2010 11:41:56.985 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.219(14872) (FastEthernet0/1 0007.8580.9edd) -> 192.168.2.1(500), 1 packet
002096: Mar 30 2010 11:41:57.984 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.208(7751) (FastEthernet0/1 0007.8580.9edd) -> 192.168.2.1(510), 1 packet
002097: Mar 30 2010 11:41:58.984 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.26(41202) (FastEthernet0/1 0007.8580.9edd) -> 192.168.2.1(520), 1 packet
Router#

Regular Expressions

Even though Cisco IOS Software does not support the grep command in this context, regular expressions can still be used to identify potential intrusions. In the following example, a regular expression was used with the include keyword to identify logs for which the source IP address was 172.16.1.92 and the destination port was 137.

Router#show logging | include 172.16.1.92.*\-\>.*\(137\)
002064: Mar 30 2010 11:41:20.659 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.92(17587) -> 192.168.2.1(137), 1 packet
002065: Mar 30 2010 11:41:21.659 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.92(58564) -> 192.168.2.1(137), 1 packet
002067: Mar 30 2010 11:41:23.663 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.92(17755) -> 192.168.2.1(137), 1 packet
Router#

Selecting Log Messages Generated by a Specific ACE

Introduced with Cisco IOS release 12.4(22)T, the ACL Syslog Correlation feature allows for every log message generated by an ACE to include a cookie. The value of the cookie can be either a system-generated hash or a user-defined string. The global configuration command ip access-list logging hash-generation enables the automatic generation of an MD5 hash value to be used as a cookie. The following is access list 185 after enabling this feature:

Router#show access-lists 185
Extended IP access list 185
    10 deny tcp 172.16.1.0 0.0.0.255 host 192.168.2.1 range domain 123 (355 matches)
    20 deny tcp 172.16.1.0 0.0.0.255 host 192.168.2.1 range 137 445 log (1432 matches) (hash = 0x279C8521)
    30 deny tcp 172.16.1.0 0.0.0.255 host 192.168.2.1 range 500 1024 log-input (1574 matches) (hash = 0xC85A617C)
    40 permit ip any any (1776 matches)
Router#

ACE sequence ID 10 on access list 185 did not receive a cookie because it does not include the log or log-input keyword. The following is an example of the syslog messages that were generated when the ACL Syslog Correlation feature was enabled:

Router#show logging | include 185
002416: Mar 30 2010 11:51:07.149 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.14(11995) -> 192.168.2.1(418), 1 packet  [0x279C8521]
002417: Mar 30 2010 11:51:08.153 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.14(7331) -> 192.168.2.1(428), 1 packet  [0x279C8521]
002418: Mar 30 2010 11:51:09.153 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.49(36426) -> 192.168.2.1(438), 1 packet  [0x279C8521]
002419: Mar 30 2010 11:51:15.353 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.229(45012) (FastEthernet0/1 0007.8580.9edd) -> 192.168.2.1(500), 1 packet  [0xC85A617C]
002420: Mar 30 2010 11:51:16.357 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.42(56954) (FastEthernet0/1 0007.8580.9edd) -> 192.168.2.1(510), 1 packet  [0xC85A617C]
002421: Mar 30 2010 11:51:17.357 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.2(25313) (FastEthernet0/1 0007.8580.9edd) -> 192.168.2.1(520), 1 packet  [0xC85A617C]

Referencing the cookie value allows the administrator to select only those syslog messages that were generated by ACE sequence ID 20 on access list 185:

Router#show logging | include 0x279C8521
002415: Mar 30 2010 11:51:06.150 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.72(5775) -> 192.168.2.1(408), 1 packet  [0x279C8521]
002416: Mar 30 2010 11:51:07.149 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.14(11995) -> 192.168.2.1(418), 1 packet  [0x279C8521]
002417: Mar 30 2010 11:51:08.153 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.14(7331) -> 192.168.2.1(428), 1 packet  [0x279C8521]
002418: Mar 30 2010 11:51:09.153 EDT: %SEC-6-IPACCESSLOGP: list 185 denied tcp 172.16.1.49(36426) -> 192.168.2.1(438), 1 packet  [0x279C8521]
Router#

Quick Identification and Impetus

The impetus for analyzing system logs can come from several sources. Often the motivation can come from network-related issues, hits to classification ACL ACEs, or the need to gather further information to support activity from another source, such IPS signatures. Below is an example from an ACL on the Cisco ASA. For the purpose of this example, a deny ACE will be used to gather more data about the denied events. It is important to note that ACE counters can have a much longer lifetime than log entries that reside in the logging buffer, therefore it is likely that there could be ACL ACE counter numbers without associated logs in the logging buffer. To clear the ACL counters, administrators can use the command clear access-list access list name counters.

Firewall# show access-list OUTSIDE  
access-list OUTSIDE; 24 elements
access-list OUTSIDE line 1 extended deny tcp host 192.168.208.63 host 192.168.150.77 range www 123 log informational interval 300 (hitcnt=96) 0x22e8ac21 
access-list OUTSIDE line 2 extended deny tcp host 192.168.208.63 host 192.168.150.77 range netbios-ssn 445 (hitcnt=1842) 0x5063b82f 
access-list OUTSIDE line 3 extended deny icmp host 192.168.208.63 host 192.168.150.77 (hitcnt=6) 0xd3f63b90

When the impetus is an ACE counter from a classification ACL, it is easiest to use the ACL name to quickly identify entries instead of the logging-specific messages. This will show log messages from the ACL OUTSIDE regardless of whether the log keyword was used in the ACL. Administrators must take care to use the correct alphanumeric case for the specific ACL, otherwise the Cisco ASA will display many connection-related messages.

Firewall# show logging | grep OUTSIDE
Aug 24 2007 09:02:35: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(51063) -> inside/192.168.150.77(83) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 09:02:35: %ASA-4-106023: Deny tcp src outside:192.168.208.63/51064 dst inside:192.168.150.77/214 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 09:02:35: %ASA-4-106023: Deny tcp src outside:192.168.208.63/51065 dst inside:192.168.150.77/260 by access-group "OUTSIDE" [0x5063b82f, 0x0]

Alternatively, if the ip audit feature is used, the regular expression could be expanded, which would enable display of logs from the access list named OUTSIDE and from IPS-related signatures.

Firewall# show logging | grep (OUTSIDE|4000[0-9][0-9])
ug 24 2007 10:07:47: %ASA-4-400014: IDS:2004 ICMP echo request from 192.168.208.63 to 192.168.150.77 on interface outside
Aug 24 2007 10:07:47: %ASA-4-106023: Deny icmp src outside:192.168.208.63 dst inside:192.168.150.77 (type 8, code 0) by access-group "OUTSIDE" [0xd3f63b90, 0x0]
Aug 24 2007 10:07:47: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(57633) -> inside/192.168.150.77(80) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 10:07:49: %ASA-4-400014: IDS:2004 ICMP echo request from 192.168.208.63 to 192.168.150.77 on interface outside
Aug 24 2007 10:07:49: %ASA-4-106023: Deny icmp src outside:192.168.208.63 dst inside:192.168.150.77 (type 8, code 0) by access-group "OUTSIDE" [0xd3f63b90, 0x0]
Aug 24 2007 10:07:49: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(57634) -> inside/192.168.150.77(80) hit-cnt 1 first hit [0x22e8ac21, 0x0]

It is also possible to look for specific ACE entries by searching for the hash (for example, 0x22e8ac21) from the ACE entry of our ACL. The hash (for example, 0x22e8ac21) identifier may not be present in some versions of code or in certain Cisco Security Appliances.

Firewall# show access-list OUTSIDE  
access-list OUTSIDE; 24 elements
access-list OUTSIDE line 1 extended deny tcp host 192.168.208.63 host 192.168.150.77 range www 123 log informational interval 300 (hitcnt=96) 0x22e8ac21 
access-list OUTSIDE line 2 extended deny tcp host 192.168.208.63 host 192.168.150.77 range netbios-ssn 445 (hitcnt=1842) 0x5063b82f 
access-list OUTSIDE line 3 extended deny icmp host 192.168.208.63 host 192.168.150.77 (hitcnt=6) 0xd3f63b90
Firewall# show logging | grep OUTSIDE
Aug 24 2007 09:02:35: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(51063) -> inside/192.168.150.77(83) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 09:02:35: %ASA-4-106023: Deny tcp src outside:192.168.208.63/51064 dst inside:192.168.150.77/214 by access-group "OUTSIDE" [0x5063b82f, 0x0]

If administrators wanted to see only the entries that corresponded to the specific ACE entry for line 2 of the ACL, they could use the hash 0x5063b82f:

Firewall# show logging | grep 5063b82f 
Aug 29 2007 11:14:55: %ASA-4-106023: Deny tcp src outside:192.168.208.63/35746 dst inside:192.168.150.77/389 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 29 2007 11:14:55: %ASA-4-106023: Deny tcp src outside:192.168.208.63/35746 dst inside:192.168.150.77/443 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 29 2007 11:14:55: %ASA-4-106023: Deny tcp src outside:192.168.208.63/35746 dst inside:192.168.150.77/256 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 29 2007 11:14:55: %ASA-4-106023: Deny tcp src outside:192.168.208.63/35746 dst inside:192.168.150.77/399 by access-group "OUTSIDE" [0x5063b82f, 0x0]

Methodology

It is important to note that, although ACL denied log messages are often of interest because they may indicate potential unauthorized attempts to access the network, ACL allowed messages may sometimes be more useful when administrators investigate incidents. In the following example, the impetus to search for ACL denied messages on the access list OUTSIDE is expanded. The results of the commands will reveal what activity has been generated that is against policy and then determine whether the attacker performed any activity that was allowed by the policy.

First administrators will use the command show logging | grep access list name to identify the source IP address that may be needed for further investigation. In this example, the source IP address of interest is 192.168.208.63.

Firewall# show logging | grep OUTSIDE 
Aug 24 2007 10:27:29: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(39675)-> inside/192.168.150.77(80) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 10:27:31: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(39676) -> inside/192.168.150.77(80) hit-cnt 1 first hit [0x22e8ac21, 0x0]

Next administrators will use show logging grep ip address to find more information about the IP address (192.168.208.63):

Firewall# show logging | grep 192.168.208.63
Aug 24 2007 10:27:22: %ASA-4-400014: IDS:2004 ICMP echo request from 192.168.208.63 to 192.168.150.70 on interface outside 
Aug 24 2007 10:27:22: %ASA-6-302020: Built ICMP connection for faddr 192.168.208.63/15343 gaddr 192.168.150.70/0 laddr 192.168.150.70/0
Aug 24 2007 10:27:22: %ASA-6-106015: Deny TCP (no connection) from 192.168.208.63/49827 to 192.168.150.70/80 flags ACK on interface outside
Aug 24 2007 10:27:22: %ASA-6-302020: Built ICMP connection for faddr 192.168.208.63/15343 gaddr 192.168.150.70/0 laddr 192.168.150.70/0
Aug 24 2007 10:27:22: %ASA-6-302015: Built inbound UDP connection 732748 for outside:192.168.208.63/49804 (192.168.208.63/49804) to inside:192.168.150.70/53 (192.168.150.70/53)
Aug 24 2007 10:27:22: %ASA-6-302015: Built inbound UDP connection 732749 for outside:192.168.208.63/49804 (192.168.208.63/49804) to inside:192.168.150.70/123 (192.168.150.70/123)
Aug 24 2007 10:27:22: %ASA-6-302015: Built inbound UDP connection 732750 for outside:192.168.208.63/49804 (192.168.208.63/49804) to inside:192.168.150.70/139 (192.168.150.70/139)
Aug 24 2007 10:27:22: %ASA-6-302020: Built ICMP connection for faddr 192.168.208.63/0 gaddr 192.168.150.70/0 laddr 192.168.150.70/0
Aug 24 2007 10:27:22: %ASA-6-302021: Teardown ICMP connection for faddr 192.168.208.63/0 gaddr 192.168.150.70/0 laddr 192.168.150.70/0
Aug 24 2007 10:27:22: %ASA-6-302020: Built ICMP connection for faddr 192.168.208.63/0 gaddr 192.168.150.70/0 laddr 192.168.150.70/0
Aug 24 2007 10:27:22: %ASA-6-302021: Teardown ICMP connection for faddr 192.168.208.63/0 gaddr 192.168.150.70/0 laddr 192.168.150.70/0
Aug 24 2007 10:27:23: %ASA-6-302015: Built inbound UDP connection 732753 for outside:192.168.208.63/49805 (192.168.208.63/49805) to inside:192.168.150.70/53 (192.168.150.70/53)
Aug 24 2007 10:27:24: %ASA-6-302021: Teardown ICMP connection for faddr 192.168.208.63/15343 gaddr 192.168.150.70/0 laddr 192.168.150.70/0
Aug 24 2007 10:27:24: %ASA-6-302021: Teardown ICMP connection for faddr 192.168.208.63/15343 gaddr 192.168.150.70/0 laddr 192.168.150.70/0
Aug 24 2007 10:27:29: %ASA-4-400014: IDS:2004 ICMP echo request from 192.168.208.63 to 192.168.150.77 on interface outside
Aug 24 2007 10:27:29: %ASA-4-106023: Deny icmp src outside:192.168.208.63 dst inside:192.168.150.77 (type 8, code 0) by access-group "OUTSIDE" [0xd3f63b90, 0x0]
Aug 24 2007 10:27:29: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(39675) -> inside/192.168.150.77(80) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 10:27:31: %ASA-4-400014: IDS:2004 ICMP echo request from 192.168.208.63 to 192.168.150.77 on interface outside 
Aug 24 2007 10:27:31: %ASA-4-106023: Deny icmp src outside:192.168.208.63 dst inside:192.168.150.77 (type 8, code 0) by access-group "OUTSIDE" [0xd3f63b90, 0x0]
Aug 24 2007 10:27:31: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(39676) -> inside/192.168.150.77(80) hit-cnt 1 first hit [0x22e8ac21, 0x0]

It appears that, in addition to several denied connections, this IP address also triggered some IPS-related alarms as well as successful UDP connections that were not denied by OUTSIDE access-list policy.

Trends

Although these syslog events do not give the same information as a full sniffer log such as TCP dump, there still is some information administrators can use to learn about the attacker. In addition, where there is access to historical data, some of these techniques can be used to find information that may evade time-based or connection-based means of identification. The following examples illustrate methods that can be used to aid in the identification of vertical and horizontal network scans.

Note: Horizontal scans will scan for a specific port or ports across a range of hosts (IP addresses), whereas vertical scans will scan a range of IP/TCP/UDP ports on a specific host or hosts (IP address/IP addresses).

The following is an example of a horizontal scan for TCP port 80 across several hosts. Note the incrementing source port from the attacker (51606-51611). This is due to standard sockets behavior. If the host has been scanning several other hosts, administrators may not see contiguous source ports such as those shown in this example. Also note that the URL "/" was accessed on host 192.168.150.70 and that 1533 bytes were transferred.

Firewall# show logging | grep 192.168.208.63
Aug 24 2007 11:15:16: %ASA-6-302013: Built inbound TCP connection 733280 for outside:192.168.208.63/51606 (192.168.208.63/51606) to inside:192.168.150.70/80 (192.168.150.70/80)
Aug 24 2007 11:15:16: %ASA-6-302014: Teardown TCP connection 733280 for outside:192.168.208.63/51606 to inside:192.168.150.70/80 duration 0:00:00 bytes 0 TCP Reset-O
Aug 24 2007 11:15:29: %ASA-6-302013: Built inbound TCP connection 733282 for outside:192.168.208.63/51607 (192.168.208.63/51607) to inside:192.168.150.60/80 (192.168.150.60/80)
Aug 24 2007 11:15:29: %ASA-6-302014: Teardown TCP connection 733282 for outside:192.168.208.63/51607 to inside:192.168.150.60/80 duration 0:00:00 bytes 0 TCP Reset-I
Aug 24 2007 11:15:33: %ASA-6-302013: Built inbound TCP connection 733283 for outside:192.168.208.63/51608 (192.168.208.63/51608) to inside:192.168.150.63/80 (192.168.150.63/80)
Aug 24 2007 11:15:33: %ASA-6-302014: Teardown TCP connection 733283 for outside:192.168.208.63/51608 to inside:192.168.150.63/80 duration 0:00:00 bytes 0 TCP Reset-I
Aug 24 2007 11:15:39: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(51609) -> inside/192.168.150.77(80) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 11:15:40: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(51610) -> inside/192.168.150.77(80) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 11:15:50: %ASA-6-302013: Built inbound TCP connection 733286 for outside:192.168.208.63/51611 (192.168.208.63/51611) to inside:192.168.150.70/80 (192.168.150.70/80)
Aug 24 2007 11:15:58: %ASA-5-304001: 192.168.208.63 Accessed URL 192.168.150.70:/
Aug 24 2007 11:15:59: %ASA-6-302014: Teardown TCP connection 733286 for outside:192.168.208.63/51611 to inside:192.168.150.70/80 duration 0:00:09 bytes 1533 TCP FINs

The following example shows a vertical scan of a specific host. It reveals that several different ports were accessed and the source port of the attacking IP is nearly contiguous, which is standard socket behavior.

Firewall# show logging | grep 192.168.208.63 
Aug 24 2007 11:23:11: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(52978) -> inside/192.168.150.77(80) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 11:23:11: %ASA-6-302013: Built inbound TCP connection 734665 for outside:192.168.208.63/52979 (192.168.208.63/52979) to inside:192.168.150.77/53 (192.168.150.77/53)
Aug 24 2007 11:23:11: %ASA-4-106023: Deny tcp src outside:192.168.208.63/52980 dst inside:192.168.150.77/256 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 11:23:11: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(52981) -> inside/192.168.150.77(113) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 11:23:11: %ASA-4-106023: Deny tcp src outside:192.168.208.63/52982 dst inside:192.168.150.77/443 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 11:23:11: %ASA-6-302013: Built inbound TCP connection 734666 for outside:192.168.208.63/52983 (192.168.208.63/52983) to inside:192.168.150.77/636 (192.168.150.77/636)
Aug 24 2007 11:23:11: %ASA-4-106023: Deny tcp src outside:192.168.208.63/52984 dst inside:192.168.150.77/389 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 11:23:11: %ASA-6-302013: Built inbound TCP connection 734667 for outside:192.168.208.63/52985 (192.168.208.63/52985) to inside:192.168.150.77/554 (192.168.150.77/554)
Aug 24 2007 11:23:11: %ASA-6-302013: Built inbound TCP connection 734668 for outside:192.168.208.63/52986 (192.168.208.63/52986) to inside:192.168.150.77/25 (192.168.150.77/25)
Aug 24 2007 11:23:11: %ASA-6-302013: Built inbound TCP connection 734669 for outside:192.168.208.63/52987 (192.168.208.63/52987) to inside:192.168.150.77/23 (192.168.150.77/23)

For cases in which IP address spoofing is strongly suspected, grepping for an IP address may not work very well, depending on the extent of the IP address spoofing. In these cases, it is better to rely on filtering based on regular expressions. Because of the near-contiguous nature of the source ports from the attacking host, it is likely that this traffic is from a single host controlled by the attacker.

Firewall# show logging | grep (106[0-2][0-9][0-9]|4000[0-9][0-9]) 
Aug 24 2007 11:43:41: %ASA-4-106023: Deny tcp src outside:192.168.208.63/55453 dst inside:192.168.150.77/256 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 11:43:41: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.20.55(55454) -> inside/192.168.150.77(80) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 11:43:41: %ASA-4-106023: Deny tcp src outside:192.168.28.68/55455 dst inside:192.168.150.77/443 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 11:43:41: %ASA-4-106023: Deny tcp src outside:192.168.18.13/55456 dst inside:192.168.150.77/389 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 11:43:41: %ASA-6-106100: access-list OUTSIDE denied tcp outside/192.168.208.63(55460) -> inside/192.168.150.77(113) hit-cnt 1 first hit [0x22e8ac21, 0x0]
Aug 24 2007 11:43:41: %ASA-4-106023: Deny tcp src outside:192.168.11.31/55457 dst inside:192.168.150.77/197 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 11:43:41: %ASA-4-106023: Deny tcp src outside:192.168.8.21/55458 dst inside:192.168.150.77/263 by access-group "OUTSIDE" [0x5063b82f, 0x0]
Aug 24 2007 11:43:41: %ASA-4-106023: Deny tcp src outside:192.168.16.231/55459 dst inside:192.168.150.77/445 by access-group "OUTSIDE" [0x5063b82f, 0x0]

Summary

Syslog messages from transit network devices can provide additional context and verification during an incident response. Often, these devices are the only place this level of detailed information exists. Although the examples in this paper were exclusive to tools available natively on the device and messages located locally in the memory buffer of the device itself, much of this analysis can be done on historical logs using common tools. Historical analysis of logs often may provide details that were not revealed using other tools and/or other forms of monitoring.

References

Understanding Access Control List Logging
http://www.cisco.com/web/about/security/intelligence/acl-logging.html

Using the Command Line Interface
http://www.cisco.com/en/US/docs/security/fwsm/fwsm31/command/reference/usecli.html#wp1046322

IOS Regular Expressions
http://www.cisco.com/en/US/docs/ios/12_2/termserv/configuration/guide/tcfaapre_ps1835_TSD_Products_Configuration_Guide_Chapter.html

Cisco IOS Software Release 12.4 Mainline Error and System Messages Documentation
http://www.cisco.com/en/US/products/ps6350/products_system_message_guides_list.html

Cisco Security Appliance System Log Messages, Version 7.2
http://www.cisco.com/en/US/docs/security/asa/asa72/system/message/syslog.html

 

This document is part of the Cisco Security Intelligence Operations.

This document is provided on an "as is" basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information on the document or materials linked from the document is at your own risk. Cisco reserves the right to change or update this document at any time.

Back to Top

Cisco Security Intelligence Operations