Table Of Contents
Implementation Note for NetFlow Collectors, Version 9.1
This document describes how to implement the use of NetFlow collectors for the ASAs and includes the following sections:
Event-Driven Data Export
Because the ASAs implement stateful tracking of flows, the tracked flows go through a set of state changes. NetFlow is used to export data about the status of a flow, and is triggered by the event that caused the state change. The tracked events include flow creation, flow denial (only flows denied by ACLs), and flow teardown.
To provide periodic byte counters for flow traffic, the ASA issues a timer per flow to generate periodic NSEL events in addition to state tracking. These periodic NSEL events, called flow update events, are usually timer-driven, which makes them more in line with traditional Netflow. However, flow update events can also be triggered by some flow state changes.
The ASA also exports syslog messages that include the same information. You can disable these syslog messages to avoid performance degradation by generating both NSEL records and syslog messages that represent the same event. For a list of redundant syslog messages, see the "Using NSEL and Syslog Messages" section in the Cisco ASA Series General Operations CLI Configuration Guide.
Most bidirectional flows are already assembled internally and are considered a single flow. The flow records reported by NSEL on the ASAs describe both directions of the flow. The data records explicitly define the source (initiator) and destination (responder) of the connection, and you can use this information to determine the direction of flow, if required by collector applications. In addition, some NSEL records include two byte counter fields, NF_F_FWD_FLOW_DELTA_BYTES and NF_F_REV_FLOW_DELTA_BYTES, which provide direction-specific traffic data.
The RFC states that templates may be sent to the user either at regular time intervals or after a set number of data records have been exported. These update intervals must be configurable. This implementation supports template updates by time interval only. Template updates based on the number of data records are not supported.
Options Template and Data Records
No options template or data records will be exported. Some fields are supported by show commands in the CLI. Collector applications must issue show commands to obtain additional information about certain fields. In addition, collectors must have unique hostnames and IP addresses; otherwise, the inspection behavior will be unpredictable. For more information, see the "Information Model" section and the Cisco ASA Series General Operations CLI Configuration Guide.
Observation Point and Observation Domain
The ASA is an Observation Domain, with each interface also an Observation Point. Flows that are created through all interfaces are exported, and no option exists to limit or filter the exported data to a specific set of interfaces. Flow that are created by external devices that connect to the ASA are also exported.
Only records for certain flows may need to be exported, For example, the ASA can generate NSEL events for flows that match an ACE. You can use this method to restrict the number of NSEL events that are generated for NetFlow. This implementation supports the filtering of NSEL events based on traffic and event type through Modular Policy Framework, with records sent to different collectors.
For example, with two collectors, you can do the following:
•Log all flow creation events to Collector 1.
•Log all flow denied events matching ACL1 to Collector 1.
•Log all events matching ACL1 to Collector 2.
If the Modular Policy Framework is not configured for NetFlow, no NSEL events are generated. For more information, see the Cisco ASA Series General Operations CLI Configuration Guide and the command reference.
This implementation of NetFlow only supports UDP payloads.
This section describes the data types and templates that are exported through NetFlow, and includes the following topics:
The list of required data elements was arrived at by consolidating the data exported by syslog messages that are generated for events that results in the export of NSEL records.
Table 1 lists the data elements that are exported from the ASAs through NSEL.
The columns include the following information:
•ID—A unique name that represents the field type
•TYPE—The value assigned for this field type
•LEN—The length of the field in records exported for the selected ASA
•DESC—A description of what the field type represents
Event IDs Field
The Event ID field describes the event that resulted in the NSEL record. Table 2 lists the values for event IDs.
Extended Event IDs Field
The extended event ID provides additional information about a particular event. This field includes a product-specific field ID (33002). Table 3 lists the values for extended event IDs.
Event Time Field
Each NSEL data record has the event time field (NF_F_EVENT_TIME_MSEC), which is the time that the event occurred in milliseconds. The NetFlow packet may consist of multiple events; however, the time that the packet is sent does not represent the time that the event occurred, because the NetFlow service waits for multiple events to pack the NetFlow packet.
Note Different events in the life of a flow may be issued in separate NetFlow packets and may arrive out-of-order at the collector. For example, the packet containing a flow teardown event may reach the collector before the packet containing a flow creation event. As a result, it is important that collector applications use the Event Time field to correlate events.
Data Records and Templates
This section describes the templates that are supported for various events and includes the following topics:
Templates describe the format of data records that are exported through NetFlow. Each flow event has several record formats or templates associated with it, as follows:
•There are different templates for different events.
•There are different templates for IPv4 and IPv6 flows under each event type.
•There are different templates for IPV44, IPV46, IPV64, and IPV66 flows under each event type.
•The flow creation/permitted event has different templates, which are based on the size of the username field associated with the flow. Different templates are required because the size of string fields is fixed in NetFlow. Having a single template with the largest possible size for string results is a waste of bandwidth, because most strings are far shorter than the maximum value. Two types of username fields are defined, which result in two types of templates in each category.
–A common username size for usernames that are less than 20 characters
–A maximum username size for usernames that are up to a maximum of 65 characters
–Each template has the Event Type and Extended Event Type fields, which can interpret or act on the event.
•The flow denied and flow deletion events have IPV46 and IPV64 templates in which the destination IP address has been translated by a NAT rule, but the source IP address has not been translated by a NAT rule; this results in different IP versions between the source and destination IP addresses. The source and destination NAT rules are not applied at the same time (the destination NAT rule is applied first), so it is possible for a NetFlow record to be generated before both NAT rules are applied or when only one NAT rule is available.
These partial NAT translation templates are not needed for flow creation and delayed flow creation events because both source and destination IP addresses need to be the same IP version for a flow to be created.
Note Template definitions are sent to all collectors, and you should use these IDs and definitions to parse data records.
Templates for Flow Creation Events
Flow creation events indicate that a flow has been created by the ASA. This event is also a log of flows that the ASA allows. Table 4 describes the templates to use for flow creation events.
Delays for Flow Creation Events
For short-lived flows, NSEL collection devices would benefit from processing a single event instead of these two events—flow-create and flow-teardown. So a configurable CLI parameter is provided to delay sending of the flow-create event. If the timer fires, the flow-create event is sent. However, if the flow is torn down before the timer expires, only the flow-teardown event is sent; no flow-create event is sent.
The flow-teardown event is extended and includes all information regarding the flow; no information is lost. New templates are introduced to accommodate the extended flow-teardown events.
Templates for Extended Flow Teardown Events
Table 5 describes the templates that are used for extended flow-teardown events.
Templates for Flow Denied Events
Flow denied events indicate that a flow has been denied. Table 6 describes the templates that are used for flow denied events.
Templates for Flow Teardown Events
Flow teardown events indicate that a flow has been terminated. Table 7 describes the templates that are used for flow teardown events.
Templates for Flow Update Events
Flow update events indicate that a flow update timer has gone off for a flow or a flow was torn down. This event functions as a periodic byte counter for flow traffic. Flow update events also use the same templates as flow teardown events, excluding those for partial NAT translation. The NF_F_FWD_FLOW_DELTA_BYTES and NF_F_REV_FLOW_DELTA_BYTES fields contain the byte counts since the last timer interval. The NF_F_FW_EXT_EVENT field is not used and is ignored in flow update records. See Table 7 for the templates that are used for flow teardown events.
Flow Update (at timer) and Flow Update (at teardown) Events
The ASA sets flow update timers for flows passing through it, and when the timers goes off, NSEL issues flow update (at timer) records. If there is no activity on the flow for the configured time interval, no flow update (at timer) records are sent for that interval. A flow update (at teardown) record is sent with a flow teardown record to capture the traffic in the last time interval. No flow update (at teardown) record is sent if there is no traffic on the flow for the last interval. In addition, no flow update (at teardown) record is sent for short-lived flows (that is, if teardown occurs before the first flow update (at timer) event occurs).
The flow update timer is not set nor is it ever set again if at the time of flow creation, no flow update collectors are configured or if during a flow update event, the flow update collectors are removed. Under these conditions, no flow update (at timer) event or flow update (at teardown) event is seen again.
Flow Update Records and Failover
An attempt to keep flow update records consistent before and after failover is made. After failover occurs, all flow update records are based on the last update from the previously active ASA. This update occurs every 15 seconds as long as traffic is flowing. Inaccuracies may appear in flow update records if failover pairs are brought up at different times, or if failover occurs before the active ASA has a chance to send a periodic update to the standby ASA.
Flow Update Events and Clustering
One major divergence occurs in how flow update events interact with failover and how they interact with clustering. In clustering, before ownership change, the flow director has a stub flow copy of the original flow, which would not have the active refresh timer set. Only after the original flow owner goes down will a full flow copy be generated with the active refresh timer set. This means it is highly likely that a noticeable time offset will occur between when the flow update timer goes off on the original flow owner and when the flow update timer goes off on the new flow owner.
After flow ownership changes in a cluster, all flow-update records are based on the last update that the flow director received. Flow information is updated every 15 seconds as long as there is traffic. Maintenance of up-to-date flow information uses the same methods as those provided for failover.
NetFlow and Failover
NetFlow data records and templates are only sent from the active (primary) ASA in an active-standby failover pair. The standby (secondary) ASA does not send any NetFlow-related information. However, after failover, the secondary ASA starts to send templates and NetFlow records for any replicated or new flows. The source IP address for each NetFlow collector connection is the same between the two ASAs, but the source port varies. This means that the NetFlow collectors are capable of differentiating packets sent from the primary unit and the secondary unit.
In an active-active failover pair, both ASAs may send NetFlow data records and templates simultaneously. Only the active unit per context sends the NetFlow packets, but the standby unit does not; much like in active-standby scenarios. The source IP address for each NetFlow collector connection is the same for an ASA context and its copy, but the source port varies.
Each ASA node (context) in the failover pair establishes its own connection to the NetFlow collector(s) and advertises its templates independently. The collector uses the source IP address and source port of the packet to differentiate between the NetFlow exporters.
NetFlow and Clustering
NetFlow is supported on both management and regular data interfaces; however, we recommend that you use management interfaces. When the NetFlow collector connection is configured on management-only interfaces, each ASA in the cluster uses its own per-unit source IP address and source port to send NetFlow packets. NetFlow may be used with both data interfaces in layer 2 mode and layer 3 mode. For data interfaces in layer 2 mode, each ASA in the cluster has the same source IP address, but the source port is different. Although layer 2 mode is designed to make a cluster appear as a single device, a NetFlow collector can differentiate between the different nodes in the cluster. For data interfaces in layer 3 mode, NetFlow operates the same way as management-only interfaces do.
Each ASA node in the cluster establishes its own connection to the NetFlow collector(s) and advertises its templates independently. The collector uses the source IP address and source port of the packet to differentiate between the NetFlow exporters.
For information about the commands that are used to configure the NSEL implementation on the ASA, see the Cisco ASA Series General Operations CLI Configuration Guide and the command reference. The commands are also used to display additional information about the fields in NSEL records.
External Partner Implementation Suggestions
This section provides examples of flows that generate events and includes information about how to implement collector support for the new NSEL fields for the ASA, and includes the following topics:
Example 1: Allowed Flow with PAT Interface
The following example shows an allowed flow that uses the PAT interface. The output interface IP address is 22.214.171.124. The user is authenticated as User A. No ACLs are specified; however, the flow is outbound, so it is allowed by default. According to Figure 1 and the description provided, a flow creation event would be issued.
Figure 1 Example of an Allowed Flow with a PAT Interface
The resulting NSEL record would include the following fields and values:
Example 2: Denied Flow on Egress with PAT Interface
The following example shows a denied flow through an egress ACL that uses the PAT interface. The output interface IP address is 126.96.36.199. The user is authenticated as User A. An input ACL (foo) allows the flow, but an output ACL (bar) denies the flow. The input ACL (foo) is specified with an object group, as shown in the following example:hostname# object-group network host_grp_1network-object host 188.8.131.52network-object host 184.108.40.206hostname (config)# access-list foo extended permit tcp object-group host_grp_1 any eq wwwhostname (config)# access-list bar extended deny tcp any anyhostname (config)# access-group foo in interface insidehostname (config)# access-group bar out interface outside
According to Figure 1 and the description provided, a flow denied event would be issued.
The resulting NSEL record would include the following fields and values:
Decoding Device Fields Through the CLI
To decode some of the field values that the ASA populates, direct interaction with the device may be required. We recommend that you use a dynamic mechanism such as expect scripts to obtain the required information from the CLI of the device that issued the event.
The device supports console, Telnet, and SSH secure shell access; however, SSH is the recommended method because of performance and security. The following sections describe fields that you need to decode, based on interaction with the ASA, and includes the following topics:
Interface ID Fields
You can also decode the Interface ID fields using SNMP GET requests from the device interface MIB. This is the only field that has MIB support.
You may use the show interface detail command to obtain a list of all the interfaces on the device. This output includes a line under each interface that corresponds to the Interface ID value sent in the NetFlow fields. In the following example, the interface number is 8.hostname(config)# show interface filter-outside detailInterface GigabitEthernet4/3 "filter-outside", is up, line protocol is upHardware is i82571EB 4CU rev06, BW 1000 Mbps, DLY 10 usecAuto-Duplex(Full-duplex), Auto-Speed(100 Mbps)MAC address 0015.1715.59c7, MTU 1500IP address 220.127.116.11, subnet mask 255.255.255.224532594 packets input, 88376018 bytes, 0 no bufferReceived 3 broadcasts, 0 runts, 0 giants0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort0 L2 decode drops675393 packets output, 53208679 bytes, 0 underruns0 output errors, 0 collisions, 0 interface resets0 late collisions, 0 deferred0 input reset drops, 0 output reset dropsinput queue (curr/max packets): hardware (36/511) software (0/0)output queue (curr/max packets): hardware (59/68) software (0/0)Traffic Statistics for "filter-outside":532594 packets input, 78636500 bytes675393 packets output, 40866215 bytes10837 packets dropped1 minute input rate 0 pkts/sec, 0 bytes/sec1 minute output rate 0 pkts/sec, 0 bytes/sec1 minute drop rate, 0 pkts/sec5 minute input rate 0 pkts/sec, 0 bytes/sec5 minute output rate 0 pkts/sec, 0 bytes/sec5 minute drop rate, 0 pkts/secControl Point Interface States:Interface number is 8Interface config status is activeInterface state is active
ACL ID Fields
The 12-byte raw ACL ID must be divided into its three constituent parts, as follows:
•The first four bytes are the ACL Name ID.
•The next four bytes are the ACL Entry ID (ACE)/Object-Group ID.
•The final four bytes are the Extended ACL Entry ID.
These individual values can be looked up in the output of the show access-list command from the ASA. The ACL Name ID is at the end of the ACL first line in this output. The ACE ID is at the end of each individual ACL entry line.
Note If you use an object-group in an access list, then the second four-byte ID is not actually the ACE ID; it is the Object-Group ID. The Extended ACE ID (the final four-byte part) refers to the actual individual ACL Entry ID. The following example shows these entries:
hostname(config)# show access-listaccess-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)alert-interval 300access-list foo; 2 elements; name hash: 0x102154c1access-list foo line 1 extended permit tcp object-group host_grp_1 any eq www 0xd0e5806eaccess-list foo line 1 extended permit tcp host 18.104.22.168 any eq www (hitcnt=4) 0x7e5ad93baccess-list foo line 1 extended permit tcp host 22.214.171.124 any eq www (hitcnt=0) 0xe0c1846baccess-list bar; 1 elements; name hash: 0x5da9bb69access-list bar line 1 extended deny tcp any any (hitcnt=41) 0x84434b4b
This example is similar to the example shown in Example 2: Denied Flow on Egress with PAT Interface. In the denied flow example, the ACL IDs are divided into their constituent parts as follows:
•NF_F_INGRESS_ACL_ID: InAcl: 0x102154c1d0e5806e7e5ad93b
where 0x102154c1 are the first four bytes, 0xd0e5806e are the second four bytes, and 0x7e5ad93b are the final four bytes.
where 0x5da9bb69 are the first four bytes, 0x84434b4b are the second four bytes, and 0x00000000 are the final four bytes.
Note Each of these IDs corresponds to lines from the show access-list command example.
From these IDs, you can deduce that access-list foo was applied on the input interface, and that access-list bar was applied on the output interface. That information is also available through the show run access-group command, but the added benefit of these ACL IDs is that you can identify the individual ACE that caused the permit or deny action. Because this flow was denied on egress (determined from the extended event code), you know that the ingress ACL ID identifies the ACE line that permitted the flow and that the egress ACL ID identifies the ACE that denied the flow.
You must hard code event codes into the collector, because the ASA only issues four different high-level event types (creation, teardown, denial, and update).
Extended Event Codes
Of the four high-level event codes, only two have extended event codes: the flow denial and flow teardown event types. For the flow denied event, the list of extended event codes in Table 3 should suffice to determine the reason why the flow was denied. However, for the flow teardown event, there are too many event codes to list in this document, and the set of reasons is quite fluid.
Obtaining Documentation and Submitting a Service Request
For information on obtaining documentation, using the Cisco Bug Search Tool (BST), submitting a service request, and gathering additional information, see What's New in Cisco Product Documentation at: http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html.
Subscribe to What's New in Cisco Product Documentation, which lists all new and revised Cisco technical documentation, as an RSS feed and deliver content directly to your desktop using a reader application. The RSS feeds are a free service.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)
© 2013 Cisco Systems, Inc. All rights reserved.