Guest

Cisco IOS Firewall

ZBFW for IOS-XE Configuration Troubleshoot Guide

Document ID: 117721

Updated: Jun 24, 2014

Contributed by Rama Darbha, Namit Agarwal, and Olivier Pelerin, Cisco TAC Engineers.

   Print
ZBFW for IOS-XE Configuration Troubleshoot Guide

Introduction

The Aggregation Services Router (ASR) 1000 is hardware-based router. The software configuration of Cisco IOS-XE® programs the hardware asics (quantum flow processor (QFP), RP, etc) to perform their functionality. This allows for higher throughput and faster functionality. The drawback to this is that it is more challenging to troubleshoot. Traditional IOS commands used to poll current sessions and drop counters via Zone-Based Firewall??(ZBFW) are no longer valid as the drops are no longer in software. This document describes how to best troubleshoot the ASR 1000, with commands that are used to poll the hardware drop counters on the ASR.


Links and Documentation

Command Reference

Cisco ASR 1000 Series Aggregation Services Routers Command References

Cisco IOS XE 3S Command References

Configuration Guide

IOS-XE 15S

Guides Overview

ZBFW Configuration Guide

Datapath Troubleshoot Steps

In order to troubleshoot datapaths, you must identify whether traffic is properly passed through the ASR and IOS-XE code. Specific to firewall features, the datapath troubleshooting follows these steps:

  1. Verify Configuration - Gather the configuration and examine the output to verify the connection.

  2. Verify Connection State - If traffic passes properly, IOS-XE will open up a connection on the ZBFW feature. This connection tracks the traffic and state information between a client and server.

  3. Verify Drop Counters - When traffic does not pass properly, IOS-XE logs a drop counter for any dropped packets. Check this output to isolate the cause of the traffic failure.

  4. Logging - Gather syslogs to provide more granular information on connection builds and packet drops.

  5. Packet Trace Dropped Packets - Use packet tracing to catch dropped packets.

  6. Debugs - Gather debugs is the most verbose option. Debugs can be obtained conditionally to confirm the exact forwarding path for the packets.

Verify Configuration

The output of show tech support firewall is summarized below:

------------------ show clock ------------------
------------------ show version ------------------
------------------ show running-config ------------------
------------------ show parameter-map type inspect ------------------
------------------ show policy-map type inspect ------------------
------------------ show class-map type inspect ------------------
------------------ show zone security ------------------
------------------ show zone-pair security ------------------
------------------ show policy-firewall stats global ------------------
------------------ show policy-firewall stats zone ------------------
------------------ show platform hardware qfp active feature firewall
datapath <submode> -----------
------------------ show platform software firewall RP <submode> ----------------

Verify Connection State

Connection information can be obtained so that all connections on ZBFW are listed. Enter this command:

ASR#show policy-firewall sessions platform           
--show platform hardware qfp active feature firewall datapath scb any any
any any any all any--
[s=session  i=imprecise channel  c=control channel  d=data channel]
 14.38.112.250 41392 14.36.1.206 23 proto 6 (0:0)    [sc]

It shows a TCP telnet connection from 14.38.112.250 to 14.36.1.206.

Note: Be aware that if you run this command, it will take a long time if there are lots of connections on the device. Cisco recommends that you run this command with specific filters as outlined here.

The connection table can be filtered down to a specific source or destination address. Use filters after the platform submode. The options to filter are:

radar-ZBFW1#show policy-firewall sessions platform ?
  all                     detailed information
  destination-port        Destination Port Number
  detail                  detail on or off
  icmp                    Protocol Type ICMP
  imprecise               imprecise information
  session                 session information
  source-port             Source Port
  source-vrf              Source Vrf ID
  standby                 standby information
  tcp                     Protocol Type TCP
  udp                     Protocol Type UDP
  v4-destination-address  IPv4 Desination Address
  v4-source-address       IPv4 Source Address
  v6-destination-address  IPv6 Desination Address
  v6-source-address       IPv6 Source Address
  |                       Output modifiers
  <cr>

This connection table is filtered so only connections that are sourced from 14.38.112.250 are displayed:

ASR#show policy-firewall sessions platform v4-source-address 14.38.112.250
--show platform hardware qfp active feature firewall datapath scb 14.38.112.250 any any any
any all any --
[s=session  i=imprecise channel  c=control channel  d=data channel]
 14.38.112.250 41392 14.36.1.206 23 proto 6 (0:0)    [sc]

When the connection table is filtered, the detailed connection information can be obtained for a more comprehensive analysis. To display this output, use the detail keyword.

ASR#show policy-firewall sessions platform v4-source-address 14.38.112.250 detail
--show platform hardware qfp active feature firewall datapath scb 14.38.112.250 any any any
any all any detail--
[s=session  i=imprecise channel  c=control channel  d=data channel]
 14.38.112.250 41426 14.36.1.206 23 proto 6 (0:0)       [sc]
 pscb : 0x8c5d4f20,  bucket : 64672, fw_flags: 0x204 0x20419441,
        scb state: active, scb debug: 0
 nxt_timeout: 360000, refcnt: 1,  ha nak cnt: 0,  rg: 0, sess id: 117753
 hostdb: 0x0, L7: 0x0, stats: 0x8e118e40, child: 0x0
 l4blk0: 78fae7a7 l4blk1: e36df99c l4blk2: 78fae7ea l4blk3: 39080000
 l4blk4: e36df90e l4blk5: 78fae7ea l4blk6: e36df99c l4blk7: fde0000
 l4blk8: 0 l4blk9: 1
 root scb: 0x0 act_blk: 0x8e1115e0
 ingress/egress intf: GigabitEthernet0/0/2 (1021), GigabitEthernet0/0/0 (131065)
 current time 34004163065573 create tstamp: 33985412599209 last access: 33998256774622
 nat_out_local_addr:port: 0.0.0.0:0 nat_in_global_addr:port: 0.0.0.0:0
 syncookie fixup: 0x0
 halfopen linkage: 0x0 0x0
 cxsc_cft_fid: 0x0
 tw timer: 0x0 0x0 0x372ba 0x1e89c181
 Number of simultaneous packet per session allowed: 25
    bucket 125084 flags 1 func 1 idx 8 wheel 0x8ceb1120

Verify Drop Counter

The drop counter output changed during XE 3.9. Before XE 3.9, the firewall drop reasons were very generic. After XE 3.9, the firewall drop reasons were extended to become more granular.

To verify drop counters, perform two steps:

  1. Confirm the global drop counters in IOS-XE. These counters will show what feature has dropped the traffic. Examples of features include Quality of Service (QoS), Network Address Translation (NAT), Firewall, etc.

  2. Once the subfeature has been identified, query the granular drop counters offered by the subfeature. In this guide, the subfeature being analyzed is the Firewall feature.

Global Drop Counters on QFP

The basic command to rely on provides all the drops across the QFP:

Router#show platform hardware qfp active statistics drop

This command shows you the generic drops globally across the QFP. These drops can be on any feature. Some example features are:

Ipv4Acl
Ipv4NoRoute
Ipv6Acl
Ipv6NoRoute
NatIn2out
VfrErr
...etc

To see all drops, include counters that have a value of zero, use the command:

show platform hardware qfp active statistics drop all

To clear the counters, use this command. It clears the output after it shows it to the screen. This command is clear on read, so the output is reset to zero after it is displayed to the screen.

show platform hardware qfp active statistics drop clear

Here is a list of basic firewall specific drop counters that are provided when this command is issued:

FirewallBackpressure
FirewallInvalidZone
FirewallL4Insp
FirewallNoForwardingZone
FirewallNonsession
FirewallPolicy
FirewallL4
FirewallL7
FirewallNotInitiator
FirewallNoNewSession
FirewallSyncookieMaxDst
FirewallSyncookie
FirewallARStandby

Firewall Feature Drop Counters on QFP

The limitation with the QFP global drop counter is that there is no granularity in the drop reasons, and some of the drop reasons such as Firewall4 get so overloaded to the point that it is of little use for troubleshooting. This has since been enhanced in IOS-XE 3.9 (15.3(2)S), where the Firewall feature drop counters were added. This gives a much more granular set of drop reasons:

ASR#show platform hardware qfp active feature firewall drop all
-------------------------------------------------------------------------------
Drop Reason                                                             Packets
-------------------------------------------------------------------------------
Invalid L4 header                                                             0
Invalid ACK flag                                                              0
Invalid ACK number                                                            0
Invalid TCP initiator                                                         0
SYN with data                                                                 0
Invalid window scale option                                                   0
Invalid Segment in SYNSENT state                                              0
Invalid Segment in SYNRCVD state                                              0
TCP out of window                                                             0
TCP extra payload after FIN                                                   0
Invalid TCP flags                                                             0
Invalid sequence number                                                       0
Retrans with invalid flags                                                    0
TCP out-of-order segment                                                      0
SYN flood drop                                                                0
Internal Error - synflood hostdb alloc fail                                   0
Synflood blackout drop                                                        0
Half-open session limit exceed                                                0
Too many packet per flow                                                      0
Too many ICMP error packets per flow                                          0
Unexpect TCP payload in 3-way handshake                                       0
Internal error - Undefined direction                                          0
SYN inside current window                                                     0
RST inside current window                                                     0
Stray Segment                                                                 0
RST sent to responder                                                         0
ICMP Internal Error - Missing NAT info                                        0
ICMP Internal Error - Fail to get ErrPkt                                      0
ICMP Internal Error - Fail to get Stats blk                                   0
ICMP Internal Error - direction undefined                                     0
ICMP packet rcvd in SCB close state                                           0
Missed IP hdr in ICMP packet                                                  0
ICMP Error Pkt has no IP or ICMP                                              0
ICMP Error Pkt exceeds burst limit                                            0
ICMP Unreachable packet exceeds limit                                         0
ICMP Error Pkt invalid sequence                                               0
ICMP Error Pkt invalid ACK                                                    0
ICMP Error Pkt too short                                                      0
Exceed session limit                                                          0
Packet rcvd in SCB close state                                                0
Packet rcvcd after CX requested teardown                                      0
CXSC not running                                                              0
Zone-pair without policy                                                      0
Same zone without Policy                                                      0
ICMP Error and Policy not present                                             0
Classification Failed                                                         0
Policy drop for non tcp/udp/icmp                                              0
PAM lookup action drop                                                        0
ICMP Error Packet TCAM missed                                                 0
Security policy misconfigure                                                  0
Internal Error - Get stat blk failed                                          0
SYN cookie max dst reached                                                    0
Internal Error - syncookie dsttbl allocation failed                           0
SYN cookie being triggered                                                    0
Fragment drop                                                                 0
Policy drop due to classification result                                      0
ICMP policy drop due to classification result                                 0
L7 segmented packet not allow                                                 0
L7 fragmented packet not allow                                                0
L7 unknown proto type                                                         0
L7 inspection returns drop                                                    0
L7 sub-channel promotion failed (no zone pair)                                0
L7 sub-channel promotion failed (no policy)                                   0
Firewall Create Session fail                                                  0
Firewall No new session allow                                                 0
Not a session initiator                                                       0
Firewall invalid zone                                                         0
Firewall AR standby                                                           0
Firewall no forwarding allow                                                  0
Firewall back pressure                                                        0
Firewall no broadcast allow                                                   0
Catch All                                                                     0

Logging

ASR logging functionality generates syslogs to record dropped packets. These syslogs provide more details on why the packet was dropped. There are two types of sysloggings:

  1. Local buffered syslogging

  2. Remote high speed logging

Local Buffered Syslogging

To isolate the cause of the drops, you can use generic ZBFW troubleshooting, such as enabling log drops. There are two ways to configure packet drop logging.

Method 1: Use inspect-global parameter-map to log all dropped packets.

parameter-map type inspect-global
     log dropped-packets

 Method 2: Use custom inspect parameter-map to log dropped packets for only specific class.

parameter-map type inspect LOG_PARAM
     log dropped-packets
! policy-map type inspect ZBFW_PMAP class type inspect ZBFW_CMAP inspect LOG_PARAM

These messages are sent to the log or console depending on how the ASR is configured for logging. Here is an example of a drop log message.

*Apr  8 13:20:39.075: %IOSXE-6-PLATFORM: F0: cpp_cp: QFP:0.0 Thread:103
TS:00000605668054540031 %FW-6-DROP_PKT: Dropping tcp pkt from GigabitEthernet
0/0/2 14.38.112.250:41433 => 14.36.1.206:23(target:class)-(INSIDE_OUTSIDE_ZP:
class-default)
due to Policy drop:classify result with ip ident 11579 tcp flag 0x2, seq
2014580963, ack 0

Limitations of Local Buffered Syslogging

  1. These logs are rate limited as per bugID CSCud09943.

  2. These logs may not print unless specific configuration is applied. For example, packets dropped by class-default packets will not logged unless the log keyword is specified:
policy-map type inspect ZBFW_PMAP
  class class-default
   drop log

Remote High Speed Logging

High speed logging (HSL) generates syslogs directly from the QFP and sends it to the configured netflow HSL collector. This is the recommended logging solution for ZBFW on ASR.

For HSL, use this configuration:

parameter-map type inspect inspect-global
   log template timeout-rate 1
   log flow-export v9 udp destination 1.1.1.1 5555

In order to use this configuration, a netflow collector capable of netflow version 9 is required. This is detailed in:

Configuration Guide: Zone-Based Policy Firewall, Cisco IOS XE Release 3S (ASR 1000) Firewall High-Speed Logging

Packet Tracing Using Conditional Matching

Enable packet tracing by turning on conditional debugs and then enabling packet tracing for these features:

ip access-list extended CONDITIONAL_ACL
  permit ip host 10.1.1.1 host 192.168.1.1
  permit ip host 192.168.1.1 host 10.1.1.1
!
debug platform condition feature fw dataplane submode all level info
debug platform condition ipv4 access-list CONDITIONAL_ACL both

Note: The match condition can use the IP address directly, as an ACL is not necessary. This will match as source or destination allowing for bidirectional traces. This method can be used if altering the configuration is not allowed.

debug platform condition ipv4 address 192.168.1.1/32

Turn on the packet-tracing feature:

debug platform packet-trace copy packet both
debug platform packet-trace packet 16
debug platform packet-trace drop
debug platform packet-trace enable

 Note: There are two ways to use this feature:

  1. If you use the command debug platform packet-trace drop, it will trace only the dropped packets.

  2. If you exclude the command debug platform packet-trace drop, it will trace any packet matching the condition, including ones that are inspected/passed by the device.

Turn on conditional debugs:

debug platform condition start

Run the test, then turn off debugs:

debug platform condition stop

Now the information can be displayed to the screen. In this example, Internet Control Message Protocol (ICMP) packets were dropped due to a firewall policy:

Router#show platform packet-trace statistics
Packets Summary
  Matched  2
  Traced   2
Packets Received
  Ingress  2
  Inject   0
Packets Processed
  Forward  0
  Punt     0
  Drop     2
    Count       Code  Cause
    2           183   FirewallPolicy
  Consume  0

Router#show platform packet-trace summary
Pkt   Input            Output           State  Reason
0     Gi0/0/2          Gi0/0/0          DROP   183 (FirewallPolicy)
1     Gi0/0/2          Gi0/0/0          DROP   183 (FirewallPolicy)

Router#show platform packet-trace packet 0
Packet: 0           CBUG ID: 2980
Summary
  Input     : GigabitEthernet0/0/2
  Output    : GigabitEthernet0/0/0
  State     : DROP 183 (FirewallPolicy)
  Timestamp
    Start   : 1207843476722162 ns (04/15/2014 12:37:01.103864 UTC)
    Stop    : 1207843477247782 ns (04/15/2014 12:37:01.104390 UTC)
Path Trace
  Feature: IPV4
    Source      : 10.1.1.1
    Destination : 192.168.1.1
    Protocol    : 1 (ICMP)
  Feature: ZBFW
    Action  : Drop
    Reason  : ICMP policy drop:classify result
    Zone-pair name  : INSIDE_OUTSIDE_ZP
    Class-map name  : class-default
Packet Copy In
  c89c1d51 5702000c 29f9d528 08004500 00540000 40004001 ac640e26 70fa0e24
  01010800 172a2741 00016459 4d5310e4 0c000809 0a0b0c0d 0e0f1011 12131415
Packet Copy Out
  c89c1d51 5702000c 29f9d528 08004500 00540000 40003f01 ad640e26 70fa0e24
  01010800 172a2741 00016459 4d5310e4 0c000809 0a0b0c0d 0e0f1011 12131415

The show platform packet-trace packet <num> decode command decodes the packet header information and contents. This feature was introduced in XE 3.11:

Router#show platform packet-trace packet all decode
Packet: 0           CBUG ID: 2980
Summary
  Input     : GigabitEthernet0/0/2
  Output    : GigabitEthernet0/0/0
  State     : DROP 183 (FirewallPolicy)
  Timestamp
    Start   : 1207843476722162 ns (04/15/2014 12:37:01.103864 UTC)
    Stop    : 1207843477247782 ns (04/15/2014 12:37:01.104390 UTC)
Path Trace
  Feature: IPV4
    Source      : 10.1.1.1
    Destination : 192.168.1.1
    Protocol    : 1 (ICMP)
  Feature: ZBFW
    Action  : Drop
    Reason  : ICMP policy drop:classify result
    Zone-pair name  : INSIDE_OUTSIDE_ZP
    Class-map name  : class-default
Packet Copy In
  c89c1d51 5702000c 29f9d528 08004500 00540000 40004001 ac640e26 70fa0e24
  01010800 172a2741 00016459 4d5310e4 0c000809 0a0b0c0d 0e0f1011 12131415
  ARPA
    Destination MAC     : c89c.1d51.5702
    Source MAC          : 000c.29f9.d528
    Type                : 0x0800 (IPV4)
  IPv4
    Version             : 4
    Header Length       : 5
    ToS                 : 0x00
    Total Length        : 84
    Identifier          : 0x0000
    IP Flags            : 0x2 (Don't fragment)
    Frag Offset         : 0
    TTL                 : 64
    Protocol            : 1 (ICMP)
    Header Checksum     : 0xac64
    Source Address      : 10.1.1.1
    Destination Address : 192.168.1.1
  ICMP
    Type                : 8 (Echo)
    Code                : 0 (No Code)
    Checksum            : 0x172a
    Identifier          : 0x2741
    Sequence            : 0x0001
Packet Copy Out
  c89c1d51 5702000c 29f9d528 08004500 00540000 40003f01 ad640e26 70fa0e24
  01010800 172a2741 00016459 4d5310e4 0c000809 0a0b0c0d 0e0f1011 12131415
  ARPA
    Destination MAC     : c89c.1d51.5702
    Source MAC          : 000c.29f9.d528
    Type                : 0x0800 (IPV4)
  IPv4
    Version             : 4
    Header Length       : 5
    ToS                 : 0x00
    Total Length        : 84
    Identifier          : 0x0000
    IP Flags            : 0x2 (Don't fragment)
    Frag Offset         : 0
    TTL                 : 63
    Protocol            : 1 (ICMP)
    Header Checksum     : 0xad64
    Source Address      : 10.1.1.1
    Destination Address : 192.168.1.1
  ICMP
    Type                : 8 (Echo)
    Code                : 0 (No Code)
    Checksum            : 0x172a
    Identifier          : 0x2741
    Sequence            : 0x0001

Debugs

Conditional Debugs

In XE3.10, conditional debugs will be introduced. Conditional statements can be used to ensure the ZBFW feature only logs debug messages relevant to the condition. Conditional debugs use Access Control Lists (ACLs) to restrict logs that match the ACL elements. Also, prior to XE3.10, the debug messages were more difficult to read. The debug output was improved in XE3.10 to make them easier to understand.

To enable these debugs, enter this command:

debug platform condition feature fw dataplane submode [detail | policy | layer4 | drop]
debug platform condition ipv4 access-list <ACL_name> both
debug platform condition start

Notice that the condition command must be set via an ACL and directionality. The conditional debugs will not be implemented until the they are started by using the command debug platform condition start. In order to turn off conditional debugs, use the command debug platform condition stop.

debug platform condition stop

In order to turn off conditional debugs, DO NOT use the command undebug all. In order to turn off all conditional debugs, enter this command:

ASR#clear platform condition all

Prior to XE3.14, ha and event debugs are not conditional. As a result, if you use the command debug platform condition feature fw dataplane submode all causes all logs to be created, independent of the condition selected here. This could create additional noise making debugging difficult.

By default, the conditional logging level is info. To increase/decrease the level of logging, use the command:

debug platform condition feature fw dataplane submode all [verbose | warning]

Gathering and Viewing Debugs

Debug files do not print to the console or monitor. All debugs are written to the harddisk of the ASR. Debugs are written to the harddisk under the folder tracelogs with the namecpp_cp_F0-0.log.<date>. To view the file where debugs are written, use the output:

ASR# cd harddisk:
ASR# cd tracelogs
ASR# dir cpp_cp_F0*
Directory of harddisk:/tracelogs/cpp_cp_F0*

Directory of harddisk:/tracelogs/

3751962  -rwx     1048795  Jun 15 2010 06:31:51 +00:00  cpp_cp_F0-0.log.5375.20100615063151
3751967  -rwx     1048887  Jun 15 2010 02:18:07 +00:00  cpp_cp_F0-0.log.5375.20100615021807
39313059840 bytes total (30680653824 bytes free)

Each debug file will be stored as a cpp_cp_F0-0.log.<date> file. These are regular text files that can be copied off the ASR using TFTP. The log file max on the ASR is 1Mb. After 1Mb, the debugs are written to a new log file. That is why each log file is timestamped to indicate the start of the file.

Log files may exist in the following locations:

harddisk:/tracelogs/
bootflash:/tracelogs/

Since log files are only displayed after they are rotated, the log file can be manually rotated using this command:

ASR#test platform software trace slot f0 cpp-control-process rotate

This will immediately create a cpp_cp log file and start a new one on the qfp. For example:

ASR#test platform software trace slot f0 cpp-control-process rotate
  Rotated file from: /tmp/fp/trace/stage/cpp_cp_F0-0.log.7311.20140408134406,
Bytes: 82407, Messages: 431

ASR#more tracelogs/cpp_cp_F0-0.log.7311.20140408134406
04/02 10:22:54.462 : btrace continued for process ID 7311 with 159 modules
04/07 16:52:41.164 [cpp-dp-fw]: (info): QFP:0.0 Thread:110 TS:00000531990811543397
:FW_DEBUG_FLG_HA:[]: HA[1]: Changing HA state to 9
04/07 16:55:23.503 [cpp-dp-fw]: (info): QFP:0.0 Thread:120 TS:00000532153153672298
:FW_DEBUG_FLG_HA:[]: HA[1]: Changing HA state to 10
04/07 16:55:23.617 [buginf]: (debug): [system] Svr HA bulk sync CPP(0) complex(0)
epoch(0) trans_id(26214421) rg_num(1)

This command allows the debug files to merge into a single file for easier processing. It will merge all files in the directory and interlace them based on time. This can help when the logs are very verbose and are created across multiple files:

ASR#request platform software trace slot rp active merge target bootflash:MERGED_OUTPUT.log
Creating the merged trace file: [bootflash:MERGED_OUTPUT.log]
including all messages

Done with creation of the merged trace file: [bootflash:MERGED_OUTPUT.log]
Updated: Jun 24, 2014
Document ID: 117721