Adaptive Security Appliance Hot Issues from Cisco TAC

Traceback: timer assert due to nf_block timer race condition, Fixed CSCtz41928

Symptom: ASASM crash happened running code version 8.5.1 with netflow enabled.
Conditions: ASASM running code version 8.5.1 with netflow enabled
Workaround: None

ASA crashes in Thread Name: ci/console after write erase command, Fixed CSCuf29783

Symptom: ASA crashes in Thread Name: ci/console when "write erase" command is issued
Conditions: Unknown
Workaround: None

Crypto IPSec SA's are created by dynamic crypto map for static peers, Fixed CSCuc75090

Symptom: When a static VPN peer adds any traffic to the crypto ACL, an SA is built even though the IP pair is not allowed in the crypto acl at the main side. Those SA's are eventually matched and setup by the dynamic crypto map instance.
Conditions: This was a intended design since day one that enabled customers to fall through in case of static crypto map didn't provide a needed crypto services. The SA need to be initiated from a statically configured peer and a dynamic crypto map instance must be configured on the receiving end.
Workaround: N/A PSIRT Evaluation: The Cisco PSIRT has assigned this bug the following CVSS version 2 score. The Base and Temporal CVSS scores as of the time of evaluation are 3.5/3.2: https://intellishield.cisco.com/security/alertmanager/cvssCalculator.do?dispatch=1&version=2&vector=AV:N/AC:M/Au:S/C:P/I:N/A:N/E:F/RL:W/RC:C No CVE ID has been assigned to this issue. Additional information on Cisco's security vulnerability policy can be found at the following URL: http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html

Crypto accelerator resets with error code 23, Fixed CSCue67198

Symptom: Crypto chip resets with MAC's native IPSEC client. %ASA-4-402124: CRYPTO: The ASA hardware accelerator encountered an error (HWErrAddr= 0x7693AB40, Core= 0, HwErrCode= 23, IstatReg= 0x40008, PciErrReg= 0x0, CoreErrStat= 0xC3, CoreErrAddr= 0x8EC19940, Doorbell Size[0]= 2048, DoorBell Outstanding[0]= 0, Doorbell Size[1]= 0, DoorBell Outstanding[1]= 0, SWReset= 3)
Conditions: Seen on ASA5585 running 9.1.1.2
Workaround: None.

ASA writes past end of file system then can't boot, Fixed CSCuc98398

Symptom: After upgrading the ASA OS the device does not boot successfully, and will continually loop the unsuccessful boot sequence. The following will be seen on the console of the ASA (The ASA and image file will vary): ----------------------------------------------------------------------------------- Evaluating BIOS Options ... Launch BIOS Extension to setup ROMMON Cisco Systems ROMMON Version (1.0(12)13) #0: Thu Aug 28 15:55:27 PDT 2008 Platform ASA5505 Use BREAK or ESC to interrupt boot. Use SPACE to begin boot immediately. Launching BootLoader... Boot configuration file contains 1 entry. Loading disk0:/asa844-9-k8.bin... Booting... Platform ASA5505 Loading... IO memory blocks requested from bigphys 32bit: 9672 ## APPLIANCE REBOOTS AUTOMATICALLY HERE ## -----------------------------------------------------------------------------------
Conditions: Cisco ASA where the disk (Compact Flash) is already close to full or is fragmented from frequent use and a new version of the OS is saved on the disk (without removing any files) and the new file is made the boot file in the configuration.
Workaround: Delete the bad file from flash, as well as any other images that are no longer in use to free up more space on the flash. Then, re-download the new file to flash - or - 1) Copy all the files off of the ASA's disk 2) Format the disk: 3) Copy the files back onto the disk, starting with the OS image you wish the ASA to boot. The second procedure (involving the re-format) is the preferred workaround, as it places the ASA image towards the beginning of the filesystem, making the chances of encountering this problem much less.

ASA reloads after issuing "show inventory" command, Fixed CSCud20887

Symptom: An Adaptive Security Appliance (ASA) 5505 or ASA Services Module (ASASM) reloads unexpectedly when issuing the show inventory command.
Conditions: ASA5505 or ASASM running 8.6(1.5) and later, 9.0(1.1) and later, or 9.1(1.1) and later software.
Workaround: None.

'show failover history' should indicate if 'write standby' was issued, Open CSCuc63634

Symptom: When issuing a 'write standby' command from the ACTIVE ASA firewall, the STANDBY firewall will re-sync its configuration citing 'configuration mismatch' as the reason. This need to be more verbose and specific about if it was caused by a user issuing 'write standby' or by some other configuration related issue.

ASA-L2TP-IPSec: PAT xlate for UDP1701 hijacks incoming L2TP conns, Open CSCug83036

Symptom: ASA drops packets from remote L2TP IPSec clients causing loss of data plane connectivity.
Conditions: L2TP IPSec configuration with dynamic/static xlate for UDP 1701
Workaround: Configure a static nat rule of the type (outside,inside) to force the ASA to own incoming L2TP connections (Example: nat (outside,inside) source static any any destination static asa-outside-ip asa-outside-ip service l2tp-port l2tp-port

SIP sessions cause CPU hogs and high CPU on standby ASA, CSCuc71272

SYMPTOM: CPU on standby ASA spikes to 100% CONDITIONS: Large number of SIP sessions through ASA to multiple destination IP addresses WORKAROUND: Use an inspection policy to limit the number of conns for SIP traffic PSIRT Evaluation: The Cisco PSIRT has assigned this bug the following CVSS version 2 score. The Base and Temporal CVSS scores as of the time of evaluation are 5.0/4.8: https://intellishield.cisco.com/security/alertmanager/cvssCalculator.do?dispatch=1&version=2&vector=AV:N/AC:L/Au:N/C:N/I:N/A:P/E:F/RL:U/RC:C Additional details about the vulnerability described here can be found at: http://tools.cisco.com/security/center/content/CiscoSecurityNotice/CVE-2012-5415 Additional information on Cisco's security vulnerability policy can be found at the following URL: http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html

%ASA-3-210007: LU allocate xlate failed on Standby unit, Fixed CSCub94479

Symptom: ASA, running 8.4.3, produces "%ASA-3-210007: LU allocate xlate failed" error message on Standby unit even if the memory has enough free space.
Conditions: unknown
Workaround: unknown

16k blocks near exhaustion - process emweb/https (webvpn), Fixed CSCue05458

Symptom: Syslog messages 321007 are seen from ASA running WebVPN: ASA-3-321007 System is low on free memory blocks of size 16384 (xx CNT out of yyyy MAX) The output of command show blocks indicates near exhaustion of 16K blocks and possible other large sizes as well, here an example with LOW mark at 0 indicating that at some point all blocks were allocated: SIZE MAX LOW CNT 0 700 570 700 [...] 2560 2052 2040 2052 4096 161 0 161 8192 142 0 130 16384 1053 0 13 65536 16 2 15
Conditions: Clientless SSL is used.
Workaround: none

Traceback: deadlock between syslog lock and host lock, Fixed CSCuc74758

Symptom: ASA traceback in thread name: Datapath
Conditions: In a rare corner case, the ASA might traceback and reload due to sysloging functions
Workaround: Disable Syslog or reduce the syslog level.

Failover Unit Stuck in Cold Standby After Boot Up, Fixed CSCua13405

Symptom: An Adaptive Security Appliance (ASA) or ASA Services Module (ASASM) configured for failover may get stuck in Cold Standby state or display HA State Progression Failure errors when establishing failover communication with the peer.
Conditions: Failover control channel communication is unstable or very aggressive failover unit poll timers are configured. The issue is still under active investigation, so the conditions may be updated.
Workaround: Manually reload the ASA to recover. Note To resolve this problem, both primary and secondary ASA must be upgraded to an image with the fix.

ASA 5585 traceback assert on bfm_to_string, Open CSCug51957
Symptom: ASA 5585 traceback under thread name DATAPATH
Conditions: Botnet is enabled
Workaround: Temporary disable botnet if possible

LU allocate xlate failed (for NAT with service port), Fixed CSCue32221
Symptom: we see the following logs on the standby : %ASA-3-210007: LU allocate xlate failed Failed to rep xlate for np/port/id/24/401 x.x.x.x /52660 - np/port/id/24/969 x.x.x.x/10051, flg: 2003000 2002002 No users are affected.
Conditions: Failover along with twice nat commands configured
Workaround: N/A

Traceback during certificate operation in IKEv2 EAP processing, Fixed CSCtq33081
Symptom: The ASA could reload when processing an Anyconnect connection with IKEv2 where any certificate operations are possible like tunnel group lookups, certificate validation, etc.
Conditions: AnyConnect IKEv2 connection to the ASA that requires certificate operations during authentication of the client.
Workaround: upgrade

ASA crashes in thread name "ssh" while running packet-tracer, CSCug85087
Symptom: ASA running 8.6(1)5 crashes in thread name "ssh" while running packet-tracer
Conditions: N/A
Workaround: There is no workaround at this time

ASA shared port-channel subinterfaces and multicontext traffic failure, Fixed CSCue59676
Symptom: An ASA configured in multi context mode, with port-channels divided into subinterfaces, may experience an issue where traffic to certain contexts will fail if the port-channel has more than one active TenGigabitEthernet member.
Conditions: This was first observed in an environment using 250 contexts with more than 300 subinterfaces.
Workaround: Reduce the number of contexts or subinterfaces. Deleting the context experience the problem and reconfiguring it sometimes resolves the issue for that context, but the problem may then move to another context.

assertion "t->stack[0] == STKINIT" failed: file "thread.c", line 774, Open CSCug82536
Symptom: Under certain circumstances, the ASA may traceback when users are logging into webvpn.
Conditions: The traceback occurs only when users are logging into webvpn.
Workaround: None.

Multicast,Broadcast traffic is corrupted on a shared interface on 5585, Fixed CSCue62422
Symptom: ASA does not update its ARP table for GARP entries received from other contexts on shared interface
Conditions: -mac-address auto prefix is changed to mac-address auto -manual change of MAC address on interface
Workaround: Clear arp on affected contexts More Info:

Traceback in Thread Name: Dispatch Unit, Open CSCug60235
Symptom: ASA may generate a traceback and reload in the dispatch unit thread
Conditions: This issue has been seen on ASA 8.4(5), other versions may also be affected
Workaround: No known workaround at this time

ASA drops some CX/CSC inspected HTTP packets due to PAWS violation, Fixed CSCug19491
Symptom: Certain HTTP connections might experience slowdowns or fail to complete if the packets are inspected by the CX module. HTTP packets might be dropped by the ASA for the ASP drop reason "TCP packet failed PAWS test (tcp-paws-fail)"
Conditions: All of the following conditions must be met to encounter this problem: 1) The traffic flow must be subjected to inspection by the ASA CX module 2) The connection must be HTTP over TCP 3) The HTTP GET message must be so big as to become segmented into multiple TCP packets. This might occur if the cookie values in the get are very long
Workaround: Using the ASA's modular policy framework, disable TCP timestamps for the connections: ! access-list http-traffic extended permit tcp any any eq www ! class-map http-class match access-list http-traffic ! tcp-map TCP-map-timestamps tcp-options timestamp clear ! policy-map global_policy ... class http-class set connection advanced-options TCP-map-timestamps !

ASA : "ERROR:Unable to create router process" & routing conf is lost, Open CSCug66457
Symptom: Standby ASA reports below error messages and loses dynamic routing configuration.
Conditions: ASA running 9.x or higher
Workaround: Issue is only seen during startup/reload of ASA. Issuing "write mem" or "write standby" resolves the issue.
Further Description ASA(config)# Beginning configuration replication from mate. ERROR: Unable to create router process, cleanup in progress ASA(config)# sh run | inc router ASA(config)#

ASA: ScanSafe for https may not allow 302 page moved response, Terminated CSCud91387
Symptom: ScanSafe for https may not allow 302 page moved response from the tower back to the client on the inside.
Conditions: ASA must be running 9.0.1 code or above with Scan Safe configured.
Workaround: None - Bypass ScanSafe for https traffic. OR Follow this link below and create a certificate on scan center and follow the steps so the blocked page can be seen on the client PC. http://www.cisco.com/en/US/docs/security/web_security/scancenter/administrator/guide/b_ScanCenter_Administrator_Guide_chapter_010010.html
Further Problem Description: 1. ScanSafe policy is configured to Block all the https traffic to Server S1 Https. 2.Client initiates the connection to Server S1 and send the Hello request ASA will hold the hello request and construct CONNECT method with X-Headers ScanSafe tower receives the CONNECT method and parse the information in the X-Headers. 3. When ScanSafe receives the CONNECT request, the IP address is checked against internal DBs to ensure policies are adhered to (not the content) Since the destination ip address "S1" in X-Headers (this is actually in the CONNECT and not in the header) is matching with policy "Block all the https traffic to Server S1" the 302 message will be sent back from ScanSafe tower. 4. For CONNECT method to be successful ASA will expect "200" response from server and any other response will be treated as "bad" CONNECT response and the connection will be closed (by the ASA). 5. The response "302" will be treated as error and connection will be closed. Tower will not pass a 200 Response, as this would mean making the request upstream, to which the policy states to block. 6. Now the Https Client will not receive the expected blocked page. This is due to the ASA closing the request as it is expecting a 200 response and not the 302 the tower sends back.

DCERPC Inspect Causes Traceback in DATAPATH Thread Due to a Page Fault, Open CSCug49347
Symptom: An Adaptive Security Appliance (ASA) or ASA Services Modules (ASASM) may reload with a traceback in DATAPATH thread due to "Page Fault" with DCERPC Inspection enabled.
Conditions: DCERPC Inspection engine enabled.
Workaround: Disable DCERPC Inspection by removing all inspect dcerpc statements under the Modular Policy Framework (MPF) configuration.

ASA upgrade from 8.4 to 9.0 changes context's mode to router, Fixed CSCug58801
Symptom: ASA fail-over pair running v8.4 in transparent mode and multiple mode trying to do a zero downtime upgrade to v9.0. After standby upgrades to v9.0 and joins the fail-over, firewall operation mode changes to router on the standby.
Conditions: ASA fail-over pair running v8.4 in transparent mode and multiple mode trying to do a zero downtime upgrade to v9.0.
Workaround: Upgrade the ASA fail-over pair from v8.4 to v9.0 by taking a down time and rebooting both the ASA's simultaneously.

Prioritize Failover Control Packets on ASA5585-X CPU Uplinks, Fixed CSCud43999
Symptom: Incoming frames that contain failover control packets may be dropped on Adaptive Security Appliance (ASA) 5585-X under a high volume of other data traffic.
Conditions: Extremely high ingress traffic volume that creates a congestion of the internal CPU uplink subsystem.
Workaround: Limit the traffic rate on the adjacent switch.

IKEv2: ASA does not clear entry from asp table classify crypto, Fixed CSCud28106
Symptom: Intermittently ikev2 Anyconnect clients are no longer able to access internal resources even though the tunnel gets built just fine.
Conditions: 1. ikev2 Anyconnect clients 2. ASA 8.4.x
Workaround: 1. increase the time period before the ip address is reused from the VPN pool 2. disconnect the user from the ASA.

ASA 5585 reloads with traceback in Thread Name: NIC status poll, Fixed CSCtu39738
Symptom: ASA 5585 may go into a boot loop with traceback in Thread Name: NIC status poll Before the box enters the traceback you will see several messages on the console that look like this: INFO: MIGRATION - Saving the startup configuration to file INFO: MIGRATION - Startup configuration saved to file 'flash:8_2_4_0_startup_cfg .sav' *** Output from config line 4, "ASA Version 8.2(4) " .....Failed to change interface status: cannot get channel *** Output from config line 442, "interface GigabitEtherne..." Failed to change interface status: cannot get channel *** Output from config line 443, " shutdown" Failed to change interface status: cannot get channel *** Output from config line 448, "interface GigabitEtherne..." Failed to change interface status: cannot get channel *** Output from config line 449, " shutdown" .Failed to change interface status: cannot get channel *** Output from config line 454, "interface GigabitEtherne..." Failed to change interface status: cannot get channel *** Output from config line 455, " shutdown" Failed to change interface status: cannot get channel Conditions: ASA 5585 only. Running 8.4.2 with an IPS SSP installed in slot 1 Workaround: Remove the IPS SSP from the chassis and the boot loop should end. The trigger for this behavior is related to using the switch on the PSU to power cycle the box. If you have an IPS blade in the chassis and you power cycle the 5585 via the switch on the PSU you may see this behavior. Call TAC to get your IPS SSP replaced. PSIRT Evaluation: The Cisco PSIRT has evaluated this issue and does not meet the criteria for PSIRT ownership or involvement. This issue will be addressed via normal resolution channels. If you believe that there is new information that would cause a change in the severity of this issue, please contact psirt@cisco.com for another evaluation. Additional information on Cisco's security vulnerability policy can be found at the following URL: http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html

Port-Channel Flaps at low traffic rate with single flow traffic, Fixed CSCtz79578
Symptom: Port-Channel flaps continously
Conditions: Observed on ASA 5585-SSP-60 under performance testing for single flow traffic
Workaround: change the channel-group mode to ON PSIRT Evaluation: The Cisco PSIRT has assigned this bug the following CVSS version 2 score. The Base and Temporal CVSS scores as of the time of evaluation are 5/4.8: https://intellishield.cisco.com/security/alertmanager/cvssCalculator.do?dispatch=1&version=2&vector=AV:N/AC:L/Au:N/C:N/I:N/A:P/E:F/RL:U/RC:C CVE ID CVE-2012-2485 has been assigned to document this issue. Additional information on Cisco's security vulnerability policy can be found at the following URL: http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html

ASA 8.4.4.1 Keeps rebooting when FIPS is enabled: FIPS Self-Test failure, Fixed CSCug14707
Symptom: with ASA 8.4.4.1 when you enable FIPS and add some basic configuration, the ASA keeps rebooting, and show this message: Reading from flash... !. Cryptochecksum (unchanged): 4c51bfab f82907ea 60c001b3 ea8ef9e3 INFO: FIPS Power-On Self-Test in process. ........................................................ ^ ERROR: % Invalid input detected at '^' marker. ^ ERROR: % Invalid input detected at '^' marker. ERROR: Crypto Map with tag "__fipstest_map" does not exist. ERROR: Transform set __fipstest_ts not found ERROR: % Incomplete command ERROR: access-list <__fipstest_acl> does not exist ERROR: FIPS Self-Test failure, fipsPostBypassTest [0:0:-1:-1:0] The same happens with 8.3(2)-13 even if this release shoud fix this defect.
Conditions: Enable FIPS
Workaround: remove FIPS enable.

ASA 5580 traceback when CSM attempts deployment, Fixed CSCtu30581
Symptom: ASA 5580 crashes when CSM attempt deployment SSLVPN/CSD is not enabled on the ASA firewall, however, when CSM (Cisco Security Manager) attempts to make a cofiguration deployment for the ASA (which contains configuration for the Default Group-Policy), the ASA5580 crashes! CSM version is 4.1 and ASA is 5580 on 8.4.2(11). Attached is the traceback information I could collect from the console of the firewall during the crash.
Conditions: Seen only when there is a functional interaction between CSM and the ASA 5580 firewall.
Workaround: None.