Resolved Caveats—Cisco ASR 1000 Series Aggregation Services Routers Release 3.7.4S
This section documents resolved issues on Cisco ASR 1000 Series Aggregation Services Routers Release 3.7.4S.
Symptom: NVRAM configuration file gets corrupted when a chassis is power cycled without a graceful shutdown.
Conditions: Power cycle an ASR chassis without graceful shutdown.
Workaround: Shutdown chassis using the reload command and make sure RP gets to ROMMON before power cycling the chassis.
Symptom: NAT overload does not work for non-directly connected destinations in MPLS-VPN configurations.
Conditions: The symptom is observed with NAT overload configured to NAT traffic coming over an MPLS VPN to internet via a VRF-enabled interface.
Workaround: There is no workaround.
Symptom: T38 fax relay call speed is set to 14.4k regardless of the "fax rate" CLI configuration.
Conditions: There is no condition.
Workaround: There is no workaround.
Symptom: CUBE logs the following message: %SIP-3-INTERNAL: Cannot insert call history entry for callID.
Conditions: Calling party cancels call before connection. The following is an example:
INVITE --------------->---------------> 100 Trying <--------------<---------------- 180 Ringing <--------------<---------------- CANCEL ---------------->---------------> 200 OK <----------------<----------------- 487 Request Cancelled <------------------<--------------- ACK
Workaround: There is no workaround.
Symptom: If there is a problem with publishing the CRL, a major problem must exist so that further publishing is suspended until the issue is addressed, and the CA requires a shut or no shut. Currently there is no error message which suggests what happened with CRL. Need to add proper error messaging to make it more interactive.
Conditions: There is no condition.
Workaround: Enter "shut" or "no shut" on the CA server should result in the CRL being published again.
Symptom: When following an upgrade from Cisco IOS Release 12.4(24)T2 to Cisco IOS Release 15.1(4)M1, crashes were experienced in PKI functions.
Conditions: This symptom is observed on a Cisco 3845 running the c3845-advipservicesk9-mz.151-4.M1 image with a PKI certificate server configuration..
Workaround: Disable Auto-enroll on the CA/RA. Manually enroll when needed.
Symptom: We send 1M prefixes to an ASR with 8G of memory. The memory status is seen as critical. However this stays as critical even days after the BGP peering is removed. The fman_rp process seems to be holding the most memory even when the BGP routes are cleared. The impact of this is unknown at this time.
Conditions: This symptom is observed on a Cisco 3845 running the c3845-advipservicesk9-mz.151-4.M1 image with a PKI certificate server configuration..
Workaround: Disable Auto-enroll on the CA/RA. Manually enroll when needed.
Symptom: Failed to join the Meetme conference.
Conditions: If there's person already join the conference.
Workaround: There is no workaround.
Symptom: The following error message appears: %SYS-3-INVMEMINT: Invalid memory action (malloc) at interrupt level.
This error message can lead the device to crash.
Conditions: Seen on HA setup.
Workaround: Remove the route list from Multicast MOH CLI so that Cu can still have music on hold and can continue the feature. Alternatively, disable MOH (no Music comes on hold).
Symptom:
– There is a discrepancy in the inbound and the outbound SA lifetime in thestandby router.
– The KB lifetime in a standby router is greater than that of the active router, when a KB lifetime rekey occurs.
– The ping will not go through after applying a dynamic crypto map.
Conditions: The issues are seen after establishing the session between the HA routers and various test conditions.
Workaround: There is no workaround.
Symptom: netsync configuration for E1 (option 1) does not working.
Conditions: Configure R0 as netsync source, the netsync source doesnt lock (only option 1), option 2 works fine.
Workaround: There is no workaround.
Symptoms: Memory leak seen with following messages:
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "VOIP_RTCP", ipl= 0, pid= 299
-Traceback= 0x25B1F0Cz 0x25AB6CBz 0x25B1029z 0x46C02Ez 0x46C89Bz 0x46BCC2z 0x471D12z 0x43EF59Ez 0x43DD559z 0x43DCF90z
%SYS-2-MALLOCFAIL: Memory allocation of 780 bytes failed from 0x46C02E, alignment 32
Conditions: The conditions are unknown.
Workaround: There is no workaround.
Symptoms: Potential memory leak is seen when handling DNS lookup response.
Conditions: This symptom occurs when handling DNS lookup response.
Workaround: There is no workaround.
Symptoms: CEF switching is not working with GRE + protected tunnel configuration.
Conditions: Packets should go through tunnel interface.
Workaround: There is no workaround.
Symptoms: Expected SPI is not populated after Authentication is configured.
Conditions: Issue is seen for IPV6.
Workaround: The problem is seen only in case of VL configuration.
Symptoms: Calls placed on hold by 3rd Party SIP Server are disconnected if media inactivity is configured.
Conditions: PRI -- GW -- SIP -- 3rd Party SIP Server. Media inactivity is configured on the SIP GW. Phone behind the call server puts the call on hold. If the 3rd Party SIP server uses RFC 3261 hold (a=inactive) the call drops. If the 3rd Party SIP server uses RFC 2543 hold (c=0.0.0.0 and a=sendonly).
Workaround: Set media inactivity timer to a large value.
Symptoms: On DMVPN HUB, some crypto maps still exist after removing Tunnel protection from tunnel interface.
Conditions: It happens with scaling test.
Workaround: There is no workaround.
Symptom: Trace backs found.
Conditions: While copying the text file from the certificate server. Accessing https://msca-root/test.txt.
Workaround: There is no workaround.
Symptom: Transform comp-lzs is not supported with current hardware configuration.
Conditions: For ikev2_sanity script,while testing the miscellaneous testcase Transform comp-lzs is not supported with current hardware configuration.
Workaround: There is no workaround.
Symptom: IKE TUNNEL HISTORY TABLE/ipsecGlobalValues/cipSecSpiStatus failed.
Conditions: It should give correct data.
Workaround: There is no workaround.
Symptom: After a rollover, RA server does not retry to obtain its rollover CS cert from the CA server.
Conditions: The issue is seen after the RA has rolled over once and its first enrollment request (post-rollover) sent to the CA server has failed for some reason.
Workaround: There is no workaround.
Symptom: CME reloads for E911 call ELIN translation for incoming FXS/FXO trunk.
Conditions: The symptom is observed from Cisco IOS interim Release 15.3(0.2)T.
Workaround: There is no workaround.
Symptom: Not able to retrieve Via header for sending OPTIONS response back.
Conditions: This issue is seen in OPTION message case.
Workaround: Use the las_option_request from ccb while retreiving Via header.
Symptom: Error message display needs cosmetic changes to follow style guide.
Conditions: In rare situation, we hit error message regarding an error situation. The message format needs to be updated to follow style guidelines.
Workaround: There is no workaround.
Symptom: SPA handle invalid message is seen after running the hw-module subslot x/y shut command on ELC.
Conditions: When multiple ELC sources are configured, such as primary and secondary network clock sources from ELC, and execute ELC shut using hw-module subslot x/y shut command, the SPA invalid handle error message is displayed.
Workaround: There is no workaround.
Symptom: Ipsec sas not setup correctly on uut1 = secp53-6.
Conditions: Negative testcase failed because expect_ncomp is 77, ncomp is 78, compf is 0(this particluar number should be 7), expect_compf is 8.
Workaround: There is no workaround.
Symptom: ESP reload on ASR1002-X and ISR4451.
Conditions: Very High traffic rate of fragmented packets recieved with NAT configured(or traffic loop).
Workaround: Eliminate unnecessary fragments using either: MTU tunning, ACL filter, diverting the packet to new interface without NAT.
Symptom: Dynamic ACL does not get applied to the interface ACL, but the user shows up in the show ip auth-proxy cache command output.
Conditions: Very High traffic rate of fragmented packets recieved with NAT configured(or traffic loop).
Workaround: Move the auth-proxy rules onto a physical interface.
Symptom: A basic call between 2 SIP phones over SIP trunk (KPML-enabled) fails.
Conditions: This symptom is observed with Cisco ISR G2 platforms.
Workaround: There is no workaround.
Symptom: Connecting from Windows 7 L2TP/IPSec client to the VPN fails when using HSRP virtual IP as a gateway IP and Error 788 is displayed.
Conditions: This symptom is observed with Cisco IOS Release 15.2(3)T or later releases, and the Windows 7 L2TP/IPsec VPN client.
Workaround: Downgrade to Cisco IOS Release 15.1(3)T.
Symptom: DVTI Virtual Access interface may flap during rekey with a large number of IKEv2/IPSec tunnels.
Conditions: IKEv2 in large scale deployment is used.
Workaround: There is no workaround.
Symptom: Traffic flow is not fine with Fragementation.
Conditions: None.
Workaround: There is no workaround.
Symptom: CLI changes in the show spi details command
Conditions: This symptom is not caused by any specific conditions.
Workaround: There is no workaround.
Symptoms: CRYPTO MAP ACL FILTERING TEST FAILED due to indent counters.
Conditions: CRYPTO MAP ACL FILTERING TEST FAILED due to indent counters.
Workaround: There is no workaround.
Symptoms:The load balancing feature of the flex-vpn solution of Cisco IOS does not provide authentication facilities to avoid non authorized member to join the load balancing cluster. Thus, an attacker may impact the integrity of the flex-vpn system by inserting a rogue cluster member and having the load balance master to forward VPN session to it. A number of secondary effect, including black-holing of some of the VPN traffic may be triggered by this issue.
Conditions: Flex-VPN with Load Balancing feature active.
Workaround: Using CoPP and interface access-list may be used to allow only trusted router to join the load balancer cluster PSIRT Evaluation: The Cisco PSIRT has assigned this bug the following CVSS version 2 score. The Base and Temporal CVSS scores as of the time of evaluation are 4.3 or 3.9
https://intellishield.cisco.com/security/alertmanager/cvssCalculator.do?dispatch=1&version=2&vector=AV:N/AC:M/Au:N/C:N/I:P/A:N/E:F/RL:W/RC:CCVE ID CVE-2012-5032 has been assigned to document this issue. Additional information on Cisco's security vulnerability policy can be found at the following URL: http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html
Symptoms: Few Mem leak seen in HSRP-CLB notification in scaled IKEv2 load-balancing cluster scenario.
Conditions: Scaled IKEv2 load-balancing cluster scenario. Crypto Load Balancer HSRP state change [Master->Slave] or [Slave->Master]. IOS TCP process cleaning up internal message which has pointer to meory which CLB has allocated.
Workaround: There is no workaround.
Symptom: If there is a "peer.. fqdn..." statement in the startup-config
For example: crypto ikev2 client flexvpn flex peer 1 fqdn <FQDN>
Then after rebooting, the "peer..." statement may be missing from the running-config.
Conditions: This occurs because at boot time, when the startup-config is parsed, there is no DNS connectivity so the DNS resolution of the FQDN fails and hence the command is not accepted.
Workaround: Remove the peer and add it again with the "dynamic" keyword, i.e.: crypto ikev2 client flexvpn flex no peer 1 fqdn <FQDN> peer 1 fqdn <FQDN> dynamic
Note This process will delay the DNS resolution of the fqdn until the VPN tunnel is built.
Symptoms: FlexVPN site to site crypto session at UP-NO-IKE status.
Conditions: Clear cry session is given during rekey, create new sa with invalid spi, invalid SPI do not delete
Workaround: Shut or no shut the tunnl interface.
Symptom: The output of the Show controller pos pm command does not show the correct SFP line type for all the POS SPAs.
Conditions: The line type is shown as LONG MM for all the SFPs inthe output of the show controller pos pm frp command.
Workaround: Execute the show hw-module subslot x/y transceiver command.
Symptoms: Router experiences crashes due to SIP due to a freed pointer in memory.
Conditions: There is no conditions.
Workaround: There is no workaround.
Symptoms: The number if IPSec SAs on the box keeps increasing.
Conditions: IPSec eekeys are happening due to volume lifetime exhaustion.
Workaround: Turn off volume based rekey.
Symptom: IOS Router Identity Certificate missing upon reboot.
Condition: Identity certificate imported into a trustpoint that does not contain the direct issuer Certificate Authority certificate.
Workaround: Import the identity certificate into the trustpoint which contains the issuer's certificate.
Symptom: A router unexpectedly reboots and a crashinfo file is generated. The crashinfo file contains an error similar to the following:
%ALIGN-1-FATAL: Illegal access to a low address 04:52:23 UTC Wed Sep 19 2012
addr=0x4, pc=0x26309630z, ra=0x26309614z, sp=0x3121BC58
Condition: This occurs when IPsec is used. More precise conditions are not known at this time.
Workaround: There is no workaround.
Symptom: Tracebacks are seen.
Condition: When protocol mode dual-stack is enabled under telephony-service and create cnf-files is executed.
Workaround: There is no workaround.
Symptom: The Cisco ASR 1000 Series Aggregation Services Router filters out the ARP requests with its own source address. This leads to ping failure between two interfaces, which belong to different vrf and own same IP subnet; for example, vrf v1 1.0.0.1/24 and vrf v2 1.0.0.2/24.
Conditions: The gigabit ethernet interface (gig0/0/0) connected b2b to another interface on same router with VRF configured on atleast one of the interfaces.
Workaround: Configure some MAC address on the gigabit ethernet interface (gig0/0/0) and then unconfigure the MAC address.
Symptom: When IKE sends KEY_MGR_CLEAR_ENDPT_SAS
during initial contact, IPSec sends KEY_ENG_DELETE_SAS
.
Conditions: on performing SSO in spoke.
Workaround: There is no workaround.
Symptom: Audio distortion for MMOH stream produced by GW, when live-feed from FXO port is used.
Conditions: Live-feed is implemented to produce MMOH stream in CME environment, where Live-Feed source is connected to an FXO port. File based Moh also to be configured, and the file needs to be in Cached state.
Workaround: Remove the file based Moh. Or have a file based Moh which will NOT get cached.
Symptoms: Static routes are not getting removed.
Conditions: This symptom is observed with Smap - Smap. Removal of CLI does not remove the static route.
Workaround: Remove the ACL before removing the SA.
Symptoms: Hung calls on FXO ports where supervisory disconnect is used to disconnect calls.
Conditions: Analog phone / device initiates disconnect. Custom CPTone is used to detect the disconnect tone that is provided to the FXO port
Workaround: Configure the analog device to use one of the default CPTones that is bundled with IOS ( country based).
Symptoms: ESP crashes in response to a show command.
Conditions: When issuing the following show command on a ASR1K 1RU, ESP5, ESP10, ESP20 and ESP40 system.
show platform hardware qfp [active | standby] infrastructure bqs [schedule|queue] qid
This only causes an ESP crash when the 'qid' specified is an internal queue. It is safe for interface or QoS created queue.
Workaround: Avoid use of the show command to display internal queues.
PSIRT Evaluation:
The Cisco PSIRT has assigned this bug the following CVSS version 2 score. The Base and Temporal CVSS scores as of the time of evaluation are 3.8 or 3.3:
https://intellishield.cisco.com/security/alertmanager/cvssCalculator.do?dispatch=1&version=2&vector=AV:L/AC:H/Au:S/C:N/I:N/A:C/E:H/RL:OF/RC:C
No CVE ID has been assigned to this issue.
Additional information on Cisco's security vulnerability policy can be found at the following URL:
http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html
Symptom: IPSec SA reset when sequence number rolls over back to 0 with anti-reply disable.
Conditions: OUT_OCT_DETECT_SEQ_OVEFLOW counter increase.
Workaround: There is no workaround.
Symptom: Multiple failed IKE negotiations result in multiple MM_KEY_EXCH states from same spoke. The older failed SA's are never deleted from the IKE SA db. This ultimately would exhaust the call admission limit set on the router.
Conditions: 3945 router running 15.(1)4M5 code.
Workaround: Staying at 15.1(4)M4 at the moment.
Symptom: When ASR1K router is equipped with FP100 or FP160 models, the conditional police might fail to work if the physical interface of the tunnel QoS changes to a different one.
Conditions: If the child policy of tunnel QoS contains "priority <kbps>" and "fair-queue" features, the police of the "priority" feature will fail to function if the physical interface is changed to a different one. Then the priority traffic would behaves like strict priority feature which might starve other traffic class. This issue is specific to certain FP models, like FP100 and FP160.
Workaround: Detach and reattach the same policy-map to tunnel interface will restore the functionality for Tunnel QoS.
Symptom: When a dynamic cryptomap is used on the Virtual Template interface, SAs do not get created and thus the testscripts fail. This issue occurs because the crypto map configurations are not added to the NVGEN, and there is no security policy applied on the Virtual Template interface.
Conditions: This symptom is observed only when a dynamic map is used on the Virtual Template interface. However, this issue is not seen when tunnel protection is used on the Virtual Template interface or when a dynamic map is used on the typical physical interface.
Workaround: Use tunnel protection on the Virtual Template interface.
Symptom: After receiving the CRCX message, the Cisco AS5400 does not send 200 ok to SSW. SSW sends the CRCX message to the Cisco AS5400 again. Between these messages, debug outputs are displayed. It seems that the call is not disconnected completely for the end point by the previous disconnect request (the DLCX is received after the CRCX message from SSW). The end point may be stuck in call_disconnecting state. Conditions: This symptom is observed only when a dynamic map is used on the Virtual Template interface. However, this issue is not seen when tunnel protection is used on the Virtual Template interface or when a dynamic map is used on the typical physical interface.
Conditions: This symptom is observed with MGCP. This issue occurs when the Cisco AS5400 receives DLCX before sending 200 ok for the first CRCX message.
Workaround: There is no workaround.
Symptom: The EIGRP routes are not coming up after removing and reenabling the tunnel interface.
Conditions: This symptom is observed when EIGRP routes do not populate properly.
Workaround: There is no workaround.
Symptom: ASR router may crash running under heavy load
Conditions: This issue is considered an extreme corner case caused by the exhaustion of resources combined with the aggressive polling of information through CLI while the system is overloaded.
Workaround: There is no workaround.
Symptom: Router crash at speed dial.
Conditions: This symptom occurs during the speed dial.
Workaround: There is no workaround.
Symptom: Router crash at speed dial.
Conditions: This symptom occurs during the speed dial.
Workaround: There is no workaround.
Symptom: GW starts to drop calls randomly if you increase simultaneous calls beyond 350.
Conditions: This symptom occurs if 350 calls are connected on GW, some doing digit collection using Cisco ASR(MRCPv2) and some playing media. Increasing a few more calls triggers the issue of call drops and total calls stay at only 350.
Workaround: A patch was provided which fixed the issue.
Symptom: An IPsec VPN tunnel fails to be established. The debug crypto ipsec command shows no output when attempting to bring up the tunnel.
Conditions: This symptom occurs when all of the following conditions are met:
– 1) The crypto map is configured on a Virtual-Template interface.
– 2) This Virtual-Template interface is configured with "ip address negotiated".
– 3) The tunnel is initiated locally (in other words, if the tunnel is initiated by the peer, it comes up correctly).
Workaround: Downgrade to Cisco IOS Release 15.2(2)T3 or earlier releases or always initiate the VPN tunnel from the peer.
Symptom: IKE responder fails to accept phase 1 proposal with rsa-sig authentication with public RSA keys and no trustpoints configured.
Conditions: An authentication mechanism of rsa-sig is configured and rsa-encr cannot be used due to hardware/software limitations.
Workaround: Use rsa-encr if supported, otherwise switch to using actual certificates with trustpoint or pre-shared keys.
Symptom: EzVPN client cannot access the Internet over the VPN. Access to Hub internal resources works fine. The ZBF firewall on the Hub drops the encryptred ESP(udp) traffic from self to out containing reply from the host on the Internet. Log on the hub:
*Dec 28 15:34:51.189: %FW-6-DROP_PKT: Dropping udp session 8.8.8.2:0 8.8.8.1:53000 on zone-pair self-out class class-default due to DROP action found in policy-map with ip ident 0
source IP and port is incorrect.
Conditions: EzVPN client behind NAT and source port is PATed - is not udp 4500. EzVPN client reaching the Internet with u-turn on the Hub. Hub has ZBF policy from self to outside permitting VPN traffic. Hub has CEF enabled.
Workaround: Remove the ZBF policy from self to outside.
Symptom: Kingpin: plim tx drop if gi0/0/0 is used as tunnel source physical interface.
Conditions: The issue occurs when Gige interface as SVT tunnel source interface and 4K QoS policy is applied to 4K SVTI tunnel.
Workaround: There is no workaround.
Symptom: In GETVPN and IPSEC redundant configuration combination, if secondary group member is reloaded in the topology, it causes TEK registration of the group member is lost once the router comes back up and HSRP does state transition to standby.
Conditions: GETVPN with IPSec Redundancy configuration.
Workaround: Wait for next rekey or issue clear crypto gdoi.
Symptom: When traffic is sent with VLAN2 tag between two ixia ports through ASR1004 as below. After executing the command show controller, input vlan errors can be found and the counter increases without any packet drops. It is also found that when show interface command is executed, the value of input errors counter under related interface is 0.
Conditions: There is no known condition for this symptom.
Workaround: There is no workaround.
Symptom: Voice XML Gateway Crashes While Handling SIP Calls - caps nack'ed.
Conditions: A fax tone getting detected on the gateway is causing the gateway to send a T.38 Fax offer on the SIP leg. However customer does not support fax calls and the gateway receives 400 Bad Request Response for the T.38 Fax Offer. When responding with a ACK for 400 Bad Request Response we are seeing a crash as for some reason ccb->pld.destVdbPtr is getting set to NULL. Accessing the NULL pointer is causing a crash.
Workaround: Remove the fax configuration in "voice service voip->sip" will prevent the crash.
Symptom: "x Calls in queue" status is not displayed on all agents in the hunt group.
Conditions: This happens when a particular agent is logged out, then the subsequent agents (i.e in the order in which they are configured a list member) do not get the status update.
Workaround: Have all the agents logged in.
Symptom: Only single L2TP IPSEC vpn client can connect to vpn when they are behind PAT device even though NAT DEMUX is configured.
Conditions: VPN clients behind PAT device.
Workaround: There is no workaround.
Symptom: Packets drop occur when performing a ping from an ASR 1001 console with packets of large size (i.e. several kilobytes).
Conditions: This issue is specific to the ASR 1001 and requires a burst of data from the Control Plane to the Forwarding Plane such that internal hardware buffers are saturated. Normal processing continues, however, there are drops when the hardware buffer is full.
Workaround: There is no workaround.
Symptom: WCCP service cannot be enabled.
Conditions: Two services are configured in same interface, and then one service is deleted while the other is inactive. Then the inactive service cannot be enabled any more.
Workaround: Do not remove a service from the interface when another service is inactive.
Symptom: ip wccp check acl outbound doesn't work on Ultra/Overlord.
Conditions: Ultra/Overlord platform
Workaround: There is no workaround.
Symptom: crypto pki export <> causes crash.
Conditions: This symptom is observed in when a SUB CA trustpoint is configured and a trustpoint is configured and enrolled to that SUB CA.
Workaround: If possible, have the trustpoint on a separate box.
Symptom: A Cisco ASR repeatedly produces a "no-input" event despite inputs provided by caller.
Conditions: The symptom is observed with the following conditions:
– IOS VXML GW running Cisco IOS Release 15.x.
– Problem seems to be triggered by a "no-match' event prior to providing
expected responses.
Debugs show the following order of events:
– GW instructs TTS server to say "please say yes or no, or press digits 1 or two".
– GW instructs ASR to recognize.
Workaround: There is no workaround.
Symptom: GTP-U drops are noticed for communication that should not have been dropped. Swisscom agrees that this might be related to some timers and pending PDP sessions that need to be terminated. Since local tests with mobile devices are all successful, Swisscom wants and needs to go for 24 hour test to see if the GTP-U drops really lead to a service impact for mobile users.
Conditions: There is no conditions.
Workaround: There is no workaround.
Symptom: A Cisco 7200 with VSA fails to encrypt traffic under specific conditions.
Conditions: The symptom is observed under the following conditions:
– Cisco 7200 has IPsec SSO configured with HSRP. Dynamic crypto map is
– configured. Remote sides have static crypto map to this device.
– All the 15.x codes to the latest Cisco IOS 15.2(4)M2 are affected.
– Issue is not seen in the Cisco IOS 12.4 codes.
– Issue not seen when IPsec SSO and HSRP are removed.
Workaround: There is no workaround.
Symptom: sVTI tunnel interface does not come up after router reboot.
Conditions: This issue happens when you reboot the router.
Workaround: Reload ESP.
Symptom: ip mtu value 1390
configured in running-configuration and startup-configuration. But after a reboot, its value was changed to 1438
.
Conditions: After a reboot.
Workaround: There is no workaround.
Symptom: The Cisco AS5350 stops processing calls on PRI with a signaling backhaul from PGW. In the packet trace, there is no q931message from PGW. Further analysis shows that as5350 sends a q_hold (0x5)message in BSM, causing peer (PGW) to stop sending signaling traffic. However, there is no BSM_resume message or BSM_reset sent after it. Hence, PGW is stuck in this condition. There was earlier defect for CSCts75818 with similar symptoms in U-state.
Conditions: This symptom is observed due to some RUDP timing issues that cause BSM session switchover.
Workaround: Reload the Cisco AS5350 (but only when CU notices the outage). Also, shutting both Ethernet interfaces may help, but this workaround has not been tested.
Symptom: ucode crashes at REM_REM_MISC_ERR_LEAF_INT_INT_REM_POP_REQ_TO_EMPTY_SCHE
Conditions: on flapping multilink interfaces
Workaround: There is no workaround.
Symptom: Some of the SPA goes to inserted (physical) state after an ISSU upgrade. This issue is not specific to any particular SPA or SIP.
Conditions: This issue is seen while doing an ISSU upgrade on a setup that has a high scale configuration. Altleast 2000 subinterfaces are configured in the router.
Workaround: This issue is not seen in the following scenarios:
Symptom: QFP load spike occurs when dropping traffic via IPv6 ACL.
Conditions: IPv6 traffic is dropped with ACL.
Workaround: Configure the no ipv6 icmp unreachable command under the receiving interface.
Symptom: The following error message may appear:
%STILE_CLIENT-4-MAX_LINK_TOUCH_WARN: F0: cpp_cp: NBAR number of flow-slinks threshold is reached, cannot allocate more memory for flow-slinks
.
This may cause some degradation in SSL based traffic.
Conditions: This message may appear under heavy SSL traffic.
Workaround: Currently there is no workaround. The classification of the SSL-based traffic should be based on the other classification mechanisms.
Symptom: VPN led does not come up when an IKEv2 tunnel is active.
Conditions: IKEv1 is not affected only IKEv2.
Workaround: There is no workaround.
Symptom: SIP call during "Call Forward No Answer" option leaks the Transcoder resource used on CUBE Example call flow: Telco -> SIP Trunk (G711alaw/G729) -> CME -> SIP phone (G711ulaw) ->NOAN -> CUE (G711ulaw)
Conditions:
– SIP Call
– Codec mis-match between two legs of the call and invokes the local transcoder resource.
– Call forward No Answer (noan) feature
Workaround: Reset the sccp session.
Symptom: SAs do not get installed in GETVPN GM.
Conditions: The symptom is observed when the key server is configured with "receive-only" SAs.
Workaround: Remove receive-only configuration at the key server.
Symptom: Traceback at DMVPN Spoke registration, DMVPN QoS policy not deployed to QFP datapath component.
Conditions: DMVPN, NHRP, QOS.
Workaround: There is no workaround.
Symptom: DSP error message printed on console, and crash takes place.
Conditions: DSP firmware (version:33.1.00) sends corrupted DSP error message to RP IOS, which leads to crash:
%SPA_DSPRM-3-DSPALARM: Received alarm indication from dsp (1/0/9).
%SPA_DSPRM-3-DSPALARMINFO: 0008 0000 0080 0000 0000 0001 7F3B FEDF
%SPA_DSPRM-3-DSPALARMINFO: ;????
%DSP-3-DSP_ALARM: SIP1/0: DSP device 2 is not responding. Trying to recover DSP device by reloading
Workaround: Downgrade to XE36, which runs firmware v. 31.1.0
Symptom: A kernel core file is generated. Process core files that were being generated are incomplete.
Conditions: The kernel core is generated when HMAN stops strobing the HW Watchdog timer. This occurs concurrently when a process with a large resident set size (IOSd) is dumping core.
Workaround: There is no workaround.
Symptom: A FlexVPN spoke configured with an inside VRF and front-door VRF may have problems with spoke-to-spoke tunnels if they are not the same. During tunnel negotiation, two virtual-access interfaces are created (while only one is needed), the one in excess may fail to cleanup correctly. As a result, the routes created by NHRP process may lead to loss of traffic, or traffic may continue to flow through the Hub.
Conditions: This symptom occurs when the VRF used on the overlay (IVRF) and the VRF used on the transport (FVRF) are not the same.
Workaround: There is no workaround.
Symptoms: DSP crash with the following console error:
%SPA_DSPRM-3-DSPALARMINFO: Checksum Failure:80000000,0000000e,d0156a80,d0156000 *Mar 14 17:56:05.851:
%SPA_DSPRM-3-DSPALARM: Received alarm indication from dsp (1/3/6).
%SPA_DSPRM-3-DSPALARMINFO: 0042 0000 0080 0000 0000 0000 4368 6563 6B73 756D 2046 6169 6C75 7265 3A38 3030 3030 3030 302C 3030 3030 3030 3065 2C64 3031 3536 6138 302C 6430 3135 3630 3030 0000 0000 0000 0000 0000
Conditions: Error occurs during an RP switchover process. The standby RP presents DSPs failing to come up.
Workaround: This command may clear up the DSPs:
Router# hw-module subslot x/y reload
Symptoms: A Cisco 3845 that is running Cisco IOS Release 15.1(4)M2 may have a processor pool memory leak in CCSIP_SPI_CONTROL.
Conditions: The conditions are not known at this time.
Workaround: There is no workaround.
Symptoms: If the call to transfer-target fails, this problem would occur.
Conditions: When an external Application is registered to UC gateway via a web-services interface,
Workaround: External application is not registered or the registered application do not subscribe for AUTHORIZE_CALL event.
Symptom: NAT64 does not work in simulator.
Conditions: This issue is not seen on hardware.
Workaround: A reboot is likely to clear the issue.
Symptom: The traffic may not be shaped correctly resulting in more traffic to leak through or the router crashes when model 3/4 subscriber policy is applied.
Conditions: The model 3 and 4 hierarchy is built incorrectly on ESP-100/200 and ASR1002X when the subscriber policy is added after the main interface is already active.
Workaround: There is no workaround.
Symptom: NA
Conditions: NA
Workaround: There is no workaround.
Symptom: Using MRCPv2 on VXML GW for CVP calls to 3rd party ASR, we have found the MRCP Client process is disappearing after a few hundred calls. This causes all future calls to fail until the VXML GW is rebooted.
A traceback is thrown in the logs at this time, indicating a memory problem.
Feb 28 00:23:23.949 JST: %SYS-2-FREEBAD: Attempted to free memory at B0D0B0D, not part of buffer pool
Traceback= 18B57F4z 2C60B0Cz 5B120B3z 4BCA9F6z 2BCCA09z 4C7692Ez 4BCAA8Bz 4C8D03Fz 4C8EE4Bz 4C85EF2z 4C85D2Fz 4C75A21z
Running 'show process' after this traceback shows the MRCP Client process is no longer running.
Conditions: The issue occurs when a Nuance server abnormally tears down MRCPv2 session in the middle of the call. MRCPv2 is needed to trigger the crash. MRCPv1 does not cause a crash.
Workaround: Set all sessionTimeout configurations to -1 on the Nuance server (In the NSSserver.cfg file). Use MRCPv1 instead of MRCPv2
Symptom: Netsync customer seeing clock in ql-failed state on one ASR-2ru.
Conditions: The issue occurred when distributing stratum 1 clock source through its network.
Workaround: If both SPAs are in the same slot, do not send the secondary config.
Symptom: Operation relying on PKI may start failing when enrolling a new trustpoint to same CA as already existing trustpoint.
Conditions: First seen with Cisco IOS 15.2(4)M1.
Workaround: Use crypto key zeroize pubkey-chain command.
Symptom: Running sh sbc FOO sbe mib mgmmediaaddresstable on standby causes CLI to hang.
Conditions: When enabled SBC-B2B redundancy.
Workaround: Do not run this command on standby.
Symptom: In SBC-B2B, after no attach/attach an adjacency, calls are rejected with 503 Service Unavailable.
Conditions: This condition occurs under the following:
– Config vrf001 on BOX1(ACTIVE) then on BOX2(STANDBY).
– Config adjacency's vrf and signaling-address, and media-address and vrf, both refer to vrf001.
– Switch-over.
– no attach/attach adjacency on BOX2(ACTIVE).
– Later calls are rejected with 503 Service Unavailable.
Workaround: Always add or change vrf related SBC config on the same box. More Info:
Symptom: In some rare situations, EzVPN client routers are seen to have an IKEv1 SA lifetime beyond 24 hours - up to "3 weeks, 3 days". This can lead to unpredictable behavior during IKEv1 phase1 renegotiation, notably this can cause the server to initiate a negotiation which would result in errors and interruptions of service over the VPN.
Conditions: There is no condition.
Workaround: There is no workaround.
Symptom: Tracebacks as follows seen during router bootup:
%SYS-2-INTSCHED: 'suspend' at level 2 -Process= "Init",
-Traceback= 4F6966C 6A708EC 890127C 6B4F924 6B4F7F8 6B4EAAC 6B4F43C 6B4F514 6DD6D4C 6DDB3A8 6A23E50 6A23F18 6A24100 57D3F94 57D42D8 4F701E4
0x4F6966C ---> process_ok_to_reschedule+288
0x6A708EC ---> process_suspend+4C
0x890127C ---> random_fill+248
0x6B4F924 ---> default_entropy_routine+9C
0x6B4F7F8 ---> hardware_entropy_source+CC
0x6B4EAAC ---> nist_instantiate+78
0x6B4F43C ---> try_create_rng+1B4
0x6B4F514 ---> nist_rng+34
0x6DD6D4C ---> cts_sap_get_key_counter+54
0x6DDB3A8 ---> cts_sap_init+C4
0x6A23E50 ---> subsys_init_routine+60
0x6A23F18 ---> subsys_init_class_internal+A8
0x6A24100 ---> subsys_init_class+8C
0x57D3F94 ---> system_init+250
0x57D42D8 ---> init_process+94
0x4F701E4 ---> ppc_process_dispatch+
Conditions: The symptom is observed during router bootup.
Workaround: There is no workaround.
Symptom: ccpp_cp_svr and fman_fp cores during mdr.
Conditions: While doing spa/SIP OIR during mdr.
Workaround: There is no workaround.
Symptom: An ESP crash occurs.
Conditions: In the rare case, where the software managed memory pools have been increased and a coalescing of buffer pools is required to create large buffers out of smaller buffers. Only a few features (MLPPP, FRF12, ESS, SSL, and IP reassem) make use of this memory.
Workaround: There is no workaround.
Symptom: CUBE reloads while testing SDP pass-through with v6.
Conditions: CUBE reloads while testing SDP pass-through with v6.
Workaround: Do not use SDP pass-through and use normal SIP processing call flows.
Symptom: A Cisco 3945E router crashes.
Conditions: The symptom is observed with the following conditions:
– Extension mobility is configured for the phone. The logout profile should not
– be configured with any number.
– In the logged out state, user has to press the "NewCall" softkey followed by
– dialing any digit between 1-9 (excluding 0).
– Instead of pressing "dial" softkey, press "AbbrDial" softkey.
Workaround: Have a proper number configured under the logout profile.
Symptom: Spurious CPLD-EHSA interrupts are seen. These interrupts are seen in cmand_R* tracelog file. Sometimes, these can also cause high CPU depending on the activity on the USB device.
Conditions: When an external USB device is attached to an Intel-x86 based RP. This includes RP2, 1RU, 2KP platforms. RP1, 2RU, 2RU-F are PPC based platforms, so these do not have this issue. On Intel x86 platforms, CPLD interrupt lines are shared with external USB devices. Spurious CPLD-EHSA interrupts are in fact USB interrupts.
Workaround: Remove external USB device from the router when not in use.
Symptom: A 3945e will crash due to a bus error with a null instance variable.
Conditions: This has been observd on a 3945e but the conditions are still unknown.
Workaround: There is no workaround.
Symptom: Handshake fails when we select Diffie Hellman cipher suite from sslvpn configuration.
Conditions: There is no condition.
Workaround: Select other than Diffie Hellman cipher suite at sslvpn.
Symptom: The ASR 1004 router crashes with:
CPPHA-3-FAULT: F0: cpp_ha: CPP:0.0
desc:ETC_ETC_LOGIC1_LEAF_INT_INT_LP_LONG_PKT_ERR det:DRVR(interrupt) class:OTHER sev:FATAL id:2694 cppstate:STOPPED res:UNKNOWN flags:0x7 cdmflags:0x0
Conditions: VASI, cryto, mpls, during normal operation (as per what is known).
Workaround: There is no workaround.
Symptom: Using local ikev2 authorisation policy, it is not possible to push prefix along with the ip address to the client. The prefix always gets pushed as 128.
Conditions: ikev2 local authorisation.
Workaround: Use radius server to push the prefix to the client.
Symptom: An NHRP resolution request is forwarded to the first NHS on the tunnel interface instead of being forwarded along the routed path.
Conditions: DMVPN phase 3 implementation.
Workaround: Use radius server to push the prefix to the client.
Symptom: FXS ports on a Cisco VG224 running Cisco IOS versions 124-24T7 or 151-4M5 will stop working randomly, user will hear a busy tone when going offhook on the analog device connected to the FXS port on the VG224. The call status will show as "ERR_WAIT4_DISC" or "ERR_WAIT4_ONHO" in the output of the command "show stcapp device summ" for that problematic FXS port.
Conditions: The Cisco VG224's FXS ports are set up as STCAPP with Cisco Unified Callmanager (CUCM) server and have the shared line feature enabled with a Cisco IP phone on the same CUCM cluster.
Workaround: Remove the "shared line" feature if feasible or issue a "shut" followed by "no shut" under the problematic FXS voice-port via the VG224's IOS command line interface (CLI) or issue a manual "reload" on the VG224 during a maintenance window.
Symptom: "playout-delay fax" CLI is not changing T.38 and modem Passthrough playout buffer to accommodate packet jitter.
Conditions: Ability to reduce the default Fax playout delay.
Workaround: There is no workaround.
Symptom: ASR1000 router crashes due to PPTP related traffic.
Conditions: Router is running on 3.9.0S. NAT PAT is configured in CGN mode on the router.
Workaround: Disable PPTP ALG in CGN mode. No ip nat service pptp.
Symptom: Call failure.
Conditions: Media antitrombone + Call farward cases + SDP passthrough.
Workaround: There is no workaround.
Symptom: In a NAT64 configuration, "show policy-map type inspect zone-pair sessions" shows NATed ipv4 address for the ipv6 host. It should show the hosts' real IP addresses, i.e. v6->v4 or v4->v6, not v4->v4.
The PD command sh plat ha qf ac fe fir da scb actually shows the scb's addresses as the real hosts' addresses, i.e. v6->v4 or v4->v6. However, the v6 host's port number is still shown as the translated v4 port number.
In the ZBFW datapath log at cpp_cp*.log, the session key printed in the debug messages is showing wrong port number. The session key is supposed to be all v4, but the port number is actually printed as v6 port number.
For the PD show scb command filter such as sh plat ha qf ac fe firewall datapath scb ipv6 3000::2 44 ::1d00:2 444, we can't use the v6 port to match the session and have to use v4 port of the v6 host to match.
Conditions: NAT64 configuration. For the issues involving v6/v4 port numbers, they are only visible if there is PAT configuration, i.e. if the v6 host's port number can be changed after NAT64 translation.
Workaround: There is no workaround.
Symptom: Over-sampling entropy source on Cavium and Quack/ACT based platforms.
Conditions: There is no condition.
Workaround: There is no workaround.
Symptom: Silent suppression of the line that is causing the difference in behavior.
Conditions: Silent suppression of the line that is causing the difference in behavior.
Workaround: Remove the silent suppression line using the lua script LVASR01#more bootflash:edit_silence_supp.lua function delete_lines(msg) for line in msg.sdp:select_by_prefix("a=silenceSupp:off"):iter() do line:delete() end end MeEditor.register(MeEditor.BEFORE_RECEIVE,"SilenceSupp",delete_lines)
.
Symptom: Router deops ESP packets with CRYPTO-4-RECVD_PKT_MAC_ERR.
Conditions: Peer router sends nonce with length 256Bytes
Workaround: There is no workaround.
Symptom: No media forwarded or media dropped for "Reprocess limit exceeded".
Conditions: This issue occurs when all the following conditions are met:
– the call is setup as nat call
– media is received before off/answer completed
– the call is modified to hairpin with other calls both on two sides
Workaround: There is no workaround.
Symptom: ASR1000 ESP may get reloaded unexpected when PfR NAT OER integration feature is enabled.
Conditions: When one of the NAT outside interface shuts down administratively with active NAT translations.
Workaround: Disable PfR NAT OER integration feature.
Symptom: When turning off a wccp service or detachin a service from an interface, the memory allocated for wccp is not freed. This can be seen using: show platform software memory qfp-control-process qfp active | section WCCP.
Conditions: None.
Workaround: There is no workaround.
Symptom: RP_Crash seen at be_interface_action_remove_old_sadb
Conditions: While unconfiguring the 4K svti sessions after the HA test.
Workaround: There is no workaround.
Symptom: Traffic decrypted on a Cisco ISR G2 series is process switched instead of staying in the CEF path.
Conditions: The symptom is observed when the hub and/or the spoke are located behind NAT or PAT.
Workaround: Disable NAT/PAT.
Symptom: Topology: S---asr1k---D1--\ | x.x.x.x/32 ------D2--/ * ISIS, fast-reroute per-prefix configured * LDP on all interfaces * x.x.x.x/32 is reachable via D1 (primary) and D2 (backup) * Sending traffic from S to x.x.x.x * S, D1, and D2 are simulated (Agilent) * Version 15.3(1)S Problem: Upon failing link asr1k-D1 (laser shut on Agilent, equivalent to pulling fiber), FRR is not triggered and traffic flow is restored when ISIS reconverges.
Conditions: The symptom is observed in IP network and when FRR is enabled and when ethernet interface is one of the primary path and protected path and when plugging out ethernet wire or remote shutdown.
Workaround: There is no workaround except changing interface type to POS/ATM.
Symptom: Topology: S---asr1k---D1--\ | x.x.x.x/32 ------D2--/ * ISIS, fast-reroute per-prefix configured * LDP on all interfaces * x.x.x.x/32 is reachable via D1 (primary) and D2 (backup) * Sending traffic from S to x.x.x.x * S, D1, and D2 are simulated (Agilent) * Version 15.3(1)S
Conditions: Upon failing link asr1k-D1 (laser shut on Agilent, equivalent to pulling fiber), asr1k quickly (<50msec) starts forwarding packets (dest x.x.x.x) to D2 (backup), but with D1's advertised label! Only after ISIS converges the packets are forwarded with the correct label (from D2).
Workaround: There is no workaround.
Symptom: ESP might crash.
Conditions: While running clear ip nat translations * after the forced removal of a NAT mapping.
Workaround: Before removing any NAT mappings, run clear ip nat trans *. And do not use the forced option when removing a NAT mapping. The following is an OK example:
ip nat inside source list 1 pool pool1 overload
Symptom: Router crash due to memory leak.
Conditions: The symptom is observed with a CME shared line feature configuration.
Workaround: Disable the shared line feature will avoid memory leak.
Symptom: I/O Leak in the middle/DSPRM buffer pools are observed
Conditions: Flex DSPs are present.
Workaround: There is no workaround.
Symptom: QFP reloads and gets stuck in reset loop until pap or cgn configuration.
Conditions: This occurs when the router is reloading when the following configurations exist: ip nat setting mode cgn and ip nat setting pap.
Workaround: Either remove PAP or CGN configuration. A fix is expected in release 3.9.1 and later.
Symptom: VTCP needs to adjust in case 10k h323 resemble packets size are received. Clear DF bit to decrease the impact on MPLS Path Selection and Limit Packet length for assembled h.323 packet to 8K.
Conditions: The following apply:
– Send 10K tcp segments from server
– pmod manipulate the 1st tcp segment into h323 realization format (03 00 length after tcp header)
– the response src port 80 and dst 1720
Workaround: Disable h323 alg.
Symptom: After applying the QoS configuration with policy-maps, the configuration is seen in show running config properly. However, on checking the QFP, the following is displayed:
sh platform hardware qfp active feature qos all output all" no interfaces are configured as QoS target(s)
When checking the matching of the packets on the interface, it is displayed as "0".
Conditions: IOS XE Version: 03.07.01.S.
Workaround: There is no workaround.
Symptom: CM tone detector being turned ON irrespective of the fax and modem features being disabled.
Conditions: CM tone detector being turned ON and being reported to the host by the DSP.
Workaround: There is no workaround.
Symptom: Topology: ========= < -----(SIP Trunk A)-----CUBE-----(SIP Trunk B)-----> CUBE is not forwarding the REINVITE message received from Trunk A to the SIP Trunk B when 491 Request Pending is received from SIP Trunk B for the previous SIP transaction.
Conditions: When 491 Request Pending is received.
Workaround: There is no workaround.
Symptom: REq/RES timeout not work as expected.
Conditions: FW session under heavy traffic 2K create/delete.
Workaround: Stop the traffic and the timer works
Symptom: ICMP v6 traffic is observed to drop.
Conditions: ICMP v6 traffic is observed to drop with cxsc configured under the zbfw policy-map. Drops are observed the zone is applied on a DMVPN tunnel.
Workaround: There is no workaround.
Symptom: %SMC-2-BAD_ID_HW: is output, and SPA is not disabled. SPA should be disabled if authentication fail.
Conditions: ASR1001 Built-in SPA.
Workaround: There is no workaround.
Symptom: ASR 1002-X acting as LNS, RP crashes after shutting down the interface that is connecting LAC.
Conditions: 5000 sessions with per-session QoS. All these sessions are setup on single L2TP tunnel.
Workaround: There is no workaround.
Symptom: GTPv1 traffic CPP crashed caused by writing protected memory
Conditions: Landslide LinuxTC ASR5K GGSN LinuxTC introduced packet delay, drop, reproduced, corrupt, reorder between GTP AIC and GGSN. During the GTPv1 traffic, CPP crash is expected, which is caused by protect memory writing.
Workaround: There is no workaround.
Symptom: CUOM could not process MOSCQEReachedMajorThreshold clear trap from CUBE SP. For MOSCqe alert clear trap, CUBE should not send CurrentLevel Varbind but should send csbQOSAlertCurrentValue Varbind.
Conditions: This condition occurs when CUBE SP generates clear trap for voice quality alerts.
Workaround: The code fix is included in CUBE 15.2(4)S4. If earlier CUBE version is used, manually clean the alarm at CUOM after root cause is rectified.
Symptom: Usernames do not show up in CCP Express. Username shows up on a router with default configuration.
Conditions: The symptom is observed on routers with configurations that break show run format.
Workaround: Use default configuration.
Symptom: Running show crypto map.
Conditions: During high CPU.
Workaround: There is no workaround.
Symptom: A DMVPN spoke router running 15.2(4)M3 and configured for Dual Hub - Dual DMVPN failover will fail to forward multicast traffic for EIGRP neighbor forming after failing from primary to backup and back to the primary. EIGRP neighbrship will fail to complete and flap on the spoke. The hub will never show any EIGRP neighborship.
Conditions: DMVPN spoke router running 15.2(4)M3 in Dual Hub - Dual DMVPN scenario and running dynamic routing protocol must failover and failback to the primary tunnel for this to occur.
Workaround: Removing "ip nhrp map multicast x.x.x.x y.y.y.y" and readding it resolves the problem.
The issue doesn't exist in 15.2(4)M1.
Symptom: 7301 router running c7301-advipservicesk9-mz.152-4.M3 is experiencing memory leak in Crypto IKMP process particularly on crypto_ikmp_config_send_ack_addr function.
Conditions: When running 7301 router and connecting EasyVPN through it, causes leak in Crypto IKMP process over time.
Workaround: Reload the router over a period of time.
Symptom: 7301 router running c7301-advipservicesk9-mz.152-4.M3 is experiencing memory leak in Crypto IKMP process particularly on crypto_ikmp_config_send_ack_addr function.
Conditions: When running 7301 router and connecting EasyVPN through it, causes leak in Crypto IKMP process over time.
Workaround: Reload the router over a period of time.
Symptom: ASR1000 RP crash after software upgrade.
Apr 20 09:53:01.396: %SYS-3-BADBLOCK: Bad block pointer 3AFDF4B0 -Traceback= 1#b3d7956825375323829953c9aa18e3e0 :10000000 6FCCF4 :10000000 6FD0A0 :10000000 1F2279C :10000000 1F1C1B0 :10000000 1F3F750 Apr 20 09:53:01.399: %SYS-6-MTRACE: mallocfree: addr, pc 33A1E15C,1011798C 33A1E15C,101178CC 33A1E15C,30000060 4C3A105C,600003E4 4C3A0834,1049C71C 4C3A0834,1049C5FC 4C3A0834,400003FC 412703FC,125DFF80 Apr 20 09:53:01.399: %SYS-6-MTRACE: mallocfree: addr, pc 412703FC,300000F6 4C29B4E0,125DFF80 4C29B47C,20005F00 33A1E15C,1011798C 33A1E15C,101178CC 33A1E15C,30000060 3AAFFF14,154DA6C4 4C1403F4,60000012 Apr 20 09:53:01.399: %SYS-6-BLKINFO: Corrupted magic value in in-use block blk 3AFDF4B0, words 60, alloc 8, InUse, dealloc 0, rfcnt 1 -Traceback= 1#b3d7956825375323829953c9aa18e3e0 :10000000 6FCCF4 :10000000 6FD0A0 :10000000 1F1D9C4 :10000000 1F227B4 :10000000 1F1C1B0 :10000000 1F3F750 Apr 20 09:53:01.402: %SYS-6-MEMDUMP: 0x3AFDF4B0: 0xF8 0x24 0x3C 0x1653EC7C Apr 20 09:53:01.402: %SYS-6-MEMDUMP: 0x3AFDF4C0: 0x8 0x8 0x3AFDF38C 0x8000003C Apr 20 09:53:01.402: %SYS-6-MEMDUMP: 0x3AFDF4D0: 0x1 0x0 0x1000001 0x3058827C %Software-forced reload Exception to IOS Thread: Frame pointer 0x30742CC8, PC = 0x87308B4 UNIX-EXT-SIGNAL: Aborted(6), Process = Check heaps -Traceback= 1#b3d7956825375323829953c9aa18e3e0 c:86FA000 368B4 c:86FA000 368B4 c:86FA000 384C8 :10000000 32FD91C :10000000 1F227BC :10000000 1F1C1B0 :10000000 1F3F750 Fastpath Thread backtrace: -Traceback= 1#b3d7956825375323829953c9aa18e3e0 c:86FA000 D9F08 c:86FA000 D9EE8 iosd_unix:887E000 1580C pthread:7DB2000 5A4C Auxiliary Thread backtrace: -Traceback= 1#b3d7956825375323829953c9aa18e3e0 pthread:7DB2000 B598 pthread:7DB2000 B578 c:86FA000 EF9C4 iosd_unix:887E000 212F4 pthread:7DB2000 5A4C PC = 0x087308B4 LR = 0x08732384 MSR = 0x0002D000 CTR = 0x07DC0D60 XER = 0x20000000 R0 = 0x000000FA R1 = 0x30742CC8 R2 = 0x30085C70 R3 = 0x00000000 R4 = 0x00006908 R5 = 0x00000006 R6 = 0x00000000 R7 = 0x08730B5C R8 = 0x0002D000 R9 = 0x3007E7F0 R10 = 0x3007E7F0 R11 = 0x30742CA0 R12 = 0x08732384 R13 = 0x18456078 R14 = 0x11F3F604 R15 = 0x00000000 R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000 R20 = 0x00000000 R21 = 0x1630C7D8 R22 = 0x18BDAA28 R23 = 0x18BDAC70 R24 = 0x18BDB3B8 R25 = 0xAB1234AB R26 = 0xAB1234CD R27 = 0x30742E58 R28 = 0x3AFDF4E0 R29 = 0x30742CE0 R30 = 0x0886A7AC R31 = 0x00000006 ========= Start of Crashinfo Collection (09:53:01 UTC Sat Apr 20 2013) ========= For image: Cisco IOS Software, IOS-XE Software (PPC_LINUX_IOSD-ADVIPSERVICESK9-M), Version 15.2(4)S1, RELEASE SOFTWARE (fc3) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2012 by Cisco Systems, Inc. Compiled Sat 06-Oct-12 11:55 by mcpre Uptime = 00:02:51
Conditions: Device configured with SBC with interchassis redundancy.
redundancy mode none application redundancy group 1 name ECS preempt priority 150 failover threshold 100 timers delay 100 control Port-channel30.8 protocol 1 data Port-channel30.9 track 1 decrement 200 track 2 decrement 200 protocol 1 name BFD timers hellotime msec 250 holdtime msec 1000.
Workaround: Do not setup B2B redundancy between XE36(or older) and XE37(or later).
Symptom: ESP fails to initialize and reboots. A message like the following will be seen on the IOS console:
*Jan 01 16:22:35.562: %CPPHA-3-INITFAIL: F0: cpp_ha: CPP 0 initialization failed - startup init (0x1)
*Jan 01 16:22:35.562: %CPPHA-3-INITFAIL: F0: cpp_ha: CPP 0 initialization failed - start CPP (0x1)
The cpp_driver tracelog contains an entry indicating the Hoover PLL failed to lock. This could be on CIF,FIF, or ICM. Here is an example from CIF:
01/01 16:22:35.120 [cpp-drv]: (ERR): COMP0053/CIF/1028: QFP0.0 - timeout waiting for Hoover TX PLL to lock.
Conditions: Router configuration or traffic pattern does not affect this problem. This software error is fixed in to XE3.7.4, XE3.9.2, XE3.10.0 and later releases.
Workaround: There is no workaround.
Symptom: ESP fails to initialize and reboots. Cman-fp indicates error due to Hoover PLL lock failure.
Conditions: Router configuration or traffic pattern does not affect this problem. This software error is fixed in to XE3.7.4, XE3.9.2, XE3.10.0 and later releases.
Workaround: There is no workaround.
Symptom:Group Member is regitering the third Key Server in its list in a redundant KS scenario, when certificate of first KS has been revoked.
Conditions: This has been observed under the following conditions:
– GM has a list of 3 or more Key server
– Certificate based authentication with OCSP validation
– First KS certificate has been revoked.
Workaround: There is no workaround.
Symptom: ASR 1001 prints following error messages and crashes: % Internal error: Connection to peer process lost %MCP_SYS-0-ASSERTION_FAILED: SIP0: cmcc: Assertion failed: Assertion failed: cman/cc/./src/cmcc_util.c:322: "bay < cmcc_max_spas_per_cc()".
Conditions: Issue show platform hardware subslot 0/3 plim statistics command in CLI.
Workaround: Not issuing show platform hardware subslot 0/3 plim command will avoid this problem.
Symptom: ZBFW syslog for passing and dropping ICMPv6 packets shows wrong value in the port number fields. The src/dst port numbers should be the ICMP type and code. In addition, the passing syslog is showing "Passing Unknown L4 protocol".
Conditions: The router is configured in 66, 64 or 46 case. syslog for pass or drop logging is enabled. Sending ICMPv6(or ICMP from v4 side) packets.
Workaround: Not issuing show platform hardware subslot 0/3 plim command will avoid this problem.
Symptom: Bursty shape rate on high bandwidth queue.
Conditions: When there are 2 vlans configured each with a single simple shape queue, one with a very high rate (ex. 400,000,000bps) and another with a very low rate (ex 128,000bps), the high rate queue's rate may be bursty and fluctuate +- 10% of the configured rate.
Workaround: Configure a hierarchical policymap on the vlans where the shape is on the parent class, not on the queue.
Symptom: NAT translations could be stranded on the standby with NAT B2B and AR configuration.
Conditions: NAT translations could be stranded on the standby with timeout of zero.
Workaround: During a MW or downtime, execute the clear ip nat trans command on the active box.
Symptom: Static routes injected through RRI (reverse-route static) are not getting removed.
Conditions: This symptom is observed when a static crypto map that has "reverse-route static" enabled is applied on two different interfaces with a local-address.
Workaround: Reload the Router.
Symptom: May 3 12:46:21.835: %SYS-2-FREEFREE: Attempted to free unassigned memory at 3EC4FF9C, alloc 350B5A70, dealloc 350B5608
-Traceback= 35D9BEC4z 350C158Cz 350AEED8z 350B081Cz 32C23084z 32C23068z
May 3 12:46:21.839: %SYS-6-MEMDUMP: 0x3EC4FF7C: 0x350B5A70 0x3EC50C58 0x3EC4FDF0 0x65E
May 3 12:46:21.839: %SYS-6-MEMDUMP: 0x3EC4FF8C: 0x0 0x350B5608 0x1000133 0x3CDD2E48%Software-forced reload
-Traceback= 0x30DF22BCz 0x30DF05F0z 0x32C3278Cz 0x35D9BEC4z 0x350C158Cz 0x350AEED8z 0x350B081Cz 0x32C23084z 0x32C23068z
Conditions: May be with Presence or Shared line feature.
Workaround: There is no workaround.
Symptom: SBC SRTP ucode crash when doing srtp-rtp interworking.
Conditions: It seems this can happen in hairpined SRTP calls, though not able to reproduce in the lab. The test scenario is: rtp----SBC-----SRTP--------SBC-------rtp
Workaround: There is no workaround.
Symptom: Incorrect statistic from SNMP OID "1.3.6.1.4.1.9.9.171.1.3.1.1", related to a number of IPSec tunnels after running "clear crypto sa / session" command.
Conditions: Configured DMVPN, running "clear crypto sa / session" command.
Workaround: Reloadof router helps to solve the issue
Symptom: Memory leak in [pfr_config].
Conditions: Performance Routing (PFR) is configured on the router.
Workaround: There is no workaround.
Symptom: ESP may reload when switching classic to CGN mode.
Conditions: ESP may reload when switching classic to CGN mode with traffic.
Workaround: There is no workaround.
Symptom: NAT timeout when used with port command does not work as expected.
Conditions: IP NAT translation port-timeout tcp <port #> <timeout value> Above CLI with ip nat translation tcp-timeout timeout value is used.
Workaround: Make use of just ip nat translation tcp-timeout timeout value command.
Symptom: The router crashes when removing and re-attaching a child policy from/to the parent or when removing and re-adding the fair-queue policy. The issue does not require traffic in the background. It could occur with a policy on a single target, so scaling is not required to hit the problem. It happens primarily on ESP-100, ASR1002-X and 1NG (Nightster). The issue does not impact ESP-5, ESP-10, ESP-20 and ESP-40, ASR1001 and ASR1002. The issue does also NOT impact the ISR and CSR platforms.
Conditions: When removing and re-applying a child policy or a policy that includes fair-queue, the hierarchy grows by one layer each time the policy is re-adding. This result is broken functionality and removing the policy would eventually result into a crash.
Workaround: The workaround is to remove the parent policy, modify the configuration then re-apply the service policy. The issue could also occur dynamically when a subscribe signs off but there is no workaround for this issue in that case.
Symptom: Crash seen on Primary RP due to Null Pointer send during Bulk Policy Map delete.
Conditions: Deleting Bulk Cos Policies.
Workaround: There is no workaround.
Symptom: When the ZBFW SYN cookie protection feature is enabled and is being triggered, the firewall will generate and send SYN packets to the server on behalf of the client. If the response from the server isn't received in time, the firewall will re-generate and resend the SYN packet. In this retransmitted SYN packet, the MSS option is missing and the sequence number is incorrect(it is one number bigger than the ISN).
Conditions: ZBFW SYN cookie protection is configured and is being triggered. Server doesn't respond in time and is causing the firewall to resend the SYN packet to the server.
Workaround: There is no workaround.
Symptom: The TCP RST packets generated by ZBFW are dropped by ZBFW on ASR box.
Conditions: TCP flow specific TCP RST packets generated by ASR to rset the connection to the client and server when "TCP packet inspection" is on.
Workaround: There is no workaround.
Symptom: Multicast RP-Announcement or RP-Advertisement packet is replicated more than one copy per incoming packet. The number of copies is equal to the number of interfaces or io items with IC flag enabled (use the show ip mfib command to get the number of IC interfaces).
Conditions: AUTO-RP filter is configured on PIM interfaces.
Workaround: There is no workaround.
Symptom: After ESP 100 reload, show policy-map interface command counters does not populate results.
Conditions: This condition occurs with an egress service policy on SPA Gigabit Ethernet interface and sending high or low priority traffic.
Workaround: Reload the SPA after FP reload.
Sympt om: If a customer configured snmp server enable traps sbc sla-violation-rev1 csbSLAViolationRev1 trap is not sent.
Conditions: This is a normal operation.
Workaround: There is no workaround.
Sympt om: FMAN-FP traceback: cgm begin batch error.
Conditions: While adding classes to the ZBFW policy.
Workaround: There is no workaround.
Symptom: ESP may reload in B2B NAT ZBFW setup.
Conditions: B2B NAT ZBFW setup with stateful traffic.
Workaround: There is no workaround.
Symptom: Exception to IOS Thread: UNIX-EXT-SIGNAL: Segmentation fault(11), Process = SBC main process.
Conditions: There is no workaround.
Symptom: The ASR1002-X Router reloads with core file reporting CGI_CSR32_CGI_OTHER_LEAF_INT__INT_ECSR_PROTOCOL_ERR interrupt.
Conditions: Only applies to the ASR1002-X Router. This software error is fixed in the IOS XE3.7.4, XE3.9.2, XE3.10.0 and later releases.
Workaround: There is no workaround.
Symptom: cpp_cp process crashes.
Conditions: Change to the parent class of a session, which causes a rate update event to be performed in the QFP hardware. At the same time, ANCP causes rate change on a VLAN shapper using mode-F QoS. The shaper rate change causes the shaper on the VLAN to be removed and then re-applied. Depending upon RP and FP CPU utilization, these events can be processed on the ESP as one QoS transaction. where the sessions parent class has a rate change event and the session is also being moved to an aggregation schedule node on the GE from the VLAN shaper schedule node. And then the shaper is re-applied to the VLAN and the session is moved back to the VLAN shaper. This all occurs in the same QoS transaction/commit on the ESP, causing the ESP to crash.
Workaround: There is no workaround.
Symptom: NAT pool exhaustion with addresses with 0 refcount.
Conditions: This condition occurs while running NAT ALG and when port allocation failure occurs.
Workaround: To recover, execute clear ip nat trans command in off hours (as this is disruptive operation).
Symptom: show ip wccp counters are not updated
Conditions: Configure more than 7 services on interface; disable some services; send traffic which match the last configured service;
Workaround: When disabling service, also delete the configuration on interface.
Symptom: Callflow: Verizon - SIP trunk - CUBE (ASR 1000) - CUSP - Genesys - Interactions IVR.
CUBE does not ACK and BYE (glare handling case) after sending Cancel and receiving 200 Ok for cancel from CUSP.
Conditions: Verizon cancelled the call 300 milliseconds (aprox) after sending the invite, it caused the 200Ok of the invite and the Cancel to cross wire between CUSP and Genesy.
By that time CUSP had already sent 200 Ok for CANCEL to CUBE, thus CUBE did not respond to the following 200 OK (for Invite).
Workaround: There is no workaround.
Symptom: Call flow: Verizon -- CUBE -- CUSP -- Genesys/IVR, transfered with SIP Refer back to PSTN hair-pining the call on CUBE.
When the call is transferred from IVR to PSTN, the codec negotiation with verizon fails, only if the original Invite received included fax capabilities, dropping the call with reason code 47 and hanging the UDP port used.
All subsequent calls that try to re-use the same UDP port for RTP stream are dropped with reason code 47 and provisn RSP fail is logged on show voip fpi stats
Conditions: Hair-pinned calls that received FAX capabilities on original SIP invite from Verizon.
Workaround: There is no workaround.
Symptom: qfp crash
Conditions: handoff from gtpv0 to gtpv1
Workaround: no More Info:
Symptom: ESP crashes.
Conditions: Subscriber session with QoS over tunnel or shaped VLAN.
Workaround: There is no workaround.
Symptom: An ASR with zone-based firewall enabled may drop SIP INVITE packets with the following drop reason:
Router#show platform hardware qfp active feature firewall drop -------------------------------------------------------------------------------
Drop Reason Packets -------------------------------------------------------------------------------
L7 inspection returns drop 1 Router#
Conditions: Application (L7) inspection for SIP must be enabled for the flow.
Workaround: Any of the following workarounds are applicable:
– Disable the port-to-application mapping for SIP with the no ip port-map sip port udp 5060 command. This prevents ZBF from treating UDP/5060 as SIP. Instead, it is treated as simple UDP.
– Use the pass action in both directions instead of inspect. This disables all inspection (even L4) for the traffic.
Symptom: QFP reloads.
Conditions: Rarely occurs when issuing show platform hard qfp active feature nat da stats command. Most likely to occur in CGN mode specifically after switching from classic to CGN mode.
Workaround: There is no workaround.
Symptom: PBHK update failure traceback from CPP-CP. AOM object download failure from FMAN-FP..
Conditions: ISG sessions have PBHK features and RP switch-over.
Workaround: There is no workaround.
Symptom: The Cisco ASR 1000 Series Aggregation Services Router sends a different Acct-Session-Id in the Access-Request and Accounting-Request for the same user.
Conditions: Flex VPN IPSEC remote access is configured.
Workaround: There is no workaround.
Symptom: ESP crashes.
Conditions: On ASR1002-X, ESP100 or ESP200 based platforms, ESP can crash when you have interfaces where the bandwidth can change dynamically and you have a hierarchical QoS policy-map applied.
Workaround: When applying a hierarchical QoS policy-map to ain interface that supports dynamic bandwidth changes, be sure to apply the QoS policy while there are no bandwidth changes to the interface at the same time.
Symptom: gtpv0 policy is not working.
Conditions: gtpv0 traffic.
Workaround: There is no workaround.
Symptoms: ESP crashes in response to a show command.
Conditions: This only causes an ESP crash when the qid specified is an internal queue. It is safe for interface or QoS-created queue. When issuing the show platform hardware qfp [active | standby] infrastructure bqs [schedule | queue] qid command on a ASR1K 1002X, ESP100/FP100, and ESP200/FP200 system.
Workaround: Avoid use of the show command to display internal queues.
PSIRT Evaluation: The Cisco PSIRT has assigned this bug the following CVSS version 2 score. The Base and Temporal CVSS scores as of the time of evaluation are 3.8/3.1: https://intellishield.cisco.com/security/alertmanager/cvssCalculator.do?dispatch=1&version=2&vector=AV:L/AC:H/Au:S/C:N/I:N/A:C/E:F/RL:OF/RC:C No CVE ID has been assigned to this issue. Additional information on Cisco's security vulnerability policy can be found at the following URL: http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html
Symptom: Cisco ASR 1000 Series Aggregation Services Router may crash when customer uses call-policy-set copy source source-address destination destination-address command to create a new call-policy-set.
Conditions: The na-src-address-table is configured within the call-policy-set. Enter this table with na-src-address-table XXX after it was created by call-policy-set copy command.
Workaround: instead of using call-policy-set copy source source-address destination destination-address command, copy and paste the text into config terminal to create a new call-policy-set.
Symptom: QFP reload may occur.
Conditions: When running NAT in CGN mode and doing a removal of a mapping.
Workaround: Switch to classic mode, to mapping removal, switch back to CGN mode.
Symptom: Unknown.
Conditions: Astro can require a core voltage of up to 1.00V. However, the voltage was defaulted to 0.9V for all Astro chips. If an Astro requires 1.0V is on a board, it is only operating at 0.9V and could fail to operate properly at speed.
Workaround: There is no workaround.
Symptom: mplssetvrf bgp routes are not coming up along with multi-vrf PBR.
Conditions: The destination address of the packet is ASR local address. Say, the packet is for us packet.
Workaround: There is no workaround.
Symptom: VTCP is not robust enough when it receives TCP segments with abnormal sequence ID. This may result in FP crash. We observed a TCP packet much older than the current window on customer network.
Conditions: Abnormal sequenced TCP segments are received when VTCP buffering current flows.
Workaround: There is no workaround.
Symptom: The Calling-Station-Id is not sent in the accounting-request.
Conditions: Easy VPN server or Flex VPN remote access is configured along with the "radius-server attribute 31 remote-id" command.
Workaround: There is no workaround.
Symptom: ESP-100 may crash continuously on an ASR1K box with cpp_svr crashes causing the FP to go down.
Conditions: Numerous QoS sessions with a single queue being created on an interface in a per-session basis on a Yoda platform (ASR1002-X/ESP100/ESP200).
Workaround: None at the moment More Info: This bug only affects Yoda platforms with large number of singe queued QoS policies being applied on a per session basis on an interface.
Symptom:
– Initiator sends identity certificate based on 'ca trustpoint' under the
– isakmp-profile.
– However, the responder does not do this. Instead it gets the identity certificate from the *first* trustpoint (out of the list of trustpoints) based on peer's cert_req payload in MM3.
Conditions:
– IKEv1 with RSA-SIg Authentication, where each Peer has two certificates issued by the same CA.
– Each Peer has isakmp profiles defined that match on certificate-map and have 'ca trustpoint' statements with self-identity as fqdn.
Workaround: There is no workaround.
Symptom: Data rate for a QoS shaped MLPPPoA/MLPPPoEoA traffic class may exceed the configured QoS shape rate.
Conditions: This issue will be apparent if a parent or child shaper is defined on the MLPPP bundle interface that is less than the configured PVC data rate.
Workaround: The user can explicitly tell the shaper to account for the ATM Cell Overhead by appending the "account user-defined 0 atm" configuration option to the shaper configuration.
Example:
shape rate rate account user-defined 0 atm
Note that if the session is already active when modifying the QoS policy-map, the session may need to be restarted for the QoS modification to take affect.
This issue will be addressed in the upcoming XE3.8, XE3.10, and later releases. This issue will not be addressed in XE3.8 and XE3.9 and will require migration to XE3.10 or later releases to pick up this fix when available.
Symptom: Call flow: Verizon -- CUBE -- CUSP -- Genesys/IVR, transfered with SIP Refer back to PSTN hair-pining the call on CUBE. When the call is put on hold to be transferred from IVR to PSTN, the CODEC negotiation fails, dropping the call with reason code 47 and hanging the UDP port used. All the subsequent calls that try to reuse the same UDP port for RTP stream are dropped with reason code 47 and provison RSP failure is logged on show voip fpi stats command.
Conditions: Hair-pinned calls that receive multiple M-Lines on the SDP received from Verizon on the original SIP Invite.
Workaround: There is no workaround. Reload of router is required to clear hung UDP ports.
Symptom: When ASR1000 connect with ISO HDLC equipment, the ATOM PW traffic could not transparent successfully.
Conditions: In L2VPN ATOM PW configuration, AC on the PE is CISCO HDLC encapsulation, and CE equipment is ISO HDLC.
Workaround:
– CE configure CISCO HDLC.
– CE configure as the FR, and PE configure as HDLC.
Symptom: The ESP crashes when updating a highly scaling configuration with a large number of flow-controllable nodes. The crash could be observed during dynamic reconfigurations such as changing the rates of a scheduling node, e.g. an ATM VC due to changing L2 shaping or QOS via MQC.
The crash could also occur due to growing a scheduling node or moving an ATM VC from one class-of-service node to another.
There are several other scenarios that could lead to a transformation of a hierarchy in order to lay out the tree correctly to meet the hardware requirements. One such example is applying a flat policy to or removing a child policy from a policy attached to an ATM VC.
Conditions: While transforming a hierarchy, there are hardware primitives used to execute the update logic safely. One of requirements for this procedure is to move flow-control from the old tree to the new tree in a particular order to prevent packets from getting out of order. The BQS resource manager had a bug that caused the update to deplete internal flow-control IDs.
Workaround: There is no workaround.
Symptom: After reloading ASR1k with redundancy RP/FP, HDLC pass through configuration remains but control flag actually lost.
Conditions: ASR1k with redundancy RP/ESP Configured HDLC pass-through and reload, or FP switchover for two times.
Workaround: Manually re-config the CLI after reload.
Enter configuration commands, one per line. End with CNTL/Z.
UUT-ASR2-1006HA(config)#platform l2vpn hdlc-pass-through
UUT-ASR2-1006HA#sh plat hard qfp ac fea xcon cli intern
Platform Xconnect global configuration
L2VPN HDLC pass through control flag: TRUE
Symptom: ASR1k:support of ignore-dtr on 4XT-Serial spa.
Conditions: There is no condition.
Workaround: There is no workaround.