The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This chapter describes some common problems and how to troubleshoot them and contains the following sections:
To clear all the AppNav-XE statistics or just certain statistics, use the following command:
Use the following debug commands to trace control plane activities:
Use the following command to display unplanned packet drops:
The following are the reasons for which packets may drop:
To display the data path CPU utilization, use the following command:
Use the following command to show statistics about the data path memory use.
Note The value for In Use DRAM memory must be less than 90 percent of the value for Total DRAM memory; otherwise, the AppNav-XE component stops optimizing new flows.
The output of the following debug command is displayed as a log file named /tmp/fp/trace/cpp_cp_ Fx -0.log under the FP shell, where Fx is either F0 or F1 depending on the active FP module. You need a shell license to access the FP shell.
If you do not have shell access, you can use the test platform software trace slot fp act cpp-control-process rotate command to force the log to flush to bootflash:tracelogs.
Each of the above categories (other than drop, which has no level) has the following four levels:
To limit the number of debug messages, we recommend that you only enable the error debug level first and then slowly reduce the debug level.
You can also use the following command to check on packets dropped by the router. The command lists all the packets that were dropped with a reason. If you see AppNav drop reasons, you can enable the debug drop command to see the actual packet drops inside the trace logs.
Use the following debug commands to trace the AppNav service node auto discovery feature:
The following debug commands are available to debug the application configuration, installation, activation, and status:
Use the following commands to debug the operation of the EZConfig program:
If traffic is not redirected properly, ensure that “service-insertion waas” is present on interfaces on which the traffic is supposed to be intercepted. Issue the show service-insertion status command to verify this.
The show service-insertion statistics connection command indicates whether traffic is passed through or redirected. If traffic is passed through instead of being redirected, use the show policy-map target service-context context_name passthru-reason command to find out the reason. For details, see the “Displaying Pass Through Reason Statistics” section.
You can also monitor the service node counters. See the “Displaying Per Service Node and Service Node Group Statistics” section.
The term “Initial Redirect” indicates that flows are being redirected to the service nodes. If the flows are not being redirected to the service nodes, maybe the policy did not cover the traffic type.
The “Initial Redirect -> Passthrough” counter indicates that the service node has decided to pass-through the flow. This is likely due to policies on the service node.
The “Redirect -> Passthrough” counter indicates that the service node later decided to pass-through the flow. This is likely due to lack of a peer WAAS device. Two WAAS devices are needed along the path to optimize a flow.
If connections are passed through and you are using an AppNav Controller group that has two or more AppNav Controllers, it is possible that the cluster state is degraded instead of operational. This means that the AppNav Controller view is not the same on each of the AppNav Controllers.
To check the cluster state and the stable AppNav Controller view on each of the AppNav Controllers, use the following command:
The reason for the difference in AppNav Controller views on the AppNav Controllers may be due to a mismatch in the AppNav Controller group configuration on the AppNav Controllers or due to a connectivity problem between the AppNav Controllers.
It is also useful to check the alarms on each of the AppNav Controllers by using the following command that also suggests corrective actions:
Cluster membership manager has detected a discrepancy in the AC view of peer ACs. Optimization will be turned off on this device for cluster consistency.
AppNav controller is unreachable.
Cluster protocol detected failure of the peer AC. This could happen due to several reasons - configuration mismatch or network issues preventing communication between the ACs or the AC actually being down.
If no traffic is redirected to a particular service node and you are using an AppNav Controller group with two or more AppNav Controllers, it is possible that the service node is excluded. This happens when the service node view is not the same on each of the AppNav Controllers.
To check the stable service node view on each of the AppNav Controllers, use the show service-insertion service-context command.
The reason for the difference in service node views could be due to a mismatch in the service node group configuration on the AppNav Controllers or due to a connectivity problem between one or more of the AppNav Controllers and the excluded service node.
To check if any service nodes are excluded or unreachable, look for the SN_excluded and SN_unreachable alarms by using the show service-insertion alarms detail support command on each of the AppNav Controllers.
This could be due to a mismatch in the VRF names for the traffic seen by the AppNav Controllers in the ACG.
Check the output of the show service-insertion statistics connection summary command for the counter for Flow Sync Failures due to vrf mismatch.
A connection might be considered “hung” for various reasons. In many cases, it helps to use telnet to simulate a connection to the server. For example, enter telnet HTTP_server 80.
If the connection hangs during the TCP 3-way handshake, verify that both the connection and the route to the service node are properly set up.
If the connection hangs after the connection was established, verify the connection along the path. Make sure that the MTU along the path is correct.
Use the show service-insertion statistics connection command on the AppNav Controller and the show statistics connection command on the service node to cross check the connections between the AppNav Controller and the service node.
Use the show platform hardware qfp active statistics drop command to check for packet drops.
You can usually see the reason for the connection reset by issuing the show statistics connection closed command and the show statistics connection closed conn-id connection_ID command on the service node. Capturing packets is also useful in analyzing the reason for the connection reset.
If you use ISR-WAAS as the service node, verify the DRE peer ID by using the show dre command on the service node and making sure that both of the ISR-WAAS DRE peer IDs are not the same. vm init is required for ISR-WAAS during configuration.
Use the show platform hardware qfp active statistics drop command to check for dropped packet.
Some older service nodes may not support all application accelerators.
Individual application accelerators, such as the video application accelerator, require a separate license.
If the system displays an ERROR_NOTIFY syslog message when you enable the service-insertion waas command on the interface, it could be that the AppNav-XE component failed to initialize due to low memory. Check the amount of memory by using the following command:
If the available memory is less than 10 percent of the total memory, the AppNav-XE component may not be able to initialize, which results in no flows being redirected.
If the output of the show policy-map target service-context waas/1 command is blank, instead of listing the AppNav policy being used, it may indicate that the system was unable to initialize.
Both the AppNav Controller and the service nodes have a limit on the number of flows that they can support. On the AppNav Controller, the limit is 2 million flows. Beyond that, all flows are passed through. If you exceed the limit, the system displays the following error message:
The flow limit may be reached in advance due to available memory. In this case, the system displays the following syslog message:
In both cases, when the existing flows are completed and the number of flows dips below the threshold, flows are optimized again.
The following errors can happen during the application installation:
These errors might occur due to an incompatibility issue between the ISR-WAAS OVA and the Cisco IOS-XE image. Ensure that you have the Cisco IOS-XE Release 3.9 image and the latest compatible OVA package. When selecting an OVA profile to use during activation, make sure that the Cisco ISR 4451-X has enough memory and storage resources for the profile selected.
If your performance is degraded, first check the CPU utilization on the application using the following command:
If the application CPU is at 100 percent, check the bandwidth and compression ratio in the ISR4451-XWAAS application. If the bandwidth is within the profile limits, enable ISR-WAAS debugging. If the application CPU is not at 100 percent, check the router CPU using the following command:
If the platform CPU is at 100 percent, follow the standard platform debug process. If the application is bound by bandwidth and there is CPU headroom on the Cisco ISR 4451-X, consider upgrading the application profile by reinstalling the ISR-WAAS application and selecting a higher profile.
Another reason for degraded performance is application disk failure. Check the ISR-WAAS installation log and ISR-WAAS alarms on WCM to see if any disk failure has occurred.
If ISR-WAAS crashes, traffic should not be impacted because the AppNav-XE component stops redirecting traffic.
Note Crashes in ISR-WAAS application accelerators do not impact connectivity because traffic goes into bypass mode.
If the AppNav Controller does not respond to a WAAS TCP trace, the system forwards the TCP trace to the service node and the service node generates a response along with a list of service nodes along the path.
You can access the ISR-WAAS application by using the following CLI command:
A sample ISR-WAAS configuration is displayed below:
Note This sample does not include the policy configuration.