Cisco Network Module Enhanced Application Performance Assurance Troubleshooting Guide - Troubleshooting Specific Scenarios [Cisco Application Networking Services Modules]

Table Of Contents

Troubleshooting Specific Scenarios

ISR/NME-APA Platform Troubleshooting

The NME-APA does not come up

The NME-APA does not power up

During installation of the NME-APA image, the NME-APA card does not come up

Cannot ping the NME-APA management interface from the ISR Router

Cannot Telnet to NME-APA management interface from other station

During installation of the NME-APA image, an error is received

The NME-APA does not see any diverted traffic

Traffic Classification Troubleshooting

No RDRs after pushing the traffic

RDR File Troubleshooting

No RDRs are received by the external entity

No RDR files are generated

The RDR files are not retrieved

SNMP Troubleshooting

Cannot get MIB values

APADC Application Troubleshooting

About Applying Configurations

Connect Screen Troubleshooting

Cannot connect to "Available" device

Cannot add a device

Device stays "Offline"

Device Configuration Screens Troubleshooting

Cannot access Device Configuration Screen

Device Configuration is erased

Apply fails

Retrieve fails

Fault Management Screen Troubleshooting

Notification is not received

Statistics Screen Troubleshooting

Installation Screen Troubleshooting

Install process fails with the error "Cannot connect to device"

Install process fails with the error "Image file type and file mismatch"

Traffic Management Screen Troubleshooting

Cannot apply TM configuration to device

Reporting Screen Troubleshooting

RDR retrieval ("RetrieveNow") fails

Cannot "Activate" scheduled task

Cannot "Add" a scheduled task

RDR Retrieval continues to be in "In Progress" state

No "Last Retrieval" information is displayed

Cannot create report instance

"Run report JPEG" is disabled for certain reports

Report results are empty

Report fails to generate output (PROCESS_ERROR)

User Management Screen Troubleshooting

Error stating FTP not available when retrieving or applying

Admin Screen Troubleshooting

Cannot Add a new APADC user

Troubleshooting Specific Scenarios

This module includes procedures that operators can use to troubleshoot issues when using the Cisco Network Enhanced Module for Application Performance Assurance (NME-APA) and the Cisco Application Performance Assurance Device Console (APADC).

•ISR/NME-APA Platform Troubleshooting

•Traffic Classification Troubleshooting

•RDR File Troubleshooting

•SNMP Troubleshooting

•APADC Application Troubleshooting

ISR/NME-APA Platform Troubleshooting

•The NME-APA does not come up

•The NME-APA does not power up

•During installation of the NME-APA image, the NME-APA card does not come up

•Cannot ping the NME-APA management interface from the ISR Router

•Cannot Telnet to NME-APA management interface from other station

•During installation of the NME-APA image, an error is received

•The NME-APA does not see any diverted traffic

The NME-APA does not come up

Step 1 Verify that the NME-APA has a service-module IP address and default gateway configured on the ISR.

The NME-APA does not power up

Step 1 Verify that the NME-APA is firmly seated in the ISR slot and that the retaining screws are tightened.

Step 2 Make sure the slot being used can support the NME-APA card.

Step 3 Verify that you are not attempting to use an NME-APA-E3 card in an ISR 2800 or an NME-APA-E2 card in an ISR 3800.

During installation of the NME-APA image, the NME-APA card does not come up

Step 1 Check the IOS version on the ISR using the show version command.

Step 2 Make sure that the IOS version supports the NME-APA card.

Step 3 If the IOS version does not support the NME-APA card, update the ISR IOS image to a version that supports the NME-APA card.

Cannot ping the NME-APA management interface from the ISR Router

Step 1 Verify that there is a valid route to and from the NME-APA. (This might require a static route to the NME-APA on the ISR if the Integrated Service Engine interface is set to ip unnumbered.)

Cannot Telnet to NME-APA management interface from other station

Step 1 Verify that there is a valid route to and from the NME-APA.

Step 2 Use the NME-APA ShellConn command i to see whether the telnetServer process is up and running.

During installation of the NME-APA image, an error is received

During installation of the NME-APA image, the following error is received:
Pycurl.error: invalid return value for write callback 
Error: Operation Aborted.
Step 1 The package name has the wrong extension. Change the extension to .pkg.

The NME-APA does not see any diverted traffic

From the ISR IOS CLI run show running-config interface integrated-Service-Engine <slot/0>. If !SCE traffic-management Disabled appears in the output, the NME-APA card is currently not enabled. This could be due to three reasons:

•NME-APA is not enabled

•NME-APA is overloaded and diversion has been explicitly disabled by the card itself

•SCOS crashed and disables traffic diversion to the NME-APA

To verify that the NME-APA is properly enabled, complete the following steps:

Step 1 Wait a few minutes and then try again. Also, verify that you have the correct card for this ISR model.

Step 2 Make sure that traffic diversion is enabled on the designated interface. You should see that the configuration, service-module apa traffic-management monitor, is set to that interface.

Step 3 Make sure that traffic is arriving from the designated interface. (Use the ISR IOS CLI command show interfaces <fastEthernet or GigabitEthernet><slot><port>counters. The counters should increase when you pump the traffic. If there is no increase, the traffic is not getting to the ISR chassis.)

Step 4 Use the NME-APA CLI command show interface fastEthernet 0/1 counters to see whether traffic has been diverted to NME-APA from ISR. The output will indicate whether the packets are diverted to NME-APA or are discarded by NME-APA.

To see if the NME-APA is overloaded and has explcitly disabled traffic diversion, look for the following messages:

A packet queue size within the Ethernet driver queue exceeding 48 MB is considered to be extremely overloaded and will cause the NME-APA to send a diversion disable message to the ISR.

Diversion of traffic will be re-enabled after 300 seconds. The NME-APA will then return to 'Full Functionality' mode.

The following sequence of messages will appear:

1. If the NME-APA is no longer inline, and you will see the following messages:
e1000_clean_rx_irq_ps::Massive overload detected. BVI FIFO size is more than 50331648 
bytes..\n", MASSIVE_OVERLOAD_THRESHOLD.
send_rbcp_message:workqueue::DIVERSION IS DISABLED, NME-APA is not inline now. Not 
receiving any packets.
send_rbcp_message:workqueue::DIVERSION will be enabled in 300 seconds. Clearing 
queue...
2. While in the disabled state, you will also see the following message in the running configuration of the NME-APA:
!APA traffic-management Disabled
3. When diversion of traffic is re-enabled and the NME-APA returns to 'Full Functionality' mode, you will see the following message:
send_rbcp_message:workqueue::DIVERSION IS ENABLED, NME-APA is inline now. Processing 
resumed...
If SCOS crashes and disables traffic diversion to the NME-APA, complete the following steps:

Step 1 Look for the following messages on the NME-APA shellConn session:
defaultFaultHandler was triggered because of signal 11 Send 
SCOS_RBCP_DIVERSION_DISABLED message...
defaultFaultHandler was triggered because of signal 11 Cannot open RBCP_RD_PIPE...
ERROR: Failed to open %
defaultFaultHandler was triggered because of signal 11 Cannot open RBCP_RD_PIPE...:
This is an unrecoverable error. A segmentation fault in the SCOS has caused the SCOS process to exit.

Step 2 Store a copy of the logs.

Step 3 Reload or reset the module

Traffic Classification Troubleshooting

No RDRs after pushing the traffic

Step 1 Make sure that traffic is diverted to NME-APA and not discarded by NME-APA.

See The NME-APA does not see any diverted traffic.

Step 2 Make sure that the packets are not filtered (bypassed or dropped) by NME-APA.

Use the NME-APA CLI debug slot 0 ppc 0 func RC_Cls to keep track of the counters. If the numbers for drop or bypass increase after the traffic is sent, packets have been filtered.

The filtered traffic from APADC is used to configure the flow filter. For example, DNS packets are bypassed by default, so no RDRs will be generated by DNS traffic.

You can also use the CLI command, debug slot 0 ppc 0 show-counters.

Step 3 Ensure that the line card is not shut down.

Step 4 Make sure that the SML application is installed.

Use the NME-APA CLI show version to see whether the information about SML application is displayed.

Step 5 If the SML application is not installed, install the PQI file from the APADC.

Step 6 Make sure that traffic is delivered to SML Application.

Check the value of Traversed PDUs from the output of the NME-APA ShellConn command, debug slot 0 ppc 0 func sr. If the value is 0, the SML application is not receiving traffic and no RDRs will be generated. In this case, it is possible that the traffic is not classified as a flow in SCOS. Turn on the debugging flag for the HwSim module.

1. Before you push traffic, enter the ShellConn command, LC_CS_on

2. After the traffic is pushed, enter the ShellConn command, LC_CS_off

Note This command should be used in a controlled environment; it may interfere with live traffic.

The log is in the file /root/tffs0/cs.txt. Send this file to DEs for analysis.

Step 7 Save the RDRs.

The RDRs can either be saved to a file (default) or sent to an external entity such as the CM or rdrHandler. If the destination IP address of the RDR-Formatter is configured, the RDRs are sent to the external entity. Otherwise, they are saved to a file. See RDR File Troubleshooting for details.

RDR File Troubleshooting

•No RDRs are received by the external entity

•No RDR files are generated

•The RDR files are not retrieved

No RDRs are received by the external entity

Step 1 Make sure that the configuration of the RDR-Formatter is correct (for instance, the IP address and the port number).

Use the NME-APA CLI command show RDR-Formatter to see whether the connection between the NME-APA and the external entity is up. If the connection is down, the RDRs are queued. Check the connection between NME-APA and the external entity. (Note that this is for debugging purposes only. The RDRs should be written to files in the normal case.)

The same command will also show the number of RDRs being queued. If no RDRs are queued, refer to Traffic Classification Troubleshooting to see why no RDRs are generated.

No RDR files are generated

Step 1 Confirm that RDRs are being generated.

Use the NME-APA CLI command show RDR-Formatter to see whether the RDRs are queued. If no RDRs are queued, refer to Traffic Classification Troubleshooting to see why no RDRs are generated.

Step 2 Use the ShellConn command, rdrShowSts, to see how much longer it has to wait before the RDRs are written to a file.

By default, these RDRs are written to a file every 30 minutes.

Step 3 After this time has elapsed, use the ShellConn command rdrShowAllFiles to see whether the file has been created.

The ShellConn command, rdrHelp, provides a list of commands for debugging RDRs.

The RDR files are not retrieved

Step 1 Make sure that the ftp server is enabled.

Use the NME-APA CLI show ip ftp-server to see whether the server is enabled.

SNMP Troubleshooting

Cannot get MIB values

Step 1 Make sure that SNMP Agent is enabled.

Use the NME-APA CLI show snmp to see whether SNMP Agent is enabled. If not, use the CLI snmp enable under configuration mode to enable it.

Step 2 Check that the console displays SNMP Server started.

If not, enter the ShellConn command debugScripts 1 from the console, and then disable and enable SNMP Agent again.

Step 3 The ShellConn commands SNMP_AGENTX_Help and SNMP_AGENT_Help provide a list of commands for SNMP debugging.

APADC Application Troubleshooting

•About Applying Configurations

•Connect Screen Troubleshooting

•Device Configuration Screens Troubleshooting

•Fault Management Screen Troubleshooting

•Statistics Screen Troubleshooting

•Installation Screen Troubleshooting

•Traffic Management Screen Troubleshooting

•Reporting Screen Troubleshooting

•User Management Screen Troubleshooting

•Admin Screen Troubleshooting

About Applying Configurations

It is important to understand that there are separate configurations for each of the following features:

•Device Management (DM)

•Traffic Management (TM)

•User Management (UM)

For example, when viewing a screen under DM, the Apply button on the screen applies only the DM configuration to the APA process. If you also want to edit the TM and/or UM configurations, you must click Apply on the TM and UM screens to push those configurations to APA.

When you log in to APA, only the DM configuration is automatically retrieved from the APA process. Click Retrieve on TM and UM screens to retrieve the TM and UM configurations from APA.

If the configuration has somehow been modified by means other than APADC, (this can happen, for example, if a change is entered via the CLI, although we do not recommend using the CLI for configuration changes), then also click Retrieve to retrieve the configurations.

In cases where different aspects of a configuration are modified on multiple pages, each page contains a Save button. It is important to click Save unless you are going to immediately Apply the modified configuration after editing. If you do not click Save and you navigate to a different page, any configuration changes made on the current page will be lost! So, if you intend to modify the configuration on several pages and then Apply all the changes at once, you must click Save on each page before moving to a new page.

Connect Screen Troubleshooting

•Cannot connect to "Available" device

•Cannot add a device

•Device stays "Offline"

Cannot connect to "Available" device

Step 1 Go to Admin >Admin User Management.

Step 2 Verify that the logged-in APADC user appears in the "User" column.

Step 3 Verify that the NME-APA trying to connect is a "configured device" for the APADC user.

Step 4 Click on the device name in the "configured device" column.

Step 5 Verify that the device user name is correctly configured and is the same as the username configured on the NME-APA (by Telnet).

Step 6 If the problem persists, create a new user account on the device and use this account to connect to the device by configuring it under Admin >Admin User Management.

Step 7 Click Help, to review integrated help for more details.

Step 8 If the problem persists, report to Cisco Systems, Inc.

Cannot add a device

Step 1 Verify that there are no more than ten devices already added to the APADC application.

Step 2 If more than ten devices are already added to the APADC application, delete an existing device to add a new device.

Step 3 Verify the correct network address for the device to be added.

Step 4 Click Help, to review integrated help for more details.

Step 5 If the problem persists, report to Cisco Systems, Inc.

Device stays "Offline"

Step 1 Ping the device from the web server running APADC.

Step 2 Restart the Tomcat Windows service from Windows Administrative Tools > Services.

Step 3 Restart the application server.

Step 4 Click Help, to review integrated help for more details.

Step 5 If the problem persists, report to Cisco Systems, Inc.

Device Configuration Screens Troubleshooting

•Cannot access Device Configuration Screen

•Device Configuration is erased

•Apply fails

•Retrieve fails

Cannot access Device Configuration Screen

Step 1 Verify that at least one device is in the "Connected" state.

Step 2 Verify that the user role allows access to this screen. Refer to the Cisco Network Module Enhanced Application Performance Assurance Troubleshooting Guide for details.

Device Configuration is erased

Step 1 Click Save before toggling to a different tab after making configuration changes.

Step 2 Click Apply to commit configuration data to NME-APA.

Step 3 Any configuration on a tab is lost if it is not saved before user moves to a different tab.

Step 4 Click Help, to review integrated help for more details.

Step 5 If the problem persists, report to Cisco Systems, Inc.

Apply fails

Step 1 Verify that the "Connected" device is still available and alive.

Step 2 Click Help, to review integrated help for more details.

Step 3 If the problem persists, report to Cisco Systems, Inc.

Retrieve fails

Step 1 Verify that the "Connected" device is still available and alive.

Step 2 Click Help, to review integrated help for more details.

Step 3 If the problem persists, report to Cisco Systems, Inc.

Fault Management Screen Troubleshooting

Notification is not received

Step 1 Verify that the Notification is not suppressed:

a. Go to the Fault Configuration screen.

b. Look for the Notification ID for the missing notification.

c. Verify that the Notification ID is present in the table.

d. Check to see that the "Suppressed" column has the value "No", rather than "Yes".

Step 2 Click Refresh.

Step 3 Verify that NME-APA is sending a Notification.

Step 4 Click Help, to review integrated help for more details.

Step 5 If the problem persists, report to Cisco Systems, Inc.

Statistics Screen Troubleshooting

All Statistics are accumulated via SNMP queries. See SNMP Troubleshooting

Installation Screen Troubleshooting

•Install process fails with the error "Cannot connect to device"

•Install process fails with the error "Image file type and file mismatch"

Install process fails with the error "Cannot connect to device"

Step 1 Ensure the Web user has credentials on the device as a level-15 user. (If the device user corresponding to the Web user has a level 0/5/10 credentials, the install process will fail.)

Install process fails with the error "Image file type and file mismatch"

The image file type selected for install (pqi or pqb) must have a filename extension of .pqi or .pqb. The files are not transferred from the user PC to the APADC server if a filename mismatch occurs. (Further semantic checks are performed in the next stage of the process to verify that the image files are of the expected type.)

Step 1 Verify that the installation file has the correct filename extension.

Traffic Management Screen Troubleshooting

Cannot apply TM configuration to device

Step 1 Check the server log for a connection-related exception.
2007-07-25 11:09:11,929  INFO [http-8080-Processor20] com.pcube.apps.engage.Engage - 
Sending configuration to SCE ... 
connection to SCE failed: server refused connection: unable to open connections - max 
number of sessions exceeded 
com.pcube.apps.engage.ConnectionFailedException: server refused connection: unable to open 
connections - max number of sessions exceeded 
	at com.pcube.apps.engage.Connection.login(Connection.java:414) 
	at com.pcube.apps.engage.Engage.login(Engage.java:138) 
	at com.cisco.scabb.servconf.mgmt.SCABB.login(SCABB.java:81) 
	at 
com.cisco.scate.web.mgt.tm.ServiceConfigManager.applyServiceConfig(ServiceConfigManager.ja
va:115) 
	at 
com.cisco.scate.web.ui.tm.service.ClassTreeAction.showClassTree(ClassTreeAction.java:159)
If there is such a connection-related exception and reapplying the TM configuration to the device still fails, the connection between the APADC server and the device is maxed out.

Step 2 Restart the APADC server (Tomcat).

Reporting Screen Troubleshooting

•RDR retrieval ("RetrieveNow") fails

•Cannot "Activate" scheduled task

•Cannot "Add" a scheduled task

•RDR Retrieval continues to be in "In Progress" state

•No "Last Retrieval" information is displayed

•Cannot create report instance

•"Run report JPEG" is disabled for certain reports

•Report results are empty

•Report fails to generate output (PROCESS_ERROR)

RDR retrieval ("RetrieveNow") fails

Step 1 Verify that the "Connected" device is still available and alive.

Step 2 Verify that it is possible to FTP to the NME-APA from the APADC server using the device username and password associated with the APADC user from the Admin screen.

If this step fails, refer to Device Configuration Screens Troubleshooting and verify users configured on the device. Retrieval will fail if an FTP attempt is made with a username and password that do not exist on the NME-APA device.

Step 3 Verify that enough hard-disk space is available on the APADC server.

Cannot "Activate" scheduled task

Step 1 Verify that the task has not expired.

if the start time is in the past, make sure that the difference between current time and the task start time is greater than the task recurrences times task interval.

Step 2 Verify that the NME-AP device is still available and alive.

Step 3 Verify that it is possible to FTP to the NME-APA from the APADC server using the username "admin" and enable15 password.

If this step fails, refer to Device Configuration Screens Troubleshooting, and reconfigure the enable15 password on the device.

Step 4 Verify that enough hard-disk space is available on the APADC server.

Cannot "Add" a scheduled task

Step 1 Verify that at least one NME-AP device is added to APADC.

Step 2 Only one scheduled task can be added at a time. If there is a task, delete the task and try again.

RDR Retrieval continues to be in "In Progress" state

Step 1 Verify that the NME-AP device is still available and alive.

Step 2 Verify that it is possible to FTP to the NME-APA from the APADC server using the username "admin" and enable15 password.

Step 3 Verify that enough hard-disk space available on the APADC server

If this step fails, refer to Device Configuration Screens Troubleshooting, and reconfigure the enable15 password on the device.

Step 4 Check the server logs for OutOfMemory Errors.

Restart the Tomcat server in the case of such errors.

No "Last Retrieval" information is displayed

Step 1 Verify that the <APADC_ROOT_DIR>/scate-config/rdrloader.xml file exists and is not empty.

Step 2 Try "RetrieveNow" to see if the information gets updated.

If the information is not updated, there could be problems retrieving data from the NME-APA. See RDR retrieval ("RetrieveNow") fails.

Step 3 If there is a "Scheduled Task", make sure that it is still active.

Cannot create report instance

Step 1 Traffic management configuration must be applied by APADC. Verify that the NME-APA TM was configured using the APADC.

Step 2 Verify that the "Last Retrieval" information (Device IP and time) on the Data Retrieval page is not "Unknown".

Step 3 Verify that there is RDR data in the DB. If the DB is empty, go to the RDR Retrieval page and perform "RetrieveNow".

"Run report JPEG" is disabled for certain reports

This behavior is by design. A few report types do not support the JPEG format and so "Run Report JPEG" is disabled. CSV reports can be generated for these report types.

Report results are empty

Step 1 Traffic management configuration must be applied by APADC. Verify that NME-APA TM was configured using the APADC. Verify that the scateinfra.INI_VALUES table is not empty in the MySQL database.

Step 2 Verify that the "Last Retrieval" was done within the last 24 hours.

Step 3 Verify that there is RDR data in the DB.

Step 4 Verify that the APADC server time zone is set correctly.

Report fails to generate output (PROCESS_ERROR)

Step 1 Verify that the report parameters do not contain invalid characters.

Step 2 Open a command prompt, cd to the SCA Reporter folder and run the report execution command. The report execution command line can be found in the APADC log.

Step 3 If the previous step successfully generates a report result, remove all optional parameters and try again.

User Management Screen Troubleshooting

Error stating FTP not available when retrieving or applying

Step 1 Try again. A new FTP connection to the device will be created to continue the operation.

Admin Screen Troubleshooting

The Admin Screen is used to:

•Add/Edit/Delete new APADC users

•Add/Edit/Delete new devices to APADC users

•Program device username, password, and access-level to APADC users

Cannot Add a new APADC user

Step 1 Verify that the maximum number of APADC users (100) has not been reached.

	Cisco Network Module Enhanced Application Performance Assurance Troubleshooting Guide
	Troubleshooting Specific Scenarios
Cisco Network Module Enhanced Application Performance Assurance Troubleshooting Guide About this Guide NME-APA Troubleshooting Concepts Troubleshooting Specific Scenarios	Download this chapter Troubleshooting Specific Scenarios Download the complete book Cisco Network Module Enhanced Application Performance Assurance Troubleshooting Guide (PDF - 800 KB) Table Of Contents Troubleshooting Specific Scenarios ISR/NME-APA Platform Troubleshooting The NME-APA does not come up The NME-APA does not power up During installation of the NME-APA image, the NME-APA card does not come up Cannot ping the NME-APA management interface from the ISR Router Cannot Telnet to NME-APA management interface from other station During installation of the NME-APA image, an error is received The NME-APA does not see any diverted traffic Traffic Classification Troubleshooting No RDRs after pushing the traffic RDR File Troubleshooting No RDRs are received by the external entity No RDR files are generated The RDR files are not retrieved SNMP Troubleshooting Cannot get MIB values APADC Application Troubleshooting About Applying Configurations Connect Screen Troubleshooting Cannot connect to "Available" device Cannot add a device Device stays "Offline" Device Configuration Screens Troubleshooting Cannot access Device Configuration Screen Device Configuration is erased Apply fails Retrieve fails Fault Management Screen Troubleshooting Notification is not received Statistics Screen Troubleshooting Installation Screen Troubleshooting Install process fails with the error "Cannot connect to device" Install process fails with the error "Image file type and file mismatch" Traffic Management Screen Troubleshooting Cannot apply TM configuration to device Reporting Screen Troubleshooting RDR retrieval ("RetrieveNow") fails Cannot "Activate" scheduled task Cannot "Add" a scheduled task RDR Retrieval continues to be in "In Progress" state No "Last Retrieval" information is displayed Cannot create report instance "Run report JPEG" is disabled for certain reports Report results are empty Report fails to generate output (PROCESS_ERROR) User Management Screen Troubleshooting Error stating FTP not available when retrieving or applying Admin Screen Troubleshooting Cannot Add a new APADC user Troubleshooting Specific Scenarios This module includes procedures that operators can use to troubleshoot issues when using the Cisco Network Enhanced Module for Application Performance Assurance (NME-APA) and the Cisco Application Performance Assurance Device Console (APADC). •ISR/NME-APA Platform Troubleshooting •Traffic Classification Troubleshooting •RDR File Troubleshooting •SNMP Troubleshooting •APADC Application Troubleshooting ISR/NME-APA Platform Troubleshooting •The NME-APA does not come up •The NME-APA does not power up •During installation of the NME-APA image, the NME-APA card does not come up •Cannot ping the NME-APA management interface from the ISR Router •Cannot Telnet to NME-APA management interface from other station •During installation of the NME-APA image, an error is received •The NME-APA does not see any diverted traffic The NME-APA does not come up Step 1 Verify that the NME-APA has a service-module IP address and default gateway configured on the ISR. The NME-APA does not power up Step 1 Verify that the NME-APA is firmly seated in the ISR slot and that the retaining screws are tightened. Step 2 Make sure the slot being used can support the NME-APA card. Step 3 Verify that you are not attempting to use an NME-APA-E3 card in an ISR 2800 or an NME-APA-E2 card in an ISR 3800. During installation of the NME-APA image, the NME-APA card does not come up Step 1 Check the IOS version on the ISR using the show version command. Step 2 Make sure that the IOS version supports the NME-APA card. Step 3 If the IOS version does not support the NME-APA card, update the ISR IOS image to a version that supports the NME-APA card. Cannot ping the NME-APA management interface from the ISR Router Step 1 Verify that there is a valid route to and from the NME-APA. (This might require a static route to the NME-APA on the ISR if the Integrated Service Engine interface is set to ip unnumbered.) Cannot Telnet to NME-APA management interface from other station Step 1 Verify that there is a valid route to and from the NME-APA. Step 2 Use the NME-APA ShellConn command i to see whether the telnetServer process is up and running. During installation of the NME-APA image, an error is received During installation of the NME-APA image, the following error is received: Pycurl.error: invalid return value for write callback Error: Operation Aborted. Step 1 The package name has the wrong extension. Change the extension to .pkg. The NME-APA does not see any diverted traffic From the ISR IOS CLI run show running-config interface integrated-Service-Engine <slot/0>. If !SCE traffic-management Disabled appears in the output, the NME-APA card is currently not enabled. This could be due to three reasons: •NME-APA is not enabled •NME-APA is overloaded and diversion has been explicitly disabled by the card itself •SCOS crashed and disables traffic diversion to the NME-APA To verify that the NME-APA is properly enabled, complete the following steps: Step 1 Wait a few minutes and then try again. Also, verify that you have the correct card for this ISR model. Step 2 Make sure that traffic diversion is enabled on the designated interface. You should see that the configuration, service-module apa traffic-management monitor, is set to that interface. Step 3 Make sure that traffic is arriving from the designated interface. (Use the ISR IOS CLI command show interfaces <fastEthernet or GigabitEthernet><slot><port>counters. The counters should increase when you pump the traffic. If there is no increase, the traffic is not getting to the ISR chassis.) Step 4 Use the NME-APA CLI command show interface fastEthernet 0/1 counters to see whether traffic has been diverted to NME-APA from ISR. The output will indicate whether the packets are diverted to NME-APA or are discarded by NME-APA. To see if the NME-APA is overloaded and has explcitly disabled traffic diversion, look for the following messages: A packet queue size within the Ethernet driver queue exceeding 48 MB is considered to be extremely overloaded and will cause the NME-APA to send a diversion disable message to the ISR. Diversion of traffic will be re-enabled after 300 seconds. The NME-APA will then return to 'Full Functionality' mode. The following sequence of messages will appear: 1. If the NME-APA is no longer inline, and you will see the following messages: e1000_clean_rx_irq_ps::Massive overload detected. BVI FIFO size is more than 50331648 bytes..\n", MASSIVE_OVERLOAD_THRESHOLD. send_rbcp_message:workqueue::DIVERSION IS DISABLED, NME-APA is not inline now. Not receiving any packets. send_rbcp_message:workqueue::DIVERSION will be enabled in 300 seconds. Clearing queue... 2. While in the disabled state, you will also see the following message in the running configuration of the NME-APA: !APA traffic-management Disabled 3. When diversion of traffic is re-enabled and the NME-APA returns to 'Full Functionality' mode, you will see the following message: send_rbcp_message:workqueue::DIVERSION IS ENABLED, NME-APA is inline now. Processing resumed... If SCOS crashes and disables traffic diversion to the NME-APA, complete the following steps: Step 1 Look for the following messages on the NME-APA shellConn session: defaultFaultHandler was triggered because of signal 11 Send SCOS_RBCP_DIVERSION_DISABLED message... defaultFaultHandler was triggered because of signal 11 Cannot open RBCP_RD_PIPE... ERROR: Failed to open % defaultFaultHandler was triggered because of signal 11 Cannot open RBCP_RD_PIPE...: This is an unrecoverable error. A segmentation fault in the SCOS has caused the SCOS process to exit. Step 2 Store a copy of the logs. Step 3 Reload or reset the module Traffic Classification Troubleshooting No RDRs after pushing the traffic Step 1 Make sure that traffic is diverted to NME-APA and not discarded by NME-APA. See The NME-APA does not see any diverted traffic. Step 2 Make sure that the packets are not filtered (bypassed or dropped) by NME-APA. Use the NME-APA CLI debug slot 0 ppc 0 func RC_Cls to keep track of the counters. If the numbers for drop or bypass increase after the traffic is sent, packets have been filtered. The filtered traffic from APADC is used to configure the flow filter. For example, DNS packets are bypassed by default, so no RDRs will be generated by DNS traffic. You can also use the CLI command, debug slot 0 ppc 0 show-counters. Step 3 Ensure that the line card is not shut down. Step 4 Make sure that the SML application is installed. Use the NME-APA CLI show version to see whether the information about SML application is displayed. Step 5 If the SML application is not installed, install the PQI file from the APADC. Step 6 Make sure that traffic is delivered to SML Application. Check the value of Traversed PDUs from the output of the NME-APA ShellConn command, debug slot 0 ppc 0 func sr. If the value is 0, the SML application is not receiving traffic and no RDRs will be generated. In this case, it is possible that the traffic is not classified as a flow in SCOS. Turn on the debugging flag for the HwSim module. 1. Before you push traffic, enter the ShellConn command, LC_CS_on 2. After the traffic is pushed, enter the ShellConn command, LC_CS_off Note This command should be used in a controlled environment; it may interfere with live traffic. The log is in the file /root/tffs0/cs.txt. Send this file to DEs for analysis. Step 7 Save the RDRs. The RDRs can either be saved to a file (default) or sent to an external entity such as the CM or rdrHandler. If the destination IP address of the RDR-Formatter is configured, the RDRs are sent to the external entity. Otherwise, they are saved to a file. See RDR File Troubleshooting for details. RDR File Troubleshooting •No RDRs are received by the external entity •No RDR files are generated •The RDR files are not retrieved No RDRs are received by the external entity Step 1 Make sure that the configuration of the RDR-Formatter is correct (for instance, the IP address and the port number). Use the NME-APA CLI command show RDR-Formatter to see whether the connection between the NME-APA and the external entity is up. If the connection is down, the RDRs are queued. Check the connection between NME-APA and the external entity. (Note that this is for debugging purposes only. The RDRs should be written to files in the normal case.) The same command will also show the number of RDRs being queued. If no RDRs are queued, refer to Traffic Classification Troubleshooting to see why no RDRs are generated. No RDR files are generated Step 1 Confirm that RDRs are being generated. Use the NME-APA CLI command show RDR-Formatter to see whether the RDRs are queued. If no RDRs are queued, refer to Traffic Classification Troubleshooting to see why no RDRs are generated. Step 2 Use the ShellConn command, rdrShowSts, to see how much longer it has to wait before the RDRs are written to a file. By default, these RDRs are written to a file every 30 minutes. Step 3 After this time has elapsed, use the ShellConn command rdrShowAllFiles to see whether the file has been created. The ShellConn command, rdrHelp, provides a list of commands for debugging RDRs. The RDR files are not retrieved Step 1 Make sure that the ftp server is enabled. Use the NME-APA CLI show ip ftp-server to see whether the server is enabled. SNMP Troubleshooting Cannot get MIB values Step 1 Make sure that SNMP Agent is enabled. Use the NME-APA CLI show snmp to see whether SNMP Agent is enabled. If not, use the CLI snmp enable under configuration mode to enable it. Step 2 Check that the console displays SNMP Server started. If not, enter the ShellConn command debugScripts 1 from the console, and then disable and enable SNMP Agent again. Step 3 The ShellConn commands SNMP_AGENTX_Help and SNMP_AGENT_Help provide a list of commands for SNMP debugging. APADC Application Troubleshooting •About Applying Configurations •Connect Screen Troubleshooting •Device Configuration Screens Troubleshooting •Fault Management Screen Troubleshooting •Statistics Screen Troubleshooting •Installation Screen Troubleshooting •Traffic Management Screen Troubleshooting •Reporting Screen Troubleshooting •User Management Screen Troubleshooting •Admin Screen Troubleshooting About Applying Configurations It is important to understand that there are separate configurations for each of the following features: •Device Management (DM) •Traffic Management (TM) •User Management (UM) For example, when viewing a screen under DM, the Apply button on the screen applies only the DM configuration to the APA process. If you also want to edit the TM and/or UM configurations, you must click Apply on the TM and UM screens to push those configurations to APA. When you log in to APA, only the DM configuration is automatically retrieved from the APA process. Click Retrieve on TM and UM screens to retrieve the TM and UM configurations from APA. If the configuration has somehow been modified by means other than APADC, (this can happen, for example, if a change is entered via the CLI, although we do not recommend using the CLI for configuration changes), then also click Retrieve to retrieve the configurations. In cases where different aspects of a configuration are modified on multiple pages, each page contains a Save button. It is important to click Save unless you are going to immediately Apply the modified configuration after editing. If you do not click Save and you navigate to a different page, any configuration changes made on the current page will be lost! So, if you intend to modify the configuration on several pages and then Apply all the changes at once, you must click Save on each page before moving to a new page. Connect Screen Troubleshooting •Cannot connect to "Available" device •Cannot add a device •Device stays "Offline" Cannot connect to "Available" device Step 1 Go to Admin >Admin User Management. Step 2 Verify that the logged-in APADC user appears in the "User" column. Step 3 Verify that the NME-APA trying to connect is a "configured device" for the APADC user. Step 4 Click on the device name in the "configured device" column. Step 5 Verify that the device user name is correctly configured and is the same as the username configured on the NME-APA (by Telnet). Step 6 If the problem persists, create a new user account on the device and use this account to connect to the device by configuring it under Admin >Admin User Management. Step 7 Click Help, to review integrated help for more details. Step 8 If the problem persists, report to Cisco Systems, Inc. Cannot add a device Step 1 Verify that there are no more than ten devices already added to the APADC application. Step 2 If more than ten devices are already added to the APADC application, delete an existing device to add a new device. Step 3 Verify the correct network address for the device to be added. Step 4 Click Help, to review integrated help for more details. Step 5 If the problem persists, report to Cisco Systems, Inc. Device stays "Offline" Step 1 Ping the device from the web server running APADC. Step 2 Restart the Tomcat Windows service from Windows Administrative Tools > Services. Step 3 Restart the application server. Step 4 Click Help, to review integrated help for more details. Step 5 If the problem persists, report to Cisco Systems, Inc. Device Configuration Screens Troubleshooting •Cannot access Device Configuration Screen •Device Configuration is erased •Apply fails •Retrieve fails Cannot access Device Configuration Screen Step 1 Verify that at least one device is in the "Connected" state. Step 2 Verify that the user role allows access to this screen. Refer to the Cisco Network Module Enhanced Application Performance Assurance Troubleshooting Guide for details. Device Configuration is erased Step 1 Click Save before toggling to a different tab after making configuration changes. Step 2 Click Apply to commit configuration data to NME-APA. Step 3 Any configuration on a tab is lost if it is not saved before user moves to a different tab. Step 4 Click Help, to review integrated help for more details. Step 5 If the problem persists, report to Cisco Systems, Inc. Apply fails Step 1 Verify that the "Connected" device is still available and alive. Step 2 Click Help, to review integrated help for more details. Step 3 If the problem persists, report to Cisco Systems, Inc. Retrieve fails Step 1 Verify that the "Connected" device is still available and alive. Step 2 Click Help, to review integrated help for more details. Step 3 If the problem persists, report to Cisco Systems, Inc. Fault Management Screen Troubleshooting Notification is not received Step 1 Verify that the Notification is not suppressed: a. Go to the Fault Configuration screen. b. Look for the Notification ID for the missing notification. c. Verify that the Notification ID is present in the table. d. Check to see that the "Suppressed" column has the value "No", rather than "Yes". Step 2 Click Refresh. Step 3 Verify that NME-APA is sending a Notification. Step 4 Click Help, to review integrated help for more details. Step 5 If the problem persists, report to Cisco Systems, Inc. Statistics Screen Troubleshooting All Statistics are accumulated via SNMP queries. See SNMP Troubleshooting Installation Screen Troubleshooting •Install process fails with the error "Cannot connect to device" •Install process fails with the error "Image file type and file mismatch" Install process fails with the error "Cannot connect to device" Step 1 Ensure the Web user has credentials on the device as a level-15 user. (If the device user corresponding to the Web user has a level 0/5/10 credentials, the install process will fail.) Install process fails with the error "Image file type and file mismatch" The image file type selected for install (pqi or pqb) must have a filename extension of .pqi or .pqb. The files are not transferred from the user PC to the APADC server if a filename mismatch occurs. (Further semantic checks are performed in the next stage of the process to verify that the image files are of the expected type.) Step 1 Verify that the installation file has the correct filename extension. Traffic Management Screen Troubleshooting Cannot apply TM configuration to device Step 1 Check the server log for a connection-related exception. 2007-07-25 11:09:11,929 INFO [http-8080-Processor20] com.pcube.apps.engage.Engage - Sending configuration to SCE ... connection to SCE failed: server refused connection: unable to open connections - max number of sessions exceeded com.pcube.apps.engage.ConnectionFailedException: server refused connection: unable to open connections - max number of sessions exceeded at com.pcube.apps.engage.Connection.login(Connection.java:414) at com.pcube.apps.engage.Engage.login(Engage.java:138) at com.cisco.scabb.servconf.mgmt.SCABB.login(SCABB.java:81) at com.cisco.scate.web.mgt.tm.ServiceConfigManager.applyServiceConfig(ServiceConfigManager.ja va:115) at com.cisco.scate.web.ui.tm.service.ClassTreeAction.showClassTree(ClassTreeAction.java:159) If there is such a connection-related exception and reapplying the TM configuration to the device still fails, the connection between the APADC server and the device is maxed out. Step 2 Restart the APADC server (Tomcat). Reporting Screen Troubleshooting •RDR retrieval ("RetrieveNow") fails •Cannot "Activate" scheduled task •Cannot "Add" a scheduled task •RDR Retrieval continues to be in "In Progress" state •No "Last Retrieval" information is displayed •Cannot create report instance •"Run report JPEG" is disabled for certain reports •Report results are empty •Report fails to generate output (PROCESS_ERROR) RDR retrieval ("RetrieveNow") fails Step 1 Verify that the "Connected" device is still available and alive. Step 2 Verify that it is possible to FTP to the NME-APA from the APADC server using the device username and password associated with the APADC user from the Admin screen. If this step fails, refer to Device Configuration Screens Troubleshooting and verify users configured on the device. Retrieval will fail if an FTP attempt is made with a username and password that do not exist on the NME-APA device. Step 3 Verify that enough hard-disk space is available on the APADC server. Cannot "Activate" scheduled task Step 1 Verify that the task has not expired. if the start time is in the past, make sure that the difference between current time and the task start time is greater than the task recurrences times task interval. Step 2 Verify that the NME-AP device is still available and alive. Step 3 Verify that it is possible to FTP to the NME-APA from the APADC server using the username "admin" and enable15 password. If this step fails, refer to Device Configuration Screens Troubleshooting, and reconfigure the enable15 password on the device. Step 4 Verify that enough hard-disk space is available on the APADC server. Cannot "Add" a scheduled task Step 1 Verify that at least one NME-AP device is added to APADC. Step 2 Only one scheduled task can be added at a time. If there is a task, delete the task and try again. RDR Retrieval continues to be in "In Progress" state Step 1 Verify that the NME-AP device is still available and alive. Step 2 Verify that it is possible to FTP to the NME-APA from the APADC server using the username "admin" and enable15 password. Step 3 Verify that enough hard-disk space available on the APADC server If this step fails, refer to Device Configuration Screens Troubleshooting, and reconfigure the enable15 password on the device. Step 4 Check the server logs for OutOfMemory Errors. Restart the Tomcat server in the case of such errors. No "Last Retrieval" information is displayed Step 1 Verify that the <APADC_ROOT_DIR>/scate-config/rdrloader.xml file exists and is not empty. Step 2 Try "RetrieveNow" to see if the information gets updated. If the information is not updated, there could be problems retrieving data from the NME-APA. See RDR retrieval ("RetrieveNow") fails. Step 3 If there is a "Scheduled Task", make sure that it is still active. Cannot create report instance Step 1 Traffic management configuration must be applied by APADC. Verify that the NME-APA TM was configured using the APADC. Step 2 Verify that the "Last Retrieval" information (Device IP and time) on the Data Retrieval page is not "Unknown". Step 3 Verify that there is RDR data in the DB. If the DB is empty, go to the RDR Retrieval page and perform "RetrieveNow". "Run report JPEG" is disabled for certain reports This behavior is by design. A few report types do not support the JPEG format and so "Run Report JPEG" is disabled. CSV reports can be generated for these report types. Report results are empty Step 1 Traffic management configuration must be applied by APADC. Verify that NME-APA TM was configured using the APADC. Verify that the scateinfra.INI_VALUES table is not empty in the MySQL database. Step 2 Verify that the "Last Retrieval" was done within the last 24 hours. Step 3 Verify that there is RDR data in the DB. Step 4 Verify that the APADC server time zone is set correctly. Report fails to generate output (PROCESS_ERROR) Step 1 Verify that the report parameters do not contain invalid characters. Step 2 Open a command prompt, cd to the SCA Reporter folder and run the report execution command. The report execution command line can be found in the APADC log. Step 3 If the previous step successfully generates a report result, remove all optional parameters and try again. User Management Screen Troubleshooting Error stating FTP not available when retrieving or applying Step 1 Try again. A new FTP connection to the device will be created to continue the operation. Admin Screen Troubleshooting The Admin Screen is used to: •Add/Edit/Delete new APADC users •Add/Edit/Delete new devices to APADC users •Program device username, password, and access-level to APADC users Cannot Add a new APADC user Step 1 Verify that the maximum number of APADC users (100) has not been reached.
	© 1992-2009 Cisco Systems, Inc. All rights reserved. Terms & Conditions \| Privacy Statement \| Cookie Policy \| Trademarks of Cisco Systems, Inc.