Troubleshooting Network and NAM Issues
This appendix addresses some common issues you might encounter while using Cisco Prime Network Analysis Module as well as how to use NAM to troubleshoot issues in your network.
This appendix contains the following sections:
•Resolving Typical NAM Issues
•Troubleshooting Login Issues
•Understanding Typical Error Messages
•Understanding NAM Behavior
•Troubleshooting WAAS Data Issues
•Using the CLI to Troubleshoot Issues
Resolving Typical NAM Issues
Q. What information should I collect and what else should I do when the NAM is not responding?
A. Determine the answers to the following questions and gather the following information:
•Does session from the switch/router CLI work?
•Does ping over EOBC (127 subnet) work?
•Does ping to the management IP address work?
•Collect output of show tech-support command from both the NAM and the switch or router.
•Collect core files.
•Check if NAM hardware is seated correctly in chassis
Perform the following tasks to troubleshoot your issue:
•Reset into maintenance image or helper
•Clear the configuration
•Reinstall the application image (using the --reformat option)
Troubleshooting Login Issues
Log into the NAM by using the username and password that the NAM administrator provided you, and click the Login button. If you are having problems logging in:
•Make sure you are using a browser that is currently supported for use with NAM:
English Firefox 3.6+ or Microsoft Internet Explorer 8+ (Microsoft Internet Explorer 7 is not supported)
•Make sure you are using a platform that is currently supported for use with NAM:
Microsoft Windows XP or Microsoft Windows 7. The Macintosh platform is not supported on this release.
•Make sure you have downloaded the most recent version of Adobe Flash.
•Clear the browser cache and restart the browser (not necessary if installing NAM for the first time).
•Make sure cookies are enabled in your browser.
•If you see the following message: "Initializing database. Please wait until initialization process finishes," you must wait until the process finishes.
•Make sure you had accepted the license agreement (WAAS VB, Nexus 1010, and SRE users only) and that the license has not expired.
To view the full documentation set (including the User Guide and Release Notes) for the Cisco NAM software, go to the NAM software Technical Documentation area on Cisco.com:
Understanding Typical Error Messages
Q. I'm waiting for the graphical data to populate on a dashboard. What does this red error "Request Error -- Please Try Again" mean?
A. This means an internal error has occurred, or the login session may have timed out.
Q. I'm waiting for the graphical data to populate on a dashboard. What does this red error "Query resulted in no data" mean?
A. The NAM does not have any data for the specified time frame and specified filter. Go to the Interactive Report (the pane on the left side of the window) and click the Filter button to check the filter settings and data sources to make sure the NAM is getting data. You can also check the Overview page to ensure the traffic is reaching the NAM. If no traffic appears, check your data sources and SPAN session configuration.
Q. What does the message "Client or NAM time is incorrect" mean?
A. The browser or client time and the NAM time must be synchronized to avoid this error.
Understanding NAM Behavior
This section includes details on how NAM works including how to navigate and use the control elements in the user interface as well as Prime NAM uses the traffic sources available on each platform.
Use these sections to understand Prime NAM behavior:
•Understanding Common Navigation and Control Elements
•Understanding How the Prime NAM Traffic Sources Work
•Frequently Asked Questions about Prime NAM Behavior
Understanding Common Navigation and Control Elements
This section includes the following topics:
•Understanding the Menu Bar
•Displaying Detailed Views
•Accessing Context Menus
•Performing a Quick Capture
•Filtering Data Using the Interactive Report
•Switching Chart Formats Using the Chart View / Table View
•Accessing Other Tasks Using Mouse-Over for Details
•Changing the Time Interval Using Zoom/Pan Charts
•Using Sort Grid to Change Sort Order
•Displaying Bits or Bytes or Packets in Charts
•Context-Sensitive Online Help
•Synchronizing the Clock on Prime NAM
Understanding the Menu Bar
For a description of common tasks in Prime NAM, see Table A-1.
Table A-1 Summary of Menu Tasks
Brings you to the Traffic Summary Dashboard (Monitor > Overview > Traffic Summary).
View summary dashboards with network traffic, application performance, site performance, and alarms information at a glance.
See various views for traffic over a time period, WAN optimization, response time, managed device, and media functions.
Configure multiple sessions for capturing, filtering, and decoding packet data, manage the data in a file control system, and display the contents of the packets.
Perform setup options needed to access Prime NAM features.
Set dashboard preferences, perform user and system administration tasks, and generate diagnostic information for obtaining technical assistance.
Displaying Detailed Views
You can access additional details from the Dashboard and Monitor and Analyze mega-menus that provide packet and bits per second data as well as identify host, application, DCSP, and other categories. Look for menus titled Details or Detailed Views. If a small arrow appears to the right of the menu selection, click it to see the submenu and the functions available. For an example of the Applications submenu see Figure A-1.
Accessing Context Menus
On most charts that appear on the dashboards, you can left-click on a colored bar of data to get a context menu, with which you can get more detailed information about that item. See Figure A-1.
Figure A-1 Context Menu Showing Application Submenus
The example above is from the Traffic Summary Dashboard, Top N Applications chart. The description to the right of "Selected Application" in the menu shows what item you had clicked on (in this case, ftp-data).
The menu items above the separator line are specific to the selected element of the Top N chart. The items below the separator line are not specific to the selected element, but apply to the Top N chart.
Performing a Quick Capture
From the Context menu of many of the bar charts that show Applications or Hosts or VLANs. you can start a Capture. For example, when you click on an Application in a bar chart (as in Figure A-1) and choose Capture, the following is done automatically:
•A memory-based capture session is created
•A software filter is created using that application
•The capture session is started
•The decode window pops open and you can immediately see packets being captured.
Note Quick Capture does not use site definition/filter.
From both the selectors in the upper left of the dashboards and from the item the user clicks on in the barchart, the following are carried into the context for the capture session:
•Data Source (if it is a DATA PORT)
If you open up the associated Capture Session and its associated Software Filter, the above settings will be shown.
Determining How to Use Sites to View Data
A site is a collection of hosts, or network endpoints, partitioned into views that help you monitor traffic and troubleshoot problems (see Configuring Sites for more detailed information).
If you have set up sites, you will be able to select a particular site to view in the Interactive Report and view data relevant to that site only. In some cases, you can select both a Client Site and a Server Site to view data pertaining to interaction between hosts at different sites.
Filtering Data Using the Interactive Report
You can use the Interactive Report on most Monitor and Analyze windows to filter the parameters of the information displayed in the dashboards. Click the Filter button to change the parameters of the information displayed in the charts.
You can choose from various parameters, such as the time range for the data being displayed. An asterisk represents required fields.
The reporting time interval selection changes depending upon the dashboard you are viewing, and the NAM platform you are using. The Prime NAM supports up to five saved Interactive Reports. Saved reports display at the bottom of the interactive report panel.
The From and To fields are only enabled when the Time Range is set to Custom.
Switching Chart Formats Using the Chart View / Table View
Using the Chart view lets you see an overview of the data in an integrated manner, and can show you trending information. To get the exact value of any data in the graphical view, hover over a data point to see the tool tip. To toggle between the two views, use the Chart and Table icons at the bottom of the panel
Accessing Other Tasks Using Mouse-Over for Details
When in Chart view, you can mouseover the chart to get more detailed information about what occurred at a specific time.
Many of the line charts in Prime NAM are dual-axis, meaning there is one metric shown on the left axis of the chart and another metric shown on the right axis of the chart.
For example, in the DCSP Group Traffic chart, Megabits per second is shown on the left axis, and Packets per second is shown on the right axis.
Changing the Time Interval Using Zoom/Pan Charts
For many charts, you can drag the beginning or end to change the time interval, as shown below.
The time interval change on the zoom/pan chart will affect the data presented in the charts in the bottom of the window. The zoom/pan time interval also affects the drill-down navigations; if the zoom/pan interval is modified, the context menu drill-downs from that dashboard will use the zoom/pan time interval.
Note In a bar chart which you can zoom/pan, each block represents data collected during the previous interval (the time stamp displayed at the bottom of each block is the end of the time range). Therefore, you may have to drag the zoom/pan one block further than expected to get the desired data to populate in the charts in the bottom of the window.
Using Sort Grid to Change Sort Order
When looking at information in Grid view, you can sort the information by clicking the heading of any column. Click it again to sort in reverse order.
Displaying Bits or Bytes or Packets in Charts
To change the display on most Monitor and Analyze charts from bits to bytes, you can use the Bits and Bytes radio buttons to specify which information you would like the chart to display. To change this preference to display bytes use the Administration > System > Preferences.
On most Monitor and Analyze charts, you can use the Bits and Packets check boxes at the top to specify which information you would like the chart to display. To change this preference to display bytes use the Administration > System > Preferences.
The Statistics legend gives you the minimum, maximum, and average statistics of the data. This will display the initial data retrieved for the selector.
Context-Sensitive Online Help
The Help link on the top-right corner of the Prime NAM interface will bring you to the Help page for that particular window of the GUI.
If available, the Help link appears on the top-right corner of each page; some pages also have a blue "i", which provides help for that specific subject.
Understanding How the Prime NAM Traffic Sources Work
The next section describes how the Prime NAM uses the supported data sources:
•Understanding How the Prime NAM Uses SPAN
•Understanding How the Prime NAM Uses VACLs
•Understanding How the Prime NAM Uses NetFlow
•Understanding How the Prime NAM Uses WAAS
•Understanding How the Prime NAM Uses PA
Understanding How the Prime NAM Uses SPAN
A switched port analyzer (SPAN) session is an association of a destination port with a set of source ports, configured with parameters that specify the monitored network traffic. You can configure up to two SPAN sessions in a Catalyst 6500 chassis. Newer Cisco IOS images may support more than two SPAN sessions. Consult the Cisco IOS document for the number of SPAN sessions supported per switch or router.
The NAM-1 platform provides a single destination port for SPAN sessions. The NAM-2 and NAM-3 platform provides two possible destination ports for SPAN and VLAN access control list (VACL) sessions. Multiple SPAN sessions to the Prime NAM are supported, but they must be destined for different ports. The Prime NAM destination ports for use by the SPAN graphical user interface (GUI) are named DATA PORT 1 and DATA PORT 2 by default. In the CLI, SPAN ports are named as shown in Table A-2.
Table A-2 SPAN Port Names
NAM-2 or NAM-3
dataport 1 and dataport 2
For more information about SPAN and how to configure it on the various Cisco NAM platforms, see Cisco.com.
Note Due to potentially very high volume of ERSPAN traffic from the source, we recommend that you do not terminate the ERSPAN session on the Cisco NAM management port. Instead, you should terminate ERSPAN on the switch, and use the switch's SPAN feature to SPAN the traffic to Cisco NAM dataports.
Synchronizing the Clock on Prime NAM
All times in the Prime NAM are typically displayed in 24-hour clock format. For example, 3:00 p.m. is displayed as 15:00.
Understanding How the Prime NAM Uses VACLs
A VLAN access control list can forward traffic from either a WAN interface or VLANs to a dataport on the NAM. A VACL provides an alternative to using SPAN; a VACL can provide access control based on Layer 3 addresses for IP and IPX protocols. The unsupported protocols are access controlled through the MAC addresses. A MAC VACL cannot be used to access control IP or IPX addresses.
There are two types of VACLs: one that captures all bridged or routed VLAN packets and another that captures a selected subset of all bridged or routed VLAN packets. Catalyst operating system VACLs can only be used to capture VLAN packets because they are initially routed or bridged into the VLAN on the switch.
A VACL can provide access control for all packets that are bridged within a VLAN or that are routed into or out of a VLAN or, with Release 12.1(13)E or later releases, a WAN interface. Unlike regular Cisco IOS standard or extended ACLs that are configured on router interfaces only and are applied on routed packets only, the VACLs apply to all packets and can be applied to any VLAN or WAN interface. The VACLs are processed in the hardware.
A VACL uses Cisco IOS access control lists (ACLs). A VACL ignores any Cisco IOS ACL fields that are not supported in the hardware. Standard and extended Cisco IOS ACLs are used to classify packets. Classified packets can be subject to a number of features, such as access control (security), encryption, and policy-based routing. Standard and extended Cisco IOS ACLs are only configured on router interfaces and applied on routed packets.
After a VACL is configured on a VLAN, all packets (routed or bridged) entering the VLAN are checked against the VACL. Packets can either enter the VLAN through a switch port or through a router port after being routed. Unlike Cisco IOS ACLs, the VACLs are not defined by direction (input or output).
A VACL contains an ordered list of access control entries (ACEs). Each ACE contains a number of fields that are matched against the contents of a packet. Each field can have an associated bit mask to indicate which bits are relevant. Each ACE is associated with an action that describes what the system should do with the packet when a match occurs. The action is feature dependent. Catalyst 6500 series switches and Cisco 7600 series routers support three types of ACEs in the hardware: IP, IPX, and MAC-Layer traffic. The VACLs that are applied to WAN interfaces support only IP traffic.
When you configure a VACL and apply it to a VLAN, all packets entering the VLAN are checked against this VACL. If you apply a VACL to the VLAN and an ACL to a routed interface in the VLAN, a packet coming into the VLAN is first checked against the VACL and, if permitted, is then checked against the input ACL before it is handled by the routed interface. When the packet is routed to another VLAN, it is first checked against the output ACL applied to the routed interface and, if permitted, the VACL configured for the destination VLAN is applied. If a VACL is configured for a packet type and a packet of that type does not match the VACL, the default action is deny.
When configuring VACLs, note the following:
•VACLs and context-based access control (CBAC) cannot be configured on the same interface.
•TCP Intercepts and Reflexive ACLs take precedence over a VACL action on the same interface.
•Internet Group Management Protocol (IGMP) packets are not checked against VACLs.
Note You cannot set up VACL using the Prime NAM interface.
For details on how to configure a VACL with Cisco IOS software, see Cisco.com.
For details on how to configure a VACL on a WAN interface and on a LAN VLAN, see VACL.
Understanding How the Prime NAM Uses NetFlow
The Prime NAM uses NetFlow as a format for the ongoing streaming of aggregated data, based on the configured set of descriptors or queries of the data attributes in NAM. NetFlow Data Export (NetFlow) is a remote device that allows you to monitor port traffic on the NAM; the Prime NAM can collect NetFlow from local or remote switch or router for traffic analysis.
To use an NetFlow data source for the Prime NAM, you must configure the remote device to export the NetFlow packets. The default UDP port is 3000, but you can configure it from the Prime NAM CLI as follows:
firstname.lastname@example.org# netflow input port ?
<port> - input NetFlow port number
The distinguishing feature of the NetFlow v9 format, which is the basis for an IETF standard, is that it is template-based. Templates provide an extensible design to the record format, a feature that must allow future enhancements to NetFlow services without requiring concurrent changes to the basic flow-record format.
For more detailed information about Prime NAM and NetFlow, see NetFlow.
For specific information about creating and managing NetFlow queries, see the Cisco Network Analysis Module API Programmer's Guide (contact your Cisco account representative if you need to refer to this document).
Understanding How the Prime NAM Uses WAAS
Cisco Wide Area Application Services (WAAS) software optimizes the performance of TCP-based applications operating in a wide area network (WAN) environment and preserves and strengthens branch security. The WAAS solution consists of a set of devices called Wide Area Application Engines (WAEs) that work together to optimize WAN traffic over your network.
When client and server applications attempt to communicate with each other, the network devices intercepts and redirects this traffic to the WAEs to act on behalf of the client application and the destination server.
WAEs provide information about packet streams traversing through both LAN and WAN interfaces of WAAS WAEs. Traffic of interest can include specific servers and types of transaction being exported. Prime NAM processes the data exported from the WAAS and performs application response time and other metrics calculations and enters the data into reports you set up.
The WAEs examine the traffic and using built-in application policies to determine whether to optimize the traffic or allow it to pass through your network not optimized.
You can use the WAAS Central Manager GUI to centrally configure and monitor the WAEs and application policies in your network. You can also use the WAAS Central Manager GUI to create new application policies so that the WAAS system will optimize custom applications and less common applications. Prime NAM is accessible from within the Central Manager interface. The Cisco Prime NAM integration with WAAS Central Manager provides for easier viewing of Prime NAM reports that are directly associated with Application Response Time measurements through the WAN, in both WAAS optimized and non-optimized environments. See WAAS Central Manager.
For more information about WAAS data sources and managing WAAS devices, see Understanding WAAS.
Understanding How the Prime NAM Uses PA
The Performance Agent (PA) can monitor interface traffic and collect, analyze, aggregate, and export key performance analytics to a Cisco NAM for further processing. PA integration with Prime NAM enables you to have a lower cost way to gain visibility into Application Response Time at the branch. Prime NAM integration with PA also reduces complexity of needing to manage a separate Prime NAM product within the branch.
Using Cisco PA, you can gain visibility into application response time and traffic statistics at remote branches. It is supported on ISR 880, ISR 890, and ISR G2 platforms with Cisco IOS Software Release 15.1(4)T. Deployed with WAAS Express, this feature allows an end-to-end view into the WAN-optimized network, delivering a cost-effective and scalable solution.
PA has the ability to consolidate and filter information before it is exported, ensuring that only contextually-required data is exported and consumed versus all data. As an example, NetFlow Export supports a number of functions, including response time and traffic analysis. Instead of exporting multiple different flows, the PA has the intelligence to consolidate, filter, and export flow data that addresses the particular user's need. Besides consolidating and filtering information, PA's mediation capabilities also includes the ability to use key Cisco IOS-embedded functionality (for example, Embedded Event Manager, or Class-Based QoS) to enrich both PA functionality and router value.
For information about configuring PA data sources, see Managing ISR PA Devices.
Frequently Asked Questions about Prime NAM Behavior
Q. How does NAM calculate network latency?
A. To calculate network latency, NAM looks at each packet and associates it to a transaction. For example, NAM looks at SYN and SYN-ACK and timestamps these packets to perform these calculations.
Q. How can vNAM be restricted to one tenant's traffic when using SPAN or ERSPAN on a Nexus 1000V?
A. VirtualNAM can be deployed per tenant so they each NAM has their own portal. NAM processes VxLAN, LISP, FabricPath, and OTV for multiple tenants.
Q. Why is the browser behaving strangely? It is displaying data for no apparent reason or is not displaying expected data.
A. Clear the browser cache, close the browser, and open a new session and try again. Also, make sure you are using a supported browser (see the Cisco Prime Network Analysis Module Release Notes).
Q. Why is the NAM performance lower than expected?
A. Disk capture will reduce the NAM performance considerably. It is due to the disk input/output speed. You will see a warning in the top right corner of the window.
Q. Why won't the system change the storage option for my capture session from disk to memory and then back to disk?
A. If you set up a capture session to disk and later modify the same packet session to save into memory, Prime NAM is unable to change the storage selection back to disk because it is in the in use state. You cannot delete the capture session to release the disk for capture. The workaround is to reboot the NAM. This has been fixed in the latest patch (patch 5) on the Cisco software download web page.
Q. What MIBs do the Prime NAM support?
Table A-3 lists the MIB objects supported by Prime NAM.
Table A-3 Supported MIBs
MIB-II: All groups except Exterior Gateway Protocol (EGP) and transmission.
RMON-MIB: Alarm and Event groups only
RMON2: trapDestTable only
CDP-MIB: Cisco Discovery Protocol1
Troubleshooting WAAS Data Issues
Q. Why does Prime NAM display the status of WAAS devices as pending?
A. Prime NAM is unable to monitor WAAS traffic until you set up WAAS monitored servers. To change the pending status, you must set up WAAS monitored servers. See your product documentation for more details.
Q. Why is no WAAS data seen in the Monitor windows?
A. Perform the following steps:
•Use the NAM GUI to verify that the Monitored Servers list is configured with the correct server IP addresses.
•Use the NAM GUI to verify that WAAS data sources have data collection enabled for applicable segments.
•Use the WAAS CLI show statistics flow filters to verify that the servers have active traffic flows that are optimized and monitored.
•Use the WAAS CLI show statistics flow mon tcpstat to verify that WAAS Flow Agent exports flow data to the correct NAM IP address.
Q. The WAAS is not sending data to the NAM, and the reports are not showing any values.
A. The WAAS will not send data unless filtering is enabled on the NAM. Enable filtering at Setup > Data Sources > WAAS > Monitored Servers, and check the "Filter Response Time for all Data Sources by Monitored Servers" check box.
Using the CLI to Troubleshoot Issues
•Locating Packet Drops
•Handling an Unresponsive NAM
•Handling an Unresponsive NAM
Locating Packet Drops
Q. How can I find out using the CLI if packets are being dropped?
A. The following CLI command shows packet drops at different layers of the NAM system at 5 minute intervals and up to the last 24 hours:
root@NAM1x-18.cisco.com# show pkt-drop-counters Hour-0
Start time of the hour: 2010-11-05 13:00 PDT
Time hardware pkts dropped FM pkts dropped ART pkts dropped
Handling an Unresponsive NAM
Q. Why is my NAM Blade not responding?
A. Do the following:
•Check the NAM IP configuration (using the CLI command show ip)
•Check VLAN configuration of management port on Sup:
analysis module <slot> management-port access-vlan <#>
•Does the session from the switch/router work?
•Does a ping to NAM mgmt IP address work?
•What is the module status on Sup/router?
Using the CLI to Troubleshoot Performance Agent (PA)
Q. Why is the NAM not receiving data from PA?
A. First troubleshoot Flexible NetFlow (FNF). Check whether FNF sent data to the NAM with show flow exporter statistics.
If FNF did not send data, check Performance Agent with show mace metrics summary. Check whether flows are exported.
If flows are not exported by PA, debug PA to identify the error with debug mace cp.