Using NAMs to Monitor VoIP Quality
Voice quality analysis has been significantly enhanced in Cisco NAM. The software is now capable of accurately measuring voice quality by using the industry-standard MOS algorithm. Call quality measurements are computed every 1 minute and made available through the GUI. Note that the voice-related screens on the NAM GUI are significantly different from previous releases. Changes have been made to provide useful information quickly and automatically, while allowing easy navigation to details.
Deployment: NAM deployments for voice quality analysis require that NAM be able to monitor VoIP packets from the calling phone to the called phone. The branch edge location in the network provides visibility into all calls entering and leaving the branch; similarly a campus edge location monitors calls crossing the campus boundary. Often, the distribution layer is a good location to deploy NAMs for this purpose, especially if specific phones or particular portions of the network are to be monitored. For example, a new Multi protocol Label Switching (MPLS) link is being piloted and three buildings that are part of Company X's headquarters are part of the pilot. In order to monitor voice quality for those three buildings, a NAM could be deployed at the distribution Catalyst 6500 that serves those users.
Note The data center is typically not an appropriate location for RTP stream analysis because calls will seldom go through the data center. However, the data center is a good location to monitor signaling messages between phones and Cisco Unified Communications Manager. NAM decodes signaling messages to track call history, caller names, phone numbers, and other relevant call details.
Use the following steps to monitor the network to make sure that call quality is good. If quality issues appear, isolate and troubleshoot the problem rapidly.
Step 1 View RTP Streams using the menu selection Analyze > Media. This chart indicates current voice quality of all RTP streams being monitored. MOS values range from 1 to 5, where 1 is poor and 5 is excellent (see the legend for a breakdown into categories-Poor, Fair, Good and Excellent). The figure below displays the Top N RTP Source and Destination endpoints. Notice that there are calls that are in the poor range.
Step 2 To isolate calls that had a poor MOS, scroll down to Top N RTP Streams and click on the chart to drill down into the RTP Stream Details. As shown in Figure 6-1, notice that the MOS value for the calls listed on top is 2.88, which is low. Further, looking at the other metrics provided in the same row (for example, row one), notice that jitter is 3.49 and the packet loss rate is 11 percent, resulting in the low MOS value. This information tells you that jitter is the root cause of the poor calls; instead, it is packet loss somewhere in the network.
Figure 6-1 Top N RTP Streams by MOS
Step 3 With the endpoints’ IP addresses, you can look at the network topology to identify where in the network the 126.96.36.199 subnet is located. For the purposes of this use case, this subnet is in Building 3 of the main campus. You know that the Building 3 distribution switch has a NAM located in it.
Navigate to that NAM and go to the menu selection Analyze > Managed Device > Interface. This page lists all interfaces and errors or discards on each interface. Look up the link that leaves Building 3 and connects to the core. That interface is likely the source of the packet loss. Check the interface for faults and fix as needed.
See Analyzing Traffic, RTP Streams and Setting Voice Signaling Thresholds.
Autodiscovery Capabilities of NAM
If you are an existing NAM 4.x or NAM 6 user, you will not need to configure the SPAN sessions, and they will be automatically created on the NAM (not on the device). If you are a new NAM 5.x user, you will need to configure SPAN or NetFlow.
SPAN or NetFlow must be already configured on the device to forward traffic to NAM for auto creating the data source. See Setting Up Prime NAM Data Sources.
Creating Custom Applications
NAM identifies applications/protocols based on the TCP/UDP port number, so if there are applications using custom ports, the NAM can be configured to identify those applications by name instead of the port.
See Creating Deeper Visibility Into Application Traffic.
Integrating NAM with Prime Infrastructure
Cisco Prime supports integrated lifecycle management of networks, services, and endpoints for Cisco borderless network, data center, and collaboration architectures with end-to-end assurance. You can use Cisco Prime Infrastructure to centrally manage the Cisco Prime NAM platforms such as the NAM appliance to track inventory, view configurations, and perform image and fault management. Prime Infrastructure also rolls up the performance intelligence from NAMs deployed across the network into a consolidated dashboard.
The following overview describes the steps to complete in Prime Infrastructure to set up NAM to view multiple NAMs on your dashboard. For details steps, see the Prime Infrastructure User Guide on Cisco.com.
Step 1 Ensure you configure NTP and DNS for all the NAMs in your network. You can now configure those without going to the CLI or logging in to the individual NAM web GUI. Use the Cisco Prime Infrastructure Device Work Center to perform this task. For detailed steps, see your Prime Infrastructure product documentation.
Step 2 Add the NAM HTTPS credentials from the Prime Infrastructure’s Device Work Center Edit Device window so that Prime Infrastructure can retrieve data from them. You must add them only after the discovery process is complete or the modules have been added to the Prime Infrastructure inventory.
If you have licensed Assurance features, most Assurance features depend on NAM data to work so this is a required step.
You can repeat this task for all NAMs from which you want Prime Infrastructure to collect data.
Step 3 To ensure that you can collect data from your NAMs using Prime Assurance, you must enable NAM data collection and configure your NetFlow-enabled switches, routers, and other devices (ISR/ASR) to export this data to Prime Infrastructure. You can do this for each discovered or added NAM, or for all NAMs at the same time.
Step 4 To manage and troubleshoot a network problem such as a suspected network attack, you can use multiple NAMs to create packet captures, save them as files, and then decode them to inspect the suspicious traffic.
For other troubleshooting tips on how to use NAM with Prime Infrastructure, see the Prime Infrastructure User Guide. For application developers who want to use the NAM REST API to connect with Prime NAM, ask your Cisco representative about using the Cisco Prime Network Analysis Module REST API.
Integrating NAM with Third Party Reporting Tools
Prime NAM integrates with the CA NetQoS SuperAgent for the purpose of aggregating Application Response Times. Prime NAM also integrates with CompuWare Vantage and InfoVista 5View for Host, Conversation, RTP, and Response Time.
Ask your Cisco representative about the Prime NAM API Programmer’s Guide to find out more about the NAM Northbound Interface, also referred to as the REST API (Application Programming Interface). The API enables you to provision Prime NAM and extract performance data.
You can write your own scripts based on the Prime NAM Northbound API, but you must perform some setup in the GUI.
For details on what data can be collected, see Using Response Time Summary.
Point to the Design Guide
Monitoring Cisco WAAS and Measuring Its Impact
Cisco Wide Area Application Services (WAAS) is a comprehensive WAN optimization solution that accelerates applications over the WAN, delivers video to the branch office, and provides local hosting of branch-office IT services. Cisco WAAS allows IT departments to centralize applications and storage in the data center while maintaining LAN-like application performance and provides locally hosted IT services while reducing the branch-office device footprint.
One of the challenges facing IT personnel who deploy WAAS is to measure and report on the benefits provided by their WAN optimization deployment. Accurate measurement provides many benefits: IT can show return on investment; IT can assess whether the improvement gained meets originally advertised expectations from the solution; and finally, IT can use WAAS ongoing for monitoring, troubleshooting, and planning information for expanding the deployment.
The NAM can monitor WAAS-optimized flows by using WAE devices as the data source. Using this capability, the NAM is able to provide visibility into optimization-related metrics for the three distinct segments that are created by WAAS: the branch, the WAN, and the data center segments.
Placing a Cisco NAM appliance at the edge of the data center is recommended for WAAS deployments. From this location in the network, the NAM can measure local metrics using SPAN technology, and for information on the remote branch segment, it relies on flow agent exports from the remote WAE device. If SM-SREs are available, deploying one at the remote branch site is very useful. This SM-SRE can provide user experience at the site before WAAS is enabled and then contrast it to user experience after WAAS is enabled. See Figure 6-2.
Figure 6-2 Cisco NAM's Ability to Analyze from Multiple Data Sources
To deploy this solution:
Step 1 Using a NAM 2x20 deployed at the data center, measure application response time before WAAS is enabled using Analyze > WAN Optimization > Top Talker Detail. The Top Talker display includes such data as utilization, concurrent connections, and average transaction time for top applications, network links, clients, and servers that are possible candidates for optimization.
Step 2 Create a WAAS Client Side and WAAS Server Side for the WAAS flows from the DC and Branch WAEs.
Step 3 The NAM provides an interactive dashboard to view the analyzed data. Figure 6-3 displays Client Transaction Time, Traffic Volume and Compression Ratio, Number of Concurrent Connections (Optimized vs. Passthru), and Multi-Segment Network Time (Client LAN - WAN - Server LAN). As you can see in the first graph, all non-optimized traffic is displayed as Passthru.
Figure 6-3 Application Performance Analysis -- Optimized
The screen shot above illustrates the significant improvement experienced by users in the branch when WAAS is turned on. Such reports are very useful to justify an investment in WAN optimization technologies and to show returns on those investments in terms of increase in employee productivity and improved user experience from remote sites.
Step 4 From the perspective of the NAM located in the data center, there are two sources of information for response time measurements. SPAN provides measurement at the data center and exports from the branch; WAAS flow or PA via Prime Infrastructure provides measurements from the branch. Using these two sources of information, the NAM at the data center can continuously monitor current response times for each branch and help IT personnel keep user experience within known bounds. When abnormal response times are detected, the NAM can be configured to send alerts to appropriate personnel with information relevant to troubleshooting the problem.
Note The NAM 2x20 in the above scenario can be substituted with the NAM Virtual Blade on the WAVE-574 and WAE-674 to obtain the same type of reports.
Using NAM to Monitor QoS/DiffServ (DSCP)
Differentiated Services (DiffServ) provides insight into how traffic is being classified by QoS and detects incorrectly marked or unauthorized traffic. The NAM identifies the application/protocol based on the type of service (ToS) bits setting. The administrator can configure DSCP Groups or use the ones provided (as shown in Figure 6-5). The voice template can be used to monitor whether voice traffic is marked properly. Figure 6-7 displays the DiffServ application statistics for all DSCP value. Looking at this, you will notice that RTP and Session Initiation Protocol (SIP) are listed, which indicates that they are not being correctly marked throughout its path.
In the following scenario, IT has deployed QoS to prioritize VoIP traffic to improve voice quality across the network. The NAMs are deployed in the data center and branches and utilized to monitor the DSCP to validate QoS policies.
Step 1 Choose Setup > Media > DSCP Groups to display the default groups.
Figure 6-5 Default DSCP Groups
Step 2 Choose Administration > System > Preferences to turn the IP TOS Flow Key on. Use caution since this option affects ART and other flow-based traffic. See Table C-59 for details.
Step 3 Choose Analyze > Traffic > DSCP to find any misclassified traffic. In Figure 6-6, the RTP protocol is displayed for ToS0 classification.
Figure 6-6 DSCP Group - ToS0
Step 4 Click on the All DSCP button to view all DSCP and applications.
Step 5 In Figure 6-7, RTP and SIP are highlighted. The protocols are listed for DSCP 0, which is incorrect since the standard classification for voice traffic is DSCP 46 and 24. This means that some of the voice traffic is misclassified on the network. You can also view the branch NAMs to investigate whether voice traffic is being misclassified.
Figure 6-7 All DSCP Table
Step 6 Left-click on the RTP graph and select Application Traffic by Host to display the clients using those protocols. This helps to troubleshoot why RTP or SIP traffic from these clients is not marked correctly. As shown in Figure 6-8, the NAM displays the IP addresses of the phones using those protocols. This helps you review the QoS policy implemented on the routers and switches between the clients.
Figure 6-8 RTP Host Table
Using NAM for Historical Trends via Interactive Report
Historical trending is an important component of network performance management. While real-time analysis provides information about events, historical trending provides visibility into event sequences. Such sequences offer valuable information about various aspects of the network such as changes in network traffic behavior, anomalies and unusual activities, and network usage in peak times versus low times. It is also helpful in planning future network upgrades, application roll outs, and hardware buildouts. Here are some things to take note of regarding NAM’s historical trending capabilities:
- Use the Interactive Report > Filter button (located on the left side of the NAM window) to look at short term and long term trends by changing the Time Range. The interactive reports can be exported or the filter setting saved for quick view in the future. The exported data can be sent via e-mail in CSV or PDF format.
- Figure 6-9 displays host traffic for the last day, and using the middle graph you can zoom down to the time range of 10:00 - 16:00 to view what other application this host is using.
Figure 6-9 Host Traffic for Last 1 Day
In the following deployment scenario, you will predict the capacity needed for a new branch build out due in six months by studying the usage of an existing branch office of a similar size. To deploy a NAM located in the branch router (ISR) of the existing branch:
Step 1 Start capturing traffic rates between the branch and the data center. View the traffic for the last month from Interactive Report > Filter > Time Range > Custom (enter a date covering a month).
Step 2 Open a conversation report from today and find a stream that has a mildly increasing trend but is unable to confirm the rate at which it is increasing (see Figure 6-10).
Figure 6-10 A Stream with a Mildly Increasing Trend
Step 3 Change the Time Range dynamically in the Interactive Report to study the trend with a granularity of one month. You may find that the pattern does show periodic increases, but it always hits a ceiling between 4.5 Kbps and 5.x Kbps (see Figure 6-11). You are then able to conclude that the ISP link needed at the new site would be similar, and so a standard T1 line would be more than sufficient for the needs of the new remote office.
Figure 6-11 The Trend Shown with a Granularity of 1 Month
Studying historical trends is a valuable exercise in planning and creating baselines in a network. Monitor and trend on business critical applications and servers. These trends should provide handy information in a variety of day-to-day decisions.
Using NAM to Evaluate Application-Level Performance Monitoring for TCP-Interactive Applications
Application Performance Response Time Analysis provides up to 45 metrics. You can configure thresholds based on many of these metrics, and receive an alert when the thresholds are passed. Thresholds should be set for critical applications or servers using Average Server Response Time, or Average Transaction Time, or Average Network Time and Average Server Network Time. These thresholds will help identify where the problem lies in the application performance, and show whether the problem is a server or network issue. Depending on the alarm, you can access the NAM to see the applications and clients accessing the server, or to check the devices in the traffic path monitoring device and interface utilization.
See Application Response Time.
See Defining Thresholds.
Using NAM to Evaluate Application-Level Performance Monitoring for UDP Real-Time Applications
The NAM monitors and analyzes RTP streams and voice calls statistics by intercepting the data collected by endpoints. So, when a phone call ends, the endpoints calculate the information and send it to the Unified Communications Manager (aka the Call Manager), the NAM collects the data (as log as it is along that path).
NAM uses the voice call statistics from the endpoint with the RTP stream to correlate the phone number with the IP address of the endpoint. Alerts are sent based on analysis of the RTP streams for MOS, Jitter, and Packet Loss.
To use NAM to monitor the application-level performance for UDP real-time applications:
Step 1 Set up thresholds to focus on which types of performance metrics you want to monitor at Setup > Alarms > Thresholds.
Step 2 View voice signaling/RTP traffic at Analyze > Media > RTP Streams or Analyze > Media > Voice Call Statistics.
See Analyzing Traffic, RTP Streams.
See Table C-29 , Voice Monitor Setup Window.
Monitoring the Nexus 1000V Switch Environment
As networks and applications move into the virtualization environment, the challenge for you is to find tools to gain insight into that environment. The NAM VSB provides that function by integrating with the Cisco Nexus 1010 virtualization appliance. Using the NAM VSB, you can gain operational visibility into the virtual switching layer and is able to see virtual machine (VM) to VM statistics. See Figure 6-12.
The Nexus 1000V switch can also be monitored by other NAM platforms running the Prime NAM software.
You are deploying applications in the ritualized environment and the Nexus 1000V switch is providing the network connectivity. The NAM VSB installed on the Nexus 1010 Virtual Services Appliance is used to monitor the environment.
Note If Nexus 1000V switches and NAMs are already deployed in the network, ERSPAN or NetFlow data source can be directed by any one of those NAMs. You should directly connect the 1000V switch and NAM to the same physical switch.
Figure 6-12 Cisco Nexus 1000V NAM Virtual Service Blade Deployment
To monitor the Nexus 1000V environment:
Step 1 Install and configure either the NAM VSB on the Nexus 1110 Virtual Services Appliance. See the Installation and Configuration Guides for the NAM on Cisco.com.
Step 2 For the NAM VSB:
1. Verify that ERSPAN or NetFlow are configured on the Cisco 1000V Switch Virtual Supervisor Module (VSM) that is providing data to NAM.
2. Configure the ERSPAN or NetFlow data source, depending on your NAM:
3. Enable all applicable monitoring parameters in NAM for ERSPAN and NetFlow. Use the Traffic Summary window to display Top N information such as applications, hosts, protocol, and server response time. You can view and display details for each of the categories listed.
4. Using the Interactive Report, configure reports for trending on the application response time, hosts, and conversation traffic patterns.
The physical and virtual interfaces table provides VM-to-VM traffic utilization. Because one virtual interface connects to one VM, the data shows which VMS are utilizing the switch resources. You can then view the hosts and conversations tables to identify the culprit utilizing the resources.
Note NAM VSB provides the same complement of features except that it supports only ERSPAN and NetFlow data sources and performs no voice monitoring and packet capture.