-
Cisco Resource Policy Management System 2.0.1 Troubleshooting Guide
-
Preface
-
Chapter 1: Cisco RPMS 2.0.1 System and Solution Troubleshooting
-
Chapter 2: Troubleshooting Issues
-
Chapter 3: Cisco RPMS System Start and Stop Procedures
-
Chapter 4: Universal Gateway IOS Debugging
-
Chapter 5: Sample Call Traces
-
Chapter 6: Common Problems and Workarounds
-
Chapter 7: Using the Cisco RPMS Diagnostic Tools
-
Glossary
-
Feedback
Table Of Contents
Using the Cisco RPMS Diagnostic Tools
Overview: Using the Diagnostic Tools
Overview: Diagnostics Report Fields
Configuration Parameters of Diagnostics Subsystem
Customer-Related Diagnostics Reports
Calculating the Customer Limit Deficit
Gateway-Related Diagnostics Reports
System-Related Diagnostics Information
Overview: Diagnosing End User Issues
Using the Cisco RPMS Diagnostics Tool to Monitor Customer Activity
Using the Cisco RPMS Diagnostics Tool to Troubleshoot Low Call Durations
Overview: Identifying Policy Enforcement Issues
Using the Cisco RPMS Diagnostics Tool to Determine the Active Call Count
Overview: Identifying Universal Gateway Issues
Using the Cisco RPMS Diagnostics Tool to Determine Universal Gateway Issues
Overview: Tracking Provisioning Changes
Overview: Viewing the Cisco RPMS History
Using the Cisco RPMS Diagnostics Tool to View the System History
Overview: Analyzing Traffic Patterns with Cisco RPMS
Overview: Analyzing Customer-Specific Traffic
Call Duration Distribution Over Time
Using the Cisco RPMS Diagnostic Tools
Topics in this chapter include:
•
Overview: Using the Diagnostic Tools
•
•
•
•
•
•
•
–
•
–
Overview: Using the Diagnostic Tools
Cisco RPMS Release 2.0.1 introduces new diagnostic features to help monitor the system's performance. The diagnostic tools have two functions: to gather information so that you can monitor the Cisco RPMS' performance, and to help diagnose system issues caused by network elements that interact with the Cisco RPMS. Both functions help track the Cisco RPMS' performance to ensure it runs smoothly, and should problems occur, help you to diagnose and repair the issues.
The features have two components:
•
•
By using the diagnostic tools, you can accomplish the following:
•
•
•
•
•
For more information on using the diagnostic tools, refer to the Cisco Resource Policy Management System Release 2.0.1 Solutions Guide.
Overview: Diagnostics Report Fields
This section defines the fields for all the diagnostics reports generated by Cisco RPMS. The diagnostics reports are classified in to the following categories.
•
•
•
The diagnostics feature is enabled by default.
Configuration Parameters of Diagnostics Subsystem
The following example shows the configuration parameters for the Diagnostics Subsystem of Cisco RPMS 2.0.1. The configuration parameters are located in the <$RPMSBASE>/config/rpms.conf file.
The sampling intervals are in milliseconds.
Example 7-1 Configuration parameters for the Diagnostics Subsystem
BaseDir= Directory used to store diagnostics file.CustCountsPolicyUsageDiagSampleTime=120000CustRejectsDiagSampleTime=300000CustCallDurationDiagSampleTime=1200000UGCallDurationDiagSampleTime=1200000UGAcctDelayDiagSampleTime=300000UGRejectsDiagSampleTime=300000UGTerminationsDiagSampleTime=1200000UGRetriesDiagSampleTime=1200000UGCallDurationSysDiagSampleTime=1200000UGCallFailureSysDiagSampleTime=300000UGMissingAcctStopsDiagSampleTime=3600000UGRetriesSysDiagSampleTime=1200000SystemStatsDiagSampleTime=1800000SystemActivityDiagSampleTime=300000SystemLatencyDiagSampleTime=300000SystemMessageRateDiagSampleTime=300000VirtualCallDuration=20Customer-Related Diagnostics Reports
Cisco RPMS logs customer-related diagnostics information in three different report files:
The reports are generated for each customer configured in Cisco RPMS under a specific directory for that customer. The reports reside in the following directory structure:
$RPMSBASE/diagnostics/<Date>/customer/<CustomerName>/<ReportType>_<TimeStamp>.csv
•
•
The data in the files are logged at periodic intervals that are configurable (in the Diagnostics section of rpms.conf). All data is incremental (i.e.. change in value since last sample time).
The following sections describe the report fields.
Customer Counts
The report logs the active call count information of each customer during the sampling interval.
Calculating the Customer Limit Deficit
The active call count for an SLA is the total number of active calls that have been admitted by meeting SLA criteria. When the active call limit reaches an SLA limit, subsequent call attempts are rejected until one of the active calls disconnect, allowing a new call to enter the system and returning to the limit.
A "virtual" call count is identical to the active call count before its limits are reached. However, when a call request is rejected by an SLA active call limit, the active call count does not change; with a virtual call count, the limit is incremented by 1. Essentially, the virtual call count tracks the count as if there were no limits. This difference between the virtual and active call counts is called the "limit deficit."
Periods of limit restriction can last for hours, yet average dial hold times last about 20 minutes. Because of this, the virtual call count also uses a virtual call duration, so that a fictitious additional call count expires after a statistical amount of time. This prevents rejected call counts from inflating forecast virtual calls.
A simple queue-based approach is used to calculate the limit deficit. The queue tracks the call duration of the "virtual" calls. Figure 7-1 illustrates the queue-based algorithm.
Figure 7-1 The queue-based algorithm
Figure 7-1 shows the following:
•
•
•
•
•
The limit deficit at any time is the sum of all the elements in the queue, so that:
To better understand this algorithm, use the following example.
Example 7-2 The Customer Limit Deficit Algorithm
Virtual Call Duration = 20 minutesSampling Interval, Ts = 2 minutesStart Time of Sampling Timer, To = 04:00:00From the values in Example 7-2, n = 20 / 2 = 10. Since n is the queue size, the values of virtual call duration and Ts is chosen so that n is a non-fractional number greater than or equal to 1.
Figure 7-2 shows the queue contents at various points in time.
Figure 7-2 Queue contents After 6 Samples
Figure 7-3 Queue contents After 10 Samples
Figure 7-4 Queue contents After 11 Samples
Table 7-2 shows the limit deficit for various samples.
Until the 10th sample, the limit deficit is merely a summation of the queue elements. However, for the 11th sample, the queue has already reached its limit. This indicates that the duration of "virtual" calls at the head of the queue have reached the virtual call duration of 20 minutes.
This is why you must calculate the queue size by dividing the virtual call duration by Ts. So, the first element is popped before the new sample value of 6 (which is the number of rejections between 04:20:00 and 04:22:00) is pushed into the queue.
Customer Rejects
The report logs rejection details of each customer during the specified sampling interval.
The report has only Limit Rejections and VPDN Rejections.
Customer Call Durations
This report logs the duration of calls for each customer during the sampling interval.
Gateway-Related Diagnostics Reports
Cisco RPMS logs gateway-related diagnostics information in the following report files:
Cisco RPMS logs gateway-related diagnostics reports in five different reports. These reports are generated for each gateway configured in the Cisco RPMS. The reports are located in the following directory structure:
$RPMSBASE/diagnostics/<Date>/gateway/<UGIP>/<ReportType>_<TimeStamp>.csv
•
•
The data in the files are logged at periodic intervals that are configurable in the [Diagnostics] section of rpms.conf. All data is incremental (i.e., change in value since last sample time).
UGCallDuration
This report logs the duration of the calls for each UG during the sampling interval.
UGAccountingDelay
The report logs information regarding delayed accounting messages that Cisco RPMS received during the sampling interval.
UGRejects
The report logs the rejections per UG during the sampling interval.
UGTerminations
The report logs information regarding the UG Terminations during the sampling interval.
The following list is for all UG (1-18) and Cisco RPMS (19-23) generated terminate causes:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
UGRetries
The report contains information UGRetries during the sampling interval.
System-Related Diagnostics Information
Cisco RPMS logs system-related diagnostics information in the following report files:
Cisco RPMS generates system-related diagnostics reports to help you understand how the system is being utilized. The reports also provide diagnostics information at the highest level. This information helps troubleshoot system issues.
Cisco RPMS logs the system-related diagnostic information in seven reports, located in the following directory structure:
$RPMSBASE/diagnostics/<Date>/system/<ReportType>_<TimeStamp>.csv
•
•
The data in the files are logged at periodic intervals configurable in the [Diagnostics] section of rpms.conf. All data is incremental (i.e., changes in value since last sample time).
PolicyUsage
The report logs the usage of the SLAs of each customer during the sampling interval.
Note
2147483647 corresponds to "No Limit" configured for Session Count Limit.
4294967294 corresponds to "No Limit" configured for both Session Count Limit and Oversubscription limit.
Message Rate
The report shows the different types of messages Cisco RPMS has received during the sampling interval.
Note
Call Stop Accept and Call Stop Reject are the messages for future use.
System Activity
The reports logs the active call count of Cisco RPMS during the sampling interval.
Table 7-12 System Activity Fields
Timestamp
Time at which the sample was collected.
Call Count
Current active call count at that sampling interval.
SystemLatency
This reports logs the latency for each messages Cisco RPMS has received during the sampling interval.
Note
SystemStats
The report logs the system related Statistics during the sampling interval.
SystemUGRetries
The report logs the UG Retries information during the sampling interval.
CellDurations
Cisco RPMS logs call duration related information in this report. The report is generated whenever calls are closed by NAS in short time. Low call durations can be caused due to NAS issues, such as faulty hardware.
MissingAcctStop
Cisco RPMS generates this report whenever NASs fail to send accounting stop to Cisco RPMS or when the accounting stop message is lost. This leads to a disparity between the active call count reported by RPMS and the number of real calls. If this happens, Cisco RPMS continues to count the calls as active even if they are dropped on the NAS. These calls are cleared only from Cisco RPMS by replacement with another call on the same port or after Active Call Timeout.
CallFailureRates
The report is generated whenever NASs experience hardware and or software problems while sending calls to Cisco RPMS. This report shows all NASs that had calls terminated due to reasons deemed erroneous. Cisco RPMS considers the following call termination codes as erroneous:
Table 7-18 Erroneous Termination Codes
2
Lost Carrier
3
Lost Service
9
NAS Error
13
Port Preempted
14
Port Suspended
15
Service Unavailable
The following are the fields and definitions for CallFailureRates.
Threshold Report
The report is generated when Cisco RPMS used resources reach its Threshold limits. This report is generated in the following cases:
•
•
•
•
This report is generated whenever these conditions become true. The report displays the respective Warning Message which occurred in the system. The report also has a complete output of the process running in the system at that instance. In addition to the report, it also has statistical information on Cisco RPMS-related processes, and their usage.
Note
Provisioning Changes
Cisco RPMS tracks all provisioning changes made to the system database by logging all changes to a "Change Log" file. The file records the time, type and details of every change.
The change log file is generated in the $RPMSBASE/diagnostics/<Date>/ directory.The file is updated whenever there is a change to the system database.
Overview: Diagnosing End User Issues
You can use the Cisco RPMS diagnostic tools to diagnose common end user problems such as receiving busy signals, no connection, or if connected, having their calls end quickly. Reasons why these issues may occur include:
•
•
Oversubscribed Policies
End users may encounter busy signals due to oversubscribed policies. Oversubscription is a configurable Cisco RPMS feature, and when limits are exceeded, the system should send them busy signals. But end users may not like receiving the busy signals; to keep the signals to a minimum, the diagnostic tools allow you, the Cisco RPMS administrator, to monitor customer activity. You can access exactly which customers have exceeded the limits and are receiving busy signals, and which other customers are quickly approaching their limits and sending rejections. This way, you can track and re-configure extra or overflow ports so that end users do not receive busy signals.
Using the Cisco RPMS Diagnostics Tool to Monitor Customer Activity
To monitor the current status of customer activity, execute the crpms_diag CLI command:
%crpms_diag mon custThe command's output provides a snapshot of call activity for all customer profiles in the system, similar to this:
Data Updated at: Jul 31, 2002 3:36 PM, PDT"Customer","Limit Used(%)","Limit","Current Count","Current Limit Deficit(%)","Current Limit Deficit","Peak Limt Deficit(%)","Peak Limit Deficit"customer1,100,1010,1010,20,202,30,303customer2,0,110,0,0,0,0,0The customers at or near their limits appear at the top of the list.
Other output provided by the command includes:
•
•
You can use these numbers to change Customer Profile configurations.
Low Call Durations
Another problem end users may encounter is low call durations; the users might not receive busy signals, but their calls may end quickly or may not connect at all.
This is a UG-related issue, caused by the UG hardware. In a large network, it might be difficult to identify problematic UGs causing low call durations.
Using the Cisco RPMS Diagnostics Tool to Troubleshoot Low Call Durations
To display errors, execute the crpms_diag err CLI command:
%crpms_diag errThe command's output provides a snapshot of errors in the system, similar to this:
RPMS Errors:No errors todayDB Errors:No errors todayRPMS Client Errors:2002-Jul-28:Call Durations. Details in /export/home/crpms/diagnostics/2002-Jul-28/system/CallDurations_2002-J ul-28_00-00-07.csvIn this example, the "RPMS Client Errors" message displays the UGs sending traffic to Cisco RPMS that are experiencing low call durations. It also displays the file which contains further details about the problematic UGs.
Overview: Identifying Policy Enforcement Issues
This section helps identify policy enforcement issues that may occur with Cisco RPMS. The main issue that may occur is an incorrect call count.
Incorrect Active Call Count
Cisco RPMS may report more active calls in the system than actual calls in the network. The disparity between the active call count reported by Cisco RPMS and the number of real calls is caused by missing or delayed accounting messages. Depending on the Cisco IOS release running, accounting messages may occasionally be delayed or dropped.
Note
If this occurs, Cisco RPMS marks the calls as active even if they are dropped on the UG. Cisco RPMS only clears the calls when another call replaces the dropped call on the same port.
Using the Cisco RPMS Diagnostics Tool to Determine the Active Call Count
To display errors, execute the crpms_diag err CLI command:
%crpms_diag errThe command's output provides a snapshot of errors in the system, similar to this:
RPMS Errors:No errors todayDB Errors:No errors todayRPMS Client Errors:2002-Jul-26:Missing Accounting Stops. Details in /export/home/crpms/diagnostics/2002-Jul-26/system/MissingAcctStop_2002 -Jul-26_00-01-44.csvIn this example, the "RPMS Client Errors" message details that UGs sending traffic to the Cisco RPMS have not been sending accounting stop messages. It prints the name of the file containing additional information about the problematic UGs.
The file shows data in the following format:
Data Updated at: Jul 27, 2002 12:00 AM, PDT"NasName","IP","Missing Acct-Stop(%)","Total Attempted Calls","Calls w/o Stop",NAS-1,10.10.1.1,100,107,107The data details the total calls received from the UG, and the percent of calls from that total which did not receive an accounting stop message. You can use these numbers to compare a low traffic UG with a higher traffic UG.
In the example above, the Cisco RPMS received 107 calls from NAS-1, and 100% of those calls missed the accounting-stop message, which means the UG is not sending the required messages to the Cisco RPMS. With this information, you can look into why the UG and ensure it is running a Cisco RPMS compatible release of the Cisco IOS software.
Overview: Identifying Universal Gateway Issues
Hardware or software problems on the UG can cause service interruptions for the Cisco RPMS. The problems may only affect a few of the UGs, and as such, may be hard to track.
However, to address this issue, Cisco RPMS tracks the call termination codes for the UGs. You can use the crpms_diag CLI command to view the information and to display errors.
Using the Cisco RPMS Diagnostics Tool to Determine Universal Gateway Issues
To display errors, execute the crpms_diag err CLI command:
%crpms_diag errThe command's output provides a snapshot of errors in the system, similar to this:
RPMS Errors:No errors todayDB Errors:No errors todayRPMS Client Errors:2002-Jul-28:Call Failure Rates. Details in /export/home/crpms/diagnostics/2002-Jul-28/system/CallFailureRates_200 2-Jul-28_00-00-07.csvIn this example, the "RPMS Client Errors" message displays information about the UGs sending traffic to Cisco RPMS that might be experiencing hardware and or software problems. It also displays the file which contains further details about the problematic UGs.
The file shows data in the following format:
Data Updated at: 16:13:40"NAS NAME","NAS IP","Call Failure Rate","Total Attempted Calls","Failed Calls"NAS-1,10.10.1.1,10,100,10The data in this example displays all UGs with calls terminated because of errors. In Cisco RPMS, the following call termination codes are considered erroneous.
Table 7-22
2
Lost carrier.
3
Lost Service
9
UG error.
13
Port preempted.
14
Port suspended.
15
Service unavailable.
Erroneous call termination codes
The sample output in the example displays that the Cisco RPMS received 100 calls from NAS-1 and that 10% of those calls terminated because of errors. There may be a problem, and you should investigate the indicated UG.
Once the UG is identified, look for a breakdown of all terminated calls by termination code for this UG in the <crpms-home>/diagnostics/<date>/nas/ <nas-ip>/ NasTerminations_<timestamp>.csv file.
Overview: Tracking Provisioning Changes
Cisco RPMS tracks all of the provisioning changes you make to the system database. The changes, such as the time, type and details, are logged to a change Log file.
The change log file is generated in the <crpms-home>/diagnostics/<date>/ directory. It displays data in the following format:
Time Action Entity Description2002-Jul-23 4:41 PM CREATE Customer Name=cust1, Description=, Call Treatment=busy, Non Overflow Limit=100, Overflow Limit=10, Non Overflow Threshold=85, Call Reject Threshold=85, Overflow Call Rejection Threshold=102002-Jul-23 4:42 PM CREATE Customer-Mapping-Criteria Customer=cust1, Dnis Group=default, Call Type=digital, SS7 Resource=None, Trunk Group=defaultOverview: Viewing the Cisco RPMS History
The CLI command crpms_diag, as described in previous sections, supports a history option. You can use the history option to display information collected over a certain period of time, and to obtain a historical perspective of the Cisco RPMS system.
Using the Cisco RPMS Diagnostics Tool to View the System History
To display the history option, execute the crpms_diag err hist CLI command. In the following example, you could execute the history option to see any UG errors that occurred in the past seven days:
%crpms_diag err hist 7Overview: Analyzing Traffic Patterns with Cisco RPMS
You can use the Cisco RPMS diagnostic tools to analyze traffic patterns and collect information useful in planning for capacity or other issues. The information you can collect includes:
•
•
Collecting System Statistics
You can use the CLI command crpms_diag to collect high-level statistics about your Cisco RPMS system. To do so, execute the command as follows:
%crpms_diag statThe command output should look similar to:
Data Updated at: Jul 24, 2002 2:01 PM, PDTStatistics Name,Today,Since RestartCallAccept,5873,265873ResourceAllocated,65422,265832ResourceFreed,65356,265356ResourceUpdate,805,265805VpdnSession,0,0VpdnConnect,0,0VpdnDisconnect,0,0NasStart,0,0NasUpdate,65,468NasStop,0,0BadPackets,0,0UnknownPackets,0,0NoCustomerFound,0,0CallDiscriminations,0,0The previous example displays information such as the various types of packets received by Cisco RPMS so far that day, and since the last restart occurred. Cisco RPMS also records the information in a file so that you can retrieve statistics for a particular day from the file generated for that day. The file is generated in the following location:
<crpms-home>/diagnostics/<date>/system/SystemStats_<time>.csvAnalyzing System-Wide Traffic
Cisco RPMS periodically records the total number of active calls in the system. The information is stored in the <crpms-home>/diagnostics/<date>/system/ SystemActivity_<time>.csv file, which is generated daily.
The entries logged in this file are comma separated, so it is easy to import the file into a charting application such as Microsoft Excel, and to plot the active calls over time to get an idea about overall traffic patterns during the day. You can also analyze patterns over a period longer than a day by concatenating multiple files before charting them.
The data recorded in the SystemActivity files looks similar to this:
TimeStamp,Call Count16:14:27,72416:14:37,90316:14:47,91016:14:57,91016:15:08,91016:15:18,91016:15:28,91016:15:38,28516:15:48,30816:15:58,48216:16:09,565Overview: Analyzing Customer-Specific Traffic
With the Cisco RPMS diagnostic tools you can also analyze customer-specific traffic. The information you can collect includes:
•
•
Call Count Over Time
By using the Cisco RPMS' diagnostic tools, you can profile a customer's traffic pattern over time. Cisco RPMS periodically records all of the active calls for each customer profile in the system. The information is generated daily, and then stored in the <crpms-home>/diagnostics/<date>/customer/<customer-name>/ CustomerCounts_<time>.csv file.
The entries logged in this file are comma separated, so it is easy to import the file into a charting application such as Microsoft Excel, and to plot the active calls over time to get an idea about overall traffic patterns for a customer during the day. You can also analyze patterns over a period longer than a day by concatenating multiple files before charting them.
The data recorded in the CustomerCounts files looks similar to this:
Customer, Cust-1, Counts for Jul-23, 2002"Timestamp","Session Limit","Oversubscription Limit","Current Count","Current Limit Deficit"16:15:36,900,10,413,23816:17:36,900,10,692,23816:19:36,900,10,690,23816:21:36,900,10,693,23816:23:37,900,10,706,23816:25:37,900,10,699,23816:27:37,900,10,692,23816:29:37,900,10,683,23816:31:38,900,10,690,23816:33:38,900,10,688,23816:35:38,900,10,696,016:37:38,900,10,688,016:39:38,900,10,687,016:41:38,900,10,674,016:43:38,1000,10,696,0Plotting the Session Limit and the Call Count shows the time of day that a customer profile went into oversubscription mode. It also displays any provisioning changes made to the Session Limit for this profile.
Call Duration Distribution Over Time
Cisco RPMS also periodically records the duration of terminated calls for each customer. The information is sorted into a number of time bins, generated daily, and stored in the <crpms-home>/diagnostics/<date>/customer/<customer-name> /CustomerCallDuration_<time>.csv file.
The entries logged in this file are comma separated, so it is easy to import the file into a charting application such as Microsoft Excel, and to plot the active calls over time to get an idea about overall traffic patterns for a customer during the day. You can also analyze patterns over a period longer than a day by concatenating multiple files before charting them.
The data recorded in the CustomerCallDuration files looks similar to:
Customer, Cust-1, Call Duration Distribution for Jul-23, 2002"Timestamp","Closed Calls",%(0-1]min,%(1-2]min,%(2-4]min,%(4-8]min,%(8-16]min,%(16-32]min, %(32-64]min,%(64-128]min,%128+min16:13:46,100,0,0,0,80,0,20,0,0,016:13:56,10,0,0,0,90,0,0,10,0,016:14:06,0,0,0,0,0,0,0,0,0,0You can plot all of the bins if you want. The graph for that would show that end users for the sample customer profile (Cust-1) stay online longer in the morning and in late evenings.
