Table Of Contents
Troubleshooting the Statistics Infrastructure
Debugging Statistics Infrastructure
debug statsd api Commands
debug statsd manager Commands
debug statsd server errors Command
Trace Commands for the Statistics Infrastructure
show statsd manager trace Command
show statsd server trace Command
show stats lib trace
Show Commands for the Statistics Infrastructure
show statsd manager info Command
show statsd collectors Command
show statsd registrations Command
show statsd requests Command
Diagnosing Problems with Statistics Values
Errors When Retrieving Data from the Statistics Manager EDM
Timeouts and Delays When Retrieving Data
Displaying Incorrect Rates for the show interfaces Command
Displaying Incorrect Data for the show Commands
Troubleshooting the Statistics Infrastructure
These sections describe the methods that you can use to diagnose problems with statistics such as wrong and missing counters:
•
Debugging Statistics Infrastructure
•
Trace Commands for the Statistics Infrastructure
•
Show Commands for the Statistics Infrastructure
•
Diagnosing Problems with Statistics Values
Note
Most of the commands used in the chapter requires a Cisco-support task id to execute them.
Debugging Statistics Infrastructure
The following sections list the debug commands that are used to diagnose problems with the statistics infrastructure:
•
debug statsd api Commands
•
debug statsd manager Commands
•
debug statsd server errors Command
debug statsd api Commands
Table 11-1 lists the debug statsd api commands that are printed from the statistics collector library and are filtered by location and process. The statistics collector library is a process which supplies the statistics to the statistics infrastructure.
Table 11-1 debug statsd api Commands
Command
|
Description
|
debug statsd api errors
|
Prints any unexpected events that occurred in the statistics infrastructure. The command is safely enabled for all processes without the risk of flooding the console.
|
debug statsd api data
|
Prints the data that the process is sending to the statistics infrastructure. If the correct data is not being reported by the show commands, this command is useful to determine where the problems reside. In particular, the problems are narrowed down to a particular statistics collector by showing it is returning incorrect data. When this command is enabled, it can generate a large volume of data and hence must be restricted to a single process.
|
debug statsd manager Commands
Table 11-2 lists the debug statsd manager commands that are printed from the statistics manager process.
Table 11-2 debug statsd manager Commands
Command
|
Description
|
debug statsd manager errors
|
Prints any unexpected events that occurred in the statistics manager.
|
debug statsd manager sysdb-edm
|
Prints the details about the embedded device manager (EDM) in the statistics manager . The EDM is an internal service that handles requests for statistics data and returns the results. If there is a problem with retrieving statistics data through a show command, the details are useful. The command shows whether the request as was received by the statistics manager and whether it was interpreted correctly.
|
debug statsd manager datarate
|
Prints the counters received by the statistics manager and the resulting packet and byte rates that are calculated. This can be used to determine whether collectors are sending the right data and analyse unexpected rate values. It is recommended to restrict this command to a single interface.
|
debug statsd server errors Command
The debug statsd server errors command prints any unexpected events that occurred in the statistics server.
Trace Commands for the Statistics Infrastructure
Trace logs are buffers that wrap data which contains a history of recent events. Unlike debugging, which must be enabled, traces are always written to the logs. Therefore, the trace logs are always available after any problem has occurred. The statistics manager trace logs must always be recovered for any problem that involves the statistics infrastructure.
These sections list the trace commands that are used for the statistics infrastructure:
•
show statsd manager trace Command
•
show statsd server trace Command
•
show stats lib trace
Note
All the statistics trace commands are included in show tech-support pfi command and need not be run separately if the tech-support output has already been collected.
show statsd manager trace Command
Table 11-3 lists the trace buffers that are included in the output of the show statsd manager trace command. Each buffer can be displayed individually or in any combination. If no argument is specified, all buffers are displayed.
Table 11-3 Buffer Descriptions for the show statsd manager trace Command
Buffer
|
Description
|
error
|
Lists all unexpected events that have occurred in the statistics manager. Events are stored based on uniqueness so that at least one copy of each different error message is retained even if the main buffer was wrapped.
|
request
|
Lists all the requests that have been made and the number of responses that have been received with the node and PID that the response came from. The output of the request buffer is used with the show statsd collectors command with the brief keyword, to match the response to a particular process.
|
collection
|
Lists all the updates that are received from the collector. The buffer wraps frequently because messages are sent periodically and contains only data about fairly recent events.
|
store
|
Lists the records that are added and removed from the internal table used by the statistics manager to store its data in the memory such that it can be recovered after a process restart or failover.
|
datatree
|
Lists the entries that have been added into or removed from the datatree that indexes the checkpoint table.
|
types
|
Lists the mappings that have been made between the feature and item strings and statistics type IDs.
|
show statsd server trace Command
Table 11-4 lists the trace buffers that are included in the output from the show statsd server trace command. Each buffer can be displayed individually or in any combination. If no argument is specified, all buffers are displayed for the statistics server on the local route processor (RP). To display information for a particular location or all locations, use the location keyword.
Table 11-4 Buffer Descriptions for the show statsd server trace Command
Buffer
|
Description
|
error
|
Lists all unexpected events that have occurred in the statistics server. Events are stored based on uniqueness so that at least one copy of each different error message is retained even if the main buffer was wrapped.
|
collectors
|
Lists the registrations that have been made to convey the statistics server.
|
contributors
|
Lists the registrations that have been made for the statistics contribution.
|
show stats lib trace
This command can be used to see the events taking place in the Statistics library for collectors.
Table 11-5 lists the buffers that are included in the output of show stats lib trace command. These can be displayed individually or in any combination if required. If you do noy specify any arguments, further it displays all buffers for all collectors on the local RP. To display information for a particular location or all locations add location <foo|all>.
Table 11-5 Buffer Descriptions for the show stats lib trace Command
Buffer
|
Description
|
api
|
Lists all the API calls that have been made to the statistics library and their results.
|
error
|
Lists all unexpected events that have occurred in the statistics server. The error buffer has a unique component so that at least one copy of each different error message is retained even if the main buffer was wrapped.
|
gsp
|
Lists all GSP activity between the stats library and stats manager .
|
Show Commands for the Statistics Infrastructure
The output from all these commands are to be collected when any statistics issue is reported. These sections provide information about the state of the statistics infrastructure:
•
show statsd manager info Command
•
show statsd collectors Command
•
show statsd registrations Command
•
show statsd requests Command
show statsd manager info Command
The show statsd manager info command provides information about each of the modules that compose the statistics manager component. The command is used to display problems in the infrastructure and examine usage patterns.
The following sample output shows the start of the output of show statsd manager info command:
RP/0/RP0/CPU0:router# show statsd manager info
Table 11-6 describes the modules whose information appears in the show statsd manager info command output.
Table 11-6 Module Descriptions for the show statsd manager info command
Module
|
Description
|
Request
|
Sends the request to the Gateway Signalling Point (GSP) group of collectors. For every request, the number of processes that the message was sent to is counted, and a response is expected for each of them.
|
Collection
|
Receives all the messages from the statistics collectors.
|
EDM
|
Handles the system database (SysDB) requests from the management applications.
|
Data Store
|
Stores and checkpoints all data in the manager.
|
Type Store
|
Stores all the statistics types that are registered by the collectors.
|
The following sample output for the Request module shows the number of requests for which all responses were received, the number of requests that timed out before all responses were received, and the number of requests that failed for another reason:
6 successful requests, 0 timeouts, 0 errors
The following sample output for the Collection module shows the total number of messages and bytes that are received. The message count is broken down into periodic updates, positive responses to the statistics requests, negative responses to the statistics requests, and control messages from GSP.
Note
Other message types may not reside in any of these categories such as type lookup requests. Therefore, the sum of these counters may not match the count of messages received.
2415 messages received, 1039608 bytes received
2369 updates, 6 responses, 38 nacks, 0 from GSP
The following sample output from the show statsd manager info command shows the EDM. The module outputs the number of GET, DATALIST, and FINDDATA requests that have been made and the total number of results that have been returned in the lists.
The output divides the requests into an internal namespace and a new-feature namespace. The requests for the new-feature namespace are subdivided with the latest data requests from the collectors, data cache requests, and snapshot requests.
The number of clear requests made through the EDM is also counted. The requests made by the clear counters command are not included.
12 GETs, 0 DATALISTs of 0 elements, 0 FINDDATAs of 0 elements
12 requests using internal namespace
0 latest, 0 cached, 0 total requests using feature namespace
The following sample output from the show statsd manager info command shows the data store module. The first line of the output shows the number of bytes that is stored in the checkpointed memory and the number of records that are used. If a single statistics structure is larger than the maximum record size, the statistics structure is chained over multiple records. Therefore, the output also shows how many of the records represent the first record in a chain.
The second line shows the number of entries in the tree that are used to access the checkpoint records. Each entry holds up to three checkpoint records for the statistics data itself, as well as any rate calculation information and any snapshot that was taken, to ensure continuity over clear counters requests.
5824 bytes stored in 52 records (51 first records)
23 entries in tree (23 with data, 23 with rates, 0 with snapshots)
The following sample output from the show statsd manager info command shows the type store module. The output provides a table of both the internally defined types and the those that are registered by the features.
Type Feature name Item name Flags
---- ------------ --------- -----
6 internal IPV4_UNICAST 0x0
7 internal IPV4_MULTICAST 0x0
8 internal IPV6_UNICAST 0x0
9 internal IPV6_MULTICAST 0x0
101 fib_stats mpls_label 0x0
104 fib_stats mpls_ln70 0x0
102 fib_stats mpls_ln0 0x0
103 fib_stats mpls_glb 0x0
show statsd collectors Command
The show statsd collectors command lists all the processes that have registered as a statistics collector, and can be filtered to a single node or a single process.
Table 11-7 lists the keywords/options for the show statsd collectors command.
Table 11-7 Output Descriptions from the show statsd collectors Command
Output
|
Description
|
brief
|
Displays a a table of all the statistics collectors, the number of connections made from the process, the number of callback functions, and the number of registrations. In addition, the table shows the number of bytes that have been sent from the collector and the number of messages that were not sent to the manager.
|
default
|
Displays more detail about each statistics collector that includes a count of the total messages that have been sent and a breakdown into particular message types. In addition, the default output lists the callback handles for each registered callback function and provides the specified collection period, the number of registrations that are associated with the handle, and whether a bulk handler is registered or not.
|
detail
|
Displays detailed output that contains the same data as the default output. In addition, it includes a table of all the registrations made and the options that are associated with them.
|
The following sample output shows the brief output:
RP/0/RP0/CPU0:router# show statsd collectors brief location 0/0/CPU0
Node PID Process Conn CB Reg Bytes Errors
---- --- ------- ---- -- --- ----- ------
0x0 131157 statsd_server 1 2 3 19020 0
0x0 131160 fib_mgr 7 14 6 700 0
0x0 192617 ipv6_nd 1 1 0 180 0
0x0 192639 clns 1 1 0 180 0
0x0 192640 arp 1 1 1 7060 0
0x0 192665 nd_partner 1 1 1 14884 0
The following sample output shows the default output:
RP/0/RP0/CPU0:router# show statsd collectors location 0/0/CPU0 pid 131157
----------------------------------
statsd_server node: 0x0 PID: 131157
Total bytes sent: 23292 Total messages sent: 87 Errors: 0
periodic updates: 83 on-demand replies: 0
nacks: 1 other messages: 3
Connections: 1 Registered callbacks: 2
1. 0x0802d9fc Options 0x0 (DEFAULT)
Period 30 secs Bulk FALSE Total registrations 0
2. 0x0802d9c8 Options 0x0 (DEFAULT)
Period 30 secs Bulk TRUE Total registrations 3
The following sample output shows the detailed output:
RP/0/RP0/CPU0:router# show statsd collectors detail location 0/0/CPU0 pid 131157
----------------------------------
statsd_server node: 0x0 PID: 131157
Total bytes sent: 25700 Total messages sent: 96 Errors: 0
periodic updates: 91 on-demand replies: 0
nacks: 1 other messages: 4
Connections: 1 Registered callbacks: 2
1. 0x0802d9fc Options 0x0 (DEFAULT)
Period 30 secs Bulk FALSE Total registrations 0
2. 0x0802d9c8 Options 0x0 (DEFAULT)
Period 30 secs Bulk TRUE Total registrations 3
1. ifhandle 0x01000100 type 6 IPV4_UNICAST opts 0x1 (REG)
2. ifhandle 0x01000100 type 8 IPV6_UNICAST opts 0x1 (REG)
3. ifhandle 0x01000100 type 10 MPLS opts 0x1 (REG)
show statsd registrations Command
The show statsd registrations command lists all the processes that return data for a specific registration. The statistics type is specified either by using the feature and item strings or by using the statistics type ID, which is found in the output from the show statsd manager info command. The output provides only the node and process ID (PID) values of the collector process. To locate the process name, you need the output from the show statsd collectors command with the brief keyword.
The following sample output is from the show statsd registrations command with the ifname keyword:
RP/0/RP0/CPU0:router# show statsd registrations ifname MgmtEth 0/0/CPU0/0 feature internal
generic
ID type: 1 (ifhandle) ID: 0x01000100 feature: internal item: generic
0x0 28696 188 0x11 (ABS|REG)
The following sample output is from the show statsd registrations command with the ifhandle keyword:
RP/0/RP0/CPU0:router# show statsd registrations ifhandle 0x01000100 type 6
ID type: 1 (ifhandle) ID: 0x01000100 stats type: 6
0x0 131160 24 0x41 (USE_SW|REG)
show statsd requests Command
The show statsd requests command parses the ltrace buffers to provide details of the most recent statistics requests that have been handled by the statistics manager.
The following sample output is from the show statsd requests command:
RP/0/RP0/CPU0:router# show statsd requests
1 unique entries (64 possible, 0 filtered)
118 wrapping entries (1024 possible, 0 filtered, 118 total)
Displaying data about the last 12 requests (0 failed)
Time taken (ms): 1 / 15 / 37
ID Start Time and Duration Size Responses NACKs First stats type, ID type, ID
-- ----------------------- ---- --------- ----- -----------------------------
1 Apr 18 02:58:16 37ms 0 3 17 2 (generic), 1 (ifhandle), 0x02000080
2 Apr 18 02:58:16 2ms 0 2 6 2 (generic), 1 (ifhandle), 0x02000000
3 Apr 18 02:58:16 1ms 0 2 6 2 (generic), 1 (ifhandle), 0x02000100
4 Apr 18 02:58:16 1ms 0 2 6 2 (generic), 1 (ifhandle), 0x02000200
5 Apr 18 02:58:16 6ms 0 1 5 2 (generic), 1 (ifhandle), 0x03000300
6 Apr 18 02:58:16 8ms 0 1 5 2 (generic), 1 (ifhandle), 0x03000600
7 Apr 18 02:58:16 7ms 0 1 5 2 (generic), 1 (ifhandle), 0x03000900
8 Apr 18 02:58:16 10ms 0 1 5 2 (generic), 1 (ifhandle), 0x03000c00
9 Apr 18 02:58:16 28ms 0 1 5 2 (generic), 1 (ifhandle), 0x05000300
10 Apr 18 02:58:18 28ms 0 1 5 2 (generic), 1 (ifhandle), 0x05000600
11 Apr 18 02:58:18 28ms 0 1 5 2 (generic), 1 (ifhandle), 0x05000900
12 Apr 18 02:58:18 34ms 0 1 5 2 (generic), 1 (ifhandle), 0x05000c00
The following sample output shows detailed output from the show statsd requests command:
RP/0/RP0/CPU0:router# show statsd requests detail
1 unique entries (64 possible, 0 filtered)
118 wrapping entries (1024 possible, 0 filtered, 118 total)
Displaying data about the last 12 requests (0 failed)
Time taken (ms): 1 / 15 / 37
ID Start Time and Duration Size Responses NACKs First stats type, ID type, ID
-- ----------------------- ---- --------- ----- -----------------------------
8 Apr 18 02:58:16 10ms 0 1 5 2 (generic), 1 (ifhandle), 0x03000c00
NodeID JID PID Order Parts Bytes
0x20 100 57434 4 0 24 NACK
0x20 102 61541 1 0 24 NACK
0x20 123 57450 3 0 24 NACK
0x20 139 57444 2 1 220 Data
0x20 171 57441 0 0 24 NACK
0x20 245 57428 5 0 24 NACK
Diagnosing Problems with Statistics Values
The statistics infrastructure is the transport and storage mechanism for statistics data. Most of the problems that are seen in this area are not caused by the infrastructure itself but by the collectors that return the data or the underlying transport mechanisms.
These sections provide information about how to diagnose problems with statistics values:
•
Errors When Retrieving Data from the Statistics Manager EDM
•
Timeouts and Delays When Retrieving Data
•
Displaying Incorrect Rates for the show interfaces Command
•
Displaying Incorrect Data for the show Commands
Errors When Retrieving Data from the Statistics Manager EDM
You need to determine whether requests from the show command or other management agents are being received and correctly interpreted by the statistics manager. Table 11-8 lists the debug commands that must be enabled while running the show command.
Table 11-8 Debug Commands for the Statistics Manager
Command
|
Description
|
debug statsd manager errors
|
Displays the errors for the statistics manager.
|
debug statsd manager sysdb-edm
|
Displays the EDM-related activity of the statistics manager.
|
Additionally, examining the diagnostic output from the show statsd manager info command before and after the request shows whether or not the request is being received by the statistics manager EDM.
Timeouts and Delays When Retrieving Data
Table 11-9 lists the show and debug commands that help you identify the client that is not responding.
Table 11-9 show Commands for Statistics Requests
Command
|
Description
|
show process blocked location all
|
Displays the statistics manager or any statistics collector process that is being blocked.
|
show process statsd manager
|
Displays whether all threads are in receive state (the expected state).
|
show statsd requests
|
Displays the details of the recent requests, including the list of collectors that responded successfully.
|
show statsd collectors brief
|
Displays the entries for collectors on all nodes. If entries are missing for any node, this indicates a problem with the GSP group on that node.
|
show statsd manager info
|
Displays the request module information that shows whether any requests have failed or timed out.
|
show statsd manager trace reverse
|
Displays the latest traces first. You should locate lines such as "Stats request 1 completed with 0/7 responses outstanding." For each request, you must also look at each response that was received, such as "Received response with 1 element(s) for request 3 from member with node ID 0x0, pid 217201." or "Received NACK with 0 element(s) for request 3 from member with node ID 0x0, pid 229528".
If there is only one response outstanding, the problem resides with the collector that did not reply ( as it is blocked, or failed to join the group). Whereas, if there are many missing responses, the problem lies with the statistics GPS group, which is broken on one or more nodes.
|
show gsp groups [name group name | location node-id]
|
Displays the number of members of the statistics group (for example, statsd_group) on all the nodes and the list of members on the specified node. All collectors are members of this group. Using the list of collectors together with the list of processes that actually responded to the request can reveal which process did not respond.
The show gsp groups command displays a member of the group (for example, statsd_mgr_lwg). All collectors are writers to the group. Correlating the writers of the group with the members of the statsd_group name can indicate which process is having problems. If the group does not exist on one or more nodes, or if some nodes do not have the statistics manager listed as a member, there is a problem with the GSP group.
|
debug statsd api errors
|
Prints errors if a collector is failing to send messages to the statistics manager. Enabling the debug statsd api errors command and repeating the request is often the fastest way to find out which collector is not responding.
|
If any process is blocked, use the run attach_process -p PID command to find out where the process is blocked.
Displaying Incorrect Rates for the show interfaces Command
Perform this task if the packet or byte rates in the show interfaces command are not displayed correctly.
SUMMARY STEPS
1.
show interfaces
2.
show statsd collectors brief
or
show statsd registrations ifname interface name feature name item name
or
show statsd manager trace
3.
debug statsd manager datarate [interface type instance]
4.
Contact Cisco Technical Support if the problem is not resolved.
DETAILED STEPS
| |
Command or Action
|
Purpose
|
Step 1
|
show interfaces
Example:
RP/0/RP0/CPU0:router# show interfaces
|
Displays whether the number of packets and bytes sent and received are correct. If the packets and bytes are not correct, see "Displaying Incorrect Data for the show Commands" section. If the packet and byte rates are correct, the problem resides with the statistics infrastructure. Go to the next step to diagnose further.
|
Step 2
|
show statsd collectors brief
or
show statsd registrations ifname interface name
feature name item name
or
show statsd manager trace
Example:
RP/0/RP0/CPU0:router# show statsd collectors brief
RP/0/RP0/CPU0:router# show statsd registrations
ifname POS 0/1/0/0 feature internal generic
RP/0/RP0/CPU0:router# show statsd registrations
ifname POS 0/1/0/0 feature internal IPV4_UNICAST
RP/0/RP0/CPU0:router# show statsd manager trace
|
Collects the data for the statistics manager. If the packet and byte rates are correct, the problem resides with the statistics infrastructure.
|
Step 3
|
debug statsd manager datarate [interface type
instance]
Example:
RP/0/RP0/CPU0:router# debug statsd manager datarate
interface POS 0/1/0/0
|
Enables the debug statsd manager datarate command to collect the output. Turn this on for a couple of minutes. You can restrict it to a single interface to reduce the amount of output.
|
Step 4
|
Contact Cisco Technical Support.
|
Send all of the output collected above to Cisco Technical Support, stating the expected rate.
|
Displaying Incorrect Data for the show Commands
In general, the statistics infrastructure does not do anything with the data that it is given other than adding and storing it. Therefore, the problems with the data values are almost always the result of errors in the collectors.
Perform this task to find out which collectors are returning incorrect data.
SUMMARY STEPS
1.
show stats manager info
2.
show statsd registrations ifname interface name type stats ID
3.
show statsd collectors brief
DETAILED STEPS
| |
Command or Action
|
Purpose
|
Step 1
|
show stats manager info
Example:
RP/0/RP0/CPU0:router# show interfaces
|
To carry out the further steps, you need the type ID. You can find this by looking at the type store module information from the show statsd manager info command. For example, IPV4_UNICAST has a value of 6.
|
Step 2
|
show statsd registrations ifname interface name type
stats ID
Example:
RP/0/RP0/CPU0:router# show statsd registrations
ifname POS 0/1/0/0 type 300
|
Displays the list of processes that have registered to collect the statistics type for the interface. Retrieve the PID and location.
|
Step 3
|
show statsd collectors brief
Example:
RP/0/RP0/CPU0:router# show statsd collectors brief
|
Displays the list of collectors to correlate the node and PID to a process name.This is the collector which is at fault.
|

Note
If the collector has specified the USE_L3 flag, the interface counters are being aggregated from the Layer 3 counters. Use the accounting keyword to find the counters for each individual protocol.
When the statistics collector process is found, you can display the data it is returning by using the debug statsd api data command with the process and location keywords.
In addition, if you are interested in data for a particular interface, you can verify what the statistics manager is receiving by using the debug statsd manager datarate command with the interface keyword.
The following example show how to identify the processes:
RP/0/RP0/CPU0:altadena#show statsd manager info | inc generic
RP/0/RP0/CPU0:altadena#show statsd registrations ifname tenGigE 0/3/0/0 type 2
ID type: 1 (ifhandle) ID: 0x01380020 stats type: 2
0x31 61558 188 0x9 (ALL|REG)
RP/0/RP0/CPU0:altadena#show statsd collectors brief | inc 57416
0x21 57416 hfr_pm 3 6 31 16840 0
0x31 57416 hfr_pm 3 6 61 27604 0
RP/0/RP0/CPU0:altadena#show statsd collectors brief | inc 61558
0x31 61558 plim_8p_10ge 1 1 3 22688 0