Cisco IOS XR Troubleshooting Guide for the Cisco XR 12000 Series Router
Troubleshooting the Statistics Infrastructure
Downloads: This chapterpdf (PDF - 436.0KB) The complete bookPDF (PDF - 2.34MB) | Feedback

Troubleshooting the Statistics Infrastructure

Table Of Contents

Troubleshooting the Statistics Infrastructure

Debugging Statistics Infrastructure

debug statsd api Commands

debug statsd manager Commands

debug statsd server errors Command

Trace Commands for the Statistics Infrastructure

show statsd manager trace Command

show statsd server trace Command

show stats lib trace

Show Commands for the Statistics Infrastructure

show statsd manager info Command

show statsd collectors Command

show statsd registrations Command

show statsd requests Command

Diagnosing Problems with Statistics Values

Errors When Retrieving Data from the Statistics Manager EDM

Timeouts and Delays When Retrieving Data

Displaying Incorrect Rates for the show interfaces Command

Displaying Incorrect Data for the show Commands


Troubleshooting the Statistics Infrastructure


These sections describe the methods that you can use to diagnose problems with statistics such as wrong and missing counters:

Debugging Statistics Infrastructure

Trace Commands for the Statistics Infrastructure

Show Commands for the Statistics Infrastructure

Diagnosing Problems with Statistics Values


Note Most of the commands used in the chapter requires a Cisco-support task id to execute them.


Debugging Statistics Infrastructure

The following sections list the debug commands that are used to diagnose problems with the statistics infrastructure:

debug statsd api Commands

debug statsd manager Commands

debug statsd server errors Command

debug statsd api Commands

Table 9-1 lists the debug statsd api commands that are printed from the statistics collector library and are filtered by location and process. The statistics collector library is a process which supplies the statistics to the statistics infrastructure.

Table 9-1 debug statsd api Commands 

Command
Description

debug statsd api errors

Prints any unexpected events that occurred in the statistics infrastructure. The command is safely enabled for all processes without the risk of flooding the console.

debug statsd api data

Prints the data that the process is sending to the statistics infrastructure. If the correct data is not being reported by the show commands, this command is useful to determine where the problems reside. In particular, the problems are narrowed down to a particular statistics collector by showing it is returning incorrect data. When this command is enabled, it can generate a large volume of data and hence must be restricted to a single process.


debug statsd manager Commands

Table 9-2 lists the debug statsd manager commands that are printed from the statistics manager process.

Table 9-2 debug statsd manager Commands 

Command
Description

debug statsd manager errors

Prints any unexpected events that occurred in the statistics manager.

debug statsd manager sysdb-edm

Prints the details about the embedded device manager (EDM) in the statistics manager . The EDM is an internal service that handles requests for statistics data and returns the results. If there is a problem with retrieving statistics data through a show command, the details are useful. The command shows whether the request as was received by the statistics manager and whether it was interpreted correctly.

debug statsd manager datarate

Prints the counters received by the statistics manager and the resulting packet and byte rates that are calculated. This can be used to determine whether collectors are sending the right data and analyse unexpected rate values. It is recommended to restrict this command to a single interface.


debug statsd server errors Command

The debug statsd server errors command prints any unexpected events that occurred in the statistics server.

Trace Commands for the Statistics Infrastructure

Trace logs are buffers that wrap data which contains a history of recent events. Unlike debugging, which must be enabled, traces are always written to the logs. Therefore, the trace logs are always available after any problem has occurred. The statistics manager trace logs must always be recovered for any problem that involves the statistics infrastructure.

These sections list the trace commands that are used for the statistics infrastructure:

show statsd manager trace Command

show statsd server trace Command

show stats lib trace


Note All the statistics trace commands are included in show tech-support pfi command and need not be run separately if the tech-support output has already been collected.


show statsd manager trace Command

Table 9-3 lists the trace buffers that are included in the output of the show statsd manager trace command. Each buffer can be displayed individually or in any combination. If no argument is specified, all buffers are displayed.

Table 9-3 Buffer Descriptions for the show statsd manager trace Command 

Buffer
Description

error

Lists all unexpected events that have occurred in the statistics manager. Events are stored based on uniqueness so that at least one copy of each different error message is retained even if the main buffer was wrapped.

request

Lists all the requests that have been made and the number of responses that have been received with the node and PID that the response came from. The output of the request buffer is used with the show statsd collectors command with the brief keyword, to match the response to a particular process.

collection

Lists all the updates that are received from the collector. The buffer wraps frequently because messages are sent periodically and contains only data about fairly recent events.

store

Lists the records that are added and removed from the internal table used by the statistics manager to store its data in the memory such that it can be recovered after a process restart or failover.

datatree

Lists the entries that have been added into or removed from the datatree that indexes the checkpoint table.

types

Lists the mappings that have been made between the feature and item strings and statistics type IDs.


show statsd server trace Command

Table 9-4 lists the trace buffers that are included in the output from the show statsd server trace command. Each buffer can be displayed individually or in any combination. If no argument is specified, all buffers are displayed for the statistics server on the local route processor (RP). To display information for a particular location or all locations, use the location keyword.

Table 9-4 Buffer Descriptions for the show statsd server trace Command 

Buffer
Description

error

Lists all unexpected events that have occurred in the statistics server. Events are stored based on uniqueness so that at least one copy of each different error message is retained even if the main buffer was wrapped.

collectors

Lists the registrations that have been made to convey the statistics server.

contributors

Lists the registrations that have been made for the statistics contribution.


show stats lib trace

This command can be used to see the events taking place in the Statistics library for collectors.

Table 9-5 lists the buffers that are included in the output of show stats lib trace command. These can be displayed individually or in any combination if required. If you do noy specify any arguments, further it displays all buffers for all collectors on the local RP. To display information for a particular location or all locations add location <foo|all>.

Table 9-5 Buffer Descriptions for the show stats lib trace Command 

Buffer
Description

api

Lists all the API calls that have been made to the statistics library and their results.

error

Lists all unexpected events that have occurred in the statistics server. The error buffer has a unique component so that at least one copy of each different error message is retained even if the main buffer was wrapped.

gsp

Lists all GSP activity between the stats library and stats manager .


Show Commands for the Statistics Infrastructure

The output from all these commands are to be collected when any statistics issue is reported. These sections provide information about the state of the statistics infrastructure:

show statsd manager info Command

show statsd collectors Command

show statsd registrations Command

show statsd requests Command

show statsd manager info Command

The show statsd manager info command provides information about each of the modules that compose the statistics manager component. The command is used to display problems in the infrastructure and examine usage patterns.

The following sample output shows the start of the output of show statsd manager info command:

RP/0/0/CPU0:router# show statsd manager info
 
   
STATS MANAGER DATA
------------------
 
   

Table 9-6 describes the modules whose information appears in the show statsd manager info command output.

Table 9-6 Module Descriptions for the show statsd manager info command 

Module
Description

Request

Sends the request to the Gateway Signalling Point (GSP) group of collectors. For every request, the number of processes that the message was sent to is counted, and a response is expected for each of them.

Collection

Receives all the messages from the statistics collectors.

EDM

Handles the system database (SysDB) requests from the management applications.

Data Store

Stores and checkpoints all data in the manager.

Type Store

Stores all the statistics types that are registered by the collectors.


The following sample output for the Request module shows the number of requests for which all responses were received, the number of requests that timed out before all responses were received, and the number of requests that failed for another reason:

Request Module
    6 successful requests,  0 timeouts,  0 errors
 
   

The following sample output for the Collection module shows the total number of messages and bytes that are received. The message count is broken down into periodic updates, positive responses to the statistics requests, negative responses to the statistics requests, and control messages from GSP.

Note Other message types may not reside in any of these categories such as type lookup requests. Therefore, the sum of these counters may not match the count of messages received.

Collection Module
    2415 messages received,  1039608 bytes received
    2369 updates,  6 responses,  38 nacks,  0 from GSP
 
   

The following sample output from the show statsd manager info command shows the EDM. The module outputs the number of GET, DATALIST, and FINDDATA requests that have been made and the total number of results that have been returned in the lists.

The output divides the requests into an internal namespace and a new-feature namespace. The requests for the new-feature namespace are subdivided with the latest data requests from the collectors, data cache requests, and snapshot requests.

The number of clear requests made through the EDM is also counted. The requests made by the clear counters command are not included.

EDM Module
    12 GETs, 0 DATALISTs of 0 elements, 0 FINDDATAs of 0 elements
    12 requests using internal namespace
    0 latest, 0 cached, 0 total requests using feature namespace
    0 clear requests
 
   

The following sample output from the show statsd manager info command shows the data store module. The first line of the output shows the number of bytes that is stored in the checkpointed memory and the number of records that are used. If a single statistics structure is larger than the maximum record size, the statistics structure is chained over multiple records. Therefore, the output also shows how many of the records represent the first record in a chain.

The second line shows the number of entries in the tree that are used to access the checkpoint records. Each entry holds up to three checkpoint records for the statistics data itself, as well as any rate calculation information and any snapshot that was taken, to ensure continuity over clear counters requests.

Data Store Module
    5824 bytes stored in 52 records (51 first records)
    23 entries in tree (23 with data, 23 with rates, 0 with snapshots)
 
   

The following sample output from the show statsd manager info command shows the type store module. The output provides a table of both the internally defined types and the those that are registered by the features.

Type Store Module
    Type  Feature name      Item name         Flags
    ----  ------------      ---------         -----
    2     internal          generic           0x0
    5     internal          srp               0x0
    6     internal          IPV4_UNICAST      0x0
    7     internal          IPV4_MULTICAST    0x0
    8     internal          IPV6_UNICAST      0x0
    9     internal          IPV6_MULTICAST    0x0
    10    internal          MPLS              0x0
    11    internal          ARP               0x0
    12    internal          IPV6_ND           0x0
    13    internal          CDP               0x0
    14    internal          CLNS              0x0
    101   fib_stats         mpls_label        0x0
    104   fib_stats         mpls_ln70         0x0
    102   fib_stats         mpls_ln0          0x0
    103   fib_stats         mpls_glb          0x0
 
   

show statsd collectors Command

The show statsd collectors command lists all the processes that have registered as a statistics collector, and can be filtered to a single node or a single process.

Table 9-7 lists the keywords/options for the show statsd collectors command.

Table 9-7 Output Descriptions from the show statsd collectors Command 

Output
Description

brief

Displays a a table of all the statistics collectors, the number of connections made from the process, the number of callback functions, and the number of registrations. In addition, the table shows the number of bytes that have been sent from the collector and the number of messages that were not sent to the manager.

default

Displays more detail about each statistics collector that includes a count of the total messages that have been sent and a breakdown into particular message types. In addition, the default output lists the callback handles for each registered callback function and provides the specified collection period, the number of registrations that are associated with the handle, and whether a bulk handler is registered or not.

detail

Displays detailed output that contains the same data as the default output. In addition, it includes a table of all the registrations made and the options that are associated with them.


The following sample output shows the brief output:

RP/0/0/CPU0:router# show statsd collectors brief location 0/0/CPU0
 
   
STATS COLLECTORS
  Node         PID  Process               Conn    CB   Reg     Bytes  Errors
  ----         ---  -------               ----    --   ---     -----  ------
  0x0       131157  statsd_server            1     2     3     19020       0
  0x0       131160  fib_mgr                  7    14     6       700       0
  0x0       192617  ipv6_nd                  1     1     0       180       0
  0x0       192639  clns                     1     1     0       180       0
  0x0       192640  arp                      1     1     1      7060       0
  0x0       192665  nd_partner               1     1     1     14884       0
 
   

The following sample output shows the default output:

RP/0/0/CPU0:router# show statsd collectors location 0/0/CPU0 pid 131157
 
   
STATS COLLECTORS
----------------------------------
statsd_server  node: 0x0     PID: 131157  
    Total bytes sent: 23292  Total messages sent: 87  Errors: 0
        periodic updates: 83        on-demand replies: 0       
        nacks:            1         other messages:    3       
    Connections: 1  Registered callbacks: 2
    1. 0x0802d9fc  Options 0x0 (DEFAULT)
        Period 30 secs  Bulk FALSE  Total registrations 0
    2. 0x0802d9c8  Options 0x0 (DEFAULT)
        Period 30 secs  Bulk  TRUE  Total registrations 3
 
   

The following sample output shows the detailed output:

RP/0/0/CPU0:router# show statsd collectors detail location 0/0/CPU0 pid 131157
 
   
STATS COLLECTORS
----------------------------------
statsd_server  node: 0x0     PID: 131157  
    Total bytes sent: 25700  Total messages sent: 96  Errors: 0
        periodic updates: 91        on-demand replies: 0       
        nacks:            1         other messages:    4       
    Connections: 1  Registered callbacks: 2
    1. 0x0802d9fc  Options 0x0 (DEFAULT)
        Period 30 secs  Bulk FALSE  Total registrations 0
    2. 0x0802d9c8  Options 0x0 (DEFAULT)
        Period 30 secs  Bulk  TRUE  Total registrations 3
        1. ifhandle 0x01000100 type    6 IPV4_UNICAST     opts 0x1 (REG)
        2. ifhandle 0x01000100 type    8 IPV6_UNICAST     opts 0x1 (REG)
        3. ifhandle 0x01000100 type   10 MPLS             opts 0x1 (REG)
 
   

show statsd registrations Command

The show statsd registrations command lists all the processes that return data for a specific registration. The statistics type is specified either by using the feature and item strings or by using the statistics type ID, which is found in the output from the show statsd manager info command. The output provides only the node and process ID (PID) values of the collector process. To locate the process name, you need the output from the show statsd collectors command with the brief keyword.

The following sample output is from the show statsd registrations command with the ifname keyword:

RP/0/0/CPU0:router# show statsd registrations ifname MgmtEth 0/0/CPU0/0 feature internal 
generic
 
   
STATS REGISTRATIONS
-------------------
ID type: 1 (ifhandle)  ID: 0x01000100  feature: internal  item: generic
  Node    PID           Size  Options
  ----    ---           ----  -------
  0x0     28696          188  0x11 (ABS|REG)
 
   

The following sample output is from the show statsd registrations command with the ifhandle keyword:

RP/0/0/CPU0:router# show statsd registrations ifhandle 0x01000100 type 6
 
   
STATS REGISTRATIONS
-------------------
ID type: 1 (ifhandle)  ID: 0x01000100  stats type: 6
  Node    PID           Size  Options
  ----    ---           ----  -------
  0x0     131157          76  0x1 (REG)
  0x0     131160          24  0x41 (USE_SW|REG)
 
   

show statsd requests Command

The show statsd requests command parses the ltrace buffers to provide details of the most recent statistics requests that have been handled by the statistics manager.

The following sample output is from the show statsd requests command:

RP/0/0/CPU0:router# show statsd requests
 
   
1 unique entries (64 possible, 0 filtered)
118 wrapping entries (1024 possible, 0 filtered, 118 total)
STATS REQUEST DATA
------------------
  Displaying data about the last 12 requests (0 failed)
                   min / avg / max
  Time taken (ms): 1 / 15 / 37
  Responses:       1 / 1 / 3
  Nacks:           5 / 6 / 17
 
   
    ID Start Time and Duration Size Responses NACKs First stats type, ID type, ID
    -- ----------------------- ---- --------- ----- -----------------------------
     1 Apr 18 02:58:16    37ms    0         3    17 2 (generic), 1 (ifhandle), 0x02000080 
     2 Apr 18 02:58:16     2ms    0         2     6 2 (generic), 1 (ifhandle), 0x02000000 
     3 Apr 18 02:58:16     1ms    0         2     6 2 (generic), 1 (ifhandle), 0x02000100 
     4 Apr 18 02:58:16     1ms    0         2     6 2 (generic), 1 (ifhandle), 0x02000200 
     5 Apr 18 02:58:16     6ms    0         1     5 2 (generic), 1 (ifhandle), 0x03000300 
     6 Apr 18 02:58:16     8ms    0         1     5 2 (generic), 1 (ifhandle), 0x03000600 
     7 Apr 18 02:58:16     7ms    0         1     5 2 (generic), 1 (ifhandle), 0x03000900 
     8 Apr 18 02:58:16    10ms    0         1     5 2 (generic), 1 (ifhandle), 0x03000c00 
     9 Apr 18 02:58:16    28ms    0         1     5 2 (generic), 1 (ifhandle), 0x05000300 
    10 Apr 18 02:58:18    28ms    0         1     5 2 (generic), 1 (ifhandle), 0x05000600 
    11 Apr 18 02:58:18    28ms    0         1     5 2 (generic), 1 (ifhandle), 0x05000900 
    12 Apr 18 02:58:18    34ms    0         1     5 2 (generic), 1 (ifhandle), 0x05000c00 
 
   

The following sample output shows detailed output from the show statsd requests command:

RP/0/0/CPU0:router# show statsd requests detail
 
   
1 unique entries (64 possible, 0 filtered)
118 wrapping entries (1024 possible, 0 filtered, 118 total)
STATS REQUEST DATA
------------------
  Displaying data about the last 12 requests (0 failed)
                   min / avg / max
  Time taken (ms): 1 / 15 / 37
  Responses:       1 / 1 / 3
  Nacks:           5 / 6 / 17
 
   
    ID Start Time and Duration Size Responses NACKs First stats type, ID type, ID
    -- ----------------------- ---- --------- ----- -----------------------------
     8 Apr 18 02:58:16    10ms    0         1     5 2 (generic), 1 (ifhandle), 0x03000c00 
      NodeID     JID        PID Order Parts Bytes
      0x20       100      57434     4     0    24 NACK
      0x20       102      61541     1     0    24 NACK
      0x20       123      57450     3     0    24 NACK
      0x20       139      57444     2     1   220 Data
      0x20       171      57441     0     0    24 NACK
      0x20       245      57428     5     0    24 NACK
 
   

Diagnosing Problems with Statistics Values

The statistics infrastructure is the transport and storage mechanism for statistics data. Most of the problems that are seen in this area are not caused by the infrastructure itself but by the collectors that return the data or the underlying transport mechanisms.

These sections provide information about how to diagnose problems with statistics values:

Errors When Retrieving Data from the Statistics Manager EDM

Timeouts and Delays When Retrieving Data

Displaying Incorrect Rates for the show interfaces Command

Displaying Incorrect Data for the show Commands

Errors When Retrieving Data from the Statistics Manager EDM

You need to determine whether requests from the show command or other management agents are being received and correctly interpreted by the statistics manager. Table 9-8 lists the debug commands that must be enabled while running the show command.

Table 9-8 Debug Commands for the Statistics Manager 

Command
Description

debug statsd manager errors

Displays the errors for the statistics manager.

debug statsd manager sysdb-edm

Displays the EDM-related activity of the statistics manager.


Additionally, examining the diagnostic output from the show statsd manager info command before and after the request shows whether or not the request is being received by the statistics manager EDM.

Timeouts and Delays When Retrieving Data

Table 9-9 lists the show and debug commands that help you identify the client that is not responding.

Table 9-9 show Commands for Statistics Requests 

Command
Description

show process blocked location all

Displays the statistics manager or any statistics collector process that is being blocked.

show process statsd manager

Displays whether all threads are in receive state (the expected state).

show statsd requests

Displays the details of the recent requests, including the list of collectors that responded successfully.

show statsd collectors brief

Displays the entries for collectors on all nodes. If entries are missing for any node, this indicates a problem with the GSP group on that node.

show statsd manager info

Displays the request module information that shows whether any requests have failed or timed out.

show statsd manager trace reverse

Displays the latest traces first. You should locate lines such as "Stats request 1 completed with 0/7 responses outstanding." For each request, you must also look at each response that was received, such as "Received response with 1 element(s) for request 3 from member with node ID 0x0, pid 217201." or "Received NACK with 0 element(s) for request 3 from member with node ID 0x0, pid 229528".

If there is only one response outstanding, the problem resides with the collector that did not reply ( as it is blocked, or failed to join the group). Whereas, if there are many missing responses, the problem lies with the statistics GPS group, which is broken on one or more nodes.

show gsp groups [name group name | location node-id]

Displays the number of members of the statistics group (for example, statsd_group) on all the nodes and the list of members on the specified node. All collectors are members of this group. Using the list of collectors together with the list of processes that actually responded to the request can reveal which process did not respond.

The show gsp groups command displays a member of the group (for example, statsd_mgr_lwg). All collectors are writers to the group. Correlating the writers of the group with the members of the statsd_group name can indicate which process is having problems. If the group does not exist on one or more nodes, or if some nodes do not have the statistics manager listed as a member, there is a problem with the GSP group.

debug statsd api errors

Prints errors if a collector is failing to send messages to the statistics manager. Enabling the debug statsd api errors command and repeating the request is often the fastest way to find out which collector is not responding.


If any process is blocked, use the run attach_process -p PID command to find out where the process is blocked.

Displaying Incorrect Rates for the show interfaces Command

Perform this task if the packet or byte rates in the show interfaces command are not displayed correctly.

SUMMARY STEPS

1. show interfaces

2. show statsd collectors brief
or
show statsd registrations ifname interface name feature name item name
or
show statsd manager trace

3. debug statsd manager datarate [interface type instance]

4. Contact Cisco Technical Support if the problem is not resolved.

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

show interfaces

Example:

RP/0/0/CPU0:router# show interfaces

Displays whether the number of packets and bytes sent and received are correct. If the packets and bytes are not correct, see "Displaying Incorrect Data for the show Commands" section. If the packet and byte rates are correct, the problem resides with the statistics infrastructure. Go to the next step to diagnose further.

Step 2 

show statsd collectors brief
or

show statsd registrations ifname interface name feature name item name
or

show statsd manager trace

Example:

RP/0/0/CPU0:router# show statsd collectors brief

RP/0/0/CPU0:router# show statsd registrations ifname POS 0/1/0/0 feature internal generic

RP/0/0/CPU0:router# show statsd registrations ifname POS 0/1/0/0 feature internal IPV4_UNICAST

RP/0/0/CPU0:router# show statsd manager trace

Collects the data for the statistics manager. If the packet and byte rates are correct, the problem resides with the statistics infrastructure.

Step 3 

debug statsd manager datarate [interface type instance]

Example:

RP/0/0/CPU0:router# debug statsd manager datarate interface POS 0/1/0/0

Enables the debug statsd manager datarate command to collect the output. Turn this on for a couple of minutes. You can restrict it to a single interface to reduce the amount of output.

Step 4 

Contact Cisco Technical Support.

Send all of the output collected above to Cisco Technical Support, stating the expected rate.

Displaying Incorrect Data for the show Commands

In general, the statistics infrastructure does not do anything with the data that it is given other than adding and storing it. Therefore, the problems with the data values are almost always the result of errors in the collectors.

Perform this task to find out which collectors are returning incorrect data.

SUMMARY STEPS

1. show stats manager info

2. show statsd registrations ifname interface name type stats ID

3. show statsd collectors brief

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

show stats manager info

Example:

RP/0/0/CPU0:router# show interfaces

To carry out the further steps, you need the type ID. You can find this by looking at the type store module information from the show statsd manager info command. For example, IPV4_UNICAST has a value of 6.

Step 2 

show statsd registrations ifname interface name type stats ID

Example:

RP/0/0/CPU0:router# show statsd registrations ifname POS 0/1/0/0 type 300

Displays the list of processes that have registered to collect the statistics type for the interface. Retrieve the PID and location.

Step 3 

show statsd collectors brief

Example:

RP/0/0/CPU0:router# show statsd collectors brief

Displays the list of collectors to correlate the node and PID to a process name.This is the collector which is at fault.


Note If the collector has specified the USE_L3 flag, the interface counters are being aggregated from the Layer 3 counters. Use the accounting keyword to find the counters for each individual protocol.


When the statistics collector process is found, you can display the data it is returning by using the debug statsd api data command with the process and location keywords.

In addition, if you are interested in data for a particular interface, you can verify what the statistics manager is receiving by using the debug statsd manager datarate command with the interface keyword.

The following example show how to identify the processes:

RP/0/0/CPU0:altadena#show statsd manager info | inc generic
    2     internal          generic           0x0
 
   
RP/0/RP0/CPU0:altadena#show statsd registrations ifname tenGigE 0/3/0/0 type 2
STATS REGISTRATIONS
-------------------
ID type: 1 (ifhandle)  ID: 0x01380020  stats type: 2
  Node    PID           Size  Options
  ----    ---           ----  -------
  0x31    57416          188  0x1 (REG)
  0x31    61558          188  0x9 (ALL|REG)
 
   
RP/0/0/CPU0:altadena#show statsd collectors brief | inc 57416
  0x21       57416  hfr_pm                   3     6    31         16840       0
  0x31       57416  hfr_pm                   3     6    61         27604       0
 
   
RP/0/0/CPU0:altadena#show statsd collectors brief | inc 61558
  0x31       61558  plim_8p_10ge             1     1     3         22688       0