Introduction
This document describes steps to troubleshoot Charging Data Records (CDRs)/General Packet Radio Service (GPRS) Tunneling Protocol Prime (GTPP) archiving in Aggregation Services Routers (ASR) 5000/ASR 5500/Virtual Packet Core.
Background Information
ASR 5000/ASR 5500/Virtual Packet Core may archive CDRs for many reasons (unable to transmit files due to IP connectivity issues, the remote server is unable to receive CDRs, various misconfigurations, etc.). An aaaproxy restart resolves the issue in many cases even if it is a Charging Gateway Function (CGF) issue. For example, if a CGF is unable to accept a particular type of message (e.g. cancel request) then after the aaaproxy restarts, the message is no longer being sent. Since the restart of aaaproxy addresses the issue, it gives a false positive as ASR 5000/ASR 5500/Virtual Packet Core being the cause. Using an external PCAP to capture traffic would help identify the cause, which in this case would be the CGF.
Problem
The show gtpp counters shows the type and counters for CDRs. The counters show archived CDRs. In the example here, the number of archived Gateway GPRS Support Node (GGSN) CDRs (GCDRs) is 144015. Multiple outputs of the show gtpp counters show if the number of archived CDRs is increasing.
[local]StarOS# show gtpp counters all
Archived GCDRs: 144015
GCDRs buffered with AAAPROXY: 0
GCDRs buffered with AAAMGR: 22354
This output shows an ongoing Serving GPRS Support Node (SGSN) CDRs (SCDRs) archiving while GCDRs archive is stable.
[local]StarOS# show gtpp counters all | grep Archive
Archived GCDRs: 176703
Archived MCDRs: 0
Archived SCDRs: 2244673
Archived S-SMO-CDRs: 0
Archived S-SMT-CDRs: 0
Archived G-MB-CDRs: 0
Archived SGW CDRs: 0
Archived WLAN CDRs: 0
Archived LCS-MT CDRs: 0
[local]StarOS# show gtpp counters all | grep Archive
Archived GCDRs: 176703
Archived MCDRs: 0
Archived SCDRs: 2244864
Archived S-SMO-CDRs: 0
Archived S-SMT-CDRs: 0
Archived G-MB-CDRs: 0
Archived SGW CDRs: 0
Archived WLAN CDRs: 0
Archived LCS-MT CDRs: 0
[local]StarOS# show gtpp counters all | grep Archive
Archived GCDRs: 176703
Archived MCDRs: 0
Archived SCDRs: 2245281
Archived S-SMO-CDRs: 0
Archived S-SMT-CDRs: 0
Archived G-MB-CDRs: 0
Archived SGW CDRs: 0
Archived WLAN CDRs: 0
Archived LCS-MT CDRs: 0
Checking syslogs for 'gtpp 52056' warning can be used to identify the context and GTPP group where archiving of CDRs is happening. This output shows that archiving is reported for context GTPP and gtpp group default.
[gtpp 52056 warning] [5/0/2399 <aaamgr:50> gr_gtpp_proxy.c:667] [context: GTPP, contextID: 6]
[software internal security system critical-info syslog] [gtpp-group default]
GTPP request with req-count 61747 retried by AAAmgr. Retry-count 3342670
Solution
1. The wrong configuration can lead to accumulation of CDRs in the archive. If CDRs/GTPP records are generated by an unintended GTPP group, and this group has an invalid configuration, archiving will occur. Verify that the configuration is present or valid for these common issues:
- "gtpp group default" in the APN configuration
- "accounting context" in GGSN, Serving Gateway (SGW), SAEGW, SGSN services
- Charging-agent IP and CGF server IP address.
- Check if CGF is up and running.
2. Check if the socket interface is up in the corresponding context. Socket creation failure can lead to CDR archiving. In order to identify such issues, test the CGF connectivity with this command. This command should be executed in the context where gtpp group is configured.
[context]StarOS# gtpp test accounting group name <name>
3. Check the RTD (round trip delay) whether Charging gateway is acknowledging the CDRs. The "show gtpp statistics verbose" shows the RTD for CGF.
4. Check the transport network to determine if it has the capacity to handle the traffic by the gateway. Delay or packet drop in the network will cause CDRs to be archived in the gateway. If the packets are dropped (resulting in re-transmission of packets from ASR 5000/ASR 5500/Virtual Packet Core, which slows down the CDR transmission rate), this will result in archived CDRs. This can be fixed by increasing the Transport link capacity or adding QoS in the network.
5. Check active records in a aaamgr instance with "debug aaamgr show archive-records instance <aaamgr_instance_id>" (it requires CLI test-commands password configured in the chassis.) on the newer software releases provides information on CDR type, context and GTPP group name for archived records on a specific aaamgr. This information helps in identifying possible misconfigurations. From below example output, it's clear that CDRs are stuck/archived in gtpp group default in context ggsn. The APN which generated these CDRs is apn wifitest. Possibly this default gtpp group in the ggsn context has an invalid configuration.
--------------------------------------------------------------------------------------
Record Type | Apn Name | Accounting Context | Group Name | Timestamp
--------------------------------------------------------------------------------------
EGCDR |wifitest |ggsn |default |Tuesday August 26 10:18:21
EGCDR |wifitest |ggsn |default |Tuesday August 26 10:23:21
EGCDR |wifitest |ggsn |default |Tuesday August 26 10:28:21
EGCDR |wifitest |ggsn |default |Tuesday August 26 10:33:22
Feedback