ASR 5000 Series Troubleshooting GTPC and GTPU and associated path failures

Available Languages

Download Options

  • PDF     (147.4 KB)
    View with Adobe Reader on a variety of devices
  • ePub     (101.4 KB)
    View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
  • Mobi (Kindle)     (279.6 KB)
    View on Kindle device or Kindle app on multiple devices
Updated:July 23, 2015
Document ID:200026

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

This article covers understanding and troubleshooting Evolved General Packet Radio Service Tunneling Protocol for Control Plane (EGTPC) / GTPC (3rd Generation Partnership Project Technical Specification 29.274), GPRS Tunnelling protocol User Plane (GTPv1-U) (3GPP TS 29.281), and most relevant to this article, Restoration Procedures (3GPP TS 23.007) issues , which revolve around such concepts as restart counters, path failures detection, and echo requests/responses. It includes a list of relevant commands and what to look for in the output, along with explanation of related configurables. This is certainly an area for which a lot of questions are raised. Note that troubleshooting specific call control issues is in itself a very extensive area of interest, but this article only mentions how to capture statistics for call control - anything beyond that (i.e. call setup issues) is not covered. Really this article focuses mainly on troubleshooting and verifying the peer connections, without which there would be no call control.  While written for operators, designers / implementers could also benefit from the content here.
Examples all use fake (changed) IP addresses from a real operator's network.

Background

Each side of a GTP connection (one of S5 (Serving Gateway (SGW) - Packet Data Network Gateway (PGW)), S11 (Multimedia Management Entity (MME) - SGW), S4 (Serving GPRS Service Node (SGSN) - Gateway GPRS Service Node (GGSN)) maintains a restart counter that is designed to alert the peer if the originating node has restarted so that that peer can take the appropriate action. While a node can use a unique counter for each peer it talks to, the design tends to be to use the SAME counter for all the peers, since when the node restarts, the counter needs to be incremented for all peers, and so tracking a separate value on a peer basis serves no extra benefit since they all need to increase by one anyway. In fact using the same restart counter for all peers is helpful in troubleshooting a large network where all the peers can be checked to make sure that the restart counters received have the same (single) value for the node those peers are talking to.
The restart counter CAN be included in all types of control messages, while the restart counter MUST be included in all echo requests and responses. As soon as a restart counter change occurs on a node, the various peers are informed about it via the next control messages sent to the respective peers. When a restart occurs, per specification, the receiving side has the option of temporarily maintaining the calls it has for a short while or in dropping them, the latter which is typically what is the configured behavior.
In the case of using echo requests, the method is to send the messages on a configured interval, and if no response is received after a configured number of retries (each after a configured timeout), the path is considered to be down and the appropriate action taken.  Configuration allows for echos to be optionally sent, and this would apply to the peer as well, whose approach doesn't necessarily have to be the same, especially when the peer could be from a different vendor and/or different carrier. In fact with regards to Echo responses, a node is only obligated to respond to an Echo request - whether it receives echos or not and with what frequency is not significant - it just needs to send an echo response for each request, and to take action if the restart counter received changes. Of course as already mentioned, IF it is also sending echo requests and doesn't receive timely responses, it needs to take action for that scenario as well.
Note that path failure detection can also be done over the GTP-U (user plane) connection via Echo, but ONLY via timeout, since there is no restart counter included in GTP-U messages like there is for GTP-C as previously discussed. GTP-U may or may not be using the same path as by control messages. If it is the same path, then GTP-U detection may not be as valuable since GTP-C detection should be sufficient.
In summary, here are the ways that path failure detection can be done:
  • Restart Counter Change (via control msg Req/Rsp or Echo)
  • GTPC Echo Req Time out
  • GTPU Echo Req Time out
  • Control Req Message Time-out (usually not configured)

Relevant Commands

In order to troubleshoot any kind of path issues, familiarize with the following commands. Following is an explanation of the relevant details of each.
  • show [egtpc-service | gtpu-service] all
  • show egtpc peers [egtp-service <service>]
  • egtpc test echo gtp-version {1 | 2} src-address <service address> peer-address <peer address>
  • show egtpc statistics [[[ (sgw-address | pgw-address | mme-address) <address>] [demux-only]] [debug-only] [verbose] | path-failure-reasons]
  • show gtpu statistics [peer-address <peer>]
  • show gtpc statistics [ggsn-service <service>] [smgr-instance <instance>]
  • show demux-mgr statistics <egtpinmgr | egtpegmgr | gtpumgr> all (SGW-egress | GGSN/ PGW,/SGW ingress | GTPU)
  • EGTPCPathFail[Clear], EGTPUPathFail[Clear] SNMP traps
  • show session disconnect-reasons - gtpc-path-failure, gtpu-path-failure, path-failure
Other commands for counting subscribers related to GTP:
show sub [summary] ggsn-service <service>
show sub [summary] gtpu-service <service>
show egtpc sessions [egtp-service <service>]
The basic command to check the status and configuration of all EGTP services is "show egtpc-service". Most of the output should be what is already known via the configuration, as well as the status should be STARTED. The unique piece of information gleaned is the "Restart Counter" which cannot be found anywhere else. Related command "show gtpu-service all" returns configurables for the GTPU service(s) that are associated with the EGTPC services.
"show egtpc peers" is very useful in identifying all of the peers, including statuses, restart counters, and current/max subscriber counts.
  • Note that this command covers both SGW and SGSN peers, whereas in the past there was a separate command for SGSN/GGSN.
  • Since this command includes all peer types, the peer type can be filtered on via the service name (Even though the keyword egtpc-service deceptively also takes a GGSN service name).
  • If there are no subscribers with the peer, then the status is Inactive and GTPC Echo disabled. In that state, for EGTPC there MAY be some peers for which the Restart counter is reported, while most may be 0.  This occurs for situations where the peer is actively sending echo requests to which ASR5K should be responding. The restart counter can be determined by running the EGTPC test command (discussed next), though in the inactive state, the counter will not be updated in "show egtpc peers" even though it was received.   Meanwhile for GGSN, the restart counter is never forgotten.
[local]PGW-GGSN> show egtpc peers egtp-service GGSN1
|||||  Service                       Restart--+   No. of
|||||  ID                            Counter  |  restarts 
|||||   |                                     |     |     Current   Max
vvvvv   v              Peer Address           v     v     sessions sessions
-----  --- -------------------------------- --- --- ----------- -- ---------

IDNKG   6                   203.0.113.69    165     0           0           1
IDNKG   6                   203.0.113.70     25     0           0           1
IDNKG   6                   203.0.113.71    165     0           0           1
IDNKG   6                   203.0.113.72     25     0           0           1
IDNKG   6                   203.0.113.73     53     0           0           1
IDNKG   6                   203.0.113.74     53     0           0           1

IDNKG   6                   192.0.2.9        46     0           0           2
IDNKG   6                   192.0.2.10       44     0           0           1
IDNKG   6                   192.0.2.11       47     0           0           2
IDNKG   6                   192.0.2.12       44     0           0           2
IDNKG   6                   192.0.2.13       54     0           0           1
IDNKG   6                   192.0.2.14       47     0           0           1

IDNKG   6                   209.165.200.225  35     1           0           3
IDNKG   6                   209.165.200.226  35     3           0           3

IDNK   6                    198.51.100.204  156     0           0           1
IDNK   6                    198.51.100.205  158     0           0           1
IDNK   6                    198.51.100.211   54     0           0           1
IDNK   6                    198.51.100.212   54     0           0           1
IDNK   6                    198.51.100.213   56     0           0           1
IDNK   6                    198.51.100.214   56     0           0           1
IDNK   6                    198.51.100.215   54     0           0           1
 Note a couple peers below that have counts much higher than the other peers. These are for SGWs that are physically located in the same site or the sister site of the PGW to which the MME funnels most calls.
[local]PGW> show egtpc peers
Monday March 30 20:10:26 UTC 2015

+----Status:                 (I) - Inactive  (A) - Active
|+---GTPC Echo:              (D) - Disabled  (E) - Enabled 
||+--Restart Counter Sent:   (S) - Sent      (N) - Not Sent
|||+-Peer Restart Counter:   (K) - Known     (U) - Unknown
||||+-Type of Node:          (S) - SGW       (P) - PGW
|||||                        (M) - MME       (G) - SGSN
|||||                        (L) - LGW       (E) - ePDG
|||||                        (C) - CGW
|||||                        (U) - Unknown 
|||||  Service                       Restart--------+  No. of 
|||||  ID                            Counter        |  restarts
|||||   |                                           |   |   Current       Max
vvvvv   v              Peer Address                 v   v   sessions    sessions
-----  --- --------------------------------------- --- --- ----------- ----------
IDNUS   5                            198.51.100.4   0   0           0           1
IDNKS   5                           198.51.100.33  75   0           0           1
IDNUS   5                           198.51.100.35   0   0           0           2
IDNUS   5                           198.51.100.65   0   0           0           1
IDNUS   5                           198.51.100.67   0   0           0           2
IDNUS   5                           198.51.100.68   0   0           0           2
IDNUS   5                           198.51.100.98   0   0           0           2
IDNUS   5                          198.51.100.100   0   0           0           1
IDNUS   5                          198.51.100.129   0   0           0           2
IDNUS   5                          198.51.100.130   0   0           0           2
IDNUS   5                          198.51.100.161   0   0           0           2
IDNUS   5                          198.51.100.194   0   0           0           2
IDNUS   5                          198.51.100.196   0   0           0           1
IDNUS   5                          198.51.100.225   0   0           0           5
IDNUS   5                          198.51.100.226   0   0           0           2
IDNUS   5                          198.51.100.228   0   0           0           1
IDNUS   5                            198.51.100.3   0   0           0           2
IDNUS   5                           198.51.100.36   0   0           0           1
IDNUS   5                           198.51.100.64   0   0           0           4
IDNUS   5                           198.51.100.66   0   0           0           2
IDNUS   5                          198.51.100.130   0   0           0           2
IDNKS   5                           198.51.100.24   2   0           0           2
AESKS   5                           198.51.100.56   3   0           1           5
IDNKS   5                           198.51.100.99  18   0           0           2
AESKS   5                          198.51.100.100   7   0           1           3
IDNKS   5                          198.51.100.131  46   0           0           2
IDNKS   5                          198.51.100.132  18   0           0           2
IDNKS   5                          198.51.100.163  41   0           0           3
IDNKS   5                          198.51.100.164   6   0           0           2
AESKS   5                          198.51.100.184  11   0           1           2
IDNKS   5                          198.51.100.216   7   0           0           2
AESKS   5                            203.0.113.24   5   0           3           8
IDNKS   5                            203.0.113.56   3   1           0          13
IDNKS   5                            203.0.113.88   2   0           0          18
IDNKS   5                           203.0.113.120   6   0           0           3
IDNKS   5                           203.0.113.152   3   0           0           4
AESKS   5                           203.0.113.184   2   0           1           9
IDNKS   5                           203.0.113.216   4   0           0           2
IDNKS   5                           203.0.113.248  10   0           0           1
IDNUS   5                             203.0.113.4   0   0           0           2
IDNUS   5                            203.0.113.33   0   0           0           4
IDNUS   5                            203.0.113.65   0   0           0           2
IDNKS   5                           203.0.113.121   5   0           0           3
IDNKS   5                           203.0.113.153   6   0           0           5
IDNUS   5                           203.0.113.162   0   0           0           3
IDNUS   5                            203.0.113.32   0   0           0           5
AESKS   5                            203.0.113.35   5   1           6          37
IDNUS   5                            203.0.113.64   0   0           0           1
IDNUS   5                            203.0.113.66   0   0           0           4
IDNUS   5                            203.0.113.97   0   0           0           1
AESKS   5                            203.0.113.98   4   0           1           4
IDNUS   5                           203.0.113.161   0   0           0           4
IDNUS   5                           203.0.113.193   0   0           0          10
IDNUS   5                           203.0.113.228   0   0           0          12
IDNUS   5                           203.0.113.233   0   0           0           4
AESKS   5                               192.0.2.4   9   0           3          10
AESKS   5                              192.0.2.24   6   0          10          19
AESKS   5                              192.0.2.32   8   0          83         116
AESKS   5                              192.0.2.33  55   0          14          21
AESKS   5                              192.0.2.65  18   0           1        2747
AESKS   5                              192.0.2.66   4   0           3        1639
AESKS   5                              192.0.2.97  44  39          33        1826
AESKS   5                              192.0.2.98   5   1           6        2075
IDNKS   5                             192.0.2.195   8   0           0           7
AESKS   5                               192.0.2.3  46   0        3230        5346
AESKS   5                               192.0.2.5  12   0          84         202
AESKS   5                               192.0.2.7   2   0          31         443
AESKS   5                              192.0.2.24   3   0         116         117
AESKS   5                              192.0.2.36  15   0        4194        6889
AESKS   5                              192.0.2.44   7   0         112         331
AESKS   5                              192.0.2.68  34   0        6104        9798
AESKS   5                              192.0.2.69   7   0        2259        5027
AESKS   5                              192.0.2.88  17   0        3899        8880
AESKS   5                             192.0.2.164  16   0      833131     1298070
AESKS   5                             192.0.2.184   3   0     1437053     2153317
AESKS   5                             192.0.2.193  21   0       11364       26577
AESKS   5                             192.0.2.195   5   0          50         162
AESKS   5                             192.0.2.226  21   0        1950        3931
AESKS   5                             192.0.2.227   6   0        4348        9332
Note that running this command on an Inter-Chassis Service Redundancy (ICSR) standby node is only relevant for active peers, in which case the numbers should be close, for example in the following pair:
PGW> show egtpc peers ... 20:51:52  (ACTIVE) PGW-ICSR> show egtpc peers ... 20:51:51 (STANDBY)
Peer RC # sessions Max Sess Peer RC # sessions Max Sess
AESKS 192.0.2.5 2 31 443 ADSKS 192.0.2.5 2 31 443
AESKS 192.0.2.24 3 117 123 ADNKS 192.0.2.24 3 117 123
AESKS 192.0.2.36 15 4223 6889 ADSKS 192.0.2.36 15 4223 6889
AESKS 192.0.2.44 7 119 331 ADSKS 192.0.2.44 7 119 331
AESKS 192.0.2.68 34 6174 9798 ADSKS 192.0.2.68 34 6171 9797
AESKS 192.0.2.69 7 2311 5027 ADSKS 192.0.2.69 7 2311 5027
AESKS 192.0.2.88 17 3914 8880 ADSKS 192.0.2.88 17 3913 8874
AESKS 192.0.2.164 16 833705 1298070 ADSKS 192.0.2.164 16 832037 1294412
AESKS 192.0.2.184 3 1438418 2153317 ADSKS 192.0.2.184 3 1435461 2147280
AESKS 192.0.2.193 21 11447 26577 ADSKS 192.0.2.193 21 11443 26568
AESKS 192.0.2.195 5 48 162 ADSKS 192.0.2.195 5 48 162
AESKS 192.0.2.226 21 1962 3931 ADSKS 192.0.2.226 21 1962 3931
AESKS 192.0.2.227 6 4465 9332 ADSKS 192.0.2.227 6 4466 9332
"egtpc test echo" is used to check a specific peer to see if it is reachable or not and must be run in the context where the service is defined.
  • Use gtp-version 1 for GGSN-SGSN connections
  • Using ping command is not a valid test, though if it is successful, it is known that there is some level of reachability. 
  • The peer restart counter (Recovery) is displayed.
  • Running this command will increment the Tx echo request / Rx Echo response counters in "show egtpc statistics"
  • The restart counter of the peer is reported after Recovery keyword (11 in example below)
  • Warning: though this should not happen in normal circumstances, if a test echo is performed on a peer and it times out for whatever reason (if all is working then it shouldn't time out), all the calls would e dropped (even though things were working just fine up to the point of doing the test).
[Ingress]PGW> egtpc test echo gtp-version 2 src-address  209.165.201.1 peer-address 198.51.100.35 
Monday November 03 15:08:12 UTC 2014
EGTPC test echo
---------------
Peer: 198.51.100.35   Tx/Rx:  1/1  RTT(ms): 1    (COMPLETE) Recovery: 11 (0x0B)
"show egtpc statistics" is for EGTPC (v2) and reports on success/failures of many EGTPC control messages (Request/Response, Tx/Rx) including Create Session, Modify Bearer, Delete Session, Downlink Data Notify, Release Access Bearer, Create Bearer, Update Bearer, Modify Bearer, Suspend, etc., and the associated responses. It includes all path management counters for Echo Request/Response Tx/Rx, and based on the timers, the growth of these timers can be predicted and used to troubleshoot if not incrementing as expected.
The following example shows echos and responses being initiated and responded to from both sides. Note that the SGW has been configured to send once per minute while the PGW has been configured for once every three minutes (both sides don't need to have the same setting) and so every three minutes there are twice the number of packets (6:16:33, 6:19:33). Keep in mind that although both sides are on the same "cycle" in this example, they do not have to be and can run independent of eachother.  The restart counters has been captured in a column as well. Note the Restart counter of 11 from the SGW's echo response matches the value from its response to the PGW's test echo from above (the test was done at different time than the packet capture but for the same peer pair).
The demux-only option captures all the path management counters (echo requests/responses) as well as control messages that never reach a sessmgr, which is generally Create Session Request/Response. Demux process etpinmgr residing on the Demux card is responsible for handling all echo request/response and initial handling of all call control messages before distributing them to the appropriate sessmgrs (randomly for new calls, and specifically for existing calls anchored on a specific sessmgr). The sessmgr-only option conversely only counts messages that get handled by sessmgrs, which does NOT include path management counters. Not specifying either qualifiers is the default of counting all values.
"show gtpc statistics" is the analogous command for SGSN/GGSN services (GTPC v1) (CPC, Update PDP Context, etc.)
"show gtpu statistics" reports stats on the user-plane traffic handled by the GTP-U service(s) that are associated with EGTPC service(s). One interesting counter is Error Indication Tx/Rx which would be sent when the receiving node has no record of the subscriber that should be associated with the Terminal Endpoint Identifier (TEID) of the packet in question which could happen for a number of reasons (The bulkstats schema/variable is GTPU1/ err-ind-tx). If the MME/SGW has not been informed that the PGW no longer has a binding to the subscriber, then calls may need to be manually cleared on the MME/SGW (or wait for them to timeout) to fully resolve  such an issue of orphaned calls.
The peer specific versions of the above stat commands are necessary when troubleshooting specific peers, which often is the case.

Path Failure detection

The EGTPCPathFail trap is the key for knowing when a path failure has occurred. Amongst some basic values that one would already assume, it reports the number of calls dropped, the old and new restart counters, and the reason for the failure. In addition, the session disconnect reason gtpc-path-failure will be incremented. 
The EGTPCPathFailClear trap will indicate the restart counter along with the reason for the path clearing. The most recent StarOS v17 is better worded to say "clear reason" instead of "failure reason" (it cleared, not failed!) to avoid confusion.
The following are path failure counts that are reported from "show egtpc stat path-failure-reasons"
[local]PGW> show egtpc statistics path-failure-reasons
Reasons for path failure at EGTPC:
 Echo Request  restart counter change:                3  
 Echo Response restart counter change:                0   echo-rsp-restart-counter-change
 No Echo Response received:                           0   no-response-from-peer
 Control message restart counter change at   demux:  23   Create Session Req Restart Counter changed or create-sess-restart-counter-change
 Control message restart counter change at sessmgr:   0   cpc-restart-counter-change or upc-restart-counter-change
 Total path failures detected:                       26
Notes:
  • Control message restart counter at sessmgr could be any type of control message (not all listed here)
  • Changed restart counter from the SGSN via the next Create PDP Context Request after that change (cpc-restart-counter-change)
  • Changed restart counter from the SGSN via the next Update PDP Context Request after that change (which would be a handoff from another SGSN) (upc-restart-counter-change)
  • Path clear failure reason sessions-add is a deceiving term - it is not a failure but a success, and it indicates that a new session has come in to establish the connection
  • If the values reported for restart counters and session counts for the PathFail and PathClear traps do not seem to line up, open a ticket with Cisco for further explanation.
It is important to understand how EGTPC/GTPC and to a lesser degree EGTPU/GTPU (as user-plane failure detection is not always configured) path failure detection is implemented to monitor the connection between a GGSN or PGW and the various peer SGSN and SGWs. Normally the connections within a service providers network are fairly stable, in which case path failures are infrequent. On the flip side, experience has shown that GTPC paths amongst roaming SGSNs connecting to GGSNs may not be so stable, and so with poor connections the result is path failures happening at a fairly frequent basis.
The following config covers timers and transmission frequency for control and echo messages which can be used to detect path failures. Same concept applies to GGSN, SGW, MME, and EGTP services:
    ggsn-service GGSN1
retransmission-timeout 20
For GTP control messages (versus the echo messages)
max-retransmissions 5
for BOTH GTP control and echo messages – how many times to retransmit before path failure detection trigger
echo-interval 120
An echo is sent every 2 minutes (120 seconds).
echo-retransmission-timeout 5
For Echos, if no response, it retransmits every 5 seconds
path-failure detection-policy gtp echo
Path failure detection is possible via gtp echo (versus not using this as a method of detecting path failure (but still sending echos)).
associate gtpu-service GGSN1_gtpu
Associates the named GTPU service GGSN1_gtpu with this GGSN service
bind ipv4-address 209.165.201.2
The user and control plane bind IP addresses are the same in the following example configs, but may not necessarily be configured as such with other service provider implementations.
 
  [local]GGSN> show ggsn-service name GGSN1

  Service name:           GGSN1
  Context:                XGWin
  Bind:                   Done
  Local IPv4 Address:     209.165.201.2     Local IP Port: 2123
  Retransmission Timeout: 20 (secs)
  Max Retransmissions:    5                
  Restart Counter:        41      
  Echo Interval:          120 (secs)
  GTPC Echo Mode                        : Default
  GTPC Echo Retransmission Timeout      : 5
  Guard Interval:         100 (secs)
  Path Failure Detection on gtp msgs: Echo

 Notes:
  • Given time 0 being when an Echo request cycle starts (every 2 minutes), if there are no responses after a TOTAL of 30 seconds (6 attempts), the path fails (0 + 5sec + (5 * 5sec))
  • Process egtpinmgr (gtpcmgr deprecated since v14) running on the demux card is responsible for processing the echo packets
  • “show gtp statistics”, section “Path Management Messages” has statistics for the echo requests/responses for the control path in BOTH directions:
[Ingress]GGSN> show gtpc statistics
Monday October 27 23:02:07 UTC 2014
 
Path Management Messages:
  Echo Request RX:        169951248   Echo Response TX:        169951248
  Echo Request TX:         40838432   Echo Response RX:         40784820

Another method of path detection is on the user plane instead of the control plane, and this is governed by the following config.
    gtpu-service GGSN1_gtpu
      no echo-interval
      max-retransmissions 4
      echo-retransmission-timeout 5
      path-failure detection-policy gtp echo
      no path-failure clear-trap gtp echo
      bind ipv4-address 209.165.201.2
    exit

Notes:
  • “no echo-interval” disables sending of echo requests for user plane and therefore CLI “path-failure detection-policy gtp echo” has no relevance – NO Path failure detection is done on GTPU
  • GGSN still does respond to echo requests from the SGSN though 
  • “show gtpu stat” and “show demux-mgr stat gtpumgr” capture echo stats
[local]PGW-GGSN> show gtpu statistics
Wednesday October 29 23:43:22 UTC 2014
  Path Management Messages:
    Echo Request Rx:           84690303  Echo Response Rx:            0
    Echo Request Tx:                  0  Echo Response Tx:     84690043
 
[local]PGW-GGSN> show demux-mgr statistics gtpumgr
Wednesday October 29 23:42:04 UTC 2014
Echo stats:
   Echo Req Rcvd:               84690303
   Echo Rsp Sent:               84690043

Various Examples of path fail traps and explanations

Usually the peer restart counter will increase by one when there is a restart. In this case, a bug in PGW caused it to override the restart counter to the value of the RC of another peer (first trap), and so when the next packet arrived over the connection, it treated it as a restart (second trap) thinking that the RC had changed. The path then clears on any successive packet (third trap). The point here is not to know about a specific bug but rather to try and be aware of any counter changes that cannot be explained.
Wed Jan 14 05:22:07 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 198.51.100.1, peer old restart counter 54,  peer new restart counter 12,  peer session count  1107240, failure reason restart-counter-change

Wed Jan 14 05:22:07 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 198.51.100.1, peer old restart counter 12,  peer new restart counter 54,  peer session count  1107207, failure reason create-sess-restart-counter-change

Wed Jan 14 05:22:07 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 198.51.100.1, peer restart counter 54,   peer session count  1107199, failure reason sessions-add
This example is from a GGSN where the peer SGSN has changed its RC from 66 to 67 via a Create PDP Context Request:
Mon Oct 27 11:48:54 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.201.2,  peer address 209.165.200.225, peer old restart counter 66,  peer new restart counter 67,  peer session count  0, failure reason cpc-restart-counter-change
 
Mon Oct 27 11:48:54 2014 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.201.2,  peer address 209.165.200.225, peer restart counter 67,   peer session count  1, failure reason sessions-add
 
This example is from a PGW where the peer ePDG has changed its RC from 69 to 70 via a Create Session Request:
Fri Jun 12 05:21:54 2015 Internal trap notification 1112 (EGTPCPathFail) context PGWin, service EGTP1, interface type pgw-ingress, self address 2001:5555:8000:fff0:132:200::,  peer address 2001:5555:8000:ffe0:123:20f::, peer old restart counter 69,  peer new restart counter 70,  peer session count  0, failure reason create-sess-restart-counter-change

Fri Jun 12 05:21:55 2015 Internal trap notification 1113 (EGTPCPathFailClear) context PGWin, service EGTP1, interface type pgw-ingress, self address 2001:5555:8000:fff0:132:200::,  peer address 2001:5555:8000:ffe0:123:20f::, peer restart counter 70,   peer session count  1, clear reason sessions-add
 
In this example, the output from various PGW nodes all connected to the same SGW show that the SGW stopped becoming reachable from the PGWs per an EGTPCPathFail trap at the various PGWs within a short time frame of eachother. Note that all show the same old restart counter as was discussed earlier that normally a node uses the same counter for all its peers. In the case of no response, the new restart counter displayed is 9, but the new counter might better be printed as undefined since the connection is down as of when the trap is triggered and so having a restart counter doesn't really make sense.
Mon Dec 22 09:07:29 2014 Internal trap notification 1112 (EGTPCPathFail) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.4,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  151, failure reason no-response-from-peer
 
Mon Dec 22 09:08:54 2014 Internal trap notification 1112 (EGTPCPathFail) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.5,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  634, failure reason no-response-from-peer

Mon Dec 22 09:09:00 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.6,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  22, failure reason no-response-from-peer

Mon Dec 22 09:07:42 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.7,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  33, failure reason no-response-from-peer

Mon Dec 22 09:08:06 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.8,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  1, failure reason no-response-from-peer
 
Mon Dec 22 09:07:34 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.9,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  725, failure reason no-response-from-peer
 
Mon Dec 22 09:08:20 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.10,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  279978, failure reason no-response-from-peer
 
Mon Dec 22 09:06:37 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.11,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  23, failure reason no-response-from-peer
 
Mon Dec 22 09:07:29 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.12,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  6, failure reason no-response-from-peer
 
Mon Dec 22 09:07:01 2014 Internal trap notification 1112 (EGTPCPathFail) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.13,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  5, failure reason no-response-from-peer
The following example shows a path failure and same new restart counter as in the previous example, but then a half hour later shows when the peer comes back up again and the counter has increased by one from 15 to 16. Over time the count increases for that peer:
Wed Jul 22 05:07:08 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.13,  peer address 198.51.100.162, peer old restart counter 15,  peer new restart counter 15,  peer session count  34, failure reason no-response-from-peer

Wed Jul 22 05:38:00 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.13,  peer address 198.51.100.162, peer restart counter 16,   peer session count  1, clear reason sessions-update

[local]PGW> show egtpc peers
Wednesday July 22 10:04:30 UTC 2015

|||||  Service           Restart--------+   No. of
|||||  ID                Counter        |   restarts
|||||   |                               |    |   Current       Max
vvvvv   v              Peer Address     v    v   sessions    sessions
-----  --- --------------------------------------- --- --- -----------
AESKS   5               198.51.100.162  16   1        1157        2186

[XGWin]PGW> egtpc test echo gtp-version 2 src-address 209.165.201.13 peer-address 198.51.100.162  
Wednesday July 22 10:52:41 UTC 2015
EGTPC test echo
---------------
Peer: 198.51.100.162                          Tx/Rx:  1/1  RTT(ms): 51   (COMPLETE) Recovery: 16 (0x10)
This example shows a repeating path failure due to CPC restart for 3 specific SGSN peers to the GGSN. The output below shows just one of the peers 209.165.200.225, but they all have the same issue. Repeated run of "show egtpc peers" and "show gtpc stat sgsn-address" also shows the alternating values. "show gtpc stat" confirms no issue with connectivity with respect to echo requests and responses, which in this case are only being initiated by the GGSN (acceptable as discussed earlier). One sidebar point to be made here is that there are often multiple pieces of data that can be collected that all corroborate to the same conclusion - one can choose to collect the data that is the most conclusive with the minimal amount of effort. 
show snmp trap history verbose | grep -E "209.165.201.31|209.165.200.225|209.165.200.246"

Mon Jun 22 22:20:51 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer old restart counter 129,  peer new restart counter 132,  peer session count  3, failure reason cpc-restart-counter-change
Mon Jun 22 22:20:51 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer restart counter 132,   peer session count  4, clear reason sessions-add
Mon Jun 22 22:20:55 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer old restart counter 132,  peer new restart counter 129,  peer session count  3, failure reason cpc-restart-counter-change
Mon Jun 22 22:20:55 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer restart counter 129,   peer session count  4, clear reason sessions-add
Mon Jun 22 22:20:58 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer old restart counter 129,  peer new restart counter 132,  peer session count  3, failure reason cpc-restart-counter-change
Mon Jun 22 22:20:58 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer restart counter 132,   peer session count  4, clear reason sessions-add
Mon Jun 22 22:21:02 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer old restart counter 132,  peer new restart counter 129,  peer session count  2, failure reason cpc-restart-counter-change
Mon Jun 22 22:21:02 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer restart counter 129,   peer session count  3, clear reason sessions-add
Mon Jun 22 22:21:03 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer old restart counter 129,  peer new restart counter 132,  peer session count  4, failure reason echo-rsp-restart-counter-change
Mon Jun 22 22:21:03 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer restart counter 132,   peer session count  5, clear reason sessions-add
Mon Jun 22 22:21:06 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer old restart counter 132,  peer new restart counter 129,  peer session count  2, failure reason cpc-restart-counter-change
Mon Jun 22 22:21:06 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer restart counter 129,   peer session count  3, clear reason sessions-add

[XGWin]GGSN> show egtpc peers | grep 209.165.200.225
|||||  Service                      Restart No. of
|||||  ID                           Counter restarts
|||||   |                              |     |        Current      Max 
vvvvv   v              Peer Address    |     |       sessions    sessions
                                       |     |            |
                                       v     v            v
Monday June 22 23:09:26 UTC 2015
AESKG   6             209.165.200.225 129 86339           2          57

[XGWin]GGSN> show egtpc peers | grep 209.165.200.225
Monday June 22 23:09:28 UTC 2015
AESKG   6             209.165.200.225 129 86341           3          57

[XGWin]GGSN> show egtpc peers | grep 209.165.200.225
Monday June 22 23:09:29 UTC 2015
AESKG   6             209.165.200.225 132 86342           7          57 

[XGWin]GGSN> show egtpc peers | grep 209.165.200.225
Monday June 22 23:09:30 UTC 2015
AESKG   6             209.165.200.225 132 86344           9          57 

[XGWin]GGSN> show egtpc peers | grep 209.165.200.225
Monday June 22 23:09:31 UTC 2015
AESKG   6             209.165.200.225 129 86345          11          57   

[XGWin]GGSN> show egtpc peers | grep 209.165.200.225
Monday June 22 23:09:32 UTC 2015
AESKG   6             209.165.200.225 129 86345           7          57 

[XGWin]GGSN> show gtpc statistics sgsn-address 209.165.200.225
Monday June 22 23:07:36 UTC 2015
SGSN Address:    209.165.200.225    Status:          Active
Total Restarts:  86287             Restart Counter: 129

Session Stats:
  Total Current:                  2   S6b Assume Positive:             0
 ...

Path Management Messages:
  Echo Request RX:                0   Echo Response TX:                0
  Echo Request TX:            22919   Echo Response RX:            22917
Looking at other peers in the subnet, they all show a RC = 132 all the time, while the broken peers alternate between 129 and 132 as just shown, which points towards these SGSN peers as potentially passing the wrong RC in the broken scenarios:
                               Restart
                               Counter
AESKG   6        209.165.200.225 129 85115         8          57  <==
IDNKG   6        209.165.200.226 132  21           0           3
AESKG   6        209.165.200.227 132  60          47          92
AESKG   6        209.165.200.228 132  53          24          59
AESKG   6        209.165.200.229 132  69          92         126
AESKG   6        209.165.200.232 132  66          21          30
AESKG   6        209.165.200.236 132  43           8          21
AESKG   6        209.165.200.238 132  49          18          32
AESKG   6        209.165.200.239 132  39           1           6
AENKG   6        209.165.200.240 132  17           1           5
AESKG   6        209.165.200.241 132  67           6          43
AESKG   6        209.165.200.242 132  56          22          29
AENKG   6        209.165.200.243 132  40           2           9
AENKG   6        209.165.200.246 129 17713         3          24  <==
AESKG   6        209.165.200.247 132  64          29          45
AESKG   6        209.165.200.250 132  62          42          56
Recall from earlier, another approach to issues like this is to check other GGSNs (or PGWs for LTE) to see if they have the same issue with those seemingly problematic SGSN peers, and in this case they do also. For GGSN3 the first run of the command showed values 132 for both peers but another run showed 129 for one of the peers. Multiple runs (not shown here for brevity) would show all GGSNs bouncing between the two RCs for those peers over time:
[local]GGSN1> show egtpc peers | grep -E "209.165.200.246|209.165.200.225"
Tuesday June 23 11:37:23 UTC 2015
AENKG   7               209.165.200.246 129 20147           1          29
AESKG   7               209.165.200.225 129 92853           2          51

[local]GGSN2> show egtpc peers | grep -E "209.165.200.246|209.165.200.225"
Tuesday June 23 11:37:23 UTC 2015
AENKG   6               209.165.200.246 129 20450           4          17
AESKG   6               209.165.200.225 129 94021           3          54

[local]GGSN3> show egtpc peers | grep -E "209.165.200.246|209.165.200.225"
Tuesday June 23 11:37:26 UTC 2015
AENKG   6               209.165.200.246 132 20211           1          39
AESKG   6               209.165.200.225 132 91996           3          59

[local]GGSN4> show egtpc peers | grep -E "209.165.200.246|209.165.200.225"
Tuesday June 23 11:37:25 UTC 2015
AENKG   6               209.165.200.246 129 20309           1          24 
AESKG   6               209.165.200.225 132 92694           9          57 


[local]GGSN3> show egtpc peers | grep -E "209.165.200.246|209.165.200.225"
Tuesday June 23 11:44:35 UTC 2015
AENKG   6                209.165.200.246 129 20242           3          39
AESKG   6                209.165.200.225 132 92224           1          59
Finally to confirm the issue is the SGSN changing the RC, monitor subscriber menu option by SGSN peer shows changing RCs. When it does change,  all calls would be disconnected with path-failure as the disconnect reason. A screenshot of a packet capture also shows this:
[XGWin]GGSN> mon sub
Monday June 22 23:34:22 UTC 2015

      y) By SGSN IP Address

      IP Address: [209.165.200.225   ]


INBOUND>>>>>  23:35:32:688 Eventid:47000(3)
GTPC Rx PDU, from 209.165.200.225:2123 to 209.165.202.129:2123 (176)
TEID: 0x00000000, Message type: GTP_CREATE_PDP_CONTEXT_REQ_MSG (0x10)
Sequence Number:: 0x1DB8 (7608)
...
                       IMSI: 300420078559902
                   Recovery: 0x81 (129)
...

***CONTROL*** 23:36:00:363 Eventid:10285 
  Disconnect Reason: path-failure


INBOUND>>>>>  23:36:48:414 Eventid:47000(3)
GTPC Rx PDU, from 209.165.200.225:2123 to 209.165.202.129:2123 (176)
TEID: 0x00000000, Message type: GTP_CREATE_PDP_CONTEXT_REQ_MSG (0x10)
Sequence Number:: 0x3E2D (15917)
...
                       IMSI: 300420125984926
                   Recovery: 0x84 (132)
...

INBOUND>>>>>  23:37:28:337 Eventid:47000(3)
GTPC Rx PDU, from 209.165.200.225:2123 to 209.165.202.129:2123 (176)
TEID: 0x00000000, Message type: GTP_CREATE_PDP_CONTEXT_REQ_MSG (0x10)
Sequence Number:: 0x3517 (13591)
...
                       IMSI: 300420094205377
                   Recovery: 0x84 (132)
...

Monday June 22 2015
INBOUND>>>>>  23:37:40:559 Eventid:47000(3)
GTPC Rx PDU, from 209.165.200.225:2123 to 209.165.202.129:2123 (176)
TEID: 0x00000000, Message type: GTP_CREATE_PDP_CONTEXT_REQ_MSG (0x10)
Sequence Number:: 0x4E47 (20039)
...
                       IMSI: 300420194755472
                   Recovery: 0x81 (129)
...

***CONTROL*** 23:37:40:755 Eventid:10285 
  Disconnect Reason: path-failure
This example shows a path failure where the problem was actually in the path and not the PGW or SGW. As a result, the RC remains the same when path re-establishes. In the second peer example the clear is a result of echo request establishment:
Thu May 28 20:47:47 2015 Internal trap notification 1112 (EGTPCPathFail) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 198.51.100.196, peer old restart counter 5,  peer new restart counter 5,  peer session count  549697, failure reason no-response-from-peer

Thu May 28 20:49:36 2015 Internal trap notification 1113 (EGTPCPathFailClear) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 198.51.100.196, peer restart counter 5,   peer session count  1, clear reason sessions-add

Thu May 28 20:49:06 2015 Internal trap notification 1112 (EGTPCPathFail) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 203.0.113.98, peer old restart counter 4,  peer new restart counter 4,  peer session count  3, failure reason no-response-from-peer
 
Thu May 28 20:50:49 2015 Internal trap notification 1113 (EGTPCPathFailClear) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 203.0.113.98, peer restart counter 4,   peer session count  0, clear reason echo-req
 
This example shows another anomaly where repeatedly after the old restart counter has reset to 2, it somehow ends up at 0 again when the path fail clears. This turned out to be a bug. Again the lesson here is if something appears odd, it should be reported.
Fri Mar 27 03:13:36 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 192.0.2.5, peer old restart counter 0,  peer new restart counter 2,  peer session count  1, failure reason create-sess-restart-counter-change

Fri Mar 27 03:13:37 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 192.0.2.5, peer restart counter 0,   peer session count  1, failure reason sessions-add

Sat Mar 28 02:20:45 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 192.0.2.5, peer old restart counter 0,  peer new restart counter 2,  peer session count  1, failure reason create-sess-restart-counter-change

Sat Mar 28 02:20:47 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 192.0.2.5, peer restart counter 0,   peer session count  1, failure reason sessions-add

Sat Mar 28 17:14:06 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 192.0.2.5, peer old restart counter 0,  peer new restart counter 2,  peer session count  1, failure reason create-sess-restart-counter-change

Sat Mar 28 17:14:06 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 192.0.2.5, peer restart counter 0,   peer session count  1, failure reason sessions-add
TAC Authored

Contributed by Cisco Engineers

  • Dave Damerjian
    Cisco Technical Services

Was this Document Helpful?

FeedbackFeedback

Contact Cisco

This Document Applies to These Products