Troubleshoot of Gx Success Rate KPI Degradation Due to an Increase in ims-authorization-failed Disconnect Reason
Available Languages
Download Options
Bias-Free Language
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
Contents
Introduction
This document describes how to troubleshoot the problem that occurs when the Gx success rate Key Performance Indicator (KPI) degradation takes place when the 'ims-authorization-failed' disconnect reason is increased.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- Hardware knowledge of 5000/5500
- StarOS
Components Used
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment.
All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
Abbreviations:
| PCRF | Policy and Charging Rules Function |
| PGW | Packet Data Network Gateway |
| CCR | Credit Control Request |
| CCA | Credit Control Answer |
| CFC | Control Function Virtual Card |
| FC | Service Function Virtual Card |
| KPI | Key Performance Indicator |
Problem
The service provider reported Gx success rate KPI Degradation in one PGW and the disconnect reason 'ims-authorization-failed' counter was found to be increasing.
Here, the 'ims-authorization-failed' disconnect reason describes the total number of sessions disconnected because of IMS authorization failures.
PGW constantly reported 'ims-authorization-failed' counter. As per the disconnect-reasons statistics, the 'ims-authorization-failed' counter which contributed to the degradation of Gx success rate.
1st iteration-
[local]PGW# show session disconnect-reasons verbose | grep -i ims-authorization-failed
Thursday June 13 16:10:39 NZST 2019
..
Disconnect Reason Num Disc Percentage
----------------------------------------------------- ---------- ----------
ims-authorization-failed 59034 1.68225
2nd iteration-
[local]PGW# show session disconnect-reasons verbose | grep -i ims-authorization-failed
Thursday June 13 16:10:59 NZST 2019
..
Disconnect Reason Num Disc Percentage
----------------------------------------------------- ---------- ----------
ims-authorization-failed 59202 1.68181 >>168 counter increased in just 20 sec of span
Troubleshoot
This section provides information to troubleshoot the problem of Gx success rate KPI degradation due to an increase in 'ims-authorization-failed' disconnect reason.
A monitor subscriber trace has been collected for the failure scenario.
From the analysis, it was found that 'ims-authorization-failed' was observed for those sessmgr instances which were associated with card 7. Overall, the ims-authorization-failed reason was received in disconnect reason.
Refer to the failure subscriber trace sequence:
The CREATE_SESSION_REQUEST message was sent from SGW to PGW.
INBOUND>>>>> From sessmgr:102 sessmgr_pgw.c:31631 (Callid 1e283e66) 16:55:07:562 Eventid:141004(3)
[PGW-S5/S2a/S2b]GTPv2C Rx PDU, from 202.73.X.Y:31552 to 202.73.A.B:2123 (262)
TEID: 0x00000000, Message type: EGTP_CREATE_SESSION_REQUEST (0x20)
Sequence Number: 0x18A861 (1615969)
GTP HEADER
Version number: 2
TEID flag: Present
Piggybacking flag: Not present
Message Length: 0x0102 (258)
INFORMATION ELEMENTS
IMSI:
Type: 1 Length: 8 Inst: 0
Value: ABXY01106562064
USER LOCATION INFO:
Type: 86 Length: 13 Inst: 0
Value:
Location type: TAI
MCC: 530
MNC: 01
TAC: 0xA101
Location type: ECGI
MCC: AB0
MNC: XY
ECI: 0x006F902
SERVING NETWORK:
Type: 83 Length: 3 Inst: 0
Value:
MCC: AB0
MNC: XY
RADIO ACCESS TECH:
Type: 82 Length: 1 Inst: 0
Value: EUTRAN (6)
SGW-CONTROL FTEID:
Type: 87 Length: 9 Inst: 0
Value:
Interface: SGW S5/S8-C
IPv4 Flag: 1
IPv6 Flag: 0
Teid: 0x84B8A061
IPV4 Addr: 202.73.X.Y
ACCESS POINT NAME:
Type: 71 Length: 28 Inst: 0
Value: internet.mnc0XY.mcc0AB.gprs
SELECTION MODE:
Type: 128 Length: 1 Inst: 0
Value: MS or network provided APN,subscr verified (0x00)
PDN TYPE:
Type: 99 Length: 1 Inst: 0
Value: IPV4
PDN ADDRESS ALLOC:
Type: 79 Length: 5 Inst: 0
Value:
PDN Type: IPV4
IPV4 Addr: 0.0.0.0
INDICATION:
Type: 77 Length: 6 Inst: 0
Value:
DAF : 0
DTF : 0(Direct tunnel disabled)
HI : 0
DFI : 0
OI : 0
ISRSI: 0(ISR not supported)
ISRAI: 0(ISR not Activated)
SGWCI: 0
SQCI : 0
UIMSI: 0(Authenticated IMSI)
CFSI : 0(F-TEID Change Indication not supported)
CRSI : 0(Location Change Reporting not supported)
PS : 1(Piggybacking feature supported)
PT : 0(GTPv2)
SI : 0
MSV : 0
RetLoc: 0
PBIC: 0
SRNI: 0
S6AF: 0
S4AF: 0
MBMDT: 0
ISRAU: 0
CCRSI: 0(CSG Information Change Reporting mechanism not supported )
CPRAI: 0
ARRL: 0
PPOF: 0
PPON / PPEI: 0
PPSI: 0
CSFBI: 0
CLII: 0
CPSR: 0
UASI: 0
DTCI: 0
BDWI: 0
PCRI: 0
AOSI: 0
AOPI: 0
EPCOSI: 0
CPOPCI: 0
S11TF: 0
PNSI: 0
WPMSI: 0
APN RESTRICTION:
Type: 127 Length: 1 Inst: 0
Value: 0
AGGREGATE MAX BIT RATE:
Type: 72 Length: 8 Inst: 0
Value:
Uplk AMBR: 64000 kbps
Dnlk AMBR: 256000 kbps
PCO:
Type: 78 Length: 32 Inst: 0
Container id: 0x8021 (IPCP)
Container length: 0x10 (16)
Container content:
Conf-Req(0), Pri-DNS=0.0.0.0, Sec-DNS=0.0.0.0
Container id: 0x000D (IPv4-DNS-Server)
Container length: 0x00 (0)
Container content:
DNS Address: Request for IPv4 DNS Address allocation
Container id: 0x000A (IP address allocation via NAS signalling)
Container length: 0x00 (0)
Container content:
IP Address allocation via NAS signalling
Container id: 0x0005 (Bearer Control Mode)
Container length: 0x00 (0)
Container content:
NCQOS BCM Indicator:
Container id: 0x0010 (Link MTU)
Container length: 0x00 (0)
Container content:
BEARER CONTEXT TO BE CREATED:
Type: 93 Length: 44 Inst: 0
Value:
EPS BEARER ID:
Type: 73 Length: 1 Inst: 0
Value: 5
SGW-DATA FTEID:
Type: 87 Length: 9 Inst: 2
Value:
Interface: SGW S5/S8-U
IPv4 Flag: 1
IPv6 Flag: 0
Teid: 0x8A948061
IPV4 Addr: 202.73.X.Y
BEARER QOS:
Type: 80 Length: 22 Inst: 0
Value:
ARP: PCI = 1 (Disabled), PL = 9, PVI = 0 (Enabled)
QCI: 6
Uplnk MBR: 0 kbps
Dnlnk MBR: 0 kbps
Uplnk GBR: 0 kbps
Dnlnk GBR: 0 kbps
UE TIME ZONE:
Type: 114 Length: 2 Inst: 0
Value:
TZ: +12:00
DST: +0 hour
CHARGING CHAR:
Type: 95 Length: 2 Inst: 0
Value: 0x0800
<additional output suprresed>
PGW sent 'No resource available' to SGW in EGTP_CREATE_SESSION_RESPONSE message.
<<<<OUTBOUND From sessmgr:102 sessmgr_egtp.c:2906 (Callid 1e283e66) 16:55:15:492 Eventid:141005(3)
[PGW-S5/S2a/S2b]GTPv2C Tx PDU, from 202.73.195.91:2123 to 202.73.195.89:31552 (33)
TEID: 0x84B8A061, Message type: EGTP_CREATE_SESSION_RESPONSE (0x21)
Sequence Number: 0x18A861 (1615969)
GTP HEADER
Version number: 2
TEID flag: Present
Piggybacking flag: Not present
Message Length: 0x001D (29)
INFORMATION ELEMENTS
CAUSE:
Type: 2 Length: 2 Inst: 0
Value:
Cause: EGTP_CAUSE_NO_RESOURCES_AVAILABLE (0x49)
PCE: 0
BCE: 0
CS: 0
BEARER CONTEXT CREATED:
Type: 93 Length: 11 Inst: 0
Value:
EPS BEARER ID:
Type: 73 Length: 1 Inst: 0
Value: 5
CAUSE:
Type: 2 Length: 2 Inst: 0
Value:
Cause: EGTP_CAUSE_NO_RESOURCES_AVAILABLE (0x49)
PCE: 0
BCE: 0
CS: 0
Overall, ims-authorization-failed reason was received in disconnect reason.
***CONTROL*** From sessmgr:102 sessmgr_func.c:5311 (Callid 1e283e66) 16:55:15:493 Eventid:10285 >>disconnect received for sessmr 102
CALL STATS: <xyz12591615@internet>, msid <ABXY01106562064>, Call-Duration(sec): 0
input pkts: 0 output pkts: 0
input bytes: 0 output bytes: 0
input bytes dropped: 0 output bytes dropped: 0
input pkts dropped: 0 output pkts dropped: 0
pk rate from user(bps): 0 pk rate to user(bps): 0
ave rate from user(bps): 0 ave rate to user(bps): 0
sust rate from user(bps): 0 sust rate to user(bps): 0
pk rate from user(pps): 0 pk rate to user(pps): 0
ave rate from user(pps): 0 ave rate to user(pps): 0
sust rate from user(pps): 0 sust rate to user(pps): 0
link online/active percent: 0
ipv4 bad hdr: 0 ipv4 ttl exceeded: 0
ipv4 fragments sent: 0 ipv4 could not fragment: 0
ipv4 input acl drop: 0 ipv4 output acl drop: 0
ipv4 bad length trim: 0
ipv4 input non-mip drop: 0 ipv4 output non-mip drop: 0
ipv4 input css drop: 0 ipv4 output css drop: 0
output gre xoff pkts drop: 0 output gre xoff bytes drop: 0
ipv4 output no-flow drop: 0
ipv4 source violations: 0 ipv4 early pdu drop: 0
ipv4 proxy-dns redirect: 0 ipv4 proxy-dns pass-thru: 0
ipv4 proxy-dns drop: 0 ipv4 proxy-dns redirect tcp connection: 0
ipv6 bad hdr: 0 ipv6 bad length trim: 0
ip source violation no acct: 0 ip source violation ignored: 0
dlnk pkts exceeded bw: 0 dlnk pkts violated bw: 0
uplnk pkts exceeded bw: 0 uplnk pkts violated bw: 0
dormancy total: 0 handoff total: 0
Disconnect Reason: ims-authorization-failed
Last Progress State: IMS Authorizing
Furthermore, sessmgr instance number is 102 that is associated with card 7 for which ims-authorization-failed counter is received.
[local]PGW# show task resources facility sessmgr instance 102
Thursday June 13 16:56:14 NZST 2019
task cputime memory files sessions
cpu facility inst used allc used alloc used allc used allc S status
----------------------- ----------- ------------- --------- ------------- ------
7/0 sessmgr 102 8.80% 100% 792.2M 2.34G 45 500 3097 32504 I good
Total 1 8.80% 792.2M 45 3097
When the configuration was checked, it was seen that the multiple diamproxy configurations were fine.
[local]PGW# show config | grep multiple
Thursday June 13 15:15:11 NZST 2019
require diameter-proxy multiple
It was then found that the 'ims-authorization-failed' was observed for the card in which diamproxy facility did not spawn up.
In this case, diamproxy facility did not spawn with card 7.
[local]PGW# show card table
Thursday June 13 14:53:53 NZST 2019
Slot Card Type Oper State SPOF Attach
----------- -------------------------------------- ------------- ---- ------
1: CFC Control Function Virtual Card Standby -
2: CFC Control Function Virtual Card Active No
3: FC 2-Port Service Function Virtual Card Active No
4: FC 2-Port Service Function Virtual Card Active No
5: FC 2-Port Service Function Virtual Card Active No
6: FC 2-Port Service Function Virtual Card Active No
7: FC 2-Port Service Function Virtual Card Active No
8: FC 2-Port Service Function Virtual Card Active No
9: FC 2-Port Service Function Virtual Card Active No
10: FC 2-Port Service Function Virtual Card Active No
11: FC 2-Port Service Function Virtual Card Active No
12: FC 2-Port Service Function Virtual Card Active No
13: FC 2-Port Service Function Virtual Card Active No
14: FC 2-Port Service Function Virtual Card Active No
15: FC 2-Port Service Function Virtual Card Standby -
16: FC 2-Port Service Function Virtual Card Active No
------- -
[local]PGW# show task resources facility diamproxy all
Thursday June 13 14:55:31 NZST 2019
task cputime memory files sessions
cpu facility inst used allc used alloc used allc used allc S status
----------------------- ----------- ------------- --------- ------------- ------
3/0 diamproxy 8 1.10% 90% 43.75M 250.0M 271 2500 -- -- - good
4/0 diamproxy 9 1.09% 90% 43.89M 250.0M 266 2500 -- -- - good
5/0 diamproxy 10 1.09% 90% 43.93M 250.0M 237 2500 -- -- - good
9/0 diamproxy 1 1.08% 90% 43.63M 250.0M 212 2500 -- -- - good
10/0 diamproxy 4 1.08% 90% 43.88M 250.0M 258 2500 -- -- - good
12/0 diamproxy 11 1.11% 90% 43.93M 250.0M 229 2500 -- -- - good
13/0 diamproxy 2 1.09% 90% 43.75M 250.0M 258 2500 -- -- - good
14/0 diamproxy 7 1.06% 90% 43.88M 250.0M 256 2500 -- -- - good
16/0 diamproxy 13 1.04% 90% 43.89M 250.0M 199 2500 -- -- - good
Solution
It is found that diamproxy facility was not spawned for card 7 although card 7 was active. In order to spawn diamproxy facility, diamctrl facility needs to be restarted or CF card switchover needs to be performed.
This action plan was implemented as suggested.
Switchover active CFC card with standby CFC card.
After the card switchover, respective diamproxy were spawned properly and Gx KPI was back to normal as per the expected values.
******** show card table all *******
Thursday June 13 22:17:54 NZST 2019<tel:2019>
Slot Card Type Oper State SPOF Attach
----------- -------------------------------------- ------------- ---- ------
1: CFC Control Function Virtual Card Active No >>CFC card 1 active post switchover
2: CFC Control Function Virtual Card Standby -
3: FC 2-Port Service Function Virtual Card Active No
4: FC 2-Port Service Function Virtual Card Active No
5: FC 2-Port Service Function Virtual Card Active No
6: FC 2-Port Service Function Virtual Card Active No
7: FC 2-Port Service Function Virtual Card Active No
8: FC 2-Port Service Function Virtual Card Active No
9: FC 2-Port Service Function Virtual Card Active No
10: FC 2-Port Service Function Virtual Card Active No
11: FC 2-Port Service Function Virtual Card Active No
12: FC 2-Port Service Function Virtual Card Active No
13: FC 2-Port Service Function Virtual Card Active No
14: FC 2-Port Service Function Virtual Card Active No
15: FC 2-Port Service Function Virtual Card Standby -
16: FC 2-Port Service Function Virtual Card Active No
--------- -
[local]PGW# show task resources facility diamproxy all
Thursday June 13 14:55:31 NZST 2019
task cputime memory files sessions
cpu facility inst used allc used alloc used allc used allc S status
----------------------- ----------- ------------- --------- ------------- ------
3/0 diamproxy 8 0.95% 90% 43.86M 250.0M 130 2500 -- -- - good
4/0 diamproxy 9 0.98% 90% 44.00M 250.0M 130 2500 -- -- - good
5/0 diamproxy 10 1.00% 90% 44.00M 250.0M 135 2500 -- -- - good
7/0 diamproxy 3 0.26% 90% 43.73M 250.0M 278 2500 -- -- - good >>diamproxy instance 3 spawn with card 7
8/0 diamproxy 5 0.31% 90% 43.73M 250.0M 277 2500 -- -- - good
9/0 diamproxy 1 1.00% 90% 43.86M 250.0M 130 2500 -- -- - good
10/0 diamproxy 4 0.98% 90% 43.96M 250.0M 133 2500 -- -- - good
11/0 diamproxy 6 0.24% 90% 43.74M 250.0M 277 2500 -- -- - good
12/0 diamproxy 11 1.04% 90% 44.01M 250.0M 144 2500 -- -- - good
13/0 diamproxy 2 0.97% 90% 43.84M 250.0M 134 2500 -- -- - good
14/0 diamproxy 7 1.04% 90% 43.99M 250.0M 138 2500 -- -- - good
16/0 diamproxy 13 0.96% 90% 43.74M 250.0M 261 2500 -- -- - good
Respective diamproxy facility were spawned after the activity and diameter messages were then exchanged properly between PGW and PCRF.
Contributed by Cisco Engineers
- Tushar PatelCisco TAC Engineer
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)