Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Problem

Troubleshoot

Solution

Introduction

This document describes how to troubleshoot the problem that occurs when the Gx success rate Key Performance Indicator (KPI) degradation takes place when the 'ims-authorization-failed' disconnect reason is increased.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Hardware knowledge of 5000/5500

- StarOS

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment.

All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Abbreviations:

PCRF

Policy and Charging Rules Function

PGW

Packet Data Network Gateway

CCR

Credit Control Request

CCA

Credit Control Answer

CFC

Control Function Virtual Card

FC

Service Function Virtual Card

KPI

Key Performance Indicator

Problem

The service provider reported Gx success rate KPI Degradation in one PGW and the disconnect reason 'ims-authorization-failed' counter was found to be increasing.

Here, the 'ims-authorization-failed' disconnect reason describes the total number of sessions disconnected because of IMS authorization failures.

PGW constantly reported 'ims-authorization-failed' counter. As per the disconnect-reasons statistics, the 'ims-authorization-failed' counter which contributed to the degradation of Gx success rate.

1st iteration-

[local]PGW# show session disconnect-reasons verbose | grep -i ims-authorization-failed

Thursday June 13 16:10:39 NZST 2019

..

Disconnect Reason                                       Num Disc    Percentage

-----------------------------------------------------  ----------  ----------

ims-authorization-failed                                    59034     1.68225

2nd iteration-

[local]PGW# show session disconnect-reasons verbose | grep -i ims-authorization-failed

Thursday June 13 16:10:59 NZST 2019

..

Disconnect Reason                                        Num Disc  Percentage

-----------------------------------------------------  ----------  ----------

ims-authorization-failed                                    59202     1.68181         >>168 counter increased in just 20 sec of span

Troubleshoot

This section provides information to troubleshoot the problem of Gx success rate KPI degradation due to an increase in 'ims-authorization-failed' disconnect reason.

A monitor subscriber trace has been collected for the failure scenario.

From the analysis, it was found that 'ims-authorization-failed' was observed for those sessmgr instances which were associated with card 7. Overall, the ims-authorization-failed reason was received in disconnect reason.

Refer to the failure subscriber trace sequence:

The CREATE_SESSION_REQUEST message was sent from SGW to PGW.

INBOUND>>>>>  From sessmgr:102 sessmgr_pgw.c:31631 (Callid 1e283e66) 16:55:07:562 Eventid:141004(3)

[PGW-S5/S2a/S2b]GTPv2C Rx PDU, from 202.73.X.Y:31552 to 202.73.A.B:2123 (262)

TEID: 0x00000000, Message type: EGTP_CREATE_SESSION_REQUEST (0x20)

Sequence Number: 0x18A861 (1615969)

GTP HEADER

Version number: 2

TEID flag: Present

Piggybacking flag: Not present

Message Length: 0x0102 (258)

INFORMATION ELEMENTS

IMSI:

Type: 1  Length: 8  Inst: 0

Value: ABXY01106562064

USER LOCATION INFO:

Type: 86  Length: 13  Inst: 0

Value:

Location type: TAI

MCC: 530

MNC: 01

TAC: 0xA101

Location type: ECGI

MCC: AB0

MNC: XY

ECI: 0x006F902

SERVING NETWORK:

Type: 83  Length: 3  Inst: 0

Value:

MCC: AB0

MNC: XY

RADIO ACCESS TECH:

Type: 82  Length: 1  Inst: 0

Value: EUTRAN (6)

SGW-CONTROL FTEID:

Type: 87  Length: 9  Inst: 0

Value:

Interface: SGW S5/S8-C

IPv4 Flag: 1

IPv6 Flag: 0

Teid: 0x84B8A061

IPV4 Addr: 202.73.X.Y

ACCESS POINT NAME:

Type: 71  Length: 28  Inst: 0

Value: internet.mnc0XY.mcc0AB.gprs

SELECTION MODE:

Type: 128  Length: 1  Inst: 0

Value: MS or network provided APN,subscr verified (0x00)

PDN TYPE:

Type: 99  Length: 1  Inst: 0

Value: IPV4

PDN ADDRESS ALLOC:

Type: 79  Length: 5  Inst: 0

Value:

PDN Type: IPV4

IPV4 Addr: 0.0.0.0

INDICATION:

Type: 77  Length: 6  Inst: 0

Value:

DAF  : 0

DTF  : 0(Direct tunnel disabled)

HI   : 0

DFI  : 0

OI   : 0

ISRSI: 0(ISR not supported)

ISRAI: 0(ISR not Activated)

SGWCI: 0

SQCI : 0

UIMSI: 0(Authenticated IMSI)

CFSI : 0(F-TEID Change Indication not supported)

CRSI : 0(Location Change Reporting not supported)

PS   : 1(Piggybacking feature supported)

PT   : 0(GTPv2)

SI   : 0

MSV  : 0

RetLoc: 0

PBIC: 0

SRNI: 0

S6AF: 0

S4AF: 0

MBMDT: 0

ISRAU: 0

CCRSI: 0(CSG Information Change Reporting mechanism not supported )

CPRAI: 0

ARRL: 0

PPOF: 0

PPON / PPEI: 0

PPSI: 0

CSFBI: 0

CLII: 0

CPSR: 0

UASI: 0

DTCI: 0

BDWI: 0

PCRI: 0

AOSI: 0

AOPI: 0

EPCOSI: 0

CPOPCI: 0

S11TF: 0

PNSI: 0

WPMSI: 0

APN RESTRICTION:

Type: 127  Length: 1  Inst: 0

Value: 0

AGGREGATE MAX BIT RATE:

Type: 72  Length: 8  Inst: 0

Value:

Uplk AMBR: 64000 kbps

Dnlk AMBR: 256000 kbps

PCO:

Type: 78  Length: 32  Inst: 0

Container id: 0x8021 (IPCP)

Container length: 0x10 (16)

Container content:

Conf-Req(0), Pri-DNS=0.0.0.0, Sec-DNS=0.0.0.0

Container id: 0x000D (IPv4-DNS-Server)

Container length: 0x00 (0)

Container content:

DNS Address: Request for IPv4 DNS Address allocation

Container id: 0x000A (IP address allocation via NAS signalling)

Container length: 0x00 (0)

Container content:

IP Address allocation via NAS signalling

Container id: 0x0005 (Bearer Control Mode)

Container length: 0x00 (0)

Container content:

NCQOS BCM Indicator:

Container id: 0x0010 (Link MTU)

Container length: 0x00 (0)

Container content:

BEARER CONTEXT TO BE CREATED:

Type: 93  Length: 44  Inst: 0

Value:

EPS BEARER ID:

Type: 73  Length: 1  Inst: 0

Value: 5

SGW-DATA FTEID:

Type: 87  Length: 9  Inst: 2

Value:

Interface: SGW S5/S8-U

IPv4 Flag: 1

IPv6 Flag: 0

Teid: 0x8A948061

IPV4 Addr: 202.73.X.Y

BEARER QOS:

Type: 80  Length: 22  Inst: 0

Value:

ARP: PCI = 1 (Disabled), PL = 9, PVI = 0 (Enabled)

QCI: 6

Uplnk MBR: 0 kbps

Dnlnk MBR: 0 kbps

Uplnk GBR: 0 kbps

Dnlnk GBR: 0 kbps

UE TIME ZONE:

Type: 114  Length: 2  Inst: 0

Value:

TZ: +12:00

DST: +0 hour

CHARGING CHAR:

Type: 95  Length: 2  Inst: 0

Value: 0x0800

<additional output suprresed>

PGW sent 'No resource available' to SGW in EGTP_CREATE_SESSION_RESPONSE message.

<<<<OUTBOUND  From sessmgr:102 sessmgr_egtp.c:2906 (Callid 1e283e66) 16:55:15:492 Eventid:141005(3)

[PGW-S5/S2a/S2b]GTPv2C Tx PDU, from 202.73.195.91:2123 to 202.73.195.89:31552 (33)

TEID: 0x84B8A061, Message type: EGTP_CREATE_SESSION_RESPONSE (0x21)

Sequence Number: 0x18A861 (1615969)

GTP HEADER

Version number: 2

TEID flag: Present

Piggybacking flag: Not present

Message Length: 0x001D (29)

INFORMATION ELEMENTS

CAUSE:

Type: 2  Length: 2  Inst: 0

Value:

Cause: EGTP_CAUSE_NO_RESOURCES_AVAILABLE (0x49)

PCE: 0

BCE: 0

CS: 0

BEARER CONTEXT CREATED:

Type: 93  Length: 11  Inst: 0

Value:

EPS BEARER ID:

Type: 73  Length: 1  Inst: 0

Value: 5

CAUSE:

Type: 2  Length: 2  Inst: 0

Value:

Cause: EGTP_CAUSE_NO_RESOURCES_AVAILABLE (0x49)

PCE: 0

BCE: 0

CS: 0

Overall, ims-authorization-failed reason was received in disconnect reason.

***CONTROL*** From sessmgr:102 sessmgr_func.c:5311 (Callid 1e283e66) 16:55:15:493 Eventid:10285   >>disconnect received for sessmr 102

CALL STATS: <xyz12591615@internet>, msid <ABXY01106562064>, Call-Duration(sec): 0

input pkts: 0                          output pkts: 0

input bytes: 0                         output bytes: 0

input bytes dropped: 0                 output bytes dropped: 0

input pkts dropped: 0                  output pkts dropped: 0

pk rate from user(bps): 0              pk rate to user(bps): 0

ave rate from user(bps): 0             ave rate to user(bps): 0

sust rate from user(bps): 0            sust rate to user(bps): 0

pk rate from user(pps): 0              pk rate to user(pps): 0

ave rate from user(pps): 0             ave rate to user(pps): 0

sust rate from user(pps): 0            sust rate to user(pps): 0

link online/active percent: 0

ipv4 bad hdr: 0                        ipv4 ttl exceeded: 0

ipv4 fragments sent: 0                 ipv4 could not fragment: 0

ipv4 input acl drop: 0                 ipv4 output acl drop: 0

ipv4 bad length trim: 0

ipv4 input non-mip drop: 0             ipv4 output non-mip drop: 0

ipv4 input css drop: 0                 ipv4 output css drop: 0

output gre xoff pkts drop: 0           output gre xoff bytes drop: 0

ipv4 output no-flow drop: 0

ipv4 source violations: 0              ipv4 early pdu drop: 0

ipv4 proxy-dns redirect: 0             ipv4 proxy-dns pass-thru: 0

ipv4 proxy-dns drop: 0                 ipv4 proxy-dns redirect tcp connection: 0

ipv6 bad hdr: 0                        ipv6 bad length trim: 0

ip source violation no acct: 0          ip source violation ignored: 0

dlnk pkts exceeded bw: 0               dlnk pkts violated bw: 0

uplnk pkts exceeded bw: 0              uplnk pkts violated bw: 0

dormancy total: 0                      handoff total: 0

Disconnect Reason: ims-authorization-failed

Last Progress State: IMS Authorizing

Furthermore, sessmgr instance number is 102 that is associated with card 7 for which ims-authorization-failed counter is received.

[local]PGW# show task resources facility sessmgr instance 102

Thursday June 13 16:56:14 NZST 2019

task   cputime        memory     files      sessions

cpu facility      inst  used  allc   used  alloc used allc  used  allc S status

----------------------- ----------- ------------- --------- ------------- ------

7/0 sessmgr        102 8.80%  100% 792.2M  2.34G   45  500  3097 32504 I   good

Total                 1  8.80%     792.2M            45         3097

When the configuration was checked, it was seen that the multiple diamproxy configurations were fine.

[local]PGW# show config | grep multiple

Thursday June 13 15:15:11 NZST 2019

require diameter-proxy multiple

It was then found that the 'ims-authorization-failed' was observed for the card in which diamproxy facility did not spawn up.

In this case, diamproxy facility did not spawn with card 7.

[local]PGW# show card table

Thursday June 13 14:53:53 NZST 2019

Slot         Card Type                               Oper State     SPOF  Attach

-----------  --------------------------------------  -------------  ----  ------

1: CFC      Control Function Virtual Card           Standby        -

2: CFC      Control Function Virtual Card           Active         No

3: FC       2-Port Service Function Virtual Card    Active         No

4: FC       2-Port Service Function Virtual Card    Active         No

5: FC       2-Port Service Function Virtual Card    Active         No

6: FC       2-Port Service Function Virtual Card    Active         No

7: FC       2-Port Service Function Virtual Card    Active         No

8: FC       2-Port Service Function Virtual Card    Active         No

9: FC       2-Port Service Function Virtual Card    Active         No

10: FC       2-Port Service Function Virtual Card    Active         No

11: FC       2-Port Service Function Virtual Card    Active         No

12: FC       2-Port Service Function Virtual Card    Active         No

13: FC       2-Port Service Function Virtual Card    Active         No

14: FC       2-Port Service Function Virtual Card    Active         No

15: FC       2-Port Service Function Virtual Card    Standby        -

16: FC       2-Port Service Function Virtual Card    Active         No

------- -

[local]PGW# show task resources facility diamproxy all

Thursday June 13 14:55:31 NZST 2019

task   cputime        memory     files      sessions

cpu facility      inst  used  allc   used  alloc used allc  used  allc S status

----------------------- ----------- ------------- --------- ------------- ------

3/0 diamproxy        8 1.10%   90% 43.75M 250.0M  271 2500    --    -- -   good

4/0 diamproxy        9 1.09%   90% 43.89M 250.0M  266 2500    --    -- -   good

5/0 diamproxy       10 1.09%   90% 43.93M 250.0M  237 2500    --    -- -   good

9/0 diamproxy        1 1.08%   90% 43.63M 250.0M  212 2500    --    -- -   good

10/0 diamproxy        4 1.08%   90% 43.88M 250.0M  258 2500    --    -- -   good

12/0 diamproxy       11 1.11%   90% 43.93M 250.0M  229 2500    --    -- -   good

13/0 diamproxy        2 1.09%   90% 43.75M 250.0M  258 2500    --    -- -   good

14/0 diamproxy        7 1.06%   90% 43.88M 250.0M  256 2500    --    -- -   good

16/0 diamproxy       13 1.04%   90% 43.89M 250.0M  199 2500    --    -- -   good

Solution

It is found that diamproxy facility was not spawned for card 7 although card 7 was active. In order to spawn diamproxy facility, diamctrl facility needs to be restarted or CF card switchover needs to be performed.

This action plan was implemented as suggested.

Switchover active CFC card with standby CFC card.

After the card switchover, respective diamproxy were spawned properly and Gx KPI was back to normal as per the expected values.

******** show card table all *******

Thursday June 13 22:17:54 NZST 2019<tel:2019>

Slot         Card Type                               Oper State     SPOF  Attach

-----------  --------------------------------------  -------------  ----  ------

1: CFC      Control Function Virtual Card           Active         No                >>CFC card 1 active post switchover

2: CFC      Control Function Virtual Card           Standby        -

3: FC       2-Port Service Function Virtual Card    Active         No

4: FC       2-Port Service Function Virtual Card    Active         No

5: FC       2-Port Service Function Virtual Card    Active         No

6: FC       2-Port Service Function Virtual Card    Active         No

7: FC       2-Port Service Function Virtual Card    Active         No

8: FC       2-Port Service Function Virtual Card    Active         No

9: FC       2-Port Service Function Virtual Card    Active         No

10: FC       2-Port Service Function Virtual Card    Active         No

11: FC       2-Port Service Function Virtual Card    Active         No

12: FC       2-Port Service Function Virtual Card    Active         No

13: FC       2-Port Service Function Virtual Card    Active         No

14: FC       2-Port Service Function Virtual Card    Active         No

15: FC       2-Port Service Function Virtual Card    Standby        -

16: FC       2-Port Service Function Virtual Card    Active         No

--------- -

[local]PGW# show task resources facility diamproxy all

Thursday June 13 14:55:31 NZST 2019

task   cputime        memory     files      sessions

cpu facility      inst  used  allc   used  alloc used allc  used  allc S status

----------------------- ----------- ------------- --------- ------------- ------

3/0 diamproxy        8 0.95%   90% 43.86M 250.0M  130 2500    --    -- -   good

4/0 diamproxy        9 0.98%   90% 44.00M 250.0M  130 2500    --    -- -   good

5/0 diamproxy       10 1.00%   90% 44.00M 250.0M  135 2500    --    -- -   good

7/0 diamproxy        3 0.26%   90% 43.73M 250.0M  278 2500    --    -- -   good      >>diamproxy instance 3 spawn with card 7

8/0 diamproxy        5 0.31%   90% 43.73M 250.0M  277 2500    --    -- -   good

9/0 diamproxy        1 1.00%   90% 43.86M 250.0M  130 2500    --    -- -   good

10/0 diamproxy        4 0.98%   90% 43.96M 250.0M  133 2500    --    -- -   good

11/0 diamproxy        6 0.24%   90% 43.74M 250.0M  277 2500    --    -- -   good

12/0 diamproxy       11 1.04%   90% 44.01M 250.0M  144 2500    --    -- -   good

13/0 diamproxy        2 0.97%   90% 43.84M 250.0M  134 2500    --    -- -   good

14/0 diamproxy        7 1.04%   90% 43.99M 250.0M  138 2500    --    -- -   good

16/0 diamproxy       13 0.96%   90% 43.74M 250.0M  261 2500    --    -- -   good

Respective diamproxy facility were spawned after the activity and diameter messages were then exchanged properly between PGW and PCRF.