Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Problem

Scenario 1

Difference in Framed-IP-Prefix AVP Value in GX_ (CCR-I) and Rx_AAR

Scenario 2

Contradiction in Gx Stale Session Timer and Session Expiration Timer Set in CPS

Scenario 3

Subscriber Session Available at PCRF, but Not in PGW Caused Due to Gx Interface Flap

Solution

Approach for Scenario 1

Approach for Scenario 2

Procedure to Set the Timer

Approach for Scenario 3

Procedure to Set the Rx_ASR Parameter

Introduction

This document describes various investigation methods to handle the Diameter Error Code: "IP-CAN_SESSION_NOT_AVAILABLE (5065)" in Cisco Policy Suite (CPS).

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Linux

- CPS

- Policy Control Charge Rule Function (PCRF)

- Diameter

Note: Cisco recommends that you must have privilege root access to CPS CLI.

Components Used

The information in this document is based on CPS 21.1.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Diameter Error Code: "IP-CAN_SESSION_NOT_AVAILABLE (5065)" is a commonly seen Diameter Experimental-Result-Code in PCRF which handles Voice over Long-Term Evolution (VoLTE) services.

This error is not generated for a healthy VoLTE system. There are a few scenarios that are discussed in this document where this error is reported and its various approaches to troubleshoot them.

In general, If the PCRF fails to do a session bind between Rx and Gx sessions, then it responds to the Application Function (AF) with an AA-Answer, that includes an Experimental-Result-Code Attribute Value Pair (AVP) set to the value " IP-CAN_SESSION_NOT_AVAILABLE (5065) ".

Problem

Scenario 1

Difference in Framed-IP-Prefix AVP Value in GX_ (CCR-I) and Rx_AAR

The basic rule for Rx to Gx Session bind is based on the same Framed-IPv6-Prefix received from Gx_Credit Control Request Initial (CCR-I) 543 while the Gx session creation and the one received from Rx_Authorization Authentication Request (AAR) for Rx Session creation.

In this scenario, it is observed that all the error samples of IP-CAN_SESSION_NOT_AVAILABLE (5065) are reported from a specific list of Framed-IPv6-Prefix. These Framed-IPv6-Prefixes are not reported in Gx_CCR-I .

This CLI provides the unique list and total count of Framed-IPv6-Prefixfor which the IP-CAN_SESSION_NOT_AVAILABLE (5065) is reverted by CPS to AF.

[root@LAB-PCRF02CO-pcrfclient01 broadhop]# zgrep -B80 "IP-CAN_SESSION_NOT_AVAILABLE" consolidated-engine.*.gz | grep "Framed-IPv6-Prefix:" | cut -f3,4,5 -d: | sort| uniq -c

6  2402:XXXX:YYYY

3  2402:PPPP:QQQQ

1  2402:JJJJ:KKKK

This CLI provides the unique list and count of Framed-IPv6-Prefix reported in Gx_CCR-I from Packet Data Network Gateway (PGW) .

[root@LAB-PCRF02CO-pcrfclient01 broadhop]# zgrep -A 50 CCR-I consolidated-engine.*.gz | grep Framed-IPv6-Prefix: | cut -f 3,4,5 -d: | sort | uniq -c

115  2402:AAAA:BBBB

116  2402:NNNN:MMMM

246  2402:SSSS:RRRR

Note: For the CLI execution, you must log in to pcrfclient01 of CPS and navigate to /var/log/broadhop/ directory.

The expectation for a healthy system is, the Framed-IPv6-Prefixreceived from Gx_CCR-I and Rx_AAR be from the same IP prefix pool allocated for Gm IP (PGW to AF).

In this scenario, it is clear that, for all the failure samples collected, the Framed-IP-Prefix reported in Rx_AAR is completely different from the Framed-IP-Prefix received in Gx_CCR-I.

Scenario 2

Contradiction in Gx Stale Session Timer and Session Expiration Timer Set in CPS

1. What is the Gx Stale session timer?

This timer is armed every time a message is received or sent for any given Gx session. When the timer expires, a Gx_Re-Auth-Request (RAR)  is sent with Re-Auth-Request-Type AVP set to AUTHORIZE_ONLY (0) for that Gx session towards PGW. If a Gx_Re-Auth-Answer (RAA) is received with Result-Code AVP value set to DIAMETER_UNKNOWN_SESSION_ID (5002) or DIAMETER_UNABLE_TO_COMPLY (5012) the Gx session is deemed stale and removed from the PCRF internal database. On any activity over the Gx interface (Gx_RAR/Gx_CCR) , the timer is reset.

2. What is a Session Expiration timer?

If no messages are received within the timer set, the session is removed from PCRF.

In the problematic scenario, the timers are set like this.

Gx Stale session timer: 180 mins

Session expiration time: 240 min

As per the timer configuration, after 180 mins a Keep alive RAR is generated and sent to PGW, and if the session is inactive for 240 mins, the session gets removed from PCRF DB.

If the Keep alive RAR is missed, due to multiple reasons [Gx interface flap, PGW out of service, or any corner case], the session reaches its expiration time, and at the same time if there is any Rx_AAR received from AF, you can see this error, and a Gx_RAR is sent to PGW with  Session-Release-Cause: UNSPECIFIED_REASON (0).

INFO   : (core) Deletion started at Wed Oct 19 07:07:03 IST 2022

WARN   : (core) Resending original message since the existing session is in deletion and this session triggered a start

Message: com.broadhop.diameter2.messages.DiameterRequestMessage

Timestamp: 2022-10-19 01:37:03,676

Application Id: Gx (16777238)

Command Code: Gx_RAR (258)

Dest host: 0006-diamproxy.lab.cisco.com

Dest realm: vpcrf.cisco.com

Device protocol: GX_TGPP

End to end id: 0 (0x00000000)

Hop by hop id: 0 (0x00000000)

Origin state: 0

Stack name: null

Origin realm: x.x.x.x

Session-Id: 0006-diamproxy.lab.cisco.com;1200190897;28719897;634ebb76-11902

Auth-Application-Id: 16777238

Re-Auth-Request-Type: AUTHORIZE_ONLY (0)

Session-Release-Cause: UNSPECIFIED_REASON (0)

Also, the Gx session is marked for soft delete at PCRF. Since PCRF failed to bind the Gx session, a Rx_AAA is responded back with Experimental-Result-Code AVP set to the value  IP-CAN_SESSION_NOT_AVAILABLE (5065).

This is the complete call flow that explains the scenario:

=========================================================================

LAB-PCRF02-qns05  [2022-10-19 07:07:03,676] ===============================================

POLICY RESULT SUCCESS:

session action = Delete

domainId = IMS

locationId = ims

SERVICES: IMS_DEFAULT

TRIGGER: Message: com.broadhop.diameter2.messages.DiameterRequestMessage

Timestamp: 2022-10-19 01:37:03,673

Application Id: Rx (16777236)

Command Code: Rx_AAR (265)

Dest realm: x.x.x.x

Device protocol: RX_TGPP

End to end id: 1274920388 (0x4bfdbdc4)

Hop by hop id: 3440610237 (0xcd138bbd)

Origin state: 0

Stack name: null

Origin host: x.x.x.x

Origin realm: x.x.x.x

Session-Id: x.x.x.x;168440;1;5.3576663599.2;pcsf.ims322-002

Auth-Application-Id: 16777236

Media-Component-Description:

Media-Component-Number: 1

Media-Sub-Component:

Flow-Number: 1

Flow-Description: permit in 17 from xxxx:xxxx:xxxx:xxxx:xxxx:xxxx  to xxxx:xxxx:xxxx:xxxx:xxxx:xxxx

Flow-Description: permit out 17 from xxxx:xxxx:xxxx:xxxx:xxxx:xxxx to xxxx:xxxx:xxxx:xxxx:xxxx:xxxx

Media-Sub-Component:

Flow-Number: 2

Flow-Description: permit in 17 from xxxx:xxxx:xxxx:xxxx:xxxx:xxxx  to xxxx:xxxx:xxxx:xxxx:xxxx:xxxx

Flow-Description: permit out 17 from xxxx:xxxx:xxxx:xxxx:xxxx:xxxx to xxxx:xxxx:xxxx:xxxx:xxxx:xxxx

Flow-Usage: RTCP (1)

Codec-Data: downlink\r\noffer\r\nm=audio 23096 RTP/AVP 108 102 8 18 100 101 116\r\na=rtpmap:108 AMR/8000\r\na=fmtp:108 mode-set=0,2,4,7; mode-change-period=2;                                        Codec-Data: uplink\r\nanswer\r\nm=audio 41660 RTP/AVP 108 116\r\na=rtpmap:108 AMR/8000/1\r\na=ptime:20\r\na=maxptime:240\r\na=rtpmap:116 telephone-event/

Media-Type: AUDIO (0)

Max-Requested-Bandwidth-UL: 37000

Max-Requested-Bandwidth-DL: 37000

Flow-Status: ENABLED (2)

RS-Bandwidth: 462

RR-Bandwidth: 1387

Subscription-Id:

Subscription-Id-Type: END_USER_SIP_URI (2)

Subscription-Id-Data: sip:+91XXXXXXXXXX

AF-Charging-Identifier: ABCA4C23EA4C0320221019772

Specific-Action: INDICATION_OF_LOSS_OF_BEARER (2)

Specific-Action: INDICATION_OF_RECOVERY_OF_BEARER (3)

Specific-Action: INDICATION_OF_RELEASE_OF_BEARER (4)

Specific-Action: IP_CAN_CHANGE (6)

Framed-IPv6-Prefix: XXXX:XXXX:XXXX:XXXX (0xxxxxxxxxxxxxxxxxxxxx)

Supported-Features:

Vendor-Id: 10415

Feature-List-ID: 1

Feature-List: 65587

Service-Info-Status: FINAL SERVICE INFORMATION (0)

Req-Received-Time: 1666143423673

QPS-Internal-Route-Record-Host: x.x.x.x

QPS-Internal-Route-Record-Realm: x.x.x.x

DEBUG MSGS:

INFO   : (core) Tagging message with ID: RX_TGPP

INFO   : (core) Successful load by secondary key: FramedIpv6PrefixKey:framedIpv6Prefix:XXXX%XXXX%XXXX%XXXX

INFO   : (core) Start session triggered

INFO   : (core) Deletion started at Wed Oct 19 07:07:03 IST 2022

WARN   : (core) Resending original message since the existing session is in deletion and this session triggered a start

INFO   : (core) domain: IMS

INFO   : (gx) Sending RAR Message on session 0006-diamproxy.lab.cisco.com;1200190897;28719897;634ebb76-11902

INFO   : (core) Adding network-id keys for missing credentials

INFO   : (custrefdata) QueryTable: TAI_BASED_POLICY Field: {ECGI=XXXXX, packname=-1, AN GW IPV4=*, RAT_TYPE=1004, IMSI_Prefix=XXXXX, TAI=4042052329, APN=XXX}

INFO   : (custrefdata) QueryTable: LAC_BASED_POLICY Field: {MCC_MNC=XXXX, packname=-1, rat=1004, CGI=*, SGSN_IP=*, CI=*, IMSI_Prefix=XXXXXX, LAC=XXXX, APN=XX}

INFO   : (core) domain: IMS

INFO   : (core) Session removed

SERVICE CALC MSGS:

USE CASES:

INFO   : (use-cases) Use case ‘Lab IMS Default', status: true, Condition: No Condition Set

SENT MESSAGES (asynchronous):

Message: com.broadhop.diameter2.messages.DiameterRequestMessage

Timestamp: 2022-10-19 01:37:03,676

Application Id: Gx (16777238)

Command Code: Gx_RAR (258)

Dest host: 0006-diamproxy.lab-spgw01.cisco.com

Dest realm: vpcrf.cisco.com

Device protocol: GX_TGPP

End to end id: 0 (0x00000000)

Hop by hop id: 0 (0x00000000)

Origin state: 0

Stack name: null

Origin realm: x.x.x.x

Session-Id: 0006-diamproxy.lab-spgw01.cisco.com;1200190897;28719897;634ebb76-11902

Auth-Application-Id: 16777238

Re-Auth-Request-Type: AUTHORIZE_ONLY (0)

Session-Release-Cause: UNSPECIFIED_REASON (0)

=========================================================================

At the same time, for the same packet, another session action: create  is processed by CPS but couldn’t bind the Gx session, and hence a 5065 Error is reverted to AF.

LAB-VPCRF02-qns05  [2022-10-19 07:07:03,696] ===============================================

POLICY RESULT SUCCESS:

session action = Create

TRIGGER: Message: com.broadhop.diameter2.messages.DiameterRequestMessage

Timestamp: 2022-10-19 01:37:03,673

Application Id: Rx (16777236)

Command Code: Rx_AAR (265)

Dest realm: x.x.x.x

Device protocol: RX_TGPP

End to end id: 1274920388 (0x4bfdbdc4)

Hop by hop id: 3440610237 (0xcd138bbd)

Origin state: 0

Stack name: null

Origin host: x.x.x.x

Origin realm: x.x.x.x

Session-Id: x.x.x.x;168440;1;5.3576663599.2;pcsf.ims322-002

Auth-Application-Id: 16777236

Media-Component-Description:

Media-Component-Number: 1

Media-Sub-Component:

Flow-Number: 1

Flow-Description: permit in 17 from xxxxxx 8975:b101 41660 toxxxxxx::2 23096

Flow-Description: permit out 17 fromxxxxxx::2 23096 toxxxxxx:b101 41660

Media-Sub-Component:

Flow-Number: 2

Flow-Description: permit in 17 from xxxxxx 8975:b101 41660 toxxxxxx::2 23096

Flow-Description: permit out 17 fromxxxxxx::2 23096 toxxxxxx:b101 41660

Flow-Usage: RTCP (1)

Codec-Data: downlink\r\noffer\r\nm=audio 23096 RTP/AVP 108 102 8 18 100 101 116\r\na=rtpmap:108 AMR/8000\r\na=fmtp:108 mode-set=0,2,4,7; mode-change-period=2;                                        Codec-Data: uplink\r\nanswer\r\nm=audio 41660 RTP/AVP 108 116\r\na=rtpmap:108 AMR/8000/1\r\na=ptime:20\r\na=maxptime:240\r\na=rtpmap:116 telephone-event/                                                 Media-Type: AUDIO (0)

Max-Requested-Bandwidth-UL: 37000

Max-Requested-Bandwidth-DL: 37000

Flow-Status: ENABLED (2)

RS-Bandwidth: 462

RR-Bandwidth: 1387

Subscription-Id:

Subscription-Id-Type: END_USER_SIP_URI (2)

Subscription-Id-Data: sip:+xxxxxxxx

AF-Charging-Identifier: ABCA4C23EA4C0320221019772

Specific-Action: INDICATION_OF_LOSS_OF_BEARER (2)

Specific-Action: INDICATION_OF_RECOVERY_OF_BEARER (3)

Specific-Action: INDICATION_OF_RELEASE_OF_BEARER (4)

Specific-Action: IP_CAN_CHANGE (6)

Framed-IPv6-Prefix: xxxx:xxxx:xxxx:xxxx

Supported-Features:

Vendor-Id: 10415

Feature-List-ID: 1

Feature-List: 65587

Service-Info-Status: FINAL SERVICE INFORMATION (0)

Req-Received-Time: 1666143423673

QPS-Internal-Route-Record-Host: x.x.x.x

QPS-Internal-Route-Record-Realm: x.x.x.x

DEBUG MSGS:

INFO   : (core) Tagging message with ID: RX_TGPP

INFO   : (rx) Could not perform Gx session binding for Framed-IP-Address null, Framed-IPv6-Prefix xxxx:xxxx:xxxx:xxxx

INFO   : (core) Start session triggered

INFO   : (rx) Could not perform Gx session binding for Framed-IP-Address null, Framed-IPv6-Prefix xxxx:xxxx:xxxx:xxxx

INFO   : (rx) Sending 265 Message on session x.x.x.x;168440;1;5.3576663599.2;pcsf.ims322-002

ERROR  : (rules) Could not perform Gx session binding: Could not perform Gx session binding 5065

SERVICE CALC MSGS:

USE CASES:

SENT MESSAGES (asynchronous):

Message: com.broadhop.diameter2.messages.DiameterResponseMessage

Timestamp: 2022-10-19 01:37:03,696

Application Id: Rx (16777236)

Command Code: Rx_AAA (265)

Dest host: x.x.x.x

Dest realm: x.x.x.x

Device protocol: RX_TGPP

End to end id: 1274920388 (0x4bfdbdc4)

Hop by hop id: 3440610237 (0xcd138bbd)

Origin state: 0

Stack name: null

Origin realm: x.x.x.x

Session-Id: x.x.x.x;168440;1;5.3576663599.2;pcsf.ims322-002

Auth-Application-Id: 16777236

QPS-Internal-Route-Record-Host: x.x.x.x

QPS-Internal-Route-Record-Realm: x.x.x.x

Experimental-Result:

Vendor-Id: 10415

Experimental-Result-Code: IP-CAN_SESSION_NOT_AVAILABLE (5065)

=========================================================================

Scenario 3

Subscriber Session Available at PCRF, but Not in PGW Caused Due to Gx Interface Flap

There is a corner case scenario, where the subscriber sessions are removed in PGW, but it is present in PCRF. This happens, when there is a Gx link flap caused by a transport layer break.

Failure call flow diagram:

Failure Scenario: IP_CAN_SESSION_NOT_AVAILABLE(5065)

Sample call flow for this error from the Engine log:

lab-qns07  [2022-08-24 00:42:48,643] ===============================================

POLICY RESULT SUCCESS:

session action = None

domainId = IMS

locationId = ims

SERVICES: IMS_DEFAULT

TRIGGER: Message: com.broadhop.diameter2.messages.DiameterRequestMessage

Timestamp: 2022-08-23 19:12:48,638

Application Id: Rx (16777236)

Command Code: Rx_AAR (265)

Dest realm: x.x.x.x

Device protocol: RX_TGPP

End to end id: 1797266185 (0x6b201b09)

Hop by hop id: 1690686712 (0x64c5d4f8)

Origin state: 0

Stack name: null

Origin host: x.x.x.x

Origin realm: x.x.x.x

Session-Id: x.x.x.x;367169;1;5.1236049266.2;pcsf-stdn.ims332-012

Auth-Application-Id: 16777236

Media-Component-Description:

Media-Component-Number: 1

Media-Sub-Component:

Flow-Number: 1

Flow-Description: permit in 17 from xxxxxx 50060 to xxxxxx::4 28532

Flow-Description: permit out 17 from xxxxxx::4 28532 to xxxxxxx 50060

Media-Sub-Component:

Flow-Number: 2

Flow-Description: permit in 17 from xxxxxx 50060 to xxxxxx::4 28532

Flow-Description: permit out 17 from xxxxxx::4 28532 to xxxxxxx 50060

Flow-Usage: RTCP (1)

Codec-Data: uplink\r\noffer\r\nm=audio 50060 RTP/AVP 104 102 105 96\r\na=rtpmap:104 AMR-WB/16000/1\r\na=fmtp:104 mode-set=0,3;mode-change-capability=2;max-

Codec-Data: downlink\r\nanswer\r\nm=audio 28532 RTP/AVP 102 96\r\na=rtpmap:102 AMR/8000/1\r\na=fmtp:102 mode-set=0,2,4,7; mode-change-period=2; mode-change-capability=2;

Media-Type: AUDIO (0)

Max-Requested-Bandwidth-UL: 72000

Max-Requested-Bandwidth-DL: 72000

Flow-Status: ENABLED (2)

RS-Bandwidth: 362

RR-Bandwidth: 1087

Subscription-Id:

Subscription-Id-Type: END_USER_SIP_URI (2)

Subscription-Id-Data: sip:+919XXXXXX

AF-Charging-Identifier: "PCSF:1-MU1-cfed-12-00000000630526b2-0000000017442828"

Specific-Action: INDICATION_OF_LOSS_OF_BEARER (2)

Specific-Action: INDICATION_OF_RECOVERY_OF_BEARER (3)

Specific-Action: INDICATION_OF_RELEASE_OF_BEARER (4)

Specific-Action: IP_CAN_CHANGE (6)

Framed-IPv6-Prefix: xxxx:xxxx:xxxx:xxxx

Supported-Features:

Vendor-Id: 10415

Feature-List-ID: 1

Feature-List: 65587

Service-Info-Status: FINAL SERVICE INFORMATION (0)

Req-Received-Time: 1661281963763

QPS-Internal-Route-Record-Host: x.x.x.x

QPS-Internal-Route-Record-Realm: x.x.x.x

DEBUG MSGS:

INFO   : (core) Tagging message with ID: RX_TGPP

INFO   : (core) Successful load by secondary key: FramedIpv6PrefixKey:framedIpv6Prefix:xxxx%3Axxxx%3Axxxx%3Axxxx

INFO   : (core) Start session triggered

INFO   : (rx) Success binding to Gx session 0001-diamproxy.ndc3-ims-lab-sbc1;1106437295;560189406;63051856-de02 by IP Address

INFO   : (TGPP) [Rel8, Rel9, Rel10, NetLoc, NetLocUntrustedWLAN] features supported for Vendor-Id 10415, Feature-List-ID 1

INFO   : (rx) Creating new diameter session x.x.x.x;367169;1;5.1236049266.2;pcsf-stdn.ims332-012

INFO   : (policy) Load CRD for Unknown Subscriber

INFO   : (location) Location found for avp matching: logical_apn\ims

WARN   : (core) Session has switched from known to unknown as subscriber could not be found

INFO   : (core) Adding unknown service code: IMS_DEFAULT for session

INFO   : (core) domain: IMS

INFO   : (core) Adding network-id keys for missing credentials

INFO   : (core) domain: IMS

INFO   : (rx) Sending AAA Message on session x.x.x.x;367169;1;5.1236049266.2;pcsf-stdn.ims332-012

INFO   : (gx) Installing preconfigured rule: _6_1_2_AF_AUDIO

INFO   : (gx) Installing preconfigured rule: _6_1_1_AF_AUDIO

INFO   : (et) Event trigger 2 added

INFO   : (et) Event trigger 5 added

INFO   : (et) Event trigger 6 added

INFO   : (et) Event trigger 7 added

INFO   : (gx) Sending RAR Message on session 0001-diamproxy.ndc3-ims-lab-sbc1;1106437295;560189406;63051856-de02

SERVICE CALC MSGS:

USE CASES

INFO   : (use-cases) Use case 'WLAN', status: false, Condition: ("Rat-Type WLAN"=false)

INFO   : (use-cases) Use case 'IMS Default', status: true, Condition: No Condition Set

SENT MESSAGES (asynchronous):

Message: com.broadhop.diameter2.messages.DiameterResponseMessage

Timestamp: 2022-08-23 19:12:48,643

Application Id: Rx (16777236)

Command Code: Rx_AAA (265)

Dest host: x.x.x.x

Dest realm: x.x.x.x

Device protocol: RX_TGPP

End to end id: 1797266185 (0x6b201b09)

Hop by hop id: 1690686712 (0x64c5d4f8)

Origin state: 0

Stack name: null

Origin realm: x.x.x.x

Session-Id: x.x.x.x;367169;1;5.1236049266.2;pcsf-stdn.ims332-012

Auth-Application-Id: 16777236

IP-CAN-Type: 3GPP_EPS (5)

RAT-Type: EUTRAN (1004)

Supported-Features:

Vendor-Id: 10415

Feature-List-ID: 1

Feature-List: 65587QPS-Internal-Route-Record-Host: x.x.x.x

QPS-Internal-Route-Record-Realm: x.x.x.x

Result-Code: DIAMETER_SUCCESS (2001)

Message: com.broadhop.diameter2.messages.DiameterRequestMessage

Timestamp: 2022-08-23 19:12:48,643

Application Id: Gx (16777238)

Command Code: Gx_RAR (258)

Dest host: 0001-diamproxy.ndc3-ims-lab-sbc1

Dest realm: vpcrf.cisco.com

Device protocol: GX_TGPP

End to end id: 0 (0x00000000)

Hop by hop id: 0 (0x00000000)

Origin state: 0

Stack name: null

Origin realm: x.x.x.x

Session-Id: 0001-diamproxy.ndc3-ims-airpcrf02-sbc1;1106437295;560189406;63051856-de02

Auth-Application-Id: 16777238

Re-Auth-Request-Type: AUTHORIZE_ONLY (0)

Charging-Rule-Install:

Charging-Rule-Definition:

Charging-Rule-Name: _6_1_2_AF_AUDIO

Rating-Group: 2

Flow-Information:

Flow-Description: permit in 17 from xxxxxx 50061 to xxxxxxx28533

Flow-Direction: UPLINK (2)

Flow-Information:

Flow-Description: permit out 17 from xxxxx 28533 to xxxxxx 50061

Flow-Direction: DOWNLINK (1)

Flow-Status: ENABLED (2)

QoS-Information:

QoS-Class-Identifier: Conversational (1)

Max-Requested-Bandwidth-UL: 1449

Max-Requested-Bandwidth-DL: 1449

Guaranteed-Bitrate-UL: 1449

Guaranteed-Bitrate-DL: 1449

Reporting-Level: RATING_GROUP_LEVEL (1)

Precedence: 0

AF-Charging-Identifier: "PCSF:1-MU1-cfed-12-00000000630526b2-0000000017442828"

Flows:

Media-Component-Number: 1

Flow-Number: 2

Charging-Rule-Install:

Charging-Rule-Definition:

Charging-Rule-Name: _6_1_1_AF_AUDIO

Rating-Group: 2

Flow-Information:

Flow-Description: permit in 17 from xxxxxx 50060 to xxxxxx 28532

Flow-Direction: UPLINK (2)

Flow-Information:

Flow-Description: permit out 17 from xxxxxxx 28532 to 2xxxxx 50060

Flow-Direction: DOWNLINK (1)

Flow-Status: ENABLED (2)

QoS-Information:

QoS-Class-Identifier: Conversational (1)

Max-Requested-Bandwidth-UL: 72000

Max-Requested-Bandwidth-DL: 72000

Guaranteed-Bitrate-UL: 72000

Guaranteed-Bitrate-DL: 72000

Reporting-Level: RATING_GROUP_LEVEL (1)

Precedence: 0

AF-Charging-Identifier: "PCSF:1-MU1-cfed-12-00000000630526b2-0000000017442828"

Flows:

Media-Component-Number: 1

Flow-Number: 1

Event-Trigger: RAT_CHANGE (2)

Event-Trigger: LOSS_OF_BEARER (5)

Event-Trigger: RECOVERY_OF_BEARER (6)

Event-Trigger: IP-CAN_CHANGE (7)

=======================================================================

lab-qns04  [2022-08-24 00:42:49,849] ===============================================

POLICY RESULT SUCCESS:

session action = None

domainId = IMS

locationId = ims

SERVICES: IMS_DEFAULT

TRIGGER: Message: com.broadhop.diameter2.messages.DiameterResponseMessage

Timestamp: 2022-08-23 19:12:49,844

Application Id: Gx (16777238)

Command Code: Gx_RAA (258)

Device protocol: GX_TGPP

End to end id: 2244560894 (0x85c947fe)

Hop by hop id: 2040440255 (0x799ea5bf)

Origin state: 1657916833

Stack name: null

Origin host: 001-diamproxy.ndc3-ims-lab-sbc1

Origin realm: vpcrf.cisco.com

Session-Id: 0001-diamproxy.ndc3-ims-lab-sbc1;1106437295;560189406;63051856-de02

Origin-State-Id: 1657916833

Error-Message: Unknown Session ID

Req-Received-Time: 0

Result-Code: DIAMETER_UNKNOWN_SESSION_ID (5002)

DEBUG MSGS:

INFO   : (core) Tagging message with ID: GX_TGPP

INFO   : (core) Successful load by primary key: diameterSessionKey:0001-diamproxy.ndc3-ims-airpcrf02-sbc1%3B1106437295%3B560189406%3B63051856-de02

INFO   : (core) Stop session triggered

INFO   : (core) Deletion started at Wed Aug 24 00:42:49 IST 2022

INFO   : (core) Soft deletion triggered for session

INFO   : (policy) Load CRD for Unknown Subscriber

INFO   : (location) Location found for avp matching: logical_apn\ims

WARN   : (core) Session has switched from known to unknown as subscriber could not be found

INFO   : (core) Adding unknown service code: IMS_DEFAULT for session

INFO   : (core) domain: IMS

INFO   : (core) Adding network-id keys for missing credentials

INFO   : (core) domain: IMS

SERVICE CALC MSGS:

USE CASES:

INFO   : (use-cases) Use case 'WLAN', status: false, Condition: ("Rat-Type WLAN"=false)

INFO   : (use-cases) Use case 'Vodafone IMS Default', status: true, Condition: No Condition Set

=========================================================================

lab-qns04  [2022-08-24 00:43:02,349] ===============================================

POLICY RESULT SUCCESS:

session action = Create

TRIGGER: Message: com.broadhop.diameter2.messages.DiameterRequestMessage

Timestamp: 2022-08-23 19:13:02,330

Application Id: Rx (16777236)

Command Code: Rx_AAR (265)

Dest realm: x.x.x.x

Device protocol: RX_TGPP

End to end id: 1811425453 (0x6bf828ad)

Hop by hop id: 1690687469 (0x64c5d7ed)

Origin state: 0

Stack name: null

Origin host: x.x.x.x

Origin realm: x.x.x.x

Session-Id: x.x.x.x;367195;1;5.1236262316.2;pcsf-stdn.ims332-012

Auth-Application-Id: 16777236

Media-Component-Description:

Media-Component-Number: 1

Media-Sub-Component:

Flow-Number: 1

Flow-Description: permit in 17 from xxxx:xxxx:xxxx:xxxx: 50032 to xxxx:xxxx:xxxx:xxxx::6 5144

Flow-Description: permit out 17 from xxxx:xxxx:xxxx:xxxx::6 xxxx to xxxx:xxxx:xxxx:xxxx: 50032

Media-Sub-Component:

Flow-Number: 2

Flow-Description: permit in 17 from 2402:3a80:2027:2db8:ac0e:ff4:dc53:af02 50033 to 2402:3a80:2000:606::6 5145

Flow-Description: permit out 17 from 2402:3a80:2000:606::6 5145 to 2402:3a80:2027:2db8:ac0e:ff4:dc53:af02 50033

Flow-Usage: RTCP (1)

Codec-Data: downlink\r\noffer\r\nm=audio 5144 RTP/AVP 96 8 18 100 101 97\r\na=rtpmap:96 AMR/8000\r\na=fmtp:96 mode-set=0,2,4,7; mode-change-period=2; mode-change-Codec-Data: uplink\r\nanswer\r\nm=audio 50032 RTP/AVP 96 97\r\na=rtpmap:96 AMR/8000/1\r\na=fmtp:96 mode-set=0,2,4,7;mode-change-capability=2;max-red=0\r\na=rtpmap:97 Media-Type: AUDIO (0)

Max-Requested-Bandwidth-UL: 37000

Max-Requested-Bandwidth-DL: 37000

Flow-Status: ENABLED (2)

RS-Bandwidth: 0

RR-Bandwidth: 0

Subscription-Id:

Subscription-Id-Type: END_USER_SIP_URI (2)

Subscription-Id-Data: sip:+xxxxxxx

AF-Charging-Identifier: GYlkAABklmEFGKS7UELMBecR

Specific-Action: INDICATION_OF_LOSS_OF_BEARER (2)

Specific-Action: INDICATION_OF_RECOVERY_OF_BEARER (3)

Specific-Action: INDICATION_OF_RELEASE_OF_BEARER (4)

Specific-Action: IP_CAN_CHANGE (6)

Framed-IPv6-Prefix: xxxxxxxxx

Supported-Features:

Vendor-Id: 10415

Feature-List-ID: 1

Feature-List: 65587

Service-Info-Status: FINAL SERVICE INFORMATION (0)

Req-Received-Time: 1661281976257

QPS-Internal-Route-Record-Host: x.x.x.x

QPS-Internal-Route-Record-Realm: x.x.x.x

DEBUG MSGS:

INFO   : (core) Tagging message with ID: RX_TGPP

INFO   : (rx) Could not perform Gx session binding for Framed-IP-Address null, Framed-IPv6-Prefix xxxxxx

INFO   : (core) Start session triggered

INFO   : (rx) Could not perform Gx session binding for Framed-IP-Address null, Framed-IPv6-Prefix xxxxxx

INFO   : (rx) Sending 265 Message on session x.x.x.x;367195;1;5.1236262316.2;pcsf-stdn.ims332-012

ERROR  : (rules) Could not perform Gx session binding: Could not perform Gx session binding 5065

SERVICE CALC MSGS:

USE CASES:

SENT MESSAGES (asynchronous):

Message: com.broadhop.diameter2.messages.DiameterResponseMessage

Timestamp: 2022-08-23 19:13:02,349

Application Id: Rx (16777236)

Command Code: Rx_AAA (265)

Dest host: x.x.x.x

Dest realm: x.x.x.x

Device protocol: RX_TGPP

End to end id: 1811425453 (0x6bf828ad)

Hop by hop id: 1690687469 (0x64c5d7ed)

Origin state: 0

Stack name: null

Origin realm: x.x.x.x

Session-Id: x.x.x.x;367195;1;5.1236262316.2;pcsf-stdn.ims332-012

Auth-Application-Id: 16777236

QPS-Internal-Route-Record-Host: x.x.x.x

QPS-Internal-Route-Record-Realm: x.x.x.x

Experimental-Result:

Vendor-Id: 10415

Experimental-Result-Code: IP-CAN_SESSION_NOT_AVAILABLE (5065)

===============================================

When you closely check the Error KPI trend for RX_AAR [IP-CAN_SESSION_NOT_AVAILABLE (5065)]  and Gx_RAR [DIAMETER_UNKNOWN_SESSION_ID (5002)], you can see the linear increment of these errors after a Gx interface flap.

Gx_RAR & Rx_AAR Failure Trend

Solution

Approach for Scenario 1

The solution to this issue must be executed outside of CPS. You must perform a reconciliation of the IP pool used in the Gm interface (the interface between SPGW to AF), and ensure the same Framed-IP-Prefixes  are used in both PGW and AF for the User Equipment (UE).

Approach for Scenario 2

Set the session expiration timer to be > (3 * Gx stale-session timer).

Procedure to Set the Timer

Step 1. Login to Policy Builder with Read/Write user.

Step 2. Set the Default Gx Stale Session Timer Minutes  as the recommended value.

Gx Stale Session Timer Configuration

Step 3. Set the Session Expiration Hours and Session Expiration Minutes  as recommended 3 * of Gx Stale session timer.

Session Expiration Timer Configuration

Step 4. Save the Repository and publish to the latest version.

Approach for Scenario 3

Introduce a Rx_ASR towards AF to clear the already created Rx Session if CPS receives Diameter Error Code=5002 in the response of Gx_RAR from PGW. In this way, if the session is not present in PGW, and CPS receives Diameter Error 5002 , along with soft deletion of the Gx session, CPS sends a Rx_ASR message towards the AF node with an Abort-Cause: 0.

Procedure to Set the Rx_ASR Parameter

Step 1. Log in to the Cluster Manager as the root user.

Step 2. Backupqns.conf file.

cp /etc/broadhop/qns.conf /etc/broadhop/qns.conf.$(date +%m%d%Y).bkp

Step 3. Edit the file /etc/broadhop/qns.conf  with vi editors and insert the line “-Dsend.Rx.ASR.For.AdditionalErrorCodes=5002”

Step 4. Build puppet script.

/var/qps/install/current/puppet/modules/qps/templates/var/broadhop/build_etc.sh

Step 5. Populate the changes in all VMs.

/var/qps/install/current/scripts/upgrade/reinit.sh

Step 6. Restart all to get the changes effective.

restartall.sh

This is the expected call flow after the required parameter change.

Success Scenario After the Change

This is the expected call flow from CPS engine log after the required parameter change.

=========================================================================

[2022-10-10 09:46:28,899] ===============================================

POLICY RESULT SUCCESS:

session action = None

domainId = F1660RxPriorityLevel

locationId = F1660IMS

SERVICES: RxPriorityLevel

TRIGGER: Message: com.broadhop.diameter2.messages.DiameterRequestMessage

Timestamp: 2022-10-10 09:46:28,332

Application Id: Rx (16777236)

Command Code: Rx_AAR (265)

Dest host: qns-site-server-1

Dest realm: cisco.com

Device protocol: RX_TGPP

End to end id: 535285358 (0x1fe7ce6e)

Hop by hop id: 655749306 (0x2715f0ba)

Origin state: 0

Stack name: null

Origin host: site-host-rx

Origin realm: site-rx-client

Session-Id: site-host-rx;132419;581;311482316600001IMSSession

Auth-Application-Id: 16777236

Media-Component-Description:

Media-Component-Number: 1

Media-Sub-Component:

Flow-Number: 1

Flow-Description: permit in ip from xxxxxxx 10000 to xxxxxxx:1 20000

Flow-Description: permit out ip from xxxxxxxxx1 20000 to xxxxxxx 10000

Flow-Usage: AF_SIGNALLING (2)

AF-Application-Identifier: lab.test.f1660.afapid01

Media-Type: VIDEO (1)

Max-Requested-Bandwidth-UL: 50000

Max-Requested-Bandwidth-DL: 50000

RS-Bandwidth: 800

RR-Bandwidth: 800

Specific-Action: INDICATION_OF_FAILED_RESOURCES_ALLOCATION (9)

Specific-Action: CHARGING_CORRELATION_EXCHANGE (1)

Specific-Action: INDICATION_OF_LOSS_OF_BEARER (2)

Specific-Action: INDICATION_OF_RECOVERY_OF_BEARER (3)

Specific-Action: INDICATION_OF_RELEASE_OF_BEARER (4)

Specific-Action: IP_CAN_CHANGE (6)

Specific-Action: ACCESS_NETWORK_INFO_REPORT (12)

Subscription-Id:

Subscription-Id-Type: END_USER_SIP_URI (2)

Subscription-Id-Data: sip:xxxxxxx

Framed-IP-Address: x.x.x.x (0x01020304)

Supported-Features:

Vendor-Id: 10415

Feature-List-ID: 1

Feature-List: 32

MIP6-Agent-Info:

MIP-Home-Agent-Host:

Destination-Realm: pcef

Destination-Host: vzr

Req-Received-Time: 1665395188271

QPS-Internal-Route-Record-Host: site-host-rx

QPS-Internal-Route-Record-Realm: site-rx-client

DEBUG MSGS:

INFO   : (core) Tagging message with ID: RX_TGPP

INFO   : (diameter)  Adding extracted AVP to policy: DiameterMessageAvp [code=Media-Type, value=1]

INFO   : (core) Successful load by secondary key: FramedIpKey:framedIp:x.x.x.x

INFO   : (core) Start session triggered

INFO   : (rx) Success binding to Gx session site-host-gx;132419;575;9001660001 by IP Address

INFO   : (TGPP) [NetLoc] features supported for Vendor-Id 10415, Feature-List-ID 1

INFO   : (rx) Creating new diameter session site-host-rx;132419;581;311482316600001IMSSession

INFO   : (gx) Installing preconfigured rule: _1_1_1_AF_VIDEO

INFO   : (et) Event trigger 2 added

INFO   : (et) Event trigger 5 added

INFO   : (et) Event trigger 6 added

INFO   : (et) Event trigger 7 added

INFO   : (et) Event trigger 28 added

INFO   : (gx) Sending RAR Message on session site-host-gx;132419;575;9001660001

SERVICE CALC MSGS:

USE CASES:

INFO   : (use-cases) Use case 'F1660_Rx_priorityLevel', status: true, Condition: No Condition Set

SENT MESSAGES (asynchronous):

com.broadhop.diameter2.actions.ISendDiameterResponse

DestinationName:

ResponseMessage: Message: com.broadhop.diameter2.messages.DiameterResponseMessage

Timestamp: 2022-10-10 09:46:28,893

Application Id: Rx (16777236)

Command Code: Rx_AAA (265)

Dest host: site-host-rx

Dest realm: site-rx-client

Device protocol: RX_TGPP

End to end id: 535285358 (0x1fe7ce6e)

Hop by hop id: 655749306 (0x2715f0ba)

Origin state: 0

Stack name: null

Origin realm: cisco

Session-Id: site-host-rx;132419;581;311482316600001IMSSession

Auth-Application-Id: 16777236

IP-CAN-Type: 3GPP_EPS (5)

RAT-Type: EUTRAN (1004)

Supported-Features:

Vendor-Id: 10415

Feature-List-ID: 1

Feature-List: 32

QPS-Internal-Route-Record-Host: site-host-rx

QPS-Internal-Route-Record-Realm: site-rx-client

Result-Code: DIAMETER_SUCCESS (2001)

com.broadhop.diameter2.actions.ISendDiameterRequest

RequestMessage: Message: com.broadhop.diameter2.messages.DiameterRequestMessage

Timestamp: 2022-10-10 09:46:28,899

Application Id: Gx (16777238)

Command Code: Gx_RAR (258)

Dest host: site-host-gx

Dest realm: site-gx-client

Device protocol: GX_TGPP

End to end id: 0 (0x00000000)

Hop by hop id: 0 (0x00000000)

Origin state: 0

Stack name: null

Session-Id: site-host-gx;132419;575;9001660001

Auth-Application-Id: 16777238

Re-Auth-Request-Type: 0

Charging-Rule-Install:

Charging-Correlation-Indicator: 0

Charging-Rule-Definition:

Charging-Rule-Name: _1_1_1_AF_VIDEO

Flow-Information:

Flow-Description: permit in ip from xxxx:xxxx:xxxx:xxxx: 10000 toxxxx:xxxx:xxxx:xxxx 20000

Flow-Information:

Flow-Description:permit in ip from xxxx:xxxx:xxxx:xxxx: 10000 toxxxx:xxxx:xxxx:xxxx 20000

QoS-Information:

QoS-Class-Identifier: 3

Max-Requested-Bandwidth-UL: 204800

Max-Requested-Bandwidth-DL: 204800

Allocation-Retention-Priority:

Priority-Level: 1

Pre-emption-Capability: 0

Pre-emption-Vulnerability: 1

Event-Trigger: 2

Event-Trigger: 5

Event-Trigger: 6

Event-Trigger: 7

Event-Trigger: 28

DestinationName: lab-1:diameter-lb

=========================================================================

[2022-10-10 09:46:29,323] ===============================================

POLICY RESULT SUCCESS:

session action = None

domainId = F1660RxPriorityLevel

locationId = F1660IMS

SERVICES: RxPriorityLevel

TRIGGER: Message: com.broadhop.diameter2.messages.DiameterResponseMessage

Timestamp: 2022-10-10 09:46:29,148

Application Id: Gx (16777238)

Command Code: Gx_RAA (258)

Device protocol: GX_TGPP

End to end id: 2577399809 (0x99a00001)

Hop by hop id: 1095304429 (0x414904ed)

Origin state: 0

Stack name: null

Origin host: site-host-gx

Origin realm: site-gx-client

Session-Id: site-host-gx;132419;575;9001660001

Auth-Application-Id: 16777238

Req-Received-Time: 0

Result-Code: DIAMETER_UNKNOWN_SESSION_ID (5002)

DEBUG MSGS:

INFO   : (core) Tagging message with ID: GX_TGPP

INFO   : (core) Lock obtained on key: diameterSessionKey:site-host-gx%3B132419%3B575%3B9001660001

INFO   : (core) Successful load by primary key: diameterSessionKey:site-host-gx%3B132419%3B575%3B9001660001

INFO   : (core) Stop session triggered

INFO   : (core) Deletion started at Mon Oct 10 09:46:29 UTC 2022

INFO   : (core) Soft deletion triggered for session

INFO   : (rx) Trigger Rx-ASR for error code 5002

INFO   : (rx) Sending ASR Message on session site-host-rx;132419;581;311482316600001IMSSession

SERVICE CALC MSGS:

USE CASES:

INFO   : (use-cases) Use case 'F1660_Rx_priorityLevel', status: true, Condition: No Condition Set

SENT MESSAGES (asynchronous):

com.broadhop.diameter2.actions.ISendDiameterRequest

RequestMessage: Message: com.broadhop.diameter2.messages.DiameterRequestMessage

Timestamp: 2022-10-10 09:46:29,323

Application Id: Rx (16777236)

Command Code: Rx_ASR (274)

Dest host: site-host-rx

Dest realm: site-rx-client

Device protocol: RX_TGPP

End to end id: 0 (0x00000000)

Hop by hop id: 0 (0x00000000)

Origin state: 0

Stack name: null

Session-Id: site-host-rx;132419;581;311482316600001IMSSession

Auth-Application-Id: 16777236

Abort-Cause: 0

DestinationName: lab-1:diameter-lb

=========================================================================