The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
CPS vDRA is a functional element that ensures that all Diameter sessions established over Gx, Rx interfaces and for unsolicited application reporting, the Sd interface for a certain IP-CAN session reach the same PCRF or destined PCRF when multiple and separately addressable PCRFs have been deployed in a Diameter realm.
Run the following command in CLI to view the diagnostics status. Verify that the status of all the nodes is in passing state.
admin@orchestrator[master-0]# show system diagnostics status
Run the following command in CLI to view the docker engines status. Verify that all docker engines are in CONNECTED state.
admin@orchestrator[master-0]# show docker engine
For messages belonging to particular interface, CPS vDRA should be ready to make diameter connection on the configured application port. As CPS vDRA acts as a server, it should be listening on ports for different applications to accept any incoming diameter requests for the application.
If you are facing problems making diameter connections, check for the following configuration:
Step 1 | Check status of
application base port on active policy director (lb). It should be listening to
diameter connections externally on VIP and internally to Policy Servers (QNS).
[root@lb01 ~]# netstat -na | grep 3868 tcp 0 0 10.77.207.100:3868 0.0.0.0:* LISTEN tcp 0 0 ::ffff:80.80.80.10:3868 :::* LISTEN |
Step 2 | Check
haproxy-diameter.cfg file for proper entries:
For Step 1 and Step 2 configuration, the entries should be as follows: [root@lb01 ~]# cat /etc/haproxy/haproxy-diameter.cfg global daemon nbproc 1 # number of processing cores stats socket /tmp/haproxy-diameter defaults timeout client 60000ms # maximum inactivity time on the client side timeout server 180000ms # maximum inactivity time on the server side timeout connect 5000ms # maximum time to wait for a connection attempt to a server to succeed log 127.0.0.1 local1 err listen diameter-int1 bind 10.77.207.100:3868 mode tcp option tcpka balance leastconn server lb01-A lb01:3868 check server lb01-B lb01:3869 check server lb01-C lb01:3870 check listen diameter-int2 bind 10.77.207.100:4868 mode tcp option tcpka balance leastconn server lb01-A lb01:4868 check server lb01-B lb01:4869 check server lb01-C lb01:4870 check listen stats_proxy_diameter lbvip01:5540 mode http option httpclose option abortonclose # enable web-stats stats enable stats uri /haproxy-diam?stats #stats auth haproxy:cisco123 stats refresh 60s stats hide-version |
Step 3 | Listen for
diameter traffic by logging into lb01 and lb02 and execute the following
command:
tcpdump -i any port 3868 -s 0 -vv |
Troubleshooting CPS vDRA consists of these types of basic tasks:
Result-Code |
Result-Code Value |
Description |
---|---|---|
Informational |
||
DIAMETER_MULTI_ROUND_AUTH |
1001 |
Subsequent messages triggered by client shall also used in Authentication and to get access of required resources. Generally used in Diameter NAS. |
Success |
||
DIAMETER_SUCCESS |
2001 |
Request processed Successfully. |
DIAMETER_LIMITED_SUCCESS |
2002 |
Request is processed but some more processing is required by Server to provide access to user. |
Protocol Errors [E-bit set] |
||
DIAMETER_COMMAND _UNSUPPORTED |
3001 |
Server returns it if Diameter Command-Code is un-recognized by server. |
DIAMETER_UNABLE _TO_DELIVER |
3002 |
Message cannot be delivered because there is no Host with Diameter URI present in Destination-Host AVP in associated Realm. |
DIAMETER_REALM_NOT _SERVED |
3003 |
Intended Realm is not recognized. |
DIAMETER_TOO_BUSY |
3004 |
Shall return by server only when server unable to provide requested service, where all the pre-requisites are also met. Client should also send the request to alternate peer. |
DIAMETER_LOOP_DETECTED |
3005 |
- |
DIAMETER_REDIRECT _INDICATION |
3006 |
In Response from Redirect Agent. |
DIAMETER_APPLICATION _UNSUPPORTED |
3007 |
- |
DIAMETER_INVALID_HDR_BITS |
3008 |
It is sent when a request is received with invalid bits combination for considered command-code in DIAMETER Header structure. For example, Marking Proxy-Bit in CER message. |
DIAMETER_INVALID_AVP_BITS |
3009 |
It is sent when a request is received with invalid flag bits in an AVP. |
DIAMETER_UNKNOWN_PEER |
3010 |
A DIAMETER server can be configured whether it shall accept DIAMETER connection from all nodes or only from specific nodes. If it is configured to accept connection from specific nodes and receives CER from message from any node other than specified. |
Transient Failures [Could not satisfy request at this moment] |
||
DIAMETER_AUTHENTICATION _REJECTED |
4001 |
Returned by Server, most likely because of invalid password. |
DIAMETER_OUT_OF_SPACE |
4002 |
Returned by node, when it receives accounting information but unable to store it because of lack of memory. |
ELECTION_LOST |
4003 |
Peer determines that it has lost election by comparing Origin-Host value received in CER with its own DIAMETER IDENTITY and found that received DIAMETER IDENTITY is higher. |
Permanent Failures [To inform peer, request is failed, should not be attempted again] |
||
DIAMETER_AVP _UNSUPPORTED |
5001 |
AVP marked with Mandatory Bit, but peer does not support it. |
DIAMETER_UNKNOWN _SESSION_ID |
5002 |
- |
DIAMETER_AUTHORIZATION _REJECTED |
5003 |
User can not be authorized. For example, Comes in AIA on s6a interface. |
DIAMETER_INVALID_AVP_VALUE |
5004 |
- |
DIAMETER_MISSING_AVP |
5005 |
Mandatory AVP in request message is missing. |
DIAMETER_RESOURCES _EXCEEDED |
5006 |
A request was received that cannot be authorized because the user has already expended allowed resources. An example of this error condition is a user that is restricted to one dial-up PPP port, attempts to establish a second PPP connection. |
DIAMETER_CONTRADICTING _AVPS |
5007 |
Server has identified that AVPs are present that are contradictory to each other. |
DIAMETER_AVP_NOT_ALLOWED |
5008 |
Message is received by node (Server) that contain AVP must not be present. |
DIAMETER_AVP_OCCURS _TOO_MANY_TIMES |
5009 |
If message contains the a AVP number of times that exceeds permitted occurrence of AVP in message definition. |
DIAMETER_NO_COMMON _APPLICATION |
5010 |
In response of CER if no common application supported between the peers. |
DIAMETER_UNSUPPORTED _VERSION |
5011 |
Self explanatory. |
DIAMETER_UNABLE _TO_COMPLY |
5012 |
Message rejected because of unspecified reasons. |
DIAMETER_INVALID_BIT _IN_HEADER |
5013 |
When an unrecognized bit in the Diameter header is set to one. |
DIAMETER_INVALID _AVP_LENGTH |
5014 |
Self explanatory. |
DIAMETER_INVALID _MESSAGE_LENGTH |
5015 |
Self explanatory. |
DIAMETER_INVALID_AVP _BIT_COMBO |
5016 |
For example, marking AVP to Mandatory while message definition doesn't say so. |
DIAMETER_NO_COMMON _SECURITY |
5017 |
In response of CER if no common security mechanism supported between the peers. |
Non-compliant Diameter requests are checked for errors in routing AVP and P-bits. The following table describes the error codes and the reasons for errors in Diameter requests:
Policy DRA Error String |
Error Code |
Sub-code |
Description |
---|---|---|---|
No application route found |
3002 |
001 |
Route List Availability Status is “Unavailable” |
Timeout triggered |
3002 |
002 |
Timeout triggered |
No peer group |
3002 |
003 |
No peer group |
Session DB Error |
3002 |
004 |
Session DB Error |
Binding DB Error |
3002 |
005 |
Binding DB Error |
No key for binding lookup |
3002 |
006 |
No key for binding lookup |
Binding not found |
3002 |
007 |
Binding not found |
Message loop detected |
3002 |
008 |
Message loop detected |
Parsing exception with message |
3002 |
009 |
Parsing exception with message |
CRD DB Error |
3002 |
010 |
CRD DB Error |
Retries exceeded |
3002 |
011 |
Retries exceeded |
No peer route |
3002 |
012 |
No peer routing rule found for a Realm-only or non-peer Destination-Host |
P-bit not set |
3002 |
013 |
P-bit in the Request message is set to “0” |
Missing Origin-Host AVP |
5005 |
014 |
Mandatory Origin-Host AVP missing |
Missing Origin-Realm AVP |
5005 |
015 |
Mandatory Origin-Realm AVP missing |
Missing Destination-Realm AVP |
5005 |
016 |
Mandatory Destination-Realm AVP missing |
No avp found in request for SLF lookup type |
3002 |
101 |
No avp found in request for SLF lookup type |
SLF DB Error |
3002 |
102 |
SLF DB Error |
SLF credential not found in DB |
3002 |
103 |
SLF credential not found in DB |
SLF Destination type not found in DB |
3002 |
104 |
SLF Destination type not found in DB |
Destination not found in SLF Mapping Table |
3002 |
105 |
Destination not found in SLF Mapping Table |
Step 1 | Run the following command to capture the packets on specific IP address:
admin@orchestrator[master-0]# debug packet-capture start ip-address 192.169.22.158 port 9100 timer-seconds 230 |
Step 2 | Run the following command to capture the packets on the host (executes tcpdump from host):
admin@orchestrator[master-0]# debug tcpdump an-dra-director-0 gen1.pcap 200s -i any port 3868 |
Step 3 | Run the following command to gather all the packet captures that were started:
admin@orchestrator[an-master]# debug packet-capture gather directory test1 |
You can view all the gathered packet captures at the following URL: https://<Master-IP>/orchestrator/downloads/debug/
Check for statistics generated at pcrfclient01/02 in /var/broadhop/stats and counters in beans at jmx terminal.
Policy Builder and CRD are inaccessible when there are multiple route entries on the master node.
This issue occurs only on OpenStack setups.
OpenStack Neutron configures multiple default routes, if the gateway is also present in the interfaces static configuration.
For example, when configuring multiple interfaces on any VM, set "gateway" for only one interface, preferably public interface.
# public network auto ens160 iface ens160 inet static address x.x.x.60 netmask 255.255.255.0 gateway x.x.x.1 # private network auto ens192 iface ens192 inet static address y.y.y.155 netmask 255.255.255.0
Run the following command to delete the default route to the internal network.
sudo route del default gw <internal network gateway IP>
For example: sudo route del default gw y.y.y.1
If the default route is not present for public network, run the following command:
ip route add default via <public network gateway IP>
For example: ip route add default via x.x.x.1