Use the following commands to start up the system after a maintaince window is completed and the VMs are powered on.

Perform a health check on both VNFs after the maintenance window is complete and the VMs are powered on. For more information, see System Health Checks.

In case of resiliency event of DB VMs, sometimes database status present on that VM takes time to update. This is due to the orchestrator thread which schedules show database status command. You need to wait for ¾ mins. This issue doesn’t always happen and is a rare one.

Non-compliant Diameter requests are checked for errors in routing AVP and P-bits. The following table describes the error codes and the reasons for errors in Diameter requests:

You can configure the HTTP response error code (such as 4xx, 5xx) corresponding to each vDRA Rest API JSON error response code for the GET binding (for example imsi, imsiApn, msisdn, msisdnApn, ipv4, ipv6). For more information about the CRD, see the CPS vDRA Configuration Guide.

If you do not configure the Rest API HTTP Error Code in the CRD, vDRA uses the default HTTP error codes for GET binding Rest API.

The following table lists the default HTTP error codes:

Run the following commands to check ping connectivity from the VM to other nodes using IPv4 and IPv6:

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# debug ping <wtc2b1fdrd02v> –n <IPv4 address>
admin@orchestrator# debug ping6 <wtc2b1fdrd02v> -n <IPv6 address>

Where:

• -n:

Run the following commands to check traceroute connectivity from the VM to other nodes:

IPv4:

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# debug traceroute <VMHOST> <IPv4address>

IPv6:

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# debug traceroute <VMHOST> -6 <IPv6address>

Use the following command to get packet capture from the VM. Specify interface and port details to avoid big packet capture files.

If you use the -i any option, you may see the same packet twice: once as it traverses the VMs interface, and again when it traverses the Docker container's virtual interface.

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# debug tcpdump wtc2b1fdrd01v test.pcap 
60s -s 0 -i ens162 port 3868
admin@orchestrator# debug packet-capture gather directory test_debug
admin@orchestrator# debug packet-capture purge

You can download the packer capture file from : https://<master ip>/orchestrator/downloads/ after logging in to https://<master ip>/

After you download the file, delete the packet capture files to clean up the disk space.

Use the following commands to monitor application logs :

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# monitor log application

Run the following command to capture SVN, CRD, logs, and save it at http://<master ip>/orchestrator/downloads/debug/tech/:

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# debug tech

Use the following command to monitor specific container logs:

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# monitor log container <container-name>

Use the following command to monitor orchestrator logs during an upgrade/downgrade:

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# monitor log container orchestrator 
| include AUDIT

If the CLI is not accessible or is giving errors when executing commands, use the following command from the master VM for more information:

cps@${DRM-hostname}:~$ docker logs orchestrator

If you know the existing password, use the following steps to change the user password in CLI:

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# aaa authentication users user fpasapi change-password
Value for 'old-password' (<string>): *****
Value for 'new-password' (<string>): ********
Value for 'confirm-password' (<string>): ********

If you do not know the password, use the following commands to reset the password:

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# config
admin@orchestrator(config)# aaa authentication users user fpasapi gid 100
 uid 9000 homedir "" ssh_keydir "" password <password>
admin@orchestrator(config-user-apiuser)# commit
Commit complete.
admin@orchestrator(config-user-apiuser)# end

If the commands show docker service or system diagnostics show errors, check the docker service for any unhealthy processes. If there are unhealthy processes, use the command monitor container logs to view logs and then restart the docker container.

Action
# node: DRA Master / DB Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# show docker service | tab | exclude HEALTHY
admin@orchestrator# show system diagnostics | tab | exclude passing
# container-name is unhealthy process container id. 
admin@orchestrator# docker restart container-id <container-name>

Check the VMs dnsmasq file to verify whether the DNS entries are present; if not, perform the following steps:

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cat /data/dnsmasq/etc/dnsmasq.conf
# If DNS entries are missing, perform the following steps:
cps@${DRM-hostname}:~$ cli
admin@orchestrator# show running-config network dns | 
save /data/config/dns_cli_backup
admin@orchestrator# config
admin@orchestrator(config)# no network dns
admin@orchestrator(config)# commit
admin@orchestrator(config)# end
admin@orchestrator# config
admin@orchestrator(config)# load merge /data/config/dns_cli_backup
admin@orchestrator(config)# commit
admin@orchestrator(config)# end
admin@orchestrator# exit
cps@${DRM-hostname}:~$ cat /data/dnsmasq/etc/dnsmasq.conf

When the master VM is deleted or redeployed for some reason, you must make it part of the existing cluster. workaround to make it part of the cluster as described in the following steps:

# node: DRA Master
# user: cps
# After the master VM is redeployed, log into the master VM, and wait til 
# cpsinstall is complete
cps@${DRM-hostname}:~$ journalctl -u cpsinstall.service –f
# Verify that the following log apperas: log <date time stamp> master-0 
bootstrap.sh[1521]: Install script completed.
# Once cpsinstall is finished; execute the following 
# commands on the master VM in the order specified. 
cps@${DRM-hostname}:~$ docker stop $(docker ps -a -q)
cps@${DRM-hostname}:~$ docker rm $(docker ps -a -q)
cps@${DRM-hostname}:~$ weave launch-router --ipalloc-init consensus=3
cps@${DRM-hostname}:~$ sudo rm -rf /data/orchestrator
cps@${DRM-hostname}:~$ sudo rm /var/cps/bootstrap-status
cps@${DRM-hostname}:~$ sudo /root/bootstrap.sh
cps@${DRM-hostname}:~$ ssh-keygen -f "/home/cps/.ssh/known_hosts" -R 
[localhost]:2024

Perform the following steps to remove a replica set member from MongoDB.

Caution

The command no database cluster deletes the configuration completely, so ensure the information is correct.

# node: DRA Master
# user: cps
cps@${DBM-hostname}:~$ cli
cps@${DBM-hostname}:~$ config
admin@orchestrator(config)# no database cluster binding shard 
binding-shard-1 shard-server fn6-1a1bs1k
admin@orchestrator(config)# commit
admin@orchestrator(config)# end
#connect to the replica set primary member container to remove the node, take a note 
#of port of the replica set
cps@${DBM-hostname}:~$ docker connect mongo-s104
root@mongo-s104:/# mongo --port 27033
rs-binding-shard-1:PRIMARY> rs.status()
#Note the name of the member from rs status output and then input it to 
#rs.remove to remove the member 
rs-binding-shard-1:PRIMARY> rs.remove
("[2606:ae00:2001:2420:8000::9]:27034")

The MongoDB logs are stored under /data/mongod-node/db on every VM that has mongod instance running.

Perform the following steps if you want to clean the database and recreate a fresh database.

Warning

All the data will be lost.

# node: DRA Master
# user: cps
cps@${DBM-hostname}:~$ cli
# Stop all the application process:
cps@${DBM-hostname}:~$ system stop
# Wait for some time till all the application proceses stop. 
# You can check the process using the commands:
# show scheduling status and show system status
# Repeat the following steps in all the database nodes
cps@${DBM-hostname}:~$ rm -rf /data/configdb/*
cps@${DBM-hostname}:~$ rm -rf /data/mongod-node/db/*
cps@${DBM-hostname}:~$ rm -rf /mmapv1-tmpfs-<port>/*
cps@${DBM-hostname}:~$ cli
# Restart the system:
cps@${DBM-hostname}:~$ system start

Perform the following steps to reset the CLI configuration:

Caution

The complete configuration will be reset.

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ docker exec -it orchestrator bash
cps@${DRM-hostname}:~$ /var/confd/bin/confd_load -D -m -l 
/data/cdb//*.xml

Use the orchestrator CLI to enable and disable application logs.

admin@orchestrator# logger set ?
Possible completions:
  <logger name>

admin@orchestrator# logger set com.broadhop.dra.service ?
Possible completions:
  debug  error  info  off  trace  warn

admin@orchestrator# logger clear com.broadhop.dra.service ?
Possible completions:
  |  <cr>

The different log levels in the order of increasing details in the log are:

• Error (error logs)

• Warn (error and warning logs)

• Info

• Debug

• Trace (all logs)

The default log level is warn.

Use the following orchestrator CLI command to view the current log levels set on a per application module basis.

admin@orchestrator# show logger level
Logger                         	Current Level
---------------------------------------------------
com.broadhop.dra.service               warn
dra.trace                 		warn
org.jdiameter                   	warn

To view application logs continuously similar to the tail –f command, use the following command:

“monitor log application”

To view application logs that were previously collected in a consolidated log file (similar to the more command), use the following command:

show log application

The following table describes the different loggers and their default log level:

Check for statistics generated at pcrfclient01/02 in /var/broadhop/stats and counters in beans at jmx terminal.

In case of call failures, check the Peer Connection, Binding Monitoring, Peer Errors, Error Result Code in Central GUI as described:

1. Log into the Central GUI as admin.

   In the Peer Monitoring, filter by the host where call failures are observed.

   If there is any problem with connection; that peer is not listed in Active Peer Endpoints screen and is listed in Inactive peers.

   

   

2. Check if the bindings are getting created. Filter the results for the imsiApn/ msisdnApn / ipv4/ ipv6 binding for which binding has to be retrieved.

   

3. Log into Central GUI/Grafana as admin and go to the Home > Application Summary.

   Check for specific errors in Grafana. The errors indicate the exact result code received from peer.

   

4. Log into Central GUI/Customer Reference Data as admin.

   Check for the descriptions of specific errors from customer reference data so that necessary action can be taken.

   

5. Log into Central GUI/Grafana as admin and go to the Home > Application Summary.

   Check for “discard” status in Grafana in DB Overload Handling graph. If entries are found in the graph, then check if maximum record limit has been set on database.

   

Use the following command to check traceroute connectivity from the VM to other nodes:

# node: DRA Director VM
# user: cps
cps@${drd-hostname}:~$ ping6 <Relay hostname configured in Policy Builder>

If there is any issue with the other vDRA, ping6 results in “timeouts.”

Use the following command to monitor exceptions in Redis or database:

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# monitor log application | include Exception

To check if there are any performance issues with vDRA, log into Central GUI as admin and check the System Health.

Monitor for any timeouts, spikes or decrease in TPS for database response times, peer response timeouts, average response timeouts.

Monitor session and binding records are uniformly distributed across all the shards from ‘Application Summary’ dashboard of DRA VNF.

Monitor Primary TPS is uniformly distributed across all the shards from 'Database Monitoring' dashboard of Binding VNF.

Use the following command to check for alerts and any issues with peer connections, low memory, low disk, or link failures.

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# show alert status | tab | include firing

Policy Builder and CRD are inaccessible when there are multiple route entries on the master node.

This issue occurs only on OpenStack setups.

OpenStack Neutron configures multiple default routes, if the gateway is also present in the interfaces static configuration.

For example, when configuring multiple interfaces on any VM, set "gateway" for only one interface, preferably public interface.

# public network
auto ens160
iface ens160 inet static
address x.x.x.60
netmask 255.255.255.0
gateway x.x.x.1

# private network
auto ens192
iface ens192 inet static
address y.y.y.155
netmask 255.255.255.0

sudo route del default gw <internal network gateway IP>

ip route add default via <public network gateway IP>

After rebooting the master and control VMs, if the Central GUI returns 503 service unavailable error, perform the following steps:

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# docker restart container-id haproxy-common-s101

Clear the browser cache and check the UI again.

If the MOG API calls fails internittently with an unauthroized message in a DRA director, then run the following commands to restart the container:

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# show network ips | include mogAPI  
admin@orchestrator# show docker service | tab | include drd02v | include haproxy-common-s
admin@orchestrator# docker restart container-id haproxy-common-s10x

If the DRA configuration API call returns null, restart the Policy Builder container as shown:

# node: DRA Master
# user: cps
cps@${DRM-hostname}:~$ cli
admin@orchestrator# show docker service | tab | 
include drc01v | include policy-builder-s
admin@orchestrator# docker restart container-id 
policy-builder-s10x

Issue: Initially, if you enter incorrect characters (encoding should be UTF-8 and not windows) as input to the expressions on the CLI (config), the subsequent configurations does not get reflected. This is because the confd does not accept incorrect characters in the confd Database and does not allow you to proceed.

Workaround: To recover, you must either remove or correct the incorrect expressions from the running configurations and then input the subsequent expressions to add the configurations.

Issue: Messages timed out when running Heap Dump of DRA process.

Condition: Taking heap dump of director and worker process. Heap dumps taken results in full GC. This in turn causes major application pause and results in message time out.

Solution: It is recommended to take the heap dump only during Maintenance Window (MW)

Issue: When Master VM is powered ON after 12 hrs it is stuck on orchestrator container throwing following logs:

2019-07-01T03:40:12.858+0000 I NETWORK [conn78] end connection 127.0.0.1:33586 (4 connections now open)
Waiting for 5s as mongod database is not yet up. Mon Jul 1 03:40:12 UTC 2019
2019-07-01T03:40:13.469+0000 I REPL [replexec-2] Canceling priority takeover callback
2019-07-01T03:40:13.469+0000 I REPL [replexec-2] Not starting an election for a priority takeover, since we are not electable due to: Not standing for election because member is not currently a secondary; member is not caught up enough to the most up-to-date member to call for priority takeover - must be within 2 seconds (mask 0x408)
2019-07-01T03:40:15.246+0000 I REPL [replexec-2] Scheduling priority takeover at 2019-07-01T03:40:26.616+0000
2019-07-01T03:40:17.919+0000 I NETWORK [listener] connection accepted from 127.0.0.1:33596 #79 (5 connections now open)

Solution: To recover the master VM, you need to execute the following commands on Master VM:

docker exec -it orchestrator bash
supervisorctl stop mongo
rm -rf /data/db/*
supervisorctl start mongo

Issue: Calls failure observed when database VNF VMs are in recovery mode.

Expected Behavior: Few call failures are expected when a shard-member recovers after restart and gets elected as new primary. The following is the expected behavior when a Binding VNF recovers after failover:

• All the shards members of the database VNF do not come up at the same time. They resynchronize with the existing shard-members and transition from STARTUP2 to Secondary to Primary state is not same for all the shards.

• Two elections for each shard are possible based on the database VM recovery time. The following is the sequence:

  1. First Election: Database VM having shard member with second highest priority completes the resynchronization first and becomes primary.

  2. Second Re-election: The shard member with highest priority completes the resynchronization and becomes primary (This behavior is as per the MongoDB replica-set protocol version 1 - pv1).

Issue: Timeout is observed when publishing is done during load.

Solution: Policy Builder publishing during load have impact on running calls.

Note	It is recommended to perform Policy Builder publishing during Maintenance Window (MW).

Issue: No users are authenticated after MongoDB auth set on DRA VNF.

Solution: After new password is set on binding VNF and DRA VNF if there are no users authenticated exception is observed, restart the binding container.

Issue: If mongod instance fails to start and displays the following error:

2019-08-26T07:11:48.012+0000 I -  [repl writer worker 7] Fatal assertion 16359 
NamespaceNotFound: Failed to apply insert due to missing collection: 
{ ts: Timestamp 1566803477000|424, t: 15, h: -4599534508984183152, v: 2, op: "i", 
ns: "ipv6bindings.ipv6bindings", o: { _id: "3001:0000:0019:3dd5", ts: 1566804446889, 
staleBindingExpiryTime: new Date(1566815246889), srk: "fPAS.CALIPER.PCRF4", fqdn: 
"client-calipers24-gx.pcef.gx", sessionid: "ClpGx3:172.16.241.40:5024:1566562707:0021653880", 
uuid: "fpas-system-22133581349" } } at src/mongo/db/repl/sync_tail.cpp 1059
2019-08-26T07:11:48.012+0000 I -        [repl writer worker 7]
 
***aborting after fassert() failure

Solution: Restart the MongoDB instances manually giving JIRA reference of mongod instance.

Issue: When Prometheus servers residing on Master/Control VMs are restarted, alerts are raised by the servers remaining in firing state. If the alerts are not resolved before Prometheus restart, the alerts remain in firing state forever even after issue is resolved.

For example,

admin@orchestrator[fPAS-site2-master-1]# show alert status | tab | include IP_NOT
IP_NOT_REACHABLE  172.26.50.114    firing    VM/VIP IP 172.26.50.114 is not reachable!    2019-08-28T14:52:49.776+00:00  -  2019-08-29T12:03:03.553+00:00
IP_NOT_REACHABLE  172.26.50.64     firing    VM/VIP IP 172.26.50.64 is not reachable!     2019-08-28T14:51:54.768+00:00  -  2019-08-29T12:03:03.548+00:00
IP_NOT_REACHABLE  172.26.50.65     firing    VM/VIP IP 172.26.50.65 is not reachable! 
admin@orchestrator[fPAS-site2-master-1]# show running-config docker | tab
ID                               ADDRESS                                                    SCHEDULING SLOTS  MODE            ID        ADDRESS
-------------------------------------------------------------------------------------------------------------------------------------------------------
fPAS-site2-app-persistence-db-7  http://engine-proxy-6662e4dc999a9e36f4f0ea2d0fbfcedf:2375  [ mongo-node ]    [ mongo-node ]  internal  172.26.50.114
fPAS-site2-control-2             http://engine-proxy-89fa384df65a6c7863252a22fcbfd696:2375  [ control-a ]     -               internal  172.26.50.65
fPAS-site2-master-1              http://engine-proxy-03b546144fadc57dc51c0fcd063375da:2375  [ master ]        -               internal  172.26.50.64
 
 
fPAS-site2-app-persistence-db-7  CONNECTED  0
fPAS-site2-control-2             CONNECTED  0
fPAS-site2-control-3             CONNECTED  0
fPAS-site2-master-1              CONNECTED  0

Solution: Manually clear the alarms using CLI.

To check if this behaviour is present on a system.

1. Login to Promethues container using CLI.

   docker connect prometheus-hi-res-s<101/102/103>
   ps aux
   

Note	Note down Prometheus start time. If it is later than “alerts” firing time, then this alert will never get cleared.

Issue: When running a call-model, it is observed that all the RAR messages initiated from server-side are routed over relay connection to other sites. From logs, it is seen that the peer is detected as active but due to some reason the siteID is detected as NULL due to which the neighbor siteID is preferred to route the messages.

Condition: 4 site cluster with peer_group and peer-route configured on the basis of FQDN HOST.

Solution: Restart the diameter-endpoint and binding containers on DRA_VNF. It can be seen from the Grafana if the response time is higher than usual for PCRF RAR traffic.

Issue: After system stop/upgrade/downgrade or site/VM restart, NTP related system diagnostics messages are observed.

Sample output with diagnostics messages:

admin@orchestrator[master-4]# show system diagnostics | tab | exclude passing
NODE CHECK ID IDX STATUS MESSAGE 
-------------------------------------------------------------------------------------------
control-plane-s101 serfHealth 1 critical Agent not live or unreachable 
ntpd-s102 service:ntp-client 1 warning unsynchronised 
ntpd-s102 service:ntp-client 2 warning polling server every 8 s 
ntpd-s103 service:ntp-client 1 warning unsynchronised 
ntpd-s103 service:ntp-client 2 warning polling server every 8 s 
ntpd-s104 service:ntp-client 1 warning unsynchronised 
ntpd-s104 service:ntp-client 2 warning polling server every 8 s 
ntpd-s105 service:ntp-client 1 warning unsynchronised 
ntpd-s105 service:ntp-client 2 warning polling server every 8 s

Observation: It is the NTP protocol behavior that it takes time to synchronize based on the time difference between the configured NTP server and local NTP client running on containers.

As it takes some time to synchronize with the NTP server, the diagnostics messages are expected to be seen during this time period. The NTP sync issues can make diameter / binding endpoints out of sync with the connected peers, which results into call failures. If the system diagnostics are still showing up ntp-related messages, then contact your Cisco Account representative.

During this time period you can execute the following commands on the master VM:

#vmware-toolbox-cmd timesync status

If time sync status is not enabled already, you can enable time it by executing vmware-toolbox-cmd timesync enable command on master VM.

On the remaining VMs, time sync status must be disabled by executing vmware-toolbox-cmd timesync disable command.

Sample output: The system diagnostics should not contain any ntp-related messages after the NTP sychronization is complete.

Issue: During system stop/start or upgrade, container on a VM isn't removed by the orchestrator.

Possible Cause: There are multiple reasons for this issue. The following are some of the causes identified:

1. Sometimes docker takes time to stop container than expected. In this case, orchestrator which is trying to remove the container times out and does retry. Same situation gets repeated continuously and hence container never gets removed.

2. Sometimes during container stop, the container isn’t stopped cleanly as volumes attached to it does not get deleted. In this case, the new container fails to start due to old volumes.

3. Sometimes during container stop, the container isn't stopped cleanly so if the docker proxy is exposing any ports for that container, those ports still remain in listening state. Hence when the new containers are started and docker proxy starts listening on exposed ports, it gets bind failures and thus the container doesn’t get started.

Solution:

1. Execute system abort-upgrade command from CLI to abort the running upgrade.

2. In case of system stop/upgrade, if the container doesn't get removed, perform the following steps:

   1. Find VM from where the containers doesn't get removed.

      Example: Execute the following command from CLI.

      show scheduling status | tab | exclude RUNNING
      monitoring 106 125 application SCHEDULING false

      monitoring 106 125 application SCHEDULING false indicates the instance ID

      admin@orchestrator[vpas-A-dra-master-0]# show docker service | tab | include 106
      diameter-endpoint 106 diameter-endpoint 19.5.0-20200506_032527.7339 
      vpas-A-dra-director-c diameter-endpoint-s106 ABORTED false

      vpas-A-dra-director-c diameter-endpoint-s106 ABORTED false indicates the VM name. In this example, the VM name is vpas-A-dra-director-c.

   2. Login to VM and use docker ps | grep <container-name> and timestamp of container to check if the container doesn't get removed.

   3. If the container is not removed, execute docker rm -f <container-name> to remove the container forcefully.

3. SSH to VM having issues and verify if the old volumes doesn't get deleted with the following steps:

   1. List all the volumes created using docker volume ls command.

      Example:

      cps@sk-persistence-db-1:~$ docker volume ls
      DRIVER              VOLUME NAME
      local               4b1e0622f0d774003c14eec9f17b98035445ef15f34fc055ebeb1aad572f1de3
      local               7d7d3b4b0ec4773427530b987475861106bc6d56dfc107feabce3f6c2afda875
      local               82e48b69a9b6687e54e8048aff2cc7af81c6754b1c44830f3e051e0fcfaaf380
      local               cps-docker-info
      local               cps-keepalived-conf
      local               cps-keepalived-service-status
      local               cps-keepalived-supervisord
      local               d7b892a933a915774ac1c883c21b70bddfea002d46dfa457dbda5f1baa0af55e
      local               faaeea115326b2981bd09f4f53fce157c1c6f327f7425ea27f92d5a371d8fcee

      Execute the following commands only for name containers. For example, cps-docker-info.

   2. Inspect volume using docker inspect <volume_name> command.

      cps@sk-persistence-db-1:~$ docker inspect cps-docker-info
      [
          {
              "CreatedAt": "2020-07-31T10:28:06Z",
              "Driver": "local",
              "Labels": {
                  "com.broadhop.orchestrator.id": "test-system"
              },
              "Mountpoint": "/var/lib/docker/volumes/cps-docker-info/_data",
              "Name": "cps-docker-info",
              "Options": {
                  "device": "tmpfs",
                  "o": "size=64m",
                  "type": "tmpfs"
              },
              "Scope": "local"
          }
      ]
      

   3. You can delete the volumes where labels is null using docker volume rm <volume_name> command.

      Example:

      docker volume rm cps-docker-info

      Volume not attached to any other volume will be safely deleted.

4. Reboot VMs.

   sudo reboot

5. Verify whether VM is joined to master VM and all the containers are scheduled as expected using show system status and show docker engine commands from CLI.

   admin@orchestrator[sk-master-binding-0]# show system status 
   system status running true
   system status upgrade false
   system status downgrade false
   system status external-services-enabled true
   system status debug false
   system status percent-complete 100.0
   admin@orchestrator[sk-master-binding-0]# show docker engine
   ID STATUS PINGS
   -----------------------------------------
   sk-control-binding-0 CONNECTED 0
   sk-control-binding-1 CONNECTED 0
   sk-master-binding-0 CONNECTED 0
   sk-persistence-db-1 CONNECTED 0
   sk-persistence-db-2 CONNECTED 0
   sk-persistence-db-3 CONNECTED 0
   sk-persistence-db-4 CONNECTED 0

   The system status percent-complete must be 100.0. The status of all the dockers must be connected.

Issue: show network ip command is not able to fetch all IPs information.

Condition: The issue happens when setups are migrated from older software (like, 18.2.0 or 19.4.0) where docker-host-info-monitor support was not present.

Possible Cause: docker-host-info-monitor containers are not coming up automatically.

show docker service | tab | include docker-host-info-monitor

Solution: Make sure all the VMs are running docker-host-info-monitor container.

If the container is not running, perform the following steps:

1. Login to the master VNF VM using ssh or console.

2. Login to orchestrator containe using docker exec -it orchestrator bash command.

3. Stop orchestration-engine service using supervisorctl stop orchestration-engine command.

4. Remove monitoring modules from database using:

   mongo orchestration --eval 'db.modules.remove({"_id": /^monitoring/})'
   mongo orchestration --eval 'db.services.remove({"_id": /^monitoring/})'

5. Start orchestration-engine service using supervisorctl start orchestration-engine command.

Issue: After downgrade tmpfs partitions names are not displayed correctly in mongo containers.

Scenarios:

• Downgrade from 20.2.0 release with tmpfs partition changes to tmpfs without changes

• Downgrade from 20.2.0 release to an older release

Condition: After completing downgrade, check whether mongo-s<instance-id> containers have unused tmpfs-partitions using the following command from CLI:

docker exec mongo-s1 "df -h"

Run the steps mentioned in the Solution if any container output shows shard word in tmpfs partitions names.

Note	tmpfs partition names should only have port numbers.

Here is a sample output.

==========output from container mongo-s105===========
tmpfs           8.3T     0  8.3T   0% /mmapv1-tmpfs-rs-shard-41
tmpfs           8.3T     0  8.3T   0% /mmapv1-tmpfs-rs-shard-21
tmpfs           8.3T     0  8.3T   0% /mmapv1-tmpfs-rs-shard-42
tmpfs           8.3T     0  8.3T   0% /mmapv1-tmpfs-rs-shard-22
tmpfs           8.3T     0  8.3T   0% /mmapv1-tmpfs-rs-shard-2
tmpfs           8.3T     0  8.3T   0% /mmapv1-tmpfs-rs-shard-1

Solution:

1. Create a file cleanup-tmpfs-partitions name and add the following contents.

   #!/bin/bash
   for CONTAINER in $(/var/broadhop/cli/list-containers mongo-s); do
     if [ ${CONTAINER} == "mongo-status" ];then
       continue
     fi
     ENGINE=`/var/broadhop/cli/resolve-container-engine ${CONTAINER}`
     if [ ! -z "${ENGINE}" ]; then
       DOCKER_CONNECTION=`/var/broadhop/cli/resolve-docker-connection ${ENGINE}`
       if [ ! -z "${DOCKER_CONNECTION}" ]; then
          echo "container name... ${CONTAINER}"
          docker -H ${DOCKER_CONNECTION} cp /var/broadhop/cli/cleanupTmpfs.sh ${CONTAINER}:/usr/local/bin/
          docker -H ${DOCKER_CONNECTION} exec -d ${CONTAINER} /usr/local/bin/cleanupTmpfs.sh
       fi
     fi
   done

2. Create a file cleanupTmpfs.sh name and add the following contents.

   #!/bin/bash
   
   count=$(df -h | grep shard | wc -l)
   echo "partitions to delete.... ${count}"
   while [ ${count} -ne 0 ]
   do
   echo "TRYING TO UNMOUNT try....${count}"
     for PARTITION in $(df -h | grep shard | tr -s ' '| cut -f 6 -d ' '); do
       echo "trying to unmount ${PARTITION}"
       umount -f ${PARTITION}
       rm -rf ${PARTITION}
     done
   
   sleep 3
   count=$(df -h | grep shard | wc -l)
   done

3. Change the permissions for both the files.

   chmod +x cleanup-tmpfs-partitions
   chmod +x cleanupTmpfs.sh

4. Copy the files to an orchestrator container.

   docker cp cleanupTmpfs.sh orchestrator:/var/broadhop/cli/
   docker cp cleanup-tmpfs-partitions orchestrator:/var/broadhop/cli/

5. Execute cleanup-tmpfs-partitions command.

   docker exec -it orchestrator /var/broadhop/cli/cleanup-tmpfs-partitions

6. Verify the cleanup of tmpfs-partitions by executing the following command from CLI.

   docker exec mongo-s1 "df -h"

   Make sure that there is no tmpfs partitions with name shard word present in any of the container outputs. tmpfs partition names should only have port numbers.

   ==========output from container mongo-s105===========
   tmpfs            26G   49M   26G   1% /mmapv1-tmpfs-27026
   tmpfs            26G   49M   26G   1% /mmapv1-tmpfs-27027
   tmpfs            26G   49M   26G   1% /mmapv1-tmpfs-27036
   tmpfs            26G   49M   26G   1% /mmapv1-tmpfs-27037
   tmpfs            26G   17M   26G   1% /mmapv1-tmpfs-27046
   tmpfs            26G   49M   26G   1% /mmapv1-tmpfs-27041
   tmpfs            26G   49M   26G   1% /mmapv1-tmpfs-27042
   admin@orchestrator[an-dbmaster]#

Issue: Binding/endpoint containers aren’t displayed when executing show docker service command.

Condition: Discrepancies with the blades hosting VMs.

Solution: Execute show docker engine command. All the VMs must be in connected state. If any VM isn’t CONNECTED, make sure to bring that VM UP.

Check the blade status (by accessing through vSphere) which is hosting the impacted VMs. Make sure that the blade is up and running fine.

Refer to the following additional details:

• Orchestrator controls scheduling of the containers and also manages the high availability. If orchestrator goes down, system can’t perform any activity.

• Consul is responsible for service discovery. It also shares the service details and state for health checks. If consul cluster isn’t running properly, then VNF lacks these features supported by consul.

• Admin DB holds backup of Grafana, SVN. In addition, it also maintains the customer-specific reference data which is used to serve TPS processing. If admin database isn’t available, it creates discrepancies in the call flows.

• Policy Builder provides GUI to add/update the policy configurations. It displays information such as peer monitoring, CRD table contents. If PB isn’t working, you’re not able to perform any configurations.

Issue: VM is stuck in JOINING state.

Analysis: From CLI, execute the following commands to display VMs status:

admin@orchestrator[Master-hostname]# show docker engine | tab | exclude CONNECTED
an-pers-db-0    JOINING  234
an-pers-db-1    JOINING  452

The following are possible reasons for VM to get stuck in JOINING state.

• VM is not reachable from master VM.

• Timestamp of VM having issue is not in sync with the master VM.

Solution 1 :

1. If the VM is not reachable from master VM, reboot the VM.

   sudo reboot

2. If the timestamp of VM having issues is not in sync with the master VM, reboot the VM.

   sudo reboot

3. Verify whether the VM is moving to CONNECTED state.

Solution 2: After reboot if the VM is still in the JOINING state, peform the following steps:

Note	The following steps must not be executed on master VM.

1. Remove old containers/volumes from the VM.

   docker stop $(docker ps -aq)
   docker rm -f $(docker ps -aq)
   sudo rm -rf /data/orchestrator
   sudo rm /var/cps/bootstrap-status

   For more information, refer to Docker Volumes Deletion section.

2. Execute bootstrapping command.

   sudo /root/bootstrap.sh

Issue: Consul containers fail when DRA is downgraded using the following command:

system downgrade version <old-dra-version>

Condition: Older incompatible consul version was used in old DRA image.

Solution:

1. Monitor consul containers using monitor log container <consul-container-name> command.

   Example:

   monitor log container consul-1
   .....
   monitor log container consul-7

2. Check for the following error in the consul container logs.

   Error starting agent: Failed to start Consul server: Failed to start Raft: failed to load any existing snapshots

3. Login to the consul container having errors using docker connect <consul-container-name> command.

   admin@orchestrator[site1-master-1]# docker connect consul-1
   root@consul-1:/#

4. Run supervisorctl status command from the consul container you logged in Step 3.

   root@consul-1:/#supervisorctl status
   consul-server                    FATAL     Exited too quickly (process log may have details)
   root@consul-1:/#

5. Clear the consul state and restart consul server using the following commands.

   root@consul-1:/# rm -rf /data/*
   root@consul-1:/# supervisorctl start consul-server
   consul-server: started
   root@consul-1:/# exit

Repeat Step 3 to Step 5 for all the consul containers that have failed.

Issue: When the number of peers connected to DRA is high (for example, 1500 peer connections), Grafana is slow (> 2 minutes) in rendering peer statistics. The delay in loading peer statistics increases the overall load time of the dashboard. For longer time ranges (for example, 24 hours), peer statistics charts could fail to load.

If Traffic By Peers panel is collapsed/minimized, dashboard loads within acceptable time.

Cause: With large number of peer connections (for example, 1500), volume of data queried to render peer statistics charts increases. This increases the time to load data and render the charts.

Solution: Load time of peer statistics charts can be improved by increasing the interval of data rendered in the charts. Duplicate the peer statistics query and create two additional queries. Change the minimum interval for the new queries to 5 mins and 10 mins respectively.

To analyze peer statistics for time ranges 1 - 6 hours, enable the query with 5 mins interval. To analyze peer statistics for time ranges > 6 hours, enable query with 10 mins interval.

To avoid delays in loading Application Summary:

• By default, all the panels (Local Peer Traffic / Remote Peer Traffic / Active Peer List / Peer Response Time / Peer Connections) under Application Summary > Traffic By Peers are disabled.

• The panels are moved under new dashboard Peer Traffic Monitor.

• Recording rules (local_peer_sum_rate, remote_peer_sum_rate, active_peers_list) are added to precompute the data and serve whenever this data is queried. By default, Trending and Planning data is disabled for these new recording rules.

When high number of peers is configured:

• Use the default expressions to query data upto max of last 2 days.

• Use the old expressions to query data beyond the last 2 days (Trending / Planning). It is recommended to use old expressions for smaller time chunks (< 6hrs) to avoid delays in loading Grafana.

Issue: Shard count is displaying more than one primary.

Condition: This happens when two primaries are encountered for the same shard in show database status command.

Solution: The following solution is to fix the database showing two primaries for the same shard, which is an edge scenario and is encountered when containers are started (upgrade, VM restarted or fresh installation):

Let shard-<ip>-<port>-MMAPv1 be the shard process running for primary in the shard, then:

cps@vpas-site-persistence-db-*:~$ docker exec -it mongo-s* bash
root@mongo-s106:/# supervisorctl stop shard-<ip>-<port>-MMAPv1
root@mongo-s106:/# rm -rf mmapv1-tmpfs-<port>/*
root@mongo-s106:/# supervisorctl start shard-<ip>-<port>-MMAPv1

Issue: Missing spikes in longevity report for 12 hours and more.

Solution: Change panel width to maximum value (24). The following screenshots are for reference purpose.

You can get all the spikes captured for 6 hours duration. So, if you need to analyse longevity report for 12 hours or more, you can grep data by grouping in 6 hours interval. The following screenshots are for reference purpose.

Issue: In consolidated-qns.log warning logs GzipFilterCustom un-expected event occurred : Committed is displayed followed by any URL.

Condition: Jetty server causes this exceptions. It's related to the working of the javax.servlet.http classes. This error can occur for many User Interfaces such as, API, CSS, JSS, gif, png fetching and so on.

Precautions: Close or refresh the associated DRA GUI URL page which is open in the browser for a long time.

Solution: Identify the URL which is mentioned in the logs and try to close or refresh the browser page associated with URL.

For example: GzipFilterCustom un-expected event occurred Committed : https://10.197.97.87/proxy/dra/api/localActivePeerEndpoints.

In this case, DRA Peer Monitoring page is open for long time, hence it is throwing java.lang.IllegalStateException. Close the page or refresh the page.

Issue: After fresh installation, scheduling is stuck at cc-monitor due to delay in insertion for license in mongo-admin.

Workaround: Execute the following to resolve this issue.

cps@vpas-A-master:~$ docker exec -it mongo-admin-a bash
root@mongo-admin-a:/# mongo
rs:PRIMARY> use sharding
switched to db sharding
rs:PRIMARY> show collections
licensedfeats
rs:PRIMARY> db.licensedfeats.insert({ "_id" : ObjectId("620a232db0f6e75a85c26697"), "featurename" : "SP_CORE", "licatt" :
"3FC5CA0ED1FCD8DABE3B9B24078DB749B0CBAA5A686436B234B53A3BDEF7676AEF777
CD2FD215B2BDE0E9392D417F8D0F2D162A6CEA04E924D2432BCC6B006D1AF98B9A9EACAFEF721DECB4C4195C020" })
WriteResult({ "nInserted" : 1 })
rs:PRIMARY> db.licensedfeats.find()
{ "_id" : ObjectId("620a232db0f6e75a85c26697"), "featurename" : "SP_CORE", "licatt" :
 "3FC5CA0ED1FCD8DABE3B9B24078DB749B0CBAA5A686436B234B53A3BDEF7676AEF777
CD2FD215B2BDE0E9392D417F8D0F2D162A6CEA04E924D2432BCC6B006D1AF98B9A9EACAFEF721DECB4C4195C020" }

Login to the CLI to view the following outputs.

docker exec cc-monitor "supervisorctl restart app"
==========output from container cc-monitor-s102===========
app: stopped
app: started
==========output from container cc-monitor-s103===========
app: stopped
app: started 

Issue: The following error may occur when Import/Publish operation is performed in Policy Builder.

Pristine text \xxxxxxxxxxxxxx\xxxxxxxxxxxxx not present\n

Solution: During import/publish, the repository is corrupted. You can create a new repository with PB backup and publish it with the desired changes.

1. Log in to CPS Central using <master-ip>/central/dra/.

   

2. Click Import/Export under Policy Builder.

3. Select Import tab. Click File to Import and select the PB backup file to import.

   

4. Enter the Import URL .

   Note	It is strongly suggested to import to a new URL and use Policy Builder to verify/publish).

5. Click Import to import the file.

6. From main page, click Policy Builder to open Choose Policy Builder Data Repository.

   

7. Click Add New Repository to open Add Repository window.

   

   Enter the name of the Repository, URL, and Local Directory.

   Note	Use the correct URL. This URL should be same as added in Step 3.

8. Log in again to CPS Central using <master-ip>/central/dra/. Click on Policy Builder and select your newly created repository.

9. Edit the Policy Builder configuration with the changes which were done for last corrupted repository to resolve the corrupted repository issue.

10. Save the changes and publish the updated configuration.

Issue: After redeployment or power ON/OFF of VMs (also observed in resiliency tests), system intermittently comes up as not healthy and VMs can move to JOINING state. In such case, weave status is displayed as waiting for IP range grant from peers and weave status ipam command shows unreachable MACs for few VMs (same VM has 2 MACs reachable and unreachable).

Solution: After each of the above scenarios (redeployment/resiliency), if any unreachable MACs are seen on VMs, perform the following steps on the same VM:

1. Remove peers that are unreachable.

   weave rmpeer <unreachable mac id>

2. Bootstrap the VM.

   docker stop $(docker ps -aq)
   docker rm -f $(docker ps -aq)
   sudo rm -rf /data/orchestrator
   sudo rm /var/cps/bootstrap-status
   sudo /root/bootstrap.sh

Issue: After login to DRA Central GUI with valid admin credentials, sometimes admin role is changed to read-only.

Solution: The following workarounds should be applied in sequential manner.

Note	Take backup of SVN repository before peforming any operation on SVN.

Workaround1: Check the disk space. The user in ReadOnly mode can also be due to the disk full issue on Control VMs. Free some space to recover from this problem.

Workaround2: Browser cache issue.

This workaround should be applied only if Workaround1 doesn't work.

1. Open DRA Central GUI in another browser.

   Or

   Delete the browser cache and restart the browser.

2. Login into DRA Central GUI again with valid admin credentials. You should have admin rights.

Workaround3: .broadhopFileRepository missing.

This workaround should be applied only if Workaround2 doesn't work.

1. Check .broadhopFileRepository in the client SVN repository.

   For example, current published repo URL is https://<master-ip>/repos/configuration/ .

2. Go to <master-ip>/repos/configuration in browser and check if .broadhopFileRepository file is missing.

   If .broadhopFileRepository file is missing, use the following steps to add .broadhopFileRepository file.

   1. Log in into SVN container using the docker exec -it svn bash command.

      This command works from control VM where SVN container exists.

   2. Create a blank .broadhopFileRepository file using the touch .broadhopFileRepository command.

   3. Import the file into repository using the following command.

      svn import --username <username> --password <password> --force -m "adding broadhop file" --no-auth-cache .broadhopFileRepository http://svn/repos/configuration/.broadhopFileRepository

3. Re-login to DRA Central GUI with valid admin credentials. You should now have admin rights.

Workaround4: Delete .broadhopFileRepository file and add the file again.

This workaround should be applied only if Workaround1 and Workaround3 doesn't work.

1. Log in into SVN container using docker exec -it svn bash command.

   This command works from control VM where SVN container exists.

2. Take backup of SVN repository.

3. Considering that you are using …/configuration SVN URL, delete .broadhopFileRepository file.

   If you are not using …/configuration SVN URL, change URL accordingly and use the following command to delete the file.

   svn delete --username <username> --password <password> --force --no-auth-cache http://svn/repos/configuration/.broadhopFileRepository --message "deleting file"

4. If you have backup of .broadhopFileRepository file, import the file in the repository using the following command:

   svn import --username admin --password admin --force -m "adding brodhop file" --no-auth-cache .broadhopFileRepository http://svn/repos/configuration/.broadhopFileRepository

   If you don't have backup of .broadhopFileRepository file, create a blank file with name .broadhopFileRepository and import the file in the repository using the following command:

   svn import --username admin --password admin --force -m "adding brodhop file" --no-auth-cache .broadhopFileRepository http://svn/repos/configuration/.broadhopFileRepository

5. Re-login to DRA Central GUI with valid admin credentials. You should now have admin rights.

The following command provides database operational status of all database clusters configured. Execute the command in operational mode.

show database status | tab

Example:

admin@orchestrator[an-dbmaster]# show database status | tab
                                                          CLUSTER
ADDRESS        PORT   NAME        STATUS     TYPE         NAME     SHARD     REPLICA SET
----------------------------------------------------------------------------------------------
192.168.11.42  27026  arbiter-21  ARBITER    replica_set  session  shard-21  rs-shard-21
192.168.11.43  27026  server-x    PRIMARY    replica_set  session  shard-21  rs-shard-21
192.168.11.44  27026  server-y    SECONDARY  replica_set  session  shard-21  rs-shard-21
192.168.11.42  27027  arbiter-22  ARBITER    replica_set  session  shard-22  rs-shard-22
192.168.11.43  27027  server-x    SECONDARY  replica_set  session  shard-22  rs-shard-22
192.168.11.44  27027  server-y    PRIMARY    replica_set  session  shard-22  rs-shard-22
192.168.11.43  27019  session     PRIMARY    shard_db     session  shdb-4    session-sharddb
192.168.11.44  27019  session     SECONDARY  shard_db     session  shdb-5    session-sharddb

admin@orchestrator[an-dbmaster]#

1. Execute the following command to find sharding database PRIMARY of particular cluster.

   show database status cluster-name session | tab | include PRIMARY | include shard_db

2. Connecting to sharding database primary member.

   Non-Mongo Auth:

   mongo --ipv6 mongodb://[2606:ae00:2001:230b::2b]:27019

   Mongo Auth:

   mongo --ipv6 mongodb://adminuser:<password>@[2606:ae00:2001:230b::2b]:27019/admin

3. After successfully connecting to sharding database primary member, execute the following step:

   For example, to validate DRA sessions sharding database metadata information:

   session-sharddb:PRIMARY> use drasessionsShardDB
   switched to db drasessionsShardDB
   session-sharddb:PRIMARY> db.shards.count()
   2
   session-sharddb:PRIMARY> db.buckets.count()
   8192
   session-sharddb:PRIMARY>
   

1. Execute the following command to find sharding database PRIMARY of particular cluster:

   show database status cluster-name session | tab | include PRIMARY | include shard_db

2. Connecting to sharding database primary member.

   Non-Mongo Auth:

   mongo --ipv6 mongodb://[2606:ae00:2001:230b::2b]:27019

   Mongo Auth:

   mongo --ipv6 mongodb://adminuser:<password>@[2606:ae00:2001:230b::2b]:27019/admin

3. Validate configured shard and zone mappings.

   Example:

   use ipv6ShardDB
   switched to db ipv6ShardDB
   binding-sharddb:PRIMARY>
   binding-sharddb:PRIMARY> db.shards.find()

   

4. Validate configured zones and ranges.

   Example:

   

5. Validate buckets/shard/range mapping.

   Example:

   

1. Make sure the database status of all the shards and sharding database members comes up with either PRIMARY, SECONDARY or ARBITER.

   show database status | tab

2. If the database status is not PRIMARY, nor SECONDARY or ARBITER, login to specific VM and check whether mongod instance is running appropriate options to enable mongo authentication or not.

   Example for working mongod instance with authentication enabled on it:

   root     15379     1  4 03:54 ?        00:16:49 mongod --keyFile=/mongodb.key 
   --storageEngine mmapv1 --nojournal --noprealloc --smallfiles --ipv6 --bind_ip_all 
   --port 27023 --dbpath=/mmapv1-tmpfs-27023 --replSet rs-shard-13 --quiet --slowms 500 
   --logpath /data/db/mongo-27023.log --oplogSize 3221 --logappend --logRotate reopen

3. If the key file is present, but the database status is not good, check whether user exists or not for that mongod instance.

   Example:

   1. Login to VM and its mongo container (container name : mongo-s<instance>).

   2. Connect to mongod with its port.

   3. Use admin user and execute the following command:

      Note	The db.getUsers() command should display adminuser and backupuser for all the non-aribiter members. For arbiter member users, connect to PRIMARY of that shard and execute db.getUsers() command.

      session-sharddb:PRIMARY> db.getUsers()
      [
              {
                      "_id" : "admin.adminuser",
                      "user" : "adminuser",
                      "db" : "admin",
                      "roles" : [
                              {
                                      "role" : "root",
                                      "db" : "admin"
                              }
                      ]
              },
              {
                      "_id" : "admin.backupuser",
                      "user" : "backupuser",
                      "db" : "admin",
                      "roles" : [
                              {
                                      "role" : "root",
                                      "db" : "admin"
                              }
                      ]
              }
      ]
      

4. If users are present, check whether mongo connection gets established manually or not by executing the following command:

   Example for Mongo Auth:

   mongo --ipv6 mongodb://adminuser:<password>@[2606:ae00:2001:230b::2b]:27019/admin

5. To validate the configurations and operational status of mongod instance, execute the following commands:

   db-authentication show-password database mongo password
   db-authentication rolling-restart-status database mongo password
   

Issue: In shard replica-set, if all the data bearing members are down at same time, they fail to elect new Primary and gets stuck in STARTUP2 state.

Run show database status | tab command on Binding VNF to check for shards without Primary member.