The SMF supports High Availability (HA) of UDP proxy. The HA model of UDP proxy is based on the keepalived virtual IP concepts.

For more information on UDP proxy redundancy, see the High Availability for the UDP Proxy section in the Pods and Services Reference chapter.

The SMF supports IPAM redundancy and load balancing for each UPF. The IPAM running in the Node Manager microservice has two IPAM instances that are associated to each UPF. When one IPAM instance is inactive, the other IPAM instance manages the address allocation requests for the UPF.

For more information on node manager redundancy, see the IPAM Redundancy Support Per UPF section in the IP Address Management chapter.

This section describes the deployment of SMF pods and its microservices.

The following figure shows the deployment model of six VMs in SMF.

In this model, the pods are deployed on VM pairs. Two replicas are available for each protocol pod (for example, rest-ep, protocol-ep, and gtp-ep). One instance is deployed on each protocol VM.

Similarly, service pods and session pods are distributed on both the service and session VMs equally. Such a distribution is controlled by labelling the VMs as well as implementing the K8 affinity and anti-affinity rules during pod scheduling.

This model ensures that, during VM reboot scenarios, at least 50% of the replicas of each pod type are available to handle user signaling.

Graceful pod restart allows pod to complete ongoing processing within 30 seconds. Abrupt pod restart will affect ongoing transactions without impact to PDU sessions.

The following is an example of VM labelling and replica configuration.

k8 label protocol-layer key smi.cisco.com/node-type value smf-proto
exit
k8 label service-layer key vm-type value smf-svc
exit
k8 label cdl-layer key smi.cisco.com/node-type value smf-cdl
exit
k8 label oam-layer key smi.cisco.com/node-type value oam
exit

endpoint pfcp
 replicas 1
 nodes    2
exit
endpoint service
 replicas 1
 nodes    2
exit
endpoint protocol
 replicas 1
 nodes    2
 vip-ip 209.165.201.28
exit
endpoint sbi
 replicas 1
 nodes    2

To verify the configuration, use the following show command:

show running-config instance instance-id instance_id endpoint 

The following is an example output of this show command.

[unknown] smf# show running-config instance instance-id 1 endpoint 
instance instance-id 1
 endpoint nodemgr
  replicas 1
  nodes    2
 exit
 endpoint gtp
  replicas 1
  vip-ip 209.165.202.149
 exit
 endpoint pfcp
  replicas                2
  enable-cpu-optimization true
  interface n4
   heartbeat
    interval               0
    retransmission-timeout 3
    max-retransmissions    5
   exit
  exit
 exit
 endpoint service
  replicas 2
 exit
 endpoint protocol
  replicas 1
  vip-ip 209.165.202.149
 exit
exit 

This command output displays the configurations related to multiple endpoints, such as endpoint names, pod replicas, nodes, and so on.

In active-active mode,

• GR deployment is transparent to the adjacent NFs.

• GR deployment contains two instances of CCG function, each instance manifest itself with a set of interface IPs.

• Each instance support sets of sessions and continue to use the same IP for session consistency.

• At a specific time period, one CCG instance can be primary only on one site and standby on the other site.

• The set of interface IPs that are associated with the CCG instance, dynamically route to the primary site of the instance.

SMF supports primary/standby redundancy in which data is replicated from the primary to standby instance. The primary instance provides services in normal operation. If the primary instance fails, the standby instance becomes the primary and takes over operation. To achieve a geographically distributed system, two primary/standby pairs can be set up where each site is actively processing traffic and standby is acting as backup for the remote site.

In an Active/Active GR deployment, consider there are two racks: Rack-1/Site-1 and Rack-2/Site-2 located in the same geographic site. All the NFs are trying to reach instance-1 and instance-2.

For NFs, both the instances are active. But in actual, instance-1 and instance-2 are divided across racks.

Rack-1/Site-1 has instance-1 and instance-2. In a pre-trigger scenario, instance-1 is local and acts as Primary and instance-2 is in Standby mode.

Rack-2/Site-2 also has instance-1 and instance-2. In a pre-trigger scenario, instance-2 is local and acts as Primary and instance-1 is in Standby mode.

In case, if Rack-1/Site-1 goes down, the traffic moves to Rack-2/Site-2. On Rack-2/Site-2 both the instances, instance-1 and instance-2 acts as Primary.

This configuration is needed to provide a geo-redundancy configuration for multiple sites. With instance ID, endpoint configurations should be configured for each Geo-Redundancy site.

Sample Configuration 1

The following is a sample configuration for endpoint VIP configuration under one instance:

config 
  instance instance-id gr_instanceId 
    endpoint endpoint_name 
        vip-ip vip_ip_address 
  exit 
exit 

Example:

config
instance instance-id 1
 endpoint sbi
  vip-ip 209.165.201.21
 exit
exit

Sample Configuration 2

The following is a sample configuration to provide information on system-id, cluster-id and slice-name under an instance:

config 
  instances instance instance_id 
   system-id system_id 
   cluster-id cluster_id 
   slice-name cdl_slice_name 
  exit 
exit 

Example:

config
 instances instance 1
  system-id smf
  cluster-id smf
  slice-name 1
 exit
exit

Note	It is recommended to have the same values for system-id, cluster-id in the instance, and app-name, cluster-name in deployment.

Only two instances can be configured on each local and remote site, and corresponding endpoints can be instantiated.

A local instance-id is the identity of the local site irrespective of the site is in GR aware or not.

Local Instance ID Configuration

The local instance is configured using the local-instance command.

local-instance instance 1

Endpoint configuration must be under instance specified by each unique instance ID.

Endpoint Configuration Example

Following are a few configuration examples.

Note	In the following example, instance-id "1" is a local instance-id, and endpoints configured under it belong to the local site. Optionally, remote site instance-id "2" can be configured for endpoints belonging to the geo-site.

instance instance-id 1
 endpoint li
  replicas 1
  nodes    2
  vip-ip 209.165.201.6
  vip-ip 209.165.201.13
 exit
 endpoint gtp
  replicas 1
  nodes    2
  retransmission timeout 5 max-retry 4
  vip-ip 209.165.201.6
  vip-ip 209.165.201.4
  interface s5
   echo interval          60
   echo retransmission-timeout 5
   echo max-retransmissions 4
  exit
  interface s2b
   echo interval 60
   echo retransmission-timeout 5
   echo max-retransmissions 4
  exit
 exit
exit
instance instance-id 2
 endpoint li
  replicas 1
  nodes    2
  vip-ip 209.165.201.6
  vip-ip 209.165.201.13
 exit
exit
endpoint gtp
  replicas 1
  nodes    2
  retransmission timeout 5 max-retry 4
  vip-ip 209.165.201.6
  vip-ip 209.165.201.5
  interface s5
   echo interval 60
   echo retransmission-timeout 5
   echo max-retransmissions 4
  exit
  interface s2b
   echo interval 60
   echo retransmission-timeout 5
   echo max-retransmissions 4
  exit
 exit
exit

Add instance for PGW FQDN corresponding to local and remote instances.

Example

Following is a configuration example.

Note	In the following example, instance-id "1" is a local instance-id, and the SMF profile configured under it belongs to the local site. Optionally, remote site instance-id "2" can be configured for FQDN belonging to the geo-site.

profile smf smf1
locality LOC1
allowed-nssai [ slice1 ]
instances 1 fqdn cisco.com.apn.epc.mnc456.mcc123
instances 2 fqdn cisco.com.apn.epc.mnc567.mcc123

This section describes how to configure the Dynamic Routing by Using BGP feature.

Configuring AS and BGP Router IP Address

To configure the AS and IP address for the BGP router, use the following commands:

config 
   router bgp local_as_number 
   exit 
exit 

NOTES:

• router bgp local_as_number —Specify the identification number for the AS for the BGP router.

  In a GR deployment, you need to configure two Autonomous Systems (AS).

  • One AS for leaf and spine.

  • Second AS for both racks: Rack-1/Site-1 and Rack-2/Site-2

Configuring BGP Service Listening IP Address

To configure the BGP service listening IP address, use the following commands:

config 
   router bgp local_as_number 
      interface interface_name 
   exit 
exit 

NOTES:

• router bgp local_as_number —Specify the identification number for the AS for the BGP router.

• interface interface_name —Specify the name of the interface.

Configuring BGP Neighbors

To configure the BGP neighbors, use the following commands:

config 
   router bgp local_as_number 
      interface interface_name 
      neighbor neighbor_ip_address remote-as as_number 
   exit 
exit 

NOTES:

• router bgp local_as_number —Specify the identification number for the AS for the BGP router.

• interface interface_name —Specify the name of the interface.

• neighbor neighbor_ip_address —Specify the IP address of the neighbor BGP router.

• remote-as as_number —Specify the identification number for the AS.

Configuring Bonding Interface

To configure the bonding interface related to the interfaces, use the following commands:

config 
   router bgp local_as_number 
      interface interface_name 
      bondingInterface interface_name 
   exit 
exit 

NOTES:

• router bgp local_as_number —Specify the identification number for the AS for the BGP router.

• interface interface_name —Specify the name of the interface.

• bondingInterface interface_name —Specify the related bonding interface for an interface. If the bonding interface is active, then the BGP gives a higher preference to the interface-service by providing a lower MED value.

Configuring Learn Default Route

If the user configures specific routes on their system and they need to support all routes, then they must set the learnDefaultRoute as true.

Note	This configuration is optional.

To configure the Learn Default Route, use the following commands:

config 
   router bgp local_as_number 
      learnDefaultRoute true/false 
   exit 
exit 

NOTES:

• router bgp local_as_number —Specify the identification number for the AS for the BGP router.

• learnDefaultRoute true/false —Specify the option to enable or disable the learnDefaultRoute parameter. When set to true, BGP learns default route and adds it in the kernel space. By default, it is false.

Configuring BGP Port

To configure the Port number for a BGP service, use the following commands:

config 
   router bgp local_as_number 
      loopbackPort port_number 
   exit 
exit 

NOTES:

• router bgp local_as_number —Specify the identification number for the AS for the BGP router.

• loopbackPort port_number —Specify the port number for the BGP service. The default value is 179.

Policy Addition

The BGP speaker pods learns many route information from its neighbors. However, only a few of them are used for supporting the outgoing traffic. This is required for egress traffic handling only, when SMF is sending information outside to AMF/PCF. Routes are filtered by configuring import policies on the BGP speakers and is used to send learned routes to the protocol pods.

A sample CLI code for policy addition and the corresponding descriptions for the parameters are shown below.

$bgp policy <policy_Name> ip-prefix 209.165.200.225 subnet 16 masklength-range 21..24 as-path-set “^65100”

This configuration controls the number of BGP speaker pods in deployment. BGP speaker advertises service IP information for incoming traffic from both the sites.

Note	Use non-bonded interface in BGP speaker pods for BGP peering. BGP peering per Proto node is supported with only two BGP routers/leafs. Considering two Proto nodes, there can be maximum of four BGP neighborships.

instance instance-id  instance_id endpoint bgpspeaker interface { bgp | bfd } internal base-port start base_port_number 

config 
instance instance-id instance_id 
 endpoint bgpspeaker 
   replicas replica_id 
   nodes node_id 
   interface bgp 
     internal base-port start base_port_number  
   exit 
   interface bfd 
     internal base-port start base_port_number 
  exit 
exit 

NOTES:

• instance instance-id instance_id —Specify the GR instance ID.

• base_port_number —Specify the port range only if logical NF is configured. This range depends on your deployment.

Example

The following is a configuration example:

instance instance-id 1
endpoint bgpspeaker
   replicas 1
   nodes    2
   interface bgp
     internal base-port start {24000}
   exit
   interface bfd
     internal base-port start {25000}
 exit

The following section provides IPAM configuraton examples.

SMF-1 Example

The following is a configuration example for SMF-1:

ipam
 instance 1
  address-pool pool-1
  vrf-name ISP
   tags
   dnn dnn-1
  exit
  ipv4
    address-range 209.165.201.1 209.165.201.31
 exit
instance 2
 address-pool pool-2
  vrf-name ISP
  tags
   dnn dnn-2
  exit
  ipv4
    address-range 209.165.202.129 209.165.202.159
 exit
exit

SMF-2 Example

The following is a configuration example for SMF-2:

ipam
 instance 1
 address-pool pool-1
  vrf-name ISP
  tags
   dnn dnn-1
  exit
  ipv4
    address-range 209.165.201.1 209.165.201.31
 exit
 instance 2
 address-pool pool-2
  vrf-name ISP
   tags
   dnn dnn-2
  exit
  ipv4
    address-range 209.165.202.129 209.165.202.159
 exit
exit

Endpoints must be configured under an instance. Two Geo-Redundancy pods are needed on each GR site. You should also configure VIP for internal and external Geo interface for ETCD/CachePod replication.

instance instance-id  instance_id endpoint geo interface { geo-internal | geo-external } vip-ip { vip_ip_address } vip-port { vip_port_number }
 

config 
instance instance-id instance_id 
 endpoint geo 
  replicas replica_id 
  nodes node_id 
  internal base-port start base_port_number 
  interface geo-internal 
   vip-ip vip_ip_address vip-port vip_port_number 
  exit 
  interface geo-external 
   vip-ip vip_ip_address vip-port vip_port_number 
  exit 
exit 
exit 

NOTES:

• instance instance-id instance_id —Specify GR instance ID. One instance ID for local site and other for remote site.

• vip-ip vip_ip_address —Specify VIP IP address for Internal/External Geo interface.

• vip-port vip_port_number —Specify VIP port number.

• internal base-port start base_port_number —Specify port range only if logical NF is configured.

Example

The following is a configuration example:

instance instance-id 1
endpoint geo
  replicas 1
  nodes 2
  internal base-port start 25000
  interface geo-internal
   vip-ip 209.165.201.8 vip-port 7001
  exit
  interface geo-external
   vip-ip 209.165.201.8 vip-port 7002
  exit
exit

To conifgure POD monitoring and failover thresholds in the GR setup, use the following configuration. The GR pod monitors the configured POD name.

config 
 geomonitor 
  podmonitor pods pod_name 
   retryCount value 
   retryInterval interval_value 
   retryFailOverInterval failover_interval 
   failedReplicaPercent percent_value 
  exit 
 exit 

NOTES:

• pods pod_name —Specify the name of the pod to be monitored. For example, Cache-pod, res-ep, and so on

• retryCount value —Specify the retry counter value to retry if pod fails to ping after which pod is marked as down. Must be an integer in the range of 1-10.

• retryInterval interval_value —Specify the retry interval in milliseconds if pod successfully pings. Must be an integer in the range of 200-10000.

• retryFailOverInterval failover_interval —Specify the retry interval in milliseconds if pod fails to ping. Must be an integer in the range of 200-10000.

• failedReplicaPercent percent_value —Specify the percent value of failed replica after which GR failover is triggered. Must be an integer in the range of 10-100.

Configuration Example

The following is an example configuration.

geomonitor podmonitor pods cache-pod
 retryCount 3
 retryInterval 5
 retryFailOverInterval 1
 failedReplicaPercent 40
exit

Remote cluster monitoring auto corrects roles (it becomes self-primary, when the remote site is in STANDBY_ERROR state) for uninterrupted traffic flow of traffic. However, this auto role correction gets done only for specific roles.

To configure this feature, use the following sample configuration:

config 
    geomonitor 
        remoteclustermonitor 
            retryCount value 
            retryInterval interval_value 
            end 

NOTES:

• retryCount value —Specify the retry count before making the current site PRIMARY. Must be an integer in the range of 1-10. The default value is 3.

• retryInterval interval_value —Specify the retry interval in the count of milliseconds, after which the remote site status gets fetched. Must be an integer in the range of 200-50000. The default value is 3000.

Configuration Example

The following is an example configuration

geomonitor remoteclustermonitor
retryCount 3
retryInterval 3000

The following command is used to monitor the traffic.

config 
 geomonitor 
  trafficMonitor 
   thresholdCount value 
   thresholdInterval interval_value 
  exit 
 exit 

NOTES:

• thresholdCount value —Specify the number of calls received for standby instance. Must be an integer in the range of 0-10000. Default value is 0.

  Both UDP-proxy and REST-EP must be considered for the counter value.

• thresholdInterval interval_value —Specify the maximum duration to hit the threshold count value in ms. Must be an integer in the range of 100-10000. Default value is 3000.

Configuration Example

The following is an example configuration

geomonitor trafficmonitor
thresholdCount 3
thresholdInterval 3000

Before deploying the CDL GR, user must configure the following:

• CDL Session Database and define the base configuration.

• Kafka for CDL.

• Zookeeper for CDL.

In CDL, along with existing GR related parameters, GR instance awareness must be enabled using a feature flag on all sites. Also, the mapping of system-id to slice names should also be provided for this feature to work on all sites.

The CDL is also equipped with Geo Replication (GR) failover notifications, which can notify the timer expiry of session data and bulk notifications to the currently active site. The CDL uses Border Gateway Protocol (BGP) through App-Infra for the GR failover notifications.

The CDL subscribes to the key value on both the GR sites. The App-Infra sends notifications to the CDL when there is any change in these key values. A key value indicates the state of the CDL System ID or the GR instance. The GR instance is mapped to the CDL slices using the CDL system ID or the GR instance ID in the key.

The system ID is mandatory on both the sites. The GR instance ID in the NF configuration must match the CDL system ID.

CDL has instance-specific data slices. It also allows users to configure instance-specific slice information at the time of bringing up.

• CDL notifies the data on expiry or upon bulk notification request from the active slices.

• CDL determines the active instance based on the notification from app-infra memory-cache.

• CDL slice is a partition within a CDL instance to store a different kind of data. In this case, NF stores a different instance of data.

Note	CDL slice name should match with the slice-name configured in GR.

To reset the GR instance role (for example, roles from STANDBY_ERROR to STANDBY to PRIMARY), use the following sample commands:

geo reset-role role role instance-id gr_instanceId 

NOTES:

• role role —Specify the new role for the given site.

  The role can be PRIMARY or STANDBY.

• instance-id gr_instanceId —Specify the GR Instance ID.

Important

The command geo reset-role triggers change in the role for the given instance on the local site. The remote site doesn't receive any message for the same command. It’s only possible to change the role for the given instance ID from STANDBY_ERROR to STANDBY and STANDBY to PRIMARY. Another role change isn’t possible.

To switch the GR role, initiate the command on the primary rack (for example, role PRIMARY to STANDBY only), and use the following command.

geo switch-role role role instance-id gr_instanceId 

NOTES:

• role role —Specify the new role for the given site.

  The roles can be PRIMARY or STANDBY. It’s mandatory to trigger manual switchover from primary role for a specific GR instance ID.

• instance-id gr_instanceId —Specify the GR Instance ID

Important

geo switch-role command triggers manual failover from one site to another site for specific instance ID. The site which triggers the failover changes from the PRIMARY role to the STANDBY_ERROR role. In between, the site which triggers the failover, sends a failover (Trigger GR) message to another site. The other site which receives the failover message changes from the STANDBY role to the PRIMARY role.

This section describes show/clear commands that help in debugging issues.

To capture messages for subscriber (gr-instance aware), use the following command:

monitor subscriber [ supi ] [ imsi ] [ imei ] (capture-duration) (internal-messages) (transaction-logs) (nf-service) (gr-instance) 

Note	In 2021.02 and later releases, the namespace keyword is deprecated and replaced with the nf-service keyword.

NOTES:

• supi —Specify the subscriber identifier.

  Example: imsi-123456789, imsi-123*

• imsi —Specify the IMSI value.

  Example: 123456789, *

• imei —Specify the IMEI value.

• Example: 123456789012345, *

• capture-duration —(Optional) Used to specify the duration in seconds during which monitor subscriber is enabled. Default value is 300 secs.

• internal-messages —(Optional) When set to yes, it enables internal messaging. By default, it is disabled.

• transaction-logs —(Optional) When set to yes, it enables transaction logging. By default, it is disabled.

  Note	Messages and transaction logs are mutually exclusive.

• namespace —Deprecated option. Use nf-service instead.

• nf-service —(Optional) Specify the NF service. Possible values are sgw, smf. Default value is none.

• gr-instance —(Optional) Monitor subscriber for a given gr-instance only.

Example

The following is a configuration example.

monitor subscriber imsi 123456789 gr-instance 1
supi: imsi-123456789
captureDuration: 300
enableInternalMsg: false
enableTxnLog: false
namespace(deprecated. Use nf-service instead.): none
nf-service: none
gr-instance: 1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   295  100    98  100   197  10888  21888 --:--:-- --:--:-- --:--:-- 29500
Command: --header Content-type:application/json --request POST --data 
{"commandname":"mon_sub","parameters":{"supi":"imsi-123456789","duration":300,
"enableTxnLog":false,"enableInternalMsg":false,"action":"start","namespace":"none",
"nf-service":"none","grInstance":1}} http://oam-pod:8879/commands
Result start mon_sub, fileName ->logs/monsublogs/none.imsi-123456789_TS_2021-04-09T09:59:59.964148895.txt
Starting to tail the monsub messages from file: logs/monsublogs/none.imsi-123456789_TS_2021-04-09T09:59:59.964148895.txt
Defaulting container name to oam-pod.
Use 'kubectl describe pod/oam-pod-0 -n smf' to see all the containers in this pod.

For more information on Monitor Protocol on SMF, see the Monitor Subscriber and Monitor Protocol section in the UCC 5G Session Management Function - Configuration and Administration Guide.

To capture packets on different interfaces (gr-instance aware), use the following command:

monitor protocol [ interface ] (capture-duration) (count) (level) (gr-instance)  

NOTES:

• interface —Interface on which PCAP is captured.

  Example: sbi, pfcp, gtpu, gtpc, gtp, radius

• list —Monitor protocol list files.

• capture-duration —(Optional) Used to specify the duration in secs during which PCAP is captured. Default value is 300 secs.

• pcap —(Optional) When set to yes, it enables PCAP file generation. By default, the value is "no" (disabled).

• gr-instance —(Optional) Monitor subscriber for a given gr-instance only.

Example

The following is a configuration example.

monitor protocol interface sbi gr-instance 1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   220  100    95  100   125   8636  11363 --:--:-- --:--:-- --:--:-- 20000
Command: --header Content-type:application/json --request POST --data 
{"commandname":"mon_pro","parameters":{"interface":"sbi","duration":300,"action":
"start","enable_pcap":false,"grInstance":1}} http://oam-pod:8879/commands
Result start mon_pro, fileName ->logs/monprologs/sessintfname_sbi_at_2021-04-30T05:26:22.712229347.txt
Starting to tail the monpro messages from file: logs/monprologs/sessintfname_sbi_at_2021-04-30T05:26:22.712229347.txt
Defaulting container name to oam-pod.
Use 'kubectl describe pod/oam-pod-0 -n cn' to see all of the containers in this pod.

For more information on Monitor Protocol on SMF, see the Monitor Subscriber and Monitor Protocol section in the UCC 5G Session Management Function - Configuration and Administration Guide.

The following section provides information on GR setup health check.

• All critical pods are in good condition to serve user traffic.

  Use the following command to check whether GR and CDL related pods are in Running state.

  kubectl get pods -n cn-cn1 -o wide | grep georeplication-pod
  kubectl get pods -n cn-cn1 -o wide | grep cdl
  kubectl get pods -n cn-cn1 -o wide | grep mirror-maker

• Keepalived pods are in healthy state to monitor all VIPs which are configured for check-interface/check-port.

  Use the following command to check whether keepalived pods in “smi-vips” namespace are in “Running” state.

  kubectl get pods -n smi-vips

• Health-check of pods related to CDL: Check the status of CDL db-endpoint, slot and indexes. All should be in STARTED or ONLINE state for both System IDs 1 and 2.

  cdl show status 
  message params: {cmd:status mode:cli dbName:session sessionIn:{mapId:0 limit:500 key: purgeOnEval:0 filters:[] nextEvalTsStart:0 nextEvalTsEnd:0 allReplicas:false maxDataSize:4096} sliceName:}
  db-endpoint {
      endpoint-site {
          system-id 1
          state STARTED
          total-sessions 4
          site-session-count 2
          total-reconciliation 0
          remote-connection-time 66h37m31.36054781s
          remote-connection-last-failure-time 2021-07-13 11:24:10.233825924 +0000 UTC
          slot-geo-replication-delay 2.025396ms
      }
      endpoint-site {
          system-id 2
          state STARTED
          total-sessions 4
          site-session-count 2
          total-reconciliation 0
          remote-connection-time 66h58m49.83449066s
          remote-connection-last-failure-time 2021-07-13 11:02:51.759971655 +0000 UTC
          slot-geo-replication-delay 1.561816ms
      }
  }
  slot {
      map {
          map-id 1
          instance {
              system-id 1
              instance-id 1
              records 4
              capacity 2500000
              state ONLINE
              avg-record-size-bytes 1
              up-time 89h38m37.335813523s
              sync-duration 9.298061ms
          }
          instance {
              system-id 1
              instance-id 2
              records 4
              capacity 2500000
              state ONLINE
              avg-record-size-bytes 1
              up-time 89h39m11.1268024s
              sync-duration 8.852556ms
          }
          instance {
              system-id 2
              instance-id 1
              records 4
              capacity 2500000
              state ONLINE
              avg-record-size-bytes 1
              up-time 89h28m38.274713022s
              sync-duration 8.37766ms
          }
          instance {
              system-id 2
              instance-id 2
              records 4
              capacity 2500000
              state ONLINE
              avg-record-size-bytes 1
              up-time 89h29m37.934345015s
              sync-duration 8.877442ms
          }
      }
  }
  index {
      map {
          map-id 1
          instance {
              system-id 1
              instance-id 1
              records 4
              capacity 60000000
              state ONLINE
              up-time 89h38m16.119032086s
              sync-duration 2.012281769s
              leader false
              geo-replication-delay 10.529821ms
          }
          instance {
              system-id 1
              instance-id 2
              records 4
              capacity 60000000
              state ONLINE
              up-time 89h39m8.47664588s
              sync-duration 2.011171261s
              leader true
              leader-time 89h38m53.761213379s
              geo-replication-delay 10.252683ms
          }
          instance {
              system-id 2
              instance-id 1
              records 4
              capacity 60000000
              state ONLINE
              up-time 89h28m29.5479133s
              sync-duration 2.012101957s
              leader false
              geo-replication-delay 15.974538ms
          }
          instance {
              system-id 2
              instance-id 2
              records 4
              capacity 60000000
              state ONLINE
              up-time 89h29m11.633496562s
              sync-duration 2.011566639s
              leader true
              leader-time 89h28m51.29928233s
              geo-replication-delay 16.213323ms
          }
      }
  }

• CDL replication status

  Check whether four gRPC connections are established between the CDL EP session pods (of each namespace) across the racks in GRPC_Connections_to_RemoteSite panel of CDL Replication Stats Grafana dashboard. Check Grafana on both racks.

  

• Admin port status between the racks for geo-replication.

  Check heartbeat messages between geo-replication pods across the racks in Periodic_Heartbeat_to_Remote_Site panel of GR Statistics Grafana dashboard.

  

• BGP/BFD link status on rack

  Check whether neighborship with BGP peers is established in BGP Peers panel of BGP, BFD Statistics Grafan dashboard.

  

  Check whether BFD link is in connected state in BFD Link Status panel of BGP, BFD Statistics Grafana dashboard.

  

• Roles of each instances are in healthy state

  Check that in each rack the roles are not in STANDBY_ERROR state at any point of time.

  • Active/Standby model: Roles should be in the following states on each rack

    Rack-1/Site-1:

    show role instance-id 1
    result "PRIMARY"
    show role instance-id 2
    result "PRIMARY"

    Rack-2/Site-2:

    show role instance-id 1
    result "STANDBY"
    show role instance-id 2
    result "STANDBY"

  • Active/Active model: Roles should be in the following states on each rack.

    Rack-1/Site-1:

    show role instance-id 1
    result "PRIMARY"
    show role instance-id 2
    result "STANDBY"

    Rack-2/Site-2:

    show role instance-id 1
    result "STANDBY"
    show role instance-id 2
    result "PRIMARY"

On Rack-1/Site-1

1. Verify that roles of both instances on Rack-1/Site-1 are in STANDBY_ERROR.

   show role instance-id 1
   result "STANDBY_ERROR"

   show role instance-id 2
   result "STANDBY_ERROR"

2. Initiate reset role for both instances on Rack-1/Site-1 to STANDBY. This step transitions the roles from STANDBY_ERROR/STANDBY_ERROR to STANDBY/STANDBY.

   geo reset-role instance-id 1 role standby
   geo reset-role instance-id 2 role standby

3. Verify that roles of both instances have moved to STANDBY on Rack-1/Site-1.

   show role instance-id 1
   result "STANDBY"

   show role instance-id 2
   result "STANDBY"

4. Initiate switch role for instance-id 1 on Rack-2/Site-2 to STANDBY with failback-interval of 30 seconds. This step transitions the roles of Rack-2/Site-2 from PRIMARY/PRIMARY to STANDBY_ERROR/PRIMARY and Rack-1/Site-1 from STANDBY/STANDBY to PRIMARY/STANDBY.

   geo switch-role instance-id 1 role standby failback-interval 30

5. Verify that roles of both instances on Rack-2/Site-2 are in STANDBY_ERROR/PRIMARY.

   show role instance-id 1
   result "STANDBY_ERROR"

   show role instance-id 2
   result "PRIMARY"

6. Verify that roles of both instances on Rack-1/Site-1 are in PRIMARY/STANDBY.

   show role instance-id 1
   result "PRIMARY"

   show role instance-id 2
   result "STANDBY"

7. Initiate reset role for instance-id 1 on Rack-2/Site-2 to STANDBY. This step transitions the roles of Rack-2/Site-2 from STANDBY_ERROR/PRIMARY to STANDBY/PRIMARY.

   geo reset-role instance-id 1 role standby

8. Verify that the roles of Rack-2/Site-2 are in STANDBY/PRIMARY.

   show role instance-id 1
   result "STANDBY"

   show role instance-id 2
   result "PRIMARY"

On Rack-2/Site-2

1. Verify that roles of both the instances on Rack-2/Site-2 are in STANDBY_ERROR.

   show role instance-id 1
   result "STANDBY_ERROR"

   show role instance-id 2
   result "STANDBY_ERROR"

2. Initiate reset role for both instances on Rack-2/Site-2 to STANDBY. This step transitions the roles from STANDBY_ERROR/STANDBY_ERROR to STANDBY/STANDBY.

   geo reset-role instance-id 1 role standby
   geo reset-role instance-id 2 role standby

3. Verify that the roles of both the instances move to STANDBY on Rack-2/Site-2.

   show role instance-id 1
   result "STANDBY"

   show role instance-id 2
   result "STANDBY"

4. Initiate switch role for instance-id 2 on Rack-1/Site-1 to STANDBY. This step transitions roles of Rack-1/Site-1 from PRIMARY/PRIMARY to PRIMARY/STANDBY_ERROR and Rack-2/Site-2 from STANDBY/STANDBY to STANDBY/PRIMARY.

   geo switch-role instance-id 2 role standby failback-interval 30

5. Verify that roles of instances on Rack-1/Site-1 are in PRIMARY/STANDBY_ERROR mode.

   show role instance-id 1
   result "PRIMARY"

   show role instance-id 2
   result "STANDBY_ERROR"

6. Verify that roles of instances on Rack-2/Site-2 are in STANDBY/PRIMARY mode.

   show role instance-id 1
   result "STANDBY"

   show role instance-id 2
   result "PRIMARY"

7. Initiate reset role for instance-id 2 on Rack-1/Site-1 to STANDBY. This step transitions the roles of Rack-1/Site-1 from PRIMARY/STANDBY_ERROR to PRIMARY/STANDBY.

   geo reset-role instance-id 2 role standby

8. Verify that roles of instances on Rack-1/Site-1 are in PRIMARY/STANDBY.

   show role instance-id 1
   result "PRIMARY"

   show role instance-id 2
   result "STANDBY"

The following section describes KPIs.

ETCD/Cachepod Replication KPIs

The following table lists ETCD/Cachepod Replication KPIs.

Geo Rejected Role Change KPIs

The following table lists Geo Rejected Role Change KPIs.

Monitoring KPIs

The following table lists monitoring KPIs.

BFD KPIs

The following table lists BFD KPIs.

Cross-rack-routing BFD Interface Monitoring

Local Interface Monitoring

GR Instance Information

Geo Maintenance Mode

The following section provides details on GR-specific bulkstats.

bulk-stats query GR-BGP-Incoming-Failed-Routes
 expression "sum(bgp_incoming_failedrouterequest_total) by (namespace, interface, service_IP, next_hop, instance_id)"
 labels     [ instance_id interface next_hop service_IP ]
 alias      gr-bgp-routes-in
exit
bulk-stats query GR-Geo-Monitoring-Failure
 expression "sum(geo_monitoring_total{ControlActionNameType=~'MonitorPod|RemoteMsgHeartbeat|
RemoteMsgGetSiteStatus|RemoteSiteRoleMonitoring|RemoteClusterPodFailure|RemoteMsgNotifyFailover|
RemoteMsgNotifyPrepareFailover|MonitorVip',status!~'success|monitoring.*'}) by (namespace, 
AdminNode, ControlActionType, ControlActionNameType,  pod, status, status_code)"
 labels     [ pod AdminNode ControlActionNameType status status_code ]
 alias      gr-geo-monitoring-failure
exit
bulk-stats query GR-Geo-Monitoring-Success
 expression "sum(geo_monitoring_total{ControlActionNameType=~'MonitorPod|RemoteMsgHeartbeat|
RemoteMsgGetSiteStatus|RemoteSiteRoleMonitoring|RemoteClusterPodFailure|RemoteMsgNotifyFailover|
RemoteMsgNotifyPrepareFailover',status=~'success|monitoring.*'}) by (namespace, AdminNode, 
ControlActionType, ControlActionNameType,  pod, status)"
 labels     [ pod AdminNode ControlActionNameType status ]
 alias      gr-geo-monitoring
exit
bulk-stats query GR-Geo-Monitoring-Total
 expression "sum(geo_monitoring_total{ControlActionNameType=~'MonitorPod|RemoteMsgHeartbeat|
RemoteMsgGetSiteStatus|RemoteSiteRoleMonitoring|RemoteClusterPodFailure|RemoteMsgNotifyFailover
|RemoteMsgNotifyPrepareFailover|MonitorVip'}) 
by (namespace, AdminNode, ControlActionType, ControlActionNameType,  pod, status)"
 labels     [ pod AdminNode ControlActionNameType status ]
 alias      gr-geo-monitoring
exit
bulk-stats query GR-Geo-Replication-Failure
 expression "sum(geo_replication_total{ReplicationNode=~'CACHE_POD|ETCD|PEER',status!='success',
ReplicationRequestType='Response'}) by (namespace, ReplicationNode, ReplicationSyncType,
ReplicationReceiver,ReplicationRequestType,status,status_code)"
 labels     [ pod ReplicationNode ReplicationReceiver ReplicationRequestType ReplicationSyncType status status_code ]
 alias      gr-geo-replication-failure
exit
bulk-stats query GR-Geo-Replication-Success
 expression "sum(geo_replication_total{ReplicationNode=~'CACHE_POD|ETCD|PEER',
status='success',ReplicationRequestType='Response'}) by (namespace, ReplicationNode, 
ReplicationSyncType,ReplicationReceiver,ReplicationRequestType,status)"
 labels     [ pod ReplicationNode ReplicationReceiver ReplicationRequestType ReplicationSyncType status ]
 alias      gr-geo-replication-success
exit
bulk-stats query GR-Geo-Replication-Total
 expression "sum(geo_replication_total{ReplicationNode=~'CACHE_POD|ETCD|PEER'}) 
by (namespace, ReplicationNode, ReplicationSyncType,ReplicationReceiver,
ReplicationRequestType,pod)"
 labels     [ pod ReplicationNode ReplicationReceiver ReplicationRequestType ReplicationSyncType ]
 alias      gr-geo-replication-total
exit
bulk-stats query GR-Trigger-ResetRole-Api
 expression "sum(geo_monitoring_total{ControlActionNameType=~'TriggerGRApi|ResetRoleApi'}) 
by (namespace, AdminNode, ControlActionType, ControlActionNameType,  pod, status, status_code)"
 labels     [ pod AdminNode ControlActionNameType status status_code ]
 alias      gr-api
exit
bulk-stats query GR-CDL-Index-Replication
 expression "sum(consumer_kafka_records_total) by (pod, origin_instance_id)"
 labels     [ origin_instance_id pod ]
 alias      gr-cdl-index-replication
exit
bulk-stats query GR-CDL-Inter-Rack-Replications-Failures
 expression "sum(datastore_requests_total{local_request='0',errorCode!='0'}) by (operation,sliceName,errorCode)"
 labels     [ sliceName operation errorCode ]
 alias      gr-cdl-inter-rack-replications
exit
bulk-stats query GR-CDL-Inter-Rack-Replications-Success
 expression "sum(datastore_requests_total{local_request='0',errorCode='0'}) by (operation,sliceName,errorCode)"
 labels     [ sliceName operation errorCode ]
 alias      gr-cdl-inter-rack-replications
exit
bulk-stats query GR-CDL-Inter-Rack-Replications-Total
 expression "sum(datastore_requests_total{local_request='0'}) by (operation,sliceName,errorCode)"
 labels     [ sliceName operation errorCode ]
 alias      gr-cdl-inter-rack-replications
exit
bulk-stats query GR-CDL-Intra-Rack-Operations-Failures
 expression "sum(datastore_requests_total{local_request='1',errorCode!='0'}) by (operation,sliceName,errorCode)"
 labels     [ sliceName operation errorCode ]
 alias      gr-cdl-intra-rack-operations
exit
bulk-stats query GR-CDL-Intra-Rack-Operations-Success
 expression "sum(datastore_requests_total{local_request='1',errorCode='0'}) by (operation,sliceName,errorCode)"
 labels     [ sliceName operation errorCode ]
 alias      gr-cdl-intra-rack-operations
exit
bulk-stats query GR-CDL-Intra-Rack-Operations-Total
 expression "sum(datastore_requests_total{local_request='1'}) by (operation,sliceName,errorCode)"
 labels     [ errorCode operation sliceName ]
 alias      gr-cdl-intra-rack-operations
exit
bulk-stats query GR-CDL-Session-Count-Per-Slice
 expression sum(avg(db_records_total{namespace=~'$namespace',session_type='total'})by(systemId,sliceName))by(sliceName)
 labels     [ sliceName ]
 alias      gr-cdl-session-count-per-slice
exit
bulk-stats query GR-CDL-Session-Count-Per-System-ID
 expression sum(avg(db_records_total{namespace=~'$namespace',session_type='total'})
by(systemId,sliceName))by(systemId)
 labels     [ systemId ]
 alias      gr-cdl-session-count-per-system-id
exit
bulk-stats query GR-CDL-Slot-Records-Per-Slice
 expression "sum(slot_records_total{pod=~'.*',systemId!=''}) by (pod, sliceName)"
 labels     [ pod sliceName ]
 alias      gr-cdl-slot-records-per-slice
exit
bulk-stats query GR-CDL-Slot-Records-Per-System-ID
 expression "sum(slot_records_total{pod=~'.*',systemId!=''}) by (pod, systemId)"
 labels     [ pod systemId ]
 alias      gr-cdl-slot-records-per-system-id
exit
bulk-stats query GR-CDL-Total-Session-Count
 expression "sum(db_records_total{namespace=~'$namespace',session_type='total'}) by (systemId,sliceName)"
 labels     [ sliceName systemId ]
 alias      gr-cdl-total-session-count
exit

For more information on GR-related statistics, see the following:

• In RADIUS statistics, you can filter GR-specific statistics using grInstId label.

  For more information, see the UCC 5G Session Management Function - Metrics Reference.

• In GTP Endpoint statistics, you can filter GR-specific statistics using gr_instance_id label.

  For more information, see the UCC 5G Session Management Function - Metrics Reference.

• In SMF statistics, you can filter GR-specific statistics using gr_instance_id label.

  For more information, see the UCC 5G Session Management Function - Metrics Reference.

• In REST Endpoint statistics, you can filter GR-specific statistics using gr_instance_id label.

  For more information, see the UCC 5G Session Management Function - Metrics Reference.

• In IPAM-related statistics, you can filter GR-specific statistics using grInstId label.

  For more information, see the UCC 5G Session Management Function - Metrics Reference.

The following section provides details on GR alerts.

BFD Alerts

The following table list alerts for rule group BFD with interval-seconds as 60.

GR Alerts

The following table list alerts for rule group GR with interval-seconds as 60.

CDL Alerts

The following table list alerts for rule group CDL with interval-seconds as 60.