The CDL endpoint pod is the front-end for receiving requests from NF application towards CDL. The CDL endpoint exposes gRPC over HTTP2 interface towards the NF client's for processing database service requests. Each pod starts with the following attributes:

• systemID: This parameter specifies the site ID (For instance: Site).

• clusterID: This parameter specifies the unique datastore ID within a site (For instance: X, Y).

The CDL endpoint receives the Create, Read, Update, Delete (CRUD) requests from the NF applications. It communicates with the CDL Index and Slot pods over GRPC to perform these operations.

The CDL endpoint learns about the other pods within a site automatically using the etcd. Also, the Cisco Data Store K8s pod is stateless in nature and does not maintain any session stickiness. The CDL endpoint receives the request from NF application and communicates with Slot and Index pods internally through GRPC to handle the request accordingly. It returns the response after the processing is complete.

When the CDL endpoint receives a Create, Delete, or Update session, the receiving container replicates it to the Slot and Index pods and sends a response only when 'n' number of writes are acknowledged (ACKed) by the Index and Slot microservices.

Note	A maximum of four virtual CPUs (vCPUs) is required for deploying the CDL endpoint pod in production environment.

The CDL Slot pod stores the actual session data. The CDL endpoint connects to all the Slot pods within the cluster to replicate the session data to all the pods. These microservices are K8s pod deployed for exposing internal gRPC interface towards the Cisco Data Store. Each pod starts with the following attributes:

• systemID: This parameter specifies the site ID (For instance: Site-1).

• clusterID: This parameter specifies the unique datastore ID within a site (For instance: Session).

• mapID: This parameter specifies the replica set ID within the cluster. (For instance: map-1, map-2,..map-n).

• instanceID:This parameter specifies the instance ID within the replica set. (For instance: map-1.instance-1,map-1.instance-2)

Each Slot pod holds a finite number of sessions and pre-allocated memory for the storing session data. Also, each replica within the replica set (mapID) has a defined anti affinity rule, which prevents the same blade form hosting multiple member or instances of the same replica set (for high availability in case of a blade or node failure). Each Slot pod maintains a timer and last updated ts. The Slot pod generates the notification callback to the client NF for taking action when the timer expires of if a conflict is detected.

Note	A maximum of two vCPUs is required for deploying the Slot pod in production environment.

In the event of pod failover and recovery, the Slot pod recovers from:

• Local Replica Member: The Slot directly reads from the gRPC stream in bulk directly to recover data.

• Remote Replica Member: When there is no local replica available for synchronization, the Slot reads the data from the remote site instances for the same map.

  The following figures depict the Slot recovery process from local and remote peers:

  

  

Data Slicing

Data slicing logically separates CDL as slices and stores the session data based on the slice name received from the Network Functions (NF).

With data slicing, one or more NFs can store different types of session data in dedicated slices of CDL.

A default slice name called session is used if the slice names are not configured.

The configuration is as follows:

cdl datastore <datastore name> slice-names [ <sliceName 1> <sliceName 2> ... <sliceName n> ]

The sample configuration is as follows:

cdl datastore session slice-names [ session1 session2 ]

Note

If the slice names are configured at the NF's ops-center or CDL's ops-center, every request from the NF must have a valid slice name. If the slice name is different from what is configured or empty, then the request is rejected with an error code. If the slice names are not configured, then the NF requests are routed to the default session. The slice names cannot be updated in a running system post deployment.

Deleting CDL Slot Data

In certain scenarios, the CDL records are found on Slot but not in the index pods. The notifications from Slot towards the application for such records do not receive the values correctly. The record in the slot is not deleted, if the index data is not deleted.

Ensure the following before deleting the CDL Slot Data:

• If the number of notifications to an application crosses a threshold value (default value of 3), a record is suspected to be stale.

• This triggers a validation check to find the corresponding record in any of the index pods (local or on any geo remote sites).

• If there is a mismatch in map ID from index, or if the map ID is not found in all index pods, then a clean-up is invoked to delete the record on local as well as remote sites.

The following parameters are introduced to delete stale records:

disable-auto-deletion:When set to true, the stale CDL records are not deleted. Auto deletion of stale records is enabled by default.

notification-retry-count:Specifies the minimum number of timer expiry notification retries sent to application without receiving an update from application. If there are no updates received even after notification-retry-count times, cdl proceeds to check if slot record is stale. The default number is 3.

The sample CDL configurations are as follows:

To disable the stale slot record auto deletion feature:

cdl datastore session
features slot-stale-session-detection disable-auto-deletion true
exit

You can change the notification-retry-count to a new value, for example 5. This indicates that the timer expiry notification tries 5 times, after which it proceeds for checking whether the data is stale.

cdl datastore session
features slot-stale-session-detection notification-retry-count 5
exit

Troubleshooting

To enable troubleshooting logs for Stale CDL Slot Data on endpoint and slot pods, use the following configuration:

cdl logging logger ep.staleRecord.session
level info
exit

cdl logging logger slot.staleRecord.session
level info
exit

The Indexing containers contain indexing data. There are two key pieces of information stored in the Index pod:

• Primary key to Slot Map ID.

• Unique secondary key to primary key mapping.

The index entries are created (an example use case) at the time of session connect and removed at the time of disconnect. Using the in-memory KV store, Indexing stores the keys in memory and provides a multi-master write. The indexing write log (set and delete operations) are also written to Kafka pods for remote site replication. One of the Index pods writes the logs in Kafka, which are then received at the remote site. The corresponding operations are also performed at the remote site.

Note	A maximum of two vCPUs is required for deploying the Index pod in production environment.

In the event of pod failover and recovery, the Index pod recovers from:

• Local Replica Member - The Index directly reads from the gRPC stream in bulk directly to recover data.

• Remote Replica Member - If there is no local replica available for synchronization, the Index reads the data from the remote site instances for the same map.

  The following figures depict the Index recovery process from local and remote peers:

  

  

The CDL uses the etcd (an open-source key-value store) as the DB service discovery. When the CDL pods (Endpoint, Slot or Index) is started, killed or shutdown, it results in updating the event to etcd by publishing the state.

The CDL endpoint is interested in knowing when a Slot or Index pod comes up or goes down. Therefore, it subscribes to the notifications for Slot and Index pod events in etcd.

Similarly, the CDL Slot or Index pod is interested in knowing when its peer replica pod (with the same map-id) comes up or goes down. Therefore, it subscribes to the notifications for corresponding Slot or Index (with same map-id) pod events in etcd.

Thus, the notifications are sent to each of the pods subscribed to these events. In addition, when a key event is added or removed, it refreshes the local map. The etcd cache is only applicable for local site events.

The Kafka pod replicates data between the local replicas and across sites for Indexing. For replication across sites, Kafak uses MirrorMaker. The Kafka pods are deployed on the session vms with the replica count set to a minimum of two for high availability.

Kafka uses the Zookeeper pod for managing Kafka clusters and coordinating with Kafka brokers.

The Mirror Maker pod geo-replicates the indexing data to the remote CDL sites. It takes data from the remote sites and publishes it to the local Kafka site for the appropriate Indexing instances to pick up.

This section describes the following call flows:

• Finding Record by Primary Key

• Finding Record by Unique Key

• Creating a Record

• Updating a Record

• Deleting a Record

• Notification to NF on Timer Expiry

• Geo Replication - Create (GEO)

• Geo Replication - Update (GEO)

• Geo Replication - Delete (GEO)

In certain scenarios, the unique keys in index pods of the CDL are stale (presumed to be deleted). The NFs may try to use these unique keys for another record, as the CDL does not show the stale record details to the NFs.

The CDL allows the operator to enable a parameter to detect the stale records and perform the necessary action on such stale records.

To detect the stale records and perform actions:

1. Identify the stale records. When a new record is created, the unique key is overwritten. To identify such records, enable the index-overwrite-detection parameter, and ensure that the unique-keys-prefix matches the prefix pattern.

2. Perform the necessary action (notify, delete or log) on the identified stale records.

   The CDL detects any unique key that is overwritten and performs one of the following actions:

   • Deletes the stale record or the entire record. The delete action is triggered only if the PurgeOnEval flag is set to false for the stale record.

   • Notifies the NF about the stale record. The notify action sends the STALE_INDEX_NOTIFICATION to the NF.

   • Logs the overwritten unique key. For the log action, the stale record is logged with the WARN log level.

   Note	If there are two different unique keys pointing to the same primary key; one with notify action and the other with delete action, then the notify action is performed.

Use the following configuration:

features index-overwrite-detection max-tps variable
features index-overwrite-detection queue-size variable
features index-overwrite-detection unique-keys-prefix uk
  action [delete-record, notify-record, log-record]
 

where,

• max-tps: Controls the rate of notification for stale records. The default is 200.

• queue-size: Controls the processing queue size of the stale records. The default is 1000.

  Note	The queue-size parameter is for both delete and notify actions. The max-tps parameter is only for the notify action.

• unique-keys-prefix: Specifies the unique-key prefix pattern along with the action that needs to be performed.

Example:

cdl datastore session
features index-overwrite-detection max-tps 250
 features index-overwrite-detection queue-size 2000
 features index-overwrite-detection unique-keys-prefix uk
 action notify-record
 exit    
  exit      
  

Troubleshooting

To troubleshoot the stale index records, set the index.overwrite.session logger to INFO level. The logs from the endpoint pods as well as the index pods help in troubleshooting.

CDL configuration:

cdl logging logger index.overwrite.session
 level info
exit
 

The following metrics are introduced:

• overwritten_index_records_deleted–maintains the total number of records deleted due to stale records identified at index.

• overwritten_index_records_skipped–maintains the total number of records detected as stale, but dropped when the queue is full while processing the records for notify or delete.

You can deploy CDL through the Network Functions (NFs) – Session Management Function (SMF) and Policy Control Function (PCF) – Ops Center.

For deploying CDL through SMF Ops Center, see Cisco Common Data Layer in SMF chapter in Ultra Cloud Core 5G Session Management Function - Configuration and Administration Guide

For deploying CDL through PCF Ops Center, see Cisco Common Data Layer in PCF chapter in Ultra Cloud Core 5G Policy Control Function -Configuration and Administration Guide

This section describes how to deploy CDL for Geo Replication (GR).

This section describes the alarms in the CDL deployment.

The following alarms are introduced to detect failure:

You can define the maximum capacity per shard in CDL using the following parameters:

• slotRecordCapacity (Default value: 1M/shard)

• indexRecordCapacity (Default value: 10M/shard)

• slotRecordsSizeCapacityInBytes (Default value: 5GB/shard)

Note	You cannot change the default capacity per shard.

Based on the following calculations, the CDL determines the maximum capacity:

Soft Limit

1. cdl_slot_record_capacity  =  No of shards * slotRecordCapacity

2. cdl_index_record_capacity = No of shards * indexRecordCapacity

3. cdl_slot_size_capacity    = No of shards * slotRecordsSizeCapacityInBytes

Hard Limit

1. cdl_slot_record_capacity = No of shards * slotRecordCapacity * 105%

2. cdl_index_record_capacity = No of shards * indexRecordCapacity * 105%

3. cdl_slot_size_capacity = No of shards * slotRecordsSizeCapacityInBytes * 105%

The CDL is overloaded whenever the system reaches its maximum capacity. Also, the CDL triggers the following alerts to protect the system from overloading:

• cdlOverloaded - major - This alert is triggered when the CDL system reaches 80% of its capacity.

• cdlOverloaded – critical - This alert is triggered when the CDL system reaches 90% of its capacity.

In addition to the alerts, the CDL endpoint rejects all the create requests with the following error:

error code 507: Datastore reached its full capacity

However, the CDL processes all the update requests till the hard limit capacity is reached. When the hard limit capacity is exceeded, the CDL rejects all the update requests.

The the existing CDL overload protection configuration is replaced with a new configuration.

Note

Earlier, the overload protection was enabled by default with hard coded limits of 1 million records per slot, 10 million records per index and 5 gb record size per slot. However, in this release, the feature is disabled by default and requires configuration to enable this feature and to configure the appropriate overload limits (optional) for each parameter.

cdl datastore session overload-protection disable true 

cdl datastore session features overload-protection enable <true/false> 

If the overload protection is enabled then the alerts are also enabled. The prometheus-rules-cdl pod is spawned. If overload-protection is disabled then the alerts are disabled. The prometheus-rules-cdl pod is removed.

Configuring Overload Protection Parameter

The following parameters are configured to set limits for overload-protection:

• cdl datastore session features overload-protection index-max-record-count <value> 

• cdl datastore session features overload-protection slot-max-record-count <value> 

• cdl datastore session features overload-protection slot-max-size <value> 

• cdl datastore session features overload-protection hard-limit-percentage <value> 

The table below lists the configuration details:

cdl datastore session features overload-protection enable <true/false> 

cdl datastore session features overload-protection index-max-record-count <value> 

cdl datastore session features overload-protection slot-max-record-count <value> 

cdl datastore session features overload-protection slot-max-size <value> 

cdl datastore session features overload-protection hard-limit-percentage <value> 

cdl datastore session features overload-protection major-alert-threshold <value> 

cdl datastore session features overload-protection critical-alert-threshold <value> 

To configure the alert percentage, run the following command:

cdl datastore session features overload-protection critical-alert-threshold <percentage> 

cdl datastore session features overload-protection major-alert-threshold <percentage> 

You can verify the alerts using the following command in CEE Ops Center CLI:

show alerts active { detail | summary }  

show alerts active detail sumary "CDL is overloaded." 

In the following example, the cdlOverloaded - major alert is triggered when the system capacity is greater than or equal to 80%

alerts active detail cdlOverloaded-major 5446095ab264
 severity    major
 type        "Processing Error Alarm"
 startsAt    2020-10-15T15:09:00.425Z
 source      System
 summary     "CDL is overloaded."

In the following example, the cdlOverloaded - critical alert is triggered when the system capacity is greater than or equal to 90%

alerts active detail cdlOverloaded-critical 5446095ab264
 severity    critical
 type        "Processing Error Alarm"
 startsAt    2020-10-15T15:09:16.425Z
 source      System
 summary     "CDL is overloaded."

Issue Description

The session data is not replicated properly to the remote site.

Identifying the Issue

The data added at one site is inaccessible at the other site.

Possible Causes

The incorrect Geo-replication configuration may cause this issue.

Resolution

To resolve this issue:

• Verify the local system-id and remote site configuration.

• Verify whether CDL endpoints and Kafka between each site are reachable.

• Verify the map, replica of index and slot at each site. They must be identical across all the sites.

This section describes how to resolve the CDL operation failure issue.

Issue Description

The NF connects to the CDL, but the session operations such as find, create, update, and delete fail.

Identifying the Issue

You can view the NF log files to identify the call failure.

Possible Causes

You might encounter this issue when the CDL Index and Slot pods are not ready.

Resolution

To resolve this issue:

• Verify whether all the pods are in ready and running state.

• The Index pods move to the ready state only if the synchronization is complete with peer replica (local or remote if available).

• The Slot pods move to the ready state only if the synchronization is complete with peer replica (local or remote if available).

• At least one slot and index pod must be available for the Endpoint to move to ready state. Even when it isn’t ready, the client accepts the GRPC connection.

This section describes how to bring up the CDL pods when they are down.

Issue Description

The CDL pods are not in the "running" state because of incorrect CDL configuration.

Identifying the Issue

Verify the "describe pods" output (Containers, Member, State, Reason, or Events) to identify whether the pods are down with the following command:

kubectl describe pods -n  <namespace> <failed pod name> 

Possible Causes

The possible causes are:

• Pods are in "pending" state.

• Pods are in "CrashLoopBackOff" failure state.

• Pods are in "ImagePullBack" failure state.

Resolution

• When the pods are in "pending" state:

  • Verify whether the k8s nodes with the label value cdl/node-type are present. Also, ensure that the number of replicas are less than or equal to the number of k8s nodes with the label value cdl/node-type.

    kubectl get nodes -l smi.cisco.com/node-type=<value of cdl/node-type, default value is 'session' in multi node setup)
    

• When the pods are in "CrashLoopBackOff" failure state:

  • Verify the status of the ETCD pods.

    kubectl describe pods -n <namespace> <etcd pod name>

    If the ETCD pods are not running, resolve the ETCD issues to bring up the pods.

• When the pods are in "ImagePullBack" failure state:

  • Verify whether the helm repository and image registry are accessible.

  • Verify whether the required proxy and DNS servers are configured.

This section describes how to move the Mirror Maker pods that remain in the "Not Ready" state.

Issue Description

The Mirror Maker pods are not in running state because of connectivity issues on the remote site.

Identifying the Issue

Verify the Describe pod's output and pod logs with the following command:

kubectl describe pods -n <namespace> <failed pod name>
kubectl logs -n <namespace> <failed pod name> [-c <container name>]

Possible Causes

Connectivity issues with remote site Kafka brokers may cause this issue.

Resolution

To resolve this issue:

• Verify whether the external IPs configured for Kafka are accurate.

• Verify whether the remote sites of Kafka are reachable through the external IPs.

This section describes how to resolve the early or delayed notification sent for purging record from CDL.

Issue Description

The notification sent for purging record from CDL is either early or delayed.

Identifying the Issue

While purging record from CDL, the notification comes either early or delayed.

Possible Causes

The nodes in the Kubernetes cluster are not time synchronized.

Resolution

To resolve this issue:

• Verify whether all the nodes in the k8s cluster are time synchronized.

• Verify the synchronization status on all the k8s nodes.

  chronyc tracking
  chronyc sources -v
  chronyc sourcestats -v

Issue Description

The number of sessions pile up because of some network issues or an unexpected number of sessions on CDL, which does not match with the peer session count.

Identifying the Issue

The session count from the CLI or Grafana dashboard displays a higher session count than expected or the sessions are rising constantly.

Possible Causes

The Create request received at CDL are more than the Delete request. Consequently, the session count keeps on growing.

Resolution

To resolve this issue, the respective NF must take appropriate action for triggering clear for the subscribers.

This section describes how to resolve unschedulable pods issue that may arise by using single node label for scheduling, node affinity, and pod placement in CDL pods.

Issue Description

The nodeType parameter controls scheduling, node affinity, and pod placement in CDL. Currently, this parameter is configured with a single node label value smi.cisco.com/node-type. When CDL is configured to its full node capacity, the pods are marked as unschedulable (sometimes) because of the single node label usage.

Identifying the Issue

The CDL pods are marked as unschedulable.

Possible Causes

The usage of single node label for scheduling, node affinity, and pod placement.

Resolution

You can control different CDL pod types using separate node labels. To create separate node labels:

1. (Mandatory) Create smi.cisco.com/node-type label on all the nodes used for CDL through SMI Cluster Manager Ops Center CLI.

2. Create labels for CDL pods as specified in the following table:

   Pods	Label	Value
   CDL Endpoint pods	smi.cisco.com/cdl-ep	true
   CDL Slot pods	smi.cisco.com/cdl-slot-<MAP Id 1..n>	true
   CDL Index pods	smi.cisco.com/cdl-index-<MAP Id 1..n>	true

   Note	This configuration is required only for user-defined scheduling.

Example

The following example shows how to create separate node labels for CDL which uses four different nodes with two endpoint replicas, four Slot maps (with two replicas each) and two Index maps (with two replicas each).

clusters user-k8s
  nodes session1
   k8s node-labels smi.cisco.com/cdl-ep true
   exit
   k8s node-labels smi.cisco.com/cdl-index-1 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-1 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-3 true
   exit
   k8s node-labels smi.cisco.com/node-type session
   exit
  exit
exit
clusters user-k8s
  nodes session2
   k8s node-labels smi.cisco.com/cdl-ep true
   exit
   k8s node-labels smi.cisco.com/cdl-index-1 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-1 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-3 true
   exit
   k8s node-labels smi.cisco.com/node-type session
   exit
  exit
exit
clusters user-k8s
  nodes session3
   k8s node-labels smi.cisco.com/cdl-ep true
   exit
   k8s node-labels smi.cisco.com/cdl-index-2 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-2 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-4 true
   exit
   k8s node-labels smi.cisco.com/node-type session
   exit
  exit
exit
clusters user-k8s
  nodes session3
   k8s node-labels smi.cisco.com/cdl-ep true
   exit
   k8s node-labels smi.cisco.com/cdl-index-2 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-2 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-4 true
   exit
   k8s node-labels smi.cisco.com/node-type session
   exit
  exit
exit

cdl label-config session
  endpoint key smi.cisco.com/cdl-ep
  endpoint value true
  slot map 1
    key smi.cisco.com/cdl-slot-1
    value true
  slot map 2
    key smi.cisco.com/cdl-slot-2
    value true
  slot map 3
    key smi.cisco.com/cdl-slot-3
    value true
  slot map 4
    key smi.cisco.com/cdl-slot-4
    value true
  index map 1
    key smi.cisco.com/cdl-index-1
    value true
  index map 2
    key smi.cisco.com/cdl-index-2
    value true
cdl datastore session
  label-config session
  endpoint replica 2
  index replica 2
  index map 2
  slot replica 2
  slot map 4
exit

You can monitor various activities in CDL through the Grafana dashboard called CDL Dashboard, which is bundled by default. The CDL Dashboard displays the TPS towards CDL for various pods like cdl-endpoint, Slot, Index and the response time taken for each operation.

A sample Grafana CDL Summary dashboard displaying the total records by Type, SliceName, SystemID and so on is shown below:

A sample Grafana CDL dashboard displaying the CDL TPS and response time graphs is shown below:

The Grafana CDL dashboard is enhanced to show the Geo Replication status. A sample Grafana dashboard displaying the Geo Replication status and other details is shown below.

The GR connection, Index replication and slot replication panels, and their descriptions are listed in the table below:

You can verify the status of the CDL and inspect the data inserted by the client using the cdl show status command in the CDL Ops Center. For more information on cdl show status command, see the UCC CDL Command Reference Guide.

The following example displays the status of all the CDL components and client's data.

[unknown] cdl# cdl show status    
message params: {cmd:status mode:cli dbName:session sessionIn:{mapId:0 limit:10000 filters:[]}}
site {
    system-id 1
    cluster-id 1
    db-endpoint {
        state STARTED
    }
    slot {
        map-id 1
        instance-id 2
        records 100
        capacity 1000000
        state STARTED
    }
    slot {
        map-id 1
        instance-id 1
        records 100
        capacity 1000000
        state STARTED
    }
    index {
        map-id 1
        instance-id 2
        records 500
        capacity 2500000
        state ONLINE
    }
    index {
        map-id 1
        instance-id 1
        records 500
        capacity 2500000
        state ONLINE
    }
}
[unknown] cdl#