The Cisco Prime Cable Provisioning (PCP) 7.4 RDU Container Deployment Guide describes the concepts and configurations of Prime Cable Provisioning Regional Distribution Unit Container (RDU-C).

The two Docker container images provided with the PCP 7.4 release:

• pcp-rdu-app:7.4 - The authoritative datastore and device configuration generation engine of PCP.

• pcp-rdu-webui:7.4 - The web-based user interface for the RDU Container.

Prime Cable Provisioning 7.4 adds support to run multiple RDU Containers in Kubernetes cluster. A new license to support the container function of RDU is now required. This license enables ability to use containerized RDU. To use the new feature function, purchase a RDU_CNT license.

The following figure shows a sample Manage License Keys page of the Admin UI, which displays the summary of licenses that has been added for the implementation:

RDU can be run as a container in a Kubernetes cluster. You can download the Prime Cable Provisioning RDU Container Docker images and relevant manifests for Prime Cable Provisioning 7.4 from, Download Software .

1. Download the following software images:

   • pcp-rdu-app_7_4.tar.gz

   • pcp-rdu-webui_7_4.tar.gz

2. Extract the files with .tar.gz extension:

   # tar -zxvf pcp-rdu-app_7_4.tar.gz

   The utility creates the following artifacts:

   a. pcp-rdu-app.tar : A tar archive of pcp-rdu-app:7.4 Docker image.

   b. manifests: A directory with manifests needed for rdu-app container.

   # tar -zxvf pcp-rdu-webui_7_4.tar.gz

   The utility creates the following artifacts:

   a. pcp-rdu-webui.tar : A tar archive of pcp-rdu-webui:7.4 Docker image.

   b. manifests : A directory with manifests needed for rdu-webui container.

3. To load the respective Docker images:

   # docker load -i pcp-rdu-app.tar

   # docker load -i pcp-rdu-webui.tar

It is recommended to verify the signature of the downloaded images. Your premises verification script can contact Cisco to download root and subCA certs. Python(3) needs to be installed.

1. Extract the public key from the public cert:

   $ openssl x509 -pubkey -noout -in PCP_RELEASE_7X-CCO_RELEASE.pem > PCP_RELEASE_7X-CCO_RELEASE.pubkey

2. To check whether the verify script is using the public key and signature files:

   $ openssl dgst -sha512 -verify PCP_RELEASE_7X-CCO_RELEASE.pubkey -signature cisco_x509_verify_release.py3.signature cisco_x509_verify_release.py3
   Verified OK

3. Verify the product image:

   $ python3 cisco_x509_verify_release.py3 -e PCP_RELEASE_7X-CCO_RELEASE.pem -s <image signature file> -i <image> -v dgst -sha512

   Image signature file	Image
   pcp-rdu-app_7_4.tar.gz.signature	pcp-rdu-app_7_4.tar.gz
   pcp-rdu-webui_7_4.tar.gz.signature	pcp-rdu-webui_7_4.tar.gz

1. A Kubernetes Cluster:

   1. The cluster should be configured with dual-stack support (IPv4 and IPv6) i.e. pod-CIDR and service-CIDR should have subnets of both IP families.

   2. The network services offered by RDU use a wider range of ports: 49187 and 49188

      By default, the range of the service NodePorts is 30000-32768. Hence, the range of NodePorts has to be expanded.

      This can be achieved by configuring kube-apiserver (--service-node-port-range=37-50000).

   Note	Cisco has tested PCP 7.4 RDU Container with Kubernetes v1.30.

2. Calico CNI installed for dual-stack support (IPv4 and IPv6).

   Calico IP Pools should have NAT Outgoing set to true (spec.natOutgoing).

   Note	Cisco has tested PCP 7.4 RDU Container with Calico v3.24 configured with IPIP encapsulation.

3. PostgreSQL Database Server:

   RDU Container acts as a client application connecting to PostgreSQL database server.

   The Database Administrator is responsible to configure and tune the PostgreSQL database server to be used by the RDU Container.

   1. A database has to be created (SQL: CREATE DATABASE) for RDU Container to store its data. RDU Container uses 'public' schema.

   2. A database user with password has to be created (SQL: CREATE USER) which RDU Container application uses to connect to the above created database. It is recommended that the user be the owner of the database.

   3. Configure PostgreSQL client authentication (pg_hba.conf) to allow connections to the database created in (a) by user created in (b) from the IP addresses of worker nodes in the Kubernetes cluster.

      DPE Container supports password-based authentication methods (scram-sha-256, md5, password) configured on PostgreSQL database server.

   Example:

   An entry in pg_hba.conf:

   'rdudbuser' database user is allowed to connect to 'rdudb' database from worker nodes in 10.0.0.0/26 network if password is correctly supplied

   #	Type	Database	User	Address	Method
   	host	rdudb	rdudbuser	10.0.0.0/26	scram-sha-256

   Note	Cisco has tested PCP 7.4 RDU Container with PostgreSQL 16 running on Linux.

4. ZooKeeper:

   Apache ZooKeeper is a distributed coordination service. RDU Containers act as client application to ZooKeeper server and use it for coordination.

5. DNS entries for IPv4 and IPv6 address of Kubernetes worker nodes.

   For RDU and CPNR Extension Points to connect to DPE Containers running on Kubernetes worker nodes, IPv4 and IPv6 address of every worker node should be available in DNS (A and AAAA records respectively).

PCP 7.4 allows you to run multiple containers of rdu-app. Typically just one container of rdu-webui is enough.

A typical PCP 7.4 deployment with RDU Container will look like:

1. A single Kubernetes cluster with RDU Containers (one rdu-webui pod and multiple rdu-app pods).

2. DPE Containers (one dpe-cli pod and multiple dpe-app pods).

3. CPNR Extension Points (CNR-EP) Containers running on a Linux node (either standalone or failover enabled).

4. PostgreSQL database cluster running on Linux machines (running a single server leads to single point of failure, so always run PostgreSQL server with replication enabled).

5. ZooKeeper ensemble running on Linux machines (running a single server leads to single point of failure, so always run ZooKeeper servers in an ensemble).

Note

Database and Coordination services are not part of the PCP software image. You have to download, install, configure and maintain them to be consumed by RDU Containers. Please see https://www.postgresql.org/download/linux/redhat/ and https://zookeeper.apache.org/releases.html. We highly recommend to configure and use highly-available deployment of PostgreSQL and ZooKeeper servers.

RDU Container deployment comprises of 2 containers:

• rdu-app
• rdu-webui

The RDU (rdu-app) container is built to use PostgreSQL database server. Please see the Requirements section for more details.

ConfigMap (rdu-app-cm) is the place to configure database connection URL.

It is recommended to run PostgreSQL database server with streaming replication enabled (e.g. one primary server + two secondary servers).

The database connection URL can be configured with multiple endpoints (host and port) to fully use replication and failover (high-availability) features, if configured on PostgreSQL database servers.

Example

/db/postgres/url=jdbc:postgresql://rdu-db-01:5432,rdu-db-02:5432,rdu-db-03:5432/rdudb

RDU Container tries to connect to the database server in the order of the hosts given in the URL. RDU Container connects to the Primary database server if available. Only if Primary connection is unsuccessful, RDU Container connects to Secondary server.

Note

When RDU Container is connected to Primary (i.e. writable) database, it can accept write (add/update/delete) batches. This means, provisioning new devices and updates to configuration is possible. When RDU Container is connected to Secondary (i.e. read-only) database, it can accept read (query) batches only. New devices provisioning and configuration updates from write batches will fail. Once Primary database is back, RDU Container will automatically connect to it and resume provisioning new devices and accepting write batches.

The RDU (rdu-app) container is built to use ZooKeeper as a coordination server. Please see the Requirements section for more details.

ConfigMap (rdu-app-cm) is the place to configure ZooKeeper connection URL and other properties.

It is highly recommended to run ZooKeeper ensemble (cluster) and connection URL can be configured with multiple endpoints (host and port) to fully use high-availability features, if configured on ZooKeeper ensemble.

Example

/zk/connection/url=zoo-01:2181,zoo-02:2181,zoo-03:2181

By default, the RDU Container logs are available at the local filesystem of the container and underlying container runtime.

Users of regular RDU are aware that the current RDU logs are available at :

/var/CSCObac/rdu/logs/rdu.log
	/var/CSCObac/rdu/logs/rdu_auth.log
	/var/CSCObac/rdu/logs/rdu_crs.log
	/var/CSCObac/rdu/logs/audit.log

Even for the RDU Container (rdu-app), the logs are available in the container at: /var/CSCObac/rdu/logs/

To read the log file in the container:

$ kubectl exec -it <pod-name> -- bash
$ cd /var/CSCObac/rdu/logs/
$ tail -f rdu.log

RDU Container (rdu-app) logs can also be viewed by command : kubectl logs <pod-name>

This is because RDU Container logs can be captured by underlying container runtime. That way, RDU Container (rdu-app) logs can be aggregated using EFK (Elasticsearch-Fluentd-Kibana) Logging Stack in Kubernetes.

Fluentd is a Log Collector which runs as a DaemonSet on each node of Kubernetes cluster. It collects logs from pods running on each node.

Elasticsearch a Log Storage which can either run as Deployment/StatefulSet on Kubernetes cluster or VM based application outside the cluster.

Kibana is a visualization tool used of log analysis which can run as Deployment on Kubernetes cluster.

If you are using EFK stack for log collection and aggregation, and want to disable logs being written to local filesystem of container, you can set the following property in ConfigMap of dpe-app (dpe-app-cm.yaml):

/server/log/2/enable=false

RDU Container (rdu-app) is built to expose the application-specific metrics in Prometheus format. This allows centralized monitoring of the rdu-app pods.

Prometheus server must be configured to scrape metrics from RDU-C pods. RDU-C pod will respond with current value of metrics.

Metrics are specific to an instance of rdu-app pod. You are responsible to configure Prometheus to scrape metrics from every rdu-app pod at regular intervals.

RDU-C support both HTTP and HTTPS schemes for metrics endpoint, based upon the communication mode enabled:

1. If rdu-app pod is running in non-secure mode (i.e. /server/rdu/unsecure/enabled=true), metrics are exposed on HTTP endpoint with port=8189

2. If rdu-app pod is running in secure mode (i.e. /server/rdu/secure/enabled=true), metrics are exposed on HTTPS endpoint with port=8190. The public certificate for this HTTPS communication is same as used by rdu-app pod for secure communication with PACE clients.

The HTTP and HTTPS endpoint are exposed via Basic authorization. A RDU administrator (user) with privilege=PRIV_RDU_READ is authorized to make requests to /metrics endpoint.

Details of metrics HTTP endpoint are:

scheme: http
   		port: 8189
   		endpoint: /metrics
  		authorization: Basic

scheme: https
   		port: 8190
   		endpoint: /metrics
  		authorization: Basic

If you are running Prometheus in Kubernetes cluster, you can use Kubernetes Service Discovery configuration to scrape targets (which uses REST API of Kubernetes).

A sample job to scrape rdu-app metrics with Prometheus using Kubernetes Service Discovery configuration:

scrape_configs:
      - job_name: 'rdu-app'
        scrape_interval: 20s
        metrics_path: /metrics
        scheme: http
        basic_auth:
          username: metricsuser
          password: metricspassword
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_container_port_name]
            action: keep
            regex: metrics
          - source_labels: [__meta_kubernetes_pod_container_name]
            action: keep
            regex: dpe-app  
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: kubernetes_pod_name

The following are the metrics that are exported: