Field Notice: FN74065 - Cisco DNA Center: etcd PKI Certificate Activation Failure Disables User Interface and Causes Other Errors - Software Upgrade Recommended

On affected releases of Cisco DNA Center, the Public Key Infrastructure (PKI) certificates for the etcd container are not activated after they automatically renew. This condition leaves the etcd container pointing to stale certificates instead of the renewed certificates. Once the stale etcd certificates expire, the web UI becomes unavailable, and other features may stop functioning.

Cisco DNA Center introduced digital certificates for etcd in Release 2.3.2.0 to ensure secure data communication with Kubernetes, both within a node and between nodes in a cluster. These certificates are valid for one year and are automatically renewed before they expire. The renewed certificates are processed by a helper container and then made available to the etcd container. In affected Cisco DNA Center releases, the etcd container does not recognize and activate those renewed certificates dynamically and continues to point to the expired certificates until etcd is restarted.

Because Kubernetes function is impacted by this problem, there are multiple symptoms, many of which result in the Cisco DNA Center node appearing unresponsive. For example, the main web user interface may stop functioning or become inaccessible. Also, SSL: CERTIFICATE_VERIFY_FAILED errors may be seen on the CLI console, as shown in the following example:

WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)'),)': /v2/keys/maglev/config/node-x.x.x.x?sorted=true&recursive=true

If the etcd certificates on a Cisco DNA Center node have already expired and the system is already experiencing symptoms, contact the Cisco Technical Assistance Center (TAC) for help. The workaround procedure to restore function requires intervention from authorized Cisco personnel. Once affected nodes have been fully returned to service, install one of the fixes that is described below to prevent the problem from recurring.

Cisco has developed a software fix for this problem.

In the following table, the left column shows the affected Cisco DNA Center releases and the right column shows the recommended solutions.

Note: This table contains new affected releases and relevant solutions that have been added since the original publication of this Field Notice on September 29, 2023.

Select the recommended solution that matches your current affected release:

CSCwe15923 Fix Script Procedure

Cisco recommends that customers who are running Cisco DNA Center releases 2.3.3.6 and earlier (except AIRGAP) upgrade to Release 2.3.3.7.

Customers who are unable to upgrade to Release 2.3.3.7 or who are running an AIRGAP release can instead download and run the CSCwe15923 Fix script for this problem. The script takes no more than three minutes to run, and the entire procedure should take no longer than 30 minutes for a single node. To download, install, and run the script, complete the following steps: 

1. Download the script tar file CSCwe15923-recovery-artifacts-2.3.3.x-RA.tar.gz from Cisco.com.
2. Transfer the downloaded file to the Cisco DNA Center node using an SCP/SFTP tool. For a three-node Cisco DNA Center cluster, transfer the script to all three nodes. The following is an example CLI command where <Node_IP_Address> is the IP address of the Cisco DNA Center cluster node:
   

   scp -P 2222 CSCwe15923-recovery-artifacts-2.3.3.x-RA.tar.gz maglev@<Node_IP_Address>:/

3. For Disaster Recovery (DR) deployments only, pause DR first, then follow the remaining steps on both the Main and Recovery sites. For instructions on how to pause DR, see the Pause Your Disaster Recovery System section of the Cisco DNA Center Administrator Guide.
4. Log in to Cisco DNA Center as the maglev user. 
5. Enter the _shell CLI command to gain full shell access.
6. Enter the maglev password when prompted.
7. Move the script tar file to the /data/tmp directory with the mv CLI command, as shown in this example:
   

   mv /home/maglev/.magshell/CSCwe15923-recovery-artifacts-2.3.3.x-RA.tar.gz  /data/tmp/

8. Enter the cd /data/tmp command to change to the /data/tmp directory.
9. Extract the contents of the tar file with the tar CLI command, as shown in this example:
   

   tar zxvf CSCwe15923-recovery-artifacts-2.3.3.x-RA.tar.gz

10. Run the script file that was extracted in Step 9 with the sudo CLI command, as shown in this example:
    

    sudo ./CSCwe15923.sh

11. For DR deployments only, rejoin DR. For instructions on how to rejoin DR, see the Rejoin Your System section of the Cisco DNA Center Administrator Guide.

Note: For a three-node cluster, repeat steps 4 - 11 for each additional node in the cluster.

Hotfix Installation Procedure

Before installing the Hotfix on your system, consult the following release notes:

• Release 2.3.3.7-HF4 Hotfix Release Notes
• Release 2.3.5.3-HF5 Hotfix Release Notes
• Release 2.3.5.4-HF3 Hotfix Release Notes

To download and install the Hotfix, complete the following steps:

Note: For DR deployments only, pause DR first, then follow the steps on both the Main and Recovery sites. For instructions on how to pause DR, see the Pause Your Disaster Recovery System section of the Cisco DNA Center Administrator Guide.

1. Click the menu icon at the top left corner of the UI and choose System > Software Management.
2. Open the list of available releases and confirm the appropriate hotfix release is present, as shown in the figure below.
3. Choose the hotfix and click Download now.
4. After the package download completes, select the package and click Install now.
5. Wait for Cisco DNA Center to complete its pre-checks, then click Install.
6. (Optional) Click More details to monitor installation progress. This link opens a slide-in pane that displays the package being installed and its progress.
7. For DR deployments only, rejoin DR. For instructions on how to rejoin DR, see the Rejoin Your System section of the Cisco DNA Center Administrator Guide.