Replace Telemetry Broker Identity Certificate

Introduction

This document describes how to replace the Server Identity Certificate on the Cisco Telemetry Broker (CTB) Manager Node.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Cisco Telemetry Broker appliance administration
• x509 Certificates

Components Used

The appliances used for this document are running version 2.0.1

• Cisco Telemetry Broker Manager Node
• Cisco Telemetry Broker Broker Node

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Configure

Certificate Requirements

The x509 certificate used by the Cisco Telemetry Broker Manager must meet these requirements:

• The Cert and Private Key must be a matching pair
• The Certificate and Private Key must be PEM-encoded
• The Private Key must not be passphrase protected

Confirm Certificate and Private Key are matching pair

Log in to the CTB Manager command line interface (CLI) as the admin user.

Note: It is possible that the files mentioned in this section do not yet exist on the system.

The sudo openssl req -in server.csr -pubkey -noout -outform pem | sha256sum command outputs the SHA-256 checksum of the public key from the Certificate Signing Request file. 

The sudo openssl pkey -in server_key.pem -pubout -outform pem | sha256sumcommand outputs the SHA-256 checksum of the public key from the private key file.

The sudo openssl x509 -in server_cert.pem -pubkey -noout -outform pem | sha256sumcommand outputs the SHA-256 checksum of the public key from the issued certficate file.

The Certificate and Private Key output must match. If a Certificate Signing Request was not used then the server_cert.pem file does not exist.

admin@ctb-manager:~$ sudo openssl req -in server.csr -pubkey -noout -outform pem | sha256sum 
3e8e6b0d397ada1be7e89be21eb555c9527741460074385730d524c60e3ae315  -
admin@ctb-manager:~$  

admin@ctb-manager:~$ sudo openssl pkey -in server_key.pem -pubout -outform pem | sha256sum
3e8e6b0d397ada1be7e89be21eb555c9527741460074385730d524c60e3ae315  -

admin@ctb-manager:~$ sudo openssl x509 -in server_cert.pem -pubkey -noout -outform pem | sha256sum 
3e8e6b0d397ada1be7e89be21eb555c9527741460074385730d524c60e3ae315  -

Confirm Private Key is not passphrase protected

Log in to the CTB Manager as the admin user. Run the ssh-keygen -yf server_key.pem command.

A passphrase is not requested if the private key does not require one.

admin@ctb-manager:~$ ssh-keygen -yf server_key.pem 
ssh-rsa {removed for brevity}
admin@ctb-manager:~$

Confirm Certificate and Private Key are PEM-encoded

Note: These validations can be performed prior to installing the certificates.

Log in to the CTB Manager as the admin user.

View the server_cert.pem file content with the sudo cat server_cert.pem command. Adjust the command to your certificate file name.

The first and last lines of the file should be -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----  respectively.

admin@ctb-manager:~$ sudo cat server_cert.pem 
-----BEGIN CERTIFICATE-----
{removed_for_brevity}
-----END CERTIFICATE-----
admin@ctb-manager:~$  

View the server_key.pem file with the sudo cat server_key.pem command. Adjust the command to your private keys file name.

The first and last lines of the file should be -----BEGIN PRIVATE KEY----- and -----END PRIVATE KEY-----  respectively.

admin@ctb-manager:~$ sudo cat server_key.pem 
-----BEGIN PRIVATE KEY-----
{removed_for_brevity}
-----END PRIVATE KEY-----
admin@ctb-manager:~$  

Self Signed Certificate

Generate Self Signed Certificate

1. Log in to the CTB Manager over a SSH (Secure Shell) as the user configured during the installation, this is usually the "admin" user.
   
   
2. Issue the sudo openssl req -x509 -newkey rsa:{key_len} -nodes -keyout server_key.pem -out server_cert.pem -sha256 -days 3650 -subj /CN={ctb_manager_ip} command.
   • Change the rsa:{key_len} with a private key length of your choice such as 2048, 4096, or 8192
   • Change the {ctb_manager_ip}with the IP of the CTB Manager Node
     
     

     admin@ctb-manager:~$ sudo openssl req -x509 -newkey rsa:4096 -nodes -keyout server_key.pem -out server_cert.pem -sha256 -days 3650 -subj /CN=10.209.35.152   
     [sudo] password for admin: 
     Generating a RSA private key
     .....................................................................................++++
     ............................................................................++++
     writing new private key to 'server_key.pem'
     -----
     admin@ctb-manager:~$  ​

3. View the server_cert.pem file with the cat server_cert.pem command, and copy the contents to your buffer so that it can be pasted to the local workstation into a text editor of choice. Save the file. You can also SCP these files off of the /home/admin  directory.
   
   

   admin@ctb-manager:~$ cat server_cert.pem 
   -----BEGIN CERTIFICATE-----
   {removed_for_brevity}
   -----END CERTIFICATE-----
   admin@ctb-manager:~$  ​

   
   
   
4. View the server_key.pem file with the sudo cat server_key.pem command, and copy the contents to your buffer so that it can be pasted to the local workstation into a text editor of choice. Save the file. You can also SCP this file off of the /home/admin directory.
   
   

   admin@ctb-manager:~$ sudo cat server_key.pem 
   -----BEGIN PRIVATE KEY-----
   {removed_for_brevity}
   -----END PRIVATE KEY-----
   admin@ctb-manager:~$  ​

Upload Self Signed Certificate

Certificate Authority (CA) Issued Certificates

Generate Certicate Signing Request (CSR) for issuance by a Certificate Authority

1. Log in to the CTB Manager over a SSH (Secure Shell) as the user configured during the installation, this is usually the "admin" user.
   
   
2. Issue the openssl req -new -newkey rsa:{key_len} -nodes -addext "subjectAltName = DNS:{ctb_manager_dns_name},IP:{ctb_manager_ip}" -keyout server_key.pem -out server.csr command. The 'extra' attributes on the last two lines can be left blank if desired.
   
   • Change the {ctb_manager_dns_name} with the DNS name of the CTB Manager Node
   • Change the {ctb_manager_ip}with the IP of the CTB Manager Node
   • Change the {key_len} with a private key length of your choice such as 2048, 4096, or 8192. 
     
     

     admin@ctb-manager:~$ openssl req -new -newkey rsa:4096 -nodes -addext "subjectAltName = DNS:ctb-manager,IP:10.209.35.152" -keyout server_key.pem -out server.csr
     Generating a RSA private key
     ......................++++
     ....................................++++
     writing new private key to 'server_key.pem'
     -----
     You are about to be asked to enter information that will be incorporated
     into your certificate request.
     What you are about to enter is what is called a Distinguished Name or a DN.
     There are quite a few fields but you can leave some blank
     For some fields there will be a default value,
     If you enter '.', the field will be left blank.
     -----
     Country Name (2 letter code) [AU]:US
     State or Province Name (full name) [Some-State]:North Carolina
     Locality Name (eg, city) []:RTP
     Organization Name (eg, company) [Internet Widgits Pty Ltd]:Cisco Systems Inc
     Organizational Unit Name (eg, section) []:TAC
     Common Name (e.g. server FQDN or YOUR name) []:ctb-manager
     Email Address []:noreply@cisco.com
     
     Please enter the following 'extra' attributes
     to be sent with your certificate request
     A challenge password []:
     An optional company name []: ​

     
     
     
3. SCP the CSR and the Key files to a local machine and provide the CSR to the CA. Issuance of the CSR by the CA in PEM format is outside the scope of this document.

Create a Certificate with Chain

The CA issues the server identity certificate in PEM format. A chain file must be created that include all chain certificates and the server identity certificate for the CTB Manager Node.

In a text editor create a file by combining the certificate that was signed in the previous step and appending all the certificates in the chain all the way including the trusted CA into a single file in PEM format in the order shown.

— BEGIN CERTIFICATE — 
{CTB Manager Issued Certificate}
— END CERTIFICATE —
— BEGIN CERTIFICATE —
{Issuing Certificate Authority Certificate}
— END CERTIFICATE —
— BEGIN CERTIFICATE —
{Intermediate Certificate Authority Certificate}
— END CERTIFICATE —
— BEGIN CERTIFICATE —
{Root Certificate Authority Certificate}
— END CERTIFICATE — 

Ensure that this new certificate file with chain file has no leading or trailing spaces, blank lines, and is in the order shown above.

Upload Certificate Authority Issued Certificate

Verify

Log in into the CTB Manager Node Web Console and navigate to Settings > TLS Certificate to see certificate details such as a new expiry date, or view the certificate details using the browser to view more detailed information such as serial numbers.

CTB Certificate Details

Verify the CTB Broker Node shows no alarms in the CTB Manager Node Web UI.

Troubleshoot

If the certificate is incomplete such as lacking the chain certificates, the CTB Broker Node Node is not able to communicate with the Manager Node and presents "Not Seen Since" in the Status column in the list of Broker Nodes.
The Broker Node will continue to replicate and distribute traffic in this state.

Log in to the CTB Manager Node CLI and issue the sudo grep -ic begin /var/lib/titan/titanium_frontend/ssl/cert.pem command to see how many certificates are in the cert.pem file.

admin@ctb-manager:~$ sudo grep -ic begin /var/lib/titan/titanium_frontend/ssl/cert.pem
[sudo] password for admin: 
3 <-- Output 
admin@ctb-manager:~$ 

The output value returned needs equal the number of CA devices in the chain plus the CTB Manager.

The output of 1 is expected if using a self signed certificate.

The output of 2 is expected if the PKI Infrastructure consists of a single Root CA that is also the Issuing CA.

The output of 3 is expected if the PKI Infrastructure consists of a Root CA, and the Issuing CA.

The output of 4 is expected if the PKI Infrastructure consists of a Root CA, a Subordinate CA, and the Issuing CA.

Compare the output to the PKI listed when viewing the certificate in another appliaction such as Microsoft Windows Crypto Shell Extensions.

PKI Infrastructure

In this image the PKI infrastructure includes a Root CA, and the Issuing CA.

The output value from the command  is expected to be 3 in this scenario.

If the output does not meet expectations, review the steps in the Create a Certificate with Chain section to determine if a certificate was missed.

When viewing a certificate in Microsoft Windows Crypto Shell Extensions it is possible that not all certificiates to be presented if the local machine does not have enough information to verify the certificate.

Issue the sudo ctb-mayday command from the CLI to generate a mayday bundle for TAC to review.