This document details steps to configure the database (db) clustering on Cisco Meeting Server (CMS) or Acano call bridges (CB).
Contributed by Joshua Alero, Cisco TAC Engineer.
Cisco recommends that you have at least 3 CB nodes to be able to create a viable db cluster.
Note: It's recommended to have an odd number of db cluster nodes as it is important for the master selection and the active failover mechanism. Another reason for this is that the master db node would be the node that has connections to the most of the db in the cluster. Up to 8 db cluster nodes have been tested at the time of the publication of this document, although it's technically possible to cluster a lot more.
Port 5432 opened on firewall.
Note: The db cluster master listens on port 5432 for connections from the client nodes, so if there's a firewall (FW) between the nodes, ensure that this port is opened.
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
There are two types of certificates for the database clustering:
1. Client: The client certificate, as the name sugest, is used by the database clients to connect to the database server (Master). This certificate must contain the string, postgres, in its Common Name (CN) field.
2. Server: The server certificate, as the name sugest, is used by the db server to connect to the postgres database.
Part 1. Certificate Creation
1. Connect with a Secure Shell (SSH) with the admin credentials to the server MMP
2. Generate Certificate Signing Request (CSR):
a. For the databasecluster client certificate:
pki csr <key/cert basename> CN:postgres
For example: pki csr databasecluster_client CN:postgres
b. For the databasecluster server certificate:
pki csr <key/cert basename> CN:<domainname>
For example: pki csr databasecluster_server CN:vngtpres.aca
Note: The CN for Part A must be postgres and Part B can be the domain name of the call bridge.
Part 2. Call Bridge Configuration
On the CB that will run the master database, follow these steps:
1. To select the interface to use:
database cluster localnode a
This will enable interface “a” to be used by the database cluster.
2. Define the client, server and root ca certificates as well as the private keys to be used by the database cluster with these commands:
Note: The same client and server certificates can be used on other CB nodes to be clustered by simply copying the private keys and certificates across to the other nodes. This is possible because the certificates contains no SAN tying them to a specific call bridge. However, it's recommend to have individual certificates for each db node.
3. Initialize this database on the local CB as the master for this database cluster:
database cluster initialize
4. On the CallBridges that would be part of the clustered db and become the db slaves run this command after you complete Steps 1 and 2 for Part 2:
database cluster join <Master CB IP address>
For example: database cluster join <10.48.36.61>
This will initiate the database synchronization and copy the database from the master peer.
Note: The local database that existed before the database cluster join command was initiated, will still continue to exist until the node is removed from the clustered database. So as long as the node is in the db cluster, its local db is not used.
Use this section to confirm that your configuration works properly.
To check the clustered database status, run this command on any of the nodes in the db cluster:
database cluster status
The output should be similar to this:
Status : Enabled Nodes: 10.48.36.61 : Connected Master 10.48.36.118 : Connected Slave ( In Sync ) 10.48.36.182 (me) : Connected Slave ( In Sync ) Node in use : 10.48.36.61
Interface : a
Certificates Server Key : dbclusterserver.key Server Certificate : dbclusterserver.cer Client Key : dbclusterclient.key Client Certificate : dbclusterclient.cer CA Certificate : vngtpRootca.cer Last command : 'database cluster join 10.48.36.61' (Success)
This section provides information you can use in order to troubleshoot your configuration.
Use this command, on the CLI, to view the current logs related to the db clustering:
The CMS log collector, see the link, provide an easy and user-friendly User Interface (UI) to collect logs from the CMS server:
Log outputs for the database usually contain the postgres string, examples as follows:
Mar 30 12:39:04 local0.warning DBMaster postgres: [2-7] #011SQL statement "INSERT INTO domains(domain_id, domain_name, tenant_id, target, priority, passcode_separator) VALUES (inp_domain_id, inp_domain_name, inp_tenant_id, existing_target, inp_priority, inp_passcode_separator)"
Mar 30 12:39:04 local0.warning DBMaster postgres: [2-8] #011PL/pgSQL function create_or_update_matching_domain(boolean,uuid,text,boolean,uuid,integer,integer,integer,text) line 61 at SQL statement
Mar 30 12:39:04 local0.warning DBMaster postgres: [2-9] #011SQL statement "SELECT * FROM create_or_update_matching_domain(TRUE, inp_domain_id, inp_domain_name, TRUE, inp_tenant_id, inp_target_true, 0, inp_priority, inp_passcode_separator)"
Mar 30 12:39:04 local0.warning DBMaster postgres: [2-10] #011PL/pgSQL function create_matching_domain(uuid,text,uuid,integer,integer,text) line 3 at SQL statement
Here are some typical db issues and solutions:
1. Database schema error on a non-master peer:
ERROR : Couldn't upgrade the schema
Status : Error
10.48.54.75 : Connected Master
10.48.54.76 : Connected Slave ( In Sync )
10.48.54.119 (me) : Connected Slave ( In Sync )
Node in use : 10.48.54.75
Interface : a
Server Key : dbclusterServer.key
Server Certificate : dbserver.cer
Client Key : dbclusterClient.key
Client Certificate : dbclient.cer
CA Certificate : Root.cer
Last command : 'database cluster upgrade_schema' (Failed)
a. First, run this command to clear the error:
database cluster clear error
b. Followed by this command to upgrade the database schema:
database cluster upgrade_schema
c. Then check to the status of the database clustering with:
database cluster status
The logs should show output similar to this:
Mar 30 11:22:45 user.notice acanosrv05 schema_builder: Upgrading schema with connect line 'connect_timeout=4 user=postgres host=127.0.0.1 port=9899 sslmode=verify-ca sslcert=/srv/pgsql/client.crt sslkey=/srv/pgsql/client.key sslrootcert=/srv/pgsql/ca.crt '
Mar 30 11:22:45 user.notice acanosrv05 schema_builder: Using database name 'cluster'
Mar 30 11:22:45 user.notice acanosrv05 schema_builder: schema build on database cluster complete
Mar 30 11:22:45 user.notice acanosrv05 schema_builder: Using CiscoSSL 1.0.1u.4.13.322-fips (caps 0x4FABFFFF)
Mar 30 11:22:45 user.notice acanosrv05 schema_builder: Using 0x1000115F
Mar 30 11:22:45 user.notice acanosrv05 schema_builder: INFO : Waiting for database cluster to settle...
Mar 30 11:22:45 user.notice acanosrv05 schema_builder: INFO : Database cluster settled
Mar 30 11:22:45 user.notice acanosrv05 schema_builder: Schema upgrade complete
Mar 30 11:22:45 user.info acanosrv05 dbcluster_watcher: Operation Complete
2. Peer nodes unable to connect to db master node:
Mar 31 10:16:59 user.info acanosrv02 sfpool: Health check 10.48.54.119: error (up = 1): could not connect to server: Connection refused|#011Is the server running on host "10.48.54.119" and accepting|#011TCP/IP connections on port 5432?|
Use these steps to collect traces to troubleshoot the connection issues:
a. Run the command pcap <interface> on the non-master (slave) node and after a few minutes, stop the capture with Ctrl-C.
b. Connect with a Secure File Transfer Protocol (SFTP) client to the server and download the .pcap file from the root directory:
c. Open the capture file on Wireshark and filter on port 5432 with tcp.port==5432 to check for traffic between the non-master peer and the db master.
d. If there's no return traffic from the server then it's likely that a FW might be blocking the port between the two servers' logical location.
Here is a typical packet capture from a working connection betwen the client and server:
In this example the client's IP is 10.48.54.119 and the server is 10.48.54.75.
For more on the troubleshooting of issues with, and questions that relate to db clustering, refer to the FAQs in these links: