Unity Connection Cluster Renegotiation Failure Troubleshooting TechNote

Introduction

This document describes the steps to verify and attempt to address the issue for cluster renegotiation failure in Cisco Unity Connection (CUC).

Prerequisites

Requirements

Cisco recommends that you have knowledge on Cisco Unity Connection.

Components Used

Cisco recommends that you have knowledge of this topic:

• Unity Connection Version 8.5 or Later

Background Information

Generally, every failure ends with this statement:

Cluster renegotiation failed.

The cluster renegotiation log cuc-cluster-rejoin_yyyy-mm-dd_hh.mm.ss.log can be found in the installation logging location through the CLI or Real-Time Monitoring Tool (RTMT).

In order to obtain this log from CLI, you need an SFTP server to transfer the file, and you need to enter this command: file get install cuc-cluster-rejoin_yyyy-mm-dd_hh.mm.ss.log

or

From the Real-Time Monitoring Tool (RTMT), choose Trace & Log Central > Collect Install Logs > Select the Node > Proceed.

In a few scenarios, the last 10 lines of the log provide the error message; Therefore, it can be viewed on the CLI. Enter this CLI command: file tail install cuc-cluster-rejoin_yyyy-mm-dd_hh.mm.ss.log to view the last 10 lines of the log.

Note: The utils cuc cluster renegotiate command is run on the Subscriber server only and copies the database from the Subscriber server to the Publisher server and establishes replication.

Troubleshoot

This section provides tips used in order to troubleshoot Cluster renegotiatio failure.

Cluster Renegotiation Failure

Problem: Scenario 1

The renegotiation command fails at the last step.

This sample output is seen on the Publisher server:

13/12/01 14:32:15 Disabling data replication...
13/12/01 14:32:28 Renegotiating ssh trusts...
13/12/01 14:32:36 Synchronizing platform and LDAP database...
13/12/01 14:35:20 Creating any missing messaging databases on the publisher...
13/12/01 14:35:23 Adding subscriber node to publisher...
13/12/01 14:35:30 Synchronizing Unity Connection databases...
13/12/01 14:43:19 Synchronizing file systems...
13/12/01 14:43:23 Synchronizing message files for mail store UnityMbxDb1...
13/12/01 14:43:25 Copying cluster DSCP configuration to publisher node...
13/12/01 14:43:27 Rebooting publisher node CUC-9A...

Cluster renegotiation failed.

Log Analysis

+ sudo -u cucluster ssh CUC-9A /usr/local/cm/bin/controlcenter.sh 
'Service Manager' stop
++ error
++ '[' 0 -eq 1 ']'
++ '[' 0 -eq 1 ']'
++ echo 'Cluster renegotiation failed.'
++ echo 'The cluster renegotiation log cuc-cluster-rejoin
_2013-12-01_14.32.15.log can be found in the installation logging location through the CLI or RTMT.'
++ exit 1

Solution

The trace analysis shows that the server is affected by Cisco bug ID CSCul75841. It fails at the last step when you try to stop the Service Manager. In order to resolve this issue, restart the Publisher server or enter this command utils cuc cluster overwritedb on the Publisher server.

Problem: Scenario 2

The upgrade fails on the Subscriber server when either of these CLI comnands are entered:

utils cuc cluster overwritedb fails either on PUB or SUB

utils cuc cluster renegotiate fails on SUB

The same issue is seen during Subscriber installation.

The root cause of this issue is that it fails to establish replication at the define server step.

Log Analysis

For Cluster renegotiation / OverwriteDB failure

+ sudo -u informix cdr define server -A /var/opt/cisco/connection/spool/ats/ -c g_ciscounity_sub1 -I g_ciscounity_sub1 -S g_ciscounity_pub
command failed -- fatal server error (100)
++ error
++ '[' 0 -eq 1 ']'
++ echo 'Cluster renegotiation failed.'

Or 

the same errors with the last line as
++ echo 'Cluster overwritedb failed.'

For subscriber install failure

Thu Oct 17 06:09:47 GMT+2 2013 + sudo -u informix cdr define server -A /var/opt/cisco/connection/spool/ats/ -c g_ciscounity_pub -I g_ciscounity_pub
Thu Oct 17 06:13:07 GMT+2 2013 command failed -- fatal server error (100)
Thu Oct 17 06:13:07 GMT+2 2013 + LOADDBRC=100Thu Oct 17 06:13:07 GMT+2 2013 + '[' 100 -ne 0 ']'Thu Oct 17 06:13:07 GMT+2 2013 + echo 'loaddb.sh return code was 100'Thu Oct 17 06:13:07 GMT+2 2013 loaddb.sh return code was 100Thu Oct 17 06:13:07 GMT+2 2013 + exit 1Thu Oct 17 06:13:07 GMT+2 2013 /opt/cisco/connection/lib/install/post.d/06_load-database had an exit code of 1error: %post(cuc-9.1.1.10000-32.i386) scriptlet failed, exit status 1

Solution

The trace analysis shows that the server is affected by Cisco bug ID CSCue78730. In order to work around this issue, contact Cisco TAC. Alternatively, upgrade the server to a fixed version of the defect.

Problem: Scenario 3

Renegotiation fails while renegotiating ssh trusts in the second step.

Log Analysis

+ echo '11/11/30 20:57:24 Renegotiating ssh trusts..
+ /opt/cisco/connection/lib/install/post.d/02_authorize-cucluster
+ . /usr/local/bin/base_scripts/icluster.sh
++ '[' -n '' ']'
++ IPM_BAD_REMOTE_FILE_ERROR=12
++ IPM_BAD_REMOTE_DIR_ERROR=13
The fingerprint for the RSA key sent by the remote host
is bb:c0:b3:a7:08:07:ef:0c:f9:86:11:1d:a2:99:5e:8a.
Please contact your system administrator.
Add correct host key in /home/sftpuser/.ssh/known_hosts to get rid of this message.
Offending key in /home/sftpuser/.ssh/known_hosts:5

Solution

Complete these steps, in order to resolve this issue:

1. Ensure that the Security Password is same on both the servers.
2. Reset the Security Password if required.
3. If the issue still exists, contact Cisco TAC to check the known_hosts file from the root.

Problem: Senario 4

Renegotiate fails at this step,
yy/mm/dd hh:mm:ss Synchronizing Unity Connection databases...
Cluster renegotiation failed.

Log Analysis

+ sudo -u informix cdr define server -A /var/opt/cisco/connection/spool/ats/ -c g_ciscounity_sub1 -I g_ciscounity_sub1 -S g_ciscounity_pub connect to g_ciscounity_sub1 failed
Incorrect password or user g_ciscounity_sub1 is not known on the database server.
(-951)command failed -- unable to connect to server specified (5)

Solution

Complete these steps, in order to resolve this issue:

1. Ensure that the Domain Name Server (DNS)/Domain name entries are correct if they are 'set' on both the servers.
2. Make sure there are no reverse DNS lookup issues with the utils diagnose testcommand.
3. If the issue still exists, contact Cisco TAC to check on various hosts file. The Issue might be with the rhosts and SQLhosts file.

Problem: Scenario 5

Renegotiation fails as it the script fails to create databases on the Publisher server.

Log Analysis

+ python - -s ciscounity /opt/cisco/connection/lib/config-modules/dbscripts/mailstore/add-missing-mbxdb-space.sh: line 37: cannot create temp file for here document: Permission denied++ error ++ '[' 0 -eq 1 ']' ++ echo 'Cluster renegotiation failed.

Solution

The trace analysis shows that the server is affected by Cisco bug ID CSCtr18463. In order to resolve this issue, upgrade the version to a fixed version.

Problem: Scenario 6

Cluster renegotiation fails due to NTP issues.

Log Analysis

+ sudo -u informix cdr define server -A /var/opt/cisco/connection/spool/ats/ -c g_ciscounity_sub1 -I g_ciscounity_sub1 -S g_ciscounity_pub
command failed -- System clocks difference is too large. 
(90)
++ error
++ '[' 0 -eq 1 ']'
++ echo 'Cluster renegotiation failed.'

Solution

In order to resolve this issue, you must fix any Network Time Protocol (NTP) issues, and assign an NTP with a good stratum value. For Unity Connection, a stratum 1 or 2 source is preferred.

Problem: Scenario 7

The following error is displayed when the Renegotiation fails.

SSH trust renegotiation failed.The security password on the publisher and subscriber servers do not match.
 Run the the CLI command "set password user security" on one or both servers to update the security password, then re-run "utils cuc cluster renegotiate".
Cluster renegotiation failed.

Solution

Complete these steps, In order to resolve this issue:

1. Enter this CLI command set password user security on one or both the servers to update the Security Password.
2. This error might occur when the subscriber's IP Address/Hostname is not entered in the publisher's system settings and the cluster page.
3. If the issue still exists, contact Cisco TAC.

Problem: Scenario 8

Renegotiation fails with this error in logs.

Log Analysis

+ /opt/cisco/connection/lib/install/post.d/sync-cucli-credentials cuc01Going to Sync cucli hash from shadow the node at cuc01...Cannot find platformConfig.xml in /usr/local/platform/conf /tmp /commonFATAL ERROR: Cannot initialize internal variable: Cannot initialize the icluster internal dataCannot initialize the iCluster Library++ error++ '[' 0 -eq 1 ']'++ echo 'Cluster renegotiation failed.'

Solution

Complete these steps, in order to resolve this issue:

1. Ensure that all the required services are functional with the use of this command utils service list.
2. Enter this command utils os secure permissiveand set both the servers in permissive mode.
3. Run the Renegotiate command on the Subscriber. Publisher reboots after renegotiation is completed.
4. Enter this command utils os secure enforce and set both the servers in enforce mode.

Problem 9

This error is displayed when the renegotiation fails.

Log Analysis

+ sudo -u cucluster ssh cuc01 'sh -lc '\''source /usr/local/cm/db/informix/local/ids.env && ontape' -s -L 0 -F -t 'STDIO'\'''
+ target_exec 't=$(mktemp); ontape -r -v -t STDIO > $t 2>&1; rc=$?; cat $t; exit $rc'
+ sudo -u cucluster ssh cuc01 ' sh -lc '\''source /usr/local/cm/db/informix/local/ids.env && t=$(mktemp); ontape -r -v -t STDIO > $t 2>&1; rc=$?; cat $t; exit $rc'\'''
Server is in an incompatible state or user authentication failed.
Physical restore failed - function read archive backup failed code -1 errno 0

Log Analysis - Before the Error

++ hostname
++ get_primary_hostname
++ is_primary
+++ get_platform_config_value CcmFirstNode
+++ xml sel -t -v /PlatformData/CcmFirstNode/ParamValue
/usr/local/platform/conf/platformConfig.xml
++ test yes == yes
++ hostname
+ /opt/cisco/connection/bin/copy-informix-instance -s ccm@cuc01
-t ccm@cuc01
source=ccm@cuc01
target=ccm@cuc01

Solution

The log analysis shows that the hostname remains the same for source and target. The issue occurs when the renegotiation command is run on the Publisher. This command should be run on the Subscriber server only.

If you run the command on the Subscriber sever and you receive the same error, the issue could be caused by the extra DB chunks added on the subscriber server. Contact TAC in order to work around this issue.