Guest

Cisco Unified Communications Manager (CallManager)

Database Replication Error in Cisco Unified Communication Manager

Cisco - Database Replication Error in Cisco Unified Communication Manager

Document ID: 100781

Updated: Aug 17, 2010

   Print

Introduction

This document describes how to resolve some of the issues that occur in the database replication status through the Cisco Unified Reporting Tool in Cisco Unified Communication Manager 6.1.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

  • Cisco Unified CallManager 5.1(3)

  • Cisco Unified Communications Manager 6.1

  • Cisco Unified Communications Manager 7.x

  • Cisco Unified Communications Manager 8.x

Components Used

The information in this document is based on these software and hardware versions:

  • Cisco Unified CallManager 5.1(3)

  • Cisco Unified Communications Manager 6.1

  • Cisco Unified Communications Manager 7.x

  • Cisco Unified Communications Manager 8.x

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Use Unifed Reports to Debug Replication

If you use Cisco Unified Communications Manager 5.1.3 or 6.1 or later, you must have access to the Cisco Unified Reports in order to debug the replication issues. These reports make it much easier to debug replication problems since you can log into one GUI (on the publisher) and see the status of all the servers in the cluster. They also allow you to look at many files that previously were accessed only with a remote account.

In order to access the Unified Reports, log onto the CCMAdmin GUI, and choose Cisco Unified Reporting from the drop down menu list on the right.

cucm_replication_error-01.gif

Choose system reports, and you see a list of reports on the left.

cucm_replication_error-02.gif

In the case of each report, use the icon on the far right to generate a new report. Old reports get stored. If you do not generate a new one, you can be looking at older data. Once the new report is generated, you can view the report.

This list shows different reports and the data that you find on each, which is important for replication setup. You can see that the data is from ALL the servers. In some cases, such as sqlhosts, it even compares the data from each server and tells you the entries that are different.

  • Unified Call Manager Database Status Report:

    Unifed CM database access 
       (whether publisher can reach subscriber DB, 
       and subscriber can reach publisher DB) 
    Unified CM Database Status 
       – gives RTMT replication counters for all servers in cluster. 
       Also has a “replication server list” that does a cdr list serv 
       on each server and displays the results. 
    Unified CM Hosts – shows the /etc/hosts file for all servers 
    Unified CM Rhosts - .rhosts files 
    Unified CM Sqlhosts – sqlhosts files 
    Unified CM Services – shows /etc/services files
  • Unified CM Cluster Overview Report:

    Unified provisioned servers 
       (shows which servers are in cluster) 
    Unified CM version – shows call manager version on publisher 
       and subscriber (be sure they match) 
    Unified CM Name Resolution 
       (check if all server names can be resolved) 
    Unified Server Connectivity 
       (shows if servers can ping each other) 
    Unified CM Time and Delay (check if publisher 
       and subscriber clocks are close enough)
    
  • Replication Server List (cdr list serv):

    Unified CM cdr list repl (shows all the replicates defined.
    This is most useful to check if all tables are set up to replicate, 
       and what options, such as ats, timestamp, etc are used.) 
    

Example of Report – in Database Status Report (sqlhosts File Data)

cucm_replication_error-03.gif

How to Determine if the Replication is Broken

Check the replication state and replication status in order to determine if the replication is broken.

Use Cisco Unified Reports: In Cisco Unified Reporting, you can generate the Unified CM Database Status report, where you can find the database replication status. Status 2 indicates that the replication is good. If you receive a status number of 3 or 4, it indicates either a broken database or that replication is not setup correctly between the publisher and subscribers. cucm_replication_error-04.gif

Replicate_State counter

Use the Syslog viewer in the Cisco Unified Communications Manager Real-Time Monitoring Tool (RTMT) in order to check the Replicate_State counter for the Number of Replicates Created and State of Replication object on all nodes. The value on each node must equal 2. This counter represents the state of replication, which includes these possible values:

  • 0 (Not Started)—No subscribers exist or the Database Layer Monitor service is not running and has not been running since the subscriber was installed.

  • 1 (Started)—Replication is currently being setup.

  • 2 (Finished)—Replication setup is completed and working.

  • 3 (Broken)—Replication failed during setup and is not working.

  • 4—Replication is not setup correctly.

Note: This information is the same for Cisco CallManager version 5.1(3) and all 6.1 versions.

Error: sqlhosts file on subscribers does not match the publisher

In a Cisco CallManager cluster that has a few subscribers and a publisher, when you check the replication status with the RTMT, you receive a status number of 3 or 4, which indicates either a broken database or that replication is not setup correctly between the publisher and the subscribers. However, the phone created in one Cisco CallManager appears in the other Cisco CallManager servers. The Cisco Unified Reporting Tool returns this error: sqlhosts file on subscribers does not match the publisher.

Note: The sqlhosts file on the subscriber only has the subscriber and publisher. The publisher sqlhosts file has all the entries for Cisco Unified CallManager 5.x . The sqlhosts file on each server must be the same in the case of Cisco Unified Communications Manager 6.x .

The Cisco Unified Reporting Tool is available in Cisco Unified CallManager 5.1(3) and Cisco Unified Communications Manager 6.1. You can access Cisco Unified Reporting from the Navigation drop down menu available in the right hand side of the Cisco Unified CallManager Administration Web page. A good replication status looks like this:

cucm_replication_error-05.gif

Solution

After you add a new subscriber to the cluster, the replication agreements and the actual replication are initiated only after the first reboot of each subscriber. So make sure you reboot the subscribers after you add them to the cluster. If you add many subscribers, reboot them one at a time in order to avoid overloading the publisher with the database replication.

Note: The sqlhosts file is present on each server and contains a reference for each Cisco Unified Communication Manager node in the cluster. If those sqlhosts files are out of sync, the SQL replication fails. Use the show tech dbstateinfo CLI command in each subscriber in order to check the local sqlhosts at the bottom of the output for any mismatch on each node.

Procedure

If the replication status is shown as 4, which indicates that the replication is not setup correctly, complete these steps:

  1. Double check the current status of the publisher in order to ensure it can establish the agreement. For this you can export and provide the database report from http://<pub>/cucreports > System Reports > Unified CM Database Replication Debug or download the report from http://<pub>/cucreports > System Reports > Unified CM Database Status > Download report .

  2. Re-initiate the agreement for each subscriber, then wait and verify the status. When the nodes start to communicate, you can reset the replication per node. In order to reset the replication in each subscriber, complete these steps:

    1. Execute the utils dbreplication stop command on all subscribers.

      Note: Ensure that you had finished Step 2a on all subscribers and then only run the command mentioned in Step 2b on the publisher.

    2. Execute the utils dbreplication stop command on the publisher.

    3. Execute the utils dbreplication clusterreset command on the publisher.

      Note: This command resets database replication on an entire cluster. If there is replication issue with only one subscriber there is no need to execute the command. You may skip Step 2d and perform Step 2e.

    4. If all subscriber servers need to be reset, issue the utils dbreplication reset all command.

    5. At the publisher server, issue the utils dbreplication reset <hostname> CLI command where <hostname> is the hostname of the subscriber server that needs to be reset.

    Now verify that the database replication status is changed to 2, which means the replication setup is complete and working.

Problem: Report Shows Version Mismatch

When running Cluster Overview Unified Report on Cisco Unified Communications Manager 6.1.2.1001-4, you see a number of version mismatches in the unified CM component version section.

Solution

This issue is identified as Mismatch in the Component Version in Unified CM Cluster Report.

This is a cosmetic defect related to how the versions are retrieved by the unified reporting system. The warning can be ignored, as it does not mean that there is actually any mismatch.

Cisco Unified Communications Manager Component Version gathers data from the Cisco Unified Communications Manager database in order to provide a summary of all versions of the component installed on each server. The summary identifies components on servers that do not match the publisher. It also identifies servers on which a component is not installed.

This issue is fixed in Cisco Unified Communications Manager version 6.1.3 and is documented in Cisco bug ID CSCsr96714 (registered customers only) .

Known Issues

Related Information

Updated: Aug 17, 2010
Document ID: 100781