Troubleshoot UCCX Backup Failing with Error Message

Introduction

This document describes how to troubleshoot Unified Contact Center Express (UCCX) failing to backup and displaying an error message.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Unified Contact Center Express
• CLI access via SSH Client
• Understanding of NTP

Components Used

The information in this document is based on this software version:

• UCCX 12.5.1

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

UCCX has stopped backing up via the scheduled task. When you try to perform a manual backup job, the backup fails with the error message "ERROR: UCCX DBReplication is not set up properly, Backup completed unsuccessfully."

UCCX Backup Failed

Troubleshooting

First, you need to confirm the health of the servers by requesting the outputs of the CLI commands from all nodes Pub and Sub:

• show status
• utils diagnose test
• show network cluster
• utils ntp status
• utils dbreplication status
• utils dbreplication runtimestate
• utils uccx dbreplication status
• utils uccx database healthcheck
• utils uccx healthcheck all

Out of the output of those CLI commands, the majority of the issues are related to NTP or DBReplication.

For optimal performance the recommended stratum level to which the PUB must be syncing with the NTP server is 4 and below. A stratum higher than 5 can cause sync issues that would in turn affect DBReplication. You can find more information on NTP Troubleshooting in the article Troubleshoot NTP on Unified Communications Manager. The article linked is for Unified Communications Manager but applies to all VOS offerings.

Here are the DBReplication Definitions to understand what is the value that you would want to see for the DBReplication status.

DBReplication Status Definitions

You can learn more on DBReplication Troubleshooting in the article Troubleshoot CUCM Database Replication Issues. The article linked is for Unified Communications Manager but applies to UCCX as well.

Note: In the article linked above, some commands that are specific to UCCX are not mentioned. As the article is more specific to CUCM. However, the overall process applies to UCCX as well. For example, utils uccx dbreplication repair all.

In this scenario, the health of the UCCX server was good.

NTP Sync Status

DBReplication Status

Log Collection

After having cleared the health and other components that could cause the issue, you can proceed to gather some additional data in the logs that were generated during the failed backup attempt. From the same screen where it showed the error message, you download the CCX Component log file that was generated and it is in-line with the failed backup.

CCXComponent Log

Log Analysis

From the logs.

Log File:
=====================================================
Server :
Feature : UCCX
Component : CCXCOMPONENT
Time Completed: 2021-10-01-12-03-21
Result Code : 24-Ontape failed to backup UCCX DB
Result String : ERROR
=====================================================

It is able to back up the components, begins the back up.

2021-10-01 12:03:15 :INFO: ------------- Begin Unified CCX DB DoBackup(db_do_backup.py) -------------

Then, checks for memory available.

2021-10-01 12:03:15 :INFO: Free space available for Unified CCX DB backup = 85622509568 bytes
2021-10-01 12:03:15 :INFO: Enough disk space found for Unified CCX DB backup

After that, it detects that it is a High Availability (HA) setup.

2021-10-01 12:03:15 :INFO: This is UCCX High Availability setup
2021-10-01 12:03:15 :INFO: Two Nodes are there in Unified CCX Cluster

Backup is initiated.

2021-10-01 12:03:15 :INFO: HA - Publisher Node. Backup initiated

2021-10-01 12:03:15 :INFO: Creating Ontape Backup of Unified CCX DB...
2021-10-01 12:03:15 :INFO: Executing [source /opt/cisco/uccx/bin/uccx_ids.env ; ontape -v -s -L 0 -t STDIO > /common/drf/ccx_comps/uccx_db/ontape_uccx_backup.gz] ...

However, it fails at this stage.

2021-10-01 12:03:19 :INFO: Results of executing [source /opt/cisco/uccx/bin/uccx_ids.env ; ontape -v -s -L 0 -t STDIO > /common/drf/ccx_comps/uccx_db/ontape_uccx_backup.gz], returnCode :[256], Output :[Using the backup and restore filter /bin/gzip -c.
Archive failed - ISAM error: An error has occurred during archive back up.

Program over.]
2021-10-01 12:03:19 :INFO: Ontape Backup failed.
2021-10-01 12:03:19 :INFO: ----- Exit Unified CCX DB DoBackup(db_do_backup.py) with returnCode = 24 : Unsucccessful -----
2021/10/01 12:03:19: DO_Backup DB Script failed to work

Corrective Action

Restart these services in CLI:

• DRF Master and DRF Local services
  • utils service restart Cisco DRF Master (On PUB)
  • utils service restart Cisco DRF Local (On Both Nodes)
• Database service
  • utils service restart Cisco Unified CCX Database (On Both Nodes)

Note: Restarting these services does not have an impact to the primary functionality of the servers. However, you can run the commands during a maintenance window.

After this, run a manual backup and verify success. A successful message appears.

Backup Completed

After a manual successful backup, the scheduled task also begins to work.

Related Information

• Troubleshoot NTP on Unified Communications Manager
• Troubleshoot CUCM Database Replication Issues
• Cisco Technical Support & Downloads