IPCC Express Troubleshooting Tips for Upgrade, Backup, and Restore Issues

Introduction

This document describes how to troubleshoot CRS upgrade, backup, and restore issues.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Cisco Unified Contact Center Express

• Cisco IP Telephony Backup and Restore System (BARS)

Components Used

The information in this document is based on Cisco Unified Contact Center Express versions 3.x, 4.x,6.x, and 7.x.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

CRS 3.x and 4.x: Common Errors Received During Backup, Restore, and Upgrade

TCP Socket closed unexpectedly

When a Backup/Restore/Upgrade (B/R/U) fails, you might receive a message (displayed in red text) on the BARS Screen that states TCP Socket closed unexpectedly. Restore/Backup/Upgrade failed.

This message is generic and displayed in case of any failure in the backup/restore/upgrade operation. It is not an indication of the TCP connection breaking off or any network connection issues between the CRS and the BARS machines.

Applet Communication Error

Problem

CRS backup/restore/patch/upgrade fails in BARS due to a timeout waiting for applet communication (CRS Java applet fails to be loaded to the browser in which BARS admin runs within 5 minutes). The BARS admin shows that it extracted the archive files in the status window, and it appears to hang for about 5 minutes before it reports the failure. The MCVD / MARC log file shows the failure reason as "timed out initializing applet's communication." This issue is documented in Cisco Bug ID CSCef91551 (registered customers only) .

This issue can occur if the browser that is used to run BARS admin does not include the required settings.

• The Java plug-in is not installed yet or it does not have the correct version of JRE or the Java plug-in installed.

  1. In the Internet Options dialog box of Internet Explorer, click the Advanced tab, and scroll down to the Java (Sun) heading.

  2. Verify that the Use Java 2 v.14.2_xx for <applet> check box is checked.

• The default security setting was modified.

  1. In the Internet Options dialog box, click the Security tab.

  2. For the Local intranet zone, click Default Level and make sure the security level is set to the default level (Medium-Low) or below.

  3. If you customized your security settings, click Custom Level, and make sure that Java permissions is not set to Disable Java. Choose one of the three safety levels instead.

  4. In the custom level dialog box, make sure Scripting of Java Applets is set to Enable or Prompt.

• The default privacy setting was modified.

  1. In the Internet Options dialog box, click the Privacy tab.

  2. Make sure the Privacy setting is set to the default level (Medium) or below.

• The proxy server that is configured in the browser is not reachable.

  1. In the Internet Options dialog box, click the Connections tab, and then click LAN settings.

  2. If a proxy server is configured, make sure it is reachable or uncheck this option for using a proxy server.

• The security warning is enabled.

  1. In the Internet Options dialog bx, click the Advanced tab, and scroll down to the Security heading.

  2. Make sure the Warn if changing between secure and not secure mode check box is unchecked.

Solution

• Check if the NIC binding on the CRS box is proper and it is NIC 1 followed by NIC 2.

• Make sure that the CRS box is reachable from the BARS server.

• Make sure that the Pop Up Blocker is turned off.

• Make sure that the guidelines mentioned in the previous section is followed.

• When prompted by the browser to download and run the Java plug-in installer, respond with yes in timely manner. The restore might still fail if the installation takes longer than 5 minutes or if the installation requires a restart of the browser. In such cases, simply restart the browser, and rerun the restore again with the same archive. Also, respond timely to any of the Internet Explorer browser pop-up dialog boxes, since CRS times out if the applet does not get loaded in the browser in 5 minutes. If it did already time out, simply restart the restore again.

If the problem persists, make sure that the settings are correct, and then complete these steps:

1. In Internet Explorer, go to Tools > Sun Java Console in order to display the Java Console.

   Note: If the version of Internet Explorer that you use does not display this in the menu bar, locate the Java logo in the Windows Taskbar, right-click the logo, and choose Open Console.

2. Once the Java Console opens, press the 5 key in order to enable debugging.

3. Use BARS from this Internet Explorer browser in order to run the restore again.

4. If the restore fails again, return to the Java console window, copy all the text, and paste it into a text file in order to save it for troubleshooting purpose.

LDAPProviderUnavailable Exception

If the backup fails with the error message, complete these steps:

1. Check the logs for the following values: LDAP_CON_WARNING and LDAP_CON_ERROR. If both values exist, the backup/restore/upgrade process failed because the LDAP does not accept connections from the Cisco CRS.

2. Make sure that the LDAP servers (CallManager(s)) are reachable from the Cisco CRS Box. Bring up the LDAP server if it is not running.

3. Restart the CRS server.

Note: This issue is documented in Cisco Bug ID CSCse15624 (registered customers only) .

Error: GET_FROM_ARCHIVE_REQUEST failed with error:-2147417842

Problem

CRS backup\restore fails when BARS server attempts to backup BARS target. The BARS trace file (located in the C:\Program Files\Cisco\Trace\BARS folder on the BARS server) displays this error:

Inside function modGetFromArchive
Connecting to \\10.10.10.38\C$
modGetFromArchive =-2147417842
GET_FROM_ARCHIVE_REQUEST failed with error: -2147417842

BARS log displays:

Staging Cisco Customer Response Solutions target Ipcc
Opening session for backup on Ipcc
Opened session successfully on Ipcc
Backup is 1% complete.
Copying /STI/Backup/CRS/clusters.properties to
     C:\DOCUME~1\CRSADM~1\LOCALS~1\Temp\_8EF792BE_4448_46CF_9403_1006E8579197_20366\GetProperties23293.properties on 10.10.10.38
[Error]	Error: unable to load clusters.properties; nested exception is: 
	com.cisco.archive.ArchiveSystemIOException: UNSPECIFIED_ERROR; Failed to retrieve /STI/Backup/CRS/clusters.properties
Session closed successfully
[Error]	Could not backup Cisco Customer Response Solutions successfully on Ipcc.

Solution

Complete these steps in order to shut down BARS on the BARS server:

1. Close all instances of Internet Explorer.

2. On the BARS server, go to Start > Programs > Administrative Tools > Component Services.

3. Expand Component Services > Computers > My Computer > COM+ Applications.

4. In the right pane, right-click BARS, and choose Shut down.

5. Restart the Internet Information Server (IIS) Admin Service from the Service Control Panel.

6. Run again the restore/backup that failed.

Specific Issues Found During Backup/Restore/Upgrade

Issue 1

If you have reached the RESTORE process, find out which step and exact percentage of RESTORE process at which upgrade process failed. There are 2 stages for Restore process: stage 1 and stage 2.

• Stage 1 is from 0 - 19% for Restore and 0-33% for Patching. During Stage1, till the BARS suspends, all the information is logged in to the CiscoMARC.log. If the upgrade process fails during this time, look in CiscoMARC.log. It is during stage 1 only the cluster level information is updated (CCNApps > clusters > profilename > clusterdependent ou). The node level information (CCNApps > clusters > profilename > Nodes > nodeid > clusterdependent ou) is updated during stage 2. When BARS suspends, it gives a list of CRS servers which need to be rebooted. Follow the process thereafter.

• Stage 2 starts after 19%, when the Cisco CRS server reboots giving acknowledgment to BARS to resume. All the information is logged in MCVD.log. Look for _FAILED in the MCVD.log in case of failures. In CRS 4.x/6.x, we use the CRS with BARS for doing a Backup/Restore/Upgrade from the previous versions like CRS 3.x/4.x.

Issue 2

Towards the end of RESTORE, BARS suspends and then waits for CRS to come up. Once it suspends, it closes the socket. BARS waits for the signal to come from the CRS server once CRS 4.x is installed. It is normal to see this message in the barbi.log:

596: Fri Aug 10 21:17:02.141 - TCPSocket::readFully err=10054
597: Fri Aug 10 21:17:02.141 - MessageReader can not read Message Header
598: Fri Aug 10 21:17:02.141 - .. release class com_cisco_archive_impl_barbi::
     AbstractSession *, refCnt: 11
599: Fri Aug 10 21:17:02.141 - .. release class com_cisco_archive_impl_barbi::
     InputStream *, refCnt: 1
600: Fri Aug 10 21:17:02.141 - .. release class com_cisco_archive_impl_barbi::
     BlockingPriorityQueue *, refCnt: 2
601: Fri Aug 10 21:17:02.141 - MessageReaderThread id=2264 completed, closed=0
602: Fri Aug 10 21:17:02.141 - .. release class com_cisco_archive_impl_barbi::
     Thread *, refCnt: 1
603: Fri Aug 10 21:17:02.141 - getMessage: null
604: Fri Aug 10 21:17:02.141 - getMessage from protocol layer returns null
605: Fri Aug 10 21:17:14.125 - TCPSocket::writeFully err=10054
606: Fri Aug 10 21:17:14.125 - HeartbeatDispatherThread returns SESSION_SOCKET_ERROR
607: Fri Aug 10 21:17:14.125 - .. release class com_cisco_archive_impl_barbi::
     AbstractSession *, refCnt: 10
608: Fri Aug 10 21:17:14.125 - .. release class com_cisco_archive_impl_barbi::
     OutputStream *, refCnt: 1
609: Fri Aug 10 21:17:14.125 - .. release class com_cisco_archive_impl_barbi::
     BlockingPriorityQueue *, refCnt: 1
610: Fri Aug 10 21:17:14.125 - HeartbeatDispatherThread id=3744 completed, closed=0
611: Fri Aug 10 21:17:14.125 - .. release class com_cisco_archive_impl_barbi::
     Thread *, refCnt

Issue 3

For Cisco CRS 4.0(4) upgrades, you must click No, I will restart my computer later radio button in step 27 of the procedure Upgrading the Cisco CRS Software in the Maintenance Complete window in order to delete the 3.x version from the registry key. If you click Yes, I want to restart, the upgrade process fails with errors, such as older version of 3.x still exists at step 28 between bullets e and f. The information above is applicable for 4.0.5 single server (co-resident) upgrades in step 31 of the procedure Upgrading the Cisco CRS Software.

Issue 4

When you upgrade from Cisco CRS 3.5 to Cisco CRS 4.0(5)/4.1(1)/6.0(1), the process fails in the Spanlink restore phase if the team names configured in the Cisco Desktop Administrator contain a slash. This issue is documented in Cisco Bug ID CSCsj23469 (registered customers only) .

Solution:

Team names configured in the Cisco Desktop Administrator cannot contain a slash. If a slash exists in any team name, complete these steps before you begin the upgrade.

1. Open Cisco Desktop Administrator, and delete the team name(s) that contain a slash.

2. Create an alternate team name without a slash and configure the same mapping for the new team name.

   Note: Failure to recreate team names without slashes might result in failure during the upgrade.

Issue 5

While troubleshooting patching issues, make sure that the path to the patch archive file in the CRS box does not contain spaces. This issue is documented in Cisco Bug ID CSCsa98554 (registered customers only) .

Issue 6

During upgrade from 3.x to 4.0.4, after successful Restore, Enterprise Data Subsystem and VOIP Monitoring subsystem are Out Of Service. Check the CDBRTool logs under C:\programfiles\Cisco\Desktop\logs on the CRS server. Look for the error CDBRAPI::RestoreAllLCCs RestoreLCCData failed. Here is the relevant log snippet:

20:59:18 09/29/2007 MAJOR     CDBRPhonebookContact_200::PutPhonebookContactToLdap: 
     AddPhonebookContactProfile failed.  Return <2>.
20:59:18 09/29/2007 MAJOR     CDBRAPI::RestorePhonebookContacts  
     PutPhonebookContactToLdap failed.
20:59:18 09/29/2007 MAJOR     CDBRAPI::RestoreLCCData  RestorePhonebookContacts failed.
20:59:18 09/29/2007 MAJOR     CDBRAPI::RestoreAllLCCs  RestoreLCCData failed.
20:59:34 09/29/2007 INFO    LC0059 LDAPConnectionMgr::EstablishConnection: Connected to 
     LDAP server on <172.24.1.13>.
20:59:35 09/29/2007 INFO      CDBRAPI::RestoreCompany RestoreCompany ended.

As a workaround, go back to the previous CRS version and remove the blank entry from the phone book in Cisco Desktop Administrator. Now, take the backup on old version of CRS, upgrade to 4.0, and then perform the restore operation.

This issue is documented by the Cisco Bug ID CSCse63244 (registered customers only) .

Note: If the return code is 19 instead of 2, make sure the employee phone book does not contain a comma or any character other than a numeric digit in the Phone Number field.

Issue 7

Problem

When you try to manually backup the UCCX 7.X application, this error is returned: * 1326 - Logon failure: unknown user name or bad password.

Solution

In order to resolve the issue, first check the MCVD logs (see Procedure for Analyzing Logs section to check the logs).

If the password used is incorrect, the UCCX uses the old credentials in order to access the share folder. Here are the workarounds for this issue:

• Keep the old credentials at the backup server site.

• If you change the user password on the backup server, update the password in the UCCX, and then reboot of UCCX server.

Otherwise, complete these steps:

1. Configure an account in your Windows backup server.

2. Create a new backup folder.

3. Assign the new user full control of the folder, and share the folder.

4. From the UCCX server backup location, set the path name to \\<backup server>\<shared folder>, the user name to <backup server>\<user id>, and the password

This issue is documented in Cisco Bug ID CSCth19279 (registered customers only) .

Logs Required for Backup/Restore/Upgrade from the BARS Server

• BARS Backup/Restore logs are stored at these locations:

  • C:\Program Files\Common Files\Cisco\Logs\BARS\Backup*.*

  • C:\Program Files\Common Files\Cisco\Logs\BARS\Restore*.*

• BARS Trace logs are stored at C:\Program Files\Cisco\Trace\BARS*.*

• BARS Barbi log is stored at C:\WINNT\system32\barbi.log

Procedure for Analyzing Logs

1. Look into the Backup (or Restore) logs located at C:\Program Files\Common Files\Cisco\Logs\BARS\Backup (or Restore) in the BARS server.

2. Based on the time stamp, look into the Trace logs. They are available at C:\Program Files\Cisco\Trace\BARS in the BARS server.

3. The Trace logs provide brief information about the exception. In order to view the details, go to the respective CRS server, and check the MCVD logs for that period. Search for backup_failed, restore_failed, upgrade_failed mnemonics in those logs for the respective operation (B/R/U) failure. If the failure occurred before BARS suspends at 19%, check the MARC logs.

4. Once you reach the mnemonic specified in the above step, you can view the exact description of the error. For example, you might see these messages:

   • Applet Communication Error

   • Data base Archive component exception

   • Spanlink Archive Component exception

   • CDBR tool failed

   These messages are informative and tell the error faced due to which B/R/U failed. Based on the component, additional logs are needed as follows (apart from the ones mentioned above):

   SL Archive component: c:\program files\cisco\desktop\log\CDBRTool.* DB Archive component:

Common Issues Faced During CRS 6.0 Backup and Restore Testing

Applet Timeout issue

Problem

The applet times out and the Restore process fails when the OK button is not clicked during the security alerts and privacy alerts. These security alerts are often displayed behind the child window on the parent BARS page window. From the Trace logs, you can locate this issue because there is a gap of exactly 5 minutes. For example:

[06:49:34 PM]   Get next message
[06:54:34 PM]   FailureResponse id=2 from Session# 19, pArchiveId={C0E85DB3-D35-
                1-40FF-AE8F-6482B9A90D3B}, errorCode=UNSPECIFIED_ERROR, statusM-
                essage=timed out initializing applet's communication

Possible Solutions

1. Manually drag the child window towards the corner of the screen, and reduce the window size, so that the centre is visible for any security alerts.

2. Keep the focus on the BARS main page, and minimize the child window. Keep track of any pop-up dialog boxes.

3. In Internet Options, reduce the security settings and privacy settings to Low before you begin the restore process. Revert back after the restore process. (This is not recommended as the implications of this action from the browser security perspective has not been verified).

Upgrade of CRS 3.5 to 6.0 in Standalone Setup

The CRS 3.5 to 6.0 upgrade must be followed as described in the Cisco Customer Response Solutions Installation Guide only. Taking a backup of CRS 3.5, re-imaging, and trying to restore it over the CRS 6.0 setup is not a valid scenario.

Since this is not a supported scenario, the only workaround is to revert back to CRS 3.5.

CRS 4.0(x) to 6.0 Upgrade

During a CRS 4.0 to 6.0 upgrade, if you have uploaded a different License Package (not the same package that was uploaded in CRS 4.0) after the upgrade, the License Package Type displays None in License Information Page in AppAdmin, and some of the AppAdmin menus will be missing.

For example, if the customer has CRS 4.1 with a standard license and upgrades to CRS 6.0 with a premium license, then after the upgrade to CRS 6.0 some menus are missing in AppAdmin. In AppAdmin > Control Center > License Information Page, the License Package Type displays None.

Solution: Change the CRS license filter value in LDAP to the new license type.

LDAP license Filter entry : CCNApps/clusters/<ProfileName>/ClsuterSpecific.xxxxx/License.xxxxx/FilterType

If the new license package is Standard , changes the FilterType to 3
If the new license package is Enhanced, changes the FilterType to 4
If the new license package is Premium, changes the FilterType to 5

After you perform the changes in LDAP, restart CRS Node Manager on the CRS Server.

Installation/Upgrade Process Left Unattended

The Installation, Upgrade, and Restore processes are very critical processes and must be followed very carefully as per the guide. At times, BARS can transition to the Not Responding state. Cisco recommends that you witness the entire process of Upgrade, Installation, and Restore.

Usage of Pre-Upgrade Tool

As described in the installation guide, you must run the Pre-Upgrade Tool (PUT) before you perform the Restore process. Its usage is to inject the CRS 6.0 license in LDAP, so that the backup archive contains the 6.0 licenses.

BARS Page Going Blank

BARS display page goes blank intermittently during the Restore process. This issue is documented by the Cisco Bug ID CSCsa82969 (registered customers only) . This is a cosmetic issue. In order to resolve this issue, refresh the child window (press F5). This should be done only on the BARS status window and not on the main BARS restore window.

Collection of BARS logs

Before you re-image the Cisco CallManager server, the BARS logs must be saved. Refer to Logs required for Backup/Restore/Upgrade for more information. The file details are mentioned in the Cisco IP Telephony Backup and Restore System (BARS) Administration Guide.

Backups fail with this error: * 86 - Unknown error occured while connecting to the host

Problem

Scheduled and manual backups are failing with error * 86 - Unknown error occured while connecting to host. Backup system accepts network path and account info, but the backup fails.

Solution

Complete these steps in order to resolve this issue:

1. Access the UCCX server and navigate to Start > Run, and type CET.

2. When the warning message appears, click NO.

3. Choose com.cisco.crs.cluster.config.ArchiveAdminConfig.

4. On the right side, double-click the record ID.

5. Click the com.cisco.crs.cluster.config.ArchiveAdminConfig tab, and clear the password under Backup Storage.

6. Click Apply.

7. Navigate to Appadmin > Tools > Backup and Restore.

8. Under Backup Storage Location, type the new password, and click Update.

After you complete these steps, you can run the backup. If the backup fails, restart the server, and try the backup again. If the backup still fails, you can navigate to CET, clear all the fields, and then type the new information for the storage location.

UCCX 7.x: BARS Backup Failure

Problem

BARS backup fails with this error message:

%MCVD-AC_SPANLINK-7-UNK:Exception thrown
while invoking and running BarsCLI:
Exception=com.cisco.archive.ArchiveException: 
BarsCLI failed to backup Spanlink config

This issue is documented in Cisco Bug ID CSCsy04635 (registered customers only) .

Solution

In order to resolve this issue, restart the Node manager.

UCCX 8.x: BARS Backup Fails at 87 %

Problem

Backup hangs at 87% with CCXCOMPONENT giving error at 30%.

Solution

In order to resolve this issue, run this command from the command-line interface:

utils service restart Cisco DRF Master

UCCX 7.x Restore From Backup Hangs at 15%

Problem

When you attempt to restore a backup of UCCX 7.x, it hangs at 15% and you receive this error message:

Since the backup was taken when HA and since this other node does not currently exist in the cluster it cannot continue.

Solution

Since the backup was taken in a high-availability environment both nodes must be in the cluster for you to restore the information. You can restore the backup files in a high-availability deployment using one of these options:

• If the high-availability setup is already in place and both nodes are added as part of the same cluster, the restore process is similar to single-node deployment; it can be done from any node and will restore data on both the nodes.

• If the high-availability setup is not in place and both the nodes are installed fresh or reimaged prior to installing Unified CCX, complete these steps in order to restore:

  1. Initiate the restore process from first node. Restore will complete 15% and will prompt you to add the second node to cluster.

  2. Add the second node through Setup wizard. Once you add the second node, restore will be complete and the high-availability setup will be ready.

Restore Fails at 69%

Problem

When you upgrade UCCX 4.5 server to 7.0, restore of UCCX 4.5 data fails with this error:

Exception occured while contacting the Call Manager com.cisco.archive.ArchiveException:
Unable to process restore request; nested exception is:
com.cisco.archive.ArchiveException: Exception thrown while downloading Recordings to the
Recording Folder:C:\Program Files\Cisco\Desktop_Audio

Exception=com.cisco.archive.impl.ArchiveFailureException: Unable to contact Call Manager.
Please make sure that the Call Manager is running and connected to the network
com.cisco.wf.spanlinkBackupRestore.SLRcrdgArchiveComponent; nested exception is:
com.cisco.archive.ArchiveException: Unable to process restore request; nested exception
is:com.cisco.archive.ArchiveException: Exception thrown while downloading Recordings to the
Recording Folder:C:\Program Files\Cisco\Desktop_Audio

Solution

This issue is documented in Cisco Bug ID CSCsr56145 (registered customers only) . The workaround is to patch the 7.0(1) system with the latest Service Release (SR) and run the restore again.

Related Information

• UCCX data restore fails with the error message "Fail to update CRS Config Datastore primary state in node 1"
• Technical Support & Documentation - Cisco Systems