Cisco Unified Contact Center Express Operations Guide, Release 10.0(1)
Serviceability
Downloads: This chapterpdf (PDF - 1.38MB) The complete bookPDF (PDF - 3.29MB) | The complete bookePub (ePub - 382.0KB) | Feedback

Serviceability

Serviceability

Cisco Unified CCX Serviceability, a web-based troubleshooting tool for Cisco Unified Contact Center Express (Unified CCX), provides the following functionality:

  • Configure alarms for local and remote Syslogs.
  • Configure trace settings for Unified CCX components. After these settings are enabled, you can collect and view trace information using the Real-Time Monitoring Tool (RTMT).
  • Configure and manage log profiles for different Unified CCX components.
  • Manage and control network services.
  • View replication status, synchronize data, and reset replication for Unified CCX servers in a cluster through Datastore Control Center.
  • Set parameters for different platform services.
  • Set Java Virtual Machine (JVM) parameters for different Unified CCX services to collect thread and memory traces.

Access Cisco Unified CCX Serviceability

Log in to Cisco Unified CCX Serviceability either with the end user credentials configured during the initial setup of CCX Administration interface, or with the application user credentials configured during install.

To access Cisco Unified CCX Serviceability, log in to Cisco Unified CCX Serviceability page using the following URL format: https://<server name or IP address>/uccxservice/.

Alarms

You can view alarm information by using the SysLog Viewer in Cisco Unified Real-Time Monitoring Tool (RTMT). See "Real-Time Monitoring Tool" section for detailed information on how to view alarm information.

Alarm Configuration

Use the Alarm Configuration web page in Unified CCX Serviceability to view and configure alarm server settings for different Unified CCX components.


Note


To find more information on the Alarm messages in the system, use Alarm Definition page in Cisco Unified Serviceability.


Alarm Configuration Settings

Use the Alarm Configuration page to modify alarm settings.

In the case of a High Availability deployment, the alarm configuration changes are automatically propagated to the second node. If the second node cannot be contacted, an alert message indicating that the update has failed on the remote node is displayed.

Following table defines the options available on this page:
Table 1 Alarm Configuration Settings

Setting

Description

Enable Alarm for Local Syslogs

Enables the alarms to be stored as syslog messages locally. This can be viewed in the Application logs within Syslog viewer from the RTMT tool.

For information about viewing logs with the SysLog Viewer, see "Real-Time Monitoring Tool" topic.

Enable Alarm for Remote Syslogs

Enable the alarm messages to be sent to the configured Syslog server.

Server Name field - Provide the IP / hostname of the Syslog server to which the system should send the alarm messages.

Alarm Event Level

Alarm event level messages range from severity 0 (most severe) to severity 7 (least severe). See the description below. When you choose a severity level, all messages of that severity level and higher are sent.

For example, if you choose ERROR_ALARM (Severity 3), all messages of severity 3, severity 2, severity 1, and severity 0 are sent. The default is INFORMATIONAL_ALARM (Severity 6), which will send messages on all severity levels starting from 6 to severity level 0.

You can choose one of the following alarm event level options from the drop-down list box:
Emergency
Systemic failures causing the whole Contact Center to be down. For example, "CCX engine crashed or went down abruptly".
Alert
Multiple components failures on the system. For example, "Telephony and RMCM subsystem out of service due to CTI provider failure".
Critical
Failures in the Major component of the system. For example "Web chat subsystem out of service".
Error
Functionality or certain scenario not working as expected. For example, "Create dialog group failed".
Warning
Some limits or threshold about to be breached. For example, "Historical reporting internal queue near capacity".
Notice
Trigger of major operation notification. For example, "Engine Shutdown initiated by Administrator"
Informational
Information about various minor event occurrences in the system. For example, "Backup Operation completed".
Debug
Detailed traces which help in debugging issues. For example, detailed information on some CCX events.

Traces

A trace file is a log file that records activity from the Cisco Unified Contact Center Express (Unified CCX) components. Trace files let you obtain specific, detailed information about the system that can help you troubleshoot problems.

The Unified CCX system also generates information about all threads that are running in the system. This information is stored in the thread dump file and is useful for troubleshooting.

Component Trace Files

The component trace file contains information about each component. You can create a trace file for any of the following Unified CCX components:

  • Cisco Unified CCX Administration
  • Cisco Unified CCX Cluster View Daemon
  • Cisco Unified CCX Editor
  • Cisco Unified CCX Engine
  • Cisco Unified CM Telephony Client
  • Cisco Unified CCX Desktop Services
  • Cisco Unified CCX Recording and Monitoring Services
  • Cisco Unified Intelligence Center Services

After configuring the information that you want to include in trace files for various services, you can collect and view them by using the trace and log central option in the Cisco Unified Real-Time Monitoring Tool. See the "Real-Time Monitoring Tool" section for detailed information.

Configure Trace Parameters

To update trace file information and to activate and deactivate logging, follow this procedure:

Procedure
    Step 1   From the Cisco Unified CCX Serviceability menu, choose Trace > Configuration.
    Step 2   From the Select Service drop-down list box, choose a service or component for which you want to configure trace. Then click Go.

    The debug levels for different Unified CCX subfacilities or services that are displayed may vary depending on the selected service.

    Step 3   Update the debug level for one or more of the libraries or subfacilities for the selected service using the check box provided and click Save.
    Step 4   To limit the number and size of the trace files, you can specify the trace output setting using the following two fields. See the following table for description and default values for these two fields.

    Field

    Description

    Maximum No. of Files

    The maximum number of trace files to be retained by the system.

    This field specifies the total number of trace files for a given service. Cisco Unified CCX Serviceability automatically appends a sequence number to the file name to indicate which file it is; for example, Cisco001MADM14.log. When the last file in the sequence is full, the trace data begins writing over the first file. The default value varies by service.

    Maximum File Size

    This field specifies the maximum size of the trace file in kilobytes or megabytes depending on the selected service. The default value varies by service.

    Caution   

    You should activate logging only for debugging and remember to deactivate logging once the debugging session is complete.

    Note   

    You will not be able to save the trace configuration if Cisco Unified Intelligence Center service on the publisher node is down.


    Trace Level Options

    A trace file that records all information for a component, such as the Unified CCX Engine, can become large and difficult to read. To help you manage the trace file, the Cisco Unified CCX system lets you specify the subfacilities for which you want to record information.

    For each component, you can select one or more Debugging trace levels. The selections in this page specify the level of details in the debugging messages that the system sends to a trace file. For instance, if you select Debugging, the system sends only the basic error messages, while if you select XDebugging5, the system will send errors, warnings, informational, debugging, verbose messages and so on in detail to the trace file.

    Trace File Location

    The Unified CCX server stores the trace files in the Log directory under the directory in which you installed the Unified CCX component. You can collect and view trace information using the Real-Time Monitoring Tool (RTMT).

    Trace File Information

    The trace files contain information in standard Syslog format. The file includes some or all of the following information for each event that it records:

    • Line number
    • Date and time the event occurred
    • Facility and subfacility (component) name
    • Severity level
    • Message name
    • Explanation
    • Parameters and values

    Log Profiles Management

    Log profile is an aggregated entity that preserves multiple trace settings of the following Unified CCX services:

    • Cisco Unified CCX Engine (Traces termed as MIVR)
    • Cisco Unified CCX Administration (Traces termed as MADM)
    • Cisco Unified CCX Cluster View Daemon (Traces termed as MCVD)
    Choose Trace > Profile from the Unified CCX Serviceability menu to access the Log Profiles Management page. You can perform the following operations using this page:
    • Create profile
    • Save as another profile
    • Enable profile
    • Delete profile
    • Save current trace settings
    • Upload profile
    • Update profile

    Log profiles in Unified CCX can be one of the following two types:

    1. System Log Profiles: These log profiles are preinstalled with Unified CCX, and you cannot modify these profiles. Following system log profiles are factory shipped with Unified CCX:
      • Default
      • Outbound
      • AppAdmin
      • Media
      • HRDM (Historical Reporting Data Manager)
      • StuckSession
      • Database
      • CallStuckInQueue
      • EDBS (Enterprise Database Subsystem)
      • Serviceability
      • RealTimeDataProblems
    2. Custom Log Profiles: If the trace settings generated by system profiles are not sufficient in a particular scenario, you can create custom log profiles for better troubleshooting. You can create and enable these custom log profiles as needed.

    Note


    • In a High Availability deployment of Unified CCX, all the log profile operations will be reflected on both the nodes in the cluster.
    • You cannot delete the profile if the selected log profile is the last-enabled profile in the system.

    Serviceability Tools

    Network Services

    Network services include services that the system requires to function and are activated by default.

    After you install your application, network services start automatically. The list of services that is displayed in the Control Center—Network Services web page depends on the license package of your Unified CCX. If you have a Unified CCX Premium license, Cisco Unified Contact Center Express (Unified CCX) Serviceability categorizes the network services into the following categories:

    • System services
    • Admin services
    • DB services
    • Desktop services
    • Finesse services

    Note


    • Unified CCX Engine Services information will be removed from UCCX Serviceability page when an invalid license is uploaded.
    • Only System and Admin services information will be visible in Unified CCX Node Services section.

    Manage Network Services

    Control Center in Cisco Unified CCX Serviceability lets you perform the following tasks:
    • Start, stop, and restart Unified CCX services
    • View and refresh the status of Unified CCX services

    Choose Tools > Control Center - Network Services from the Unified CCX Serviceability menu to perform the actions on services.


    Tip


    You may need to manage services in both Cisco Unified CCX Serviceability and Cisco Unified Serviceability to troubleshoot a problem. The Cisco Unified Serviceability services are described in the Cisco Unified Serviceability Administration Guide.



    Note


    You cannot start or stop Cisco Unified CCX Serviceability service from the Unified CCX Serviceability web interface and you need to use CLI. For a list of services that you can start and stop through the CLI and for information about how to perform these tasks, see "Command Line Interface Reference" section.


    Manage Datastores

    Datastores are components that allow you to manage and monitor historical, agent, repository, and configuration data across all servers in the Unified CCX cluster.

    Use the Datastore Control Center to perform the following functions:
    • Obtain an overview of the datastores in the cluster and their relationships.
    • Manage the data store read/write access.
    • Monitor and control the replication state (available only for Agent, Historical, and Repository datastores).

    Note


    Support for High Availability and remote servers is available only in multiple-server deployments.


    The Unified CCX cluster uses publisher/subscriber database model for data replication across the system. Under normal circumstances, the database master acts as the source of data and the other node acts as the target for the data. In other words, the database master is the publisher and the other node is the subscriber.


    Note


    In the Tools > Datastore Control Center > Datastores page, the first node that is installed in the cluster is marked as publisher (with an icon marked P). In Unified CCX, the term "publisher" is used to denote only the first node in the cluster and does not indicate that node is the source of the data. Typically, the database master node acts as the source and the other node acts as the destination.


    Synchronize Datastore

    Use this procedure to synchronize data for each datastore between the two nodes in case of a mismatch.

    Procedure
    Choose Tools > Datastore Control Center > Datastores and click Synchronize Data
    Note   

    Synchronize datastore does not update any mismatch in the configuration datastore.


    Manage Replication Between Nodes

    The Replication Servers menu option in Datastore Control Center allows you to view replication status and reset the replication between two nodes for the data stores across all servers in the cluster. This menu will be available only in a High Availability deployment.

    By default, replication between two nodes is removed if they are not able to synchronize with each other due to network outage for a substantial period of time. This time depends on the amount of requests sent from one node to the other and the load on servers. If the replication stops due to network outage, an alert is sent to the administrator so that the administrator can take corrective action.

    If the replication is removed, the administrator can go to Tools > Datastore Control Center > Replication Servers submenu from the Cisco Unified CCX Serviceability menu and click Reset Replication. This ensures that the replication is established between the nodes and the data synchronization (repair) process is initiated. Click Check Details icon to monitor status of the repair.

    If the network outage did not result in the replication setup being removed, after the network is functional, the synchronization of data between the databases will be automatic. For outages that last a few seconds, typically the administrator need not take any action, and the system will be able to synchronize automatically.


    Note


    Even though the replication between the nodes is removed, data can still be written to the database, which is accessible to the Unified CCX engine.


    When the subscriber node is not functional and it is required to make configuration updates from the publisher, you can disable Config Data store (CDS) and Historical Data store (HDS) on the subscriber using Disable CDS and HDS icon or button. After the subscriber node is functional, you can enable CDS and HDS on the subscriber using the same toggle button.

    Caution


    Any configuration in Application Administration and Historical data on the Subscriber node will be overwritten, when CDS is enabled again.


    Replication During Network Outage

    By default, replication between two nodes is removed if they are not able to synchronize with each other due to network outage for a substantial period of time. This time depends on the amount of requests sent from one node to the other and the load on servers. If the replication stops due to network outage, an alert is sent to the administrator so that the administrator can take corrective action.

    If the replication is removed, the administrator can go to Tools > Datastore Control Center > Replication Servers submenu from the Cisco Unified CCX Serviceability menu and click Reset Replication. This ensures that the replication is established between the nodes and the data synchronization (repair) process is initiated. Click Check Details icon to monitor status of the repair.

    If the network outage did not result in the replication setup being removed, after the network is functional, the synchronization of data between the databases will be automatic. For outages that last a few seconds, typically the administrator need not take any action, and the system will be able to synchronize automatically.


    Note


    Even though the replication between the nodes is removed, data can still be written to the database, which is accessible to the Unified CCX engine.


    Update Parameters

    Use the Service Parameters page to view and update different services in Unified CCX servers. Ensure the following prerequisites are met before configuring the parameters:
    • The servers are configured.
    • The service is available on the servers.

    Caution


    Some changes to service parameters may cause system failure. Cisco recommends that you do not make any changes to service parameters unless you fully understand the feature that you are changing or unless the Cisco Technical Assistance Center (TAC) specifies the changes.


    The following services are supported in Unified CCX:
    • Cisco AMC Service
    • Cisco DRF Local
    • Cisco DRF Master
    • Cisco Log Partition Monitoring Tool
    • Cisco RIS Data Collector
    • Cisco Serviceability Reporter
    • Cisco Trace Collection Service

    For more information, see Cisco Unified Serviceability Administration Guide available here:

    http:/​/​www.cisco.com/​en/​US/​partner/​products/​sw/​voicesw/​ps556/​prod_​maintenance_​guides_​list.html

    Configure Performance Monitoring of Unified CCX Servers

    Use the Performance Configuration and Logging page to configure Java Virtual Machine (JVM) parameters and dump Thread and Memory traces for performance monitoring of Unified CCX servers.

    Use the following procedure to configure JVM parameters for a particular service on a particular server.

    Procedure
      Step 1   Choose Tools > Performance Configuration and Logging and select the server and a service for which you want to get the JVM options.
      Step 2   Click Dump Thread Trace to dump the thread traces for the selected service in the selected server. You can collect the corresponding jvm.log from the log folder for that facility using Real-Time Monitoring Tool (RTMT).
      Step 3   Click Dump Memory Trace to dump the memory traces. This creates the following two logs in the log folder for that facility.
      1. Memory-<facility name>-<time stamp>.hprof (for heap dump)
      2. histo-<facility name> <time stamp>.log (for histogram)
      Step 4   You can change the JVM options by clicking Enable or Disable radio buttons in this page.

      Click Update JVM Options to update the new settings for selected service on selected node.


      Simple Network Management Protocol

      Simple Network Management Protocol (SNMP) is an industry-standard interface for exchanging management information between network devices. SNMP enables you to monitor and manage the Unified CCX system. You also can set up SNMP traps to automatically notify any high-severity messages and errors that are generated by the Cisco Unified CCX system.

      You can configure the SNMP settings using the Cisco Unified Serviceability web interface.

      SNMP Management Information Base (MIB)

      A Management Information Base (MIB) designates a collection of information that is organized hierarchically. MIBs are made up of managed objects, which are referenced by object identifiers. Managed objects are made up of one or more object instances, which are essentially variables. MIBs provide status monitoring, provisioning and notification.

      Table 2 SNMP MIBs

      MIB

      Agent Service

      CISCO-VOICE-APPS-MIB

      Cisco Unified CCX Voice Subagent

      CISCO-CDP-MIB

      Cisco CDP Agent

      CISCO-SYSLOG-MIB

      Cisco Syslog Agent

      SYSAPPL-MIB

      System Application Agent

      MIB-II

      MIB2 Agent

      HOST-RESOURCES-MIB

      Host Resources Agent


      Note


      • Unlike previous releases of Unified CCX, the SysAppl MIB implementation in Unified CCX 9.0(1) and later versions does not distinguish between Unified CCX activated services and deactivated services. It shows all the installed services in Unified CCX.
      • In Unified CCX 9.0(1) and later versions, the SysAppl MIB will not provide the Unified CCX subsystem information and their status information. You can view the subsystem and their status information through Cisco Unified CCX Serviceability web interface.
      • Syslog messages can also be sent as SNMP traps using the CISCO-SYSLOG-MIB. Refer to the section on CISCO-SYSLOG-MIB for details. They can be correlated to the failure of important features of Unified CCX.

      The following section describes CISCO-VOICE-APPS-MIB. For more information about other CCX supported MIBs, see Cisco Unified CM SNMP chapter in the Cisco Unified Serviceability Administration Guide available here:

      http:/​/​www.cisco.com/​en/​US/​partner/​products/​sw/​voicesw/​ps556/​prod_​maintenance_​guides_​list.html

      CISCO-VOICE-APPS-MIB

      The CISCO-VOICE-APPS-MIB provides information associated with the installed workflow applications provisioned on the Unified CCX Server. It also provides information on the supported SNMP Traps on Unified CCX. You can manage CISCO-VOICE-APPS-MIB through Unified CCX Serviceability web interface.

      Unified CCX Voice Subagent

      Cisco Unified CCX Voice Subagent service implements the CISCO-VOICE-APPS-MIB. Cisco Unified CCX Voice Subagent Service communicates with the SNMP Master Agent through Cisco Unified CCX SNMP Java Adaptor. The Cisco Unified CCX SNMP Java Adaptor service should be up and running for the Unified CCX Voice Subagent to work properly.

      For more information about the CISCO-VOICE-APPS-MIB, refer to this URL: ftp:/​/​ftp.cisco.com/​pub/​mibs/​v2/​CISCO-VOICE-APPS-MIB.my.


      Note


      • In Unified CCX, while exposing the Unified CCX workflow information through CISCO-VOICE-APPS-MIB, only one trigger per application row will be returned when doing a walk on the workflow table (cvaWorkflowInstallTable object). If there are multiple triggers associated with a Workflow application, these will be shown as separate entries (rows).

      SNMP Traps

      Subsystems which are the functional blocks for of Unified CCX send out alarms which are routed to the Syslog or as SNMP Traps. SNMP Traps are generated when any Unified CCX Subsystem/module or processes starts or stops, a runtime failure occurs for a module. These can be tracked for each major component to track the health of the Unified CCX system.

      The following Traps are supported as part of the CISCO-VOICE-APPS-MIB:

      Trap Name

      Description

      cvaModuleStart

      A cvaModuleStart notification signifies that an application module or subsystem has successfully started and transitioned into in-service state.

      cvaModuleStop

      A cvaModuleStop notification signifies that an application module or subsystem has stopped. If failure cause is known then it will be specified as part of the Trap message.

      cvaModuleRunTimeFailure

      cvaModuleRunTimeFailure notification signifies that a run time failure has occurred. If failure cause is known then it will be specified as part of the Trap message.

      cvaProcessStart

      A cvaProcessStart notification signifies that a process has just started.

      cvaProcessStop

      A cvaProcessStop notification signifies that a process has just stopped.

      The ModuleStart and ModuleStop traps are generated when the key Unified CCX services including Cisco Unified CCX Engine, Cisco Unified CCX Cluster View Daemon and Cisco Unified CCX Administration and their modules/subsystems are started and stopped respectively.

      The ProcessStart and ProcessStop traps are generated when the key Unified CCX services including Cisco Unified CCX Engine, Cisco Unified CCX Cluster View Daemon and Cisco Unified CCX Administration are started and stopped.

      You can configure the notification destinations by using the SNMP Notification Destination Configuration page in Cisco Unified Serviceability.


      Note


      SNMP Traps are not generated for events when the Unified CCX services and/or their subsystems goes Out of Service or comes In Service. Rather these events are sent as Remote Syslog messages and can be viewed through any third-party Syslog Viewers. You can refer to the list of CCX services and their subsystems/modules from the Cisco Unified CCX Serviceability under Tools - Control Center Network Services.



      Note


      • Unified CCX does not support SNMP trap V3 notifications.
      • CISCO-VOICE-APPS-MIB does not support INFORM notifications.

      For all notifications, the system sends traps immediately if the corresponding trap flags are enabled. Before you configure notification destination, verify that the required SNMP services are activated and running. Also, make sure that you configured the privileges for the community string/user correctly.

      More Info on SNMP

      For more information related to SNMP such as SNMP Version 1, Version 2C, Version 3, SNMP system group configuration, SNMP informs and SNMP trap parameters, see Cisco Unified Serviceability Administration Guide available here:

      http:/​/​www.cisco.com/​en/​US/​partner/​products/​sw/​voicesw/​ps556/​prod_​maintenance_​guides_​list.html