Before your network becomes operational, you can take several proactive steps to make troubleshooting easier, including:
•Produce network topology diagrams to help you isolate potential sources of problems.
•Synchronize the date and time on all servers.
•Set trace/logging levels on key devices so that diagnostic information is available when problems occur.
•Create IVR flowcharts that illustrate how calls are routed between agents and sites.
Network Topology Diagrams
One of the first lines of defense is possessing current topology information. One of the most important pieces of topology information is a detailed network diagram (usually created using Microsoft Visio or a similar application). At a minimum, your network topology diagrams should include the following information:
•The name assigned to each major device (typically the DNS name)
•IP addresses for all devices in the network
–Addresses for each router, core and access switch
–Addresses for all telephony and application servers, including the IP address for each server in a Cisco Unified Communications Manager cluster
–DHCP address range for addresses assigned to endpoints such as IP phones and agent workstations
•Phone extension number ranges assigned to sets of agents or users, as well as the main inbound dial-up numbers for each location. This information is useful in resolving dial plan configuration errors.
•WAN IP and PSTN links between sites.
This information is critical for isolating which components are involved in a particular problem. For medium- to large-sized networks, you may want to take a "layered" approach in your diagrams. Create a high-level diagram that illustrates the overall physical layout of your network, including all sites and the links between them. Then for each site create additional diagrams that show detailed addressing information, port numbers and dial plan configurations.
Tip Frequent adds, changes and upgrades to your network can quickly make these diagrams out-of-date. Inaccurate diagrams slow down the troubleshooting process and may lead to misdiagnosing the problem. Remember to keep these diagrams as current as possible.
Figure 1 shows a typical high-level topology diagram for a two sites in a contact center network. Agents accepting calls from customers are located in a central distribution site in Kansas, while the equipment supporting interactive voice is located in a data center in San Francisco.
Figure 1 Contact Center Network Topology Diagram Example
Synchronizing Server Date and Time
The best resources for diagnosing problems within your network are the debug and trace log files produced by individual Cisco devices. Tracing can be enabled on multiple devices and the log file output compared to isolate problems. In order to correlate messages for the same activity in different log files, you must compare the message timestamps and the source device MAC and IP addresses (there is no universal call ID value shared between Cisco devices). You should synchronize every device to the same date and time source so that the timestamps match. To accomplish this synchronization, set each device to obtain its date and time from the same Network Time Protocol (NTP) source.
For Cisco IOS-based devices (switches, routers or voice gateways), you can configure each device to act as a NTP client and periodically poll a master NTP source using the following command:
ntp server ip-address [version number] [key keyid] [source interface] [prefer]
Additional IOS commands are available to establish a device as a NTP peer (operating as the master source for other devices), as well as setting up NTP broadcasting instead of polling. See the Cisco IOS Configuration Fundamentals Command Reference for details on these IOS commands.
Recommended Trace/Logging Settings
In order to have diagnostic information available when you begin to research problems, you must configure devices in your network to capture signaling, processing and other activity in log files.
Cisco Unified Communications Manager Trace Settings
Trace settings for Cisco Unified Communications Manager servers are maintained using the Cisco Communications Manager Serviceability graphical interface. There are two ways to set trace logging levels for Unified Communications Manager services:
•Customize trace levels for individual parameters: This approach offers a high-degree of control and flexibility over the trace output. However, in order to use this approach you should understand not only the significance of each parameter, but also the impact of tracing on Unified Communications Manager server performance. For example, setting trace levels to "Error" has a minimal impact to CPU cycles while leaving the "Detail" level set for long periods of time may impact call processing. For instructions on setting individual trace levels, see the Cisco Unified Serviceability Administration Guide for Cisco Unified Communications Manager, "Configuring Trace" chapter .
•Apply predefined trace levels: This approach allows you to quickly enable and disable tracing for each Unified Communications Manager service based on predefined levels. You can also use these default troubleshooting trace settings in combination with customized settings to temporarily override the your custom settings. For instructions on using the Troubleshooting Trace Settings option in the Cisco Unified Communications Manager Serviceability interface, see the Cisco Unified Serviceability Administration Guide for Cisco Unified Communications Manager, "Configuring Troubleshooting Trace Setting Configuration" chapter .
Debug Trace Settings for CRS and IP IVR JTAPI Client
If you encounter any problems with CRS, activate the following debug trace settings to generate debug logs:
•For CRS issues: SS_TEL, SS_ICM, and LIB_ICM.
•For JTAPI Client issues: Enable all Trace Levels and select all debug levels except MISC_DEBUGGING.
However, deactivate the above trace settings if you experience any degradation in performance during heavy load situations.
In a contact center environment, another tool that can help you troubleshoot call processing problems is a flowchart that traces the call routing process based on the interactive voice response (IVR) menu choices that callers make. Figure 2 traces the processing of an incoming call received by a central distribution site. The call receives a voice treatment prompting the call to select between three menu options or hold for an agent. The flowchart indicates which set of agents receives the call based on the menu option selected, as well as describing the capacity (number of agents) in the particular skill set group.
For calls received during high traffic periods, with more than 20 calls queued up for agents, the flowchart indicates the announcements played to the callers and how the calls are routed. This type of flowchart is useful for troubleshooting problems reported by external users (customers). While Figure 2 shows a fairly simple example, where calls stay within a particular site, some contact center applications may overflow calls to other sites. In those cases, the overflow calls may traverse an IP WAN to a secondary site and may be handled by additional devices. In situations like that, you need to also view a network topology diagram for the secondary site to trace the call processing.
Figure 2 Interactive Voice Response Flowchart Example