Possible Errors & Troubleshooting Tips
This section documents errors that customers may encounter and how to fix them.
Table 1-1 Possible Errors and Reasons
Customer pod has no display
1. Check whether monitor is working and the power is connected.
2. If monitor was connected after the IEC was booted up, reboot the IEC.
3. Try connecting the monitor to IEC’s alternate video port (VGA or HDMI).
IEC displays ‘Management failure: Product VEP is not found’
The IEC has an older version of firmware. Upgrade the IEC's firmware.
IEC displays 'Network Error'
If LAN cable was connected after the IEC was booted, reboot the IEC. Check the network connection. If it is DHCP-based, check whether the DHCP server is correctly leasing an IP address to the IEC.
IEC displays 'Kiosk Phone is out of Service'
Customer pod-side video endpoint
Customer pod-side video endpoint is unregistered from the UCM.
IEC reboots in certain cases
Customer pod-side video endpoint
The following are some of the scenarios in which reboot is initiated from REM:
- Whenever there is a power failure, REM reboots the IEC to identify all peripherals that are connected.
- If customer pod-side video endpoint goes down and comes up, REM reboots the IEC.
- If the Cobra application is not working normally, REM reboots the IEC.
IEC is not streaming video such as the On Hold Video or a video that was pushed from an expert
RTMP Compliant Streaming Server
1. Check RTMP Compliant Streaming Server is up.
2. If the agent is not registered, the call will not be connected to the agent who then will not be able to load READ. Also, the Kiosk will not move to the connected screen.
Agent Desktop displays 'Checking for Active Session' continually, even after accepting the call
Contact Center CAD Admin Workflow
1. Validate Expert configuration in REAC.
2. Verify CAD Workflow settings.
Additional issues and how to solve them are described below.
Unable to Bring Up Kiosk Flex Application on the Customer Pod’s Home Page
The call flow that is related to the customer pod’s display page is the following:
1. While the IEC boots up, the IEC contacts the IEM for its home page URL (startup URL in the IEM) which is hosted on REM.
2. The IEC goes to the URL destination listed in the startup URL of the IEM’s policy.
3. The REM (the destination of the URL) requests the IEC to load the Kiosk Flex application.
4. While requesting to load the Kiosk Flex application, the Kiosk Flex application queries the IEC for its serial number.
5. The IEC responds with its serial number.
6. If the IEC’s serial number matches what is configured in the REM, the Kiosk Flex application is loaded into the IEC and the IEC displays the home page with a connect button on the customer pod.
7. The IEC also displays a static image on the TP video endpoint (e.g. office hours).
Figure 1-3 Kiosk Flex Application Call Flow
If you are unable to bring up the Kiosk Flex application, the following are possible reasons:
- The IEC was configured with the wrong IEM host information. Verify that the IEM’s URL is correct in the IEC and the IEM shows that the IEC is active.
- The URL configured in the IEM is incorrect. First put the default kiosk serial number in rem.properties and run the configuration tool. Then register the kiosk with the default IEC in REAC. Verify that the URL is correct by typing “https://<REM_IP>:8443/reic” in a browser. You should see the same homepage with the connect button on the browser window and be able to click the connect button with a mouse. As a result, a call will try to connect with the video endpoint using the UCCX/UCCE Pilot number.
- The IEC does not have the correct policy applied. Verify that the policy with the REM’s URL as the startup URL was applied to the IEC in the IEM.
- The IEC is unable to reach the IEM. This could be a network problem (e.g. Local LAN issue, WAN issue, or cable issue). If the REM’s FQDN is used, it could be that DNS server is either incorrectly configured or not reachable. Ping the REM from IEC’s console to ensure that it is pingable.
- The LAN connection may not have been up when the IEC was booting up. Reboot the IEC.
- The web service may not be up. Verify that all nine web services (listed below) are up by entering the URL "https://<REM_IP>:8443/resc/services/listServices".
1. Print and Video Share Service
2. File Upload Service
3. Spring Init
4. Customer Service
5. Admin Service
6. Notification Service
8. Virtual Agent Services
9. Reporting Service
- The IEC’s serial number is listed incorrectly in the REM. Verify that the REM’s Kiosk configuration window is showing the correct serial number of the IEC.
- Check the IEC’s event log from IEM for more information. Refer to the “Events Tab in the IEM” section in the Serviceability Administration chapter for detailed information on how to check an IEC’s logs.
Unable to Establish Call from Customer Pod (‘System Error’ Message)
When a customer presses the Connect button, the call flow is the following:
1. The Kiosk application invokes the REM’s call connect web service.
2. The REM requests the CUCM to set up a call via JTAPI between the customer pod’s video endpoint DN and UCCX/UCCE Pilot DN.
3. The CUCM initiates a call from the customer pod’s TP video endpoint to UCCX/UCCE Pilot DN and the call is established.
Figure 1-4 Connect Button Call Flow
When a call is unable to be established, the following are possible reasons:
Configuration data is incorrect in the REM:
– The TelePresence DN of the customer pod’s video endpoint is listed incorrectly in REM. Check the Kiosk configuration in the REAC.
– The IVR Phone number is listed incorrectly in the REM. Check the Expert Type configuration in the REAC.
– The customer pod’s video endpoint is not listed in the CUCM’s application user (ragent) control list. Include the customer pod’s video endpoint into the REM’s control list.
– The CUCM’s application user (ragent) either does not exist or the user’s password is not matching the password in the REM configuration.
– The REM has incorrect CUCM credentials (username, password, IP addresses).
– The REM may have duplicate entries for the customer pod’s video-endpoint (DN).
- Problem with the JTAPI links to either the REM or to UCCX/UCCE:
– Check that the JTAPI link between REM and CUCM is up and running.
– Check that the JTAPI link between CUCM and UCCX/UCCE is up and running by checking the CTI Route Point’s status in the CUCM administration page.
– The customer pod’s video endpoint is not in the active state (e.g. upgrading firmware or rebooting state).
– The UCCX/UCCE ports are maxed out and hence unable to establish a call with UCCX/UCCE (e.g. UCCX/UCCE getting more calls than it can support).
Customer Pod Displays ‘Error: Management Server is not reachable’ Message
The message “Management Server is not reachable” indicates that the IEM IP Address is not correct or inaccessible. This message may also appear due to a network problem, a proxy server configuration error, or an incorrect IEM URL. Check if the firewall policy is blocking access.
If the IEM is down but the IEC has accessed the startup URL previously, it will load the startup URL from its cache. In other words, the failure of the IEM does not prevent the IEC from functioning. If any configuration changes are needed, then IEM has to be active for pushing the new policy configurations to the IEC.
Customer Pod Displays ‘Startup URL is not configured’ Message
The following are the possible reasons and resolutions for the message “Startup URL is not configured”:
- IEC does not have a policy (initial configuration that includes startup URL) enforced in IEM. Verify the IEC has the correct policy applied and the IEC has been rebooted.
- Another possibility is that the IEC is not registered in the IEM, instead it is in standalone mode. Check the IEM to ensure that the proper serial number is added for the IEC.
- If the IEC is not rebooted after configuration changes, reboot the IEC.
IEC Displays ‘Cannot register’ Message
Click the Show Details button to reveal information about service that is disabled.
This is due to the fact the IEM is not enabled for registration.
Step 1 Log into the IEM as root/administrator user. Otherwise, users cannot see the Maintenance link. The Maintenance link is not shown to regular users.
Step 2 Click the Maintenance link.
Step 3 Click Server Settings.
Step 4 Check the Device gateway enabled check box.
Step 5 Click Apply.
Customer Pod Touch Screen Function is Not Working Correctly
If the touch screen (customer pod) is not working correctly:
1. Ensure that the USB interface cable is plugged into the IEC and the touch screen.
2. Use the calibration utility to recalibrate the screen.
a. Press Ctrl+Alt+S.
b. Enter the DMC (Device Maintenance Code).
c. Click Calibrator.
3. Reboot the system if the touch screen USB cable was not connected before boot time.
Customer Pod Displays Virtual Keyboard When Call Connects
When connecting to a remote expert, the touchscreen connected to the IEC displays a virtual keyboard on the customer pod. This is due to an incorrect policy applied to the IEC. Disable the following configuration settings in the policy that is applied to the IEC. If there is no policy applied to the IEC, then these changes should be set in the IEC profile configuration.
Step 1 Log into the IEM.
Step 2 Go to the policy that is applied to the IEC or the IEC’s profile.
Step 3 Go to keyboard > virtual > enabled property.
Step 4 Set enabled value to false.
Step 5 Go to browser > input > popup > keyboard > enabled property.
Step 6 Set enabled value to false.
Step 7 Save the policy by clicking the Apply button.
Step 8 Reboot the IEC to activate the policy on it.
Customer Pod Displays ‘Server is down’ Message
There are multiple possible reasons for this error message:
1. Check the IEC's Policy or Profile settings in the IEM with respect to startup URL configuration.
1. Check REM-UCM JTAPI configuration (application username/password in UCM and REM).
2. The REM’s tomcat service may be down.
3. The most likely reason for this error message is that the REM’s service is not up and running. A successful REM installation should show nine services (Print and Video Share Service, File Upload Service, Spring Init, Customer Service, Admin Service, Notification Service, Version, Virtual Agent Services, and Reporting Service). To list those services, go to https://<REM_IP>:8443/resc/services/listServices where “<REM_IP>” is the IP address of the REM.
If you see only one service, you need to check resc.log. Check that the JTAPI link between the REM and UCM is up and running. If the link is not up and running, the UCM username and password that is listed in the rem.properties file are not what is configured in UCM application user page.
Try the following to resolve the issue:
1. Restart the Tomcat service. See the Cisco Remote Expert Manager Installation Guide for instructions.
2. Look for the message ‘bad login’ in the resc.log.
[root@localhost ~]# grep --color 'bad login' /var/rem/resc/logs/resc.log
2012-06-08 22:33:44,442 ERROR [pool-2-thread-1] org.apache.axis2.deployment.ServiceDeployer - The VirtualAgentServices.aar service, which is not valid, caused Error creating bean with name 'virtualAgentService' defined in class path resource [applicationContext.xml]: Cannot resolve reference to bean 'callService' while setting bean property 'callService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'callService' defined in class path resource [applicationContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.cisco.big.call.CallService]
: Constructor threw exception; nested exception is com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider -- bad login or password.
If it is a bad login or password error, correct the JTAPI username and password either in the rem.properties file or in the UCM application configuration page. If the rem.properties file is modified to correct this username and password issue, execute the IAS. See the Cisco Remote Expert Manager Installation Guide for instructions.
Customer Pod Displays ‘Service Temporarily Unavailable’ Message
This error occurs when the IEC cannot pull the home page from the REM because the startup URL configured in the IEM is not reachable or the REM server or services are down.
To resolve this error:
- Verify the REM is up and functioning by checking this URL on a browser: https://<REM_IP>:8443/reic where “<REM_IP>” is the IP address of the REM.
- Check the IEC’s event log in the IEM. Verify the IP address of the REM as configured in the IEM policy that is applied to the IEC or the IEC’s profile.
Customer Pod Displays ‘Kiosk is not registered’ Message
If the customer pod does not display the ‘Connect homepage’ message but instead displays the ‘Kiosk is not registered’ message, the reason for this error is due to the fact that REAC does not have an entry for this IEC in its Kiosk menu page. Add the IEC to the REAC. If it is already added, verify that the IEC’s serial number is correct.
The resc.log file will display this issue as ‘unique serialNumber’:
[root@localhost ~]#grep --color 'unique serialNumber:' /var/rem/resc/logs/resc.log
When the IEC is added or updated with the correct serial number, the resc.log should display ‘kiosk: <serialnumber> is alive’.
Connected Peripherals are Not Detected by the IEC
Peripherals such as a printer, scanner, keyboard, or mouse must be connected to the IEC before it is booted up in order for the IEC to detect them. If you connect a peripheral after the IEC has booted up, reboot the IEC to detect that peripheral.
IEC is Not Reflecting the Applied Policy
After applying the policy, IEC needs to be rebooted to have the policy enforced. Also check the IEM to ensure that the IEC has the proper policy applied to it.
IEC’s Profile Configuration is Not Active
If there is a policy assigned to the IEC, its configuration take precedence over the IEC’s profile configuration. If the IEC’s profile configuration is required instead, remove the applied policy.
Figure 1-5 Profile and Policy Hierarchy for IECs
Customer Pod Displays ‘System is not available, Please try after some time’ Message
If the customer pod displays the ‘System is not available, Please try after some time’ message, it could be due to the video endpoint at the customer pod is not in the UCM’s REM application user’s control list.
[root@localhost ~]# grep --color 'not in provider' /var/rem/resc/logs/resc.log
com.cisco.jtapi.InvalidArgumentExceptionImpl: Address 2512 is not in provider's domain.
To resolve this issue, go to the UCM application user configuration that is used by the REM in its rem.properties file. Add the video endpoint into its control list.
Use the Unix “less” command to view and search the resc.log file for RE call issues. Within the “less” command, search for the following strings to find call issues:
- ‘Making a Call connect via Jtapi’ is the resc.log entry indicating a new RE session attempt.
- ‘ObserverThread’ are resc.log entries containing ObserverThread that correspond to the client pod video endpoint initiating and participating in the RE session.
- ‘AgentObserver’ are resc.log entries containing AgentObserver that correspond to the expert pod’s video endpoint participating in the RE session.
The following command shows which the DNs that successfully got connected in the past:
[root@localhost ~]# grep --color 'CallConnected' /var/rem/resc/logs/resc.log
The result of the command is the following output:
2012-06-07 22:15:25,632 INFO [ObserverThread(com.cisco.big.call.BIGObserver@6f221448)] com.cisco.big.call.BIGObserver - CallConnected2512
2012-06-07 22:15:25,634 INFO [ObserverThread(com.cisco.big.call.BIGObserver@6f221448)] com.cisco.big.call.BIGObserver - CallConnected1134
2012-06-07 22:15:43,017 INFO [ObserverThread(com.cisco.big.call.BIGObserver@6f221448)] com.cisco.big.call.BIGObserver - CallConnected2504
2012-06-07 2:22:24,026 INFO [ObserverThread(com.cisco.big.call.BIGObserver@1539d980)] com.cisco.big.call.BIGObserver - CallConnected2512
Customer Pod Displays ‘Expert not available’ Message
Whenever the IEC is rebooted or powered up, the REM will check reachability to the IEC’s local video endpoint that is configured in the REAC’s Kiosk menu page. If the IEC’s video endpoint is not pingable from the REM, the REM instructs the IEC to display ‘Expert not available’.
This may also happen when the REM or REAC’s Kiosk page has multiple instances of the same IEC.
If you check the resc.log, you will see following error messages:
[root@localhost ~]#grep --color 'is not reachable' /var/rem/resc/logs/resc.log
WARN [http-bio-80-exec-60] com.cisco.big.admin.util.VepManagementUtil - ------ **** CTS for kiosk: 656015330015 is not reachable 4 ****
WARN [http-bio-80-exec-60] com.cisco.big.admin.util.VepManagementUtil - ------ **** CTS for kiosk: 656015330015 is not reachable 5 ****
WARN [http-bio-80-exec-60] com.cisco.big.admin.util.VepManagementUtil - **** ---- CTS for kiosk:656015330015 is dead. after 5 tries
A similar message will also be logged in the IEC’s Event screen in the IEM.
Customer Pod Displays Request to Provide Feedback After Pressing the Connect Button
If the customer pod displays a screen requesting the customer to provide feedback about the session after the customer presses the Connect button, the wrong CC Pilot DN was configured or the DN is not working. Contact Center could also be possibly down or busy or the UCM-Contact Center integration has issues.
Check the resc.log for Called Address:Unknown:
[ObserverThread(com.cisco.big.call.BIGObserver@5b7127)] com.cisco.big.call.BIGObserver - BIGObserver - Curent Called Address:Unknown
INFO [ObserverThread(com.cisco.big.call.BIGObserver@5b7127)] com.cisco.big.call.BIGObserver - BIGObserver - Curent Calling Address:2512
INFO [http-bio-80-exec-73] com.cisco.big.call.CallService - Time taken for Call Connect: 4
INFO [ObserverThread(com.cisco.big.call.BIGObserver@5b7127)] com.cisco.big.call.BIGObserver - Setting the session to status 2
INFO [ObserverThread(com.cisco.big.call.BIGObserver@5b7127)] com.cisco.big.call.BIGObserver - Call disconnected
Customer Pod Does Not Display the Home Page
There are several possible reasons why the customer pod does not display the Home page:
- The IEC does not have the policy with the startup URL pointing to the REM applied to it. Apply that policy to the IEC in the IEM.
- If IEC does not have a policy but using its profile in the IEM for the startup URL, verify that the startup URL in the profile is pointing to the REM.
Note Use of policies is the preferred method for configuring IEC properties.
- Verify that the IEC has been added to the REAC’s Kiosk menu.
- If the IEC was updated recently but not rebooted, the IEC needs to be rebooted.
- Check if REM Server is reachable and web services are accessible.
Once you have fixed the issue above, verify that the content in the Home page is correct by following these steps:
Step 1 Check which images will load in the IEC by going to the following link: https://<REM_IP>:8443/resc/services/VirtualAgentServices/getKioskDetailsBySerialNum?serialNumber=<Serial Number>. The PNG files are the ones that actually display on the customer pod.
Step 2 Those PNG files displayed are in fact mapped to specific Contact Center Pilot DNs, Expert Types are grouped together and the IVR (DN) number can be modified altogether for an Expert Type group.
Step 3 Check the IEC’s event log in the IEM to see if the request from the IEC was successful.
Note The IP address used in the request is also in its response. This would help to identify the REM IP address used by the IEC in its request.
IEC Reboots Twice
In certain circumstances, such as during a power failure at the branch, the IEC reboots twice. This is expected because the IEC was designed to only detect those peripherals that are connected to it when it is rebooted. In a power failure scenario, the TP video endpoint takes longer time to reboot compared to the IEC. As a result, the IEC does not detect the TP video endpoint connected to its HDMI port. In order to circumvent this issue, IEC reboots again after it finds the TP video endpoint is reachable.
The following resc.log shows that the REM rebooted the IEC:
2012-06-11 16:45:40,326 INFO [http-bio-80-exec-144] com.cisco.big.common.util.CommandExec - reboot
[root:localhost ~]$ echo "RE_CHANNEL_TERM";
Broadcast message from root@localhost (pts/2) (Mon Jun 11 09:45:28 2012):
The system is going down for reboot NOW!
Unable to See Wait Video While Call is in the Queue
The Wait Video that is streamed when the call is in a queue is handled by REM. The call flow when a customer call is being queued is as follows:
1. When there is no expert available, the UCCX/UCCE puts the call in a queue.
2. Based on the JTAPI event, the RECS requests the media server (Adobe) to stream a pre-configured video file to the IEC.
Possible reasons for the Wait video not playing are:
- The Wait video has not been added in the REM. Go to /opt/apache-tomcat-7.0.23/webapps/reic/assets/video and ensure that the Wait video has been added.
- If a particular video has been provided as the Wait video for an expert type in REAC and still does not play, check the following:
– The video format provided in REAC in the Video tab is correct (https or rtmp format).
– The video filename in REAC is the same as that given in the Adobe server (Folder Path: Adobe > webroot > vod > filename).
Unable to See On-Hold Video While Call is On Hold
The On-Hold video is handled by the external media server such as Adobe Media Server (AMS). There are two deployment models: customers can install AMS on a different server or customers can install AMS and REM on the same server.
Possible reasons for the On-Hold video not playing are:
- The Adobe server does not have the requested media file. Check the REAC Video tab to ensure that the file’s URL is provided in the correct format.
- The file is not present in the Adobe media server location (Folder Path: Adobe > webroot > vod > filename). Check if the filename provided in the REAC is the same as that stated in this location.
- Only filenames of types.flv,.f4v,.mp4 and.mov are supported. Video clips are streamed via the rtmp protocol. Only filenames and formats of these types would play if provided as the On-Hold video. Convert the video format or change the filename if necessary.
Unable to Bring Up Static Graphic Page in TP (During Non-TP Calls)
The IEC can display a static image or stream a video file to the TP video endpoint when it is not being used. If this feature is not working, a configuration mismatch is the likely cause.
Step 1 Verify the TP video endpoint device configuration in UCM is correct and make sure that the ‘Days Display Not Active’, ‘Display On Time’, and ‘Display on Duration’ settings are correctly configured.
Step 2 Verify the REM’s Content configuration page has correct image file names and hours.
Step 3 Go to the following URL: https://<REM_IP>:8443/reic/dualcontent.html
This URL response should display the graphics that supposed to show up at the current time.
Step 4 Verify in the IEM that the IEC shows dual images (one image for the customer pod and the other image is for TP video endpoint when it is not on a call). Click Devices in the left pane and then click the Show Screenshot icon located at the top of the center pane.
If the dual images show up in the IEM, then the issue is probably with the TP video endpoint.
Unable to Upload and Update Images to REM
The static graphic images and video files can be uploaded to the REM from REAC. However, such uploading fails from certain browsers. Only use supported browsers while accessing the REAC.
READ is Not Showing Up
When the expert answers the call, the READ will be displayed in the Cisco Agent Desktop’s integrated browser. If it does not, the following are possible reasons:
- UCCX/UCCE does not have a Premium Package license. The CAD needs the Premium Package license.
- Cisco Agent Workflow is not correctly configured. Verify that the Cisco Agent Workflow is correctly configured to bring up the READ from the REM server.
Note The configuration parameters in the HTTP Action Setup are case sensitive so ensure that the values entered into fields are correct.
- The webservices are not running. This link should show nine active services: https://<REM_IP>:8443/resc/services/listServices.
- Verify each of the call events.dation of READ. For different call event states, the expert’s desktop will display different web pages.
Note Replace <rem_server_ip> with the actual IP address and replace <agent_dn> with the actual user ID.
– Not Ready: https://<rem_server_ip>:8443/read/Common.jsp?agentDn=<agent_dn>&state=0&request=welcome
– Ready: https://<rem_server_ip>:8443/read/ Common.jsp?agentDn=<agent_dn>&state=1&request=welcome
– Ringing: https://<rem_server_ip>:8443/read/desktoppage?agentDn=<agent_dn>&calling=<ivr_queue_id>
– Dropped: https://<rem_server_ip>:8443/read/Common.jsp?agentDn=<agent_dn>&request=disconnect
– Logout: https://<rem_server_ip>:8443/read/Common.jsp?request=logout
eREAD Document Panel Has Changed Its Size
If the document panel size does not appear as expected on the agent’s screen, the READ_GADGET_WIDTH and READ_GADGET_HEIGHT properties have been changed in the REM Properties file.
The default values are READ_GADGET_WIDTH = ‘100%’ and READ_GADGET_HEIGHT = ‘790px’. The minimum value for READ_GADGET_HEIGHT is 500px. Once you have changed their values, restart the Finesse Tomcat service for the changes to take effect.
Video Call is Not Established Between Different Types of TelePresence Video Endpoints
When an expert is available, the customer call that has been queued in UCCX/UCCE will be redirected to the available expert’s video endpoint. If the video endpoints used between the customer and the expert are of different types (i.e. one is an EX90 and the other is a C40), there could be interoperability issues associated with MTP.
UCM-based MTP does not support video and if the end-to-call uses MTP resources, the resulting call will be an audio call. There are two workarounds:
1. Remove MTP resources being used in such calls between customer-side video endpoint and expert’s video endpoint.
2. If MTP is needed for some other reasons, use IOS-based MTP with pass-through configuration. IOS-based pass-through feature supports video calls getting established.
Experts are not Getting Registered in REAC
While registering experts in REAC, if the registration fails and shows both nodes (in a dual node setup) in red, do the following:
Step 1 Go to the console of the VM in VMWare VSphere.
Step 2 Type the command system-config-network.
Step 3 Go to the option Edit DNS Configuration.
Step 4 Check if the Hostname is different in both these VMs. The Hostname must be unique for each VM. If it is the same name, change the Hostname in one of the VMs.
Step 5 Validate the UCM configuration.
Step 6 Restart the network by typing the command /etc/init.d/network restart.
Note If the hostname has changed, restart Tomcat for REM to function properly.
REIC (Cobra Browser) Hangs
When the REIC hangs or goes into a limbo state due to some reason, use the following command to clear cache: https://<Virtual IP>:8443/resc/services/AdminService/cleanCallCache
The administrator can clean the call cache of kiosks if a session hangs. When a session hangs, a new call cannot be initiated to or from the customer pod. By cleaning the call cache, the session’s status in the Session tab is changed to “Completed”, which then allows the customer pod to start or receive a new call.
Step 1 Click the Clean Call Cache tab.
A Status box appears in the lower right corner of the screen indicating that the call cache was cleaned.
REM Error Message
If you see the error message “Address <DN> is not in provider's domain”, the device whose Directory Number (DN) is <DN> is not in the control-list of application user (ragent) in CUCM’s configuration. Add the device in the CUCM under the appropriate Application User.
EX90 Firmware Error Message
If you see the error message “The agent or workflow-initiated action request failed”, update the EX90 firmware to TC6.1.
Performance indicators from within the virtual machines are still valid. For the UC applications that support it, use RTMT or the perfmon data for analysis of the performance of the UC application. Data from these tools provides view of the guest performance such as disk, CPU, memory, etc.
Move to the VMware infrastructure when there is a need to get the perspective from the ESXi host. Use the vSphere Client to view data.
- If vCenter is available, historical data is available through the client.
- If vCenter is not available, live data from the host is available through the client.
Just like some of the UC applications, vCenter can be configured to save more performance data. The more historical data saved, the bigger disk space needed by the database used by vCenter.
Note This is one of the main areas where you need vCenter rather than going directly to the ESXi host for performance data. vCenter can save historical data that the ESXi host does not keep.
The configurations to change the amount historical data saved by vCenter is located in the vSphere client under Administration > Server Settings. For each interval duration, the duration that the statics will be saved for (days, weeks, months, or years) and the statistics level can be set. The statistic levels range from 1 to 4 with level 4 containing the most data. Look at the data size estimates to make sure there is enough space to keep all of the statistics.
VMware Performance Indicators
The following table lists the performance indicators to monitor and view from a VMware perspective when a virtual machine’s performance is not optimal. Most counters are from the ESXi host, which can give a perspective of VM interactions and overall host and data store utilization.
Table 1-2 VMware Performance Indicators
less than 80%
less than 3%
general trend is stable
Kernel command latency
less than 3ms
Physical device command latency
less than 20ms
Average commands issued per second
less than LUN capacity
Receive packets dropped / Transmit packets dropped
High CPU usage could be due to a small number of VMs using all of the resources or too many VMs running on the host. IF there are too many VMs running on the host, see if CPU reservations are in use (see oversubscription section). To isolate a CPU issue for a particular VM, consider moving it to another ESXi host.
To view the CPU performance indicators, go to the ESXi host's performance tab and select the advanced button. Under chart options, select CPU, timeframe, and then only the host (not individual cores) to view overall CPU usage on the host.
Each virtual machine CPU usage can also be seen from the Virtual Machines tab on the host.
To get a view of the reservations set by all of the VMs, use the Resource Allocation tab of the cluster.
Note This is only available via vCenter.
The guidelines do not support memory sharing between VMs. To verify, the swapping and ballooning counters should be set to zero. If a given VM does not have enough memory and there are not memory issues on the specific host, consider increasing the VM's memory.
To view the memory performance indicators, go to the ESXi host's performance tab and select the advanced button.
Under chart options, select memory and timeframe, and then select the following counters:
- Used memory
- Swap used
Used memory can be used to look at general trends. Swap and balloon should always be “0”, otherwise memory sharing is being used, which should not be the case.
Bad disk performance often shows up as high CPU usage. IOPS data can provide information on how hard the application and VM is working the disks. Specific activities can cause spikes in IOPS such as upgrades and database maintenance. If VMs running on the same datastore are all doing these activities at the same time, the disks might not be able to keep up. IOPS data can be seen from vCenter or the SAN. Disk latency (response time) is a good indicator of disk performance.
Step 1 To view the disk performance indicators, go to the ESXi host's performance tab.
Step 2 Choose the advanced button.
Step 3 Choose the appropriate datastore, which can be found on the datastore page.
Step 4 Under chart options, select disk and timeframe.
Step 5 Choose the following counters:
- Physical device command latency
- Kernel command latency
- Average commands issued per second
The kernel counter should not be greater than 2-3ms. The physical device counter should not be greater than 15-20 ms. The ‘Average commands issued per second’ counter can be used if IOPS are not available from the SAN. IOPS should be considered if it looks like datastore is overload. This IOPS data is viewable from the host and each VM.
Note For NFS datastores, the physical and kernel latency data is not available. Starting in VMware 4.0 update 2 and beyond the esxtop command can be used to view NFS counters and in particular the guest latency. The guest latency is a summation of the physical device and kernel latencies.
Generally, network performance issues can be seen by dropped packets. If dropped packets are seen from an ESXi host, the network infrastructure should be investigated for the issue, which might include a virtualized switch (Nexus 1000V). In ESXi 4.1, issues have been seen with large file transfers (e.g. SFTP/FTP transfers). For this issue, the Large Receive Offload options need to be disabled on the ESXi host. This setting is found on the host's Configuration tab > Advanced Settings > Net.*.
Note There are several LRO settings on this page and all of them need to be disabled. If a VM has been cloned and uses static MAC addresses, verify there are not duplicate MAC addresses in the network.
To view the network performance indicators, follow these steps:
Step 1 Go to the ESXi host's performance tab.
Step 2 Choose the advanced button.
Step 3 Under chart options, select Network and timeframe.
Step 4 Choose the following counters:
- Receive packets dropped
- Transmit packets dropped
The main thing to check is that no packets are getting dropped in the network.
Note Advanced network debugging and configuration can be done on Nexus 1000v if used, which requires vCenter.
CPU Oversubscription Implications on Performance
Some deployments allow CPU oversubscription, which is the sharing of CPU cores between VMs. In this case, a few more data points need to be considered. First, look to see if CPU reservations are set for the VM. Cisco has recommended CPU reservations while using CPU oversubscription for some applications which are part of the official OVA. Second, look at the CPU ready time (i.e. how long a VM is waiting to run on a core). This counter can be converted into a percentage. General guidelines are to keep the percentage below 6% and then anything above 3% should be monitored for desired response times.
On the C-series UCS servers there have been issues with the write cache battery backup. If this battery is not operating correctly, performance will suffer. Use a tool like WBEM Command Line Interface (wbemcli) to verify the battery is okay. The following output is an example of using the wbemcli:
wbemcli ei -noverify 'https://root:<password>@<ESXi Host IP>:5989/root/cimv2:VMware_HHRCBattery'
Go to the following link to learn more about troubleshooting virtualized environments: