This document describes how to troubleshoot Cisco Customer Voice Portal (CVP) Voice eXtensible Markup Language (VXML) Server OutofMemory, License and Call flow issues.
Cisco recommends that you have knowledge of these topics:
Cisco Unified Contact Center Enterprise (UCCE)
Cisco Voice Portal
VXML Server Deployment
Automatic Speech Recognition (ASR) and Text To Speech (TTS)
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Troubleshoot Out of Memory Issues
One of the most common problems in CVP is when the VXML Server runs out of memory and report or/and crashed with the error "java.lang.OutOfMemoryError".
Till CVP 11.0 version, There is a restriction on total size of the memory allocated to VXML Server, as shown in the image.
All this configuration of Heap Memory , PermGen and Native memory is configured in HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun2.0 \VXMLServer\Parameters\Java\Options.
Categorization of Out of Memory Issues
Error: OutofMemory "java.lang.OutOfMemoryError"
JVM Thread : Unable to create JVM thread
Solution: Troubleshoot Custom application / Increase Heap Memory ONLY if required
For 11.0 : Install ES 15 and then ES-25 for known VXML issue
Error: "lang.OutOfMemoryError: unable to create new native thread"
Solution: Increase Permgen to 256 ( MAX)
For 11.5 : Default PermGenSize is 512 ( Refer CSCvc71931)
logs files: hs_err_pid*.log
Integration of 3rd party libraries with JVM ( Refer to how to deploy 3rd party libraries in Presenation)
Look for possible Java heapdump (*.hprof) at C:\Cisco\CVP\VXMLServer\logs.
Look for PermGen errors: C:\Cisco\CVP\VXMLServer\Tomcat\logs.
Look for JVM Crash L hss_err_pid_*: C:\Cisco\CVP\VXMLServer\Tomcat\bin & C:\windows\system32.
Use jConsole for online monitor.
Use VisualVM in order to monitor the VXML Server and the application performance.
Monitor Heap and permgen
Take thread dump - Look for blocked threads
Heap Dump / Snapshot
Use Eclipse MAT , when the VXML Server already crashed and you would like to know the reason for the crash.
Load hprof into MAT. Navigate to File > Open Heap Dump
Report will display problem suspects in leak
Troubleshoot VXML Upgrade Issues
Other common issues happens while VXML upgrades, refer to release notes and compatibility matrix for more detailed info.
Consider different TOMCAT versions, it's important to note that Custom Application Development should follow this Class Heirarchy when you upgrade the application. A 3rd pary library which works fine in one TOMCAT version may not be able to do so in new TOMCAT.
Note: Avoid the use of CVP_War to place custom / 3rd Party Jar files.
Troubleshoot Non-Memory Issues (Call flow, ASR and TTS)
Collect the logs from these locations:
C:\Cisco\CVP\logs\VXML - Look these logs and track session ID for license related issues / VXML call across multiple application
AdminLogger (for issues related to deployment of Application)
Global Call Logger ( VXML Application Call issues)
Error Logger (Global Application Error logger , logs error related to application execution)
Tomcat logs (useful to debug application which causes issues with integrationg with Tomcat)