Troubleshoot CVP VXML Server

Introduction

This document describes how to troubleshoot Cisco Customer Voice Portal (CVP) Voice eXtensible Markup Language  (VXML) Server OutofMemory, License and Call flow  issues.

Prerequisites

Requirments

Cisco recommends that you have knowledge of these topics:

• Cisco Unified Contact Center Enterprise (UCCE)
• Cisco Voice Portal
• VXML Server Deployment
• Automatic Speech Recognition (ASR)  and Text To Speech (TTS)

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Troubleshoot Out of Memory Issues

One of the most common problems in CVP is when the VXML Server  runs out of memory and report or/and crashed with the error "java.lang.OutOfMemoryError".

Till CVP 11.0 version, There is a restriction on total size of the memory allocated to VXML Server, as shown in the image.

All this configuration of Heap Memory , PermGen and Native memory is configured in HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun2.0 \VXMLServer\Parameters\Java\Options.

Categorization of Out of Memory Issues

• Crashes 
  • Heap Issues:

    • Error: OutofMemory "java.lang.OutOfMemoryError"

      • JVM Thread : Unable to create JVM thread

      • Solution: Troubleshoot Custom application / Increase Heap Memory ONLY if required

      • For 11.0 : Install ES 15 and then ES-25 for known VXML issue

    • PermGen Issues:

      • Error: "lang.OutOfMemoryError: unable to create new native thread"

      • Solution: Increase Permgen to 256 ( MAX)

      • For 11.5 : Default PermGenSize is 512 ( Refer CSCvc71931)

  • JVM Crash:

    • logs files: hs_err_pid*.log

    • Integration of  3rd party libraries with JVM ( Refer to how to deploy 3rd party libraries in Presenation)

Logs Related

Look for possible Java heapdump (*.hprof) at C:\Cisco\CVP\VXMLServer\logs.

Look for PermGen errors: C:\Cisco\CVP\VXMLServer\Tomcat\logs.

Look for JVM Crash L hss_err_pid_*: C:\Cisco\CVP\VXMLServer\Tomcat\bin & C:\windows\system32.

Tools

• JConsole
• VisualVM
• Eclipse MAT

JConsole

Use jConsole for online monitor.

VisualVM

Use VisualVM in order to monitor the VXML Server and the application performance.

• Monitor Heap and permgen
• Monitor PermGen
• Application Snapshot
• Take thread dump - Look for blocked threads
• Heap Dump / Snapshot

Elipse MAT

Use Eclipse MAT , when the VXML Server already crashed and you would like to know the reason for the crash.

• Load hprof into MAT. Navigate to File > Open Heap Dump
• Report will display problem suspects in leak

Troubleshoot VXML Upgrade Issues

Other common issues happens while VXML upgrades, refer to release notes and compatibility matrix for more detailed info.

Consider different TOMCAT versions, it's important to note that Custom Application Development should follow this Class Heirarchy when you upgrade the application. A 3rd pary library which works fine in one TOMCAT version may not be able to do so in new TOMCAT. 

Note:  Avoid the use of  CVP_War to place custom / 3rd Party Jar files.

Troubleshoot Non-Memory Issues (Call flow, ASR and TTS)

Collect the logs from these locations:

• C:\Cisco\CVP\logs\VXML - Look these logs and track session ID for license related issues / VXML call across multiple application
• C:\Cisco\CVP\VXMLServer\logs
  • AdminLogger (for issues related to deployment of Application)
  • Global Call Logger ( VXML Application Call issues)
  • Error Logger (Global Application Error logger , logs error related to application execution)
• C:\Cisco\CVP\VXMLServer\Tomcat\logs
  • Tomcat logs (useful to debug application which causes issues with integrationg with Tomcat)
• Application Logs: C:\Cisco\CVP\VXMLServer\applications\>Application>\logs
  • Activity : Application execution
  • Error : Error  logs with in Application
  • Admin: Common