Troubleshoot Call Drops on a CVP Courtesy Callback Deployment

Introduction 

This document describes how to troubleshoot call drops while caller is on queue in a Customer Voice Portal (CVP)  Courtesy Callback deployment.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• CVP Call Server
• CVP Voice Extensible Markup Language (VXML) Server
• CVP Call Studio Applications
• VXML Gateways

Components Used

The information in this document is based on these software versions:

• CVP 10.5(1)
• CVP Call  Studio 10.5(1)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Problem

In a CVP Courtesy Callback deployment, after the original caller is called back and while the caller waits in queue for an agent the call drops.

Troubleshoot

Step 1. Collect ActivityLogs from CallbackWait and CallbackQueue applications on the CVP VXML Server. You can find these logs on the directories:

C:\Cisco\CVP\VXMLServer\applications\CallBackWait\logs\ActivityLog\and  C:\Cisco\CVP\VXMLServer\applications\CallBackQueue\logs\ActivityLog\.

Step 2. Find the bad call in the CallbackQueue ActivityLogs. You can search for error or warnig to find the bad call for the specific timestamp. 

Snippet CallbackQueue ActivityLogs:

10.85.137.68.1469202885038.5788.CallbackQueue_custom,07/22/2016 11:59:24.656,Queue1,element,warning,A session has timed out after 3 minutes. This is most likely caused by a start of call class or action element at the top of the callflow not completing before the voice browser's fetch timeout occurred. To resolve it ensure this class executes in a timely manner or run it in the background. Session timeouts may also occur under high load or if there are issues with a load balancer or voice browser. 
10.85.137.68.1469202885038.5788.CallbackQueue_custom,07/22/2016 11:59:24.656,Queue1,custom,Callback_Leave_Queue,ELEMENT_ENTRY 
10.85.137.68.1469202885038.5788.CallbackQueue_custom,07/22/2016 11:59:24.656,Queue1,custom,Callback_Leave_Queue,ELEMENT_EXIT 
10.85.137.68.1469202885038.5788.CallbackQueue_custom,07/22/2016 11:59:24.656,,end,how,app_session_complete 
10.85.137.68.1469202885038.5788.CallbackQueue_custom,07/22/2016 11:59:24.656,,end,result,timeout

Step 3. As you can see in the ActivityLogs, a warning message is found, which indicates that the session has timeout. This is reported in the VXML Gateway logs as a badfetch error.

Step 4. Collect Tomcat logs from the VXML Server. You can find the Tomcat logs in the directory C:\Cisco\CVP\VXMLServer\Tomcat\logs

java.lang.NullPointerException
      at org.apache.coyote.http11.InternalNioOutputBuffer.flushBuffer(InternalNioOutputBuffer.java:240)
      at org.apache.coyote.http11.InternalNioOutputBuffer.endRequest(InternalNioOutputBuffer.java:128)
      at org.apache.coyote.http11.AbstractHttp11Processor.endRequest(AbstractHttp11Processor.java:1586)
      at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1022)
      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)
      at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1600)

As you see in the Tomcat server logs, there are null pointer exceptions at NIO (Non-blocking Input/Output).

Note: NIO is a collection of JAVA Application Programming Interface (API) used for intensive Input/Output (I/O) operations.

Step 5. Check the network connectivity between CVP VXML Server and CVP VXML Gateway. In most scenarios, when this Tomcat error is reported the VXML gateway and the CVP VXML Server are in different subnets.

Solution

Step 1. Ensure that fetchtimeout is configured to minimum 60 seconds. Follow these steps if you have  not configured fetchtimeout.  

• Add the VoiceXML property fetchtimeout to the root document.
• In Unified Call Studio, right click on the desired project and choose Properties.
• Select on Call Studio - Root Doc Settings.
• Under VoiceXML Property enter fetchtimeout, and under Value enter the desired timeout. For example for 60 seconds enter 60s

Step 2. Modify the Tomcat server.xml file to include  useSendfile="false".  You can find this file in C:\Cisco\CVP\VXMLServer\Tomcat\conf\ directory.

For instance:

<Connector port="7000" useSendfile="false" redirectPort="7443" protocol="org.apache.coyote.http11.Http11NioProtocol" maxHttpHeaderSize="8192" executor="tomcatThreadPool" acceptCount="1500"/>

<!-- A "Connector" using the shared thread pool-->


<!-- <Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" /> -->


<!-- Define a SSL HTTP/1.1 Connector on port 8443 This connector uses the JSSE configuration, when using APR, the connector should be using the OpenSSL style configuration described in the APR documentation -->

Note: This is Tomcat issue and not attributed to a CVP product. Refer CSCus07896 for more details.

Step 3. To address the packet delays when different subnets are used, there is a recommendation to change the windows registry key, TcpAckFrequency to 1.

Note:This recommendation is to address a network issues (if any) for the CVP solution that uses different subnet. Refer to CSCuq07550 for more details.