This document describes a loss of heartbeats between the Voice Response Unit Peripheral Interface Manager (VRU PIM) and the Customer Voice Portal (CVP) server. This caused a failover and intermittent issues.
pim1 Error receiving data from VRU.
Last API Error [10054]: An existing connection was
forcibly closed by the remote host.
pim1 TCP connection to VRU has been broken.
Mar 30 2013 19:36:46.105 -0500:
%CVP_8_5_ICM-1-LOGMSG_ICM_SS_STATE:
Shutting down VRU PIM connection. Transition to
partial service. [id:2006]
Mar 30 2013 19:36:46.136 -0500:
%CVP_8_5_MSGBUS-3-MESSAGING_LAYER:
ConnectionServer(GED125)::
terminateConnection on plugin(GED125)
with connection(Socket[addr=/161.135.182.16,
port=4335,localport=5000])
due to: Plugin was stopped by the application [id:1]
Mar 30 2013 19:36:46.531 -0500: %CVP_8_5_IVR-7-CALL:
{Thrd=http-8000-1} VXMLManager:generateVXML:
CALLGUID=E1D13C7998D111E288360013C39AE710
Generated VXML from template 'PlayMediaIOS.template' for
client: 161.135.211.38 clientType: IOS
Mar 30 2013 19:36:57.328 -0500:
%CVP_8_5_ICM-6-LOGMSG_ICM_SS_GENERAL_INFO: Missed 2 VRU PIM
heartbeats. Closing session and waiting for new connection
from PIM. [id:2007]
Collect Perfmon (CSV format) from both affected servers. In this case it was the CVP server and the Peripheral Gateway (PG) server where the affected VRU was hosted. Open perfmon on a local system. Identify the time frame in which a heartbeat is missing or a gap in communication (logs) appears. Select the Deferred Procedure Call (DPC) rate and identify if there was upward movement at the time frame. In this scenario, there was an increase from 0 to 10 at the exact second the gap in logging was present (refer to the figure). If you verify that the log gap coincides with the DPC spikes (no matter the percentage), then the DPC is the probable culprit of the dropped User Datagram Protocol (UDP) packets.
% DPC Time shows the percentage of time that the processor spent to receive and service deferred procedure calls (DPCs) in the sample interval time period. DPCs are interrupts that run at a lower priority than standard interrupts. % DPC Time is a component of % Privileged Time because DPCs are executed in privileged mode. They are counted separately and are not a component of the interrupt counters. This counter displays the average busy time as a percentage of the sample time.
Refer to Windows Server Processor Object- By clicking on the link, you will be directed to a third party website that is not affiliated with Cisco.
The Ndis.sys driver queues the DPC routines at a low importance level on the same processor that services the interrupt service routine (ISR). Therefore, the UDP related DPC routine goes to the end of the queue, and this DPC routine is processed last. Additionally, the DPC queue of the processor may not be empty, and these DPCs for other I/O drivers are processed first. If the DPC rate is sufficiently high for all I/O drivers, not just for NDIS, there could be a noticeable delay.
Under a heavy stress situation, this delay could cause the system to drop packets when the Ethernet adapter's receive buffers fill while the receive buffers wait for the queued DPC routine to finish.
Refer to Applications that use the UDP protocol may encounter poor performance on a computer that is running Windows Server 2003- By clicking on the link, you will be directed to a third party website that is not affiliated with Cisco.
This affects UDP traffic only.
The normal suspects such as NIC settings, TCP offloading, and VM snapshots could also cause similar issues.
Refer to Applications that use the UDP protocol may encounter poor performance on a computer that is running Windows Server 2003- By clicking on the link, you will be directed to a third party website that is not affiliated with Cisco.
A supported hotfix is available from Microsoft. However, the hotfix is intended to correct only the problem that is described in this article. Apply this hotfix only to systems that experience the problem described in this article. This hotfix might receive additional testing. Therefore, if you are not severely affected by this problem, Cisco recommends that you wait for the next software update that contains this hotfix.
If the hotfix is available for download, there is a "Hotfix download available" section at the top of the Knowledge Base article. If the section does not appear, contact Microsoft Customer Service and Support to obtain the hotfix.
To apply this hotfix, your computer must run Windows Server 2003 Service Pack 2 (SP2).
You must restart the computer after you apply this hotfix.
You do not have to make any change to the registry.
The English version of this hotfix has the file attributes (or later file attributes) that are listed in these tables. The dates and times for these files are listed in Coordinated Universal Time (UTC). When you view the file information, it is converted to local time. To find the difference between UTC and local time, use the Time Zone tab in the Date and Time item in Control Panel.
For all Supported x86-based Versions of Windows Server 2003
File name | File version | File size | Date | Time | Platform |
Ndis.sys | 5.2.3790.4524 | 210,432 | 04-Jun-2009 | 13:29 | x86 |
For all Supported x64-based Versions of Windows Server 2003 and of Windows XP
File name | File version | File size | Date | Time | Platform |
---|---|---|---|---|---|
Ndis.sys | 5.2.3790.4524 | 361,984 | 04-Jun-2009 | 17:48 | x64 |
For all Supported Itanium-based Versions of Windows Server 2003
File name | File version | File size | Date | Time | Platform |
---|---|---|---|---|---|
Ndis.sys | 5.2.3790.4524 | 646,656 | 04-Jun-2009 | 17:49 | IA-64 |