Troubleshoot Finesse Call Overlap Issue with CVP Unreachable Destination Table

Introduction

This document describes how to troubleshoot Finesse call overlap issue that is closely related to CVP unreachable table in Cisco Customer Voice Portal (CVP) server.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• CVP Call Server
• CVP Operations, Administration, Maintenance and Provisioning (OAMP)
• Cisco Unified Communications Manager (CUCM)

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Cisco Finesse Not Ready - Call Overlap status is very a very common issue seen on Finesse desktop. Agents miss two calls and they are put in Not Ready - Call Overlap status. This is very likely related to CVP server and it's unreachable table mechanism. Cisco Unified Communications Manager (CUCM) server has a big role here as well.

Problem

When CVP receives specific Session Initiation Protocol (SIP) messages from CUCM (5XX and 4XX) or any other SIP endpoint, CVP deems the server is in trouble and stops to route calls to the SIP element. The SIP element (in this case, CUCM) is placed into the unreachable table.

There can be multiple conditions that affect this behaviour:

1. Heartbeats are enabled on CVP SIP Server Group. Recovery can be faster than three minutes.

2. Heartbeats are not enabled (CSCuy30406). Recovery can take upto three minutes.

3. Cisco Unified IP Communicator (CIPC) registered with private IP (CSCuy10282).

In this example, if CVP receives SIP 503 Service Unavailable from any CUCM server, it blocks to route calls to that endpoint as well as any other call related signal for the entire time the element is in the unreachable table. 

Since Agent PG is successfully able to reserve the agent via Finesse and agent phone does not receive call from CVP and CUCM, agent is seen in Call Overlap state after it misses two such calls.

CVP Call Server Log Analysis

CVP shows this error when the issue happens:

“Can’t connect to SIP server."

46253: 10.38.249.20: Sep 25 2018 16:32:16.635 -0600: %_Connection-4-com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.Connection: DsTcpNBConnection(): I/O Exception: java.net.SocketTimeoutException at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:109) at com.dynamicsoft.DsLibs.DsSipLlApi.DsTcpNBConnection.<init>(DsTcpNBConnection.java:259) at com.dynamicsoft.DsLibs.DsSipLlApi.DsSipTcpNBConnection.<init>(DsSipTcpNBConnection.java:120) at com.dynamicsoft.DsLibs.DsSipLlApi.DsSipConnectionFactory.createConnection(DsSipConnectionFactory.java:417) at com.dynamicsoft.DsLibs.DsSipLlApi.DsSipConnectionFactory.createConnection(DsSipConnectionFactory.java:239) at com.dynamicsoft.DsLibs.DsSipLlApi.DsTransportLayer.getConnection(DsTransportLayer.java:768)

CVP sent the call to 46003 (Agent extension) on CUCM

53137541.005 |18:07:17.797 |AppInfo |SIPTcp - wait_SdlReadRsp: Incoming SIP TCP message from 192.168.1.10 on port 32798 index 331135 with 1907 bytes:
[16836932,NET]
INVITE sip:46003@UCCE-Cluster.cisco.com;transport=tcp SIP/2.0
Via: SIP/2.0/TCP 192.168.1.10:5060;branch=z9hG4bKFxuPfAXH7ydZItoxHcH6Ig~~1036543
Max-Forwards: 69
To: <sip:46003@UCCE-Cluster.cisco.com;transport=tcp>

CUCM sent the call to 192.168.1.11 

53137602.001 |18:07:17.805 |AppInfo  |SIPTcp - wait_SdlSPISignal: Outgoing SIP TCP message to 192.168.1.11 on port 5060 index 3[16836934,NET]

INVITE sip:46003@192.168.1.11:5060 SIP/2.0

Via: SIP/2.0/TCP 192.168.1.12:5060;branch=z9hG4bK3cff18401b2380

From: <sip:+18005550000@192.168.1.12>;tag=6266176~faa91654-786e-4b55-bb8e-f71e4b7d1200-52767741

To: <sip:46003@192.168.1.11>

Date: Mon, 29 Feb 2016 18:07:17 GMT

Call-ID: 43327e00-6d4188d5-2eb80e-5a0a2818@192.168.1.12

Supported: timer,resource-priority,replaces

Min-SE:  1800

User-Agent: Cisco-CUCM10.5

Allow: INVITE, OPTIONS, INFO, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY

CSeq: 101

INVITE

Expires: 180

Allow-Events: presence

Supported: X-cisco-srtp-fallback,X-cisco-original-called

Call-Info: <urn:x-cisco-remotecc:callinfo>;x-cisco-video-traffic-class=MIXED

Cisco-Guid: 1127382528-0000065536-0000197816-1510615064

Session-Expires:  1800

P-Asserted-Identity: <sip:+18005550000@192.168.1.12>

Remote-Party-ID: <sip:+18005550000@192.168.1.12>;party=calling;screen=yes;privacy=off

Contact: <sip:+18005550000@192.168.1.12:5060;transport=tcp>

Max-Forwards: 68

Content-Type: application/sdp

Content-Length: 235 


v=0
o=CiscoSystemsCCM-SIP 6266176 1 IN IP4 192.168.1.12
s=SIP Call
c=IN IP4 192.168.1.13
b=TIAS:64000
b=AS:64
t=0 0
m=audio 23092 RTP/AVP 0 101
a=ptime:20
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15

192.168.1.11 replied back with 503 service unavailable.

53137618.002 |18:07:18.783 |AppInfo |SIPTcp - wait_SdlReadRsp: Incoming SIP TCP message from 192.168.1.11 on port 5060 index 3 with 589 bytes:
[16836936,NET]
SIP/2.0 503 Service Unavailable
Via: SIP/2.0/TCP 192.168.1.12:5060;branch=z9hG4bK3cff18401b2380
From: <sip:+18005550000@192.168.1.12>;tag=6266176~faa91654-786e-4b55-bb8e-f71e4b7d1200-52767741
To: <sip:46003@192.168.1.11>;tag=11566332~61d9dc41-d668-45b0-8140-083ab0d25e9d-72414340
Date: Mon, 29 Feb 2016 18:07:17 GMT
Call-ID: 43327e00-6d4188d5-2eb80e-5a0a2818@192.168.1.12
CSeq: 101 INVITE
Allow-Events: presence
Server: Cisco-CUCM10.5
Reason: Q.850; cause=41
Remote-Party-ID: <sip:46003@192.168.1.11;user=phone>;party=x-cisco-original-called;privacy=off
Content-Length: 0

CUCM sent 503 service unavailable back to CVP

53137648.001 |18:07:18.786 |AppInfo |SIPTcp - wait_SdlSPISignal: Outgoing SIP TCP message to 192.168.1.10 on port 32798 index 331135
[16836938,NET]
SIP/2.0 503 Service Unavailable
Via: SIP/2.0/TCP 192.168.1.10:5060;branch=z9hG4bKFxuPfAXH7ydZItoxHcH6Ig~~1036543
From: +18005550000 <sip:+18005550000@192.168.1.10:5060>;tag=ds6ab6b28c
To: <sip:46003@UCCE-Cluster.cisco.com;transport=tcp>;tag=6266175~faa91654-786e-4b55-bb8e-f71e4b7d1200-52767740
Date: Mon, 29 Feb 2016 18:07:17 GMT
Call-ID: 187EB7FDDE4611E5BECDA0ECF93A8A80-145676923777928840@192.168.1.10
CSeq: 1 INVITE
Allow-Events: presence
Server: Cisco-CUCM10.5
Reason: Q.850; cause=41
Content-Length: 0

Since CVP call server tracks down unreachable endpoints (In sip.properties, SIP.UseDsUnreachableDestinationTable = true), CVP added CUCM server into unreachable destination table and hold down any traffic to that server for 180 seconds. Once the servers in the sip server group gets 503 service unavailable, CVP no longer routes calls through that sip server group.

For CVP unreachable table, please see CSCuy30406 . Unreachable table was disabled in CVP.

This is the entry in CVP logs that shows elements added to unreachable destination table:

6510: x.x.x.x: May 05 2020 05:04:12.726 +1000: %_ConnectionManagement-3-com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.ConnectionManagement:  UnreachableDestinationTable - add (192.0.2.22:5060:2. Reason=DsSipClientTransactionImpl:response=DS_RESPONSE_SERVICE_UNAVAILABLE). Current count: 0

Solution

Disable CVP Unreachable table

https://www.cisco.com/c/en/us/support/docs/customer-collaboration/unified-customer-voice-portal/200912-Disable-the-CVP-Unreachable-Destination.html

Configure SIP Options Ping heartbeat on SIP Server Group:

https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/contact_center/customer_voice_portal/cvp11_6/configuration/guide/cvvp_b_configuration-guide-for-cisco-unified/cvvp_b_configuration-guide-for-cisco-unified_chapter_01111.html#CCVP_RF_H0F1315C_00

CVP 11.6 ES7 added paremeters to further tweak unreachable destination table:

CVP 11.6 ES7 Release Notes

• When CVP receives a 503 Service Unavailable , it first checks reason header of this 503 based on values in “SIP.System.ExcludedCauseCodeFromUnreachableTable =”. If there is no match, then CVP puts destination to unreachable table.
• If there is match, then CVP does not put destination to unreachable table.
• The values which in “SIP.System.ExcludedCauseCodeFromUnreachableTable =” purely depends on your environment. In another word what kind of Q.850 cause code is generated in your environment. Cause code 47 is very common with CUCM that is related to media resources.
  
  

Based on the CVP log snippet, here's one example to tweak unreacahble table parameters:

SIP.System.ExcludedCauseCodeFromUnreachableTable = 34,41,47

SIP/2.0 503 Service Unavailable
Via: SIP/2.0/TCP 192.168.1.14:5060;branch=z9hG4bKdGDMWjvRtZecqdhibRZLPw~~75244713
To: <sip:4170155@sip.cucm.cisco.com;transport=tcp>;tag=45652992~bc86a266-334e-4442-9a19-dc23b0bbb5f7-54664086
From: 0561714593 <sip:0561714593@192.168.1.14:5060>;tag=dse0b5c430
Call-ID: 2C87B293952911EAA6EFAB2EBAE6904A-158946732828459275@192.168.1.14<mailto:2C87B293952911EAA6EFAB2EBAE6904A-158946732828459275@192.168.1.14>
CSeq: 1 INVITE
Content-Length: 0
Date: Thu, 14 May 2020 14:51:06 GMT
Allow-Events: presence
Server: Cisco-CUCM11.5
Session-ID: 00000000000000000000000000000000;remote=60c552d91509577789316085283c3155
Reason: Q.850; cause=41