Configure and Troubleshoot CVP SIP Heartbeats

Requirements

Cisco recommends that you have knowledge of these topics:

• Cisco Unified Customer Voice Portal (CVP)

Components Used

The information in this document is based on these software versions:

• Cisco Unified Customer Voice Portal (CVP) 12.5
• Session Initiation Protocol (SIP)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Overview

A Server group consists of one or more destination IP addresses (endpoints) of the same type, which is identified by a Server group Domain Name. The Server Groups add a heartbeat mechanism with endpoints for SIP. This feature enables faster failover on call control to eliminate delays due to failed endpoints.

• CVP SIP Server group uses the SIP Options method for the heartbeat mechanism.
• OPTIONS allows a user agent (UA) to query another UA or a proxy server as to its capabilities. This allows a client to discover information about the supported methods, content types, extensions, codecs, and so on. without “ringing” the other party.
• Most implementations use OPTIONS as a SIP ping mechanism. SIP Ping RFC

Configuration

You must perform these procedures to enable the CVP SIP Heartbeat for the SIP end points:

Step 1. Access OAMP Web Page.

Step 2. Navigate System -> SIP Server Group.

Step 3.See list of SIP servers if you added already, or you can add those new end points.

Step 4. Navigate to Heartbeat Properties tab.

Step 5. Configure as shown here. Once you enable the Use Heartbeats to Endpoints, default values gets populated.

Note:  SIP Errors (503 - Service Unavailable, 480 - Temporarily Unavailable, 600 - Busy) every Static 3 minutes downtime. The Up and Down Endpoint Heartbeat Interval is between any two heartbeats; however, it is not between heartbeats to the same endpoint. The SIP Server Group does not wake up at specific interval and send a heartbeat for all elements since this approach can result in CPU utilization issues. It also takes more resources to track heartbeats for many endpoints.

For example, for 3 total elements across all SIP Server Groups to proactively send a heartbeat to each element at 30000ms (30 seconds) intervals, you have to set the Endpoint Heartbeat Interval to 10000ms (10 seconds). It is less deterministic for reactive mode since elements that are currently down can fluctuate, so the heartbeat interval fluctuates, with it. To turn off ping when the element is UP, set the UP interval to zero (reactive pinging). To turn off ping when the element is down, set the DOWN interval to zero (proactive ping). To ping when the element is either UP or DOWN, set both the intervals to greater than zero (adaptive ping).

Step 6. SIP Server group Ports. Also to be aware if SIP Server Group Heartbeat is not detected or no sip options message is seen on wireshark to the configured destinations.

Be aware if Call Server SIP Subsystem in Call Server SIP tab is configured to listen on port 5060, Then you cannot configure SIP Server Groups for the Heartbeat to listen on the same port, it has to be different port.

Note:  The issue is that the tool does not tell or warn the user that the port is already configured for either the sip subsystem or the sip server group heartbeat. The sip server group heartbeat options does not work and there is no way to tell with the logs. It simply marks the SRV Group all green regardless if it is up or down as no sip option message is sent so the user thinks the servers are all up and it could not be the case. The listen ports for SIP Call Server and SIP Server Group heartbeat must be different as they are different processes so the same port cannot be used twice.

Step 7. Override Host: Is Resource Allocation Information (RAI)?. If you want to use RAI from the gateway, you have to make sure to associate the override host with the RAI group in the GW Deploy and Configure with the Trunk Utilization Feature with Customer Voice Portal .

How it Works

Demonstrate the Message flow between SIP endpoints and CVP call Server.

Diagnostic

Step 1. Navigate to the call server Diag page. Use any browser https://<callserver-ip>:8000/cvp/diag. 

Step 2. Verify the SIP, ICM, IVR state supposed to be IN Service, but we see here as Partial Service.

Step 3. You can use any of the links to check the error caused for partial service.

Step 4. Enable SIP trace to collect the logs and check. Follow the link and refer to CVP section.

Troubleshoot

SIP Options message generated by CVP are not shown in the logs. Wireshark traces or Capture on the WSM is required.

Scenario 1

One of the SIP Server group endpoint is Unreachable.

Usually, you can find the Unreachable table turned true or false from the CVP Call Server logs.

5: 10.201.224.41: Aug 19 2022 12:36:49.655 -0700: %CVP_12_5_SIP-6-SIP_INFO:  B2BUA using timeout=180  seconds for DsUnreachableDestinationTable [id:5000] 
58: 10.201.224.41: Aug 19 2022 12:36:50.827 -0700: %CVP_12_5_SIP-6-SIP_INFO:  Using property: DsUnreachableDestinationTableTimer=180 [id:5000] 
130: 10.201.224.41: Aug 19 2022 12:36:50.843 -0700: %CVP_12_5_SIP-6-SIP_INFO:  Using property: UseDsUnreachableDestinationTable=true [id:5000] 

From the Wireshark, you can see:

Scenario 2

Messages in CVP logs

14: 10.201.198.7: Sep 26 2021 01:21:25.988 -0500: %_ConnectionManagement-3-com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.ConnectionManagement:  UnreachableDestinationTable - remove (10.201.198.21:5060:1). Current count: 3  
16: 10.201.198.7: Sep 26 2021 01:21:25.988 -0500: %_ConnectionManagement-3-com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.ConnectionManagement:  UnreachableDestinationTable - remove (10.201.198.21:5060:2). Current count: 2  
18: 10.201.198.7: Sep 26 2021 01:21:25.988 -0500: %_ConnectionManagement-3-com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.ConnectionManagement:  UnreachableDestinationTable - remove (10.201.198.21:5061:4). Current count: 1  
217: 10.201.198.7: Sep 26 2021 01:21:25.988 -0500: %CVP_12_5_SIP-6-SIP_INFO:  /10.201.198.21 endpointUnreachableClear com.dynamicsoft.DsLibs.servergroups.EndpointEvent[source=DEFAULT:10.201.198.21:5060:1 numTries=3] [id:5000] 

No Heartbeats enabled – CUCM Service unavailable

CUCM endpoint goes to unreachable table for 3 minutes.

Recommendation

• Setup SIP heartbeats
• Check the logs to see what CUCM sends and ignore this code (Sip.properties)
• Review: SIP.System.ExcludedCauseCodeFromUnreachableTable = 47,25
  

  Any destination added to unreachable Table on 503 response does further filter to cause code sent in Reason-header. Fix added a property in the SIP.properties as SIP.System.ExcludedCauseCodeFromUnreachableTable = 47,25

  If the cause-code matches the incoming cause-code, then destination would not be added to unreachable Table.
• Review: SIP.DsUnreachableDestinationTableTimer = 15
  

  Unreachable table timer is configurable now SIP.Properties.

  Review : Disable the CVP Unreachable Destination Table Tracking - Cisco

Note:  Set SIP.UseDsUnreachableDestinationTable = false cannot help to detect actual SIP endpoint outage or congestions. This is a workaround. Add OPTIONS in order to investigate the root cause of the problem.