Troubleshoot Call Failures on TC Endpoints Registered to Cisco CallManager

Introduction

This document explains some of the common call failure issues faced with Tandberg Codec (TC) endpoints registered to Cisco CallManager and suggested solutions.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

How to Capture H.323 debug Logs

Note: Ensure Secure Socket Host (SSH) session output is captured.

1. SSH into the codec CLI and enter these commands:
   • log ctx H.323Packet debug 9
   • log output on (This outputs all the logs to the SSH session terminal session screen.)
2. Start a call and recreate the problem.
3. Enter the log output off and log ctx H.323Packet debug off commands.

How to Capture Session Initiation Protocol (SIP) debug Logs

Note: Ensure SSH session output is captured.

1. SSH into the codec CLI and enter these commands:
   • log ctx SIPPacket debug 9
   • log output on (This outputs all the logs to the SSH session terminal session screen.)
2. Start a call and recreate the problem.
3. Enter the log output off and log ctx SIPPacket debug off commands.

How to Collect Packet Capture/ Endpoint Logs from TC Endpoints

1. From the Web GUI choose Diagnostics > Log files and enable extended logging with full packet capture.
2. Start a call and recreate the problem. Note that packet capture can only be enabled for 3 minutes.
3. From the Web GUI choose Diagnostics > Log files and download the complete log archive and packet capture.

Other Information Required

• Complete call flow with all devices involved
• Called and Calling number
• Date and time of the issue occurred

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Problem: Call Failures due to Call Search Space (CSS)/Partition Issue on the CallManager

Calls between two endpoints registered to the Cisco Unified Communications Manager (CUCM) might fail due to a CSS/Partition issue on the CUCM.

Capture the calling endpoint SIP logs. This "404 Not Found" message appears on the Endpoint SIP logs that come from the CUCM:

 |SIP/2.0 404 Not Found
 Via: SIP/2.0/TCP 172.16.2.55:5060;branch=z9hG4bK26e12a6fbed832;received=172.16.2.55
 Call-ID: 77fec00-564180a1-1eec8b-370210ac@172.16.2.55
 CSeq: 101 INVITE
 From: <sip:1502@172.16.2.55>;tag=158127671
 To: <sip:4659@172.16.2.53>;tag=654ba920aeef9e74
 User-Agent: Cisco-CUCM10.5
 Content-Length: 0

Solution

Complete these steps in order to check the CSS of the Calling endpoint and the partition of the Called endpoint. Ensure the CSS of the Calling endpoint has the partition of the Called endpoint.

You can assign a CSS at the Device and Line level on the endpoint:

1. Choose Device > Phone, select the endpoint and click on the line, and check the Calling Search Space (CSS) at the line level. In this example, no CSS is configured at the line level. However if there is a CSS at the directory number level, either of the CSSs have to have partiton of the called number:

   

2. Check the CSS asigned on the phone level. Choose Device > Phone and select the calling endpoint in question:

   

3. Check the partition of the Called number. Choose Device > Phone, select the called device, click on the line, and check the Route Partion:

   

4. After you verify the Partiton and CSS on both endpoints, check if the CSS of the calling device has the partition of the called device:

   

   If not, this could be the cause of the "404 Not Found" error.

Problem: SIP Call Drop after 15 Minutes (or after any Specific Time)

Generally call drops at specific time intervals are caused by SIP timers or TCP timeout configured on firewalls, routers, and so forth.

Solution

When the call disconnects at exactly 15 minutes, the common problem seen is the TCP timeout configured on the network (firewalls, routers) is less than the SIP session expires timer. By default on CallManager, SIP Session Expire Timer is set to 1800 seconds.

In order to verify this, choose Cisco Unified CM Administration > System > Service Parameters > Cisco Call Manager Service >  Look for - SIP Session Expires Timer.

All the endpoints registered to CUCM use this timer. When the endpoint is on call with another remote endpoint, one of the parties has to refresh the session and sends a re-INVITE or UPDATE. This refresh has to be sent before half of the Session Expires Timer ( 1800/2 = 900 seconds = 15minutes). If there is no refresh message received, the call is disconnected.

Check for session timer in the initial INVITE. A refresh (INVITE / UPDATE) should be received before this time expires:

|INVITE sip:+1234@10.108.64.22:5060;transport=tcp SIP/2.0
 Via: SIP/2.0/TCP 10.110.68.38:5060;branch=z9hG4bK00eed555
 Call-ID: dbfe0000-4491f669-9fd00-16406c0a@10.108.64.22
 CSeq: 1 INVITE
 Contact: <sip:30048@example.com;gr=urn:uuid:f7a3a098-ead8-5512-85ef-26ae544d6547
   >;isfocus;x-cisco-tip
 From: "TP Conference 30048 - Test" <sip:30048@10.110.68.6>;tag=86251172C3B60000
 To: <sip:1234@10.108.64.22>;tag=25983910~226bf657-9d6c-4ad9-98a2-cf842fe1d733-52629917
 Max-Forwards: 70
 Route: <sip:proxy-call-id=53a00ced-68e1-4ecd-872b-1edbb9abc75b
 @10.110.68.6:5060;transport=tcp;lr>
 Route: <sip:proxy-call-id=53a00ced-68e1-4ecd-872b-1edbb9abc75b
 @10.110.68.6:5060;transport=tcp;lr>
 Allow: INVITE,ACK,CANCEL,OPTIONS,UPDATE,INFO,SUBSCRIBE,NOTIFY,BYE
 User-Agent: TANDBERG/518 (TC6.2.0.20b1616)
 Supported: timer,outbound,record-aware,X-cisco-callinfo
 Session-Expires: 1800;refresher=uac

Based on the initial User Agent Client /User Agent Server (UAC/UAS) negotiation, one of the endpoints refreshes the session when it sends a Re-INVITE. If the refresher is UAC, the initiator of the call has the responsibility to refresh the session. If the refresher is UAS, the server has to refresh the session. Collect the SIP debug logs from both endpoints and check these items:

Example: Call made from Party A to CUCM to Party B. If the refresher is UAC on Party A and UAS on Party B:

1. Party A has to send the re-INVITE / UPDATE to the CUCM.
2. CUCM has to send a re-INVITE / UPDATE to Party B.
3. Party B receives the re-INVITE and responds to that message with a 200 OK.
4. CUCM has to send 200 OK to Party A.

If one endpoint sends the re-INVITE message to the CUCM, CUCM sends a re-INVITE to the other Party. However, if this is not received by the remote side then this could be because of some network devices in between. It is highly possible that the re-INVITE/response does not get to one of the sides due to SIP inspection or network settings.

If the endpoints do not initiate the re-INVITE, it could be a problem with the endpoint. Involve Cisco Technical Assitance Center (TAC) in order to investigate further.

Problem: H.323 Call Drops after any Specific Time

As with SIP, in H.323 calls call drops at a specific time interval occur usually due to network or firewall timeout configuration.

Solution

In H.323 calls, a Round Trip Delay Request (RTDR) message is sent every 30 seconds between endpoints along with sequence numbers . For each request, a response is expected.

Cisco Endpoint utilizes the RTDR/Round Trip Delay Response message, which is a part of the H.245 Multimedia System Control Message. This keps the H.245 TCP session alive during the call which is used for active call management. If the endpoint receives a response for RTDR initially and no response is received during the call, the endpoint terminates the call.

In this scenario, collect the H.323 debug logs and Endpoint Logs in order to isolate the issue. From the H.323 debug logs, check for the RTDR request and response messages and find out if it drops.

In this example output, the endpoint sends an RTDR request to the remote endpoint and it does not receive a response from the remote end. It therefore disconnects the call:

 014-09-23T21:37:01+10:00 corevcs1 tvcs: UTCTime="2014-09-23 11:37:01,
 711"Module="network.H.323" Level="DEBUG":  Dst-ip="10.0.20.11"  
 Dst-port="11012" Sending H.245 PDU: value MultimediaSystemControlMessage
 ::= request : roundTripDelayRequest : {   sequenceNumber 120

The requests and responses can be tracked with sequenceNumbers.

This example from the endpoint logs shows the cause for the disconnection:

2977610.83 H.323Call I: H.323_call_handler::handleDiscInd(p=349, s=1) 
Received disconnectindication (Cause: 12:18, H.323 cause: 3:18)- 
NetworkRejected Q85012977610.84 MC I: RemoteParticipant::
reevalRefMode(p=349,ch=2) set ref [Video (2): vid-off0x0@0.0  0k ] 
q= auto, t60=600012977610.84 ModesController I: ModesController::
resetRateLimit(ch=2)12977610.84 MC I: RemoteParticipant::modeChanged
(p=349, ch=2): ModesController wants torun mode: Video (2): vid-off 0x0@0.0  0k

Problem: Call failure due to Media Resource Allocation Failure

In the case of video calls, calls that fail due to a Media Resource alocation failure are seen. For example, if the calling and called endpoint do not support a common codec then a transcoder is required , for a Dual Tone Multi Frequency (DTMF) mismatch a Media Termination Point (MTP) is required on the Call Manager.

Solution

For video transcoding, a Packet Voice Digital Module (PVDM3) Digital Signal Processor (DSP) transcoder is required as transcoders on PVDM2 do not support video. If a transcoder/MTP is not available, a 503 Service unavailable message would be sent to the endpoint:

SIP/2.0 503 Service UnavailableVia: SIP/2.0/TCP 10.101.15.13:
5060;branch=z9hG4bK954956da2012413dfb6ef80d6bc9e373.1;rportFrom:
 <sip:3550@10.102.254.4>;tag=47c4717d0db85e1aTo:
 <sip:1281@10.102.254.4>;tag=176803~66dd1c7a-eac9-42af-a69b-
18da1695a800-31478649Date: 
Wed, 19 Feb 2014 16:10:05 GMTCall-ID: 
c05df2acedcafd063eb5cf947ebc1efcCSeq: 100 INVITEAllow-Events: 
presenceReason: Q.850;cause=47Content-Length: 0 

In order to resolve this, check the Media Resource Group/Media Resource Group List (MRG/MRGL) configuration and ensure video transcoder/MTP is available. An MRGL can be assigned to a device at the Phone Level or the Device Pool Level:

1. On the CallManger choose Device > Phone and select the device that has the problem and check the Device Pool and the MRGL settings:

   

2. If the MRGL setting on the phone is None, then you have to check the Device pool settings to make sure there is a transcoder.
3. Choose System > Device Pool and select the device pool assigned to the device:

   

4. Choose Media Resources > Media Resource Group List and select the MRGL assigned at the phone level / Device pool level and check the MRGs:

   

5. Note the MRGs and choose Media Resources > Media Resource Group and select the noted MRGs. Ensure you have a PVDM3 hardware transcoder/MTP added.

   

Problem: Call Failures due to Insufficient Bandwidth

More often than not there are scenarios where a call gets disconnected because of insufficient bandwidth configuration in Region/Location on the device in CUCM. When the region is set to a low bandwidth that the endpoint cannot support, CallManager sends a "488 Not Acceptable Media" with cause 125 which means "Out of Bandwidth" or "Insufficient Bandwidth" after the SIP media negotiation happens.

You need to caputure the SIP logs on the endpoint as described and look for this message:

1459.81 SipPacket I: PacketDump: Proto: SIP, Direction: Incoming, Name: 488 
Not Acceptable Media, CSeq: 100 INVITE, RemoteAddress: 10.106.85.219:5060, 
CallId: 207b6ddb148ddf900ae2e2f844115837, Time: 1459811
	1459.81 SipPacket   SIP/2.0 488 Not Acceptable Media
	1459.81 SipPacket   Via: SIP/2.0/TCP 10.106.85.231:56280;
branch=z9hG4bK64e2eb4a1a3afd5f956a1547eb1c05ad.1;rport
	1459.82 SipPacket   Call-ID: 207b6ddb148ddf900ae2e2f844115837
	1459.82 SipPacket   CSeq: 100 INVITE
	1459.82 SipPacket   From: <sip:4657@example.com>;tag=2d98ee2065ba492d
	1459.82 SipPacket   To: <sip:1112@10.106.85.219>;
tag=10543~8c84fc84-78bb-de4d-3ac7-da2a9cab63d5-19683975
	1459.83 SipPacket   Server: Cisco-CUCM10.5
	1459.83 SipPacket   Date: Sun, 07 May 2015 14:36:41 GMT
	1459.83 SipPacket   Allow-Events: presence
	1459.83 SipPacket   Warning: 370 10.106.85.219 "Insufficient Bandwidth"
	1459.83 SipPacket   Reason: Q.850 ;cause=125
	1459.83 SipPacket   Content-Length: 0
	1459.83 SipPacket   
	1459.83 SipStack I: SipDialog(ui=3,s=9) sendInviteRejToStack (488:Not Acceptable Media)
	1459.84 SipCall I: sip_call_handler::handleSIPMCallRej(3/9/-1): Call rejected
 (cause: Not Acceptable Media)
	1459.84 MainEvents I: CallDisconnectRequested(p=3) remoteURI='sip:1112@10.106.85.219'
 cause=[normal('') 'LocalDisconnect']
	1459.84 MainEvents I: ParticipantLeftConference(c=2,p=3)
	1459.85 APPL_Media ERROR: AudioCtrlImpl::execute_disconnectInputOutput 
No mixer for (p=1,ch=61)
	1459.85 MainEvents I: CallDisconnected(p=3) remoteURI='sip:1112@10.106.85.219' 
causeToLocal=[disconnected('Not Acceptable Media') 'RemoteDisconnect'] 
causeToRemote=[normal('') 'LocalDisconnect']

Solution

If this issue happens, check the Region configured on both the endpoints and check the Region relationship between them:

1. Choose Device > Phone and select both devices. Check the Device Pool assigned to the devices:

   

2. Once you check the Device Pool, choose System > Device Pool on the CUCM and check the Region configured on both Device Pools:

   

3. Choose System > Region Information > Regions and check the Region Relationship. Check the audio video bandwidth on the region and ensure that the endpoint can operate at the audio/video bandwidth as selected:

   

In the above screenshots it is assumed that one endpoint is in the Region "Trunk Region" and the other is in the "Local Endpoints Region".

Another workaround is to try the video call as an Audio Call if bandwidth for the video call bandwidth is insufficient. Use this procedure in order to check and configure:

1. Choose Device > Phone and select the calling Device with the problem. Check if the parameter in this screenshot is checked. If it is unchecked, check it so that a video call falls back to audio in case of bandwidth issues:

   

   This issue could happen because of Location Settings on the CallManager .

   Locations can be assigned at the Phone Level or at the Device Pool level (Phone Level takes higher priority).

2. In order to check Phone Level location settings, choose Devices > Phones and check the location on both calling and called endpoint:

   

   The location can also be applied at the device pool level. Therefore, first check the device pool of both endpoints:

   

3. Choose System > Device Pool. On the device pool, check the location assigned on both the calling and called endpoints. In this example no location is assigned at the device pool level. The phone location configuration is used:

   

4. Check if sufficient bandwidth is configured between the calling and called endpoints location. In this example one endpoint is assumed to be in Local Endpoints location and the other one is in the Hub_None location and bandwidth for audio / video and immersive calls are all configured as Unlimited:

   

There could be other reasons for disconnection. See page 178 of Cisco Unified Communications Manager Call Detail Records Administration Guide, Release 10.0(1) for Disconnection cause codes.