Troubleshoot Connectivity Issues Between CVP OAMP Server And CVP Call Server

Introduction

This document describes how to troubleshoot connectivity issues between Cisco Voice Portal (CVP) Operation Console (OAMP) server with CVP call server. 

Here is the list of common connectivity issues between these two servers:

1.  What does it happen when Save and Deploy button pressed in CVP OAMP page?

2.  What components do we need to verify if OAMP server can't reach CVP call server?

3.  Which files are changed in re-image process?

Contributed by Mingze Yan, Cisco TAC Engineer.

Edited by Sahar Modares, Cisco TAC Engineer.

Prerequisties

Requirements

Cisco recommends that you have knowledge of these topics:

• CVP

Components Used

• CVP Call Server 11.5
• CVP OAMP 11.5

What does it happen when Save and Deploy button pressed in CVP OAMP page?

In order to understand the what happens when Save and Deploy button is pressed, you need to see what is shown when new call server was installed. After CVP Call Server is installed, there is no CVP subsystem defined and OAMP server does not allocate any message bus.

In order to verify the configuration, navigate to \Cisco\CVP\conf\system.properties file. The file does not include subsystem information.

CVPServlet.upgradeProperties = false /
#msg.sendType = NEW_CALL
#msg.sendTopic = CVP.ICM.CC.RESP.SYS_SIP1
#stub.ipaddress = localhost
#stub.sendport  = 6000
#stub.readport  = 7000
CVPServlet.upgradeProperties = false

From same folder, there is no plugin defined in the file MessageAdapter.properties.

#MessageAdapter.MsgBus001.filter.stubFilter1 = CVP.SIP.CC.*
##--------------- STUB SS (SYS_STUB1) MESSAGE BUS SIDE PLUGIN -------------------
#MessageAdapter.MsgBus001.plugin.MSGBUS_STUB1.class = com.cisco.msg.msgbus.inprocess.InProcessPlugin
#MessageAdapter.MsgBus001.plugin.MSGBUS_STUB1.param.pluginType = IN_PROCESS
#MessageAdapter.MsgBus001.plugin.MSGBUS_STUB1.param.adapterName = MsgBus001
#MessageAdapter.MsgBus001.plugin.MSGBUS_STUB1.param.adapterType = MessageBus
#MessageAdapter.MsgBus001.plugin.MSGBUS_STUB1.param.mirrorPluginName = SYS_STUB1
#MessageAdapter.MsgBus001.plugin.MSGBUS_STUB1.filter.exclude.1 = filterAll
#MessageAdapter.MsgBus001.plugin.MSGBUS_STUB1.filter.include.1 = stubFilter1

 

##--------------- STUB SS (SYS_STUB1) PLUGIN -------------------
#MessageAdapter.STUB1.active = true
#MessageAdapter.STUB1.plugin.SYS_STUB1.class = com.cisco.msg.msgbus.inprocess.InProcessPlugin
#MessageAdapter.STUB1.plugin.SYS_STUB1.param.pluginType = IN_PROCESS
#MessageAdapter.STUB1.plugin.SYS_STUB1.param.adapterName = STUB1
#MessageAdapter.STUB1.plugin.SYS_STUB1.param.mirrorPluginName = MSGBUS_STUB1
#MessageAdapter.STUB1.plugin.SYS_STUB1.param.adapterType = STUB

From CVP logs, this is what it shows when there is no subsystem defined:

16: 10.66.75.101: Jul 07 2017 09:28:27.726 +1000: %CVP_11_5_Infrastructure-0-START:  No Subsystems have been started.  Have you configured your server yet? ; Make sure <CVP_HOME>\conf\propFileList.properties is loading "system" and that <CVP_HOME>\conf\system.properties lists at least one subsystem to load. [id:1033]

When CVP Call server is configured in OAMP GUI page, these changes happen:

1. The OAMP server pushes message bus information to CVP call server, also adds subsystem and plug-in.

2. These files are modified after the changes were applied:

3. From system.properties file, message bus is allocated by OAMP server, which is globally significant. Each CVP call server uses different message bus to communicate with OAMP:

CVPServlet.upgradeProperties = false
MessagingLayer.adapterName = MsgBus003
CVPServlet.subsystem.0.Class = com.cisco.cvp.central.CVPController
CVPServlet.subsystem.0.ID = CVPCTL3
CVPServlet.subsystem.1.Class = com.cisco.cvp.icm.ICMSubsystem
CVPServlet.subsystem.1.ID = ICM3
CVPServlet.subsystem.2.Class = com.cisco.cvp.sip.SipSubsystem
CVPServlet.subsystem.2.ID = SIP3
CVPServlet.subsystem.3.Class = com.cisco.cvp.ivr.IVRSubSystem
CVPServlet.subsystem.3.ID = IVR3
cmdManager.timeout = 10000
cmdManager.commandPrefix = cmd /c
cmdManager.checkServiceCmd = net start
cmdManager.cmd_separator = &
cmdManager.expected = expected

4. From ORM.Properties file, control adapterName is added:

#For http client
orm.org.apache.logLevel = ERROR
httpclient.logLevel = ERROR
orm.controller.adapterName = CVPCTL3

5. Inside messageAdapter.properties file, plugins is added, and port numbers are allocated to each plugin.

# --------------- ORM OUTPROCESS PLUGIN -------------------
MessageAdapter.ORM.active = true
MessageAdapter.ORM.plugin.SYS_ORM.class = com.cisco.msg.msgbus.outprocess.OutProcessClientPlugin
MessageAdapter.ORM.plugin.SYS_ORM.param.pluginType = OUT_PROCESS
MessageAdapter.ORM.plugin.SYS_ORM.param.adapterName = ORM
MessageAdapter.ORM.plugin.SYS_ORM.param.adapterType = OAMP
MessageAdapter.ORM.plugin.SYS_ORM.param.primaryServerHostname = 10.66.75.101
MessageAdapter.ORM.plugin.SYS_ORM.param.primaryServerPort = 23000
MessageAdapter.ORM.plugin.SYS_ORM.param.socketReadTimeout = 0
MessageAdapter.ORM.plugin.SYS_ORM.param.waitForApplication = 3
MessageAdapter.ORM.plugin.SYS_ORM.param.mirrorPluginName = MSGBUS_ORM3

6. In wrapper.conf, Java addition setting rmi.server is added:

# Java Additional Parameters
wrapper.java.additional.1= -Djavax.net.ssl.keyStore=C:/Cisco/CVP/conf/security/.ormKeystore
wrapper.java.additional.2= -Djavax.net.ssl.keyStorePassword=<NEED_TO_MODIFY>
wrapper.java.additional.3= -Djavax.net.ssl.keyStoreType=JCEKS
wrapper.java.additional.4= -Djavax.net.ssl.trustStore=C:/Cisco/CVP/conf/security/.keystore
wrapper.java.additional.5= -Djavax.net.ssl.trustStorePassword=<NEED_TO_MODIFY>
wrapper.java.additional.6= -Djavax.net.ssl.trustStoreType=JCEKS
wrapper.java.additional.7= -Dcom.sun.management.config.file=../conf/orm_jmx.properties
wrapper.java.additional.8= -Dccbu.logging.config.file=log4j_orm.xml
wrapper.java.additional.9= -Djava.rmi.server.hostname=10.66.75.101

What components do we need to verify if OAMP server can't reach CVP call server?

When CVP configuration is saved from OAMP, this error message can be displayed:

"Device could not be reached because of connection timeout"

In order to connect OAMP to CVP call server, OAMP needs to communicate with two ports in CVP call sever, which are 2099 (fixed port) and a random rmi port (1024- 65535).

Use jconsole to test connectivity (C:\Cisco\CVP\jre\bin\jconsole.exe):

If the test fails with this error message "Connection Failed:Retry?", these two ports must be checked:

1. Port number 2099
2. The rmi random port. (which in our test bed OS allocated port number 1042).

Here is the error message when port 1042 is blocked:

Here is the packet capture when jconsole test works fine:

Because of this random port, CVP Operation console server can't be deployed outside of a firewall while all other Unified CVP components reside inside the firewall.

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/contact_center/customer_voice_portal/srnd/11_0/CCVP_BK_11366AC9_00_1101-design-guide/CCVP_BK_11366AC9_00_1101-design-guide_chapter_01100.html

The behavior was described in this defect:

CSCve32007 -Firewall in between Operations Console Server and CVP Components

In order to find rmi random number, use these steps:

Step1. Use 2099 to find PID number.

Step 2. Use the PID number to find out all the ports which are used.

Example:

Note: The connection between OAMP and call server is bi-directional. Same command can be used for local ports on OAMP servers.

Which files are changed in re-image process?

After reimage process, four files are changed:

• messageAdapter.properies
• orm.propeties
• orm.xml
• system.propeties

Here are the changes for these four files:

• In messageAdapter.properies, plugins that was added in previous section are deleted
• In orm.propeties, orm.controller.adapterName = CVPCTL3 is removed
• In orm.xml, all subsystem added in previous section are deleted
• In system.propeties, the subsystem was added in previous section is deleted
  

If you click save and deployment from OAMP page, CVP call server isn't updated, as OAMP still in use with the message bus ID communicate with Call Server, but call server has no such message bus information.

After CVP call server deletion from OAMP, three files are changed:

• messageAdaptor
• Orm 
• System

• messageAdaptor file is empty
  
• Orm adds this line orm.controller.adapterName = 
  
• System file adds this line MessagingLayer.adapterName =

When you re-add the CVP call server in OAMP, then the configuration is updated in the CVP call server.

 The updated files are almost the same as first time deployment, except for the file orm_jmx.

javax.net.debug = all
com.sun.management.jmxremote.ssl.need.client.auth = false
com.sun.management.jmxremote.authenticate = false
com.sun.management.jmxremote.port = 2099
com.sun.management.jmxremote.ssl = false

So, re-image deletes message bus information, which causes OAMP can't update CVP call server. Therefore, Call server must be deleted to allow Call server to accept new message bus allocation. 

Attached configuration files and packet capture for reference.