Contents

Introduction

Prerequisites

Requirements

Components Used

Problem: HCM-F Administrator is not able to assign the UC cluster to the PLM instance

Solution 1

Verify

Solution 2

Verify

Solution 3

Verify

Solution 4

Verify

Solution 5

Verify

Problem: HCM-F Administrator is not able to assign the UC cluster to the PLM instance

Solution

Verify

Introduction

This document describes general troubleshooting steps for Hosted License Manager (HLM) and Prime License Manager (PLM) while adding a new Unified Application product instance into PLM via HCM-F. Moreover, perform the checklist before you open a TAC case.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Cisco Hosted Collaboration Solution (HCS)

- Cisco Hosted Collaboration Mediation Function (HCM-F)

- Cisco Prime License Manager (PLM)

Components Used

The information in this document is based on these software and hardware versions:

- HCM-F 10.6.2

- PLM 10.5.2

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Problem: HCM-F Administrator is not able to assign the UC cluster to the PLM instance

The HCM-F Job fails with the error shown in the image:

In order to troubleshoot this issue, HLM detailed logs are required as several conditions can lead to the same GUI error.

Log in into the HCM-F, CLI enables the logs level and collects the logs.

Run the command:

set trace tracelevel

Enter the tracelevel(use CLI "show tracelevels" to find allowed tracelevels) ::                              Detailed

Enter the servicename :: Cisco HCS License Manager Service

Reproduce the issue and collect the logs.

file get activelog /hcs/HLM/*

Solution 1

Add the Service provider IP address in the Cluster application Publisher. The  HCM-F can't connect to the UC application  because the IP service provider is missing.

2016-09-21 18:25:21,659 INFO  [264674] UCAppDeploymentModeConnection: Opening secure connection to: https://null:8443/platform-services/services/DeploymentModeService?wsdl

2016-09-21 18:25:21,659 INFO  [264674] UCAppDeploymentModeConnectionPort successfully opened

2016-09-21 18:25:21,688 INFO  [264674] UCAppDeploymentModeConnection: Setting userName for SOAP connection to: ccmadmin

2016-09-21 18:25:21,688 INFO  [264674] UCAppDeploymentModeConnection: Accepting Certificate from UC Application

2016-09-21 18:25:21,688 INFO  [264674] UCAppDeploymentModeConnection.getDeploymentMode Called

2016-09-21 18:25:21,695 ERROR [264674] UCAppAPI.getDeploymentMode() CaughtException: Error writing to XMLStreamWriter.

2016-09-21 18:25:21,695 ERROR [264674] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest  Error writing to XMLStreamWriter.

2016-09-21 18:25:21,696 DEBUG [264674] hlmSDRUtil.updateJob

2016-09-21 18:25:21,750 DEBUG [264674] hlmSDRUtil.beginTransaction

Verify

In order to verify, navigate to Application Management > Cluster Application

Solution 2

Enable the IP connectivity between HCM-F and UC application on port 8443.

The HCM-F configuration is fine but HLM module can't open a tcp connection toward the UC application. Please refer to the Verify section how troubleshoot further this issue

2016-09-21 18:34:33,745 INFO  [264677] UCAppDeploymentModeConnection: Opening secure connection to: https://10.48.50.72:8443/platform-services/services/DeploymentModeService?wsdl

2016-09-21 18:34:33,746 INFO  [264677] UCAppDeploymentModeConnectionPort successfully opened

2016-09-21 18:34:33,764 INFO  [264677] UCAppDeploymentModeConnection: Setting userName for SOAP connection to: ccmadmin

2016-09-21 18:34:33,764 INFO  [264677] UCAppDeploymentModeConnection: Accepting Certificate from UC Application

2016-09-21 18:34:33,764 INFO  [264677] UCAppDeploymentModeConnection.getDeploymentMode Called

Failed to connect :

2016-09-21 18:35:09,801 ERROR [264677] UCAppAPI.getDeploymentMode() CaughtException: Error writing to XMLStreamWriter.

2016-09-21 18:35:09,801 ERROR [264677] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest  Error writing to XMLStreamWriter.

2016-09-21 18:35:09,801 DEBUG [264677] hlmSDRUtil.updateJob

2016-09-21 18:35:09,856 DEBUG [264677] hlmSDRUtil.beginTransaction

2016-09-21 18:35:09,859 DEBUG [264677] hlmSDRUtilcommitTransaction

2016-09-21 18:35:09,990 INFO  [264677] AgentFSM::currentState changed from null0 to stopped

2016-09-21 18:35:09,990 INFO  [264677] Agent route no longer needed...moving it to stopped routes

2016-09-21 18:35:09,990 INFO  [264677] Route removed from context >HLM(be75ce6c-f4ca-464e-aa0f-8204f6cb1fa3)<

Verify

- Verify the UC application is up and running.

- Verify the connectivity between UC app and HCM-F.

admin:utils network ping XXX.YYY.WWW.ZZZ

- Verify port 8443 is not blocked by firewall. Collect the packet capture and verify the TCP hand shake is successfully established.

admin:utils network capture eth0 file filename coount 10000 size all port 8443 host ip xxx.yyy.www.zzz

where xxx.yyy.www.zzz is the UC app service provide space

- Verify the below link is accessible.

https://UC_APP_IP_ADDRRESS:8443/platform-services/services/DeploymentModeService?wsdl

Solution 3

- Re type the platform UC application password in order to match with the UC application one because HLM can't send the message.

- Start the Platform Administrator web service in the UC application

2016-10-06 16:24:49,592 INFO  [252833] UCAppDeploymentModeConnection: Opening secure connection to: https://10.48.50.14:8443/platform-services/services/DeploymentModeService?wsdl

2016-10-06 16:24:49,592 INFO  [252833] UCAppDeploymentModeConnectionPort successfully opened

2016-10-06 16:24:49,616 INFO  [252833] UCAppDeploymentModeConnection: Setting userName for SOAP connection to: ccmadmin

2016-10-06 16:24:49,616 INFO  [252833] UCAppDeploymentModeConnection: Accepting Certificate from UC Application

2016-10-06 16:24:49,616 INFO  [252833] UCAppDeploymentModeConnection.getDeploymentMode Called

2016-10-06 16:24:51,877 ERROR [252833] UCAppAPI.getDeploymentMode() CaughtException: Could not send Message.

2016-10-06 16:24:51,878 ERROR [252833] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest  Could not send Message.

2016-10-06 16:24:51,878 DEBUG [252833] hlmSDRUtil.updateJob

2016-10-06 16:24:51,939 DEBUG [252833] hlmSDRUtil.beginTransaction

2016-10-06 16:24:51,942 DEBUG [252833] hlmSDRUtilcommitTransaction

2016-10-06 16:24:52,054 INFO  [252833] AgentFSM::currentState changed from null0 to stopped

2016-10-06 16:24:52,054 INFO  [252833] Agent route no longer needed...moving it to stopped routes

2016-10-06 16:24:52,054 INFO  [252833] Route removed from context >HLM(ad495160-864d-4b74-8aec-cc708036b5e0)<

2016-10-06 16:24:52,054 DEBUG [252833]

Verify

- Verify the Platform Administrative Web service is running on the UC applications

Navigate to Cisco Unified Serviceability > Platform Service > Platform Administrative Web Service

- Verify the Platform Administrative Web service replies back while accessing to the wsdl file

https://UC_APP_IP_ADDRRESS:8443/platform-services/services/DeploymentModeService?wsdl

Solution 4

Contact cisco TAC in order to change back the PLM deployment mode into HCS

2016-10-07 11:37:55,581 INFO  [238120] UCAppDeploymentModeConnection: Opening secure connection to: https://10.48.52.10:8443/platform-services/services/DeploymentModeService?wsdl

2016-10-07 11:37:55,581 INFO  [238120] UCAppDeploymentModeConnectionPort successfully opened

2016-10-07 11:37:55,608 INFO  [238120] UCAppDeploymentModeConnection: Setting userName for SOAP connection to: ccmadmin

2016-10-07 11:37:55,608 INFO  [238120] UCAppDeploymentModeConnection: Accepting Certificate from UC Application

2016-10-07 11:37:55,608 INFO  [238120] UCAppDeploymentModeConnection.getDeploymentMode Called

2016-10-07 11:37:55,980 DEBUG [238120] UCAppDeploymentModeConnection.getDeploymentMode Response Recieved

2016-10-07 11:37:55,980 INFO  [238120] UCAppDeploymentModeConnection.getDeploymentMode Completed returning value: Enterprise

2016-10-07 11:37:55,980 INFO  [238120] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest  Found current UC app deployment mode to be: Enterprise

2016-10-07 11:37:55,980 INFO  [238120] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest  Found current Global deployment mode to be: HCS

2016-10-07 11:37:55,980 INFO  [238120] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest  Setting the UC app deployment mode to be: HCS

2016-10-07 11:37:55,981 INFO  [238120] UCAppDeploymentModeConnection: Opening secure connection to: https://10.48.52.10:8443/platform-services/services/DeploymentModeService?wsdl

2016-10-07 11:37:55,981 INFO  [238120] UCAppDeploymentModeConnectionPort successfully opened

2016-10-07 11:37:55,993 INFO  [238120] UCAppDeploymentModeConnection: Setting userName for SOAP connection to: ccmadmin

2016-10-07 11:37:55,993 INFO  [238120] UCAppDeploymentModeConnection: Accepting Certificate from UC Application

2016-10-07 11:37:55,993 INFO  [238120] UCAppDeploymentModeConnection.setDeploymentMode Called

2016-10-07 11:37:56,183 DEBUG [238120] UCAppDeploymentModeConnection.setDeploymentMode Response Recieved

2016-10-07 11:37:56,189 INFO  [238120] UCAppRestartSystemConnection: Opening secure connection to: https://10.48.52.10:8443/platform-services/services/RestartSystemService?wsdl

2016-10-07 11:37:56,205 INFO  [238120] UCAppRestartSystemConnection: Setting userName for SOAP connection to: ccmadmin

2016-10-07 11:37:56,205 INFO  [238120] UCAppRestartSystemConnection: Accepting Certificate from UC Application

2016-10-07 11:54:12,248 INFO  [238120] ELMApi: initialized, connected to plm-hcs10

2016-10-07 11:54:12,248 DEBUG [238120] ELMApi: createCluster called - force registration is enabled.

2016-10-07 11:54:12,248 DEBUG [238120] ELMApi:Creating Cluster with clusterName: test-cluster-Z|HostName: 10.48.52.10|userid: ccmadmin|password: ******|Type: HUCM

2016-10-07 11:54:12,360 DEBUG [238120] ELMApi: createCluster  Response status = 400

2016-10-07 11:54:12,360 ERROR [238120] ELMApi: createCluster completes with an error. LM resp status: 400

2016-10-07 11:54:12,360 ERROR [238120] HLMClusterLicenseAgent:  Failed to assign cluster test-cluster-Z due to an internal error in LM plm-hcs10

2016-10-07 11:54:12,360 DEBUG [238120] hlmSDRUtil.updateJob

2016-10-07 11:54:12,412 DEBUG [238120] hlmSDRUtil.beginTransaction

2016-10-07 11:54:12,415 DEBUG [238120] hlmSDRUtilcommitTransaction

2016-10-07 11:54:12,538 INFO  [238120] AgentFSM::currentState changed from null0 to stopped

2016-10-07 11:54:12,539 INFO  [238120] Agent route no longer needed...moving it to stopped routes

2016-10-07 11:54:12,539 INFO  [238120] Route removed from context >HLM(8032fcf3-9368-4644-a46f-46445247e893)<

2016-10-07 11:54:12,539 DEBUG [238120] +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

2016-10-07 11:54:12,539 DEBUG [238120]  Agent info : HLM(8032fcf3-9368-4644-a46f-46445247e893): RED

Assign test-cluster-Z to plm-hcs10 is Failed

Verify

Verify the PLM deployment mode by browsing the below URL. PLM platform user id and password is required

https://PLM_IP_Address /elm-resources/

Solution 5

Enable the IP connectivity between HCM-F and PLM on port 8443. From logs HLM can't connect to the PLM

2016-10-18 12:17:47,119 DEBUG [264677] ELMApi:Creating Cluster with clusterName: cluster delta|HostName: 10.48.55.26|userid: administrator|password: ******|Type: HUCM

2016-10-18 12:17:47,904 DEBUG [43] KeepAliveConsumerProcessor::process -- enter

2016-10-18 12:17:47,905 DEBUG [43] KeepAliveConsumerProcessor::process -- received broadcast message for service sdrcnf

2016-10-18 12:17:47,905 DEBUG [43] noChange -- sdrcnf is Alive

2016-10-18 12:17:47,905 DEBUG [43] KeepAliveMonitor::setExpiresBy:

2016-10-18 12:17:47,905 DEBUG [43]   now:         18/10/2016 12:17:47.905

2016-10-18 12:17:47,905 DEBUG [43]   expected by: 18/10/2016 12:19:47.905

2016-10-18 12:17:47,905 DEBUG [71] KeepAliveConsumerProcessor::process -- enter

2016-10-18 12:17:47,906 DEBUG [71] KeepAliveConsumerProcessor::process -- received broadcast message for service sdrcnf

2016-10-18 12:17:47,906 DEBUG [71] noChange -- sdrcnf is Alive

2016-10-18 12:17:47,906 DEBUG [71] KeepAliveMonitor::setExpiresBy:

2016-10-18 12:17:47,906 DEBUG [71]   now:         18/10/2016 12:17:47.906

2016-10-18 12:17:47,906 DEBUG [71]   expected by: 18/10/2016 12:19:47.906

2016-10-18 12:17:50,188 ERROR [264677] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest  javax.ws.rs.WebApplicationException: javax.xml.bind.MarshalException

- with linked exception:

[java.net.NoRouteToHostException: No route to host]

2016-10-18 12:17:50,188 DEBUG [264677] hlmSDRUtil.updateJob

2016-10-18 12:17:50,247 DEBUG [264677] hlmSDRUtil.beginTransaction

2016-10-18 12:17:50,249 DEBUG [264677] hlmSDRUtilcommitTransaction

2016-10-18 12:17:50,356 INFO  [264677] AgentFSM::currentState changed from null0 to stopped

2016-10-18 12:17:50,356 INFO  [264677] Agent route no longer needed...moving it to stopped routes

Verify

Verify the connectivity between PLM and HCM-F

admin:utils network ping XXX.YYY.WWW.ZZZ

Problem: HCM-F Administrator is not able to assign the UC cluster to the PLM instance

The job fails with the error shown in the image

;

Solution

Add Platform credential either on HCM-F or into CUCDM

2016-10-06 16:26:48,508 DEBUG [264674] hlmSDRUtilcommitTransaction

2016-10-06 16:26:48,624 ERROR [264674] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest  no platform credential is configured for test-cluster-Y

2016-10-06 16:26:48,624 DEBUG [264674] hlmSDRUtil.updateJob

2016-10-06 16:26:48,675 DEBUG [264674] hlmSDRUtil.beginTransaction

2016-10-06 16:26:48,677 DEBUG [264674] hlmSDRUtilcommitTransaction

2016-10-06 16:26:48,798 INFO  [264674] AgentFSM::currentState changed from null0 to stopped

2016-10-06 16:26:48,798 INFO  [264674] Agent route no longer needed...moving it to stopped routes

2016-10-06 16:26:48,798 INFO  [264674] Route removed from context >HLM(1b32a922-3619-44b0-8004-7f6fc930c419)<

2016-10-06 16:26:48,798 DEBUG [264674] +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

2016-10-06 16:26:48,798 DEBUG [264674]  Agent info : HLM(1b32a922-3619-44b0-8004-7f6fc930c419): RED

Assign test-cluster-Y to plm-hcs10 is Failed

2016-10-06 16:26:48,798 DEBUG [264674]  AgentMessageProcessor::Agent is stopped after process, removed it from persistence store

Verify

Verify the Platform credential are configured into HCM-F

Navigate to  Application Management > Cluster Application > Credential > Platform

or

Navigate to Administration > Default Credential > UC application type ( CUCM/CUCXN) Platform