The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes general troubleshooting steps for Hosted License Manager (HLM) and Prime License Manager (PLM) while adding a new Unified Application product instance into PLM via HCM-F. Moreover, perform the checklist before you open a TAC case.
Cisco recommends that you have knowledge of these topics:
The information in this document is based on these software and hardware versions:
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
The HCM-F Job fails with the error shown in the image:
In order to troubleshoot this issue, HLM detailed logs are required as several conditions can lead to the same GUI error.
Log in into the HCM-F, CLI enables the logs level and collects the logs.
Run the command:
set trace tracelevel
Enter the tracelevel(use CLI "show tracelevels" to find allowed tracelevels) :: Detailed
Enter the servicename :: Cisco HCS License Manager Service
Reproduce the issue and collect the logs.
file get activelog /hcs/HLM/*
Add the Service provider IP address in the Cluster application Publisher. The HCM-F can't connect to the UC application because the IP service provider is missing.
2016-09-21 18:25:21,659 INFO [264674] UCAppDeploymentModeConnection: Opening secure connection to: https://null:8443/platform-services/services/DeploymentModeService?wsdl 2016-09-21 18:25:21,659 INFO [264674] UCAppDeploymentModeConnectionPort successfully opened 2016-09-21 18:25:21,688 INFO [264674] UCAppDeploymentModeConnection: Setting userName for SOAP connection to: ccmadmin 2016-09-21 18:25:21,688 INFO [264674] UCAppDeploymentModeConnection: Accepting Certificate from UC Application 2016-09-21 18:25:21,688 INFO [264674] UCAppDeploymentModeConnection.getDeploymentMode Called 2016-09-21 18:25:21,695 ERROR [264674] UCAppAPI.getDeploymentMode() CaughtException: Error writing to XMLStreamWriter. 2016-09-21 18:25:21,695 ERROR [264674] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest Error writing to XMLStreamWriter. 2016-09-21 18:25:21,696 DEBUG [264674] hlmSDRUtil.updateJob 2016-09-21 18:25:21,750 DEBUG [264674] hlmSDRUtil.beginTransaction
In order to verify, navigate to Application Management > Cluster Application
Enable the IP connectivity between HCM-F and UC application on port 8443.
The HCM-F configuration is fine but HLM module can't open a tcp connection toward the UC application. Please refer to the Verify section how troubleshoot further this issue
2016-09-21 18:34:33,745 INFO [264677] UCAppDeploymentModeConnection: Opening secure connection to: https://10.48.50.72:8443/platform-services/services/DeploymentModeService?wsdl 2016-09-21 18:34:33,746 INFO [264677] UCAppDeploymentModeConnectionPort successfully opened 2016-09-21 18:34:33,764 INFO [264677] UCAppDeploymentModeConnection: Setting userName for SOAP connection to: ccmadmin 2016-09-21 18:34:33,764 INFO [264677] UCAppDeploymentModeConnection: Accepting Certificate from UC Application 2016-09-21 18:34:33,764 INFO [264677] UCAppDeploymentModeConnection.getDeploymentMode Called Failed to connect : 2016-09-21 18:35:09,801 ERROR [264677] UCAppAPI.getDeploymentMode() CaughtException: Error writing to XMLStreamWriter. 2016-09-21 18:35:09,801 ERROR [264677] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest Error writing to XMLStreamWriter. 2016-09-21 18:35:09,801 DEBUG [264677] hlmSDRUtil.updateJob 2016-09-21 18:35:09,856 DEBUG [264677] hlmSDRUtil.beginTransaction 2016-09-21 18:35:09,859 DEBUG [264677] hlmSDRUtilcommitTransaction 2016-09-21 18:35:09,990 INFO [264677] AgentFSM::currentState changed from null0 to stopped 2016-09-21 18:35:09,990 INFO [264677] Agent route no longer needed...moving it to stopped routes 2016-09-21 18:35:09,990 INFO [264677] Route removed from context >HLM(be75ce6c-f4ca-464e-aa0f-8204f6cb1fa3)<
admin:utils network ping XXX.YYY.WWW.ZZZ
admin:utils network capture eth0 file filename coount 10000 size all port 8443 host ip xxx.yyy.www.zzz
where xxx.yyy.www.zzz is the UC app service provide space
https://UC_APP_IP_ADDRRESS:8443/platform-services/services/DeploymentModeService?wsdl
2016-10-06 16:24:49,592 INFO [252833] UCAppDeploymentModeConnection: Opening secure connection to: https://10.48.50.14:8443/platform-services/services/DeploymentModeService?wsdl 2016-10-06 16:24:49,592 INFO [252833] UCAppDeploymentModeConnectionPort successfully opened 2016-10-06 16:24:49,616 INFO [252833] UCAppDeploymentModeConnection: Setting userName for SOAP connection to: ccmadmin 2016-10-06 16:24:49,616 INFO [252833] UCAppDeploymentModeConnection: Accepting Certificate from UC Application 2016-10-06 16:24:49,616 INFO [252833] UCAppDeploymentModeConnection.getDeploymentMode Called 2016-10-06 16:24:51,877 ERROR [252833] UCAppAPI.getDeploymentMode() CaughtException: Could not send Message. 2016-10-06 16:24:51,878 ERROR [252833] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest Could not send Message. 2016-10-06 16:24:51,878 DEBUG [252833] hlmSDRUtil.updateJob 2016-10-06 16:24:51,939 DEBUG [252833] hlmSDRUtil.beginTransaction 2016-10-06 16:24:51,942 DEBUG [252833] hlmSDRUtilcommitTransaction 2016-10-06 16:24:52,054 INFO [252833] AgentFSM::currentState changed from null0 to stopped 2016-10-06 16:24:52,054 INFO [252833] Agent route no longer needed...moving it to stopped routes 2016-10-06 16:24:52,054 INFO [252833] Route removed from context >HLM(ad495160-864d-4b74-8aec-cc708036b5e0)< 2016-10-06 16:24:52,054 DEBUG [252833]
Navigate to Cisco Unified Serviceability > Platform Service > Platform Administrative Web Service
https://UC_APP_IP_ADDRRESS:8443/platform-services/services/DeploymentModeService?wsdl
Contact cisco TAC in order to change back the PLM deployment mode into HCS
2016-10-07 11:37:55,581 INFO [238120] UCAppDeploymentModeConnection: Opening secure connection to: https://10.48.52.10:8443/platform-services/services/DeploymentModeService?wsdl 2016-10-07 11:37:55,581 INFO [238120] UCAppDeploymentModeConnectionPort successfully opened 2016-10-07 11:37:55,608 INFO [238120] UCAppDeploymentModeConnection: Setting userName for SOAP connection to: ccmadmin 2016-10-07 11:37:55,608 INFO [238120] UCAppDeploymentModeConnection: Accepting Certificate from UC Application 2016-10-07 11:37:55,608 INFO [238120] UCAppDeploymentModeConnection.getDeploymentMode Called 2016-10-07 11:37:55,980 DEBUG [238120] UCAppDeploymentModeConnection.getDeploymentMode Response Recieved 2016-10-07 11:37:55,980 INFO [238120] UCAppDeploymentModeConnection.getDeploymentMode Completed returning value: Enterprise 2016-10-07 11:37:55,980 INFO [238120] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest Found current UC app deployment mode to be: Enterprise 2016-10-07 11:37:55,980 INFO [238120] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest Found current Global deployment mode to be: HCS 2016-10-07 11:37:55,980 INFO [238120] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest Setting the UC app deployment mode to be: HCS 2016-10-07 11:37:55,981 INFO [238120] UCAppDeploymentModeConnection: Opening secure connection to: https://10.48.52.10:8443/platform-services/services/DeploymentModeService?wsdl 2016-10-07 11:37:55,981 INFO [238120] UCAppDeploymentModeConnectionPort successfully opened 2016-10-07 11:37:55,993 INFO [238120] UCAppDeploymentModeConnection: Setting userName for SOAP connection to: ccmadmin 2016-10-07 11:37:55,993 INFO [238120] UCAppDeploymentModeConnection: Accepting Certificate from UC Application 2016-10-07 11:37:55,993 INFO [238120] UCAppDeploymentModeConnection.setDeploymentMode Called 2016-10-07 11:37:56,183 DEBUG [238120] UCAppDeploymentModeConnection.setDeploymentMode Response Recieved 2016-10-07 11:37:56,189 INFO [238120] UCAppRestartSystemConnection: Opening secure connection to: https://10.48.52.10:8443/platform-services/services/RestartSystemService?wsdl 2016-10-07 11:37:56,205 INFO [238120] UCAppRestartSystemConnection: Setting userName for SOAP connection to: ccmadmin 2016-10-07 11:37:56,205 INFO [238120] UCAppRestartSystemConnection: Accepting Certificate from UC Application 2016-10-07 11:54:12,248 INFO [238120] ELMApi: initialized, connected to plm-hcs10 2016-10-07 11:54:12,248 DEBUG [238120] ELMApi: createCluster called - force registration is enabled. 2016-10-07 11:54:12,248 DEBUG [238120] ELMApi:Creating Cluster with clusterName: test-cluster-Z|HostName: 10.48.52.10|userid: ccmadmin|password: ******|Type: HUCM 2016-10-07 11:54:12,360 DEBUG [238120] ELMApi: createCluster Response status = 400 2016-10-07 11:54:12,360 ERROR [238120] ELMApi: createCluster completes with an error. LM resp status: 400 2016-10-07 11:54:12,360 ERROR [238120] HLMClusterLicenseAgent: Failed to assign cluster test-cluster-Z due to an internal error in LM plm-hcs10 2016-10-07 11:54:12,360 DEBUG [238120] hlmSDRUtil.updateJob 2016-10-07 11:54:12,412 DEBUG [238120] hlmSDRUtil.beginTransaction 2016-10-07 11:54:12,415 DEBUG [238120] hlmSDRUtilcommitTransaction 2016-10-07 11:54:12,538 INFO [238120] AgentFSM::currentState changed from null0 to stopped 2016-10-07 11:54:12,539 INFO [238120] Agent route no longer needed...moving it to stopped routes 2016-10-07 11:54:12,539 INFO [238120] Route removed from context >HLM(8032fcf3-9368-4644-a46f-46445247e893)< 2016-10-07 11:54:12,539 DEBUG [238120] +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2016-10-07 11:54:12,539 DEBUG [238120] Agent info : HLM(8032fcf3-9368-4644-a46f-46445247e893): RED Assign test-cluster-Z to plm-hcs10 is Failed
Verify the PLM deployment mode by browsing the below URL. PLM platform user id and password is required
https://PLM_IP_Address /elm-resources/
Enable the IP connectivity between HCM-F and PLM on port 8443. From logs HLM can't connect to the PLM
2016-10-18 12:17:47,119 DEBUG [264677] ELMApi:Creating Cluster with clusterName: cluster delta|HostName: 10.48.55.26|userid: administrator|password: ******|Type: HUCM 2016-10-18 12:17:47,904 DEBUG [43] KeepAliveConsumerProcessor::process -- enter 2016-10-18 12:17:47,905 DEBUG [43] KeepAliveConsumerProcessor::process -- received broadcast message for service sdrcnf 2016-10-18 12:17:47,905 DEBUG [43] noChange -- sdrcnf is Alive 2016-10-18 12:17:47,905 DEBUG [43] KeepAliveMonitor::setExpiresBy: 2016-10-18 12:17:47,905 DEBUG [43] now: 18/10/2016 12:17:47.905 2016-10-18 12:17:47,905 DEBUG [43] expected by: 18/10/2016 12:19:47.905 2016-10-18 12:17:47,905 DEBUG [71] KeepAliveConsumerProcessor::process -- enter 2016-10-18 12:17:47,906 DEBUG [71] KeepAliveConsumerProcessor::process -- received broadcast message for service sdrcnf 2016-10-18 12:17:47,906 DEBUG [71] noChange -- sdrcnf is Alive 2016-10-18 12:17:47,906 DEBUG [71] KeepAliveMonitor::setExpiresBy: 2016-10-18 12:17:47,906 DEBUG [71] now: 18/10/2016 12:17:47.906 2016-10-18 12:17:47,906 DEBUG [71] expected by: 18/10/2016 12:19:47.906 2016-10-18 12:17:50,188 ERROR [264677] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest javax.ws.rs.WebApplicationException: javax.xml.bind.MarshalException - with linked exception: [java.net.NoRouteToHostException: No route to host] 2016-10-18 12:17:50,188 DEBUG [264677] hlmSDRUtil.updateJob 2016-10-18 12:17:50,247 DEBUG [264677] hlmSDRUtil.beginTransaction 2016-10-18 12:17:50,249 DEBUG [264677] hlmSDRUtilcommitTransaction 2016-10-18 12:17:50,356 INFO [264677] AgentFSM::currentState changed from null0 to stopped 2016-10-18 12:17:50,356 INFO [264677] Agent route no longer needed...moving it to stopped routes
Verify the connectivity between PLM and HCM-F
admin:utils network ping XXX.YYY.WWW.ZZZ
The job fails with the error shown in the image
;
Add Platform credential either on HCM-F or into CUCDM
2016-10-06 16:26:48,508 DEBUG [264674] hlmSDRUtilcommitTransaction 2016-10-06 16:26:48,624 ERROR [264674] HLMClusterLicenseAgent: .null0_AssignClusterToELMRequest no platform credential is configured for test-cluster-Y 2016-10-06 16:26:48,624 DEBUG [264674] hlmSDRUtil.updateJob 2016-10-06 16:26:48,675 DEBUG [264674] hlmSDRUtil.beginTransaction 2016-10-06 16:26:48,677 DEBUG [264674] hlmSDRUtilcommitTransaction 2016-10-06 16:26:48,798 INFO [264674] AgentFSM::currentState changed from null0 to stopped 2016-10-06 16:26:48,798 INFO [264674] Agent route no longer needed...moving it to stopped routes 2016-10-06 16:26:48,798 INFO [264674] Route removed from context >HLM(1b32a922-3619-44b0-8004-7f6fc930c419)< 2016-10-06 16:26:48,798 DEBUG [264674] +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2016-10-06 16:26:48,798 DEBUG [264674] Agent info : HLM(1b32a922-3619-44b0-8004-7f6fc930c419): RED Assign test-cluster-Y to plm-hcs10 is Failed 2016-10-06 16:26:48,798 DEBUG [264674] AgentMessageProcessor::Agent is stopped after process, removed it from persistence store
Verify the Platform credential are configured into HCM-F
Navigate to Application Management > Cluster Application > Credential > Platform
or
Navigate to Administration > Default Credential > UC application type ( CUCM/CUCXN) Platform