소개

    이 문서에서는 문제 해결 프로세스와 함께 HCM-F(Hosted Collaboration Mediation Fulfillment)를 업그레이드한 후 발생한 문제에 대해 설명합니다.

    문제의 근본 원인은 만료된 Tomcat 인증서와 관련이 있습니다. HCM-F 어플라이언스를 업그레이드하려는 많은 고객은 기존 HCM-F Tomcat 인증서가 만료되어 재생성되지 않은 경우 이 문제에 직면하게 됩니다.

    사전 요구 사항

    요구 사항

    다음 주제에 대한 지식을 보유하고 있으면 유용합니다.

    • Cisco HCS(Hosted Collaboration Solution)
    • Cisco HCM-F(Hosted Collaboration Mediation Fulfillment)

    사용되는 구성 요소

    이 문서의 정보는 다음 소프트웨어 및 하드웨어 버전을 기반으로 합니다.

    • HCM-F 11.5.4

    이 문서의 정보는 특정 랩 환경의 디바이스를 토대로 작성되었습니다.이 문서에 사용된 모든 디바이스는 초기화된(기본) 컨피그레이션으로 시작되었습니다.현재 네트워크가 작동 중인 경우, 모든 명령어의 잠재적인 영향을 미리 숙지하시기 바랍니다.

    문제

    HCM-F(11.5.1~11.5.4) 및 스위치 버전을 업그레이드한 후 그래픽 사용자 인터페이스(GUI) 페이지에 더 이상 연결할 수 없으며 서비스가 중지됨 상태입니다.

    수행된 문제 해결

    • 서비스 상태를 확인하는 동안 UI(웹 사용자 인터페이스) 및 Tomcat 관련 서비스가 STOPPED 상태인 것으로 확인되었습니다.
    admin:utils service list 
    Requesting service status, please wait...
    System SSH [STARTED]
    Name Service Cache [STOPPED] Service Not Activated
    Entropy Monitoring Daemon [STARTED]
    Cisco SCSI Watchdog [STARTED]Service Manager [STARTED]
    Service Manager is running
    Getting list of all services
    >> Return code = 0
    Cisco AMC Service[STARTED]
    Cisco Audit Event Service[STARTED]
    Cisco CDM Database[STARTED]
    Cisco CDP[STARTED]
    Cisco CDP Agent[STARTED]
    Cisco Certificate Expiry Monitor[STARTED]
    Cisco Configuration Manager[STARTED]
    Cisco DRF Local[STARTED]
    Cisco DRF Master[STARTED]
    Cisco HCS Admin UI[STOPPED] Component is not running
    Cisco HCS CSF UI Service[STOPPED]  Component is not running
    Cisco HCS CUCDMSync Service[STARTED]
    Cisco HCS ConfigMgr[STARTED]
    Cisco HCS DMA-SA Service[STARTED]
    Cisco HCS Data Access Manager Service[STARTED]
    Cisco HCS Fulfillment Service[STARTED]
    Cisco HCS IPA Service[STARTED]
    Cisco HCS Intelligent Loader Service[STOPPED]  Component is not running
    Cisco HCS License Manager Service[STARTED]
    Cisco HCS NBI REST FF Web Service[STOPPED] Component is not running
    Cisco HCS NBI REST SDR Web Service[STOPPED]  Component is not running
    Cisco HCS NBI REST UCMon Web Service[STOPPED]  Component is not running
    Cisco HCS North Bound Interface Web Service[STOPPED]  Component is not running
    Cisco HCS Provisioning Adapter Service[STARTED]
    Cisco HCS SDR Change Notification Service[STARTED]
    Cisco HCS SDR UI[STOPPED]  Component is not running
    Cisco HCS SI UI[STOPPED]  Component is not running
    Cisco HCS SSO SP Service[STOPPED]  Component is not running
    Cisco HCS Service Inventory[STOPPED]  Component is not running
    Cisco HCS UCPA Service[STARTED]
    Cisco HCS UCSMSync Service[STARTED]
    Cisco HCS VCenterSync Service[STARTED]
    Cisco JMS Broker[STARTED]
    Cisco Log Partition Monitoring Tool[STARTED]
    Cisco Platform Manager Service[STOPPED]  Component is not running
    Cisco RIS Data Collector[STARTED]
    Cisco RTMT Web Service[STOPPED]  Component is not running
    Cisco Syslog Agent[STARTED]
    Cisco Tomcat[STARTED]
    Cisco Tomcat Stats Servlet[STOPPED]  Component is not running
    Cisco Trace Collection Service[STARTED]
    Cisco Trace Collection Servlet[STOPPED]  Component is not running
    Host Resources Agent[STARTED]
    MIB2 Agent[STARTED]
    Platform Administrative Web Service[STOPPED]  Component is not running
    SNMP Master Agent[STARTED]System Application Agent[STARTED]
    Cisco HCS UC Monitor Service[STOPPED]  Service Not ActivatedPrimary Node =true
    • CiscoSyslog에서 여러 웹 관련 서비스에 대한 HTTP(Hypertext Transfer Protocol) 404 오류가 표시됩니다.
    admin:file view activelog /syslog/CiscoSyslog 

Jun 17 16:14:19 hcs862-hcm-f local7 2 : 6: hcs862-hcm-f.XXXXXXXX.net:
Jun 17 2019 14:14:19.781 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Cisco RTMT Web Service][Reason=HTTP 404 Error][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Failed to start service.
Jun 17 16:14:19 hcs862-hcm-f local7 2 : 7: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:19.781 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco RTMT Web Service][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service terminated.
Jun 17 16:14:20 hcs862-hcm-f local7 2 : 8: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:20.791 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Cisco Trace Collection Servlet][Reason=HTTP 404 Error][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Failed to start service.
Jun 17 16:14:20 hcs862-hcm-f local7 2 : 9: hcs862-hcm-f.XXXXXXXXtest.net: Jun 17 2019 14:14:20.791 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco Trace Collection Servlet][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service terminated.
Jun 17 16:14:21 hcs862-hcm-f local7 2 : 10: hcs862-hcm-f.XXXXXXXXtest.net: Jun 17 2019 14:14:21.796 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Cisco Tomcat Stats Servlet][Reason=HTTP 404 Error][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Failed to start service.
Jun 17 16:14:21 hcs862-hcm-f local7 2 : 11: hcs862-hcm-f.XXXXXXXXtest.net: Jun 17 2019 14:14:21.796 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco Tomcat Stats Servlet][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service terminated.
Jun 17 16:14:22 hcs862-hcm-f local7 6 : 12: hcs862-hcm-f.XXXXXXXXtest.net: Jun 17 2019 14:14:22.800 UTC :  %UC_GENERIC-6-ServiceStarted: %[ServiceName=Cisco CDP][ProcessID=8426][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service started.
Jun 17 16:14:23 hcs862-hcm-f local7 2 : 13: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:23.804 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Cisco HCS Admin UI][Reason=HTTP 404 Error][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Failed to start service.
Jun 17 16:14:23 hcs862-hcm-f local7 2 : 14: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:23.804 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco HCS Admin UI][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service terminated.
Jun 17 16:14:24 hcs862-hcm-f local7 2 : 15: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:24.808 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Cisco HCS SDR UI][Reason=HTTP 404 Error][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Failed to start service.
Jun 17 16:14:24 hcs862-hcm-f local7 2 : 16: hcs862-hcm-f.XXXXXXXXtest.net: Jun 17 2019 14:14:24.808 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco HCS SDR UI][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service terminated.
Jun 17 16:14:25 hcs862-hcm-f local7 6 : 17: hcs862-hcm-f.XXXXXXXXtest.net: Jun 17 2019 14:14:25.812 UTC :  %UC_GENERIC-6-ServiceStarted: %[ServiceName=Cisco Certificate Expiry Monitor][ProcessID=8774][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service started.
Jun 17 16:14:25 hcs862-hcm-f local7 2 : 18: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:25.813 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Cisco HCS NBI REST SDR Web Service][Reason=HTTP 404 Error][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Failed to start service.
Jun 17 16:14:25 hcs862-hcm-f local7 2 : 19: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:25.813 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco HCS NBI REST SDR Web Service][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service terminated.
    • Catalina의 Tomcat.out에서 CCMTrustCerts 컨텍스트 문제를 보고합니다.
    SEVERE: Context [/CCMTrustCerts] startup failed due to previous errors
Jun 18, 2019 2:59:49 PM org.apache.catalina.startup.HostConfig deployDirectoryINFO: Deployment of web application directory /common/log/taos-log-a/tomcat/webapps/CCMTrustCerts has finished in 8,042 ms end of the file reachedoptions: q=quit, n=next, p=prev, b=begin, e=end (lines 1221 - 1223 of 1223) :INFO: Deploying web application directory /common/log/taos-log-a/tomcat/webapps/manager
Jun 18, 2019 2:59:33 PM org.apache.catalina.startup.SetContextPropertiesRule beginWARNING: [SetContextPropertiesRule]{Context} Setting property 'debug' to '0' did not find a matching property.
Jun 18, 2019 2:59:33 PM org.apache.tomcat.util.digester.SetPropertiesRule beginWARNING: [SetPropertiesRule]{Context/Realm} Setting property 'resourceName' to 'TomcatUsersDatabase' did not find a matching property.
Jun 18, 2019 2:59:41 PM org.apache.catalina.startup.TldConfig executeINFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jun 18, 2019 2:59:41 PM org.apache.catalina.core.StandardContext startInternalSEVERE: One or more Filters failed to start. Full details will be found in the appropriate container log file
Jun 18, 2019 2:59:41 PM org.apache.catalina.core.StandardContext startInternalSEVERE: Context [/manager] startup failed due to previous errors
Jun 18, 2019 2:59:41 PM org.apache.catalina.startup.HostConfig deployDirectoryINFO: Deployment of web application directory /common/log/taos-log-a/tomcat/webapps/manager has finished in 7,483 ms
Jun 18, 2019 2:59:41 PM org.apache.catalina.startup.HostConfig deployDirectoryINFO: Deploying web application directory /common/log/taos-log-a/tomcat/webapps/CCMTrustCerts
Jun 18, 2019 2:59:49 PM org.apache.catalina.startup.TldConfig executeINFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jun 18, 2019 2:59:49 PM org.apache.catalina.core.StandardContext startInternalSEVERE: One or more Filters failed to start. Full details will be found in the appropriate container log file
Jun 18, 2019 2:59:49 PM org.apache.catalina.core.StandardContext startInternal options: SEVERE: Context [/CCMTrustCerts] startup failed due to previous errors
Jun 18, 2019 2:59:49 PM org.apache.catalina.startup.HostConfig deployDirectoryINFO: Deployment of web application directory /common/log/taos-log-a/tomcat/webapps/CCMTrustCerts has finished in 8,042 ms
    • CLI(Command-Line Interface)에서 "utils diagnose test"를 실행하면 tomcat_connectors 모듈에 오류가 표시되므로 Tomcat에 대한 내부 연결이 작동하지 않습니다. 
    test - tomcat_connectors   : Failed - The HTTP and HTTPS ports are not responding to local requests. 
Please collect all of the Tomcat logs for root cause analysis: file get activelog tomcat/logs/*
    • Tomcat 인증서는 3월 26일 만료됩니다.그러나 HCM-F를 업그레이드하는 동안 또는 그 이전에 문제가 관찰되지 않았습니다.
    Jun 19 12:00:00 hcs862-hcm-f local99 0 : 42: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 19 2019 10:00:00.19 UTC :  %UC_CERT-0-CertExpired: %[Message=Certificate expiration Notification. Certificate name:tomcat.der Unit:tomcat Type:own-cert Expiration:Sun Mar 26 12:30:59:000 CES][AppID=Cisco Certificate Monitor][ClusterID=][NodeID=hcs862-hcm-f]: Certificate has Expired and needs to be changed at the earliest
Jun 19 12:00:00 hcs862-hcm-f local99 0 : 43: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 19 2019 10:00:00.21 UTC :  %UC_CERT-0-CertExpired: %[Message=Certificate expiration Notification. Certificate name:ipsec.der Unit:ipsec Type:own-cert Expiration:Sun Mar 26 12:30:57:000 CEST ][AppID=Cisco Certificate Monitor][ClusterID=][NodeID=hcs862-hcm-f]: Certificate has Expired and needs to be changed at the earliest

    솔루션

      1. Tomcat 인증서 재생성:

    admin:set cert regen tomcat 
WARNING: This operation will overwrite any CA signed certificate previously imported for  tomcat
Proceed with regeneration (yes|no)? yes
Successfully Regenerated Certificate for tomcat.
You must restart services related to tomcat for the regenerated certificates to become active.

     2 .  HCM-F 서버 다시 시작:

    admin:utils system restart
 Do you really want to restart ?Enter (yes/no)? yes

    확인

    • 서비스 상태 확인:
    admin:utils service list 
    Requesting service status, please wait...
    System SSH [STARTED]
    Name Service Cache [STOPPED] Service Not Activated
    Entropy Monitoring Daemon [STARTED]
    Cisco SCSI Watchdog [STARTED]
    Service Manager [STARTED]Service Manager is running
    Getting list of all services
    >> Return code = 0
    Cisco AMC Service[STARTED]
    Cisco Audit Event Service[STARTED]
    Cisco CDM Database[STARTED]
    Cisco CDP[STARTED]
    Cisco CDP Agent[STARTED]
    Cisco Certificate Expiry Monitor[STARTED]
    Cisco Configuration Manager[STARTED]
    Cisco DRF Local[STARTED]
    Cisco DRF Master[STARTED]
    Cisco HCS Admin UI[STARTED]
    Cisco HCS CSF UI Service[STARTED]
    Cisco HCS CUCDMSync Service[STARTED]
    Cisco HCS ConfigMgr[STARTED]
    Cisco HCS DMA-SA Service[STARTED]
    Cisco HCS Data Access Manager Service[STARTED]
    Cisco HCS Fulfillment Service[STARTED]
    Cisco HCS IPA Service[STARTED]
    Cisco HCS Intelligent Loader Service[STARTED]
    Cisco HCS License Manager Service[STARTED]
    Cisco HCS NBI REST FF Web Service[STARTED]
    Cisco HCS NBI REST SDR Web Service[STARTED]
    Cisco HCS NBI REST UCMon Web Service[STARTED]
    Cisco HCS North Bound Interface Web Service[STARTED]
    Cisco HCS Provisioning Adapter Service[STARTED]
    Cisco HCS SDR Change Notification Service[STARTED]
    Cisco HCS SDR UI[STARTED]
    Cisco HCS SI UI[STARTED]
    Cisco HCS SSO SP Service[STARTED]
    Cisco HCS Service Inventory[STARTED]
    Cisco HCS UCPA Service[STARTED]
    Cisco HCS UCSMSync Service[STARTED]
    Cisco HCS VCenterSync Service[STARTED]
    Cisco JMS Broker[STARTED]
    Cisco Log Partition Monitoring Tool[STARTED]
    Cisco Platform Manager Service[STARTED]
    Cisco RIS Data Collector[STARTED]
    Cisco RTMT Web Service[STARTED]
    Cisco Syslog Agent[STARTED]
    Cisco Tomcat[STARTED]
    Cisco Tomcat Stats Servlet[STARTED]
    Cisco Trace Collection Service[STARTED]
    Cisco Trace Collection Servlet[STARTED]
    Host Resources Agent[STARTED]MIB2 Agent[STARTED]
    Platform Administrative Web Service[STARTED]
    SNMP Master Agent[STARTED]
    System Application Agent[STARTED]
    Cisco HCS UC Monitor Service[STOPPED]  
    Service Not ActivatedPrimary Node =true