Understand and Troubleshoot Finesse BOSH Implementation
Available Languages
Contents
Introduction
This document describes the architecture behind Finesse connections that use BOSH and how BOSH connection problems can be diagnosed.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- Cisco Finesse
- Unified Contact Center Enterprise (UCCE)
- Unified Contact Center Express (UCCX)
- Web browser developer tools
- Windows and/or Mac administration
Components Used
The information in this document is based on these software and hardware versions:
- Cisco Finesse 9.0(1) - 11.6(1)
- UCCX 10.0(1) - 11.6(2)
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
Connections that use Bidirectional-streams Over Synchronous HTTP are called BOSH.
Understand Finesse BOSH Implementation
Understand XMPP
Extensible Messaging and Presence Protocol (XMPP) (also known as Jabber) is a stateful protocol in a client-server model. XMPP allows for fast delivery of small pieces of structured eXtensible Markup Language (XML) data from one entity to another. XMPP/Jabber is extensively used in instant messaging (IM) and presence applications.
All XMPP entities are identified by their Jabber ID (JID).
JID Addressing Scheme: user@domain/resource
|
user |
client username on the XMPP server or name of the conference room |
|
domain |
XMPP server fully qualified domain name (FQDN) |
|
resource |
identifier of the user's specific entity/endpoint (for example, laptop, smartphone, and so on), a session identifier, or pubsub node name |
XMPP messages are called stanzas. There are three core stanzas in XMPP:
1. <message>: one direction, one recipient
2. <presence>: one direction, publish to many
3. <iq>: info/query - request/response
All stanzas have to and from addresses and most stanzas also have type, id, and xml:langattributes.
|
Stanza Attribute |
Purpose |
|
to |
destination JID |
|
from |
source JID |
|
type |
purpose of the message |
|
id |
unique identifier used to link a request with a response for <iq> stanzas |
|
xml:lang |
defines the default language for any human readable XML in the stanza |
Example XMPP Message
<message to='person1@example' from='person2@example' type='chat'>
<subject> Team meeting </subject>
<body>Hey, when is our meeting today? </body>
<thread>A4567423</thread>
</message>
XMPP Implementation with Finesse
If a web application needs to work with XMPP, multiple issues arise. Browsers don’t support XMPP over Transmission Control Protocol (TCP) natively, so all XMPP traffic must be handled by a program which runs inside the browser. Web servers and browsers communicate via HyperText Transfer Protocol (HTTP) messages, so Finesse and other web applications wrap XMPP messages inside of HTTP messages.
The first difficulty with this approach is that HTTP is a stateless protocol. This means that each HTTP request is not related to any other request. However, this problem can be addressed by applicative means--for example through the use of cookies/post data.
The second difficulty is the unidirectional behavior of HTTP. Only the client sends requests, and the server can only respond. The server’s inability to push data makes it unnatural to implement XMPP over HTTP.
This problem does not exist in the original XMPP Core specification (RFC 6120), where XMPP is bound to TCP. However, if you want to address the problem with XMPP bound to HTTP, for example, because Javascript can send HTTP requests, there are two possible solutions. Both require a bridge between HTTP and XMPP.
The proposed solutions are:
1. Polling (legacy protocol): repeated HTTP requests asking for new data defined in XEP-0025: Jabber HTTP Polling
2. Long polling also is known as BOSH: transport protocol that emulates the semantics of a long-lived, bidirectional TCP connection between two entities by efficiently using multiple synchronous HTTP request/response pairs without requiring the use of frequent polling defined in XEP-0124: HTTP Binding and extended by XEP-0206: XMPP Over BOSH
Finesse implements BOSH as it is quite efficient from the server load point of view and traffic-wise. The reason to use BOSH is to cover up the fact that the server does not have to respond as soon there is a request. The response is delayed up to a specified time until the server has data for the client, and then it is sent as a response. As soon as the client gets the response, the client makes a new request and so forth.
The Finesse desktop client (web application) establishes a stale BOSH connection over TCP port 7443 every 30 seconds. After 30 seconds, if there are no updates from the Finesse Notification Service, the Notification service sends an HTTP reply with a 200 OK and a (nearly) empty response body. If the Notification Service has an update on the presence of an agent or a dialog (call) event, for example, the data is sent immediately to the Finesse web client.
Example Finesse XMPP Reqeust/Response
This example shows the first XMPP message request response shared between the Finesse client and Finesse server to set up the BOSH connection.
Finesse client request:
<body xmlns="http://jabber.org/protocol/httpbind" xml:lang="en-US" xmlns:xmpp="urn:xmpp:xbosh" hold="1" ver="1.9" to="fin1.ucce.local" wait="30" xmpp:version="1.0" from="47483648@fin1.ucce.local" rid="704654808"/>
Finesse server response:
<body xmlns="http://jabber.org/protocol/httpbind" xmlns:stream="http://etherx.jabber.org/streams" authid="26779701" sid="26779701" secure="true" requests="4" inactivity="60" polling="5" wait="30" hold="1" ack="704654808" maxpause="300" ver="1.6"><stream:features><mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl"><mechanism>PLAIN</mechanism></mechanisms><compression xmlns="http://jabber.org/features/compress"><method>zlib</method></compression><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"/><session xmlns="urn:ietf:params:xml:ns:xmpp-session"/></stream:features></body>
To summarize:
- The Finesse web client has a stale HTTP connection (http-bind) set up to the Finesse server via TCP port 7443. This is known as a BOSH long poll.
- The Finesse Notification Service is a presence service that posts updates regarding the state of an agent, call, and so on.
- If the Notification service has an update, it replies to the http-bind request with the state update as an XMPP message in the HTTP response body.
- If there are no state updates 30 seconds after receiving the http-bind request, the Notification Service replies without any state updates to allow the Finesse web client to send another http-bind request. This serves as a way for the Notification service to know that the Finesse web client is still able to connect to the Notification Service and that the agent did not close their browser or put their computer to sleep, and so on.
Understand Finesse XMPP Messages and XMPP Nodes
Finesse also implements XMPP specification XEP-0060: Publish-Subscribe. The purpose of this specification is to allow the XMPP server (Notification service) to get information published to XMPP nodes (topics) and then to send XMPP events to entities subscribed to the node. In the case of Finesse, the Computer Telephony Integration (CTI) server sends CTI messages to the Finesse web service to tell Finesse about configuration updates such as, but not limited to, agent or Contact Service Queue (CSQ) creation or information about a call. This information is then converted into an XMPP message that the Finesse web service publishes to the Finesse Notification service. The Finesse Notification service then sends XMPP over BOSH messages to agents that are subscribed to certain XMPP nodes.
Some of Finesse API objects that are defined in the Finesse Web Services Developer Guide are XMPP nodes. Agent and supervisor Finesse web clients can subscribe to event updates for some of these XMPP Nodes in order to have up-to-date information about real-time events (such as call events, state events, and so on). This table shows the XMPP nodes that are pubsub enabled.
|
Finesse API Object |
Purpose |
Subscription |
|
/finesse/api/User/<LoginID> |
Shows the state and team mapping of the agent |
Agents and Supervisors |
|
/finesse/api/User/<LoginID>/Dialogs |
Shows the call(s) handled by the agent |
Agents and Supervisors |
|
/finesse/api/User/<LoginID>/ClientLog |
Used to capture client logs from the Send Error Report button |
Agents and Supervisors |
|
/finesse/api/User/<LoginID>/Queue/<queueID> |
Shows queue statistics data (if enabled) |
Agents and Supervisors |
|
/finesse/api/Team/<TeamID>/Users |
Shows the agents that belong to a certain team including state information |
Supervisors |
|
/finesse/api/SystemInfo |
Shows the state of the Finesse server. Used to determine if failover is needed |
Agents and Supervisors |
Example 1: Use Pidgin to View Finesse XMPP Nodes
Step 1. Download and install the XMPP client Pidgin.
Step 2. Navigate to Accounts > Modify > Basic and configure the Login Options:
- Protocol: XMPP
- Username: LoginID for any agent
- Domain: FQDN of Finesse server
- Resource: Placeholder - any value can be used, for example, test
- Password: Agent password
- Check the Remember password checkbox
Step 3. Navigate to Accounts > Modify > Advanced and configure:
- Connection security: Use encryption if available
- Check the Allow plaintext auth other unencrypted streams.
- Connection port: 5222. Use the default port 5222. This port is required for external XMPP clients. Finesse desktop clients use 7443. Do not use port 7443.
- Connection server: Finesse server FQDN
Step 4. Navigate to Tools > Plugins and enable the XMPP Console.
Step 5. Navigate to Tools > XMPP Console > XMPP Console to open the XMPP Console.
Step 6. Execute this <iq> message to see all of the XMPP nodes that exist.
<iq type='get' from='<loginID>@<Finesse_FQDN>/test' to='pubsub.<Finesse_FQDN>' id='testId1'
<query xmlns='http://jabber.org/protocol/disco#items'/>
</iq>For example:
<iq type='get' from='47483648@fin1.ucce.local/test' to='pubsub.fin1.ucce.local' id='testId1'>
<query xmlns='http://jabber.org/protocol/disco#items'/>
</iq>
In a lab environment with two agents and two CSQs configured, this output is contained in the Finesse response:
<iq type='result' id='testId1' from='pubsub.fin1.ucce.local' to='47483648@fin1.ucce.local/test'>
<query xmlns='http://jabber.org/protocol/disco#items'>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/Team/5000/Users'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483648/Dialogs'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483651'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483651/ClientLog'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483649/Queues'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/Team/5001/Users'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483650/ClientLog'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/Queue/1'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483648/ClientLog'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483648/Queues'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483650'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483650/Queues'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483651/Dialogs'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483648'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/Team/1/Users'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483649'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483651/Queues'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483649/ClientLog'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/Queue/0'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483649/Dialogs'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/User/47483650/Dialogs'/>
<item jid='pubsub.fin1.ucce.local' name='' node='/finesse/api/SystemInfo'/>
</query>
</iq>Example 2: Use Browser Developer Tools Network Tab to View HTTP Messages
Each browser has a set of developer tools. The Network tab of the developer tools shows the HTTP messages sent and received by the Finesse web client (browser). For example, this image shows how the Finesse web client sends a SystemInfo request which checks Finesse Tomcat status every minute as a failover check. Additionally, the http-bind messages from the BOSH connection are also displayed. The Finesse server sends back a response within 30 seconds if there are no updates to publish on the XMPP nodes the web client is subscribed to.
Troubleshoot BOSH Disconnect Error Message
When a BOSH disconnect occurs, the error Lost connection to {Finesse Server FQDN}. Please wait for a reachable Finesse Server to be found... is displayed in a red banner at the top of the Finesse desktop.
This message is displayed because at this time, no XMPP subscription events can be received from the Cisco Finesse Notification Service. Hence, state information and call details cannot be displayed on the agent desktop.
For UCCX, 60 seconds after the browser disconnects, the agent is put into a Logout state. The agent can be in the Ready or Not Ready state for the logout to happen.
For UCCE, Finesse takes up to 120 seconds to detect when an agent closes the browser or the browser crashes and Finesse waits 60 seconds before sending a forced logout request to the CTI server which causes the CTI server to put the agent in a Not Ready state. Under these conditions, Finesse can take up to 180 seconds to sign out the agent. Unlike in UCCX, the agent moves into a Not Ready state instead of the Logout state.
For more information on UCCE Finesse Desktop behavior, refer to the Desktop Behavior section of the Cisco Finesse Failover Mechanisms chapter in the Cisco Finesse Administration Guide.
Log Analysis
The Finesse and UCCX Notification service logs can be collected via RTMT or via the CLI:
file get activelog /desktop recurs compress
Debug Notification Service Logs
Enable Notification Service debug logs of Unified Contact Center Express (UCCX), as shown:
admin:utils uccx notification-service log enable
WARNING! Enabling Cisco Unified CCX Notification Service logging can affect system performance
and should be disabled when logging is not required.
Do you want to proceed (yes/no)? yes
Cisco Unified CCX Notification Service logging enabled successfully.
NOTE: Logging can be disabled automatically if Cisco Unified CCX Notification Service is restarted.
Enable Notification Service debug logs of Unified Contact Center Enterprise (UCCE) (Finesse Standalone), as shown:
admin:utils finesse notification logging enable
Checking that the Cisco Finesse Notification Service is started...
The Cisco Finesse Notification Service is started.
Cisco Finesse Notification Service logging is now enabled.
WARNING! Cisco Finesse Notification Service logging can affect system performance
and should be disabled when logging is not required.
Note: Logging can be disabled automatically if you restart the Cisco Finesse Notification Service
These logs are in the /desktop/logs/openfire folder and are named debug.log.
As shown in the image, the Notification Service (Openfire) debug.log shows the http binding with desktop along with the IP address and port of the agent PC.
As shown in the image, the last active 0 ms shows that session is still active.
Openfire closing the idle session indicates the agent logout can trigger in 60 seconds where Finesse can send a forced logout with a reason code of 255 to the CTI server. The actual behavior of the desktop under these conditions depends on the setting for Logout on Agent Disconnect (LOAD) in UCCE. In UCCX, this is always the behavior.
If the Fineese client does not send http-bind messages to the Finesse server, the logs can show the session up time and show the session close.
2017.06.17 00:14:34 Session (id=f382a015) was last active 0 ms ago: 1001003@xxxxx.xxxx.xxx.cisco.com/desktop 2017.06.17 00:15:04 Session (id=f382a015) was last active 13230 ms ago: 1001003@xxxxx.xxxx.xxx.cisco.com/desktop 2017.06.17 00:15:34 Session (id=f382a015) was last active 43230 ms ago: 1001003@xxxxx.xxxx.xxx.cisco.com/desktop 2017.06.17 00:16:04 Session (id=f382a015) was last active 63231 ms ago: 1001003@xxxxx.xxxx.xxx.cisco.com/desktop 2017.06.17 00:17:04 Unable to route packet. No session is available so store offline. <message from="pubsub. xxxxx.xxxx.xxx.cisco. com" to="1001003@xxxxx.xxxx.xxx.cisco.com.cisco.com" id="/finesse/api/User/1001003__1001003@xxxxx.xxxx.xxx.cisco.com__o5Aqb"><event xmlns="http://jabber.org/protocol/pubsub#event"><items node="/finesse/api/User/1001003"><item id="0d78a283-466d-4477-a07e-6e33a856fce388"><notification xmlns="http://jabber.org/protocol/pubsub"><Update>
Info Notification Service Logs
These logs are in the /desktop/logs/openfire folder and are named info.log. If the Fineese client does not send http-bind messages to the Finesse server, the logs can show the the session become inactive.
2017.06.17 00:16:04 Closing idle session (id=f382a015): 1001003@xxxxx.xxxx.xxx. cisco.com/desktop
after inactivity for more than threshold value of 60 2017.06.17 00:16:04 A session is closed for 1001003@xxxxx.xxxx.xxx. cisco.com/desktop
Webservices Logs
These logs are in the /desktop/logs/webservices folder and are named Desktop-webservices.YYYY-MM-DDTHH-MM-SS.sss.log. If the Fineese client does not send http-bind messages to the Finesse server within the specified amount of time, the logs can show the the agent presense become unavailable and 60 seconds later, a presense driven logout can occur.
0000001043: XX.XX.XX.XXX: Jun 17 2017 00:16:04.630 +0530: %CCBU_Smack Listener Processor (1)-6-PRESENCE_NOTIFICATION_RECIEVED: %[FROM JID=1001003@xxxxx.xxxx.xxx.cisco.com/desktop][PRESENCE_TYPE=unavailable]:Finesse received a presence notifcation 0000000417: XX.XX.XX.XXX: Jun 17 2017 00:16:04.631 +0530: %CCBU_Smack Listener Processor (1)-6-UNSUBSCRIBE_REQUEST_SUCCESS: %[NodeId=/finesse/api/User/1001003/ClientLog][user_id=1001003@xxxxx.xxxx.xxx.cisco.com]: Sucessfully unsubscribed from a node on the XMPP server 0000001044: XX.XX.XX.XXX: Jun 17 2017 00:16:04.631 +0530: %CCBU_Smack Listener Processor (1)-6-AGENT_PRESENCE_MONITOR: %[message_string=Adding agent 1001003 into the expiry hash.]: 0000001051: XX.XX.XX.XXX: Jun 17 2017 00:16:35.384 +0530: %CCBU_pool-8-thread-1-6-AGENT_PRESENCE_MONITOR: %[message_string=[Expired] Removed agent from cache 1001003]: 0000001060: XX.XX.XX.XXX:: Jun 17 2017 00:17:04.632 +0530: %CCBU_CoreImpl-worker12-6-PRESENCE DRIVEN LOGOUT: %[agent_id=1001003]: Performing CTI Logout on basis of the agents unavailable presence 0000001061: XX.XX.XX.XXX:: Jun 17 2017 00:17:04.633 +0530: %CCBU_CoreImpl-worker12-6-MESSAGE_TO_CTI_SERVER: %[cti_message=Invoke id :39 , agentstate :
1, workmode : 0, reason code: 255, forceflag :1, agentcapacity: 1, agentext: 1001003, agentid: 1001003, supervisorid: null, ssoFlag=false][cti_message_name=SetAgentStateReq]: Message going to the backend cti server 0000001066: XX.XX.XX.XXX:: Jun 17 2017 00:17:04.643 +0530: %CCBU_CTIMessageEventExecutor-0-6-DECODED_MESSAGE_FROM_CTI_SERVER: %[cti_message=CTIAgentStateEvent [skillGroupState=1 (LOGOUT), stateDuration=0,
skillGroupNumber=-1, skillGroupPriority=0, agentState=1 (LOGOUT), eventReasonCode=255, numFltSkillGroups=0, CTIClientSignature=null, agentID=1001003, agentExtension=1001003, agentInstrument=null, agentID_Long=1001003,
duration=null, nextAgentState=null, fltSkillGroupNumberList=[], fltSkillGroupIDList=[], fltSkillGroupPriorityList=[], fltSkillGroupStateList=[], MRDId=1, agentMode=0]CTIMessageBean [invokeID=null, cti_sequence_id=105,
msgID=30, timeTracker={"id":"AgentStateEvent","CTI_MSG_RECEIVED":1497638824642,"CTI_MSG_DISPATCH":1497638824643}, msgName=AgentStateEvent, deploymentType=CCX]][cti_response_time=1][dispatch_phase=DnD-CHECKPOINT-3B]:
Decoded Message to Finesse from backend cti server
Common Reasons for BOSH Disconnect
BOSH connections are setup by the web client, and the Finesse server determines if the agent presence is unavailable. These issues are almost always client side issues relating to the browser, agent computer, or network as the onus of starting up the connection is up to the client.
Problem - Agents Disconnect at Different Times (Client Side Issue)
Recommended Actions
Check for these issues:
1. Network issue:
- Review firewall rules and logs -- TCP port 7443 must not be blocked or throttled
- Use an HTTP web traffic sniffer like Fiddler® or Wireshark® to confirm that the browser sends http-bind requests over TCP port 7443 and receives responses
- Check all network devices/interfaces between the agent computer and the Finesse server for excessive delay or packet drops
- Traceroute can be useful to determine the path and determine delays
- On a Microsoft® Windows® PC: tracert {Finesse Server IP | Finesse Server FQDN}
- On a Mac®: traceroute {Finesse Sever IP | Finesse Server FQDN}
- On Cisco IOS® software, the interface statistics can be checked: show interfaces
- Traceroute can be useful to determine the path and determine delays
- Collect Finesse Client logs for a test agent. Client logs can be collected in three ways:
- Browser web console logs
- Press the Send Error Report button on the Finesse page and collect the Finesse server logs. The logs are located in /desktop/logs/clientlogs.
- Log in via https://<Finesse-FQDN>/desktop/locallog and collect the logs after the issue occurs.
Every minute, the client connects to the Finesse server to calculate drift and network latency:
<PC date-time with GMT offset>: : <Finesse FQDN>: <Finesse server date-time with offset>:
Header : Client: <date-time>, Server: <date-time>, Drift: <drift> ms, Network Latency (round trip): <RTT> ms
2019-01-11T12:24:14.586 -05:00: : fin1.ucce.local: Jan 11 2019 11:24:14.577 -0600: Header : Client: 2019-01-1
In case of any log collection issues, refer to Troubleshoot Cisco Finesse Desktop Persistent Logging Problem
2. Unsupported browser and/or version:
Use supported browser/version and settings as per the compatibility matrices:
3. Browser stuck condition due to content/processing of other tab/window:
Check the agent workflow to see if they:
- Commonly have other tabs or windows up that are constantly running other real-time applications such as music/video streaming, WebSocket connections, custom Customer Relationship Management (CRM) web clients, etc
- Have a very large number of tabs or windows open
- Have disabled browser caching
- Have kept their browser running for a long time and do not close the browser at the end of the workday
4. Computer put to sleep:
Check to see if the agent puts their computer to sleep before logging out of Finesse or if their computer sleep setting timer is very low.
5. High CPU or high memory issue on client computer:
- If the agent browser runs in a shared environment such as Microsoft Windows Remote Desktop Services, Citrix® XenApp®, Citrix XenDesktop®, determine if the browser performance depends on the number of users running the browser at the same time
- Ensure that the proper memory and CPU resources is configured based on the number of users
- Check computer resource utilization issues:
- Windows:
- Windows PowerShell Get-Counter command that checks % of CPU time, Megabytes of memory available, and % of memory in use every 2 seconds: Get-Counter -Counter "\Processor(_Total)\% Processor Time","\Memory\Available MBytes","\Memory\% Committed Bytes In Use" -SampleInterval 2 -Continuous
- Alternative to using PowerShell to view the Windows' performance counters, Windows Perfomance Monitor can be used
- Task Manager can be used to view live CPU and memory statistics globally and on a process-by-process basis
- Mac:
- Terminal Top command that checks live total CPU and memory: top
- Check processes and sort by CPU utilization: top -o CPU
- Check processes and sort by memory utilization: top -o MEM
- Activity Monitor can be used to view live CPU and memory statistics globally and on a process-by-process basis
- Terminal Top command that checks live total CPU and memory: top
- Windows:
6. 3rd party gadgets performing unexpected, problematic activity in background:
Test the Finesse desktop behavior with all 3rd party gadgets removed.
7. NTP issue on server or client:
- Check utils ntp status on the Finesse publisher server to ensure the NTP server stratum is 4 or lower
- In the client logs, check drift and network latency
Problem - All Agents Disconnect at the Same Time (Server Side Issue)
Recommended Actions
Check for these issues:
1. Cisco Unified Communications Manager CTIManager service disconnect. If all CTIManager providers for UCCX are shutdown or crash, UCCX agents see the red banner error. UCCE agents do not see the red banner if this happens, but calls fail to route propertly to the agents.
- Check to see if the Cisco CTIManager service is started on the CUCM servers used as CTI providers
- Check to see if the Cisco CTIManager service crashed via the Event Viewer - Application logs on RTMT to see if the Cisco CTIManager service crashed
- To collect event viewer logs on RTMT, navigte to System > Tools > Trace and Log Central > Collect Files > Select System Services/Applications > Event Viewer-Application Log.
- To collect the Event Viewer-Application logs on CLI: file get activelog /syslog/CiscoSyslog* abstime hh:mm:MM/DD/YY hh:mm:MM/DD/YY
- To view core dumps on the CLI: utils core active list
2. Finesse/UCCX Notification Service stopped or crashed:
- Check the Event Viewer-Application Logs for Notification Service errors or to see if the service was stopped
- Check to see if the Notification service is up: utils service list
- Check the times the Notification service shut down: file search activelog /desktop/logs/openfire "Openfire stopped"
- Check the times the Notification service started: file search activelog /desktop/logs/openfire "HTTP bind service started"
- Check for Notification service memory dumps that resulted from a crash: file list activelog /desktop/logs/openfire/*.hprof
- Check to see if the Notification service is listening for traffic on TCP port 7443: show open ports regexp 7443.*LISTEN
- Check to see if these defects are applicable (these defects would cause login failure for agents logging in and for agents already logged in, those agents would see the red banner Finesse disconnect message):
- Cisco bug ID CSCva72280 - Finesse Tomcat and Openfire Crash for invalid XML characters
- Cisco bug ID CSCva72325 - UCCX: Finesse Tomcat and Openfire Crash for invalid XML characters
Restart Cisco Finesse Tomcat and Notification Service if a crash is suspected. This is only recommended in a network down situation, otherwise, these restarts disconnect agents from the Finesse server.
Steps for UCCE:
- utils service stop Cisco Finesse Tomcat
- utils service stop Cisco Finesse Notification Service
- utils service start Cisco Finesse Tomcat
- utils service start Cisco Finesse Notification Service
Steps for UCCX:
- utils service stop Cisco Finesse Tomcat
- utils service stop Cisco Unified CCX Notification Service
- utils service start Cisco Finesse Tomcat
- utils service start Cisco Unified CCX Notification Service
Use Fiddler
Configuring Fiddler can be a somewhat challenging task without understanding the steps needed and understanding how Fiddler works. Fiddler is a man-in-the-middle web proxy that stands between the Finesse client (web browser) and the Finesse server. Due to the connections secured between the Finesse client and Finesse server, this adds a layer of complexity to the Fiddler configuration in order to view secured messages.
Common Fiddler Issue
Since Fiddler stands in between the Finesse client and Finesse server, the Fiddler application needs to create signed certificates for all Finesse TCP ports that require certificates:
Cisco Finesse Tomcat service certificates
- Finesse publisher server TCP 8445 (and/or 443 for UCCE)
- Finesse subscriber server TCP 8445 (and/or 443 for UCCE)
Cisco Finesse (Unified CCX) Notification Service certificates
- Finesse publisher server TCP 7443
- Finesse subscriber server TCP 7443
HTTPS decryption must be enabled for Fiddler to dynamically generate certificates on behalf of the Finesse server. This is not enabled by default.
If HTTPS decryption is not configured, the initial tunnel connection to the Notification service is seen, but the http-bind traffic is not. Fiddler only shows:
Tunnel to <Finesse server FQDN>:7443
Then, the Finesse certificates signed by Fiddler must be trusted by the client. If these certificates are not trusted, moving past the Establishing encrypted connection... stage of Finesse login is not possible.
In some cases, accepting the certificate exceptions from the login does not work, and the certificates need to be trusted by the browser manually.
Example Configuration Steps
Step 1. Download Fiddler
Step 2. Enable HTTPS decryption. Navigate to Tools > Options > HTTPS and check the Decrypt HTTPS traffic checkbox.
Step 3. A warning message box opens to ask to trust the Fiddler Root Certificate. Select Yes.
Step 4. A warning message box opens with the message "You are about to install a certificate from a certification authority (CA) claiming to represent: DO_NOT_TRUST_FiddlerRoot... Do you want to install this certificate?". Select Yes.
Step 5. Manually add the Finesse publisher and subscriber certificates to the computer or browser certificate trust store. Ensure ports 8445, 7443, and (only for UCCE) 443. For example, on Firefox, this can be done simply without downloading certificates from the Finesse Operating System Administration page:
Options > Find in Options (search) > Certificates > Servers > Add Exception > Location > Enter https://<Finesse server>:port for the relevant ports for both Finesse servers.
Step 6. Log into Finesse and see the http-bind messages leave the Finesse client to the Finesse Server via Fiddler.
In the example provided, the first 5 messages show http-bind messages that were responded to by the Finesse server. The first message contains 1571 bytes of data returned in the message body. The body contains an XMPP update regarding an agent event. The final http-bind message has been sent by the Finesse client, but has not gotten a response from the Finesse server. This can be determined when you see that the HTTP result is null (-) and the number of bytes in the response body is null (-1).
Closer view of the data:
Response body for XMPP message:
Use Wireshark
Wireshark is a commonly used packet sniffing tool that can be used to sniff and decode HTTPS traffic. HTTPS traffic is HTTP traffic secured over Transport Layer Security (TLS). TLS provides integrity, authentication and confidentiality between to hosts. It is used commonly in web applications, but it can be used with any protocol that uses TCP as the transport layer protocol. Secure Sockets Layer (SSL) is the former version of the TLS protocol, which is no longer used as it is insecure. These names are often used interchangeably, and the Wireshark filter used for SSL or TLS traffic is ssl.
Step 1. On the agent's Windows PC navigate to Control Panel > System and Security > System > Advanced system settings Environmental Variables...
Step 2. Navigate to User variables for user <username> > New...
Create a variable named SSLKEYLOGFILE.
Create a file to store the SSL premaster secret in a private directory: SSLKEYLOGFILE=</path/to/private/directory/with/logfile>
Step 3. If Firefox or Chrome are open, close the applications. After they are reopened, they can start writing to the SSLKEYLOGFILE.
Step 4. On Wireshark, navigate to Edit > Preferences...
Navigate to Protocols > SSL.
Step 5. Enter the location of the premaster secret log filename configured in Step 2.
Step 6. Use Wireshark filter tcp.port==7443 && ssl, the secured HTTP communication between the Finesse client and Finesse server (Notification Service) is seen decrypted.
Related Defects
- Cisco bug ID CSCva72280 - Finesse Tomcat and Openfire Crash for invalid XML characters
- Cisco bug ID CSCva72325 - UCCX: Finesse Tomcat and Openfire Crash for invalid XML characters
Related Information
- XMPP Specifications
- XEP-0124: BOSH
- XEP-0060: Publish-Subscribe
- Firefox Web Console
- Microsoft Edge Web Console
- Chrome Web Console
- Windows PowerShell
- Windows Perfomance Monitor
- Troubleshooting Input Queue Drops and Output Queue Drops
- Windows Task Manager
- Mac Terminal
- Mac Activity Monitor
- Fiddler Download
- Fiddler Configuration
- Wireshark Download
- Wireshark SSL Decryption
- Technical Support & Documentation - Cisco Systems
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
2.0 |
31-May-2023 |
Updated Title, Introduction, PII, Biased Language, SEO, Alt Text, Branding Requirements, Style Requirements, Machine Translation, Gerunds, Formatting. |
1.0 |
23-Jun-2017 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)