Introduction
This document describes multiple troubleshooting scenarios for Cisco Finesse Server or any other Cisco Voice Operating System (VOS) based product with the use of java based command line interface (CLI).
Note: The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Troubleshoot Cisco VOS Based Finesse Server With CLI
Generic Overview
Cisco VOS CLI provides a set of commands that allows to accomplish certain troubleshoting tasks in a very effecient way.
Clean Up Disk Space
A very common task for any VOS based server is to clean up space in the common partition. This command can be used for that.
First, list the log files and specify a path and optional keys.
admin:file list activelog /desktop/logs/webservices date detail
11 Aug,2016 20:26:23 1,927,946 Desktop-webservices.2016-08-11T15-53-39.646.startup.log.zip
11 Aug,2016 21:44:42 1,105,322 Error-Desktop-webservices.2016-08-11T15-53-40.166.startup.log.zip
12 Aug,2016 01:48:58 1,741,638 Desktop-webservices.2016-08-11T20-26-23.164.log.zip
12 Aug,2016 03:35:30 1,126,078 Error-Desktop-webservices.2016-08-11T21-44-42.664.log.zip
12 Aug,2016 07:11:33 1,753,064 Desktop-webservices.2016-08-12T01-48-58.164.log.zip
12 Aug,2016 09:26:17 1,125,984 Error-Desktop-webservices.2016-08-12T03-35-30.164.log.zip
12 Aug,2016 12:34:08 1,741,476 Desktop-webservices.2016-08-12T07-11-33.664.log.zip
12 Aug,2016 15:17:05 1,125,790 Error-Desktop-webservices.2016-08-12T09-26-17.664.log.zip
Then remove the old log files. Optionally you can specify noconfirm key to avoid and not press yes as a confirmation.
admin:file delete activelog /desktop/logs/webservices/*webservices.2016-08-11* noconfirm
files: found = 4, deleted = 4
Audit CLI Commands Entered
This command lists all VOS CLI commands entered. Even the commands executed unsuccessfully.
From the output, you can see that the system was restarted, tomcat-trust certificate was imported and Tomcat service was restarted after that.
admin:file search activelog platform/log/cli* "running command" ignorecase recurs
Searching path: /var/log/active/platform/log/cli*
2016-11-03 09:59:12,558 INFO [main] sdMain.main - running command -> [utils system restart ]
2016-11-03 09:38:52,623 INFO [main] sdMain.main - running command -> [set cert import trust tomcat]
2016-11-03 09:39:30,978 INFO [main] sdMain.main - running command -> [utils service restart Cisco Tomcat]
2016-11-03 09:41:16,664 INFO [main] sdMain.main - running command -> [show cert own tomcat]
Search completed
Note: On some VOS based servers ignorecase key is not available.
Review Logs Online
Find a Relevant Log File
In case of a specific error message displayed in Finesse client, a search option case be used to determine the exact timestamp and the log file.
admin:file search activelog /desktop/logs/ "100011" recurs ignorecase
Searching path: /var/log/active//desktop/logs/
/var/log/active//desktop/logs/realm/realm.log:1289383032: Nov 18 2016 08:14:15.076 +0100:
[http-bio-8445-exec-10] INFO CCRealmConfig.finesseAuthenticate(): Unable to authenticate user: 100011
Search completed
The output shows the exact log file, timestamp with msec precision and the error message generated "Unable to authenticate user: 100011".
Review the File Online
Next step after you find relevant log file is to review it. And use file dump option for big log files is not very beneficial. Instead file view can be used, so you can navigate straight to the end of the file.
admin:file view activelog /desktop/logs/realm/realm.log
0: Aug 11 2016 15:52:14.423 +0200: [localhost-startStop-1] INFO
Loaded realm log config file at: /opt/cisco/desktop/conf/realm/log4j.xml
55577: Aug 11 2016 15:53:10.000 +0200: [Infrastructure Initialization, completed] INFO
Loaded realm log config file at: /opt/cisco/desktop/conf/realm/log4j.xml
options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 374) :
956921239: Nov 14 2016 11:53:13.283 +0100: [http-bio-8445-exec-12] INFO
CCRealmConfig.finesseAuthenticate(): Unable to authenticate user: 1012
1206572479: Nov 17 2016 09:14:04.523 +0100: [http-bio-8082-exec-6] INFO
CCRealmConfig.finesseAuthenticate(): Unable to authenticate user: MEDIASENSE_API
1217996207: Nov 17 2016 12:24:28.251 +0100: [http-bio-8082-exec-10] INFO
CCRealmConfig.finesseAuthenticate(): Unable to authenticate user: MEDIASENSE_API
1289292089: Nov 18 2016 08:12:44.133 +0100: [http-bio-8445-exec-22] INFO
CCRealmConfig.finesseAuthenticate(): Unable to authenticate user: 5678
1289383032: Nov 18 2016 08:14:15.076 +0100: [http-bio-8445-exec-10] INFO
CCRealmConfig.finesseAuthenticate(): Unable to authenticate user: 100011
end of the file reached
options: q=quit, n=next, p=prev, b=begin, e=end (lines 361 - 374 of 374) :
Browse Directory in VOS
Can be used for multiple reasons. Amongst them are checks that the log file is generated to prove the process runs, check logs size for disk clean up or self-study.
This command will give a list of log files filtered by the name provided. Option detail displays date and size. An additional key date sorts the output based on the time files were generated.
admin:file list activelog /desktop/logs/webservices/*2016-08-12* det date
12 Aug,2016 07:11:33 1,753,064 Desktop-webservices.2016-08-12T01-48-58.164.log.zip
12 Aug,2016 09:26:17 1,125,984 Error-Desktop-webservices.2016-08-12T03-35-30.164.log.zip
12 Aug,2016 12:34:08 1,741,476 Desktop-webservices.2016-08-12T07-11-33.664.log.zip
12 Aug,2016 15:17:05 1,125,790 Error-Desktop-webservices.2016-08-12T09-26-17.664.log.zip
12 Aug,2016 17:56:43 1,741,988 Desktop-webservices.2016-08-12T12-34-08.664.log.zip
12 Aug,2016 21:07:52 1,125,050 Error-Desktop-webservices.2016-08-12T15-17-05.164.log.zip
12 Aug,2016 23:19:18 1,753,089 Desktop-webservices.2016-08-12T17-56-43.663.log.zip
13 Aug,2016 02:58:40 1,125,382 Error-Desktop-webservices.2016-08-12T21-07-52.664.log.zip
13 Aug,2016 04:41:53 1,742,441 Desktop-webservices.2016-08-12T23-19-18.664.log.zip
dir count = 0, file count = 9
Check Platform History
Basic diagnostic information about the server upgrades, Cisco Options Package (COP) file installations and reboots is very helpful. Especially during the troubleshoot process.
admin:file dump install system-history.log
=======================================
Product Name - Cisco Finesse
Product Version - 11.0.1.10000-24
Kernel Image - 2.6.32-431.20.3.el6.x86_64
=======================================
08/11/2016 12:46:19 | root: Install 11.0.1.10000-24 Start
08/11/2016 06:02:15 | root: Boot 11.0.1.10000-24 Start
08/11/2016 15:45:52 | root: Install 11.0.1.10000-24 Success
08/11/2016 15:45:53 | root: Boot 11.0.1.10000-24 Start
08/26/2016 12:50:14 | root: Cisco Option Install finesse-cce.1101.ES05.10000.cop Start
08/26/2016 13:25:51 | root: Cisco Option Install finesse-cce.1101.ES05.10000.cop Success
08/26/2016 13:51:14 | root: Restart 11.0.1.10000-24 Start
08/26/2016 13:51:40 | root: Boot 11.0.1.10000-24 Start
10/19/2016 17:16:48 | root: Shutdown 11.0.1.10000-24 Start
10/20/2016 12:17:10 | root: Boot 11.0.1.10000-24 Start
Download Logs
In case there is a need to pull the logs from the system for further analysis, then this command can be used. It is irreplaceble when Cisco Tomcat web server is down and Real-Time Monitoring Tool (RTMT) can not be used.
You can pull the logs based on time filter. In this example for the last two hours.
admin:file get activelog /desktop/logs/webservices/ reltime hours 2 compress
This command can take significantly long time,
and can also affect the system wide IOWAIT on your system.
Would you like to proceed [y/n]?y
Please wait while the system is gathering files info ...done.
Sub-directories were not traversed.
Number of files affected: 1
Total size in Bytes: 3460015
Total size in Kbytes: 3378.921
Would you like to proceed [y/n]? y
SFTP server IP:
Also, logs can be pulled based on the name filter. In this example, only files with log extension are pulled from install directory. The files are not compressed.
admin:file get install /*.log
Please wait while the system is gathering files info ...done.
Sub-directories were not traversed.
Number of files affected: 10
Total size in Bytes: 2640585
Total size in Kbytes: 2578.6963
Would you like to proceed [y/n]? n
Files transfer cancelled.
Another useful case is to collect the logs for a root cause analysis for a specific time range.
admin:file get activelog /desktop/logs/ abstime 09:00:11/13/16 09:00:11/14/16 recurs
Please wait while the system is gathering files info ...done.
Sub-directories were traversed.
Number of files affected: 2
Total size in Bytes: 7446444
Total size in Kbytes: 7271.918
Would you like to proceed [y/n]?
Note: Ensure to verify total size value and if it is too big either make lesser time range or put more specific directory.
Feedback