The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This chapter provides information about Trace and Log Central.
The Trace and Log Central feature in RTMT allows you to configure on-demand trace collection for a specific date range or an absolute time. You can collect trace files that contain search criteria that you specify and save the trace collection criteria for later use, schedule one recurring trace collection and download the trace files to a SFTP or FTP server on your network, or collect a crash dump file.
Note | From Cisco Unified Serviceability, you can also edit the trace setting for the traces on the node that you have specified. Enabling trace settings decreases system performance; therefore, enable Trace only for troubleshooting purposes. |
After you collect the files, you can view them in the appropriate viewer within the real-time monitoring tool. You can also view traces on the node without downloading the trace files by using the remote browse feature. You can open the trace files by either selecting the internal viewer that is provided with Unified RTMT or choosing an appropriate program as an external viewer.
Follow this procedure to import the node certificates.
You can import the server authentication certificate that the certificate authority provides for the node or for each node in the cluster.
We recommend that you import the certificates before using the trace and log central option. If you do not import the certificates, the Trace and Log Central option displays a security certificate for the nodes each time that you sign in to Unified RTMT and access the Trace and Log Central option. You cannot change any data that displays for the certificate.
Follow this procedure to view Trace and Log Central options in Unified RTMT.
Note | From any option that displays in the tree hierarchy, you can specify the services and applications for which you want traces, specify the logs and servers that you want to use, schedule a collection time and date, configure the ability to download the files, configure zip files, and delete collected trace files. |
Note | For devices that support encryption, the SRTP keying material does not display in the trace file. |
The Collect Files option in Trace and Log Central collects traces for services, applications, and system logs on the server or on one or more servers in the cluster.
Note | The services that you have not activated also appear, so you can collect traces for those services. |
RTMT supports the throttling of critical Trace and Log Central operations and jobs, whether they are running on demand, scheduled, or automatic. The throttling slows the operations when I/O utilization is in high demand for call processing, so call processing can take precedence.
When you make a request for an on-demand operation when the call processing node is running under high I/O conditions, the system displays a warning that gives you the opportunity to abort the operation. You can configure the I/O rate threshold values that control when the warning displays with the following service parameters (in Cisco RIS Data Collector service):
The system compares the values of these parameters against the actual system CPU and IOWait values. If the goal (the value of the service parameter) is lower than the actual value, the system displays the warning.
This feature enables the ROS (Recoverable Outstream) library to support the compressed output of tracefiles. The files get compressed as they are being generated. The benefits of tracefile compression include the following:
Use the enterprise parameter Trace Compression to enable or disable trace compression. The default value for this parameter is Disabled. For information about setting the values of enterprise parameters, see the "Enterprise Parameters Configuration" chapter in the System Configuration Guide for Cisco Unified Communications Manager .
Caution | Compressing files adds additional CPU cycles. Enabling the Trace Compression enterprise parameter can negatively affect overall call throughput by as much as 10 percent. |
You can recognize compressed files by their .gz extension (.gzo if the file is still being written to). To open a compressed file, double-click the filename, and the file opens in the log viewer.
Use the Collect Files option in Trace and Log Central to collect traces for services, applications, and system logs on one or more nodes in the cluster. You specify date and time range for which you want to collect traces, the directory in which to download the trace files and whether to delete the collected files from the node.
Follow this procedure to collect traces using the trace and log central feature.
Note | The services that you have not activated also appear, so you can collect traces for those services. |
Use the Query Wizard if you want to collect trace files that contain search criteria that you specify or you want to use trace collection criteria that you saved for later use.
Perform one or more of the following actions:
Follow this procedure to collect installation and upgrade logs.
Step 1 | Choose
.
The Trace & Log Central window appears. |
Step 2 | In the Trace
& Log Central tree hierarchy, double-click
Collect
Install Logs.
The Collect Install Logs wizard appears. |
Step 3 | In the Select Servers Options box, specify from which server you would like to collect the install logs. |
Step 4 | In the Download File Options, specify the directory where you want to download the log file. To specify the directory in which you want to download the log files, click Browse next to the Download File Directory field, navigate to the directory, and click Open. The default specifies <rtmt_users_directory>. |
Step 5 | Click Finish. |
The Trace Collection Query Wizard allows you to collect and download trace files that contain search criteria that you specify as well as to save trace collection criteria for later use. To use the Trace Collection Query Wizard, perform the following procedure.
Note | You can open a maximum of five concurrent files for viewing within Trace and Log Central. This includes using the Query Wizard, Local Browse, and Remote Browse features. |
Perform one or more of the following tasks:
Step 1 | Display the Trace and Log Central options. | ||||||
Step 2 | In the Trace & Log Central tree hierarchy, double-click Query Wizard. The Query wizard appears. | ||||||
Step 3 | In the Query Wizard Options window, click one of the following radio buttons, and then click Next to modify the query.
| ||||||
Step 4 | If you clicked the Saved Query radio button and chose a query, the criteria that you specified for query display. If necessary, modify the list of services/applications for which you want to collect traces. If you clicked the Create Query radio button, you must choose all services/applications for which you want to collect traces.
| ||||||
Step 5 | Cisco Unity Connection users go to Step 7. For Unified Communications Manager or Cisco Business Edition, choose the services and application logs in which you are interested by checking all check boxes that apply in the Select CallManager Services/Applications tab. | ||||||
Step 6 | Click Next. Unified Communications Manager users go to Step 8. | ||||||
Step 7 | In the Select CUC Services/Application tab, choose the services and application logs in which you are interested by checking all check boxes that apply, then click Next. | ||||||
Step 8 | In the Select System Logs tab, choose the logs in which you are interested by checking all check boxes that apply, then click Next. | ||||||
Step 9 | In the Query Time Options box, specify the time range for which you want to collect traces. Choose one of the following options:
| ||||||
Step 10 | To search by phrases or words that exist in the trace file, enter the word or phrase in the Search String field. If you want to search for an exact match to the word or phrase that you entered, check the Case Sensitive check box. | ||||||
Step 11 | In the Call Processing Impact Options box, specify the level of impact you want the string search activity to have on call processing, then click Next. From the Select Impact Level drop down list box, select Low, Medium, or High. Low impact causes the least impact on call processing but yields slower results. High impact causes the most impact on call processing but yields faster results. | ||||||
Step 12 | In the Action Options window, choose one of the following actions: | ||||||
Step 13 | Choose one of the following options: | ||||||
Step 14 | Check the check box next to the type of query that you want to create from the following options, and then click Finish. | ||||||
Step 15 | Browse to the location to store the query, enter a name for the query in the File Name field, and click Save. | ||||||
Step 16 | Perform one of the following actions:
| ||||||
Step 17 | After the query execution completes, perform one or more of the following tasks: |
You can use the Schedule Collection option of the trace and log central feature to schedule up to six concurrent trace collections and to download the trace files to a SFTP or FTP server on your network, run another saved query, or generate a syslog file. To change a scheduled collection after you have entered it in the system, you must delete the scheduled collection and add a new collection event. To schedule trace collection, perform the following procedure.
Note | You can schedule up ten trace collection jobs, but only six trace collection can be concurrent. That is, only six jobs can be in a running state at the same time. |
Perform one or more of the following actions:
Step 1 | Open the Trace and Log Central options. | ||||||
Step 2 | In the Trace & Log Central tree hierarchy, double-click Schedule Collection. The Schedule Collection wizard appears.
| ||||||
Step 3 | Cisco Unity Connection users go to Step 4. For Unified Communications Manager or Cisco Business Edition, perform one of the following actions in the Select CCM Services/Application tab:
For Cisco Business Edition, go to Step 4. For Unified Communications Manager, go to Step 5. | ||||||
Step 4 | In the Select CUC Services/Application tab, perform one of the following actions:
| ||||||
Step 5 | In the Select System Services/Application tab, perform one of the following actions:
| ||||||
Step 6 | Specify the node time zone and the time range for which you want to collect traces. The time zone of the client machine provides the default setting for the Select Reference Server Time Zone field. All the standard time zones, along with a separate set of entries for all time zones that have Daylight Saving settings, appear in the Select Time Zone drop-down list box. | ||||||
Step 7 | To specify the date and time that you want to start the trace collection, click the down arrow button next to the Schedule Start Date/Time field. In the Date tab, choose the appropriate date. In the Time tab, choose the appropriate time. | ||||||
Step 8 | To specify the date and time that you want to end the trace collection, click the down arrow button next to the Schedule End Date/Time field. In the Date tab, choose the appropriate date. In the Time tab, choose the appropriate time.
| ||||||
Step 9 | From the Scheduler Frequency drop-down list box, choose how often you want to run the configured trace collection. | ||||||
Step 10 | From the Collect Files that are generated in the last drop-down list boxes, specify the time (in minutes, hours, days, weeks, or months) prior to the current time for which you want to collect traces. | ||||||
Step 11 | To search by phrases or words that exist in the trace file, enter the word or phrase in the Search String field. The tool searches for a match to the word or phrase that you enter and collects those files that match the search criteria. If you want to search for an exact match to the word or phrase that you entered, check the Case Sensitive check box | ||||||
Step 12 | To create a zip file of the trace files that you collect, check the Zip File check box. | ||||||
Step 13 | To delete collected log files from the node, check the Delete Collected Log Files from the Server check box. | ||||||
Step 14 | Choose one or more of the following actions: | ||||||
Step 15 | In the SFTP/FTP Server Parameters group box, enter the node credentials for the node where the trace and log central feature downloads the results and click Test Connection. After the trace and log central feature verifies the connection to the SFTP or FTP server, click OK. The Download Directory Path field specifies the directory in which the trace and log central feature stores collected files. By default, the trace collection stores the files in the home directory of the user whose user ID you specify in the SFTP or FTP parameters fields: /home/<user>/Trace. You can choose Localhost download option when downloading traces. This option is available only for Cisco Intercompany Media Engine servers. If you download trace files to the local host directories on the Cisco Intercompany Media Engine server, you can offload the files to a remote SFTP server by using the file get CLI command.
| ||||||
Step 16 | If you chose the Run Another Query Option, click the Browse button to locate the query that you want to run, and click OK.
| ||||||
Step 17 | Click Finish. A message indicates that the system added the scheduled trace successfully.
| ||||||
Step 18 | Click OK. | ||||||
Step 19 | To view a list of scheduled collections, click the Job Status icon in the Trace portion of the Quick Launch Channel.
|
Follow this procedure to view the trace collection event status and to delete scheduled trace collections.
Step 1 | Open the Trace & Log Central tree hierarchy. | ||
Step 2 | Double-click Job Status. The Job Status window appears. | ||
Step 3 | From the Select a Node drop-down list box, choose the server for which you want to view or delete trace collection events. This list of scheduled trace collections appears. | ||
Step 4 | To delete a scheduled collection, choose the event that you want to delete and click Delete.
|
Follow this procedure to collect a core dump of trace files:
Step 1 | Open the Trace & Log Central tree hierarchy. | ||||||
Step 2 | Double-click Collect Crash Dump. The Collect Crash Dump wizard appears.
| ||||||
Step 3 | Cisco Unity Connection users go to Step 4. For Unified Communications Manager or Cisco Business Edition, perform one of the following actions in the Select CCM Services/Application tab:
Go to Step 4 for Cisco Business Edition or go to Step 5 for Unified Communications Manager. | ||||||
Step 4 | In the Select CUC Services/Application tab, perform one of the following actions:
| ||||||
Step 5 | In the Select System Services/Application tab, perform one of the following actions:
| ||||||
Step 6 | In the Collection Time group box, specify the time range for which you want to collect traces. Choose one of the following options:
| ||||||
Step 7 | From the Select Partition drop-down list box, choose the partition that contains the logs for which you want to collect traces. Cisco Unified Serviceability stores the logs for the version of application that you are logged in to in the active partition and stores the logs for the other version (if installed) in the inactive directory. When you upgrade from one version of your product that is running on the Linux platform to another version, and you restart the node with the new version, Cisco Unified Serviceability moves the logs of the previous version to the inactive partition and stores logs for the newer version in the active partition. If you log in to the older version, Cisco Unified Serviceability moves the logs for the newer version to the inactive partition and stores the logs for the older version in the active directory.
| ||||||
Step 8 | To specify the directory in which you want to download the trace files, click the Browse button next to the Download File Directory field, navigate to the directory, and click Open. The default specifies <rtmt_install_directory>\<server name or server IP address>\<download time> where <rtmt_install_directory> specifies the directory where RTMT is installed. | ||||||
Step 9 | To create a zip file of the crash dump files that you collect, choose the Zip File radio button. To download the crash dump files without zipping the files, choose the Do Not Zip Files radio button.
| ||||||
Step 10 | To delete collected crash dump files from the node, check the Delete Collected Log Files from Server check box. | ||||||
Step 11 | Click Finish. A message appears that states that you want to collect core dumps. To continue, click Yes.
|
The audit user can collect, view, and delete the audit logs. The end user can view the audit logs.
Note | Only a user with an audit role can delete the audit logs. |
Step 1 | Open the Trace & Log Central tree hierarchy. | ||||||||||||
Step 2 | Double-click Collect Audit Logs. The Collect Audit Logs Action Options wizard appears. | ||||||||||||
Step 3 | Perform one of the following actions in the Action Options window: | ||||||||||||
Step 4 | Click Next. The Nodes Selection Options wizard appears. | ||||||||||||
Step 5 | Perform one of the following actions in the Action Options window:
| ||||||||||||
Step 6 | Click Finish. Proceed with one of the following selections: | ||||||||||||
Step 7 | The Remote Browse is Ready window appears. Click the Close button. The Nodes pane appears. | ||||||||||||
Step 8 | On the left side of the Nodes pane, double-click the Nodes folder. Navigate through the tree hierarchy until the Audit App folder appears. | ||||||||||||
Step 9 | After the audit log file names appear in the pane on the right side of the window, you can either right-click the mouse to select the type of program that you would like to use to view each file or double-click the selected file to display the file in the default viewer. | ||||||||||||
Step 10 | Select an audit log file and perform one of the following actions: | ||||||||||||
Step 11 | To download audit logs, click Next. The Download Audit Logs window appears. | ||||||||||||
Step 12 | In the Nodes Selection Options pane, select one of the following: | ||||||||||||
Step 13 | In the Collection Time pane, select one of the following radio buttons:
| ||||||||||||
Step 14 | In the Download File Options pane, select one of the following options: | ||||||||||||
Step 15 | Click Finish. You have completed the steps for the download of audit logs. | ||||||||||||
Step 16 | The Schedule Download of Audit Logs window appears. |
After you collect trace files and download them to your PC, you can view them with a text editor that can handle UNIX variant line terminators such as WordPad on your PC, or you can view them by using the viewers within Unified RTMT.
Tip | Do not use NotePad to view collected trace files. |
Follow this procedure to display the log files that you collected with the Trace and Log Central feature. If you zipped the trace files when you downloaded them to your PC, you need to unzip the files to view them by using the viewers within Unified RTMT.
Note | You can open a maximum of five concurrent files for viewing within Trace and Log Central, which includes using the Query Wizard, Local Browse, and Remote Browse features. |
Collect the required traces files. See topics related to collecting trace files, downloading trace files using Query Wizard, and scheduling trace collection for instructions.
Step 1 | Open Trace and Log Central. |
Step 2 | Double-click Local Browse. |
Step 3 | Browse to the directory where you stored the log file and choose the file that you want to view. |
Step 4 | To display the results, double-click the file. If the file type has a viewer that is already associated with it, the file opens in that viewer. Otherwise, the Open With dialog box appears. |
Step 5 | Click the program that want to use to view the file. If your preferred program is not on the list, choose another program by clicking Other. If you want to use this program as your default viewer, check the Always use this program to open these files check box. Unified RTMT displays the file in the appropriate viewer for the file type. If no other appropriate viewer applies, Unified RTMT opens files in the Generic Log Viewer. |
After the system generates trace files, you can view them on the node by using the viewers within Unified RTMT. You can also use the remote browse feature to download the traces to your PC.
Follow this procedure to display and download the log files on the node with the Trace and Log Central feature.
Note | You can open a maximum of five concurrent files for viewing within Trace and Log Central. This includes using the Query Wizard, Local Browse, and Remote Browse features. |
Collect the required traces files. See topics related to collecting trace files, downloading trace files using Query Wizard, and scheduling trace collection.
Step 1 | Open the Trace and Log Central options. | ||||||
Step 2 | Double-click Remote Browse. | ||||||
Step 3 | Choose the appropriate radio button, and click Next.
| ||||||
Step 4 | Cisco Unity Connection users go to Step 5. For Unified Communications Manager or Cisco Business Edition, perform one of the following actions in the Select CCM Services/Application tab:
Go to Step 5 for Cisco Business Edition or go to Step 6 for Unified Communications Manager. | ||||||
Step 5 | In the Select CUC Services/Application tab, perform one of the following actions:
| ||||||
Step 6 | In the Select System Services/Application tab, perform one of the following actions:
| ||||||
Step 7 | Cisco Unity Connection users go to Step 8. For Unified Communications Manager or Cisco Business Edition, perform one of the following actions in the Select CCM Services/Application tab:
| ||||||
Step 8 | In the Select CUC Services/Application tab, perform one of the following tasks:
| ||||||
Step 9 | In the Select System Services/Application tab, perform one of the following tasks:
| ||||||
Step 10 | Click Finish. | ||||||
Step 11 | After the traces become available, a message appears. Click Close. | ||||||
Step 12 | Perform one of the following actions: |
You can view the QRT log files by either viewing the files on the server or by downloading the files onto your computer.
Note | This section applies only to Cisco Unified Communications Manager. |
You can view the IP phone problem reports that the Quality Report Tool generates by using the QRT viewer. QRT serves as a voice-quality and general problem-reporting tool for Cisco Unified IP Phones. After you collect the QRT log files, you can use the following procedure to list and view Cisco Unified Communications Manager IP Phone problem reports by using the QRT viewer. The QRT viewer allows you to filter, format, and view phone problem reports that are generated. For more information about how to configure and use QRT, see the System Configuration Guide for Cisco Unified Communications Manager .
Collect or view the Quality Report Tool (QRT) log files. See topics related to collecting trace files, scheduling trace collection, and downloading trace files using either Query Wizard or the Remote Browser.
Step 1 | Display the log
file entries by using the Query Wizard, the Remote Browse, or the Local Browse
option in Trace and Log Central.
The QRT Viewer window appears.
| ||||
Step 2 | From the Extension drop-down list box, choose the extension or extensions that you want the report to include. | ||||
Step 3 | From the Device drop-down list box, choose the device or devices that you want the report to include. | ||||
Step 4 | From the Category drop-down list box, choose the problem category that you want the report to include. | ||||
Step 5 | From the
Select
Fields drop-down list box, choose the fields that you want the
report to include.
| ||||
Step 6 | To view the report in the QRT Report Result pane, click Display Records. |
The real-time trace option of the Trace and Log Central feature allows you to view the current trace file that is being written on the server for each application. If the system has begun writing a trace file, the real-time trace starts reading the file from the point where you began monitoring rather than at the beginning of the trace file. You cannot read the previous content.
The real-time trace provides the option to view real-time data and monitor user events.
The view real-time data option of the trace and log central feature allows you to view a trace file as the system writes data to that file. You can view real-time trace data in the generic log viewer for up to ten services, with a limit of three concurrent sessions on a single node. The log viewer refreshes every 5 seconds. As the traces are rolled into a new file, the generic log viewer appends the content in the viewer.
Note | Depending on the frequency of the traces that a service writes, the View Real Time Data option may experience a delay before being able to display the data in the generic log viewer. |
Step 1 | Open the Trace & Log Central tree hierarchy. | ||||
Step 2 | Double-click Real Time Trace.
| ||||
Step 3 | Double-click View Real Time Data. The View Real Time Data wizard appears. | ||||
Step 4 | From the Nodes drop-down list box, choose the node for which you want to view real-time data and click Next. | ||||
Step 5 | Choose the product, service, and the trace file type for which you want to view real-time data.
| ||||
Step 6 | Click Finish. The real-time data for the chosen service displays in the generic log viewer. | ||||
Step 7 | Perform one of the following actions: | ||||
Step 8 | Repeat this procedure to view data for additional services. A message appears if you attempt to view data for too many services or too many services on a single node. | ||||
Step 9 | After you finish with viewing the real-time data, click Close on the generic log viewer.
|
The monitor user event option of the trace and log central feature monitors real-time trace files and performs a specified action when a search string appears in the trace file. The system polls the trace file every 5 seconds. If the search string occurs more than once in one polling interval, the system performs the action only once.
If you want to generate an alarm when the specified search string exists in a monitored trace file, enable the LogFileSearchStringFound alert.
Step 1 | Open the Trace & Log Central tree hierarchy. | ||||||||||||||||||
Step 2 | Double-click Real Time Trace.
| ||||||||||||||||||
Step 3 | Double-click Monitor User Event. The Monitor User Event wizard appears. | ||||||||||||||||||
Step 4 | Perform one of the following actions:
| ||||||||||||||||||
Step 5 | Choose the node that you want the system to monitor from the Nodes drop-down list box and click Next. | ||||||||||||||||||
Step 6 | Choose the product, service, and the trace file type that you want the system to monitor and click Next.
| ||||||||||||||||||
Step 7 | In the Search String field, specify the phrases or words that you want the system to locate in the trace files. The tool searches for an exact match to the word or phrase that you enter. | ||||||||||||||||||
Step 8 | Specify the node time zone and the time range (start and end date and time) for which you want the system to monitor trace files. The time zone of the client machine provides the default setting for the Select Reference Server Time Zone field. All the standard time zones, along with a separate set of entries for all time zones that have Daylight Saving settings, display in the Select Time Zone drop-down list box. Trace and Log Central downloads the files with a time range that is based on your Selected Reference Server Time Zone field. If you have nodes in a cluster in a different time zone, TLC adjusts for the time change and get files for the same period of time. For example, if you specify files from 9:00 a.m. to 10:00 a.m. and you have a second node (node x) that is in a time zone that is one hour ahead, TLC downloads files from 10:00 a.m. to 11:00 a.m. from node x. To set the date range for which you want to monitor traces, choose the drop-down list box in the From Date/Time and To Date/Time fields. | ||||||||||||||||||
Step 9 | Choose one or more of the following actions that you want the system to perform when it encounters the search string that you specified in the Search String field:
The system polls the trace files every 5 seconds and performs the specified actions when it encounters the search string. If more than one occurrence of the search string occurs in a polling interval, the system performs the action only once. The following message appears: If trace compression is enabled, there might be a delay in catching the event after it occurs, due to buffering of data. | ||||||||||||||||||
Step 10 | Click Finish. |
Follow this procedure to edit trace settings for Unified RTMT.
Note | The Error radio button is the default setting. |
In Unified Communications Manager 8.0 onward, the log compression feature only compresses the following log files:
The other log files are not compressed and are written directly to the hard disk.
The compressed files have a .gz extension. The file that is being actively written to the disk will have a .gzo extension.
All the CLI commands used to view and tail the files will work on the compressed files and will automatically uncompress them for viewing or tailing. The only difference is in specifying file names with the .gz and .gzo extension.
The following option is available with the file tail command:
file tail activelog cm/trace/cti/sdl recent
The recent option, when used with a compressed directory, continually tails the most recent log file. You do not need to switch to a newer log file when the currently written-to log file is closed, so it is an infinite and ongoing tail. This option is only available with the compressed log files.
The log files are compressed in the gzip format. For uncompressing the log files, the open source program 7-Zip is available at http://www.7-zip.org, and works on all Windows platforms. You can use 7-Zip on any computer, including a computer in a commercial organization. You do not need to register or pay for 7-Zip. On a Linux platform, you can use the gzip or gunzip commands.