The system generate alert messages to notify administrator when a predefined condition is met, such as when an activated service goes from up to down. The system can send alerts as e-mail/epage.
RTMT, which supports alert defining, setting, and viewing, contains preconfigured and user-defined alerts. Although you can perform configuration tasks for both types, you cannot delete preconfigured alerts (whereas you can add and delete user-defined alerts). The Alert menu comprises the following menu options:
Alert Central—This option comprises the history and current status of every alert in the system.
You can also access Alert Central by clicking the Alert Central icon in the hierarchy tree in the system drawer.
Set Alert/Properties—This menu category allows you to set alerts and alert properties.
Remove Alert—This menu category allows you to remove an alert.
Enable Alert—With this menu category, you can enable alerts.
Disable Alert—You can disable an alert with this category.
Suspend cluster/node Alerts—This menu category allows you to temporarily suspend alerts on a particular server or on an entire cluster (if applicable).
Clear Alerts—This menu category allows you to reset an alert (change the color of an alert item to black) to signal that an alert has been handled. After an alert has been raised, its color will automatically change in RTMT and will stay that way until you manually clear the alert.
The manual clear alert action does not update the System cleared timestamp column in Alert Central. This column is updated only if alert condition is automatically cleared.
Clear All Alerts—This menu category allows you to clear all alerts.
Reset all Alerts to Default Config—This menu category allows you to reset all the alerts to the default configuration.
Alert Detail—This menu category provides detailed information on alert events.
Config Email Server—In this category, you can configure your e-mail server to enable alerts.
To configure RTMT to send alerts via e-mail, you must configure DNS. For information on configuring the primary and secondary DNS IP addresses and the domain name in Cisco Unified Communications Manager Server Configuration, see the "DHCP Server Configuration" chapter in the Cisco Unified Communications Manager Administration Guide.
Config Alert Action—This category allows you to set actions to take for specific alerts; you can configure the actions to send the alerts to desired e-mail recipients.
In RTMT, you configure alert notification for perfmon counter value thresholds and set alert properties for the alert, such as the threshold, duration, frequency, and so on. RTMT predefined alerts are configured for perfom counter value thresholds as wells as event (alarms) notifications.
You can locate Alert Central under the Tools hierarchy tree in the quick launch. Alert Central provides both the current status and the history of all the alerts in the system.
Alert Central Displays
Unified RTMT displays both preconfigured alerts and custom alerts in Alert Central. Unified RTMT organizes the alerts under the applicable tabs: System, CallManager, Cisco Unity Connection, and Custom.
You can enable or disable preconfigured and custom alerts in Alert Central; however, you cannot delete preconfigured alerts.
The following list comprises the preconfigured system alerts:
ServerDown (Applies to Unified Communications Manager clusters)
Because none of the audit events is alert worthy, the CriticalAuditEventGenerated alert cannot be triggered.
The following list comprises the preconfigured CallManager alerts.
The following list comprises the
Connection alerts. These alerts apply only to
Cisco Business Edition
Brain Resolution Failed)
The first six
alerts apply only to
Connection cluster configurations.
Business Edition 5000 does not support a
Connection cluster configuration.
You can configure both preconfigured and user-defined alerts in Unified RTMT. You can also disable both preconfigured and user-defined alerts in Unified RTMT. You can add and delete user-defined alerts in the performance-monitoring window; however, you cannot delete preconfigured alerts.
Severity levels for Syslog entries match the severity level for all Unified RTMT alerts. If Unified RTMT issues a critical alert, the corresponding Syslog entry also specifies critical.
The following table provides a list of fields that you may use to configure each alert; users can configure preconfigured fields, unless otherwise noted.
Table 1 Alert Customization
High-level name of the monitoring item with which Unified RTMT associates an alert
Descriptive name. For preconfigured alerts, you cannot change this field. See topics related to Alert Central displays for a list of preconfigured alerts.
Description of the alert
You cannot edit this field for preconfigured alerts. See topics related to Alert Central displays for a list of preconfigured alerts.
Source of the performance counter
You cannot change this field. You can associate only one instance of the performance counter with an alert.
Condition to raise alert (value is...)
Specify up < - > down, less than #, %, rate greater than #, %, rate. This field is applicable only for alerts based on performance counters.
Value Calculated As
Method used to check the threshold condition
Specify value to be evaluated as absolute, delta (present - previous), or % delta. This field is applicable only for alerts based on performance counters.
Condition to raise alert (how long value threshold has to persist before raising alert)
Options include the system sending the alert immediately or after a specified time that the alert has persisted. This field is applicable only for alerts based on performance counters.
Number of Events Threshold
Raise alert only when a configurable number of events exceeds a configurable time interval (in minutes).
For ExcessiveVoiceQualityReports, the default thresholds equal 10 to 60 minutes. For RouteListExhausted and MediaListExhausted, the defaults equal 0 to 60 minutes. This field is applicable only for event based alerts.
(Applies to Unified Communications Manager)
Cluster or list of servers to monitor
Cisco Unified Communications Manager servers, Cisco TFTP server, or first server. This field is applicable only for non-clusterwide alerts.
When you deactivate both the Cisco CallManager and Cisco TFTP services of a server, the system considers that server as removed from the currently monitored server list. When you reactivate both Cisco CallManager and Cisco TFTP services, that server is added back, and its settings are restored to default values.
Alert Action ID
ID of alert action to take (System always logs alerts no matter what the alert action.)
Alert action is defined first (see the Alert Customization topic). A blank field indicates that e-mail is disabled.
Enable or disable alerts.
Options include enabled or disabled.
Resets alert (change the color of an alert item from red to black) to signal that the alert is resolved
After an alert is raised, its color automatically changes to black and remains until you manually clear the alert. Use Clear All to clear all alerts.
(Applies to Unified Communications Manager)
Displays the detail of an alert (not configurable)
For ExcessiveVoiceQualityReports, RouteListExhausted, and MediaListExhausted, up to 30 current event details display in the current monitoring interval if an alert is raised in the current interval. Otherwise, the previous 30 event details in the previous interval displays. For DChannel OOS alert, the list of outstanding OOS devices at the time the alert was raised appears.
Alert Generation Rate
How often to generate alert when alert condition persists
Specify every X minutes. (Raise alert once every X minutes if condition persists.)
Specify every X minutes up to Y times. (Raise alert Y times every X minutes if condition persists.)
User Provide Text
Administrator to append text on top of predefined alert text
For viewing purposes (for example, show only Sev. 1 alerts)
Specify defaults that are provided for predefined (for example, Error, Warning, Information) alerts.
Alert Action Setup
In RTMT, you can configure alert actions for every alert that is generated and have the alert action sent to e-mail recipients that you specify in the alert action list.
The following table provides a list of fields that you will use to configure alert actions. Users can configure all fields, unless otherwise marked.
Table 2 Alert Action Configuration
Alert Action ID
ID of alert action to take.
Specify descriptive name.
List of e-mail addresses. You can selectively enable or disable an individual e-mail in the list.
Automatic Trace Download Activation
Some preconfigured alerts allow you to initiate a trace download based on the occurrence of an event. You can automatically capture traces when a particular event occurs by checking the Enable Trace Download check box in Set Alert/Properties for the following alerts:
CriticalServiceDown: CriticalServiceDown alert is generated when any service is down. CriticalServiceDown alert monitors only those services that are listed in RTMT Critical Services.
The Unified RTMT backend service checks status (by default) every 30 seconds. If service goes down and comes back up within that period, CriticalServiceDown alert may not be generated.
CodeYellow: This alarm indicates that Cisco Unified Communications Manager initiated call throttling due to unacceptably high delay in handling calls.
CoreDumpFileFound: CoreDumpFileFound alert is generated when the Unified RTMT backend service detects a new Core Dump file.
You can configure both CriticalServiceDown and CoreDumpFileFound alerts to download corresponding trace files for troubleshooting purposes. This setup helps preserve trace files at the time of crash.
Trace Download may affect services on the node. A high number of downloads adversely impacts the quality of services on the node.
The alert log stores the alert, which is also stored in memory. The memory is cleared at a constant interval, leaving the last 30 minutes of data in the memory. When the service starts or restarts, the last 30 minutes of the alert data load into the memory by the system reading from the alert logs on the server or on all servers in the cluster (if applicable). The alert data in the memory is sent to the RTMT clients on request.
Upon RTMT startup, RTMT shows all logs that occurred in the last 30 minutes in the Alert Central log history. The alert log is periodically updated, and new logs are inserted into the log history window. After the number of logs reaches 100, RTMT removes the oldest 40 logs.
The following filename format for the alert log applies: AlertLog_MM_DD_YYYY_hh_mm.csv.
The alert log includes the following attributes:
Time Stamp: Time when RTMT logs the data
Alert Name: Descriptive name of the alert
Node: Server name for where RTMT raised the alert
Alert Message: Detailed description about the alert
Type: Type of the alert
Description: Description of the monitored object
Severity: Severity of the alert
PollValue: Value of the monitored object where the alert condition occurred
Action: Alert action taken
Group ID: Identifies the source of the alert
The first line of each log file comprises the header. Details of each alert are written in a single line, separated by a comma.
Log Partition Monitoring Tool
Log Partition Monitoring (LPM), which is installed automatically with the system, uses configurable thresholds to monitor the disk usage of the log partition on a server. The Cisco Log Partition Monitoring Tool service starts automatically after installation of the system.
Every 5 minutes, Log Partition Monitoring uses the following configured thresholds to monitor the disk usage of the log partition and the spare log partition on a server:
LogPartitionLowWaterMarkExceeded (% disk space): When the disk usage is above the percentage that you specify, LPM sends out an alarm message to syslog and an alert to RTMT Alert central. To save the log files and regain disk space, you can use trace and log central option in RTMT.
LogPartitionHighWaterMarkExceeded (% disk space): When the disk usage is above the percentage that you specify, LPM sends an alarm message to syslog and an alert to RTMT Alert central.
SparePartitionLowWaterMarkExceeded (% disk space): When the disk usage is above the percentage that you specify, LPM sends out an alarm message to syslog and an alert to RTMT Alert central. To save the log files and regain disk space, you can use trace and log central option in RTMT.
SparePartitionHighWaterMarkExceeded (% disk space): When the disk usage is above the percentage that you specify, LPM sends a n alarm message to syslog and an alert to RTMT Alert central.
In addition, Cisco Log Partitioning Monitoring Tool service checks the server every 5 seconds for newly created core dump files. If new core dump files exist, Cisco Log Partitioning Monitoring Tool service sends a CoreDumpFileFound alarm and an alert to Alert Central with information on each new core file.
To utilize log partition monitor, verify that the Cisco Log Partitioning Monitoring Tool service, a network service, is running on Cisco Unified Serviceability on the server or on each server in the cluster (if applicable). Stopping the service causes a loss of feature functionality.
When the log partition monitoring services starts at system startup, the service checks the current disk space utilization. If the percentage of disk usage is above the low water mark, but less than the high water mark, the service sends a alarm message to syslog and generates a corresponding alert in RTMT Alert central.
To configure Log Partitioning Monitoring, set the alert properties for the LogPartitionLowWaterMarkExceeded and LogPartitionHighWaterMarkExceeded alerts in Alert Central.
To offload the log files and regain disk space on the server, you should collect the traces that you are interested in saving by using the Real-Time Monitoring tool.
If the percentage of disk usage is above the high water mark that you configured, the system sends an alarm message to syslog, generates a corresponding alert in RTMT Alert Central, and automatically purges log files until the value reaches the low water mark.
Log Partition Monitoring automatically identifies the common partition that contains an active directory and inactive directory. The active directory contains the log files for the current installed version of the software (Cisco Unified Communications Manager or Cisco Unity Connection), and the inactive directory contains the log files for the previous installed version of the software. If necessary, the service deletes log files in the inactive directory first. The service then deletes log files in the active directory, starting with the oldest log file for every application until the disk space percentage drops below the configured low water mark. The service does not send an e-mail when log partition monitoring purges the log files.
After the system determines the disk usage and performs the necessary tasks (sending alarms, generating alerts, or purging logs), log partition monitoring occurs at regular 5 minute intervals.
Access Alert Central and Set Up Alerts
By using the following procedure, you can perform tasks, such as access Alert Central, sort alert information, enable, disable, or remove an alert, clear an alert, or view alert details.
Perform one of the following tasks:
On the Quick Launch Channel, do the following:
In the tree hierarchy, double-click Tools.
Click the Alert Central icon.
Choose System > Tools > Alert > Alert Central.
The Alert Central monitoring window displays and shows the alert status and alert history of the alerts that the system has generated.
Perform one of the following tasks:
Set alert properties.
Configure e-mails for alert notification.
Configure alert actions.
Sort alert information in the Alert Status pane. Click the up/down arrow that displays in the column heading.
For example, click the up/down arrow that displays in the Enabled or In Safe Range column.
You can sort alert history information by clicking the up/down arrow in the columns in the Alert History pane. To see alert history that is out of view in the pane, use the scroll bar on the right side of the Alert History pane.
To enable, disable, or remove an alert, perform one of the following tasks:
From the Alert Status window, right-click the alert and choose Disable/Enable Alert (option toggles) or Remove Alert, depending on what you want to accomplish.
Highlight the alert in the Alert Status window and choose System > Tools > Alert > Disable/Enable (or Remove) Alert.
You can remove only user-defined alerts from RTMT. The Remove Alert option appears grayed out when you choose a preconfigured alert.
To clear either individual or collective alerts after they get resolved, perform one of the following tasks:
After the Alert Status window displays, right-click the alert and choose Clear Alert (or Clear All Alerts).
Highlight the alert in the Alert Status window and choose System > Tools > Alert > Clear Alert (or Clear All Alerts).
After you clear an alert, it changes from red to black.
To reset alerts to default configuration, perform one of the following tasks:
After the Alert Status window displays, right-click the alert and choose Reset Alert to Default Config, to reset that alert to the default configuration.
Choose System > Tools > Alert > Reset all Alerts to Default Config, to reset all the alerts to the default configuration.
To view alert details, perform one of the following tasks:
After the Alert Status window displays, right-click the alert and choose Alert Details.
Highlight the alert in the Alert Status window and choose System > Tools > Alert > Alert Details.
After you have finished viewing the alert details, click OK.
Set Up Alert Properties
The following procedure describes how to set alert properties.
Open Alert Central.
From the Alert Status window, click the alert for which you want to set alert properties.
Perform one of the following actions:
Right-click the alert and choose Set Alert/Properties.
Choose System > Tools > Alert > Set Alert/Properties.
For clusterwide alerts, the Enable/Disable this alert on following server(s) box does not show up in the alert properties window. Clusterwide alerts include number of registered phones, gateways, media devices, route list exhausted, media list exhausted, MGCP D-channel out of service, malicious call trace, and excessive quality reports.
To enable the alert, check the Enable Alert check box.
From the Severity drop-down list box, choose the severity of the alert.
From the Enable/Disable this alert on following server(s) pane, check the Enable check box of the servers on which you want this alert to be enabled.
For preconfigured alerts, the Description information pane displays a description of the alert.
In the Threshold pane, enter the conditions in which the system triggers the alert.
In the Duration pane, click one of the following radio buttons:
Trigger alert only when below or over. . . radio button—If you want the alert to be triggered only when the value is constantly below or over the threshold for a specific number of seconds; then, enter the seconds.
Trigger alert immediately—If you want the system to trigger an alert immediately.
In the Frequency pane, click one of the following radio buttons:
Trigger alert on every poll—If you want the alert to be triggered on every poll.
Trigger up to <numbers> of alerts within <number> of minutes—If you want a specific number of alerts to be triggered within a specific number of minutes. Enter the number of alerts and number of minutes.
In the Schedule pane, click one of the following radio buttons:
24-hours daily—If you want the alert to be triggered 24 hours a day.
Start time/Stop time—If you want the alert to be triggered within a specific start and stop time. Enter the start and stop times.
If you want to enable e-mail for this alert, check the Enable Email check box.
To trigger an alert action with this alert, choose the alert action that you want to send from the drop-down list box.
To configure a new alert action, or edit an existing one, click Configure.
To add a new alert action, continue to Step 18. To edit an existing alert action, skip to Step 25.
In the Name field, enter a name for the alert action.
In the Description field, enter a description of the alert action.
To add an e-mail recipient, click Add.
In the Enter email/epage address field, enter an e-mail or e-page address of the recipient that you want to receive the alert action.
The Action Configuration window shows the recipient(s) that you added, and the Enable check box appears checked.
To delete an e-mail recipient, highlight the recipient and click Delete. The recipient that you chose disappears from the recipient list.
When you finish adding all the recipients, click OK. Skip to Step 27.
To edit an existing alert action, highlight the alert action and click Edit.
The Action Configuration window of the alert action that you chose appears.
Update the configuration and click OK. Continue to Step 27.
After you finish alert action configuration, click Close.
For alerts, such as CriticalServiceDown and CodeYellow, that allow trace download, perform the following procedure:
In the Alert Properties: Trace Download window, check the Enable Trace Download check box.
The SFTP Parameters Dialog window appears. Enter the IP address, a username, password, port and download directory path where the trace will be saved. To ensure that you have connectivity with the SFTP server, click Test Connection. If the connection test fails, your settings will not get saved.
To save your configuration, click OK.
In the Trace Download Parameters window, enter the number and frequency of downloads. Setting the number and frequency of download will help you to limit the number of trace files that will be downloaded. The setting for polling provides the basis for the default setting for the frequency.
Enabling Trace Download may affect services on the server. Configuring a high number of downloads will adversely impact the quality of services on the server.
To delete an alert action, highlight the action, click Delete, and click Close.
You may want to temporarily suspend some or all alerts; you can suspend alerts on a particular node or on an entire cluster. For example, if you are upgrading your system to a newer release, suspend alerts until the upgrade completes, so that you do not receive e-mails and e-pages during the upgrade.
Follow this procedure to suspend alerts in Alert Central.
Choose System > Tools > Alert > Suspend cluster/node Alerts.
Per node suspend states do not apply to clusterwide alerts.
Perform one of the following actions:
To suspend all alerts in the cluster, click the Cluster Wide radio button and check the Suspend all alerts check box.
To suspend alerts per server, click the Per Server radio button and check the Suspend check box of each server on which you want alerts to be suspended.
To resume alerts, choose Alert > Suspend cluster/node Alerts and uncheck the suspend check boxes.
Set Up E-Mails for Alert Notification
Perform the following procedure to configure e-mail information for alert notification.
To configure RTMT to send alerts through e-mail, you must configure DNS. For information on configuring the primary and secondary DNS IP addresses and the domain name in Cisco Unified Communications Manager Server Configuration, see the DHCP Server Configuration chapter in the Cisco Unified Communications Manager Administration Guide.
Unified Communications Manager clusters only: Because Unified Communications Manager generates the e-mail notifications, you can verify that the mail server that you configure can be reached from the Unified Communications Manager platform with the CLI command: utilsnetworkping<mail server>
Choose System > Tools > Alert > Config Email Server.
The Mail Server Configuration window appears.
Enter the address of the mail server in the Mail Server field.
Enter the port number of the mail server in the Port field.
Enter the address of the intended recipient in the Enter e-mail/epage address field.
Repeat this step as necessary to enter all intended e-mail recipients.
By default, RTMT_Admin@domain is used, where domain is the domain of the host server.
Set Up Alert Actions
The following procedure describes how to configure new alert actions.
Display Alert Central.
Choose System > Tools > Alert > Config Alert Action.
Perform Step 17 in the Set up alert properties task to add, edit, or delete alert actions.
Set Up Global A-Mail List for Alert Notifications
The following procedure describes how to configure all precanned alerts at once for sending to one or more e-mail destinations. This procedure uses the initial "Default" alert action setting that is assigned to all alerts by default at installation.
Follow this procedure to configure a recipient list for all precanned alerts without having to set an alert action for each alert. When you add e-mail destinations to the Default alert action list, all pre-canned alerts get sent to those recipients, as long as all alerts continue to use the Default alert action.
To configure a new alert action for a specific alert, you can use the Set Alerts/Properties option, which displays when you right-click an alert. You can also reconfigure existing alert actions with this option.
Any time you update an alert action, the changes apply to all alerts that are configured with that alert action. For example, if all alerts use the "Default" alert action, updating the alert action "Default" will impact all alerts.
You cannot remove the "Default" alert action. For all other alert actions, the system allows you to delete an alert action only when it is not associated with other alerts. If an alert action is associated with multiple alerts, you must reassign a new alert action to those alerts before you can delete the alert action.
Click Alert Central in the QuickLaunch Channel.
The Alert Central window displays.
Click System > Tools > Alert > Config Alert Action.
The Alert Action box displays.
Select Default (highlight the item) in the Alert Action list and click Edit.
The Action Configuration box displays.
Enter the description of the default list.
Click Add to add a recipient. The Input box displays.
Enter an e-mail destination that is to receive all alerts. Click OK.
The e-mail address displays in the Recipients list in the Action Configuration box; the destination is enabled by default.
You can disable an e-mail destination at any time by clicking the check box next to the destination to disable it. To completely remove a recipient from the list, highlight the recipient in the list and click Delete.
Return to Step 5 to add additional e-mail destinations, as required.
You can disable e-mails for an alert at any time by highlighting the alert in the Alert Central window, right-clicking the alert, and using the Set Alert/Properties selections to deselect Enable Email.