Table Of Contents
Managing Faults
9.1 What Is Fault Management?
9.1.1 What the NE Provides
9.1.2 Fault Notification and Maintenance
9.1.3 Root Cause Analysis
9.2 Where Can I Get Information on Affected Services and Customers?
9.2.1 Viewing the Alarm Browser
9.2.2 Filtering Data in the Alarm Browser
9.2.3 Adding Alarm Notes
9.2.4 Viewing the Alarm Log
9.2.5 Filtering Data in the Alarm Log
9.3 What Fault Information Can I See?
9.3.1 How Are Alarms Displayed?
9.3.2 Suppressing Alarms
9.4 Is the Service Working?
9.4.1 Locating Alarms
9.4.2 Setting Up and Viewing Alarm Configuration Parameters
9.4.3 Creating Alarm Profiles—CTC-Based NEs
9.4.4 Applying Alarm Profiles—CTC-Based NEs
9.4.5 Managing Alarm Profiles—CTC-Based NEs
9.4.6 Using the Recovery Properties Pane
9.5 Where Is the Fault?
9.5.1 Sources of Information
9.5.2 Identifying and Monitoring Alarms
9.5.3 Using Visual and Audible Alarm Notifications
9.5.4 Sources of Events
9.5.5 Enabling Fault Synchronization—CTC-Based NEs
9.6 How Can I Find the Root Cause of the Fault?
9.6.1 Setting Up Error Logs
9.6.2 Viewing the Error Log
9.6.3 Filtering Data in the Error Log
9.6.4 Managing the NE Audit Trail—CTC-Based NEs
9.6.5 Setting Debug Options
9.7 Who Is Responsible for Managing the Fault?
9.7.1 Acknowledging and Unacknowledging Alarms
9.7.2 Configuring Alarm Acknowledgement and Alarm Notes
9.8 How Can the Fault Be Fixed?
9.8.1 Clearing a Security Violation Alarm—ONS 15327, ONS 15454 SONET, ONS 15454 SDH
9.8.2 Performing a System Reset—ONS 15600 SONET and ONS 15600 SDH
9.8.3 Restarting an ONS 15800, ONS 15801, or ONS 15808 Laser
9.8.4 Exporting Alarms and Events to a Text File
Managing Faults
This chapter includes the following information:
•
What Is Fault Management?
•Where Can I Get Information on Affected Services and Customers?
•What Fault Information Can I See?
•Is the Service Working?
•Where Is the Fault?
•How Can I Find the Root Cause of the Fault?
•Who Is Responsible for Managing the Fault?
•How Can the Fault Be Fixed?
9.1 What Is Fault Management?
Fault management is the process of locating, diagnosing, and correcting network problems. This is important for increasing network reliability and effectiveness, and for increasing the productivity of network users. Fault management is more than just handling emergencies. It provides functions for managing problems with services and handling customer-facing service problems.
Efficient fault management can:
•Save repair costs through efficient fault detection, location, and correction
•Improve customer care through efficient trouble administration
•Improve service availability and equipment reliability through proactive maintenance and through measurement, review, and corrective action
One responsibility of fault management is to detect faults. A piece of equipment, a transmission medium, a software module, or a database is said to be in a fault state if it cannot perform its intended function and meet all of the requirements placed on that function. The onset of a fault is called a failure event and is usually signaled by one or more alarm reports. The termination of a fault state is called a clear event.
Fault management is responsible for determining, from a variety of information sources, the root cause of a fault, and for its repair. In certain cases, the root cause of a fault might be in a connecting network. In such cases, fault management is responsible for reporting the problem through appropriate channels.
Service assurance is the overall process of ensuring that the purchased level of service is delivered. The Element Management System (EMS) plays a key role in maintaining the health of both NEs and transmission facilities. This is done in conjunction with other systems, typically at the network management layer and service management layer. The EMS can be the primary repository of detailed history of NE-specific faults and events, technician action, and performance data.
The steps for successful fault management are:
1. Identify a problem by gathering data about the state of the network (polling and trap generation).
2. Restore any services that have been lost.
3. Isolate the cause, and decide if the fault should be managed.
4. Correct the fault if possible.
9.1.1 What the NE Provides
Currently deployed, intelligent NEs provide the management system with the following, which are required for effective fault management:
•Detection of the four main types of failure:
–Equipment failures through failure detection mechanisms built into the hardware, and through routine exercises and diagnostics.
–Software failures—Detected through failure of software checks, and through routine audits.
–Communication failures—Detected through defects detected in the incoming signal or outgoing signal characteristics. Defects include line coding errors, framing bit errors, parity errors, cyclic redundancy check errors, and addressing errors. Signal characteristics include optical or electrical power, analog signal-to-noise ratio, and deviation from required voltage or wavelength.
–Environment failure.
•Notification of failure—NEs notify a management system when a failure occurs by generating an alarm report. The NE can also report a summary of current fault states, or replay its log of historical failures and clears.
•Notification of changes in operational state of the NEs components—If a component of the NE is in a fault state, a management system should not receive further alarms, alerts, or scheduled performance data from that component.
9.1.2 Fault Notification and Maintenance
Fault notification and maintenance can be proactive or reactive:
•Proactive notification—Where X notifies Y of a problem regarding a service delivered from X to Y.
•Reactive maintenance—Where Y contacts X to query X on potential problems in X's domain.
9.1.2.1 Proactive Maintenance
Automated detection tests and surveillance software enable rapid initiation of the repair process, sometimes even before customers have noticed a problem. This is called proactive maintenance and promotes customer satisfaction.
Proactive maintenance consists of functions and processes associated with the detection, analysis, isolation, and resolution of problems by means that are independent of customer trouble reports. The problems might be faults or degradations in equipment or transmission media.
The goals of proactive maintenance are to:
•Detect and fix service quality problems before the customer calls to establish a trouble report, or at least to start the repair process before the customer calls, thereby minimizing the time, as perceived by the customer, before service is restored.
•Maintain the transport network at a high level of quality by identifying the facilities that perform relatively poorly and rehabilitating them.
9.1.2.2 Reactive Maintenance
Reactive maintenance is required when a failure occurs. This type of problem can be time-consuming and costly. It requires accurate administration of trouble reports, rapid analysis and repair of service-affecting faults, and notifications to customers of restoration of service, all of which also promote customer satisfaction.
9.1.3 Root Cause Analysis
The root cause is the most basic reason for an undesirable condition or problem that, if eliminated or corrected, would have prevented the problem from existing or occurring. The outcome of the root cause analysis is not a restatement of the most obvious symptom, but is the result of a methodical analysis of the problem situation, leading to the most basic cause.
Root cause analysis captures additional information about defects for the purpose of identifying preventive actions.
9.2 Where Can I Get Information on Affected Services and Customers?
The first thing to do in fault management is to identify what services and which customers are affected by the fault. CTM provides a number of options for viewing this information, including:
Table 9-1 describes where to obtain information on affected services and customers.
Table 9-1 Obtaining Information on Affected Services and Customers
Where
|
Description
|
For Further Information
|
Dashboard
|
Shows useful alarm and NE information in one easily accessible location.
|
See 1.3.1 Dashboard, page 1-4.
|
Tooltips
|
Visible when you position the cursor over a managed object (domain, group, subnetwork, NE, board, link, and so on). The tooltip displays an actual alarm count for critical, major, minor, and warning severities.
|
—
|
Domain Explorer
|
Domain Explorer is the home window and provides a logical view of the network plus alarm, connectivity, and operational status
|
See 1.3.2 Domain Explorer, page 1-5.
|
Subnetwork Explorer
|
Similar in appearance and function to the Domain Explorer. A key difference is that the Subnetwork Explorer provides a single-level grouping of NEs based on network partitions and subnetworks.
|
See 1.3.3 Subnetwork Explorer, page 1-13.
|
Network Map
|
Displays a geographical layout of the network.
|
See 1.3.6 Network Map, page 1-19.
|
Alarm Browser
|
Displays standing alarms and conditions in the managed domain that are assigned a severity level of critical, major, minor, or warning. It also shows cleared alarms that are not acknowledged.
|
See Viewing the Alarm Browser.
|
Alarm Log
|
The Alarm Log contains alarms that have transitioned from the Alarm Browser.
|
See Viewing the Alarm Log.
|
9.2.1 Viewing the Alarm Browser
The Alarm Browser (Figure 1-5 on page 1-17) displays standing alarms and conditions in the managed domain that are assigned a severity level of critical, major, minor, or warning. It also shows cleared alarms that are not acknowledged. The Alarm Browser and Alarm Log views provide a robust listing of all current and historical alarms and events. See Exporting Alarms and Events to a Text File for information about the Alarm Log.
To display the Alarm Browser, select an NE, group, subnetwork, or domain node from the Domain Explorer, Subnetwork Explorer, Network Map, or NE Explorer; then, choose Fault > Alarm Browser (or click the Open Alarm Browser tool from the Dashboard).
Note No alarms or events are generated in the Alarm Browser if Oracle shuts down.
Note Use the toolbar icons to manage the alarm display. See Appendix A, "Icons and Menus Displayed in CTM" for an explanation of each toolbar icon.
Table 9-2 describes the fields in the Alarm Browser.
Table 9-2 Field Descriptions for the Alarm Browser Window
Field
|
Description
|
Alarm ID
|
Unique number that the system uses to identify a particular alarm.
|
Perceived Severity
|
Perceived severity of the selected alarm (critical, major, minor, or warning). The background color also indicates the severity, where:
•Red = Critical
•Orange = Major
•Yellow = Minor
•Blue = Warning
•Green = Cleared
Note For the ONS 15501, ONS 15530, and ONS 15540, alarms that are shown in the CLI as informational are shown in CTM as warnings.
|
Acknowledged
|
Whether the selected alarm has been acknowledged by the user. Values are Yes and No.
|
Note
|
Any notes that were entered for the selected alarm. This field also shows the login name of the user who entered the note and the time when the note was entered.
|
Source ID
|
Name of the NE where the selected alarm occurred.
|
Probable Cause
|
Probable cause of the selected alarm. Displays Not Applicable/Unknown if no additional information is available.
|
Affected Object
|
Name of the object where the selected alarm occurred.
|
Module Name
|
Name of the module where the selected alarm occurred.
|
Physical Location
|
Physical location of the equipment where the selected alarm occurred, such as rack, subrack, slot, and port numbers.
|
CTM Alarm Time Stamp
|
Date and time when the alarm occurred on the server.
|
NE Alarm Time Stamp
|
Date and time when the alarm occurred on the NE.
|
Service Affecting
|
Whether the alarm condition is service affecting (SA); values are Yes and No.
|
CTM Clear Time Stamp
|
Date and time when the alarm was cleared on the server.
|
NE Clear Time Stamp
|
Date and time when the alarm was cleared on the NE.
|
Acknowledged Time Stamp
|
Date and time when the user acknowledged the selected alarm.
|
Acknowledged Username
|
Login name of the user who acknowledged the selected alarm.
|
Alarm Status
|
Status of the selected alarm (active or cleared).
|
Description
|
Additional information about the selected alarm. If there is no additional information, this field is blank.
|
9.2.2 Filtering Data in the Alarm Browser
Step 1 In the Alarm Browser window, choose File > Filter (or click the Filter Data tool). The Filter dialog box opens.
Step 2 Specify the filter parameters described in Table 9-3.
Table 9-3 Field Descriptions for the Alarm Browser Filter Dialog Box
Tab
|
Description
|
CTM Alarm Time (time zone)
|
Allows you to filter alarm data for a specified time period, ranging from the past hour to the past 6 months. Additionally, you can click the User Specified radio button to specify an exact filter starting and ending time by month, day, year, and hour. The time zone can be Greenwich Mean Time (GMT), a user-defined offset from GMT, or local time, depending on what is specified in the User Preferences dialog box. If you want to filter alarms and the time period is not important, click No Time Specified. Click From Now Onward to set the filter time to start immediately and continue until you change filter parameters.
|
Source ID
|
Allows you to move NEs back and forth between the list of Available Source IDs and Selected Source IDs and then run the alarm filter. If you have the appropriate user permissions, you can filter CTM EMS alarms by selecting CTM and adding it to the Selected Source ID list.
If more than 100 NEs are selected, the Source ID tab becomes gray and all devices are included in the filter criteria you specify.
|
Module Name
|
Allows you to specify which module types you want to include in the alarm filter. The modules displayed depend on the NE selection in the Domain Explorer tree when the Alarm Browser is opened. Use the Add and Remove buttons to filter the display to specific modules. The Alarm Browser displays alarms for modules listed under Selected Module Name.
|
Affected Object
|
Allows you to specify which objects you want to include in the alarm filter. All objects are displayed regardless of the NE selected in the Domain Explorer tree when the Alarm Browser is opened. Use the Add and Remove buttons to filter the display to specific objects. The Alarm Browser displays alarms for objects listed under Selected Affected Object. To filter NE-specific EMS alarms, include CTM in the Selected Affected Object list.
|
PS
|
Allows you to filter alarm data based on perceived severity (Critical, Major, Minor, and Warning) and alarm status (Active and Cleared). You can filter on acknowledged alarms, unacknowledged alarms, or both. You can also filter service-affecting alarms, nonservice-affecting alarms, and/or alarms where the service-affecting status is not known.
|
Physical Location
|
Allows you to filter alarm data based on the physical location of an NE or its components. To view the tab, an NE must be selected in the Domain Explorer tree. The filters that are available depend on the NE that is selected when the Alarm Browser is opened. For example, if you select an ONS 15800, you can filter alarms by rack, subrack, and slot. If you select a CTC-based NE, you can filter alarms by slot and port.
|
ID
|
Allows you to filter alarm data based on a specific starting and ending alarm ID. Check the Disregard All Other Filter Criteria check box to base the filter on only the starting and ending alarm ID.
|
NE Alarm Time (time zone)
|
Allows you to filter alarm data for a specified time period, ranging from the past hour to the past 6 months. Additionally, you can click the User Specified radio button to specify an exact filter starting and ending time by month, day, year, and hour. The time zone can be GMT, a user-defined offset from GMT, or local time, depending on what is specified in the User Preferences dialog box. If you want to filter alarms and the time period is not important, click No Time Specified. Click From Now Onward to set the filter time to start immediately and continue until you change filter parameters.
|
Step 3 After making your selections, click OK to run the filter.
9.2.3 Adding Alarm Notes
Use the Alarm Note dialog box to read any comments that have been entered for the selected alarm and to enter additional comments. In the Alarm Browser window, select an alarm and choose Fault > Show Alarm Note (or click the Show Alarm Note tool). Table 9-4 provides descriptions.
Table 9-4 Field Descriptions for the Alarm Note Dialog Box
Field
|
Description
|
Note
|
Provides space to type comments about the selected alarm. To add your comments to the previous comments, click the Append radio button. To overwrite the previous comments, click Replace. To delete the comments, click Delete.
Note You can enable and disable the ability to overwrite or delete alarm notes in the Control Panel > UI Properties pane.
|
History
|
Comments that were entered by previous users.
|
9.2.4 Viewing the Alarm Log
The Alarm Log (Figure 1-6 on page 1-18) contains alarms that have transitioned from the Alarm Browser. Cleared alarms are transitioned when you acknowledge them or when automatic acknowledgment has been enabled (in the Control Panel > User Interface Properties pane). In addition, the Alarm Log shows a history of cleared and acknowledged alarms and all transient conditions (also known as events or autonomous nonalarmed messages). Events are placed directly into the Alarm Log; they do not appear in the Alarm Browser. By default, the Alarm Log shows alarm and event information that occurred during the last 4 hours.
To view the Alarm Log, select a node in the Domain Explorer tree and choose Fault > Alarm Log. Table 9-5 describes the fields in the Alarm Log.
Table 9-5 Field Descriptions for the Alarm Log Window
Field
|
Description
|
ID
|
Unique number that the system uses to identify a particular alarm or event.
|
Source ID
|
Name of the NE or EMS where the selected alarm or event occurred.
|
Affected Object
|
Name of the object where the selected alarm or event occurred. For NE-specific alarms, the affected object field displays "CTM." For the non-NE specific alarms:
•Maximum number of login attempts exceeded alarm—The affected object field displays the user ID associated with the alarm.
•CTM self-monitor alarm—The affected object field displays the threshold parameter associated with the alarm.
|
Module Name
|
Name of the module where the selected alarm or event occurred.
|
Physical Location
|
Physical location of the equipment where the selected alarm or event occurred, such as rack, subrack, slot, and port numbers.
|
Probable Cause
|
Probable cause of the selected alarm or event.
|
Perceived Severity
|
Severity of the alarm before it was cleared. Perceived Severity is listed as:
•Critical (CR)
•Major (MJ)
•Minor (MN)
•Warning (WR)
•Indeterminate (IN)
The background color of the column indicates the alarm status, where:
•Green = Cleared alarms
•Purple = Indeterminate events
Note Indeterminate events are transient events that do not have a severity indicated by the source NE. Indeterminate events do not have a cleared condition.
|
Service Affecting
|
Whether the alarm or event is service affecting. Values are:
•Yes if the alarm is service affecting
•No if the alarm is not service affecting
•N/A if no information is provided by the NE
|
CTM Time Stamp (time zone)
|
Date and time when the alarm or event occurred on the CTM server.
|
CTM Clear Time (time zone)
|
Date and time when the alarm was cleared on the CTM server.
|
CTM Duration
|
Amount of time required to clear an alarm (CTM clear time - CTM time) in dddd:hh:mm:ss format.
|
NE Time Stamp (time zone)
|
Date and time when the alarm or event occurred on the NE.
|
NE Clear Time (time zone)
|
Date and time when the alarm or event was cleared on the NE.
|
Description
|
Brief description of the selected alarm or event. If no description is entered, this field is blank.
|
Acknowledged Username
|
Login name of the user who acknowledged the alarm or event.
Note If the alarm acknowledgement is set to Automatic and you can manually acknowledge an alarm, the Acknowledged Username is not overwritten when the alarm clears.
|
Acknowledged Time
|
Date and time when the alarm or event was acknowledged.
Note If the alarm acknowledgement is set to Automatic and you can manually acknowledge an alarm, the Acknowledged Time is not overwritten when the alarm clears.
|
Note
|
Any notes that were entered for the selected alarm or event. This field also shows the login name of the user who entered the note and the time stamp when the note was entered.
|
9.2.5 Filtering Data in the Alarm Log
Step 1 In the Alarm Log window, choose File > Filter (or click the Filter Data tool). The Filter dialog box opens.
Step 2 Specify the filter parameters described in Table 9-6.
Table 9-6 Field Descriptions for the Alarm Log Filter Dialog Box
Tab
|
Description
|
CTM Time Stamp (time zone)
|
Allows you to filter alarm and event data for a specified time period, ranging from the past hour to the past 6 months. Additionally, you can click the User Specified radio button to specify an exact filter starting and ending time by month, day, year, and hour. The time zone can be GMT, a user-defined offset from GMT, or local time, depending on what is specified in the User Preferences dialog box. If you want to filter alarms and events and the time period is not important, click No Time Specified. Click From Now Onward to set the filter time to start immediately and continue until you change filter parameters.
|
Source ID
|
Allows you to move NEs back and forth between the list of available source IDs and selected source IDs and then run the filter. If you have the appropriate user permission, you can filter CTM EMS alarms and events by selecting CTM and adding it to the Selected Source ID list.
If more than 100 NEs are selected, the Source ID tab becomes gray and all devices are included in the filter criteria you specify.
|
Module Name
|
Allows you to specify which modules you want to include in the filter. The modules displayed depend on the NE selection in the Domain Explorer tree when the Alarm Log is opened. Use the Add and Remove buttons to filter the display to specific modules. The Alarm Log displays events for modules listed under Selected Module Name.
|
Affected Object
|
Allows you to specify which objects you want to include in the filter. The objects displayed depend on the NE selection in the Domain Explorer tree when the Alarm Log is opened. Use the Add and Remove buttons to filter the display to specific objects. The Alarm Log displays events for entities listed under Selected Affected Object.
To filter NE-specific EMS alarms, select CTM and add it to the Selected Affected Object list.
|
PS
|
Allows you to filter data based on the perceived severity (PS) of the alarm or event. Additionally, you can filter service-affecting alarms and events, nonservice-affecting alarms and events, and/or alarms and events where the service-affecting status is not known.
|
Physical Location
|
Allows you to filter data based on the physical location of an NE or its components. To view the tab, an NE must be selected in the Domain Explorer tree. The filters that are available depend on the NE selected. For example, if you select an ONS 15800, you can filter events by rack, subrack, and slot. If you select a CTC-based NE, you can filter events by slot and port.
|
ID
|
Allows you to filter data based on alarm or event ID. Enter a starting ID and an ending ID; then, run the filter to see only the alarms or events that occurred within the specified range of IDs. Check the Disregard All The Other Filter Criteria check box to ignore all other filter specifications. In addition, you can filter the view to only alarms, only events, or both alarms and events.
|
NE Alarm Time (time zone)
|
Allows you to filter alarm data for a specified time period, ranging from the past hour to the past 6 months. Additionally, you can click the User Specified radio button to specify an exact filter starting and ending time by month, day, year, and hour. The time zone can be GMT, a user-defined offset from GMT, or local time, depending on what is specified in the User Preferences dialog box. If you want to filter alarms and the time period is not important, click No Time Specified. Click From Now Onward to set the filter time to start immediately and continue until you change filter parameters.
|
Step 3 After making your selections, click OK to run the filter.
9.3 What Fault Information Can I See?
An alarm is represented by a notification from a managed NE that a certain condition has just occurred. These alarms usually represent error conditions on NEs. Each alarm is associated with the NE for which it provides notification, and an NE can have a number of alarms related to itself at any time.
Each NE shown in the Domain Explorer tree has a corresponding alarm icon that indicates the highest severity alarm that affects the NE. Management domain nodes and group nodes have alarm icons that reflect the highest alarm condition of the NEs contained in the domain or group.
The user-defined Domain Explorer views have "bubble-up" alarm severity propagation and drill-down capabilities to isolate fault conditions and identify service-delivery impact.
9.3.1 How Are Alarms Displayed?
The color of the border surrounding a component, or the background color, indicates the operational status of the component. When status changes, the border or the background color changes as indicated in Table A-3 on page A-8.
9.3.1.1 Understanding How CTM Displays the Affected Object Field
CTM displays the Affected Object field in the Alarm Browser and Alarm Log windows in the same way as for the Interface field in PM tables. See 10.4.6 Understanding How CTM Displays the Interface Field, page 10-28.
Note the following exceptions:
•For the ONS 1580x, the Affected Object field is set to Equipment.
•For the ONS 155xx, the Affected Object field can be:
–wavepatch—The alarm is related to the optical client side of the card.
–wave—The alarm is related to the optical connection with the OADM (filter) card.
–The actual interface object, which changes depending on the card. For example, WaveEthernetPhy for a 10 Gb card, EsconPhy for an ESCON card, and so on.
9.3.2 Suppressing Alarms
Alarm suppression is useful when the NE is under maintenance.
Caution If multiple CTC or TL1 sessions are open, alarms in all other open sessions are also suppressed.
9.3.2.1 Suppressing Alarms at the Card Level—CTC-Based NEs
Step 1 In the Domain Explorer window, select a CTC-based NE and choose Configuration > NE Explorer.
Step 2 In the NE Explorer tree, click the specific card.
Step 3 In the card slot property sheet of the NE Explorer window, click the Identification tab.
Step 4 Check the Suppress Alarms check box.
Step 5 Click Apply.
Note For the ONS 15800, ONS 15801, and ONS 15808 NEs, you can suppress alarms at the card level by setting the card as Out of Service. See 5.11.3 Tagging an ONS 15800, ONS 15801, or ONS 15808 Module as Out of Service, page 5-191.
9.3.2.2 Suppressing Alarms at the Node Level—CTC-Based NEs
Step 1 In the Domain Explorer window, select a CTC-based NE and choose Configuration > NE Explorer.
Step 2 In the node property sheet of the NE Explorer window, click the Alarm tab.
Step 3 Click the Alarm Behavior subtab.
Step 4 Check the Suppress Alarms check box.
Step 5 Click Apply.
9.4 Is the Service Working?
Network devices report symptoms of problems by generating events. An event in this context is a message indicating that a device or application in your network has discovered something of note. The network devices generate many types of events automatically. In addition, you can use thresholds to define or modify the conditions under which events are generated. A threshold is a trigger, set up on a continuous data stream, that is a point of interest that generates events when that point is satisfied.
The events generated need to be analyzed to determine whether they represent a fault condition or a problem in your network.
It is important to generate events when there is a problem. It is also important to limit the number of events generated to prevent an excessive load on the network. CTM performs a number of self-monitoring tasks where threshold limits can be set. The threshold limits are set in the Self Monitor table. (See 10.3.12 Using the Self Monitor Table, page 10-14.) If a threshold is crossed, an EMS alarm is generated.
9.4.1 Locating Alarms
You can locate the equipment for an existing alarm from either the Alarm Browser or the Alarm Log.
Step 1 In the Domain Explorer, choose Fault > Alarm Browser or Alarm Log (or click the Open Alarm Browser or Open Alarm Log tool).
Step 2 In the Alarm Browser or Alarm Log, click an alarm condition and choose Fault > Locate Alarm/Event. The NE Explorer opens and displays the property sheet of the alarmed equipment.
9.4.2 Setting Up and Viewing Alarm Configuration Parameters
Use the Alarm Configuration pane to configure and view alarm severities for system parameters. The Self Monitor table displays information about threshold parameters. These parameters are collected and evaluated based on the NE model types.
Note Changing the EMS alarm severities can affect the alarm status seen by users on the EMS's OSS interfaces.
Step 1 In the Domain Explorer window, choose Administration > Control Panel.
Step 2 Click Alarm Configuration to open the Alarm Configuration pane. Table 9-7 describes the tabs and fields in the Alarm Configuration pane.
Note Nonthreshold alarms do not have an alarm threshold. Instead, nonthreshold alarms occur when a condition occurs, such as loss of connectivity to an NE. Use the Non-Threshold EMS Alarms tab to set the severity level for which a nonthreshold alarm should be raised when that condition occurs.
Step 3 After making your selections, click Save.
Table 9-7 Field Descriptions for the Alarm Configuration Pane
Field
|
Description
|
Threshold EMS Alarms Tab
|
Poll Frequency
|
Threshold alarms are raised when their limit exceeds the value set for Critical, Major, Minor, or Warning thresholds. For example, you can set threshold alarms for disk usage for 90%, 80%, 70%, and 60%, meaning a warning alarm is raised when the disk becomes 61% full and a critical alarm is raised when the disk becomes 91% full. The server checks these parameters at every polling interval that is set in the Poll Frequency field.
|
Parameter Name
|
Name of the parameter:
•Base Circuit Creation Time (seconds)—Time it takes to create a base circuit.
•CPU Usage (%)—Percentage of CPU time utilized for executing user, system, and I/O tasks.
•Circuit Creation Time Per Hop (seconds)—Time it takes for CTM to create an end-to-end circuit.
•Config Resynch Time (seconds)—Time it takes for CTM to collect alarm and inventory information from the NE.
•Disk Usage (%)—Percentage of disk space used in a particular partition. CTM database and partitions are monitored separately.
•Memory Usage RAM (%)—Percentage of RAM memory used for all processes of the system.
•Memory Usage SWAP (%)—Percentage of SWAP memory used for all processes of the system.
•NE Synch Time (seconds)—Time it takes to synchronize CTM server with the NEs.
•PM Collection Time (seconds)—Time it takes to collect PM data for an NE. It is the sum of the time it takes to read the PM data and the time it takes to update the database.
•Prune Time 15 min PM (seconds)—Time it takes to prune 15-minute PM data.
•Prune Time 1 day PM (seconds)—Time it takes to prune 1-day PM data.
•Prune Time Audit Log (seconds)—Time it takes to prune audit log data.
•Prune Time Audit Trail Log (seconds)—Time it takes to prune audit trail log data.
•Prune Time Error Log (seconds)—Time it takes to prune error log data.
•Prune Time FM (seconds)—Time it takes to prune FM data.
•Prune Time Job Monitor (seconds)—Time it takes to prune job monitor data.
•Prune Time Purge NE (seconds)—Time it takes to prune NE purge data.
•Prune Time Server Monitor (seconds)—Time it takes to prune server monitor data.
|
Enable
|
Whether or not the corresponding parameter in the Parameter Name column is enabled (checked) or disabled (unchecked). When checked, it enables monitoring for the selected parameter. If an EMS threshold alarm is outstanding when you disable monitoring, CTM clears the alarm.
|
Critical
|
Amount of time, in minutes, that must elapse before triggering a critical alarm.
|
Major
|
Amount of time, in minutes, that must elapse before triggering a major alarm.
|
Minor
|
Amount of time, in minutes, that must elapse before triggering a minor alarm.
|
Non-Threshold EMS Alarms Tab
|
Parameter Name
|
Name of the parameter.
|
Severity
|
Click the appropriate cell and select the alarm severity level from the available options (Critical, Major, Minor, or Warning) for each of the corresponding parameters listed in the Parameter Name column. If an EMS alarm is outstanding when you change its severity, the outstanding EMS alarm's severity remains the same. The next time the alarm is raised, the severity changes to the new setting.
Note If an alarm is outstanding when you change the severity level, the outstanding alarm's severity level stays the same and the new severity level takes effect the next time the alarm is raised.
|
9.4.3 Creating Alarm Profiles—CTC-Based NEs
Use the Create Alarm Profile dialog box to create new alarm profiles for CTC-based NEs.
Step 1 In the Domain Explorer tree, select a CTC-based NE and choose Configuration > NE Explorer.
Step 2 In the node properties sheet, click the Alarm tab.
Step 3 In the Profile subtab, click the Create button. The Create Alarm Profile dialog box opens. Table 9-8 provides descriptions.
Step 4 After making your selections, click OK.
Table 9-8 Field Descriptions for the Create Alarm Profile Dialog Box
Field
|
Description
|
Enter the Profile Name
|
Enter the name of the new alarm profile.
|
Condition
|
Condition of the alarm.
|
Severity
|
Select a severity for the new alarm from the list. Alarm severities include:
•Not Reported (NR)—A raise or clear of the condition is not sent to clients, but is tracked on the NE. You can retrieve a complete list of all raised conditions, including Not Reported as well as Not Alarmed, Critical, Major, and Minor, by using the RTRV COND TL1 command, or its equivalent.
•Not Alarmed (NA)—A raise or clear of the condition is sent to clients as a nonalarmed TL1 message (REPT EVT). The message has no severity and no service affecting flag.
•Minor (MN)—The alarm is a minor alarm.
•Major (MJ)—The alarm is a major alarm.
•Critical (CR)—The alarm is a critical, traffic-affecting alarm.
Note For Critical, Major, and Minor alarms, a raise or clear of the condition is sent to clients as an alarmed message (REPT ALM). This message includes a service affecting flag, which is On or Off. If a normally service affecting condition is raised in a nonservice affecting situation (for example, the nontraffic-bearing side of a protect pair), the condition is sent as Minor even if the profile says Major or Critical.
•UNSET—The value of this alarm corresponds to the value of the default alarm profile on the node. UNSET is useful when transferring alarm profiles between different versions of software.
•Transparent—The alarm behaves according to its parent object:
–If you set an alarm as Transparent on a port, it uses the card profile.
–If you set an alarm as Transparent on a card, it uses the node profile.
–If you sent an alarm as Transparent on a node, it uses the default profile.
|
9.4.4 Applying Alarm Profiles—CTC-Based NEs
The alarm profiles feature allows you to change default alarm severities by creating unique alarm profiles for individual CTC-based NEs. A profile can be applied to any node on the network. Alarm profiles must be stored on a node before they can be applied to a node, card, or port.
The two reserved profiles include the Default profile, which sets severities to standard Telcordia GR-253 settings, and the Inherited profile, which sets all alarm severities to transparent (TR). If an alarm has an Inherited profile, it inherits (copies) its severity from the same alarm at the next level. For example, a card with an Inherited alarm profile copies the severities used by the node that contains the card. The Inherited profile is not available at the node level.
In the card view, the Alarm Behavior subtab displays the alarm profiles of the selected card. In the node view, the Alarm Behavior subtab displays the alarm profiles for the node. Alarms form a hierarchy. A node-level alarm profile applies to all cards in the node, except those that have their own profiles. A card-level alarm profile applies to all ports on the card, except those that have their own profiles.
At the node level, apply profile changes on a card-by-card basis or set a profile for the entire node. At the card level, apply profile changes on a port-by-port basis or set the profiles for all ports on that card simultaneously.
Step 1 In the Domain Explorer window, click a CTC-based NE and choose Configuration > NE Explorer.
Step 2 In the node properties sheet, click the Alarm > Alarm Behavior tab. In the card properties sheet, click the Alarm Behavior tab.
Step 3 Click Update to view the latest Alarm Profile list.
Step 4 To apply an alarm profile at the card view, click the appropriate row in the Alarm Profile column for the port desired. To apply an alarm at the node view, click the appropriate row in the Profile column for the card desired.
Step 5 Choose the appropriate alarm profile from the drop-down list.
Step 6 Repeat these steps for each port that is being assigned a profile.
Step 7 Click Apply.
Note In the Alarm Behavior tab > Alarm Profile drop-down list, you can select any profile and click Force to all ports to apply this profile to all ports. But when the profile is saved, the Alarm Profile value returns to Inherited. This is expected behavior, because the drop-down list does not represent a value on the NE. After setting the alarm profile value on the ports, the Alarm Profile field returns to the first selectable value in the drop-down list.
9.4.5 Managing Alarm Profiles—CTC-Based NEs
Use the Alarm Profiles Management wizard to download an alarm profile from either an NE or from a local file.
Note This feature is available for CTC-based NEs with software release 3.4 and later.
Step 1 In the Domain Explorer window, choose Configuration > CTC-based SONET NEs or CTC-based SDH NEs > Alarm Profiles Management. The Alarm Profiles Management dialog box opens. Table 9-9 provides descriptions.
Step 2 Select an available profile from the NE by clicking the From NE radio button or from a file by clicking the From File radio button.
Step 3 Complete one of the following options:
•If you clicked the From NE radio button, select the NE and alarm profile from the lists.
•If you clicked the From File radio button, specify the file from a local drive by clicking the Local radio button, or from a server by clicking the Server radio button.
Step 4 Click Next.
Step 5 Edit the alarm severity for each alarm condition by clicking a row in the Alarm Severity column.
Step 6 Click Next.
Step 7 Enter the profile name and save the new alarm profile to the NE by clicking the Save to NE(s) radio button, or to a file by clicking the Save to File radio button.
Step 8 Complete one of the following options:
•If you clicked Save to NE(s), select the NE(s) on which to apply the new alarm profile. Check the Apply to Selected NE(s) check box if you want to set the alarm profile as the current profile.
•If you clicked Save to File, you can either save the file locally by clicking the Local radio button and specifying the directory path, or save it to a server by clicking the Server radio button and specifying the server name.
Step 9 Click Finish.
CTM schedules a job for this action. The alarm profile is downloaded to each selected NE and set, if so selected. This is tracked as a separate task in the Job Monitor table.
Table 9-9 Field Descriptions for the Alarm Profiles Management Wizard
Field
|
Description
|
Select a Profile
|
From NE
|
Select From NE if the alarm profile is on an NE. If you select From NE, the From File options are not accessible.
|
Select NE
|
Select the NE where the alarm profile exists.
|
Select Profile
|
Select an alarm profile from the list.
|
From File
|
Select From File if the alarm profile file is on your PC or a server. If you select From File, the From NE options are not accessible.
|
Local
|
Select Local if the alarm profile file is on your PC. Enter the path for the file, or click Browse to search for it.
|
Server
|
Select Server if the alarm profile file is located on a server. Use the drop-down list to select a server.
|
Edit the Profile
|
Alarm Condition
|
List of alarm conditions for the selected alarm profile.
|
Alarm Severity
|
Click the field to select a new alarm severity for the alarm condition. Severities are:
•Not Reported (NR)—A raise or clear of the condition is not sent to clients, but is tracked on the NE. You can retrieve a complete list of all raised conditions, including Not Reported as well as Not Alarmed, Critical, Major, and Minor, by using the RTRV COND TL1 command, or its equivalent.
•Not Alarmed (NA)—A raise or clear of the condition is sent to clients as a nonalarmed TL1 message (REPT EVT). The message has no severity and no service affecting flag.
•Minor (MN)—The alarm is a minor alarm.
•Major (MJ)—The alarm is a major alarm.
•Critical (CR)—The alarm is a critical, traffic-affecting alarm.
Note For Critical, Major, and Minor alarms, a raise or clear of the condition is sent to clients as an alarmed message (REPT ALM). This message includes a service affecting flag, which is On or Off. If a normally service affecting condition is raised in a nonservice affecting situation (for example, the nontraffic-bearing side of a protect pair), the condition is sent as Minor even if the profile says Major or Critical.
•Unset—The value of this alarm corresponds to the value of the default alarm profile on the node. Unset is useful when transferring alarm profiles between different versions of software.
•Transparent—The alarm behaves according to its parent object:
–If you set an alarm as Transparent on a port, it uses the card profile.
–If you set an alarm as Transparent on a card, it uses the node profile.
–If you sent an alarm as Transparent on a node, it uses the default profile.
|
Save the Profile
|
|
Profile Name
|
Name of the selected alarm profile.
|
Save to NE(s)
|
To save the alarm profile to one or more NEs, select Save to NE(s). If you select Save to NE(s), the Save to File options are not accessible.
|
Available NE(s)
|
Select one or more NEs in the Available NE(s) list and click Add to move them to the Selected NE(s) list.
|
Selected NE(s)
|
Select one or more NEs in the Selected NE(s) list and click Remove to move them to the Available NE(s) list.
|
Apply to Selected NE(s)
|
When checked, applies the alarm profile as the node-level alarm profile.
|
Overwrite the Profile
|
If a profile of the same name already exists, check this check box to overwrite the profile with the new profile.
|
Save to File
|
To save the defaults to a file on your PC or a server, select Save to File. If you select Save to File, the Save to NE(s) options are not accessible.
|
Local
|
Select Local to save the file on your PC. Enter the path for the file, or click Browse to search for it.
|
Server
|
Select Server to save the file on a server. Enter a server path in the field.
|
9.4.6 Using the Recovery Properties Pane
You can use the Recovery Properties pane in the Control Panel window to mark a service as critical for process monitoring purposes. If a critical process stops running or fails to poll monitoring services for a long time, the server shuts down and the client generates an alarm.
You can also use the Recovery Properties pane to list the servers that clients will log into if access to the primary server is disrupted.
Step 1 In the Domain Explorer window, choose Administration > Control Panel.
Step 2 Click Recovery Properties to open the Recovery Properties pane. Table 9-10 provides descriptions.
Step 3 Complete the following substeps to set a process as critical:
a. Click the Process Monitoring tab.
b. Check the Critical check box beside each service to indicate that the service is critical.
c. Click Save.
Step 4 Complete the following substeps to set up session recovery:
a. Click the Session Recovery tab.
b. In the Session Recovery area, fill in the Wait Time and No. of Retries fields.
c. In the Server List area, specify the servers that CTM clients will log into if access to the primary server is disrupted.
d. Click Save.
Table 9-10 Field Descriptions for the Recovery Properties Pane
Field
|
Description
|
Session Recovery Tab
|
Wait Time
|
If disruption to the primary CTM server occurs, the Wait Time field provides the number of seconds the client waits before attempting to log into the backup server.
|
No. of Retries
|
If the first backup CTM server is not available, this field provides the number of times the client should retry logging in before moving to the next server on the list.
|
Server List
|
Lists the CTM servers that CTM clients will log into if access to the primary CTM server is disrupted. The Add button adds a CTM server to the CTM server list. The Remove button removes a CTM server from the list.
If access to the primary CTM server is disrupted, the CTM client tries to log back into the primary server by default. If the CTM client cannot log back into the primary server, and if the server list contains server names, the client tries to log into each backup server in the order listed until the last server in the list is reached, or until the client successfully logs into a server.
|
Process Monitoring Tab
|
Service Name
|
Displays the process monitoring service name.
|
Critical
|
If checked, the selected service is designated as critical for process monitoring.
Note Service Manager, Oracle Service, and OS Agent are permanently critical. You cannot uncheck the Critical check boxes for these services.
|
9.5 Where Is the Fault?
You need to be able to quickly troubleshoot problems on the network, identify when network capacity is being reached, and provide information to management on the number and types of devices in use. If the network goes down, one of the first things you will need to know is what devices are running on the network. You will want to know the names, addresses, and interfaces associated with each device in order to begin troubleshooting the problem. The more information you have in one central place about all of the devices, the easier it is to locate the necessary information, resolve problems quickly, and provide detailed information to interested parties.
9.5.1 Sources of Information
Fault management receives and processes information from the following sources:
•Autonomous reports of failures from NEs
•Trouble reports from customers and peer systems
•Results of diagnostics, exercises, and audits from NEs
•Impairment indications from performance management
•Network configuration data from configuration management
CTM constantly updates the alarm status of the network based on the alarm and event notifications sent by the monitored NEs and generated by the EMS itself. It performs alarm synchronization with the NE each time the connection to the NE is established or re-established and the NE is in service.
9.5.2 Identifying and Monitoring Alarms
To identify and monitor alarms on groups of NEs:
Step 1 In the Domain Explorer tree, select the management domain node or a group node. If it shows a critical, major, minor, or warning alarm icon, it means that one or more NEs within the management domain or group are experiencing an alarm.
Step 2 Select the management domain node or group node and choose Fault > Alarm Browser (or right-click the node and choose Alarm Browser in the popup menu). This opens the Alarm Browser window, which shows all the NEs in the management domain or group that are experiencing an alarm.
To identify and monitor alarms on a specific NE:
Step 1 Select an in-service NE in the Domain Explorer tree that shows an alarm icon.
Step 2 Choose Fault > Alarm Browser.
Step 3 For the CRS-1, ONS 15200, ONS 15216, ONS 1580x, and CTC-based NEs, you can also choose Configuration > NE Explorer to view alarms on the NE or on specific modules. The Module View tab displays a graphic of the module that is installed in the slot. The number of critical, major, minor, and warning alarms for the module is displayed under Alarm Status. (Alarms also display when you move the mouse pointer over the graphic.)
9.5.3 Using Visual and Audible Alarm Notifications
Step 1 In the Domain Explorer window, choose Edit > User Preferences. The User Preferences dialog box opens.
Step 2 In the Event Notification tab, in the Show Notification Dialog For area, select whether or not an alert popup opens when a specific alarm or informational event occurs on NEs in the management domain or in the application.
The Event Notification dialog box opens whenever a new alarm or event occurs. According to your User Preferences selection, you will receive popup notification about alarms by severity and information on events from the NE or from CTM. The Event Notification popup remains open until one of the following occurs:
•You click OK to close the dialog box.
•It is replaced by an Event Notification dialog box with a higher severity.
•You click Disable on the popup window itself to disable additional popups.
Table 9-11 describes the fields in the Event Notification dialog box.
Step 3 In the Play Audible Notification For area, select whether or not an audible alert is sounded when a specific alarm or informational event occurs on the NE or in the application. You can also select whether or not a continuous audible alert is sounded when there is an update in the Dashboard. Check the Continuous Alarm for Dashboard notifications check box.
Note To stop the continuous audible alert, choose Fault > Stop Continuous Beep in the Domain Explorer.
Step 4 Check the Save current settings check box and click OK.
Table 9-11 Field Descriptions for the Event Notification Dialog Box
Field
|
Description
|
Source
|
Name of the source where the alarm or event originated.
|
Time
|
Date and time that you received the Event Notification popup.
|
Category
|
Category of alarm or event. Alarm categories include Critical, Major, Minor, or Cleared. Event categories include NE event (if the event occurred on an NE) or EMS event (if the event occurred on CTM).
|
Probable Cause
|
Probable cause of the alarm or event.
|
Affected Object
|
Object that is affected by the alarm or event.
|
Description
|
Description of the alarm or event.
|
Service Affecting
|
Whether the alarm or event affects service.
|
9.5.4 Sources of Events
There are two sources of events: EMS-generated alarms and OSS (SNMP, CORBA GateWay, TL1). These event sources are described in the following sections:
•Chapter 9 "Managing Faults."
•Chapter 12, "Managing Southbound and Northbound Interfaces."
9.5.4.1 EMS-Generated Alarms
In addition to reporting NE-generated alarms, the EMS monitors and reports alarms and events on the EMS itself; for example, loss of connectivity to NE, and so on.
The EMS monitors and reports the NE-specific alarms and events (see Table 9-12) and non-NE-specific alarms and events (see Table 9-13).
Note NE-specific alarms and events can be viewed and accessed by users who are assigned to the particular NE.
Table 9-12 NE-Specific Alarms and Events
NE-Specific Alarms and Events
|
Description
|
Loss of communication to an NE
|
When the system detects loss of connectivity to an NE, an EMS alarm is generated in the Alarm Browser. This EMS alarm is cleared when the system reestablishes connectivity to the NE or when the NE is marked Out of Service.
|
Automatic or manual memory backup failure
|
If an automatic or manual memory backup job fails, an EMS alarm is generated in the Alarm Browser. An individual EMS alarm is generated for each memory backup failure that occurs. All instances of the backup-related EMS alarms are cleared (for that particular NE) when the memory backup succeeds or when the NE is marked Out of Service.
|
CTM-to-NE authentication failure
|
If the system attempts to log into an NE and fails, an alarm is generated. This alarm indicates that the username and password are no longer valid.
|
Switching IP addresses
|
For the ONS 1580x NEs, when an active IP address switches from the primary to the secondary IP address, an alarm is generated. It is only cleared when the address is switched back to the primary IP address or when communication to the ONS 1580x NE is lost.
|
Failed PM data retrieval
|
For CTC-based NEs, an alarm is generated for every PM data retrieval failure. PM 15-minute retrieval fail alarms are generated if the system has not retrieved 15-minute PM data after the number of times to retrieve PM data has been reached. These alarms can be cleared manually or cleared automatically if a PM lost alarm is generated or if PM data is retrieved (PM collection should be set to 15 Min Robust).
|
Lost PM data
|
For CTC-based NEs, an alarm is generated for all PM lost data. A lost PM alarm is generated when:
•The EMS cannot collect PM data for 15 minutes or 1 day and the NE's PM collection is set to either 15 Min or 1 Day.
•The EMS cannot collect 15-minute PM data after 8 hours or 1-day PM data after 2 days and the NE's PM collection is set to 15 Min Robust or 1 Day Robust. If there are outstanding PM retrieval fail alarms, these alarms are cleared and the PM lost alarm is generated.
These alarms can be cleared manually.
|
Table 9-13 Non-NE-Specific Alarms and Events
Non-NE-Specific Alarms and Events
|
Description
|
Maximum number of login attempts exceeded
|
By default, users have a maximum of five login attempts. The user account is locked after the fifth unsuccessful login attempt and an EMS alarm is generated in the Alarm Browser. The alarm is cleared once the user account is unlocked or the account is deleted.
|
System self-monitor alarm
|
Threshold parameters such as CPU usage, memory usage, disk usage, circuit creation time, PM collection time, and resynch time are collected and evaluated to monitor the server performance. An alarm is generated if any of these parameters cross their threshold values with their corresponding severity level. The alarms are cleared only after the corresponding parameter value falls below the minor threshold. Subsequent threshold crossings for the same parameter do not generate additional alarms. Only the severity level is changed to indicate the current severity level for the specific parameter.
Note Alarms associated with circuit creation, configuration resynchronization, NE synchronization, and PM data collection indicate that the load on the system is high. Reduce the load on the system before proceeding. Alarms associated with pruning times also indicate that the load on the system is high. Reschedule pruning at a time when the system has less activity.
|
9.5.5 Enabling Fault Synchronization—CTC-Based NEs
Fault synchronization allows the system to collect the alarm history of an NE that has been marked Out of Service for some time and is marked In Service again. Alarm history is displayed in the Alarm Browser.
Step 1 In the Domain Explorer window, choose Administration > Control Panel and expand NE Service.
Step 2 Choose ONS 15310 CL/ONS 15327/ONS 15454/ONS 15600 or ONS 15454 SDH/ONS 15600 SDH.
Step 3 In the Robust Fault Synchronization area, check the appropriate check box:
•Enable ONS 15310
•Enable ONS 15327
•Enable ONS 15454
•Enable ONS 15600
•Enable ONS 15454 SDH/ONS 15600 SDH
Step 4 Click Save.
9.6 How Can I Find the Root Cause of the Fault?
Root cause analysis captures additional information about defects for the purpose of identifying preventive actions.
In some cases, the alarm report or set of alarm reports generated by a fault are sufficient to indicate the root cause. But often, the information in the alarm messages must be supplemented or confirmed by information from customer trouble reports, diagnostics and exercises of equipment, audits of software and databases, and testing of circuits. Tests of equipment are called diagnostics, which are designed to identify the root cause of a fault; exercises, which isolate a unit or subsystem and verify that it can perform its intended function; and audits, which verify the integrity of software.
The EMS should correlate events and determine the faults that exist in the network. To correlate events means to look for relationships between them.
9.6.1 Setting Up Error Logs
The Logging Properties pane allows you to control the volume of messages that are created by the server. All changes take effect immediately and do not require restart of the server. To reduce the amount of information logged to the database, turn off entire components (CTM, CTM GateWay/SNMP, and CTM GateWay/TL1).
Step 1 In the Domain Explorer window, choose Administration > Control Panel.
Step 2 Click Logging Properties to open the Logging Properties pane. The pane displays the information shown in Table 9-14.
Step 3 In the Error Log Properties area, choose the error level that will be included in the error log for each component.
Step 4 Click Save.
Table 9-14 Field Descriptions for the Logging Properties Pane
Field
|
Description
|
Error Log Properties
|
CTM Server and GateWay/SNMP
|
Choose the error level to include in the error log for services related to the CTM server and CTM GateWay/SNMP. Critical, major, minor, and informational errors are logged to the database; trace and debug information is logged to a log file.
Caution CTM performance will degrade significantly if the trace or debug option is left on. All operations will slow down and you might lose alarm and event notifications. Use trace or debug only when troubleshooting with a customer support engineer.
Note Trace and debug information does not appear in the error log.
|
SM Service
|
Choose the error level to include in the error log for the SM service. Critical, major, minor, and informational errors are logged to the database; trace and debug information is logged to a log file.
Caution CTM performance will degrade significantly if the trace or debug option is left on. All operations will slow down and you might lose alarm and event notifications. Use trace or debug only when troubleshooting with a customer support engineer.
Note Trace and debug information does not appear in the error log.
|
SNMP Trap Service
|
Choose the error level to include in the error log for the SNMP Trap service. Critical, major, minor, and informational errors are logged to the database; trace and debug information is logged to a log file.
Caution CTM performance will degrade significantly if the trace or debug option is left on. All operations will slow down and you might lose alarm and event notifications. Use trace or debug only when troubleshooting with a customer support engineer.
Note Trace and debug information does not appear in the error log.
|
Preferences
|
Log File Directory
|
Choose the directory where the log files are saved. The default directory is /opt/CiscoTransportManagerServer/log.
|
Max Error Log File Size
|
Select the maximum size of the error log file, in megabytes (MB). You can select a default log size for each log file.
|
Max Debug Log File Size
|
Select the maximum size of the debug log file, in MB. You can select a default log size for each log file.
|
Max Initialization Log File Size
|
Select the maximum size of the initialization log file, in MB. You can select a default log size for each log file.
|
Archiving
|
Enable Log File Archiving
|
If checked, allows you to archive log files to facilitate longer term collection of logging information. When a log file is about to wrap, you can save the regular log file to a separate archive file.
|
Archive Directory
|
Specify the directory where archive log files are saved. The name of the archived file incorporates a time stamp (when the file is created) to ensure a unique filename.
|
Max No. of Archived Log Files
|
Specify the maximum number of log files to archive on the server. The range is 1 to 25 files.
|
9.6.2 Viewing the Error Log
The error log shows CTM server error information that is useful for debugging purposes. In most cases, the error log is requested by service personnel for debugging a problem on the CTM server. The error log captures abnormal and significant events based on severity level.
As the default, the error log displays information about significant events that occurred during the last four hours. You can change the default time period in the User Preferences dialog box.
To open the error log, choose Administration > Error Log in the Domain Explorer. Table 9-15 describes the fields in the error log.
Table 9-15 Field Descriptions for the Error Log
Column Name
|
Description
|
CTM Time Stamp
|
Date and time when the error occurred on the CTM server.
|
Module
|
Name of the module where the error occurred.
|
Severity
|
Severity level of the error:
•Critical, Major, Minor, or Informational—When set to any of these severity levels, all messages corresponding to critical, major, and minor severity levels are logged to the database and all informational messages are stored in the log file.
•Debug or Trace—When set to debug or trace, all informational and higher messages are logged to the database. All debug and trace messages are logged to the log files.
|
Submodule
|
Name of the submodule where the error occurred.
|
Filename
|
Name of the file where the error occurred. Cisco technical support engineers use this information for troubleshooting.
|
Line
|
Exact line where the error occurred. Cisco technical support engineers use this information for troubleshooting.
|
Message
|
Text of the error message.
|
Note Each ONS NE service and ONS PM service has an error level selection. For example, choose Control Panel > NE Service or PM Service > ONS 155XX. In the Error Level drop-down list, choose the error level to include in the error log for the selected ONS NE service or ONS PM service. Click Save.
By default, all messages are logged to the following files in the /opt/CiscoTransportManagerServer/log directory:
•CTMTL1FWDerror.log
•CTMerror.log
•HFRNEService-number.log
•HFRNEService-numberError.log
•HFRPMService-number.log
•HFRPMService-numberError.log
•CAT65xxNEService-number.log
•CAT65xxNEServiceError-number.log
•ONS15200NEService-number.log
•ONS15200NEService-numberError.log
•ONS15216NEService-number.log
•ONS15216NEService-numberError.log
•ONS15302NEService-number.log
•ONS15302NEService-numberError.log
•ONS15305NEService-number.log
•ONS15305NEService-numberError.log
•ONS1530xPMService-number.log
•ONS1530xPMService-numberError.log
•ONS15454NEService-number.log
•ONS15454NEService-numberError.log
•ONS15454SDHNEService-number.log
•ONS15454SDHNEService-numberError.log
•ONS15454SDHPMService-number.log
•ONS15454SDHPMService-numberError.log
•ONS15454PMService-number.log
•ONS15454PMService-numberError.log
•ONS155xxNEService-number.log
•ONS155xxNEService-numberError.log
•ONS155xxPMService-number.log
•ONS155xxPMService-numberError.log
•ONS15600SDHPMService-number.log
•ONS15600SDHPMService-numberError.log
•ONS15600PMService-number.log
•ONS15600PMService-numberError.log
•ONS15800NEService-number.log
•ONS15800NEService-numberError.log
•ONS15801NEService-number.log
•ONS15801NEService-numberError.log
•ONS15808NEService-number.log
•ONS15808NEService-numberError.log
•ONS158xxPMService-number.log
•ONS158xxPMService-numberError.log
•UnmanagedNEService-number.log
•UnmanagedNEService-numberError.log
•SnmpTrapService.log
•SnmpTrapServiceError.log
•CORBAGWService.log
•CORBAGWServiceError.log
After resetting the error log level to Critical, Major, Minor, or Informational, remove the log files to free disk space. Each time a new log file is started, a backup of the previous file is kept in the log-file.bak file. Remove the backup file at any time.
9.6.3 Filtering Data in the Error Log
Step 1 In the error log, choose File > Filter (or click the Filter Data tool). The Filter dialog box opens.
Step 2 Specify the filter parameters described in Table 9-16.
Step 3 After making your selections, click OK to run the filter.
Table 9-16 Field Descriptions for the Error Log Filter Dialog Box
Tab
|
Description
|
CTM Time Stamp (time zone)
|
Allows you to filter error log data for a specified time period, ranging from the past hour to the past 6 months. You can click the User Specified radio button to specify an exact filter starting and ending time by month, day, year, and hour. The time zone can be GMT, a user-defined offset from GMT, or local time, depending on what is specified in the User Preferences dialog box. If you want to filter error log data and the time period is not important, click No Time Specified.
|
Modules
|
Allows you to filter error log data by CTM module.
|
Submodules
|
Allows you to select CTM server submodules to filter error log data.
|
Severity
|
Allows you to filter error log data based on severity level: Critical, Major, Minor, and Informational.
|
9.6.4 Managing the NE Audit Trail—CTC-Based NEs
The Audit Trail table is a security tool used to investigate unauthorized activities after they occur so that proper remedial action can be taken. It displays audit trail information for CTC-based NEs.
Audit trail entries might be missing from the Audit Trail table because of the following reasons:
•There is heavy provisioning activity on the NE that the local audit trail log on the NE might wrap between polling intervals, and records will be missed for collection.
•The server is down or communications to the NE are unavailable for an extended period. The chance that the log on the NE will wrap before the system can collect the records is increased.
Note The audit trail collection interval can be set in the applicable NE Service pane in the Control Panel. See Changing the Audit Trail Collection Interval.
Note To collect the latest and most reliable audit trail information for a specific NE, the time has to be synchronized with the SNTP server. Do not manually set the clock on CTC-based NEs.
9.6.4.1 Viewing the Audit Trail Table
To view the Audit Trail table, choose Administration > CTC-based NEs > Audit Trail Table. Table 9-17 provides descriptions.
Table 9-17 Field Descriptions for the Audit Trail Table
Field
|
Description
|
NE ID
|
ID of the selected NE.
|
Sequence Number
|
NE-generated record ID.
|
NE Username
|
NE user ID.
|
Time Stamp
|
Date and time.
|
Description of Operation
|
Description of the audit trail operation.
|
Status of Operation on NE
|
Status of the audit trail operation. Statuses are Passed, Failed, and Aborted.
|
9.6.4.2 Filtering Data in the Audit Trail Table
Step 1 In the Audit Trail table, choose File > Filter (or click the Filter Data tool). The Filter dialog box opens.
Step 2 Specify the filter parameters described in Table 9-18.
Step 3 After making your selections, click OK to run the filter.
Table 9-18 Field Descriptions for the Audit Trail Table Filter Dialog Box
Tab
|
Description
|
Time Stamp (time zone)
|
Allows you to filter audit trail data for a specified time period, ranging from the past hour to the past 6 months. Additionally, you can click the User Specified radio button to specify an exact filter starting and ending time by month, day, year, and hour. The time zone can be GMT, a user-defined offset from GMT, or local time, depending on what is specified in the User Preferences dialog box. Click No Time Specified if you want to filter audit trail data and the time period is not important.
|
NE ID
|
Allows you to move NEs back and forth between the list of available NEs and selected NEs. The filter runs on the NEs in the Selected NE ID list.
|
Username
|
Allows you to move users back and forth between the list of available users and selected users. The filter runs on the users in the Selected Users list.
|
Sequence Number
|
Allows you to enter a starting and ending sequence number for filtering. Check the Disregard All Other Filter Criteria check box to base the filter on only the starting and ending sequence number.
Note If the sequence number reaches 59999, CTM collects audit records starting with 1 in the next collection interval.
|
Operation Status
|
Select an operation status for filtering. Operation statuses are Passed, Failed, and Aborted.
|
9.6.4.3 Enabling or Disabling Audit Trail Collection
Step 1 In the Domain Explorer tree, select a CTC-based NE.
Step 2 In the Audit Trail State field of the Status tab in the Network Element Properties pane, choose Enabled or Disabled from the pull-down menu.
Note By default, the Audit Trail State field is set to Disabled.
Step 3 Click Save.
9.6.4.4 Changing the Audit Trail Collection Interval
Step 1 In the Domain Explorer, choose Administration > Control Panel.
Step 2 In the Control Panel, expand the NE Service and select ONS 15310 CL/ONS 15327/ONS 15454/ONS 15600 or ONS 15454 SDH/ONS 15600 SDH.
Step 3 In the Audit Trail Collection Interval field, enter the collection interval time. The default is 30 minutes.
Step 4 Click Save.
9.6.5 Setting Debug Options
In CTM, the debug option gives you information to investigate, diagnose, and fix a problem. Specifying debug options allows you to choose parameters to display in the debug log.
Step 1 In the Domain Explorer, choose File > Debug Options.
Step 2 Specify the debug options. Table 9-19 describes the fields in the Debug Options dialog box.
Step 3 After making your selections, click Apply.
Table 9-19 Field Descriptions for the Debug Options Dialog Box
Field
|
Description
|
Modules
|
Available, Selected
|
Select modules that will display debug messages. Use the Add and Remove buttons to move modules to the Selected list or remove modules from the list.
|
Debug Level
|
Fatal
|
Instructs the Debug Log to display messages with a severity level of at least Fatal.
|
Warning
|
Instructs the Debug Log to display messages with a severity level of at least Warning.
|
Info
|
Instructs the Debug Log to display messages with a severity level of at least Info.
|
Debug
|
Instructs the Debug Log to display messages with a severity level of at least Debug.
Caution CTM performance will degrade significantly if the Debug option is left on. All operations will slow down and you might lose alarm and event notifications. Use Debug only when troubleshooting with a Cisco customer support engineer.
|
Trace
|
Instructs the Debug Log to display messages with a severity level of Trace.
Caution CTM performance will degrade significantly if the Trace option is left on. All operations will slow down and you might lose alarm and event notifications. Use Trace only when troubleshooting with a Cisco customer support engineer.
|
Display Options
|
File
|
Check the File check box to write the Debug Log to a specific file. You can click Browse to browse for a local client directory for the Debug Log. After you specify the filename, the log is stored at <filename>0.log and <filename>1.log when <filename>0.log is filled to its maximum size.
By default, the Debug Log is saved at C:\Cisco\TransportManagerClient<version_number>\debug\ctmc-debug0.log or /opt/CiscoTransportManagerClient<version_number>/debug/ctmc-debug0.log. The dialog box shows the filename without the number 0 or 1, which is appended by default by the Java debugging APIs.
Note <version_number> is replaced by the version number of the installed CTM client.
|
Max File Size
|
Enter the maximum file size for the Debug Log, in bytes.
|
Telnet
|
Check the Telnet check box to write the Debug Log to a Telnet port.
Note Telnet to the IP address of the CTM client workstation, not the CTM server.
|
Console
|
Check the Console check box to write the Debug Log to the console.
|
9.7 Who Is Responsible for Managing the Fault?
To manage faults effectively, you must know who is taking responsibility for managing each case. CTM offers the following options:
•Acknowledging and Unacknowledging Alarms
•Configuring Alarm Acknowledgement and Alarm Notes
9.7.1 Acknowledging and Unacknowledging Alarms
The alarm acknowledgement feature acknowledges selected alarms or all alarms with a single click.
Step 1 In the Domain Explorer window, select an NE and choose Fault > Alarm Browser. This opens the Alarm Browser window for the selected NE.
The Alarm Browser window lists critical, major, minor, and warning alarms that have not been cleared or cleared alarms that have not been acknowledged.
Step 2 Select the alarms to be acknowledged and choose Fault > Acknowledge Alarms (or click the Acknowledge Selected Alarm(s) tool. Click Yes in the confirmation dialog box. Click the Refresh Data tool to see the changes. A check mark icon provides a visual indication of acknowledged alarms.
Step 3 Click the Acknowledge Selected Alarm(s) tool again to unacknowledge the selected alarms. Click Yes in the confirmation dialog box. Click Refresh Data to see the changes. The check mark is removed, indicating that the alarm has been unacknowledged.
Note Alarm unacknowledgement is disabled by default. Make sure to enable the alarm unacknowledgement feature in the Control Panel before unacknowledging an alarm. See Configuring Alarm Acknowledgement and Alarm Notes for more information.
Step 4 To acknowledge all alarms in the view, choose Fault > Acknowledge All Alarms (or click the Acknowledge All Alarms tool). Click Yes in the confirmation dialog box.
9.7.2 Configuring Alarm Acknowledgement and Alarm Notes
Use the UI Properties pane to configure alarm acknowledgment and enable or disable the alarm note feature.
Step 1 In the Domain Explorer window, choose Administration > Control Panel and click UI Properties.
Step 2 In the Fault Management area, select either Manual or Automatic for alarm acknowledgement.
•If you choose Manual, alarms must be acknowledged manually. Cleared alarms move from the Alarm Browser to the Alarm Log once they are acknowledged.
•If you choose Automatic, the server automatically acknowledges alarms when they are cleared and moves them from the Alarm Browser to the Alarm Log.
Note Active alarms are not automatically acknowledged.
If the alarms are initially set to Manual Alarm Acknowledgement, and then you switch to Automatic Alarm Acknowledgement, all the alarms in the Alarm Browser will be cleared and acknowledged automatically. This might take a while, depending on the number of alarms in the database that have not been acknowledged manually.
You can still acknowledge alarms manually even if Automatic Alarm Acknowledgment is set.
Step 3 Use the Overwrite Alarm Notes option to enable or disable the ability to overwrite alarm notes created by another user.
Step 4 Select either Enable or Disable for alarm unacknowledgement.
•If you choose Enable, you can unacknowledge alarms in the Alarm Browser.
•If you choose Disable, alarms cannot be unacknowledged in the Alarm Browser.
Step 5 Click Save.
9.8 How Can the Fault Be Fixed?
After receiving an alarm, the EMS or the user must take some sort of action on each of the faults. This can include logging the fault, delivering it to an appropriate tracking application, alerting key personnel to a critical fault, or implementing a repair.
9.8.1 Clearing a Security Violation Alarm—ONS 15327, ONS 15454 SONET, ONS 15454 SDH
Step 1 In the Domain Explorer window, select an ONS 15327, ONS 15454 SONET, or ONS 15454 SDH NE and choose Configuration > NE Explorer (or click the Open NE Explorer tool).
Step 2 In the NE Explorer window, choose Fault > Clear Security Violation Alarm. A message appears if there are no security violations or intrusion alarms on the NE or if the operation has been accepted.
Step 3 Click OK in the message box.
9.8.2 Performing a System Reset—ONS 15600 SONET and ONS 15600 SDH
Step 1 In the Domain Explorer tree, select the ONS 15600 SONET or ONS 15600 SDH NE for which to view the configuration.
Step 2 Choose Configuration > NE Explorer (or click the Open NE Explorer tool).
Step 3 In the NE Explorer, choose File > System Reset.
9.8.3 Restarting an ONS 15800, ONS 15801, or ONS 15808 Laser
The ONS 1580x NEs include an automatic laser shutdown feature where the laser is automatically shut down to a safe power level in the event of a fiber break. You can manually restart an ONS 15800, ONS 15801, or ONS 15808 laser.
Note This procedure applies only to the following amplifiers:
•Infrared-band Booster Amplifier (ONS 15800, ONS 15801)
•Infrared-band Booster Amplifier-10 Gbps (ONS 15800, ONS 15801)
•Transmit Power Amplifier-Infrared Band (ONS 15800, ONS 15801)
•Pre-Line Amplifier-Infrared Band (ONS 15800, ONS 15801)
•Optical Booster Amplifier (ONS 15808)
•Optical Extra Pump (ONS 15808)
•Optical External Pump (ONS 15808)
Step 1 Select an ONS 15800, ONS 15801, or ONS 15808 NE and choose Configuration > NE Explorer.
Step 2 In the tree view of the NE Explorer window, select the card.
Step 3 In the property sheet of the NE Explorer window, click the Action tab.
Step 4 In the Laser Off Control section, click Enable.
Step 5 Click Execute.
9.8.4 Exporting Alarms and Events to a Text File
Use the Event Export Manager to export alarms and events to a text file as they occur. In addition, you can use the Event Export Manager to set various export parameters to refine the export. See 1.5.8 Exporting Alarms and Events, page 1-36.
