Cisco ACE XML Gateway User Guide (Software Version 6.0)
Monitoring System Status

Table Of Contents

Monitoring System Status

About Logged Information

Event Logging

Configuring Event Logging

Viewing the Event Log

Monitoring Performance

Filtering Performance Data by Time

Viewing Performance Information

Exporting Performance Information to a File

Service Health

Message Logging

Configuring Message Logging

Viewing the Message Traffic Log

Compliance Reports

Viewing and Enabling Compliance Logging

Generating Compliance Reports


Monitoring System Status


This chapter describes how to monitor the health and activities of the system. It covers these topics:

About Logged Information

Event Logging

Monitoring Performance

Message Logging

Compliance Reports

About Logged Information

The ACE XML Gateway and Manager include a rich set of features for monitoring system activities. The features include the Manager Dashboard, which presents customizable views of dynamic traffic statistics, the performance monitor, extensive error logging, the audit log, which shows policy changes in the Manager, and the incidents report.


Note This chapter describes the monitoring tools available in the Manager web console. For information on using external tools to monitor the system, such as SNMP and syslog, see the Cisco ACE XML Gateway Administration Guide.


The logs can enhance system security by providing information on potentially malicious traffic crossing your network. It identifies requests that match a variety of attack signatures, including signatures designed to match SQL injection attacks or command injection attacks. The log can also be used to identify problems with backend infrastructure, since server processing errors are captured and reported in the logs. The performance reporting tools can help you tune your system for best performance.

The Manager Dashboard displays a summary of the information provided by the logs. As the first page that appears after a successful login, it alerts you to conditions that may require attention, such as possible attacks. It can be customized to display the graphs of interest to you. Graphs are available that present the transaction rate, errors, and latency by service definition.

The types of logs in the ACE XML Gateway system include:

The event log records data about system events that affect the processing and administrative activity of the ACE XML Gateway and Manager. Examples of events recorded by the event log are message transactions, system startup and shutdown, authentication of web console users, deployment of policies, and a variety of errors and other activities.

The performance log keeps a variety of statistics on traffic in the system intended to assist performance analysis. It provides information on transaction count, processing time, backend round-trip time, and more. This information appears in the Performance Monitor and in the graphs that can be added to the Traffic Monitor section of the Manager Dashboard.

The audit log shows user activity in the ACE XML Manager web console.

The message traffic log records information on request and response messages. Depending on the logging configuration, the log may contain statistics about the message traffic, complete copies of all messages, or both statistics and messages.

Logged information on a busy system can occupy a considerable amount of disk space on the appliance. To prevent resource exhaustion, when the log files on the appliances take up a particular amount of disk space, older log files are automatically deleted to make more space. This feature is intended to prevent unexpected shutdown of the appliance. However, it's preferable to have log files copied to backup storage and removed from the appliance at regular intervals using a managed process. This way, the logged information is recoverable if necessary. For this purpose, you can set up a Shell script that moves the files off the appliance at regular intervals. For more information on disk management, see the Cisco ACE XML Gateway Administration Guide.

Event Logging

The event log provides detailed information on the activities of the ACE XML Gateway and Manager. It displays information on traffic processing activities as well as on the internal operation of the ACE XML Manager and ACE XML Gateway. These events include control events (such as policy deployment), error notifications, and other events important to the operation of the system. This information can help you diagnose problems in the policy or network configuration of the system.

The system can write to the event log at several levels of detail. Each successively higher level of detail records more information. The logging levels are:

Table 32-1 Event logging levels

Level
Description

Alert

Critical system conditions that require immediate attention to prevent system failure.

Error

Error conditions that cause incorrect results or incorrect system behavior.

Warning

Conditions that appear to be incorrect and may cause unexpected system behavior or other undesirable results.

Notice

Normal but significant conditions, such as receipt or delivery of a message. This level of reporting produces one line of output for each message processed under normal conditions.

Info

Significant processing stages in the normal handling of message traffic; at this level each message processed should produce several lines of output.

Debug

All information the ACE XML Gateway or Manager can report. Among other things, this level logs the body of every message the ACE XML Gateway processes.

Note that the debug-level information shown for a message may contain sensitive information, including passwords passed in a request. In general, this level of logging should be used only in testing or troubleshooting scenarios.


It's important to consider that a busy ACE XML Gateway can generate a large number of event log records. Event information is passed to the Manager via syslog, which, as a UDP protocol, offers best-effort delivery only. In extremely busy systems or in stress-testing scenarios, it's possible for event log information to be lost.

At the higher levels of detail—Notice, Info, and Debug—the system records so much information that it may affect the performance of the ACE XML Gateway. These logging levels are useful when investigating a problem, but should be avoided on an ongoing basis in a production system.

Configuring Event Logging

Event logs items are generated by both the ACE XML Gateway and Manager. The types of events they generate are:

The ACE XML Gateway event logs provide information mainly on the message processing activities of the system.

The ACE XML Manager event logs provide information on administrative activities in the system.

In general, the ACE XML Manager event logs are useful to system administrators, while the ACE XML Gateway logs are helpful to both administrators and developers who are creating and testing service definitions in the policy.

The log level at which events are recorded can be separately configured for the Gateway and Manager.


Note If the Manager controls multiple clusters, the Event Log displays Gateway events only for the Gateways in the current cluster. Manager events are shown for all clusters. For Manager events, the log description indicates the cluster affected by the event, by cluster name. For more information, see Chapter 34, "Managing Gateway Clusters."


To set the event logging level, take the following steps:


Step 1 Log in to the web console as an Administrator user or a Privileged user with the Operations role.

Step 2 Display the System Management page in either of the following ways:

Click the System Management link in the navigation menu, or

If you're already viewing the Event Log page, click one of the edit links at the far right of the Current ... Event Logging pane.

The ACE XML Manager displays the System Management page.

Step 3 Choose a value from the Log all Manager events of type menu for Manager logging, or from the Log all Manager events of type menu for Gateway logging.

Step 4 Click the Set Log Level button next to the menu to confirm the new settings.


The new settings take effect immediately.

Viewing the Event Log

To view the event log, click the Event Log link in the Reports & Tools section of the navigation menu. By default, the ACE XML Manager displays events in the last hour. The search and filter tools at the top of the Event Log Viewer enable you to filter the logs that are displayed. For example, you can choose to view only event generated for a particular ACE XML Gateway instance. You can also search by message GUID, the globally unique identifier assigned to a given message transaction by the ACE XML Gateway. In this case, the Event Log Viewer displays only events associated with the request or response with that ID.

Monitoring Performance

The Performance Monitor provides extensive performance information on the system, including message count, sizes, and processing time. The performance monitor can help you identify bottlenecks in the system and optimize performance at the ACE XML Gateway and backend infrastructure.

Information is presented on the page by handler group and endpoint. For each item, a variety of performance statistics are shown. For descriptions of each statistical category, see the online help accessed from the Performance Monitor page.

If identity reporting is enabled, the monitor displays information by identity of the user accessing the service. To have identity-specific information appear in the monitor, the identity tracking feature needs to be enabled for the authenticator associated with the service. (For information on enabling identity tracking, see "Enabling Identity Reporting Features" section on page 6-89.)

Figure 32-1 Performance Information


Note It is important to note that statistics shown in the monitor should be regarded as approximate in some cases. In particular, messages that result in certain types of errors may not cause relevant statistics to be incremented as would be expected. Furthermore, the occurrence of this issue varies depending upon whether the message is handled by the Reactor or Flex Path. For example, a response handled by the Reactor that produces a validation error does not increment the BackendCount statistic as may be expected. Similarly, a backend service error (returning a 500 SOAP fault for the request) for a message on the Flex Path does not increment the ErrorCount statistic as expected.


Filtering Performance Data by Time

The performance monitor includes controls that let you filter the information by time in various ways. Time filtering affects the console view as well what information is exported to file. You can show statistics by:

A set time period ending at the present time, such as over the last hour or the last seven days.

A time period starting at a set time, such as at 10AM and ending at the present time.

A set time period ending in the past, such as from 10AM to 8PM on a given date.

When analyzing performance data, it is important to consider that the Manager's physical capacity for performance information is not unlimited. When the Manager's performance data capacity is reached, oldest performance information is lost. To conserve space in order to minimize this effect, the Manager consolidates information from smaller time frames into larger time frames over time, in effect, lowering the resolution of performance data as it ages. Therefore, while you can query the Manager performance information for a short-time span from a relatively distant time period of its operation, it's possible that the data returned is actually representative of a larger time period than requested. In this event, a notice at the top of the page indicates that the specified resolution is not available. Also, the actual values are reflected in the time filter fields at the top of the page.

The rate at which this data consolidation or loss occurs varies depending on the nature of the traffic in the system. It is worth noting that the most significant factor in reaching the performance capacity is the number of separate virtual services and, in particular, the use of identity reporting rather than the volume of traffic at the Gateway.

As a rough guideline, for a policy with about 100 virtual services, each of which gets constant traffic flow (about a request every ten seconds) and with identity tracking disabled, the Manager may be expected to reach its performance data capacity in seven to eight months. For a policy with just ten virtual services and no identity tracking, the Manager may be able to retain performance data without loss for several years.

Data consolidation, on the other hand, may occur after several hours. Given ten virtual services that each receive a message every ten seconds, data would be consolidated into a five-minute time frame after about six-and-a-half hours. Eight days later, data from the five-minute time frames would be consolidated into a single one-hour time frame, and so on.

If you request information in the Performance Monitor for a time interval at a resolution for which data is not available, the interface presents the closest time range that is available, and indicates that time range at the top of the page.

If maintaining historical performance information is important to you, you should export performance data to a file regularly. The Manager supports performance data export in CSV and XML formats.

When the Manager consolidates performance information into records that correspond to a day, it does so along day boundaries determined in GMT. Alternatively, you can have the Manager draw day-record boundaries at a specific time zone relative to GMT. To do so, set the Message statistics "day" boundary field in the Gateway Settings page of the Manager.

Viewing Performance Information

To view performance information:


Step 1 Log in to the web console as an Administrator user, Privileged user, or Policy View user.

Step 2 Click the Performance Monitor link in the Reports & Tools section of the navigation menu.


The Performance Monitor page lists performance statistics for the service definitions in the policy sorted into handler groups. By default the page displays statistics for all virtual services in your policy.

The handler group row shows total statistics for all virtual services in that group. Under the group name, statistics are broken down by each service.


Note For a multiple operation virtual service, statistics are not available for each operation in the virtual service, only for the entire virtual service.


You can use the controls at the top of the page to filter what information is displayed in various ways, such as by Gateway or time period.

There are a few points to note regarding these statistics:

The Request Processing and Response Processing times represent the amount of time it takes the ACE XML Gateway to perform validation, consumer authentication, transformation, or any other processing steps specified by the policy on the message.

The Service Latency column shows the time it takes from the point at which the ACE XML Gateway sends the request to the backend service until is receives the response. It does not include the time the ACE XML Gateway spends processing the message.

The total time it takes for message processing—including request processing, response processing, and service round trip—is indicated in the Processing Latency column.

These categories are shown in Figure 32-2.

Figure 32-2 Performance statistics categories

The times indicated in the Performance Monitor are based on time-to-first-byte. This means that the timer starts when the first byte of the message is received by the Gateway, and ends when the first byte is transmitted to the network from the Gateway. Accordingly, the values can be affected by network conditions, particularly if messages are composed of multiple packets.

For information on each performance category, see the online help for the performance monitor page.

Exporting Performance Information to a File

If left on the ACE XML Manager of a busy ACE XML Gateway system, performance data is eventually lost. When the amount of performance data reaches the Manager's capacity, the oldest information is deleted to make space for new information. If you need to retain information indefinitely, you can export performance information to a file.

In addition to providing a mechanism for saving performance data indefinitely, the performance data export feature provides access to richer information than that provided in the Performance Monitor interface, with additional statistical categories for message processing times.

Performance data can be exported as XML data or to a comma-separated values (CSV) file. As in the Performance Monitor, statistics in the exported file are grouped by handler.


Note When viewing performance monitor, note that handlers that have been moved between subpolicies are identified by internal object number, rather than by handler name, for their activity in the former subpolicy.


It is important to note that the information in exported files is presented differently from the performance monitor. The exported performance information should be considered raw data, in that it is not processed or organized for human-readability.

Note the following differences between exported data and the performance monitor:

Virtual services that have received traffic in the selected time frame are listed in the file. Virtual services that have not received requests do not appear in the generated file.

The performance monitor shows message processing totals for each handler group. The exported file does not show total values in the same way; instead, it contains a record for each virtual service. If identity reporting is enabled, it contains a record for each identity that accessed the service, with a request count for that identity.

The exported data file includes records for requests that were not serviced due to an error. They are indicated by an error count field with a value greater than 1.

In addition to the time to first byte measurement shown in the Performance Monitor, the exported file shows measurements for time-to-last-byte for each request and response.

To export performance data to an XML or CSV file:


Step 1 While logged into the web console as an Administrator user, Privileged user, or Policy View user, click the Performance Monitor link in the Reports & Tools section of the navigation menu.

Step 2 Use the Gateway and time controls to filter the information to be exported to the exported file.

In addition to affecting the view in the Performance Monitor, the filter controls, such as time spans, control what information is exported to a file.

Step 3 Click Update View.

Step 4 Choose the format of the output file, either:

XML, for an XML format file

CSV, for a comma delimited file

This choice does not affect what information is generated, only its format.

Step 5 Click Export Raw Data.

Step 6 In the File Save dialog, choose a file location and name for saving the export file.


After you save it, the file is generated and downloaded to the file location you specified.

The exported file contains all of the information shown in the Performance Monitor, plus some additional statistical categories. This information includes message error counts, such as access failures, and information on message size.

The XML file indicates the time frame represented by the data in the file with the Report element. The element has a queryStartTime and queryEndTime attribute, which indicates the time period for which performance data was captured for the file.

The file provides extensive details on time-based performance measures. Note the following points on this performance data:

Message timings are shown in microseconds (the Performance Monitor shows time in milliseconds).

Time measurements include the following statistics:

Time-to-first byte (TTFirst) is the time from when the Gateway receives the first byte of a message, off the network, until the time it starts sending the first byte of the message. The times shown in the Performance Monitor are time-to-first byte.

Time-to-last byte (TTLast) is the time from when the Gateway receives the last byte of a message until it sends the last byte of the message

In the names of the statistics categories, you can determine the message processing stage measured by the following identifiers:

Req is the request processing time, the amount of time the ACE XML Gateway spends processing the consumer request. An example is MinReqTTFirst.

Resp is the response processing time, the amount of time the ACE XML Gateway spends processing the response from the backend service. An example is MinRespTTFirst.

Source is the backend message roundtrip time, from when the outgoing request is sent to the service until the response is received back from the service. An example is MinSourceTTFirst.

Roundtrip is the total message processing time, which includes request processing, response processing, and the roundtrip to the backend service. An example is MinRoundtripTTFirst.

For a description of each statistical category, see the online help for the web console.

Service Health

The ACE XML Manager provides a summary display of the status of all service definitions in the policy. You can review this status display to quickly understand how heavily each handler is loaded and whether the ACE XML Gateway is reporting errors on any of the handlers. Service Health provides a quick way to locate a particular handler and to view its status.

To view the status of a handler, take the following steps:


Step 1 Log in to the web console as an Administrator user, Privileged user, or Policy View user.

Step 2 Click the Service Health link in the Reports & Tools section of the navigation menu.


The Service Health page lists each handler in the policy organized by handler group, along with a summary of the status for each. By default the page displays status for all defined handlers.

Use the filter controls at the top of the page to narrow the displayed items to only certain Gateway appliances or to traffic handled in a specified interval.

The Status column indicates whether the virtual service is functioning normally and without errors (OK, if operating normally), allowing you to quickly identify problems.

The Error Summary columns shows a brief summary of errors messages logged against each handler. Other columns show the current logging level for handler and a count of messages processed.

You can view the current logging level in the Message Logging column. For handlers on the Flex Path I/O processor in the Gateway, you can change the logging level using the menu in this column.


Note Handlers not on the Flex Path do not have this option, since only statistics logging is supported in these cases.


Message Logging

The ACE XML Gateway records information about each message that it processes. The recorded information is stored in the message-traffic log. You can use the ACE XML Manager's Message Traffic Log page to view, filter, and search the recorded information.

For message traffic, the ACE XML Gateway can record information at one of three levels:

statistics only

log message bodies of outbound messages

log message bodies of inbound and outbound messages.

Virtual services set to log statistics only record only cumulative totals describing the sizes of messages and the time taken to process them. Objects configured for outbound messages record a complete copy of each outbound message processed. Those configured for inbound and outbound logging record complete copies of the message bodies for both inbound and outbound messages.

Because different handlers may be set to different logging levels, the information recorded in Message Log entries is not uniform. When examining recorded message data keep in mind that some entries will have been made by handlers at one logging level, and some at another.

The exact information recorded depends on how each handler is configured. You can use an editor accessible from each handler's information page to configure the handler to record more or less information about the messages it processes. At its most detailed setting a handler records everything it knows about each message, including a full copy of the message itself. This most-detailed setting requires some caution; recording a complete copy of many large messages can fill up the largest disk very quickly.

Alternatively, you can configure a handler to record only processing statistics—a count of messages handled together with size and processing times. This statistics-only setting is much faster than recording whole messages, but provides less information about individual messages, which can be valuable when you are debugging problems.

In general, if performance is important, it is suggested that you use statistics only logging for production systems, but more verbose message logging for policy testing or development.

Configuring Message Logging

The message logging level in the policy is a property of a virtual service. You can configure message logging when the object is created or change it as follows.


Step 1 As an Administrator user or Privileged user with the Operations role, log in to the ACE XML Manager web console and open the Virtual Services browser.

Message logging can be enabled by individual service definition object or for several service definitions at a time by handler group.

Step 2 To set up message logging for a single service definition

a. Click the virtual service object or handler that handles the messages for which you would like to enable message logging.

b. Click the Edit link next to the General settings header.

c. From the Default Message Logging menu, choose log bodies of inbound and outbound messages.

Step 3 To configure message logging by handler group

a. In the Virtual Services browser, click on the name of the handler group.

b. In the Set message logging levels for all members to menu, choose log bodies of inbound and outbound messages.

Step 4 Click Save Changes and deploy the policy to have the change take effect.


Message logging in an active system can consume a significant amount of disk space on the appliance. You can limit the amount of space consumed by message logging from the System Management > Gateway Settings page. Configure the limit using the option labelled Delete old log files when total message log disk usage exceeds.

Viewing the Message Traffic Log

To view the message traffic log, take the following steps:


Step 1 Log in to the ACE XML Manager web console as a user that has the Message Traffic Log role.

Step 2 Click the Message Traffic Log link in the Reports & Tools section of the navigation menu.

The Message Traffic Log page appears.

Step 3 Search the log for messages of interest:

Set the controls to filter search results by consumer, handler, service GUID and various other criteria.

For details, see the remainder of this section.

Click the Update View button.

The Search Results pane displays log entries that meet the search criteria.


The ACE XML Manager's Message Traffic Log page lists the contents of the message log and provides tools for filtering and searching entries. By default the Message Traffic Log page displays every entry in the log.

The log on a busy Gateway can be quite large, and the sheer number of entries can present an obstacle to troubleshooting. The search tabs at the top of the page enables you to construct queries that instead display only those log entries that match the query.

The search tabs are:

Simple Search, the one displayed by default, provides the most commonly used search options.

Advanced Search displays additional search options, such as search by service, consumer, or request or response attribute.

GUID Search search lets you search for a specific message when you know the Globally Unique Identifier (GUID) assigned to it by the ACE XML Gateway. The GUID search option is most useful when you are tracing the ACE XML Gateway's handling of a specific message you have identified by inspecting the log.

User Search lets you search by user identifier. This option requires that you have enabled user-identity features for a particular authenticator. For more information on enabling user information in logs, see "Identity Reports" section on page 6-89.

The page displays information about log entries in a list, limiting the number of entries shown to the number you set in the search tool. By default it shows 25 log entries per page. Each entry shows the time the entry was made, along with other information about it, such as the IP address of the particular Gateway appliance that recorded it, the message type, the handler, the authenticator that accepted the request, and so on. The exact information depends on the logging configuration of the recording handler at the time the entry was made.

For more information about a particular entry, click the Details link at the right end of the entry's row.

Compliance Reports

Standardized accounting requirements, such as the Sarbanes-Oxley act (or SOX, for short), require organizations to be able to review and report on business processes transactions that have occurred. The ability to account for changes to policies applicable to business processes is important as well.

As a development and enforcement point for such policies, as well as the Gateway for the type of traffic subject to such requirements, the ACE XML Gateway is well-positioned for acquiring and reporting information needed for compliance requirements.

The ACE XML Gateway provides tools that help you meet compliance requirements, including tools for setting log levels on policy objects and for generating compliance reports, as described in this section.

Viewing and Enabling Compliance Logging

In most cases, adhering to compliance reporting requirements requires the ability to audit transactions between business processes. To comply with these types of requirements, therefore, you must first enable message logging in some form on the relevant traffic handlers in the ACE XML Gateway policy.

The Compliance Report page provides an overview of the logging configuration of each service policy object (that is, the handler and service descriptor) in the currently deployed policy.

To view logging status for the policy:


Step 1 Click the Compliance Report link in the Reports & Tools area of the navigation menu.

The Compliance Report tabbed pane appears. The Logging Settings tab indicates whether logging is enabled for the handlers and service descriptors in the current policy.

Logging options can be set to either:

enabled—The Default Message Logging property for the handler or service descriptor is set to log headers of outbound messages, or to a level sufficient for compliance reporting.

disabled—The Default Message Logging property for the handler or service descriptor is set at log statistics only (no message content), which, for relevant services, is a level insufficient for compliance reporting purposes.

Step 2 To change the compliance logging status for a particular virtual service, click the edit link in the Compliance Logging column of the virtual service.

The General Information editing page appears.

Step 3 Choose from one of the following log levels from the Default Message Logging menu:

log headers of outbound messages

log bodies of outbound messages

log bodies of inbound and outbound messages

Step 4 Click Save Changes.

The information page for the handler appears.

Step 5 To return to the Compliance Report page, click the Compliance Report link in the navigation menu.

Notice that the status for the handler remains at the original setting until you have deployed the policy.


When finished, the ACE XML Gateway logs transaction information for traffic handled by the virtual service you configured. The information will appear in the compliance reports you subsequently generate, as described in "Generating Compliance Reports" section.

Generating Compliance Reports

To generate a compliance report:


Step 1 In the Reports & Tools area of the Navigation menu, click the Compliance Report link.

The Compliance Report tabbed pane appears. The tabs shows the logging status for the service objects, by handler, in your policy. For more information on setting and understanding compliance logging, see "Generating Compliance Reports" section.

Other tabs in the Compliance Report page are:

Test Messages. Shows information relating to traffic issued from the test browser within the ACE XML Manager web console.

Message Traffic. Information relating to traffic issued from the test browser within the web console.

Policy Changes. Shows activities associated with ACE XML Gateway administrators, in particular, it shows changes to the ACE XML Gateway policy.

Event Log. Display the event logs.

Step 2 To change the time span of messages shown in the log report panes (the also affects what messages are included in generated report):

a. Choose the new time span from the Show pane. You can either choose to show message generated in the most recent 7, 30, or 60 days, or specify a date range by choosing the date range option.

b. Click Update View.

Step 3 To generate an XML formatted report of all activities, click the Export as XML button.


The generated log file appears in a pop-up browser window. Notice that the log file reflects the time frame you specified in the previous step. Also, the ACE XML Manager generates a composite log file—it contains message traffic, test activities, administrative activities and events—as XML formatted data.