-
Cisco ACNS Software Configuration Guide for Locally Managed Deployments, Release 5.2
-
Title
-
Preface
-
Chapter 1: Introduction
-
Chapter 2: Understanding the Basics
-
Chapter 3: Getting Started
-
Chapter 4: Deployment Scenarios for Standalone Content Engines
-
Chapter 5: Configuring Transparent Redirection for Standalone Content Engines
-
Chapter 6: Configuring Conventional Caching Services for Standalone Content Engines
-
Chapter 7: Configuring WMT Streaming Media Services on Standalone Content Engines
-
Chapter 8: Configuring RTSP Media Services on Standalone Content Engines
-
Chapter 9: Configuring Content Authentication and Authorization on Standalone Content Engines
-
Chapter 10: Configuring URL Filtering on Standalone Content Engines
-
Chapter 11: Configuring ICAP on Standalone Content Engines
-
Chapter 12: Configuring the Rules Template on Standalone Content Engines
-
Chapter 13: Configuring Primary and Backup Proxy Servers for Standalone Content Engines
-
Chapter 14: Configuring Advanced Transparent Caching Features on Standalone Content Engines
-
Chapter 15: Configuring Additional Network Interfaces and Bandwidth on Standalone Content Engines
-
Chapter 16: Configuring Login Authentication and Authorization on Standalone Content Engines
-
Chapter 17: Configuring AAA Accounting on Standalone Content Engines
-
Chapter 18: Creating and Managing IP Access Control Lists for Standalone Content Engines
-
Chapter 19: Viewing and Modifying TCP Stack Parameters on Standalone Content Engines
-
Chapter 20: Monitoring and Troubleshooting
-
Appendix A: Content Engine GUI Menu Options
-
Appendix B: Reference Material for Standalone Content Engine Deployments
-
Table Of Contents
Monitoring and Troubleshooting
Monitoring Standalone Content Engines
Monitoring Standalone Content Engines with the Cisco Discovery Protocol
Monitoring Standalone Content Engines with SNMP
Understanding the Different Versions of SNMP
Downloading MIB Files to Standalone Content Engines
Enabling the SNMP Agent on Standalone Content Engines
Defining SNMP Users for Standalone Content Engines
Configuring Standalone Content Engines to Send SNMP Traps
Disabling the SNMP Agent on Standalone Content Engines
Disabling SNMP Traps on Standalone Content Engines
Monitoring Standalone Content Engines with the ACNS Software Alarms
Checking for Application Liveness
Configuring SNMP Alarm Traps on Standalone Content Engines
Displaying Information About ACNS Software Alarms
Displaying Details About ACNS Software Alarms
Displaying the History of ACNS Software Alarms
Displaying the Status of ACNS Software Alarms
Monitoring Transactions with Standalone Content Engines
Displaying Statistics for Particular Protocols
Monitoring Critical Disk Drives on Standalone Content Engines
Specifying the Disk Error-Handling Threshold
Manually Marking and Unmarking Content Engine Disk Drives
Displaying Details About the Current Disk Configuration
Removing All Disk Partitions on a Single Disk Drive
Using ACNS Software Transaction Logs
Enabling Squid-Style Transaction Logging
Enabling Extended Squid-Style Transaction Logging
Enabling Apache-Style Transaction Logging
Enabling Custom Format Transaction Logging
Logging Windows Domain with Authenticated Usernames
Exporting Transaction Log Files
Exporting Transaction Logs to External FTP or STFP Servers
Changing the Transaction Logging Export Settings on Standalone Content Engines
Restarting Export After Receiving a Permanent Error from the External FTP Server
Disabling Transaction Logging Export on Standalone Content Engines
Monitoring HTTP Request Authentication Failures in Real Time
Configuring the Remote Syslog Host for Real-Time Transaction Logging
Configuring a Transaction Log Facility when Logging to a Remote Syslog Host
Specifying the Transaction Log Entry Type when Logging to a Remote Syslog Host
Displaying the Transaction Log Configuration for Standalone Content Engines
Using the System Logging Feature
Displaying the Current Configuration for System Logging
Configuring System Logging on Standalone Content Engines
Configuring System Logging to the Console
Configuring System Logging to Disk
Configuring System Logging to Remote Syslog Hosts
Mapping Syslog Priority Levels to RealProxy Error Codes
Creating HTTP Custom Error Pages for Standalone Content Engines
Troubleshooting with the HTTP CLI Client
Troubleshooting with the Log Viewing Tool
Troubleshooting with the Telnet Client
Troubleshooting with Traceroute
Monitoring and Troubleshooting
This chapter describes how to monitor and troubleshoot locally managed deployments (standalone Content Engines). This chapter contains the following sections:
•
Monitoring Standalone Content Engines
•
•
•
•
•
•
•
Note
For information about monitoring and troubleshooting a Content Router, Content Distribution Manager, or a Content Engine that is registered with a Content Distribution Manager (as opposed to standalone Content Engines that are not registered with a Content Distribution Manager), refer to the Cisco ACNS Software Configuration Guide for Centrally Managed Deployments, Release 5.2.
Showing Inventory
Cisco Content Engines are embedded with following three identification items:
•
•
•
This identity information is stored in non-volatile memory. Each Content Engine has a unique device identifier (UDI). The UDI = PID + VID + SN.
The UDI is electronically accessed by the product operating system or network management application to enable identification of unique hardware devices. Consequently, the data integrity of the UDI is vital to customers. This means that the UDI that is programmed into the Content Engine's non-volatile memory is equivalent to the UDI that is printed on the product label and on the carton label. This UDI is also equivalent to the UDI that can be viewed through any electronic means, and in all customer-facing systems and tools.
In ACNS 5.2 software, the show inventory EXEC command was added to allow the administrator of a Content Engine to view the Content Engine's UDI. Currently, there is only CLI access to the UDI; there is no SNMP access to the UDI information.
All of the Content Engine models that are supported in ACNS 5.2 software, support this new show inventory feature.
On newer Content Engine models, you can display the Content Engine's UDI by using a single command, the show inventory EXEC command.
CE-565# show inventoryPID: CE-565-K9 VID: 0 SN: serial number
where serial number is the serial number of the Content Engine. If the version number is available, then it is displayed; otherwise, zero "0" is displayed (as shown above).
On older Content Engine models (for example, the CE-507 or CE-2636), you must use the show tech-support and show inventory EXEC commands to display the Content Engine's UDI.
CE-507# show inventoryPlease look at 'sh tech-support' for information!CE-507# show tech-support
Note
Monitoring Standalone Content Engines
It is important that you monitor your Content Engines in order to gauge their performance and to identify any signs that you need to tune their configurations or deploy additional Content Engines. This section describes how use the Simple Network Management Protocol (SNMP) and ACNS software alarms to monitor standalone Content Engines.
Several tools are available to monitor the performance of standalone Content Engines that are running ACNS software, Release 5.2 or later. This set of tools includes the Cisco Discovery Protocol (CDP), SNMP, and ACNS software alarms. For more information, see the following sections:
•
•
•
Monitoring Standalone Content Engines with the Cisco Discovery Protocol
Cisco Discovery Protocol (CDP) is a device discovery protocol that runs on all Cisco-manufactured devices. With CDP, each device within a network sends periodic messages to all other devices within the network. Devices listen to periodic messages sent by others in order to learn about neighboring devices and determine the status of their interfaces.
With CDP, network management applications can learn the device type and the SNMP agent address of neighboring devices. Applications are then able to send SNMP queries within the network. Also, CiscoWorks2000 discovers the Content Engine by noticing the CDP packets that are sent by the Content Engine after booting.
Content Engine-related tasks require that the Content Engine platform support CDP in order to be able to notify the system manager of the existence, type, and version of the Content Engine platform.
The following example enables CDP implementation on a standalone Content Engine with a single CLI command:
ContentEngine(config)# interface FastEthernet 0/0 cdp enableMonitoring Standalone Content Engines with SNMP
Simple Network Management Protocol (SNMP) is an interoperable standards-based protocol that allows for external monitoring of the Content Engine through an SNMP agent.
An SNMP-managed network consists of three primary components: managed devices, agents, and management systems.
•
•
•
Each Content Engine that is running ACNS 5.x software has an SNMP agent that is responsible for gathering information about the Content Engine's device configuration and activity. Before you can access this SNMP information, you must have deployed an SNMP management application on a management station. This SNMP management station is referred to as the SNMP host because it uses SNMP to send the device agent an SNMP Get request to obtain information from the Content Engine.
The SNMP management station and the device agent (the SNMP agent on the Content Engine) use SNMP to communicate, as follows:
1.
2.
Note
3.
Figure 20-1 illustrates these SNMP operations when the Content Engine has been standalone.
Figure 20-1 SNMP Components with a Standalone ACNS Content Engine
Understanding the Different Versions of SNMP
ACNS 5.x software supports the following versions of SNMP:
•
•
•
SNMP Security Models and Security Levels
SNMPv1 and SNMPv2c do not have any security (that is, authentication or privacy) mechanisms to keep SNMP packet traffic on the wire confidential. As a result, packets on the wire can be detected and SNMP community strings compromised.
To solve the security shortcomings of SNMPv1 and SNMPv2c, SNMPv3 provides secure access to Content Engines by authenticating and encrypting packets over the network. The SNMP agent in the ACNS 5.x software supports SNMPv3 as well as SNMPv1 and SNMPv2c.
The following security features are provided in SNMPv3:
•
•
•
SNMPv3 provides security models as well as security levels. A security model is an authentication process that is set up for a user and the group in which the user resides. A security level is the permitted level of security within a security model. A combination of a security model and a security level determines which security process is used when an SNMP packet is handled. Three security models are available: SNMPv1, SNMPv2c, and SNMPv3.
Table 20-1 describes the combinations of security models and security levels.
The SNMPv3 agent can be used in the following modes:
•
•
•
Using SNMPv3, users can securely collect management information from their SNMP agents without fear that the data has been tampered with. Also, confidential information, such as SNMP set packets that change a Content Engine's configuration, can be encrypted to prevent their contents from being exposed on the wire. Also, the group-based administrative model allows different users to access the same SNMP agent with varying access privileges.
Supported MIBs
The ACNS 5.x software implementation of SNMP supports the following MIBs:
•
•
•
•
•
•
•
Note
In ACNS 5.2 software, six generic alarm traps were added to this MIB for SNMP and Node Health Manager integration. For a list of these six generic alarm traps, see Table 20-5.In ACNS software, Release 5.1 or later, you can use IP access control lists (ACLs) to control SNMP access on a Content Engine. For more information about defining IP ACLs, see "Creating and Managing IP Access Control Lists for Standalone Content Engines."
Downloading MIB Files to Standalone Content Engines
You can download the MIB files for all of the MIBS that are supported by a standalone Content Engine that is running ACNS 5.x software, from the following Cisco FTP site:
ftp://ftp.cisco.com/pub/mibs/v2
The MIB objects that are defined in each MIB are described in the MIB files at the above FTP site are self explanatory.
Enabling the SNMP Agent on Standalone Content Engines
By default, the SNMP agent on a standalone Content Engine is disabled and an SNMP community string is not defined. The SNMP community string is used as a password for authentication when accessing the SNMP agent on the standalone Content Engine. In order to be authenticated, the Community Name field of any SNMP message sent to the standalone Content Engine must match the SNMP community string defined on the standalone Content Engine.
The SNMP agent on the standalone Content Engine is enabled when an SNMP community string is defined on the Content Engine. You can use the Content Engine GUI or CLI to define an SNMP community string and enable the SNMP agent, as follows:
•
•
ContentEngine(config)# snmp-server community comaccessIf the SNMPv3 protocol is going to be used for SNMP requests, the next step is to define an SNMP user account that can be used to access a standalone Content Engine through SNMP. For more information on how to create an SNMPv 3 user account on a standalone Content Engine, see the "Defining SNMP Users for Standalone Content Engines" section.
Defining SNMP Users for Standalone Content Engines
When defining SNMP users for standalone Content Engines, note the following:
•
•
Defining SNMPv3 Users
You can use either the Content Engine GUI or the CLI to define SNMPv3 user accounts on standalone Content Engines.
To use the Content Engine GUI to configure an SNMPv3 user account on a standalone Content Engine, follow these steps:
Step 1
Step 2
Step 3
a.
b.
c.
Tip
d.
–
–
e.
f.
Step 4
Step 5
Step 6
To use the Content Engine CLI to define an SNMPv3 user account on a standalone Content Engine (the SNMP server), use the snmp-server user global configuration command. Use the no form of this command to remove SNMP access.
snmp-server user name group [auth {md5 password [priv password] | sha password [priv password]} | remote octetstring [auth {md5 password [priv password] | sha password [priv password]}]]
Table 20-2 describes the parameters for the snmp-server user command.
In the following example, an SNMPv3 user account is created on the Content Engine. The SNMPv3 user is named acme and belongs to the group named admin. Because this SNMP user account has been set up with no authentication password, the SNMP agent on the Content Engine will not perform authentication on SNMP requests from this user.
ContentEngine(config)# snmp-server user acme adminConfiguring Standalone Content Engines to Send SNMP Traps
You can use either the Content Engine GUI or the CLI to configure standalone Content Engines to send SNMP traps.
From the Content Engine GUI, choose System > SNMP. The SNMP window appears. From the SNMP window, do one of the following:
•
For example, you must define the SNMP trap host by specifying the host name or IP address of the SNMP trap host that will be sent in the SNMP trap messages from the Content Engine.
•
For more information about configuring SNMPv3 user accounts, see the "Defining SNMPv3 Users" section. For information about the fields in the SNMP windows, click the HELP button in the window.
When using the Content Engine CLI to configure a standalone Content Engine to send SNMP traps, note the following:
•
•
To use the Content Engine CLI to configure SNMP traps on a standalone Content Engine, follow these steps:
Step 1
snmp-server group name {v1 [notify name] [read name] [write name] | v2c [notify name] [read name] [write name] | v3 {auth [notify name] [read name] [write name] | noauth [notify name] [read name] [write name] | priv [notify name] [read name] [write name]}}
where:
Step 2
ContentEngine(config)# snmp-server enable trapsIf you do not enter the snmp-server enable traps command, no traps are sent. Use the no form of this command to disable all SNMP traps or only SNMP authentication traps.
Step 3
Note
The following shows how to configure the Content Engine to send all SNMP traps to the host 172.31.2.160 using the community string public.
ContentEngine(config)# snmp-server host 172.31.2.160 publicStep 4
In the following example, comaccess is specified as the password.
ContentEngine(config)# snmp-server community comaccess
Tip
The snmp-server community string global configuration command provides view-based access control for SNMPv1, SNMPv2c, and SNMPv3 but also continues to provide backward compatibility between different versions.
In ACNS 5.x software prior to Release 5.1, the snmp-server community string global configuration command did not have an option to create a community string that allows SNMP messages to execute a set operation on a MIB object. A rw option has been introduced for this purpose. Also, the previous version of the SNMP agent did not provide selective access control to MIB objects. Access to any MIB object was denied or granted based on authentication of the SNMP community string.
With the introduction of view-based access control, it is now possible to configure a community string that grants access to only part of the MIB subtree. To provide backward compatibility with the previous version of this command, a default read group or default write group (if the rw option is specified on the command line) is associated with the community string if no group name is specified. Both of these default groups are hidden from users and not displayed in the configuration file or in the show snmp group EXEC command, but are created during initialization of the SNMP agent
Disabling the SNMP Agent on Standalone Content Engines
To disable the SNMP agent on standalone Content Engines, enter the no snmp-server global configuration command.
ContentEngine(config)# no snmp-serverTo disable the SNMP agent and to remove the previously defined community string, enter the no snmp-server community global configuration command.
ContentEngine(config)# no snmp-server communityDisabling SNMP Traps on Standalone Content Engines
To disable all SNMP traps on standalone Content Engines, enter the no snmp-server enable traps global configuration command.
ContentEngine(config)# no snmp-server enable trapsTo disable the sending of the MIB-II SNMP authentication trap, you must enter the no snmp-server enable traps snmp authentication command.
Monitoring Standalone Content Engines with the ACNS Software Alarms
Traditionally SNMP is used to report error conditions by generating SNMP traps. ACNS 5.x continues to use this monitoring mechanism, as described in the "Monitoring Transactions with Standalone Content Engines" section.
In ACNS 5.2 software, the Node Health Manager feature was added. The Node Health Manager enables ACNS applications to raise alarms to draw attention in error/significant conditions. The Node Health Manager, which is the data repository for such alarms, aggregates the health and alarm information for the applications, services (for example, the cache service) and resources (for example, disk drives) that are being monitored on the Content Engine. For example, this new feature gives you a mechanism to determine if a monitored application (for example, the HTTP proxy caching service) is alive on the Content Engine. These alarms are referred to as "ACNS software alarms."
In ACNS 5.2 software, the following Content Engine applications can generate an ACNS software alarm:
•
•
•
Alarms that have been raised on a Content Engine can be listed using the Content Engine CLI commands in Table 20-3. SNMP traps are sent all raised and cleared alarms. The type of SNMP trap sent varies by alarm.
Note
In ACNS 5.2 software, several CLI commands were added to allow you to systematically drill down to the source of an ACNS software alarm (the cause of the problem). See Table 20-3. You can use these CLI commands to identify the source of a problem without wading through numerous ACNS software logs.
Table 20-3 List of Show Alarms CLI Commands
show alarm
Display a list of all currently raised ACNS software alarms (critical, major, and minor alarms) on the Content Engine.
See the "Displaying Information About ACNS Software Alarms" section.
show alarm critical
Display a list of only currently raised ACNS software critical alarms on the Content Engine.
See the "Displaying Information About ACNS Software Alarms" section.
show alarm major
Display a list of only currently raised ACNS software major alarms on the Content Engine.
See the "Displaying Information About ACNS Software Alarms" section.
show alarm minor
Display a list of the currently raised ACNS software minor alarms on the Content Engine.
See the "Displaying Information About ACNS Software Alarms" section.
show alarm detail
Display detailed information about the currently raised ACNS software alarms.
See the "Displaying Details About ACNS Software Alarms" section.
show alarms history
Display a history of ACNS software alarms that have been raised and cleared on the Content Engine. The CLI retains the last 100 alarm raise and clear events only.
See the "Displaying the History of ACNS Software Alarms" section.
show alarms status
Display the counts for the currently raised ACNS software alarms on the Content Engine. Also lists the alarm overload state and the alarm overload settings.
See the "Displaying the Status of ACNS Software Alarms" section.
Note
Alarm Severity Levels
There are three levels of ACNS software alarms. See Table 20-4.
Note that the output of the show alarms history EXEC command includes the severity of the ACNS software alarm. See sample below.
ContentEngine# show alarms historyOp Sev Alarm ID Module/Submodule Instance-- --- -------------------- -------------------- --------------------1 C Mi servicedead nodemgr mediacache2 C Mi servicedead nodemgr cache3 R Mi servicedead nodemgr mediacache4 R Mi servicedead nodemgr cache5 C Mi servicedead nodemgr rpc_httpd6 R Mi servicedead nodemgr rpc_httpd7 C Mi servicedead nodemgr rpc_httpd8 R Mi servicedead nodemgr rpc_httpd9 C Mi servicedead nodemgr mediacache10 C Mi servicedead nodemgr cache11 R Mi servicedead nodemgr mediacache12 R Mi servicedead nodemgr cache13 C Mi servicedead nodemgr cache14 C Mi servicedead nodemgr mediacache15 C Mi servicedead nodemgr rtspg16 R Mi servicedead nodemgr cache17 R Mi servicedead nodemgr mediacache18 R Mi servicedead nodemgr rtspgOp - Operation: R-Raised, C-ClearedSev - Severity: Cr-Critical, Ma-Major, Mi-MinorAlarm Overload
Content Engines that are running ACNS software, Release 5.2 and later can track the rate of incoming alarms from the Node Health Manager. If the rate of incoming alarms exceeds the high water mark (HWM), then the Content Engine enters an alarm overload state. When a standalone Content Engine is in an alarm overload state, then the following occurs:
•
•
•
Checking for Application Liveness
The Node Manager tracks the liveness of applications that it spawns on the Content Engine (for example, the HTTP cache application, the WMT streaming application, and the RTSP gateway (RTSPG) streaming application). When the Node Manager detects the death of a spawned application, it raises an alarm. An application is considered dead when the Node Manager does not receive keepalives from the application.
When an application dies, the Node Manager raises a servicedied alarm to report the condition, and then it restarts the service. If the service continues to run for a little while (typically 10 seconds) the 'servicedied' alarm is cleared.
If the application dies again after restarting, the servicedied alarm continues to be raised and the Node Manager attempts to restart it. Restarts are done typically a maximum of 10 times by the Node Manager. After the that the Node Manager raises a svcdisabled alarm for the service, clears the 'servicedied' alarm, and it stops restarting the service.
To restart the service, you must unconfigure and configure the features (for example, in the case of the NTP service, enter the no ntp server hostname | IP address global configuration command to unconfigure the NTP service, and then enter the ntp server hostname | IP address global configuration command to reconfigure the NTP service.
Configuring SNMP Alarm Traps on Standalone Content Engines
You can configure a Content Engine to generate an SNMP trap for a specific alarm condition. You can configure the generation of SNMP alarm traps on standalone Content Engines based on the following:
•
•
In ACNS 5.2 software, the following six generic alarm traps were added to the CISCO-CONTENT-ENGINE-MIB (the CCE MIB). See Table 20-5.
Note
These six general alarm traps provide SNMP and Node Health Manager integration. Each of these six alarm traps can be enabled or disabled through the Content Engine CLI. In ACNS 5.2 software, the snmp-server enable traps global configuration command was extended to include an alarm option (see below).
ContentEngine(config)# snmp-server enable traps alarm ?clear-critical Enable clear-critical alarm trapclear-major Enable clear-major alarm trapclear-minor Enable clear-minor alarm trapraise-critical Enable raise-critical alarm trapraise-major Enable raise-major alarm trapraise-minor Enable raise-minor alarm trapIn the following example, the Content Engine (the SNMP server) is configured to generate an SNMP trap if a critical alarm is cleared.
ContentEngine(config)# snmp-server enable traps alarm clear-criticalDisplaying Information About ACNS Software Alarms
Use the show alarm EXEC command to display information about all of the currently raised critical, major, and minor alarms for a standalone Content Engine. If there are no alarms currently raised on the Content Engine, the output indicates "None." See sample output below.
ContentEngine# show alarmCritical Alarms:----------------NoneMajor Alarms:-------------NoneMinor Alarms:-------------NoneYou can also display information for only a specific level of ACNS software alarms that are currently raised on the Content Engine, as follows:
•
•
•
Note
Displaying Details About ACNS Software Alarms
To display details about the currently raised SNMP alarms, use the show alarm detail EXEC command. This command allows you to drill down and obtain more information about a particular alarm.
Displaying the History of ACNS Software Alarms
Use the show alarms history EXEC command to display a history of ACNS software alarms that have been raised and cleared on a standalone Content Engine.
ContentEngine# show alarms historyOp Sev Alarm ID Module/Submodule Instance-- --- -------------------- -------------------- --------------------1 C Mi servicedead nodemgr mediacache2 C Mi servicedead nodemgr cache3 R Mi servicedead nodemgr mediacache4 R Mi servicedead nodemgr cache5 C Mi servicedead nodemgr rpc_httpd6 R Mi servicedead nodemgr rpc_httpd7 C Mi servicedead nodemgr rpc_httpd8 R Mi servicedead nodemgr rpc_httpd9 C Mi servicedead nodemgr mediacache10 C Mi servicedead nodemgr cache11 R Mi servicedead nodemgr mediacache12 R Mi servicedead nodemgr cache13 C Mi servicedead nodemgr cache14 C Mi servicedead nodemgr mediacache15 C Mi servicedead nodemgr rtspg16 R Mi servicedead nodemgr cache17 R Mi servicedead nodemgr mediacache18 R Mi servicedead nodemgr rtspgOp - Operation: R-Raised, C-ClearedSev - Severity: Cr-Critical, Ma-Major, Mi-MinorUse the show alarms history detail support EXEC command to display additional information about an alarm.
Content Engine# show alarms history detail supportOp Sev Alarm ID Module/Submodule Instance-- --- -------------------- -------------------- --------------------1 C Mi servicedead nodemgr rtspgJul 2 18:22:04.577 UTC, Processing Error Alarm, #000001, 2000:330004nodemgr: The rtspg service has died./alm/min/nodemgr/-service_name-/servicedead:-service name- service has died.Explanation:The node manager found the specified service to be dead.Attempts will be made to restart this service.Action:Examine the syslog for messages relating to cause of servicedeath. The alarm will be cleared if the service staysalive and does not restart in a short while.2 R Mi servicedead nodemgr rtspgJul 2 18:21:54.231 UTC, Processing Error Alarm, #000001, 2000:330004nodemgr: The rtspg service has died./alm/min/nodemgr/-service_name-/servicedead:-service name- service has died.Explanation:The node manager found the specified service to be dead.Attempts will be made to restart this service.Action:Examine the syslog for messages relating to cause of servicedeath. The alarm will be cleared if the service staysalive and does not restart in a short while.Op - Operation: R-Raised, C-ClearedSev - Severity: Cr-Critical, Ma-Major, Mi-MinorDisplaying the Status of ACNS Software Alarms
Use the show alarms status EXEC command to display the counts for all currently raised alarms on the Content Engine. As the sample output below shows, the number of currently raised ACNS software alarms. The output also includes information about the alarm overload settings (for example, if overload detection is currently enabled or disabled on the Content Engine).
ContentEngine# show alarms statusCritical Alarms : 0Major Alarms : 0Minor Alarms : 0Overall Alarm Status : NoneDevice is NOT in alarm overload state.Device enters alarm overload state @ 10 alarms/sec.Device exits alarm overload state @ 1 alarms/sec.Overload detection is DISABLED.Monitoring Transactions with Standalone Content Engines
Typically, Content Engine administrators are interested in what types of requests have been made of the Content Engine and what the results of these requests were. For example, if streaming media is a source of revenue for a company, then the company needs a way to track which customer is accessing which content, how long a user viewed the content, and at what viewing quality. Because these companies charge their customers to stream on-demand content and live broadcasts, they must rely on logged information as the basis for billing their customers for their content access services.
The software logs that record requests that are serviced by a Content Engine are referred to as "transaction logs." Typical fields in the transaction log are the date and time when a client request was made, the URL that was requested, whether it was a cache hit or a cache miss, the type of request, the number of bytes transferred, and the source IP address.
Transaction logs are generally used for the following purposes:
•
•
•
•
•
•
In ACNS 5.2 software, support for Windows Media Services 9 logging was added. The Windows Media Services 9 Series provides a more robust logging model than Windows Media Services Version 4.1.
You can log to a predefined format (for example, Squid, Extended Squid, or Apache, or a custom transaction log formats that allows you to log additional fields). The contents of the transaction logs can be exported to an external server using FTP at periodic intervals. You can also configure log rotation policies.
Standalone Content Engines that are running ACNS 5.x software can record all errors and access activities for reporting purposes. In ACNS 5.x software, each content service module (for example, the HTTP module, the WMT server, the FTP proxy process, and the TFTP server) on the Content Engine provides logs of the requests that were serviced. For example, logging for the following types of requests are provided:
•
•
•
•
•
•
•
Note
Displaying Statistics for Particular Protocols
For each content transport protocol, there is a corresponding show protocol-name statistics EXEC command that displays the statistics for that particular protocol. For example, you can use the show statistics http EXEC commands to display important statistical information about the HTTP requests that the Content Engine has serviced.
ContentEngine# show statistics http ?cluster Display healing mode statisticsims Display If-Modified-Since statisticsmiss-reason Display miss/revalide/no-store reason statisticsobject Display object statisticsperformance Display performance statisticsproxy Display proxy mode statisticsrequests Display request statisticssavings Display savings statisticsusage Display usage statisticsSee Table 20-6 for a list of the show protocol-name statistics EXEC commands. For detailed information about this commands, refer to the Cisco ACNS Software Command Reference, Release 5.2 publication.
The following shows how to use the show statistics https requests EXEC command to display monitoring statistics about the HTTPS requests that a Content Engine has serviced.
ContentEngine# show statistics https requestsHTTPS StatisticsTotal % of Total---------------------------------------------------Total connections: 1328 -Tunneled (CONNECT): 0 0.0Tunneled (wccp): 0 0.0SSL terminated: 0 0.0Connection errors: 0 0.0Total bytes: 8013157 -Bytes received from client: 1602824 20.0Bytes sent to client: 6410333 80.0Bytes received from server: 0 0.0The following shows how to use the show statistics http proxy outgoing EXEC command to display monitoring statistics for successful and failed outgoing proxy requests.
ContentEngine# show statistics http proxy outgoingHTTP Outgoing Proxy StatisticsIP PORT ATTEMPTS FAILURES--------------------------------------------------------------------10.10.10.10 1 49026 49026The following shows how to use the show statistics http requests EXEC command to display statistics about the HTTP requests that a Content Engine has received.
ContentEngine# show statistics http requestsStatistics - RequestsTotal % of Requests---------------------------------------------------Total Received Requests: 525979748 -Forced Reloads: 501468 0.1Client Errors: 81834 0.0Server Errors: 149808 0.0URL Blocked (Reset): 514998075 97.9URL Blocked: 0 0.0Sent to Outgoing Proxy: 0 0.0Failures from Outgoing Proxy: 0 0.0Excluded from Outgoing Proxy: 0 0.0ICP Client Hits: 0 0.0ICP Server Hits: 0 0.0If-Range Hits: 32 0.0HTTP 0.9 Requests: 677 0.0HTTP 1.0 Requests: 524097101 99.6HTTP 1.1 Requests: 1881966 0.4HTTP Unknown Requests: 4 0.0Non HTTP Requests: 0 0.0Non HTTP Responses: 1380 0.0Chunked HTTP Responses: 1631953 0.3Http Miss Due To DNS: 31050 0.0Http Deletes Due To DNS: 12914 0.0Objects cached for min ttl: 575986 0.1The following is sample output from the show statistics http performance EXEC command. The command output displays performance statistics regarding the HTTP requests that Content Engine has serviced.
ContentEngine# show statistics http performanceStatistics - PerformanceAvg Min Max Last-------------------------------------------------------------Requests / Second: - - 677 3Bytes / Second: - - 5995814 81801Seconds / Request: 0.067 0.000 15453.547 1.499Seconds / Hit: 0.308 0.000 979.442 0.158Seconds / Miss: 0.066 0.000 15453.547 1.572-------------------------------------------------------------Object Size: 150.2 Avg0 Min718732317 Max21386.0 LastThe following is sample output of the show statistics http savings EXEC command. The command output displays statistics regarding the savings because the Content Engine serviced HTTP requests from its local HTTP cache (cache hits) instead of retrieving the content from the origin web server.
ContentEngine# show statistics http savingsStatistics - SavingsRequests Bytes-----------------------------------------------------------Total: 525980242 79047534484Hits: 1966223 19865155481Miss: 524014019 59182379003Savings: 0.4 % 25.1 %The following is sample output of the show statistics rtsp proxy media-real requests EXEC command. The command output displays RealMedia caching statistics for the RTSP requests that Content Engine has serviced.
ContentEngine# show statistics rtsp proxy media-real requestsMedia Cache Statistics - RequestsTotal % of Requests---------------------------------------------------Total Received Requests: 0 -Demand Cache Hit: 0 0.0Demand Cache Miss: 0 0.0Demand Pass-Through: 0 0.0Live Split: 0 0.0Live Pass-Through: 0 0.0The following is sample output of the show statistics rtsp proxy media-real savings EXEC command. The command output displays the number of media cache hits and misses for RealMedia content, and the amount of savings because content was served from the local cache rather than being retrieved multiple times from the origin streaming server.
ContentEngine# show statistics rtsp proxy media-real savingsMedia Cache Statistics - SavingsRequests Bytes-----------------------------------------------------------Total: 525980242 79047534484Hits: 1966223 19865155481Miss: 524014019 59182379003Savings: 0.4 % 25.1 %Monitoring Critical Disk Drives on Standalone Content Engines
In order to operate properly, the Content Engine depends on the following critical disk drives:
•
•
The sysfs partition is used to store log files, including transaction logs, system logs (syslogs), and internal debugging logs. It can also be used to store image files and configuration files on a Content Engine.
Note
When a Content Engine is booted and a critical disk drive is not detected at system startup time, the ACNS system on the Content Engine runs at a degraded state. If one of the critical disk drives goes bad at run time, the ACNS system can exhibit symptoms such as the applications malfunctioning, hanging, or crashing, or the ACNS system hanging or crashing. Consequently, it is vital that the critical disk drives on a Content Engine are monitored and that disk drive errors are reported.
With an ACNS system, a disk device error is defined as any of the following events:
•
•
•
In ACNS 5.2 software, the ability to monitor Content Engine disk drives was added. Disk status is now recorded in flash (non-volatile storage). When an error on a Content Engine disk device occurs, a message is written to the system log (syslog) if the sysfs partition is still intact, and an SNMP trap is generated if SNMP is configured on the Content Engine.
In addition to tracking the state of critical disk drives, you can define a disk device error-handling threshold on the Content Engine. If the number of disk device errors reaches the specified threshold, the corresponding disk device is automatically marked as "bad." The ACNS system does not stop using the "bad" disk device immediately; it stops using the "bad" disk drive after the next reboot.
If the specified threshold is exceeded, the Content Engine either records this event or reboots. If the automatic reload feature is enabled and this threshold is exceeded, then the ACNS system automatically reboots the Content Engine. For more information about specifying this threshold, see the "Specifying the Disk Error-Handling Threshold" section.
Note
Specifying the Disk Error-Handling Threshold
In ACNS 5.2 software, a disk error-handling threshold option was added. This threshold determines how many disk errors can be detected before the disk drive is automatically marked as "bad." By default, this threshold is set to 10.
To change the default threshold, use the disk error-handling threshold global configuration command. Valid values are from 0 through 100. Specify 0 if you never want the disk drive to be marked as "bad."
In the following example, five disk drive errors for a particular disk drive (for example, disk00) will be allowed before the disk drive is automatically marked as "bad."
ContentEngine(config)# disk error-handling threshold 5If the bad disk drive is a critical disk drive, and the automatic reload feature (disk error-handling reload command) is enabled, then the ACNS software marks the disk drive as "bad" and the Content Engine is automatically reloaded. After the Content Engine is reloaded, a syslog message and an SNMP trap are generated.
By default, the automatic reload feature is disabled on a Content Engine. To enable the automatic reload feature, use the disk error-handling reload global configuration command.
ContentEngine(config)# disk error-handling reloadAfter enabling the automatic reload feature, use the no disk error-handling reload global configuration command to disable it.
ContentEngine(config)# no disk error-handling reloadManually Marking and Unmarking Content Engine Disk Drives
A disk drive on a Content Engine will remain in the "Not used" state until you manually unmark it as follows:
•
disk add diskname [sysfs {remaining | disk-space}] [cfs {remaining | disk-space}] |
[mediafs {remaining | disk-space}]•
The following scenario shows how to mark disk03 as "bad," reload the Content Engine, then unmark disk03 as a "bad" so that it can be used again.
1.
Content Engine# disk mark ?WORD Disk name (e.g. disk00, disk01,..)Content Engine# disk mark disk03 ?bad Mark as bad disk drive, don't use itgood Mark as good disk driveContent Engine# disk mark disk03 baddisk03 is marked as bad.It will be not used after reload.2.
Content Engine# show disks details(*) Disk drive won't be used after reload.......disk03: Normal (h00 c00 i03 l00 - Int DAS) 70001MB( 68.4GB) (*)FREE: 70001MB( 68.4GB)......3.
Content Engine# reloadProceed with reload?[confirm]......4.
Content Engine# show disks details(*) Disk drive won't be used after reload.......disk03: Not used (*)......5.
Content Engine# disk mark disk03 gooddisk03 is marked as good.It will be used after reload.6.
Content Engine# show disks details......disk03: Not used......Displaying Details About the Current Disk Configuration
Use the show disks details EXEC command to display detailed information about a Content Engine's current disk configuration.
ContentEngine# show disks detailsdisk00: Normal (IDE disk) 38160MB( 37.3GB)disk00/04: PHYS-FS 15137MB( 14.8GB) mounted internallydisk00/04: MEDIAFS 532MB( 0.5GB) mounted internallydisk00/05: SYSFS 1023MB( 1.0GB) mounted at /local1disk00/06: CFS 15359MB( 15.0GB)System use: 6130MB( 6.0GB)FREE: 16MB( 0.0GB)disk01: Not presentNo NAS share is attached to this device.Disk drives that are currently marked as "bad" are shown as "Not used" in the output of the show disks details EXEC command. Future "bad" disk drives (drives that will not be used after the next time the Content Engine is reloaded) are shown with an asterisk (*). In the following case, disk03 is a future "bad" disk drive that will not be used after the Content Engine is reloaded.
ContentEngine# show disks details(*) Disk drive won't be used after reload.......disk03: Normal (h00 c00 i03 l00 - Int DAS) 70001MB( 68.4GB) (*)FREE: 70001MB( 68.4GB)......Removing All Disk Partitions on a Single Disk Drive
Use the disk delete-partitions EXEC command to remove all disk partitions on a single disk drive.
Caution
Typically, this command is used when you want to add a new disk drive that was previously used with another operating systems (for example, a Microsoft Windows or Linux operating system). When asked if you want erase everything on the disk, specify "yes" to proceed, as shown below.
ContentEngine# disk delete-partitions disk03This will erase everything on disk. Are you sure? [no] yesUsing ACNS Software Transaction Logs
Administrators of standalone Content Engines (caching and streaming engines) are often interested in what types of requests have been made of the Content Engine and what the results of these requests were. For example, if streaming media is a source of revenue for a company, then the company needs a way to track which customer is accessing which content, how long a user viewed the content, and at what viewing quality. Because these companies charge their customers to stream on-demand content and live broadcasts, they must rely on logged information as the basis for billing their customers for their content access services.
Standalone Content Engines that are running ACNS 5.x software can record all errors and access activities. In ACNS 5.x software, each content service module (for example, the HTTP module, the WMT server, the FTP proxy process, and the TFTP server) on the Content Engine provides logs of the requests that were serviced. These logs are referred to as "transaction logs."
Typical fields in the transaction log are the date and time when a request was made, the URL that was requested, whether it was a cache hit or a cache miss, the type of request, the number of bytes transferred, and the source IP address. Transaction logs are generally used for the following purposes:
•
•
•
•
•
•
The reliable production, storage, and management of transaction logs is vital in the billing and cost analysis and provisioning cases.
The Windows Media Services 9 Series provides a more robust logging model than Windows Media Services Version 4.1. In ACNS 5.2 software, support for Windows Media Services 9 logging was added.
You can log to a predefined format (for example, Squid, Extended Squid, or Apache, or a custom transaction log formats that allows you to log additional fields). The contents of the transaction logs can be exported to an external server using FTP. You can also configure log rotation policies.
Note
In ACNS 5.2 software, there is a new transaction log feature that provides a real-time transaction log capability to another device. You can configure a Content Engine to send HTTP transaction log messages to a remote syslog server so that you can monitor HTTP transaction authentication failures in "real time." For more information on this topic, see the "Monitoring HTTP Request Authentication Failures in Real Time" section.
By default, transaction logging is disabled on the Content Engine that is running ACNS software. You can use either the Content Engine GUI or the CLI to enable transaction logging on the Content Engine. When transaction logging is enabled through the Content Engine GUI, the Squid log format is used as shown in the sample output below.
ContentEngine(config)# transaction-logs ?archive Configure archive parametersenable Enable transaction log featureexport Configure file export parametersfile-marker Add entries to translog indicating the file begin and endformat log file format (default squid)log-windows-domain Log Windows domain with authenticated username ifavailablesanitize Mask end user identities in log fileTable 20-7 lists the default settings for transaction logging on a standalone Content Engine.
Table 20-7 Default Settings for Transaction Logging for Standalone Content Engines
Archive
Disabled
See the "Archiving the Working Log" section.
Transaction logging
Disabled
Export compression
Disabled
Export of
transaction logDisabled
File marker
Disabled
Use to add entries to the transaction log file to indicate the beginning and the end of a file.
Sanitized transaction logging
Disabled
Archive interval
Every day, every one hour
See the "Archiving the Working Log" section.
Archive maximum
file size2,000,000 kilobytes
See the "Archiving the Working Log" section.
Export interval
Every day, every one hour
Transaction log format
Squid native log format
Note
For more information about working with ACNS transaction logs, see the following sections:
•
•
•
•
•
Enabling Transaction Logging
In ACNS 5.x software, you can choose the Squid, Extended Squid, or Apache transaction log formats, or you can use a custom log format that allows you to log additional fields. (See Table 20-8.)
Table 20-8 List of Supported Transaction Log Formats
Transaction Log FormatSquid
Extended Squid
See the "Enabling Extended Squid-Style Transaction Logging" section.
Apache
See the "Enabling Apache-Style Transaction Logging" section.
Custom
See the "Enabling Custom Format Transaction Logging" section.
Note
In ACNS 5.2 software, the ability to send HTTP transaction log messages to a remote syslog server was added. This allows you to monitor HTTP transaction authentication failures in real time. The existing transaction logging to the local file system remains unchanged. For more information about real-time transaction logging, see the "Monitoring HTTP Request Authentication Failures in Real Time" section.
Enabling Squid-Style Transaction Logging
Use the transaction-logs format squid global configuration command to enable transaction logging in Squid-style format.
ContentEngine(config)# transaction-logs format squidThe Squid-style log format is the default format for transaction logging in the Content Engine. The Squid log file format used is the native log file format associated with the Squid-1.1 access.log file format.
The Squid log file format is as follows:
time elapsed remote host code/status bytes method URL rfc931 peer status/peer host typeA Squid log format example looks like this:
1012429341.115 100 172.16.100.152 TCP_REFRESH_MISS/304 1100 GET http://www.cisco.com/images/homepage/news.gif - DIRECT/www.cisco.com -Squid transaction logs are a valuable source of information about cache workloads and performance. The logs record not only access information but also system configuration errors and resource consumption, such as memory and disk space.
Table 20-9 lists the fields associated with the Squid-style format.
Note
Enabling Extended Squid-Style Transaction Logging
Use the transaction-logs format extended-squid global configuration command to enable transaction logging in Extended Squid format.
ContentEngine(config)# transaction-logs format extended-squidThe Extended Squid format logs the associated username for each record in the log file in addition to the fields logged by the Squid-style format, and is used for billing purposes. In this format the Rfc931 field associated with the Squid format (Table 20-9) is used to log the authorized user. This field always contains a "-" (dash) if no user information is available.
An Extended Squid-style log file format example looks like this:
1012429341.115 100 172.16.100.152 TCP_MISS/302 184 GET http://www.cisco.com/ cgi-bin/login myloginname DIRECT/www.cisco.com -Enabling Apache-Style Transaction Logging
Use the transaction-logs format apache global configuration command to enable transaction logging in Apache style format.
ContentEngine(config)# transaction-logs format apacheThe Apache-style log file format is as follows:
remotehost rfc931 authuser date request status bytesAn Apache-style log file forma looks like this:
172.16.100.152 - - [Wed Jan 30 15:26:26 2002] "GET/http://www.cisco.com/images/homepage/support.gif HTTP/1.0" 200 632This format is the Common Log File (CLF) format defined by the World Wide Web Consortium (W3C) working group. This format is compatible with many industry-standard log tools. For more information, refer to the W3C Common Log Format website at http://www.w3.org/Daemon/User/Config/Logging.html.
Table 20-10 lists the fields associated with the Apache Common Log file format.
Enabling Custom Format Transaction Logging
Use the transaction-logs format custom global configuration command to use a log format string to log additional fields that are not included in the predefined native Squid or Extended Squid formats, or the Apache Common Log file (CLF) format. The log format string is a string which can contain the tokens listed in Table 20-11 and which mimics the Apache log format string. The log format string can contain literal characters that are copied into the log file.
•
•
•
Table 20-11 lists the acceptable format tokens for the log format string.
•
•
•
•
–
–
For example, "%400,501{User-Agent}i" logs the User-Agent header value on 400 errors and 501 errors (Bad Request, Not Implemented) only, whereas "%!200,304,302{Referer}i" logs the Referer header value on all requests that did not return a normal status.
The custom format currently supports the following request headers:
•
•
•
•
The output of each of the following Request, Referer, and User-Agent format tokens specified in the custom log format string is always enclosed in double quotation marks in the transaction log entry:
%r
%{Referer}i
%{User-Agent}i
The %{Cookie}i format token is generated without the surrounding double quotation marks, because the Cookie value itself can contain double quotes. The Cookie value can contain multiple attribute-value pairs that are separated by spaces. For this reason, it is recommended that when the Cookie format token is used in a custom format string, it be positioned as the last field in the format string. This positioning of the Cookie format token allows it to be more easily parsed by transaction log reporting tools. Alternatively, if you use the format token string "\'%{Cookie}i\'", the Cookie header can be surrounded by single quotes.
The following command can be entered to generate the well-known Apache Common Log file format:
transaction-logs format custom "[%{%d}t/%{%b}t/%{%Y}t:%
{%H}t:%{%M}t:%{%S}t %{%z}t] %r %s %b %{Referer}i %{User-Agent}i"The following transaction log entry example in the Apache Common Log file format is configured using the preceding custom format string:
[11/Jan/2003:02:12:44 -0800] "GET http://www.cisco.com/swa/i/ site_tour_link.gif HTTP/1.1" 200 3436 "http://www.cisco.com/" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"
Table 20-11 Custom Log File Format String Values
%...a
IP address of the requesting client.
%...A
IP address of the server from which the object was served (for example, the origin server or the outgoing proxy in the case of a cache miss or 0.0.0.0 in the case of a cache hit).
%...B
%...bBytes sent, excluding HTTP headers.
%...c
Connection status when response is completed, where:
X = Connection was aborted before the response was completed.
+ = Connection can be kept alive after the response is sent.
- = Connection is closed after the response is sent.%...f
Filename.
%...h
Remote host (IP address of the requesting client is logged).
%...H
Request protocol.
%...{Foobar}i
Contents of Foobar: header lines in the request that is sent to the server. The value of Foobar can be one of the following headers: User-Agent, Referer, Host, or Cookie.
%...l
Remote log name. Not implemented on the Content Engine, so a dash (-) is logged.
%...m
Request method.
%...p
Canonical port of the server servicing the request. Not applicable on the Content Engine, so a dash (-) is logged.
%...P
Process ID of the child that serviced the request.
%...q
Query string (that is preceded by a ? if a query string exists; otherwise, it is an empty string).
%...r
First line of the request.
%...s
Status. The translog code always returns the HTTP response code for the request.
%...t
Time in common log time format (or standard English format).
%...{format}t
Time in the form given by the format token specified in Table 20-12.
%...T
Time consumed to serve the request in seconds (a floating point number with 3 decimal places).
%...u
Remote user.
%...U
URL path requested, not including query strings.
%...v
%...VValue of the host request header field reported if the host appeared in the request. If the host did not appear in the host request header, the IP address of the server specified in the URL is reported.
Table 20-12 specifies the format token for the date and time of the format token %...{format}t that is listed in Table 20-11.
Logging Windows Domain with Authenticated Usernames
If your Content Engine is configured for NTLM authentication and uses the Extended Squid-style or custom format, the transaction-logs log-windows-domain global configuration command records the Windows domain name and username in the username field of the transaction log. If the domain name is available, both the domain name and the username are recorded in the username field, in the form domain\username. If only the username is available, only the username is recorded in the username field. If neither a domain name nor a username is available, a "-" (dash) is recorded in the field.
The Windows domain name that is used for NTLM authentication appears in the username field of the transaction log. The username appears in the format "domain\username" in the Extended Squid-style and custom transaction log formats that contain the username using the %u format token. (The %u format token specifies the day of the week as a decimal; the range is 1 to 7 with Monday being 1.)
Use the no transaction-logs log-windows-domain global configuration command to negate logging NTLM parameters to the transaction log in Extended Squid-style or Custom formats.
Sanitizing Transaction Logs
You can disguise the IP address of clients in the transaction log file. The default is that transaction logs are not sanitized. A sanitized transaction log disguises the network identity of a client by changing the IP address in the transaction logs to 0.0.0.0. You can enable the sanitize feature in transaction logging, through the following interfaces:
•
•
ContentEngine(config)# transaction-logs sanitizeThe no form of this command disables the sanitize feature. Note that the transaction-logs sanitize command does not effect the client IP (%a) value associated with a custom log format string that is configured with the CLI (that is, configured with the transaction-logs format custom string global configuration command in which string is the quoted log format string that contains the custom log format). To hide the identity of the client IP in the custom log format, either hard code "0.0.0.0" in the custom log format string or exclude the %a token, which represents the client IP, from the format string.
Exporting Transaction Log Files
In order to facilitate the postprocessing of cache log files, it is possible to export transaction logs to an external host.
This feature allows log files to be automatically exported by FTP to an external host at configurable intervals. The username and password used for FTP are configurable, as is the directory to which the log files are uploaded.
The log files automatically have a filename of:
type_ipaddr_yyyymmdd_hhmmss.txt
where
•
•
•
Note
Exporting Transaction Logs to External FTP or STFP Servers
You can use the Content Engine GUI or CLI to export transaction logs to external FTP or secure FTP (SFTP) servers:
From the Content Engine GUI, choose Caching > Transaction Logs. Use the displayed Transaction Logs window to export transaction logs to an FTP or SFTP server. For more information about how to use the Transaction Logs window, click the HELP button in the window.
To use the Content Engine CLI to enable exporting of transaction logs to an external FTP or SFTP server, follow these steps:
Step 1
Step 2
transaction-logs export ftp-server {hostname | server-ip-address} login password directory
where:
•
The Content Engine translates the host name with a DNS lookup and then stores the IP address in the configuration.
•
•
•
Note
In this example, the Content Engine is configured to export its transaction logs to two FTP servers.
ContentEngine(config)# transaction-logs export ftp-server 10.1.1.1 mylogin mypasswd /ftpdirectoryContentEngine(config)# transaction-logs export ftp-server myhostname mylogin mypasswd /ftpdirectoryStep 3
transaction-logs export sftp-server {hostname | server-ip-address} login password directory
where:
•
The Content Engine translates the host name with a DNS lookup and then stores the IP address in the configuration.
•
•
•
Step 4
The compressed filename has a .gz extension in the filename. This feature uses less disk space for the archived files on both the Content Engine and the FTP and SFTP export servers also require less bandwidth during export.
Changing the Transaction Logging Export Settings on Standalone Content Engines
After you have specified the transaction logging export configuration (as described in the "Enabling Transaction Logging"), you can change a username, password, or directory, as follows:
Use the transaction-logs export ftp-server global configuration command, with the new parameters to change the current configuration for an FTP server.
Use the transaction-logs exportsftp-server global configuration command, with the new parameters to change the current configuration for an SFTP server.
As the following example shows, you can change the username, password, or directory by reentering the entire line using the new parameters (for example, mynewname, mynewpass, or newftpdirectory).
ContentEngine(config)# transaction-logs export ftp-server 10.1.1.1 mynewname mynewpass /newftpdirectoryTo delete an FTP server from the current configuration, use the no form of the transaction-logs export ftp-server global configuration command.
ContentEngine(config)# no transaction-logs export ftp-server 10.1.1.1To delete an SFTP server from the current configuration, use the no form of the transaction-logs export sftp-server global configuration command.
ContentEngine(config)# no transaction-logs export sftp-server sftphostnameRestarting Export After Receiving a Permanent Error from the External FTP Server
When an FTP server returns a permanent error to a standalone Content Engine, the archive transaction logs are no longer exported to that server. You must reenter the Content Engine transaction log export parameters for the misconfigured server to clear the error condition. The show statistics transaction-logs EXEC command displays the current state of readiness for transaction log export.
A permanent error (Permanent Negative Completion Reply, RFC 959) occurs when the FTP command to the server cannot be accepted, and the action does not take place. Permanent errors can be caused by invalid user logins, invalid user passwords, and attempts to access directories with insufficient permissions or directories that do not exist.
In the following example, an invalid user login parameter was included in the transaction-logs export ftp-server global configuration command. The show statistics transaction-logs EXEC command shows that the Content Engine failed to export archive files.
ContentEngine# show statistics transaction-logsTransaction Log Export Statistics:Server:172.16.10.5Initial Attempts:1Initial Successes:0Initial Open Failures:0Initial Put Failures:0Retry Attempts:0Retry Successes:0Retry Open Failures:0Retry Put Failures:0Authentication Failures:1Invalid Server Directory Failures:0To restart the export of archive transaction logs, you must reenter the transaction-logs export ftp-server parameters.
ContentEngine(config)# transaction-logs export ftp-server 172.16.10.5 goodlogin pass /ftpdirectoryArchiving the Working Log
Depending upon where the sysfs is mounted, the following log files are logged to a working log on the local disk of a standalone Content Engine as follows:
•
–
–
•
–
–
You can specify the interval at which the working log should be cleared by moving the data to an archive log. The archive log files are located on the local disk in the /local1/logs/ or /local2/logs/ directory depending upon where the sysfs is mounted.
Because multiple archive files are saved, the filename includes the time stamp when the file was archived. Because the files can be exported to an FTP or SFTP server, the filename also contains the IP address of this Content Engine.
The archive filename format is:
celog_IPADDRESS_YYYYMMDD_HHMMSS.txt
Use the transaction-logs archive global configuration commands to archive the working logs.
transaction-logs archive interval seconds
transaction-logs archive interval every-day {at hour:minute | every hours}
transaction-logs archive interval every-hour {at minute | every minutes}
transaction-logs archive interval every-week [on weekdays at hour:minute]
transaction-logs archive max-file-size filesize
See Table 20-13 for a description of these command parameters.
Disabling Transaction Logging Export on Standalone Content Engines
To disable the transaction logging export feature on a standalone Content Engine while retaining the transaction logging export configuration (for example, such configuration information about the FTP or SFTP servers as their IP addresses), use the no form of the transaction-logs export enable global configuration command.
ContentEngine(config)# no transaction-logs export enableMonitoring HTTP Request Authentication Failures in Real Time
In ACNS 5.2 software, the ability to send HTTP transaction log messages to a remote syslog server was added. This allows you to monitor the remote syslog server for HTTP request authentication failures in real time. This real-time transaction log feature allows you to monitor transaction logs in real time for particular errors such as HTTP request authentication errors. The existing transaction logging to the local file system remains unchanged.
Note
To support the real-time transaction logging feature, the following CLI commands were added in the ACNS 5.2 software release:
[no] transaction-logs logging enable
[no] transaction-logs logging host {hostname | ipaddr} [port port-num rate-limit msgs-per-sec]
[no] transaction-logs logging facility fac-name[no] transaction-logs logging entry-type entry-type [request-auth-failures | all]
These CLI commands allow you to specify a remote syslog host that will receive transaction log messages in real time. The configurable rate-limiting option (the rate-limit option) was added to limit the rate at which the transaction logger is allowed to send messages to the remote syslog server (messages per second). You can configure the Content Engine to send transaction log messages in real time to one remote syslog host.
The following are the default settings for the real-time transaction logging feature:
•
•
•
•
•
•
–
–
The message format of the transaction log entry to the remote syslog host is the same as in the transaction log file and pre-pended with Cisco's standard syslog header information
Apr 22 20:10:46 ce-host cache:%CE-TRNSLG-6-460012: <translog formatted msg>where:
•
•
•
•
To include the username and domain name in the transaction log, use the following Content Engine CLI command:
ContentEngine(config)# transaction-logs log-windows-domainThis will cause the Windows domain name that is used for NTLM authentication to appear in the username field of the transaction log. The username appears in the format "domain\username" in the Extended Squid-style and custom transaction log formats that contain the username using the %u format token. Proxy request authentication failures in the HTTP transaction logs are reported as "401/407" errors and include the username. This type of error indicates an HTTP authentication failure. These errors are also included in the system syslog.
Configuring the Remote Syslog Host for Real-Time Transaction Logging
To configure a standalone Content Engine to send transaction log messages in real time to a remote syslog host, use the transaction-logs logging host global configuration command.
In ACNS 5.2 software, Content Engine system syslog messages were added to report communication errors with the remote syslog host that is configured for transaction logging. These new syslog messages are in the error message range: %CE-TRNSLG-6-460013 to %CE-TRNSLG-3-460016. Note that the last error message (%CE-TRNSLG-3-460016), shows level "3" (for error-level messages) instead of "6" (for information-level messages). Information-level messages are reported when messages are dropped due to rate limiting and the number of dropped messages are reported.
Configuring a Transaction Log Facility when Logging to a Remote Syslog Host
When logging transactions to a remote syslog host, use the transaction-logs logging facility global configuration command to configure a transaction log facility.
ContentEngine(config)# transaction-logs logging facility ?auth Authorization systemdaemon System daemonskern Kernellocal0 Local uselocal1 Local uselocal2 Local uselocal3 Local uselocal4 Local uselocal5 Local uselocal6 Local uselocal7 Local usemail Mail systemnews USENET newssyslog Syslog itselfuser User processuucp UUCP systemTo create a separate log on the remote syslog host for real-time transaction log entries, configure a unique facility (for example, "local1") on the Content Engine.
ContentEngine(config)# transaction-logs logging facility local1On the remote syslog host, configure messages from this unique facility to be logged in a separate file. The following is an example if the remote syslog host is a UNIX server:
1.
local1.=info /var/log/translog-messages2.
*.info;mail.none;news.none;authpriv.none;cron.none;local1.none /var/log/messagesOn a UNIX system, help on the syntax for the /etc/syslog.conf file is under the command "man syslog.conf."
On a UNIX system, the port for the syslog daemon is defined in /etc/services:
syslog 514/udpIf a port other than port 514 is configured on the syslog host, you must configure the same port on the Content Engine (the transaction-logs logging host {hostname | ipaddr} [port port-num] global configuration command).
Specifying the Transaction Log Entry Type when Logging to a Remote Syslog Host
With ACNS software, Release 5.2 or later, you can configure the Content Engine to send only the transactions associated with HTTP request authentication failures, or to send all of the transactions.
Typically, organizations are interested in only HTTP request authentication failures for security purposes. By monitoring these types of authentication failures in real time, it enables organizations to identify which end users failed to be authenticated through the Content Engine.
ContentEngine(config)# transaction-logs logging entry-type ?all Log all transaction messages to remote syslog hostrequest-auth-failures Log transactions CE failed to authenticate with the auth serverNote that only the authentication failure transaction that is associated with an end user who is attempting to contact the authentication server is logged. The "pending" transactions that are waiting for a response from the transaction that contacted the authentication servers are not logged. This approach provides you with the information needed to determine which user fails to authenticate with the Content Engine and minimizes the traffic to the syslog host. In order for you to track which users failed to authenticate, you must configure a transaction log format that logs the username by configuring either Extended Squid-style format or the custom log format with the custom format token %u. For more information about specifying the format of the transaction log, see the "Enabling Transaction Logging" section.
When the transaction-logs logging enable global configuration command is specified, the logging of only HTTP request authentication failures is the default. If you want to change this default and log all transactions, then you must enter the transaction-log logging entry-type all global configuration command on the Content Engine. However, if you log all transactions, there may be a significant UDP drop rate if your syslog host cannot handle the rate of incoming traffic.
Displaying the Transaction Log Configuration for Standalone Content Engines
To display information about the current configuration of transaction logging on a standalone Content Engine, use the show transaction-log or show transaction-logging EXEC commands. Both of these EXEC commands display the same output. Note that transaction log file information is displayed for HTTP and WMT MMS caching proxy transactions, as well as TFTP and ICAP transactions. See sample output below.
Note
ContentEngine# show transaction-logTransaction log configuration:---------------------------------------Logging is enabled.End user identity is visible.File markers are disabled.Archive interval: every-day every 1 hourMaximum size of archive file: 2000000 KBLog File format is squid.Windows domain is not logged with the authenticated usernameExporting files to ftp servers is disabled.File compression is disabled.Export interval: every-day every 1 hourHTTP Caching Proxy logging to remote syslog host is disabled.Remote syslog host is not configured.Facility is the default "*" which is "user".Log HTTP request authentication failures with auth server to remote syslog host.HTTP Caching Proxy Transaction Log File InfoWorking Log file - None existingWMT MMS Caching Proxy/Server Transaction Log File InfoWorking Log file - size : 584age: 404Archive Log file - mms_export_10.1.1.21_20040622_230415 size: 584Archive Log file - mms_export_1.1.1.1_20040623_205825 size: 584Translog directory doesn't exist. Maybe because /local1 has no sysfs mounted.Translog directory doesn't exist. Maybe because /local1 has no sysfs mounted.TFTP Transaction Log File InfoWorking Log file - size : 88age: 403Archive Log file - tftp_server_10.1.1.21_20040622_230415 size: 88Archive Log file - tftp_server_1.1.1.1_20040623_205826 size: 88ICAP Transaction Log File InfoWorking Log file - size : 61age: 403Archive Log file - icap_10.1.1.21_20040622_230415 size: 61Archive Log file - icap_1.1.1.1_20040623_205826 size: 61Using the System Logging Feature
Use the system logging feature to set specific parameters for the system log file (syslog). This file contains authentication entries, privilege levels, and administrative details. System logging is always enabled internally. The system log file is located on the system file system (sysfs) partition as /local1/syslog.txt.
By default, system logging is enabled on a standalone Content Engine. Table 20-14 lists the default settings for system logging.
You can use the Content Engine GUI or CLI to configure standalone Content Engines to send varying levels of event messages to disk, console, or remote syslog hosts. See the "Configuring System Logging on Standalone Content Engines" section for information about how to change the default syslog settings.
Displaying the Current Configuration for System Logging
To display the current syslog host configuration on a standalone Content Engine, enter the show logging EXEC command.
ContentEngine# show loggingSyslog to host is enabled.Priority for host logging to 1.2.1.1:514 is set to: warningSyslog to console is disabledPriority for console logging is set to: warningSyslog to disk is enabledPriority for disk logging is set to: noticeFilename for disk logging is set to: /local1/syslog.txtSyslog facility is set to syslogSyslog disk file recycle size is set to 10000000Configuring System Logging on Standalone Content Engines
You can use the Content Engine GUI or the CLI to configure system logging on standalone Content Engines. As part of the configuration, you specify whether the Content Engine is to send varying levels of messages to any of the following: to disk, to console, or to up to four remote syslog hosts.
From the Content Engine GUI, choose System > Syslog. Use the displayed Syslog window to configure system logging for the Content Engine. For more information about how to use the Syslog window, click the Help button in the window to access the context-sensitive help.
From the Content Engine CLI, use the logging global configuration command to set specific parameters for the syslog.
ContentEngine(config)# logging ?console Log to consoledisk Store log in a filefacility Facility parameter when log to hosthost Log to host (maximum of 4 hosts)See the following sections for more information about how to use the Content Engine CLI to configure system logging on standalone Content Engines:
•
•
•
Configuring System Logging to the Console
System logging can be configured to send various levels of messages (priority levels) to the console. Use the logging console priority global configuration command to configure system logging to the console and to specify the various levels of messages that should be sent to the console.
logging console {enable | priority loglevel }
See Table 20-15 for a description of these command parameters.
Note
This example shows the last few lines of the syslog.txt file using the type-tail EXEC command, which only lists the last few lines of text in a file.
ContentEngine# type-tail syslog.txtJan 18 17:50:03 ContentEngine Host[3766]: authentication failure; (uid=0) -> aaHH for content_engine_config serviceJan 18 17:50:05 ContentEngine login[3766]: Failed login session from 172.16.1.1 for user aaHH: Authentication service cannot retrieve authentication info.Jan 18 18:39:05 ContentEngine Host[6787]: set privilege level to `0'Jan 18 18:39:05 ContentEngine login: user login on 1 from 172.16.66.148Configuring System Logging to Disk
System logging can be configured to send various levels of messages (priority levels) to disk. Use the logging disk priority global configuration command to configure system logging to disk and to specify the various levels of messages that should be sent to disk.
logging disk {enable | filename filename | priority loglevel | recycle size}
See Table 20-16 for a description of these command parameters.
Configuring System Logging to Remote Syslog Hosts
In ACNS 5.1 software, logging to only a single remote syslog host was supported, and the following two commands were used to configure a single remote syslog host for a standalone Content Engine:
ContentEngine(config)# logging host hostnameContentEngine(config)# logging priority priorityIn ACNS 5.2 software, you can configure a Content Engine to send varying levels of messages to up to four remote syslog hosts. To accommodate this change, the ACNS 5.1.x logging host priority priority global configuration command (shown above) is deprecated, and the logging host hostname global configuration command is extended as follows:
ContentEngine(config)# [no] logging host hostname [priority priority-code | port port |rate-limit limit]where:
•
•
ContentEngine(config)# logging host 1.2.3.4 priority ?alert (1) Immediate action neededcritical (2) Critical conditionsdebug (7) Debugging messagesemergency (0) System is unusableerror (3) Error conditionsinformation (6) Informational messagesnotice (5) Normal but significant conditionswarning (4) Warning conditions
Note
•
•
Use the logging host global configuration command to configure the Content Engine to send varying levels of syslog messages to up to four external syslog hosts. In the following example, the Content Engine is configured to send messages that have a priority code of "error" (level 3) to the remote syslog host that has an IP address of 172.31.2.160.
ContentEngine(config)# logging host 172.31.2.160 priority errorMapping Syslog Priority Levels to RealProxy Error Codes
RealProxy generates error messages and writes them to the RealProxy log file. These error messages are captured by the ACNS software and passed to the system log file. The correspondence between the RealProxy error codes and the syslog priority levels are shown in Table 20-17.
Using CiscoWorks2000
CiscoWorks2000 (CW2K) is a Cisco product that provides a suite of management applications used to manage most Cisco devices. The Content Engine can interoperate with CiscoWorks2000 without any modification in the following ways:
•
•
•
•
You can enable or disable syslog message generation in CiscoWorks2000-compliant format through either the Content Engine GUI or the CLI. For example, to use the Content Engine CLI to configure a CW2K as a remote syslog host, use the logging host hostname global configuration command, as described in the "Configuring System Logging on Standalone Content Engines" section.
Creating HTTP Custom Error Pages for Standalone Content Engines
ACNS software, Release 5.1 or later allows you to create HTTP customized error pages. If you create these customized pages, then the standalone Content Engine displays the appropriate customized error page instead of the default error message when proxy errors occur.
Table 20-18 describes the custom error messages and their usage for standalone Content Engines.
You can use the Content Engine GUI or the CLI to create HTTP customized error pages for any of the error messages listed in Table 20-18.
•
•
http custom-error-page download message url | reset {all | message} | upload {ip-address | hostname} dirname filename message
Table 20-19 describes the parameters for the http custom-error-page command.
Table 20-19 Parameters for the http custom-error-page CLI Command
download
Copies the custom error message file to the Content Engine from the specified URL. To change the text for a specific message, use this option to identify the message you want to change, and specify the URL that is the source for the custom message file. The custom message file can be up to 16 KB in size and is used instead of the standard message page for the specified message.
url
Specifies the source of the custom error file. The file size cannot exceed 16 KB.
message
Specifies the type of custom error message (for example, ftp-put-error). See Table 20-18 for a list of these custom error messages
reset
Reverts to the default error page.
message
Specifies the message that you want to return to the default page
upload
Uploads the custom error message file to the specified host, directory, and file.
ip-address
Specifies the IP address of the host to which to copy the error page.
hostname
Specifies the host name to which to copy the error page. The host should be reachable and allow copying a file to the specified directory.
dirname
Specifies the directory name to which to copy the error page.
filename
Specifies the filename to which to copy the error page.
The following command copies a custom error message page to the Content Engine for the cache-read-error message:
ContentEngine# http custom-error-page download http://www.myserver.com/errors/cache-read-error.txt cache-read-errorThe following command copies the current contents of the cache-read-error message to a file in the errors directory on the host with the IP address 192.168.1.1:
ContentEngine# http custom-error-page upload 192.168.1.1 /errors cache-read-error.txt cache-read-errorThe following command resets the cache-read-error message to the default text:
ContentEngine# http custom-error-page reset cache-read-errorTroubleshooting with the HTTP CLI Client
In ACNS 5.2 software, an HTTP CLI client was added. This new capability allows you to test connectivity and debug caching issues. The test-url EXEC command provides debugging capabilities to the end user from the Content Engine CLI. With this command, you can test the URL accessibility and connectivity through the web server and the proxy server. The functionality of the URL over the HTTP, HTTPS, and FTP protocols can be tested (see below). This command tests the functionality of a URL over the HTTP and FTP proxy servers. The command output is displayed to the standard output (the console).
ContentEngine# test-url ?ftp For request over FTPhttp For request over HTTPhttps For request over HTTPSSee Table 20-20 for a description of the test-url commands.
When using the test-url EXEC command, you can specify the test URL (test-url protocol test-URL EXEC command) in either of the following formats (see below):
http://domain.com/path or http://user:password@domain.com/path
In the following example, the test URL (www.cisco.com) is an HTTP request. The request succeeds.
ContentEngine# test-url http http://www.cisco.com--17:34:50-- http://www.cisco.com/=> `/dev/null'Len - 21 , Restval - 0 , contlen - 0 , Res - 134728056Resolving www.cisco.comdone.Connecting to www.cisco.com[192.168.0.0]:80... connected.HTTP request sent, awaiting response...1 HTTP/1.1 200 OK2 Date: Fri, 25 Jun 2004 17:09:06 GMT3 Server: Apache/1.0 (Unix)4 Content-Type: text/html5 Set-Cookie: CP_GUTC=172.16.0.0.133781088183346835; path=/; expires=T19-Jun-29 17:09:06 GMT; domain=.cisco.com6 Via: 1.1 Application and Content Networking System Software 5.2.07 Connection: Close(-1 to go)[<=> ] 0 --.--K/sen - 1185 ELen - 0 Keepalive - 0[ <=> ] 56,249 1.38M/s17:34:50 (1.38 MB/s) - `/dev/null' saved [56249]In the following example, the test URL (gmail.google.com) is an HTTPS request that fails because the host is not found.
ContentEngine# test-url https https://gmail.google.com head-only--17:43:13-- https://gmail.google.com/=> `/dev/null'Len - 25 , Restval - 0 , contlen - 0 , Res - 134728056Resolving gmail.google.com...Resolving gmail.google.com failed: Host not found.In the following example, the test URL is an FTP-over-HTTP request using the Content Engine as an that has an IP address of 172.16.0.0 as the FTP proxy for the request.
ContentEngine# test-url ftp ftp://1234567:pAB6rB7x@smartfilter.com use-ftp-proxy ? WORD Proxy-url. Format http://proxyIp:proxyPort orhttp://proxyUser:proxypasswd@proxyIp:proxyPortContentEngine# test-url ftp ftp://1234567:pAB6rB7x@smartfilter.com use-ftp-proxy 172.16.0.0:8080Troubleshooting with the Log Viewing Tool
In ACNS 5.2 software, the less EXEC command was added. Use this command to specify the filename of the ACNS software log that you want to view.
ContentEngine# less ?WORD file nameTroubleshooting with the Telnet Client
In ACNS 5.2 software, a Telnet CLI client was added. This Telnet client allows you to specify a destination port. This allows you to test websites by attempting to telnet to the website from the Content Engine CLI. To support this feature, the telnet EXEC command was added.
ContentEngine# telnet ?Hostname or A.B.C.D Remote host or IP addressTroubleshooting with Ping
To send echo packets for diagnosing basic network connectivity on networks, use the ping EXEC command.
ping {hostname | ip-address} | PING options
In ACNS 5.2 software, the functionality of the ping EXEC command was expanded to support all of the standard ping options as command arguments. (For more information on the standard ping options, refer to the Linux man page.)
ContentEngine# ping ?LINE Destination Host or IP Address (or PING options)To use the ping EXEC command with the hostname argument, be sure that DNS functionality is configured on your Content Engine. To force the timeout of a nonresponsive host, or to eliminate a loop cycle, press Ctrl-C.
In the following example, the host with an IP address of 172.19.131.189 is pinged from the Content Engine CLI.
ContentEngine# ping 172.19.131.189PING 172.19.131.189 (172.19.131.189) from 10.1.1.21 : 56(84) bytes of data.64 bytes from 172.19.131.189: icmp_seq=0 ttl=249 time=613 usec64 bytes from 172.19.131.189: icmp_seq=1 ttl=249 time=485 usec64 bytes from 172.19.131.189: icmp_seq=2 ttl=249 time=494 usec64 bytes from 172.19.131.189: icmp_seq=3 ttl=249 time=510 usec64 bytes from 172.19.131.189: icmp_seq=4 ttl=249 time=493 usec--- 172.19.131.189 ping statistics ---5 packets transmitted, 5 packets received, 0% packet lossround-trip min/avg/max/mdev = 0.485/0.519/0.613/0.047 msIn ACNS 5.2 software, the functionality of the ping command was expanded to support all of the standard ping options. (For more information on the standard ping options, refer to the Linux man page.)
ContentEngine# ping ?LINE Destination Host or IP Address (or PING options)Troubleshooting with Traceroute
Traceroute is a widely available utility on most operating systems. Much like ping, it is a valuable tool for determining connectivity in a network. Ping allows the user to find out if there is a connection between two end systems. Traceroute does this as well, but additionally lists the intermediate routers between the two systems. Users can therefore see the routes that packets can take from one system to another.
Use the traceroute EXEC command to find the route to a remote host, when either the host name or IP address is known.
ContentEngine# traceroute yahoo.comtraceroute to 66.218.71.113 (66.218.71.113), 30 hops max, 38 byte packets************10 p3-3.paloalto-cr2.bbnplanet.net (4.0.26.13) 3.219 ms 2.001 ms 2.097 ms11 p7-1.paloalto-nbr2.bbnplanet.net (4.0.6.77) 3.133 ms 1.949 ms 2.076 ms12 p4-0.paloalto-nbr1.bbnplanet.net (4.0.5.65) 2.755 ms 2.204 ms 2.037 ms13 p1-0.paix-bi2.bbnplanet.net (4.0.6.98) 2.928 ms 2.146 ms 2.334 ms14 p1-0.xpaix17-level3.bbnplanet.net (4.0.1.70) 3.397 ms 3.631 ms 3.081 ms15 gige10-0.ipcolo4.SanJose1.Level3.net (64.159.2.42) 3.334 ms 2.999 ms 2.388 ms16 cust-int.level3.net (64.152.69.18) 3.871 ms 3.031 ms *17 ge-3-3-0.msr1.pao.yahoo.com (216.115.101.42) 3.695 ms ge-2-3-0.msr2.pao.yahoo.com (216.115.101.46) 6.998 ms *18 vl16.bas1.scd.yahoo.com (66.218.64.146) 6.282 ms 5.091 ms 5.162 ms19 w2.rc.scd.yahoo.com (66.218.71.113) 6.028 ms 5.782 ms 5.544 ms
