Cisco ACNS Software Deployment and Configuration Guide, Release 5.0
Chapter 11: Monitoring and Troubleshooting the Content Delivery Network
Download this chapter
Chapter 11: Monitoring and Troubleshooting the Content Delivery NetworkChapter 11: Monitoring and Troubleshooting the Content Delivery Network
Download the complete book
Cisco ACNS Software Deployment and Configuration Guide, Release 5.0Cisco ACNS Software Deployment and Configuration Guide, Release 5.0 (PDF - 7 MB)
give_us_feedback

Table Of Contents

Monitoring and Troubleshooting the Content Delivery Network

Monitoring CDN Devices

Viewing Content Engine Statistics

Viewing Device Group Statistics

Viewing Routing Statistics

Viewing the System Message Log

Troubleshooting Using the Content Distribution Manager GUI show Command Utility

Troubleshooting Content Router Configurations

Recovery Procedures

Recovering the System Software

Recovering a Lost Administrator Password

Recovering from Missing Disk-Based Software

Replacing a Failed Disk Drive

Performing Backup and Restore for the Content Distribution Manager Configuration Management Subsystem Database


Monitoring and Troubleshooting the Content Delivery Network

This chapter provides information on monitoring and troubleshooting the CDN by viewing device statistics, viewing system message logs, and accessing troubleshooting tools.

This chapter contains the following sections:

•Monitoring CDN Devices

•Viewing the System Message Log

•Troubleshooting Using the Content Distribution Manager GUI show Command Utility

•Troubleshooting Content Router Configurations

•Recovery Procedures

•Performing Backup and Restore for the Content Distribution Manager Configuration Management Subsystem Database

Monitoring CDN Devices

The Monitoring area of the Content Distribution Manager GUI allows you to view statistics for your Content Engines, device groups, and Content Routers.

Viewing Content Engine Statistics

It is often useful to be able to assess the performance of Content Engines across your network. You can do this using the Content Engine statistics feature, available from the Monitoring area of the Content Distribution Manager GUI.

The Content Engines statistics feature enables you to view, at a glance, which Content Engines are online, as well as assess their available resources, the volume of traffic being routed to them, and their performance in serving requests. The information displayed is based on a snapshot of your CDN taken on the quarter hour. The statistics displayed represent the state of your Content Engines for the previous quarter hour.

To view Content Engine statistics for your CDN, follow these steps:

Step 1 From the Content Distribution Manager GUI, choose Monitoring > Statistics.

Step 2 In the Contents pane, click Content Engines, and then choose one of the following submenu options to view the Content Engine statistics for that option:

•Cisco Streaming Engine

•HTTP

•Real Proxy

•Routing

•WMT

Step 3 To print the statistics data, click the Printer icon.

Table 11-1 explains the meaning of each Content Engine statistic presented in the GUI.

Table 11-1 Content Engine and Device Group Statistics 

Content Engine Property
Description
Cisco Streaming Engine

Bandwidth (bits/sec)

Current bandwidth output by the server in bits per second.

Total Bytes

Total bytes output by the server since it started.

Total Packets

Total packets output by the server since it started.

RTSP Connections

Number of clients currently connected over RTSP.

All Connections

Number of clients currently connected since startup.

Updated

Time stamp indicating when the statistics were updated.

HTTP

Requests/Sec

Number of requests per second.

Bytes/Sec

Number of bytes per second.

Service Time

Length of time that the Content Engine has been up.

Hit Rate

Average number of content items per minute successfully served from the cache of the Content Engine or from all the Content Engines in the channel or virtual CDN during the preceding quarter hour.

Updated

Time stamp indicating when the statistics were updated.

RealProxy

Requests/Sec

Number of requests per second.

Bytes/Sec

Number of bytes per second.

Hit Rate

Average number of content items per minute successfully served from the cache of the Content Engine or from all the Content Engines in the channel or virtual CDN during the preceding quarter hour.

Updated

Time stamp indicating when the statistics were updated.

Routing

HTTP Redirects

Total number of HTTP redirects since startup.

RTSP Redirects

Total number of RTSP redirects since startup.

Keep Alives

Port number open for communication between the Content Engine and the Content Distribution Manager.

Updated

Time stamp indicating when the statistics were updated.

WMT

Requests/Sec

Number of requests per second.

Bytes/Sec

Number of bytes per second.

Hit Rate

Average number of content items per minute successfully served from the cache of the Content Engine or from all the Content Engines in the channel or virtual CDN during the preceding quarter hour.

Updated

Time stamp indicating when the statistics were updated.


Viewing Device Group Statistics

To view device group statistics for your CDN, follow these steps:

Step 1 From the Content Distribution Manager GUI, choose Monitoring > Statistics.

Step 2 In the Contents pane, click Device Groups, and then choose one of the following submenu options to view the Content Engine statistics for that option:

•Cisco Streaming Engine

•HTTP

•Real Proxy

•Routing

•WMT

Statistics for device groups are the same as for Content Engines. See Table 11-1 for an explanation of the meaning of each device group statistic presented in the GUI.

Step 3 To print the statistics data, click the Printer icon.

Viewing Routing Statistics

To view routing statistics, follow these steps:

Step 1 From the Content Distribution Manager GUI, choose Monitoring > Statistics.

Step 2 In the Contents pane, click Content Routers, and then choose one of the following submenu options to view the Content Engine statistics for that option:

•Routing Requests

•Routing Redirects

Step 3 To print the statistics data, click the Printer icon.

Table 11-2 explains the meaning of each Content Router statistic presented in the GUI.

Table 11-2 Content Router Statistics 

Content Router Property
Description
Routing Requests

Total Requests

Total number of content requests received by simplified hybrid routing.

Http Requests

Number of ASX and traditional HTTP web requests received.

Rtsp Requests

Number of RTSP requests received.

Updated

Time stamp indicating when the statistics were updated.

Routing Redirects

Total Requests

Total number of content requests received by simplified hybrid routing.

Reqs Redirected

Total number of content requests received by simplified hybrid routing that were redirected.

Reqs Not Directed

Total number of content requests received by simplified hybrid routing that were not redirected.

Updated

Time stamp indicating when the statistics were updated.


You can also view additional content routing statistics through the Content Distribution Manager GUI, by choosing Device > Content Routers > Show/Clear Commands > Show Commands. The show commands are further documented in the Cisco ACNS Software Command Reference, Release 5.0 publication.

Viewing the System Message Log

Using the system message log feature of the Content Distribution Manager GUI, you can view information about events that have occurred in your CDN. To view logged information for your CDN, follow these steps.

Note The Content Distribution Manager logs messages only of the severity level "critical" or higher from registered nodes.

Step 1 From the Content Distribution Manager GUI, choose Monitoring > Logs.

The System Message Log window appears. (See Figure 11-1.)

Figure 11-1 System Message Log (Showing All Messages)

Step 2 Choose one of the following types of messages to display from the System Message Log drop-down list:

•All

•CLI

•Critical

•Database

A table listing and describing each message is displayed.

Step 3 Click a column heading to sort the messages chronologically, by node type, node name, module, severity, or message text. By default, messages are listed chronologically. (See Figure 11-1.)

Note If no name is available for a node, the name displayed is "Unavailable." This might occur if the node has been deleted or has been reregistered with Cisco ACNS 5.0 software.

Step 4 If you have many event messages, you might need to view multiple pages to view the activity that you are interested in. Click the forward (>>) and back (<<) buttons to move between pages. Alternatively, click the link for a specific page number to jump to that page.

Note You can choose the number of rows to be displayed in the System Message Log window by choosing a number from the Rows drop-down list.

Troubleshooting Using the Content Distribution Manager GUI show Command Utility

To use the Content Distribution Manager GUI show command utility, follow these steps:

Step 1 From the Content Distribution Manager GUI, choose Devices > CDM (or Devices > Content Engines or Devices > Content Routers).

Step 2 Click the Edit icon next to the name of the device that you want to issue a show command for.

Step 3 In the Contents pane, click Show/Clear Commands and then click Show Commands.

Step 4 Choose a show command from the drop-down list.

Step 5 Enter arguments for the command, if any. (Refer to the Cisco ACNS Software Command Reference, Release 5.0 publication for more command information.)

Step 6 Click Submit. A window appears, displaying the show command output for that device.

Troubleshooting Content Router Configurations

Since there are quite a few configuration setups required for the Content Router to redirect the request properly, you might see some content request errors from the Content Router when the configuration is not quite complete. Some areas to look at when troubleshooting are as follows:

•DNS delegation

–Is the requested domain delegated to the Content Router on the D-proxy of the user's end-system? Check with the system administrator to delegate a domain.

•Content Router routing properties

–Is the Content Router activated? See the "Activating Content Engines or Content Routers in the Content Distribution Manager GUI" section to activate a Content Router.

–Is a default coverage zone set for a Content Engine, or is there a CDN-wide coverage zone file or a local coverage zone file set for the Content Router or routing Content Engine? See the "Importing Coverage Zone Files" section to set a coverage zone file. See the "Selecting a Default Coverage Zone" section to set a default coverage zone.

–If a coverage zone entry is for a routing Content Engine, does the "agent" field contain the name of the routing Content Engine? See the "Coverage Zones and Coverage Zone Files" section for a description of agent fields in coverage zone files. See the "Scenario 3: Multiple Content Engines Behind a NAT Firewall" section for an example of a coverage zone entry for a routing Content Engine.

–Is the content request from an end system covered by a Content Engine in a coverage zone based on the default coverage zone or the coverage zone file? This Content Engine is the "serving Content Engine." See the "Coverage Zones and Coverage Zone Files" section under "Understanding Simplified Hybrid Routing and Coverage Zones" in Chapter 4 for information on how to select a serving Content Engine for a client.

–Is the serving Content Engine activated? See the "Activating Content Engines or Content Routers in the Content Distribution Manager GUI" section to activate a Content Engine.

–Is there a channel created for the requested domain and a serving Content Engine assigned to this channel? See the "Working with Channels" section.

–Is the serving Content Engine alive? Use the CLI show statistics content-routing ce cename command to show the status of a Content Engine. (Refer to the Cisco ACNS Software Command Reference, Release 5.0 publication.)

•Content pre-positioned on a Content Engine

–Is there a manifest file assigned to the channel associated with the serving Content Engine? See the "Working with Manifest Files" section.

–Is the manifest file accessible from the Content Distribution Manager? See the "Creating and Modifying Channels" section.

–Is there any syntax error in the manifest file? See the "Manifest File Structure and Syntax" section.

–Is the requested content specified in the manifest file? See the "Specifying a Single Content Item" section.

–If the requested content is streaming media, is the application server enabled with the correct license key? See the "Enabling RealProxy" section, the "Enabling RealSubscriber" section, and the "Enabling the Cisco Streaming Engine" section.

–Has disk space been configured for the mediafs to store streaming media objects or the cdnfs to store nonstreaming content? See the "Configuring Disk Space" section.

–Has the requested content been successfully acquired by the Content Engine? See the "Listing Website Content Using the Spider Script" section.

Recovery Procedures

This section discusses how to recover a corrupted system image, how to recover a lost password, how to recover from missing disk-based software, and how to recover from catastrophic failure on a Content Engine, Content Router, or Content Distribution Manager.

Recovering the System Software

The Content Engine, Content Router, and Content Distribution Manager have a resident rescue system image that is invoked should the image in Flash memory be corrupted. A corrupted system image can result from a power failure that occurs while a system image is being written to Flash memory. The rescue image can download a system image to the main memory of the device and write it to Flash memory.

To install a new system image using the rescue image, perform the following steps:

Step 1 Download the system image file (*.sysimg) to a host that is running an FTP server.

Step 2 Establish a console connection to the device and open a terminal session.

Step 3 Reboot the device by toggling the power on/off switch.

The rescue image dialog appears. The following example demonstrates how to interact with the rescue dialog (user input is denoted by entries in bold typeface): 

This is the rescue image. The purpose of this software is to let

you download and install a new system image onto your system's

boot flash device. This software has been invoked either manually

(if you entered `***' to the bootloader prompt) or has been

invoked by the bootloader if it discovered that your system image

in flash had been corrupted.

To download an image, this software will request the following

information from you:

    - which network interface to use

    - IP address and netmask for the selected interface

    - default gateway IP address

    - server IP address

    - which protocol to use to connect to server

    - username/password (if applicable)

    - path to system image on server


Please enter an interface from the following list:

    0: FastEthernet 0/0

    1: FastEthernet 0/1

    0

Using interface FastEthernet 0/0

Please enter the local IP address to use for this interface:

[Enter IP Address]: 172.16.22.22

Please enter the netmask for this interface:

[Enter Netmask]: 255.255.255.224

Please enter the IP address for the default gateway:

[Enter Gateway IP Address]: 172.16.22.1

Please enter the IP address for the FTP server where you wish

to obtain the new system image:

[Enter Server IP Address]: 172.16.10.10

Please enter your username on the FTP server (or 'anonymous'):

[Enter Username on server (e.g. anonymous)]: anonymous

Please enter the password for username 'anonymous' on FTP server (an email address): 

Please enter the directory containing the image file on the FTP server:

[Enter Directory on server (e.g. /)]: /

Please enter the file name of the system image file on the FTP server:

[Enter Filename on server]: ACNS-5.0.0-K9.sysimg

Here is the configuration you have entered:

Current config:

          IP Address: 172.16.22.22

             Netmask: 255.255.255.224

     Gateway Address: 172.16.22.1

      Server Address: 172.16.10.10

            Username: anonymous

            Password: 

     Image directory: /

      Image filename: ACNS-5.0.0-K9.sysimg

Attempting download...

Downloaded 10711040 byte image file

A new system image has been downloaded.

You should write it to flash at this time.

Please enter 'yes' below to indicate that this is what you want to do:

[Enter confirmation ('yes' or 'no')]: yes

Ok, writing new image to flash

..................................................................................Finished 
writing image to flash.

Enter 'reboot' to reboot, or 'again' to download and install a new image:

[Enter reboot confirmation ('reboot' or 'again')]: reboot

Restarting system.


Initializing memory. Please wait.

Step 4 Log in to the device as username admin. Verify that you are running the correct version by entering the show version command. 

Username: admin 

Password: 


Console> enable

Console# show version

Copyright (c) 1986-2003 by Cisco Systems, Inc.

Cisco Application and Content Networking Software Release 5.0 

Compiled 18:48:10 Feb 26 2003 by (cisco)


System was restarted on Wed Feb 26 22:12:25 2003.

The system has been up for 1 day, 5 hours, 26 minutes, 53 seconds.

Recovering a Lost Administrator Password

If an administrator password is forgotten, lost, or misconfigured, perform the following steps to reset the password on the device.

Note There is no way to restore a lost administrator password. You must reset the password to a new one, as described in this procedure.

Step 1 Establish a console connection to the device and open a terminal session.

Step 2 Reboot the device. While the device is rebooting, watch for the following prompt and press Enter when you see it: 

Cisco ACNS boot:hit RETURN to set boot flags:0009

Step 3 When prompted to enter bootflags, enter this value: 

0x8000

For example: 

Available boot flags (enter the sum of the desired flags):

0x4000 - bypass nvram config

0x8000 - disable login security

[CE boot - enter bootflags]:0x8000

You have entered boot flags = 0x8000

Boot with these flags? [yes]:yes


[Display output omitted]

Step 4 When the device completes the boot sequence, you are prompted to enter the username to access the CLI. Enter the default administrator username (admin): 

Cisco Content Engine Console

Username: admin

Setting the configuration flags to 0x8000 lets you into the system, bypassing all 
security. Setting the configuration flags field to 0x4000 lets you bypass the NVRAM 
configuration.

Step 5 Once you see the CLI prompt, set the password for the user using the username password command in global configuration mode.

The following example shows the different options and parameters for the username command. You can specify that the password be either clear text or encrypted. The user in the example chose to have an encrypted password. 

ContentEngine# configure

ContentEngine(config)# username ?

  WORD  User name

ContentEngine(config)# username conwilso ?

  password        Specify the password for the user

  privilege       Set user privilege level

  samba-password  Set user's Windows sharing password

ContentEngine(config)# username conwilso password ?

  0     Specifies clear-text password (default)

  1     Specifies type 1 encrypted password

  WORD  User password (clear text)

ContentEngine(config)# username conwilso password 0 ?

  WORD  User password (encrypted)

ContentEngine(config)# username conwilso password 0 mypassword ?

  uid   User Id

  <cr>  

ContentEngine(config)# username conwilso password 0 mypassword uid ?

  <2001-65535>  User Id

ContentEngine(config)# username conwilso password 0 mypassword uid 2001 ?

  <cr>  

ContentEngine(config)#

Step 6 Use the write memory command in EXEC mode to save the configuration change. 

ContentEngine(config)# exit

ContentEngine# write memory

Step 7 Optionally, reboot your device by using the reload command. 

ContentEngine# reload

Rebooting is optional; however, you might want to reboot to ensure that the boot flags are reset, and to ensure that subsequent console administrator logins do not bypass the password check.

Note In ACNS software, the bootflags are reset to 0x0 on every reboot.

Recovering from Missing Disk-Based Software

This section describes the recovery procedures to use if for some reason the software installation on the first disk drive (disk00) is corrupt or missing.

This situation is most likely to occur only if you replaced disk00 in your Content Engine, Content Router, or Content Distribution Manager. By design, the software installation on disk00 cannot be corrupted by a system crash or a power failure.

If the system disk (disk00) fails or is missing, the software continues to run; however, it runs in a degraded mode in which HTTP proxy and related HTTP features still work, but most other features fail.

To recover from this condition, follow these steps:

Step 1 Remove the Content Engine record from the Content Distribution Manager GUI.

a. From the Content Distribution Manager GUI, choose Devices > Content Engines.

b. Click the Edit icon next to the name of the Content Engine that you wish to delete. The browser window refreshes, displaying the Modifying a Content Engine window.

c. Click the Trash icon. You are prompted to confirm your decision.

d. Click OK to execute your request. The Content Engine is removed from the Content Distribution Manager GUI.

Note The Content Engine registration record needs to be deleted from the Content Distribution Manager in order for the Content Engine to complete reregistration after it comes back online. The Content Distribution Manager will not register a device if the device already appears in the record as registered.

Step 2 Power down the device and replace the failed or missing disk00 with a new, blank disk.

Step 3 After the new disk is installed, power up the device.

Step 4 From a console or through a Telnet session, check the startup messages that appear on your screen.

If there is a problem with disk00 or the disk-based software, a message similar to the following appears: 

*****

Your first disk is not in standard configuration.

You might need to run 'disk recover' from the CLI.

*****

Step 5 Log in as admin. 

Cisco Content Engine Console


Username: admin

Password: 

System Initialization Finished.



CE-507 con now available



Press RETURN to get started!

Step 6 Enter the disk recover command to create the file systems on disk00 that are for internal system use. 

CE-507# disk recover

This will erase everything on disk00. Are you sure? [no]yes

System filesystems appear to have been installed.

Please verify your software installation with 'show flash'

and install a new image if necessary.

CE-507#

CE-507# show flash

/diamond/bin/exec_show_flash: could not open /swstore/manifest: No such file or directory

Your software installation is damaged.

Please run 'copy ftp install' to install a new image.

Step 7 Enter the copy ftp install command to download and install a new system image. 

ContentEngine# copy ftp install ftp-server remotefiledir remotefilename

For example: 

CE# copy ftp install vista /users2/gid/bfc/boot ce507-ACNS-5.0.0-K9.bin

Enter username for remote ftp server: gid

Enter password for remote ftp server: 

Initiating FTP download...

printing one # per 1MB downloaded

Sending: USER gid

vista.cisco.com FTP server (Version wu-2.6.0(1) Mon Feb 24 10:30:36 EST 2003) ready.

Password required for gid.

Sending: PASS *****  

User gid logged in.

Sending: TYPE I

Type set to I.

Sending: PASV

Entering Passive Mode (128,107,193,244,173,205)

Sending: CWD /users2/gid/bfc/boot

CWD command successful.

Sending PASV

Entering Passive Mode (128,107,193,244,173,205)

Sending: RETR ruby.bin

Opening BINARY mode data connection for ruby.bin (102794344 bytes).

##########################################################################################

Installing phase3 bootloader...

Installing system image to flash: image on flash identical to new image, no work required

The new software will run after you reload.

#

ContentEngine# show flash

ACNS software version (disk-based code): ACNS-5.0.0-b130


System image on flash:

Version: 5.0.0


System flash directory:

System image: 98 sectors

Bootloader, rescue image, and other reserved areas: 26 sectors

128 sectors total, 4 sectors free.

Step 8 Enter the disk config command to define file system space allocations on disk00. For example: 

ContentEngine# disk config sysfs 10% cfs 20% mediafs 20% cdnfs 50%

Note Disk allocation percentages or values should reflect the anticipated usage for each file type.

Alternatively, you can run the disk config command after you reboot the software.

Step 9 Use the reload command to reboot the software with the new disk and new system image running. 

ContentEngine# reload

Step 10 Register the device with the Content Distribution Manager by using the cms enable command in global configuration mode. 

CE-507# configure

CE-507(config)# cms enable

Replacing a Failed Disk Drive

When adding or replacing a disk drive other than disk00, you need to use the disk add command to make the system aware of the additional disk space and to reallocate the file systems, if necessary.

Note In ACNS 5.0 software, the disk add command does not support disk00 but supports disk01 or higher, where the drive in the slot is a blank new replacement disk. Use the disk recover command rather than the disk add command to add disk00.

Performing Backup and Restore for the Content Distribution Manager Configuration Management Subsystem Database

The Content Distribution Manager stores CDN-wide device configuration information in its configuration management subsystem (CMS) database. You can manually back up the CMS embedded database contents for greater system reliability.

To back up the CMS database for the Content Distribution Manager, use the cms database backup EXEC command. For database backups, you need to specify the following items:

•Location, password, and user ID

•Dump format in PostgreSQL plain text syntax

Note The naming convention for backup files includes the time stamp.

To backup and restore the CMS database on the Content Distribution Manager, follow these steps:

Step 1 Back up the CMS database to a file. 

CDM# cms database backup 

creating backup file with label \Qbackup' 

backup file local1/acns-db-9-22-2002-17-36.dump is ready. use \Qcopy' commands to move the 
backup file to a remote host.

Step 2 Save the file to a remote server by using the copy disk ftp command. This command copies the file from the local disk to a remote ftp server. For example, 

CDM# cd /local1

CDM# copy disk ftp 10.86.32.82 /incoming acns-db-9-22-2002-17-36.dump 
acns-db-9-22-2002-17-36.dump 


Enter username for remote ftp server:ftp

Enter password for remote ftp server:*******

Initiating FTP upload...

Sending:USER ftp

10.86.32.82 FTP server (Version wu-2.6.1-18) ready.

Password required for ftp.

Sending:PASS *******  

User ftp logged in.

Sending:TYPE I

Type set to I.

Sending:PASV

Entering Passive Mode (10,86,32,82,112,221)

Sending:CWD /incoming

CWD command successful.

Sending PASV

Entering Passive Mode (10,86,32,82,203,135)

Sending:STOR acns-db-9-22-2002-17-36.dump

Opening BINARY mode data connection for acns-db-9-22-2002-17-36.dump.

Transfer complete.

Sent 18155 bytes

Step 3 Delete the existing CMS database. 

CDM# cms database delete

Step 4 Restore the CMS database contents from the backup file. For example, 

CDM# cms database restore acns-db-9-22-2002-17-36

Step 5 Enable CMS. 

CDM# cms enable