Cisco Internet Streamer CDS 2.5 Software Configuration Guide
Troubleshooting
Downloads: This chapterpdf (PDF - 134.0KB) The complete bookPDF (PDF - 4.65MB) | Feedback

Troubleshooting

Table Of Contents

Troubleshooting

Troubleshooting Service Router Configurations

Troubleshooting the Distribution Hierarchy

Troubleshooting Content Acquisition

Enabling the Kernel Debugger


Troubleshooting


This appendix provides information on troubleshooting. The following topics are covered in this appendix:

Troubleshooting Service Router Configurations

Troubleshooting the Distribution Hierarchy

Troubleshooting Content Acquisition

Enabling the Kernel Debugger

For more troubleshooting tools, see Chapter 7 "Monitoring the Internet Streamer CDS."

Troubleshooting Service Router Configurations

Because there are many steps required for the Service Router to redirect the request properly, you might see some content request errors from the Service Router when the configuration is not quite complete. Here are some areas to look at when troubleshooting:

DNS delegation

Is the requested domain delegated to the Service Router on the DNS server that is authoritative for the parent domains? The Service Router's DNS name should be forward resolvable. Check with the system administrator to delegate a domain.

Service Router routing properties

Is the Service Router activated? See the "Activating a Service Router" section to activate a Service Router.

Is a default coverage zone set for a Service Engine, or is there a CDS network-wide Coverage Zone file or a local Coverage Zone file set for the Service Router? See the "Coverage Zone File Registration" section to set a Coverage Zone file. See the "Configuring the Service Engine" section to set a default coverage zone.

Is the content request from an end system covered by a Service Engine in a coverage zone based on the default coverage zone or the Coverage Zone file? This Service Engine is the "serving Service Engine." See the "Coverage Zone File" section for information on coverage zones. See "Creating Coverage Zone Files," for information on creating a Coverage Zone file.

Is the serving Service Engine activated? See the "Activating a Service Engine" section to activate a Service Engine.

Is there a delivery service created for the requested domain and a serving Service Engine assigned to this delivery service? See the "Creating Delivery Service" section.

Is the serving Service Engine alive? Use the show statistics service-routing se command to show the status of a Service Engine. See the "Using show and clear Commands" section.

Content prefetched on a Service Engine

Is a Manifest file assigned to the delivery service associated with the serving Service Engine? See the "Working with Manifest Files" section.

Is the Manifest file accessible from the CDSM? See the "Identifying Content Using a Manifest File" section.

Is there any syntax error in the Manifest file? See the "Manifest File Structure and Syntax" section.

Is the requested content specified in the Manifest file? See the "Specifying a Single Content Item" section.

If the requested content is streaming media, is the protocol engine enabled? See the "Application Control" section.

For general information, use the show statistics service-router all command.

Troubleshooting the Distribution Hierarchy

Because distribution-related problems are design-dependent, your initial strategy is to discover whether or not the correct Service Engine is sending content in the correct distribution path.

To determine which Service Engines are in the distribution path of a particular Service Engine, use the show distribution remote traceroute EXEC command, as shown in the following example:

ce1# show distribution remote traceroute ?
  forwarder-next-hop  next forwarder along the path
  unicast-sender      check status for unicast sender
 
   
ce1# show distribution remote traceroute forwarder-next-hop ?
  delivery-service-id  Delivery-service-id of a Delivery Service
 
   
ce1# show distribution remote traceroute forwarder-next-hop delivery-service-id 133 ?
  max-hop          Trace route till specified number of hops is reached
  trace-till-good  traceroute till probe is good or the object is found
  trace-till-root  traceroute till the acquirer
 
   
ce1# show distribution remote traceroute forwarder-next-hop delivery-service-id 133 
trace-till-root
 
   
Hop NextHop_SEId  NextHop_SEName NextHop_SEIp      GenID Status/Reason
--- ------------  -------------- ------------      ----- -------------
  1    1100                ce3     10.255.0.43     1       LOC-LEAD
  1    1100                ce3 128.107.193.183     1       LOC-LEAD (Reached RootCE)
 
   

To verify that the Service Engine is reachable and that it is in the distribution hierarchy, use the show distribution remote traceroute EXEC command, as shown in the following example:

se1# show distribution remote traceroute unicast-sender delivery-service-id 133 ?
  cdn-url           check the object on remote SE using cdn-url
  probe             probe the remote unicast sender
  relative-cdn-url  check the object on remote SE using relative-cdn-url
 
   
se1# show distribution remote traceroute unicast-sender delivery-service-id 133 probe 
?
  max-hop          Max-hop to traceroute to
  trace-till-good  traceroute till probe is good or the object is found
  trace-till-root  traceroute till the root se
 
   
se1# show distribution remote traceroute unicast-sender delivery-service-id 133 probe 
trace-till-root
Polling .... se3 [10.255.0.43]  Fwdr_Id:1100
Polling .... se3 [128.107.193.183]  Fwdr_Id:1100
(Reached RootSE)

Troubleshooting Content Acquisition

To monitor acquisition progress and to troubleshoot, use the following commands from the Content Acquirer CLI:

Use the show acquirer delivery-services EXEC command to obtain delivery service information, such as the delivery service ID and delivery service name, that you need to enter in other show acquirer commands, such as the show acquirer progress command. In the following example, the delivery service ID is 793 and the delivery service name is group01-cifs.

SE# show acquirer delivery-services
Querying Database.......
Acquirer information for all delivery services:
--------------------------------------
Delivery-service-id       : 793
Delivery-service-Name     : group01-cifs
WebSite-Name     : group01-cifs
Root-CE-Type     : Configured
State            : Enabled
Disk Quota       : 200 MB
Origin FQDN      : cdn.allcisco.com
Delivery-service Priority : 500
Manifestfile-TTL : 5
Manifestfile-URL : ftp://10.1.1.1/cifs.xml
 
   

Use the show acquirer EXEC command to make sure that the acquirer process on the Content Acquirer is working correctly, and that the device is using the expected amount of bandwidth for acquisition. The following example shows that the acquirer is running properly and that the device is configured with unlimited bandwidth for acquisition of content.

SE# show acquirer 
Acquirer is running OK
Current Acquisition Bandwidth:Not Limited
 
   

Use the show acquirer progress EXEC command to check how far the acquisition of content has progressed. A specific delivery service ID or delivery service name can be specified to obtain the progress for a specific delivery service. In the following example, the acquirer has already acquired 2237 items.

SE# show acquirer progress delivery-service-id 793
Querying Database.......
 
   
Acquirer progress information for delivery service ID:793 
Delivery-service-Name:group01-cifs
-----------------------------------------------------------------
 
   
Acquired Single Items   :         0 / 0          
Acquired Crawl Items    :     2237 / 2500     -- start-url=www.mtv.com// 
 
   

Use the show statistics acquirer delivery-service-id or show statistics acquirer delivery-service-name EXEC command to obtain the detailed acquisition statistics for a given delivery service. In the following example, there was an error acquiring two items.

SE# show statistics acquirer delivery-service-id 793
Querying Database.......
 
   
Statistics for Delivery Service Delivery-service-id :793 Delivery-service-Name 
:group01-cifs
---------------------------------------------------------
 
   
Manifest:
---------
        Fetch Errors    :0
        Parsing Errors  :0
        Parsing Warnings:0
 
   
Acquisition:
-------------
        Total Number of Acquired Objects        :2237
        Total Disk Used for Acquired Objects    :981511280 Bytes
        Total Number of Failed Objects          :2
        Total Number of Re-Check Failed Objects :0 
 
   

Use the show statistics acquirer errors delivery-service-id or show statistics acquirer errors delivery-service-name EXEC command to see the reasons why the errors occurred. In the following example, one error occurred because there was a problem acquiring the URL. The other error occurred because the disk quota for the delivery service configured in the Content Distribution Manager GUI would have been exceeded if the specified URL had been acquired. You can increase the delivery service disk quota to correct this error.

SE# show statistics acquirer errors delivery-service-id 793
Querying Database.......
 
   
Acquisition Errors for the Delivery Service ID:793
-------------------------------------
Crawl job:start-url http://www.mtv.com//
Crawl Errors
---------------
Internal Server Error(500):http://cgi.cnn.com/entries/intl-emailsubs-confirm
Exceeded Disk Quota(703):http://www.cdt.org/copyright/backgroundchart.pdf 
 
   

If more detailed troubleshooting of content acquisition is required, you can increase the debug level of the acquirer using the debug acquirer trace EXEC command. The logs are written to local1/errorlog/acquirer-errorlog.current.

To verify that an expected object has been pre-positioned on the Service Engine, use the show distribution object-status EXEC command, as shown in the following example:

SE# show distribution object-status 
http://172.18.81.168/Videos/SM-final%20Innebandy%202003.wmv
 
   
        ========== Website Information ==========
Name:		        RTPServer5
Origin Server FQDN:							172.18.81.168
Service Routing FQDN:							N/A
Content UNS Reference #:							1
        ========== Delivery Services Information ==========
        *** Delivery Service 1903 (name = A_Multicast) ***
 
   
Object Replication
------------------
Replication: 							Done
File State:		    					Ready for distribution
Multicast for Delivery Service: 							Not Enabled
Replication Lock:			 				Received by Unicast-Receiver/Acquirer
Reference Count:	 						1
Total Size:							2756437
Transferred Size:				 			2756437
MD5 of MD5:	 				 		tjS#DxqE5oUc024Z8XtFDw..
Source Url:					 		http://172.18.81.168/Videos/SM-final%20Innebandy%202003.wmv
Source Last Modified Time:	Wed Jan  7 19:03:48 2004
 
   
Object Properties
-----------------
Redirect To Origin:							Yes
Requires Authentication:							No
Alternative URL:                         
Serve Start Time:							N/A
Serve End Time:							N/A
Play servers:							HTTP HTTPS WMT
Content Metadata:							None
Content uns_id:							NgcJTCU#JaY4ZGIPbsrONw..
Content gen-id:							1768:1136512329:2 
 
   
========== CDNFS Information ==========
Internal File Name:	 
/disk00-04/d/http-172.18.81.168-k5bsm1o+y14jgiqsvwaohq/19/19f6d5cec7266c33f419709dc28c
8d9b.0.data.wmv
Actual File Size:						 	2756437 bytes
MD5 of MD5 (Re-calculated):	tjS#DxqE5oUc024Z8XtFDw..
Content metadata:	 						None
Metadata match with:							Delivery Service 1903
Number of Source-urls:							1
 
   
Source-url to CDN-object mapping:
Source-url: 							http://172.18.81.168/Videos/SM-final%20Innebandy%202003.wmv
Used by CDN object:							---- Yes ----
Internal File Name:	 
/disk00-04/d/http-172.18.81.168-k5bsm1o+y14jgiqsvwaohq/19/19f6d5cec7266c33f419709dc28c
8d9b.0.data.wmv
Actual File Size: 							2756437 bytes
 
   
 ========== CDNFS lookup output ==========
CDNFS File Attributes:
  Status                 	3  (Ready)
  File Size              	2756437 Bytes
  Start Time             	null
  End Time               	null
  Allowed Playback via   	HTTP WMT HTTPS
  Last-modified Time     	Wed Jan  7 19:03:48 2004
  cache-control          	max-age=864000
  cdn_uns_id             	NgcJTCU#JaY4ZGIPbsrONw..
  content-type           	video/x-ms-wmv
  etag                   	"042e6fa50d5c31:b39"
  file_duration          	65
  last-modified          	Wed, 07 Jan 2004 19:03:48 GMT
  server                 	Microsoft-IIS/6.0
  x-powered-by           	ASP.NET
Internal path to data file: 
/disk00-04/d/http-172.18.81.168-k5bsm1o+y14jgiqsvwaohq/19/19f6d5cec7266c33f419709dc28c
8d9b.0.data.wmv
 
   

By comparing fields, such as Total Size, Transferred Size, and Source URL in the Object Replication output and Actual File Size and Source URL in the Source-URL to CDN-Object Mapping output, you can determine whether or not the object that is stored is the same as the object that was requested.

To view the file directory structure on the Service Engine and verify the physical file on the disk, you can use the cdnfs browse EXEC command, as shown in the following example:

SE# cdnfs browse
 
   
------  CDNFS interactive browsing  ------
dir, ls:   list directory contents
cd,chdir:  change current working directory
info:      display attributes of a file
more:      page through a file
cat:       display a file
exit,quit: quit CDNFS browse shell
 
   
/>ls
                    srv2-static.cisco.com/
                    http-172.18.81.151-glfc4h-b9gywnf5rlnfweg/
                    http-172.18.81.163-og5o21u178nrhw1mctgtiq/
                    172.18.81.163/
                    file--xrnfwxifgu62jtiwtyixvg/
                    172.18.81.168/
                    http-172.18.81.168-k5bsm1o+y14jgiqsvwaohq/
                    172.18.81.155/
/>cd 172.18.81.168/
/172.18.81.168/>ls
376 Bytes           	manifest-Delivery_service_1903.xml-lEYmrfnjt2o5GbUNwLCApA
                    	Videos/
/172.18.81.168/>cd Videos
/172.18.81.168/Videos/>ls
2756437 Bytes       	SM-final Innebandy 2003.wmv <===============Physical file on disk
/172.18.81.168/Videos/>quit
SE#
 
   

Enabling the Kernel Debugger

Cisco CDS software allows you to enable or disable access to the kernel debugger from the CDSM. Once enabled, the kernel debugger is automatically activated when kernel problems occur.


Note The "hardware watchdog" is enabled by default and automatically reboots a device that has stopped responding for over ten minutes. Enabling the kernel debugger disables the "hardware watchdog."


If the device runs out of memory and kernel debugger (KDB) is enabled, the KDB is activated and dump information. If the KDB is disabled and the device runs out of memory, the syslog reports only dump information and reboots the device.

To enable the kernel debugger, do the following:


Step 1 Choose Devices > Devices > General Settings > Troubleshooting > Kernel Debugger. The Kernel Debugger page is displayed.

Step 2 To enable the kernel debugger, check the Enable check box, and click Submit.