Troubleshooting
This appendix provides information on troubleshooting. The following topics are covered in this appendix:
•
Troubleshooting Service Router Configurations
•
Troubleshooting the Distribution Hierarchy
•
Troubleshooting Content Acquisition
•
Enabling the Kernel Debugger
For more troubleshooting tools, see Chapter 7 "Monitoring the Internet Streamer CDS."
Troubleshooting Service Router Configurations
Because there are many steps required for the Service Router to redirect the request properly, you might see some content request errors from the Service Router when the configuration is not quite complete. Here are some areas to look at when troubleshooting:
•
DNS delegation
–
Is the requested domain delegated to the Service Router on the DNS server that is authoritative for the parent domains? The Service Router's DNS name should be forward resolvable. Check with the system administrator to delegate a domain.
•
Service Router routing properties
–
Is the Service Router activated? See the "Activating a Service Router" section to activate a Service Router.
–
Is a default coverage zone set for a Service Engine, or is there a CDS network-wide Coverage Zone file or a local Coverage Zone file set for the Service Router? See the "Coverage Zone File Registration" section to set a Coverage Zone file. See the "Configuring the Service Engine" section to set a default coverage zone.
–
Is the content request from an end system covered by a Service Engine in a coverage zone based on the default coverage zone or the Coverage Zone file? This Service Engine is the "serving Service Engine." See the "Coverage Zone File" section for information on coverage zones. See "Creating Coverage Zone Files," for information on creating a Coverage Zone file.
–
Is the serving Service Engine activated? See the "Activating a Service Engine" section to activate a Service Engine.
–
Is there a delivery service created for the requested domain and a serving Service Engine assigned to this delivery service? See the "Creating Delivery Service" section.
–
Is the serving Service Engine alive? Use the show statistics service-routing se command to show the status of a Service Engine. See the "Using show and clear Commands" section.
•
Content prefetched on a Service Engine
–
Is a Manifest file assigned to the delivery service associated with the serving Service Engine? See the "Working with Manifest Files" section.
–
Is the Manifest file accessible from the CDSM? See the "Identifying Content Using a Manifest File" section.
–
Is there any syntax error in the Manifest file? See the "Manifest File Structure and Syntax" section.
–
Is the requested content specified in the Manifest file? See the "Specifying a Single Content Item" section.
–
If the requested content is streaming media, is the protocol engine enabled? See the "Application Control" section.
For general information, use the show statistics service-router all command.
Troubleshooting the Distribution Hierarchy
Because distribution-related problems are design-dependent, your initial strategy is to discover whether or not the correct Service Engine is sending content in the correct distribution path.
•
To determine which Service Engines are in the distribution path of a particular Service Engine, use the show distribution remote traceroute EXEC command, as shown in the following example:
ce1# show distribution remote traceroute ?
forwarder-next-hop next forwarder along the path
unicast-sender check status for unicast sender
ce1# show distribution remote traceroute forwarder-next-hop ?
delivery-service-id Delivery-service-id of a Delivery Service
ce1# show distribution remote traceroute forwarder-next-hop delivery-service-id 133 ?
max-hop Trace route till specified number of hops is reached
trace-till-good traceroute till probe is good or the object is found
trace-till-root traceroute till the acquirer
ce1# show distribution remote traceroute forwarder-next-hop delivery-service-id 133
trace-till-root
Hop NextHop_SEId NextHop_SEName NextHop_SEIp GenID Status/Reason
--- ------------ -------------- ------------ ----- -------------
1 1100 ce3 10.255.0.43 1 LOC-LEAD
1 1100 ce3 128.107.193.183 1 LOC-LEAD (Reached RootCE)
•
To verify that the Service Engine is reachable and that it is in the distribution hierarchy, use the show distribution remote traceroute EXEC command, as shown in the following example:
se1# show distribution remote traceroute unicast-sender delivery-service-id 133 ?
cdn-url check the object on remote SE using cdn-url
probe probe the remote unicast sender
relative-cdn-url check the object on remote SE using relative-cdn-url
se1# show distribution remote traceroute unicast-sender delivery-service-id 133 probe
?
max-hop Max-hop to traceroute to
trace-till-good traceroute till probe is good or the object is found
trace-till-root traceroute till the root se
se1# show distribution remote traceroute unicast-sender delivery-service-id 133 probe
trace-till-root
Polling .... se3 [10.255.0.43] Fwdr_Id:1100
Polling .... se3 [128.107.193.183] Fwdr_Id:1100
Troubleshooting Content Acquisition
To monitor acquisition progress and to troubleshoot, use the following commands from the Content Acquirer CLI:
•
Use the show acquirer delivery-services EXEC command to obtain delivery service information, such as the delivery service ID and delivery service name, that you need to enter in other show acquirer commands, such as the show acquirer progress command. In the following example, the delivery service ID is 793 and the delivery service name is group01-cifs.
SE# show acquirer delivery-services
Acquirer information for all delivery services:
--------------------------------------
Delivery-service-id : 793
Delivery-service-Name : group01-cifs
WebSite-Name : group01-cifs
Root-CE-Type : Configured
Origin FQDN : cdn.allcisco.com
Delivery-service Priority : 500
Manifestfile-URL : ftp://10.1.1.1/cifs.xml
•
Use the show acquirer EXEC command to make sure that the acquirer process on the Content Acquirer is working correctly, and that the device is using the expected amount of bandwidth for acquisition. The following example shows that the acquirer is running properly and that the device is configured with unlimited bandwidth for acquisition of content.
Current Acquisition Bandwidth:Not Limited
•
Use the show acquirer progress EXEC command to check how far the acquisition of content has progressed. A specific delivery service ID or delivery service name can be specified to obtain the progress for a specific delivery service. In the following example, the acquirer has already acquired 2237 items.
SE# show acquirer progress delivery-service-id 793
Acquirer progress information for delivery service ID:793
Delivery-service-Name:group01-cifs
-----------------------------------------------------------------
Acquired Single Items : 0 / 0
Acquired Crawl Items : 2237 / 2500 -- start-url=www.mtv.com//
•
Use the show statistics acquirer delivery-service-id or show statistics acquirer delivery-service-name EXEC command to obtain the detailed acquisition statistics for a given delivery service. In the following example, there was an error acquiring two items.
SE# show statistics acquirer delivery-service-id 793
Statistics for Delivery Service Delivery-service-id :793 Delivery-service-Name
:group01-cifs
---------------------------------------------------------
Total Number of Acquired Objects :2237
Total Disk Used for Acquired Objects :981511280 Bytes
Total Number of Failed Objects :2
Total Number of Re-Check Failed Objects :0
•
Use the show statistics acquirer errors delivery-service-id or show statistics acquirer errors delivery-service-name EXEC command to see the reasons why the errors occurred. In the following example, one error occurred because there was a problem acquiring the URL. The other error occurred because the disk quota for the delivery service configured in the Content Distribution Manager GUI would have been exceeded if the specified URL had been acquired. You can increase the delivery service disk quota to correct this error.
SE# show statistics acquirer errors delivery-service-id 793
Acquisition Errors for the Delivery Service ID:793
-------------------------------------
Crawl job:start-url http://www.mtv.com//
Internal Server Error(500):http://cgi.cnn.com/entries/intl-emailsubs-confirm
Exceeded Disk Quota(703):http://www.cdt.org/copyright/backgroundchart.pdf
•
If more detailed troubleshooting of content acquisition is required, you can increase the debug level of the acquirer using the debug acquirer trace EXEC command. The logs are written to local1/errorlog/acquirer-errorlog.current.
•
To verify that an expected object has been pre-positioned on the Service Engine, use the show distribution object-status EXEC command, as shown in the following example:
SE# show distribution object-status
http://172.18.81.168/Videos/SM-final%20Innebandy%202003.wmv
========== Website Information ==========
Origin Server FQDN: 172.18.81.168
Service Routing FQDN: N/A
Content UNS Reference #: 1
========== Delivery Services Information ==========
*** Delivery Service 1903 (name = A_Multicast) ***
File State: Ready for distribution
Multicast for Delivery Service: Not Enabled
Replication Lock: Received by Unicast-Receiver/Acquirer
Transferred Size: 2756437
MD5 of MD5: tjS#DxqE5oUc024Z8XtFDw..
Source Url: http://172.18.81.168/Videos/SM-final%20Innebandy%202003.wmv
Source Last Modified Time: Wed Jan 7 19:03:48 2004
Requires Authentication: No
Play servers: HTTP HTTPS WMT
Content uns_id: NgcJTCU#JaY4ZGIPbsrONw..
Content gen-id: 1768:1136512329:2
========== CDNFS Information ==========
Internal File Name:
/disk00-04/d/http-172.18.81.168-k5bsm1o+y14jgiqsvwaohq/19/19f6d5cec7266c33f419709dc28c
8d9b.0.data.wmv
Actual File Size: 2756437 bytes
MD5 of MD5 (Re-calculated): tjS#DxqE5oUc024Z8XtFDw..
Metadata match with: Delivery Service 1903
Source-url to CDN-object mapping:
Source-url: http://172.18.81.168/Videos/SM-final%20Innebandy%202003.wmv
Used by CDN object: ---- Yes ----
Internal File Name:
/disk00-04/d/http-172.18.81.168-k5bsm1o+y14jgiqsvwaohq/19/19f6d5cec7266c33f419709dc28c
8d9b.0.data.wmv
Actual File Size: 2756437 bytes
========== CDNFS lookup output ==========
Allowed Playback via HTTP WMT HTTPS
Last-modified Time Wed Jan 7 19:03:48 2004
cache-control max-age=864000
cdn_uns_id NgcJTCU#JaY4ZGIPbsrONw..
content-type video/x-ms-wmv
etag "042e6fa50d5c31:b39"
last-modified Wed, 07 Jan 2004 19:03:48 GMT
Internal path to data file:
/disk00-04/d/http-172.18.81.168-k5bsm1o+y14jgiqsvwaohq/19/19f6d5cec7266c33f419709dc28c
8d9b.0.data.wmv
By comparing fields, such as Total Size, Transferred Size, and Source URL in the Object Replication output and Actual File Size and Source URL in the Source-URL to CDN-Object Mapping output, you can determine whether or not the object that is stored is the same as the object that was requested.
•
To view the file directory structure on the Service Engine and verify the physical file on the disk, you can use the cdnfs browse EXEC command, as shown in the following example:
------ CDNFS interactive browsing ------
dir, ls: list directory contents
cd,chdir: change current working directory
info: display attributes of a file
more: page through a file
exit,quit: quit CDNFS browse shell
http-172.18.81.151-glfc4h-b9gywnf5rlnfweg/
http-172.18.81.163-og5o21u178nrhw1mctgtiq/
file--xrnfwxifgu62jtiwtyixvg/
http-172.18.81.168-k5bsm1o+y14jgiqsvwaohq/
376 Bytes manifest-Delivery_service_1903.xml-lEYmrfnjt2o5GbUNwLCApA
/172.18.81.168/>cd Videos
/172.18.81.168/Videos/>ls
2756437 Bytes SM-final Innebandy 2003.wmv <===============Physical file on disk
/172.18.81.168/Videos/>quit
Enabling the Kernel Debugger
Cisco CDS software allows you to enable or disable access to the kernel debugger from the CDSM. Once enabled, the kernel debugger is automatically activated when kernel problems occur.
Note
The "hardware watchdog" is enabled by default and automatically reboots a device that has stopped responding for over ten minutes. Enabling the kernel debugger disables the "hardware watchdog."
If the device runs out of memory and kernel debugger (KDB) is enabled, the KDB is activated and dump information. If the KDB is disabled and the device runs out of memory, the syslog reports only dump information and reboots the device.
To enable the kernel debugger, do the following:
Step 1
Choose Devices > Devices > General Settings > Troubleshooting > Kernel Debugger. The Kernel Debugger page is displayed.
Step 2
To enable the kernel debugger, check the Enable check box, and click Submit.