Cisco TV CDS 2.4 RTSP Software Configuration Guide for the Cisco ISM (Integrated Service Module) Line Card
Troubleshooting
Downloads: This chapterpdf (PDF - 630.0KB) The complete bookPDF (PDF - 9.24MB) | Feedback

Troubleshooting

Table Of Contents

Troubleshooting

General Issues

File System

CDSM

Vault, Streamer, and ISV

Log Files

Linux Log Files

CServer Log Files

RTSP Log Files

Server Configuration Files

Description of the .arroyorc Settings

Description of the setupfile Settings

Identifying the Software Versions or Releases

Linux OS Version

CDS-Related Releases

Using ifstats to Monitor Traffic

Kernel Crash

Disk Drive Issues

CDSM GUI Disk Monitor Page Reports a Disk Warning

Memory Issues

Network

Interface Information

Startup Issues

BIOS Settings—Operating System Hangs or Goes into KDB Mode

Serial Console Port Settings

Required Services Not Starting or Running Correctly

Management and Database Issues

System Health

Cannot Access the CDSM GUI

CDSM GUI Does Not Register the Vaults and Streamers

Database Monitoring

Ingest Issues

Ingest Interface

FSI Starts but Cannot Ingest Content

FSI Does Not Start

Content Not Ingesting

Asset Ingest Manager Restart Causes Endless Loop

Bad Content

Network

Content Processing Issues

Listing Content

Content Mirroring

Verifying GOIDs

Trick-Mode Issues

Cache-Fill Issues

Rules for ISV Interoperability with Vaults and Streamers

Network

Stream Stops Playing at the Same Place or Does Not Play at All

Streaming and Playout Issues

Listing of Streams

No Streaming

Stream Not Playing

Poor Video or Audio Quality

No Video Displayed

RTSP Cannot Start

Video Stops Playing

No Video Playing on the STB

CDS Is Streaming but No Video Is Playing on the STB

Session Messaging

Log File Search Tips

Codes

Database Issues

Database Replication

CDSM GUI Does Not Report All the Ingested Content

Errors in Log File

Many Log Files

Corruption Recovery

Advanced Features and Applications

Live Multicast Ingest

Ingest with Media Scheduler

Ingest without Media Scheduler

Ingest Troubleshooting

Frequently Asked Questions

Reliability and Availability

Serviceability and Manageability

Content

Other

CDS Content Quality Guidelines

Supported Elementary Stream Types

Scrambling

Transport Bit Rate

Stream Length

Format Restrictions

Preferred Formats


Troubleshooting


This appendix presents troubleshooting procedures for the CDS by showing the symptoms, probable causes, and recommended actions for a variety of issues. The topics covered in this appendix include:

General Issues

Startup Issues

Management and Database Issues

Ingest Issues

Content Processing Issues

Cache-Fill Issues

Streaming and Playout Issues

Session Messaging

Database Issues

Advanced Features and Applications

Frequently Asked Questions

CDS Content Quality Guidelines

There are a variety of possible combinations of CDS topologies, backoffice environments, middleware, and so on. The engineers using this troubleshooting appendix are expected to know their system well enough that they can extrapolate the relevant troubleshooting guidelines. With all connectivity issues, physical integrity of cables and ports should be verified, as well as VLAN configuration if applicable.

All Linux commands described in this appendix require console access to the server, or Secure Shell (SSH) access to the server.


Caution Do not attempt to access the Linux command line unless you are familiar with the CDS, the Linux operating system, and have an understanding of the Linux command line.


Note It is important to verify at each step that the correct user account is being used. The root and isa user accounts are the only ones required to manipulate the files. The root user account uses the # symbol as a prompt. The isa user account uses the $ symbol as a prompt. We strongly recommend that you change these passwords as soon as possible by using the passwd command.


General Issues

This section describes the CDS file system, log files, configuration files, and general troubleshooting methods. This section includes the following:

File System

Log Files

Server Configuration Files

Identifying the Software Versions or Releases

Using ifstats to Monitor Traffic

Kernel Crash

Disk Drive Issues

Memory Issues

Network

File System

The CDSM file system differs from the file system on the other servers (Vault, Streamer, ISV).

CDSM

The CDSM has the following directory structures:

/arroyo/asmrpt—Contains comma-separated values (CSV) files that are created by extracting information from the database every 24 hours. These files are accessible through the Reports > Archived Data page. The asm_archiver job must be installed and added to the crontab for these files to be generated. For more information, see the "Archived Data" section on page 5-25.

/arroyo/db—Contains the database binaries, this roughly maps to the /home/isa/Berkeley directory on Streamers and Vaults.

/arroyo/db/DATADIR—Contains the database files and indexes.

/arroyo/image—The staging area for CDS software image files. This directory also includes backup directories when a software upgrade is performed on the server.

/arroyo/msa—Contains the Managed Services Architecture (MSA) logs that are created by extracting information from the database. The logs are processed by the iVAST MSA agent.

/arroyo/www—Contains the HTTP files for the CDSM GUI. The subdirectory arroyo/www/htdocs, contains the PHP files for the CDSM GUI.

/arroyo/www/modules—The link library for htdocs files.

/home/isa/—Contains configuration and log files.

Report Archiving

The CSV files are generated every 24 hours and are deleted when they are older than 30 days. The CSV files are stored in the /arroyo/asmrpt directory. For the CSV files to be generated, the report archiver needs to be installed and configured. The CSV files are accessible by going to the /arroyo/asmrpt directory, or by using an FTP client with the username "asmrpt" and the password "asmrpt."

Vault, Streamer, and ISV

The Vault, Streamer, and ISV have the following directory structures:

/arroyo/db

/arroyo/log

/arroyo/test/

/arroyo/archive

/home/isa

In addition to the above directories, the Vault, Streamer, and ISV have the following directories specific to the RTSP environment:

/home/isa/bss

/home/isa/bss/bin—Contains the VOD server binaries

/home/isa/bss/database—Contains the database files

/home/isa/bss/etc—Contains the configuration files for the binaries

/home/isa/bss/lib—Contains the shared application libraries

/home/isa/bss/log—Contains the application log files

/home/isa/bss/log/archive—Contains the archived log files

/home/isa/bss/run—Contains process ID (PID) files for running the application

/home/isa/bss/scripts—Contains general scripts

Log Files

There are three types of log files in an RTSP environment:

Linux Log Files

CServer Log Files

RTSP Log Files

The CServer log files are automatically archived and moved to the /arroyo/archive directory when the disk storage reaches a certain level. The RTSP log files are automatically archived and moved to the /home/isa/bss/log archive directory whenever the FSI or RTSP process is restarted, or the log file reaches close to 50 MB. A total of nine revisions are kept of each log file, with the eight oldest being compressed and moved to the archive directory.

Linux Log Files

The Linux operating system has the following useful log files:

/var/log/debugmessages—Syslog messages

/var/log/messages—Includes useful bootup status messages

CServer Log Files

The CDS has the following useful log files:

/arroyo/log/c2k.log.<date>—This log has information about content read issues. The date extension for the log filename has the format of yyyymmdd (for example, 20090115 is January 15, 2009). To increase the verbosity of this log file, use the following command:

# echo "6" > /proc/calypso/tunables/c2k_verbosedump

/arroyo/log/protocoltiming.log.<date>—Provides information about any network interface issues and any disk issues.

/arroyo/log/avsdb.log.<date>—Provides information about any database issues.

/arroyo/log/statsd.log.<date>—Provides system statistics information.

/arroyo/log/stresstest.log.<date>—Provides CPU uptime information.

/root/avslauncher.log.<date>—Provides information about the startup of the avslauncher module.

Other CServer log files that may be useful are the following:

/arroyo.log/controlblocktiming.log.<date>

/arroyo.log/debug.log.<date>

/arroyo.log/decommissioned.log.<date>

/arroyo.log/deleted.log.<date>

/arroyo.log/executiontiming.log.<date>

/arroyo.log/objectRepair.log.<date>

/arroyo.log/serverinfo.log.<date>

/arroyo.log/streamevent.log.<date>

/arroyo.log/systemstats.log.<date>


Note The files with the extension <date> use the format yyyymmdd. The date is the Coordinated Universal Time (UTC) date.


CServer Error Codes

CServer error codes that appear in the c2k.log.<date> file do not necessarily mean an error has occurred. An actual error has "err" listed in the entry, as opposed to "out" or "ntc." Following is a list of important CServer error and status codes:

Error Codes

5—Completion of a task.

25—Insufficient resources.

Status Codes

0—Content is okay (cnOK).

1—Stream has ended (cnEnd).

2—Stream has been paused (cnPaused).

3—Error has occurred (cnError).

4—Next element is being processed (cnNextElement).

5—Live content has resumed (cnResumeLive).

6—Next content object is being processed (cnNextContent).

7—Next iteration is being processed (cnNextIteration).

9—There has been a failover (cnFailover) .

8—Stream has been destroyed (cnDestroyed).

Protocoltiming Warning Messages

Table A-1 describes some of the warning messages that might be seen in the Protocoltiming log.

Table A-1 Protocoltiming Warning Messages 

Warning Message
Description

WARNING: Fill transmit bus hold offs

System bus is overloaded or network transmissions are not occurring fast enough and transmission of data is being delayed. The counts following these numbers may be low, this is not a concern because the delay is only 2 microseconds (ms). However, if the counts are high, this can cause stream data delivery problems.

WARNING: Fill Data Wait

Vault or Caching Node is unable to deliver data to a waiting Caching Node or Streamer because the data is not yet available. If the numbers are low, this is not a concern because the delay is only 2 ms. If the counts are high, this can cause stream data delivery problems.

WARNING Data Low

Data being streamed has less than 100 ms buffered ahead of the current stream point. Normally there should be a 2-second elasticity buffer for data being transmitted, except for a short interval when the stream first starts and the data is still "bursting" to fill the elasticity buffer. There are no problems as a result of this warning, but it is a precursor to the Fill Data Wait warning.

WARNING: Disk Refetches

Warning does not indicate any problems with streaming content, just that the disk bandwidth was not being used as efficiently as possible.

WARNING: No capacity 5 percent

Server was not accepting any new requests, which were sent to it during five percent of the preceding ten-second sample period, because it was out of capacity. Other statistics in the protocoltiming log need to be examined to determine why the server determined it was out of capacity. If Caching Nodes or Streamers are unable to find an alternate server to provide the data they need, a stream failure may occur.

WARNING: Cannot stripe disk writes

Indicates one of the following two conditions:

Some disk drives are completely full and data can no longer be written to them.

The disk system is under a full-bandwidth load such that some drives are fully committed to reading stream data and are never getting any time to write data to the drive.

The data storage pattern is not efficient when this happens because the data can no longer be spread equally across all the drives. Check other load statistics to determine why the disk system cannot stripe to some drives is useful in determining why these warnings are occurring.

WARNING: Mirror Recovery degraded - some remote vaults (0:1) are inaccessible

The configured mirroring has not occurred because 1 or more required Vaults are down, or a partner Vault is up but configured to be in a different Vault Group.


RTSP Log Files

The Linux user isa is the owner of the application files in an RTSP environment. To change from the root use the su - isa command.

All application log files are located in /home/isa/bss/log directory. When a log file reaches around 50 MB in size, it is archived and moved to the archive directory. When the RTSP or FSI is restarted, all current log files are archived and moved to the archive directory. The latest archived log files remain uncompressed, while the rest are gzipped. The archive directory always maintains the ten latest archives of each log file. The older log files are deleted.

If the RTSP application or the FSI application does not start, check the following files for possible reasons:

rtsp.err

rtsp.out

fsi.err

fsi.out

These files are located in the /home/isa/bss/log/ directory.

The CDS RTSP environment has the following useful log files:

/home/isa/bss/log/fsi.log

/home/isa/bss/log/rtsp.log

/home/isa/bss/log/aim.log

/home/isa/bss/log/library.log—Logs the cache2app library information

To change the log levels while the RTSP application is not running, edit the arroyo-env.sh file.

$ cd /home/isa/bss/scripts/
$ vi arroyo-env.sh

Change the value for the ARROYO_LOG_LEVEL variable to OFF, LOW, or HIGH. The default setting is LOW.

To change the log level whether the RTSP application is running or not, use the RTSP Setup page. For more information, see the "Configuring RTSP Setup" section on page 3-75.

Log File—rtsp.out

The rtsp.out log file stores the processing state and associated session ID of each thread.

The thread state logging is controlled by the THREAD_MON_FREQ in the arroyo-env.sh script in the /home/isa/bss.scripts directory. The THREAD_MON_FREQ parameter is used as a time interval to print the thread state and the corresponding value, which could range from 0 to 65535. The default value for the THREAD_MON_FREQ parameter is 30. The logging can be turned off by setting THREAD_MON_FREQ to 0.

Following is an example of the rtsp.out log file:

13 Apr 2010 23:06:36,407002|Thread named ThreadPoolThread0 is PROCESSING request with sessionid : 172452576 
13 Apr 2010 23:06:36,407113|Thread named ThreadPoolThread1 is PROCESSING request with sessionid : 242353532 
13 Apr 2010 23:06:36,407149|Thread named ThreadPoolThread2 is NOT PROCESSING any requests 
13 Apr 2010 23:06:36,407184|Thread named ThreadPoolThread3 is NOT PROCESSING any requests 
13 Apr 2010 23:06:36,407220|Thread named ThreadPoolThread4 is NOT PROCESSING any requests 
13 Apr 2010 23:06:36,407255|Thread named ThreadPoolThread5 is NOT PROCESSING any requests 

Server Configuration Files

The server configuration settings are stored in the .arroyorc file and the setupfile file. This section describes the different parameters for each file.


Note This section is informational only. All changes to the configuration files should be accomplished through the initial configuration and CDSM GUI.


Table A-2 lists the CDSM GUI ID names and maps them to the CServer names in the setupfile and .arroyorc files.

Table A-2 ID Names in the CDSM GUI and CServer Files 

CDSM GUI ID Name
CServer Files ID Name

Array ID on the Array Name page

groupid

Group ID on the Server-Level pages

groupid

Stream Group ID on the Server Setup page

arrayid

Cache Group ID on the Server Setup page

arrayid

Vault Group ID on the Server Setup page

arrayid

Stream Group ID on the Configuration Generator page

arrayid


Description of the .arroyorc Settings

This section describes the different line entries of the .arroyorc file. The .arroyorc file is located in the /home/isa directory and is created during the initial configuration procedure outlined in the Cisco ASR 9000 Series Aggregation Services Router ISM Line Card Hardware Installation Guide.

self

This number represents what type of server the CDE is:

0 = ISV (also known as SSV)

1 = Vault

2 = Streamer

3 = CDSM

groupid

All servers that are part of the same CDS system (managed by one CDSM) have the same group ID. This group ID should be unique across an enterprise. The purpose of the group ID is to allow servers in a group to recognize each other as belonging to the same group. If two server groups were on the same VLAN and they had the same group number they would conflict and cause issues. This is much more likely to be an issue in a lab environment with shared resources than an actual production deployment but this should still be managed.

serverid

Every server in the group has to have a unique ID ranging from 1 to 255. It is a good idea to use a standardized numbering solution; for example, all 1xx serverids are Streamers and all 2xx server IDs are Vaults.

vault

This parameter has the IP address of a Vault in the system. Each "vault" line represents an individual Vault. There may be multiple vault lines.

streamer

This is the IP address of a Streamer in the system. Each "streamer" line represents an individual Streamer. There may be multiple streamer lines.

controller

This is the IP address of the CDSM. There is only one controller line. This line is not needed in the file for the CDSM, but is used on Vaults and Streamers to point to the CDSM.

mirroring

This controls local mirroring, which is to say this determines the number of copies of a given piece of content that is stored locally.

partno

This allows the server to identify itself properly to the CDSM. The CDSM can then display the appropriate server graphic in the GUI and manage the appropriate number of disks, Ethernet ports, and so on.

mgmtif

The index of the management interface starting at eth0. Typically this remains 0.

ingestif

This parameter is only for Vaults. The index of the ingest interface starting at eth0. Typically this remains 0 but may have the value of 1 as well.

dbdomsock

This is the "file handle" where the applications address messages intended for the database.

dbnetport

This is the port number where the applications address messages intended for the database.

controlif

The index of the stream control interface starting at eth0. This is an optional configuration that is used when you want to separate the Setup and Control interface.

Description of the setupfile Settings

This section describes the different parameters of the setupfile file. The setupfile file is located in the /arroyo/test directory. Some values for the parameters in the setupfile file are set during the initial configuration (serverid, groupid, streamer vault), others are set by using the CDSM.


Note The localip # line entry has been deprecated. Ignore this line entry.


Required Settings

The following line entries are required in every setupfile file:

serverid #

An identifier that uniquely identifies the server within a group of servers identified by the group ID. See the "serverid" section for more information.

groupid #

An identifier that identifies the group of servers within the CDS. See the "groupid" section for more information.

streamer <0 or 1> vault <0 or 1>

To run the server as a Streamer, set streamer to 1, otherwise set streamer to 0. To run the server as a Vault, set vault to 1, otherwise set vault to 0. Setting both streamer and vault to 0 is not a valid option.

service address <ip in dot notation> setup <setup portno> control <control portno>

The service address is used to specify whether this server can assume the role of the Setup server, the Control server, or both the Setup and Control servers for the specified IP address. This parameter applies only to Streamers.

setup portno—A value of 0 means the server is not available to assume the role of the Setup server for the specified IP address. A value of 1 means to use the default port number 3300.

control portno—A value of 0 means the server is not available to assume the role of the Control server for the specified IP address. A value of 1 means to use the default port number 9000.

e1000 <index>: streaming <0 or 1> fill <0 or 1> ip <ip in dot notation> tport <transport portno> cport <cache portno> tgid <transport groupid>

The e1000 is used to configure the network interfaces for cache-fill and transport/streaming. Each "e1000" line represents an individual Ethernet port. Include one line per interface.

index—Refers to the interface index as known to the e1000 driver. In the case of servers with the Lindenhurst chipset, this matches one for one with the number for the eth# interface.

 streaming—For transport/streaming. A value of 1 means this interface is used for streaming, otherwise set streaming to 0.

fill—For cache fill. A value of 1 means this interface is used for cache fill. Otherwise, set fill to 0.

ip—Each interface requires a source IP address. This assumes Layer 3 networks only.

tport—The transport port number used as the source in transporting (streaming) packets. A value of 0 means to use the default port number 1026 (unless affected by the optional default source IP entry).

cport—The cache port number used as the source in caching (fill) packets. A value of 0 means to use the default port number 48879 (unless affected by the optional default source IP entry).

tgid—The transport group ID for this interface. The transport group ID is used in conjunction with the TransportGroupIdTable file located in the /arroyo/test directory to determine which interface to use to transport the packet. This is based on destination IP address or subnet of the packets. The default value is 0, which means this interface is available to any transport group. Any other value means the interface is dedicated to a particular transport group.

vault mirror copies <number of copies>

The Vault mirror copies is a numeric value representing the number of copies of each content to store on the Vaults.


Note The transport group ID (tgid) has been deprecated. Use the SubnetTable instead. See the "Network" section.


Optional Settings

The following line entries are optional in the setupfile file:

management eth #

Specifies the interface used for management. The default is eth0.

ingest eth #

Specifies the interface used for live ingests (FTP push or UDP capture). By default, the management interface is used. This parameter is applicable only to Vaults.

e1000 adapters: maxrate <rate in Mbps>

Controls the maximum transmit bandwidth on this interface, either for streaming, for caching, or for both. The default is 975 Mbps.

ibg adapters

The maximum transmit bandwidth of the ibg adapters. The default is 975 Mbps.

disks #

Specifies the number of hard drives (disks) installed on a server. The default is 12 disks for a Streamer, and 24 disks for a Vault. If you have a server with 12 disks, you must add this entry and specify 12 disks; otherwise, warning messages stating disks are non-operational are logged to the protocoltiming log file.

test #

Specifies the test mode of the server. The default is 4, which means to run the server in production mode.

cache_dscp #

Used to set the DSCP bits on cache-fill packets. The default value is 0.

cache_ecn #

Used to set the ECN bits on cache-fill packets. The default value is 0. This parameter should not be used.

transport_dscp #

Used to set the DSCP bits on transport/streaming packets. The default value is 0.

transport_ecn #

Used to set the ECN bits on transport/streaming packets. The default value is 0. This parameter should not be used.

trickspeedsv2 # # # # # # # #

Used to specify up to 8 speeds for generating trick objects during ingest. This parameter is applicable only to Vaults. The defaults are 5, -5, 10, -10, 32, and -32. The highest speeds are -127. An entry larger than 127 defaults to 127. A value of 0 is ignored.

ftpout if eth # max utilization mbps # max sessions #

Used to specify which interface on the Vault is used for FTP Out, the maximum bandwidth utilization for all active sessions (in Mbps), and the maximum number of simultaneous sessions allowed. By default the maximum sessions is 0, meaning that FTP Out is not allowed. The default for bandwidth utilization is 0, which means unlimited usage. The default interface selected is the management interface. For FTP Out to function properly, the entire content must exist on the Vault.

bypass_isacheck <0 or 1>

Used in the RTSP environments to bypass global object ID (GOID) checks at startup.

arrayid #

Specifies the array this server belongs to. The array ID is used in conjunction with the StreamDestinationMap file located in the /arroyo/test directory to determine which play servers are available for selection. The selection is based on the destination IP address or subnet of the packets. The default value is 0.

remote site manager <ip address> for arrayid #

Specifies the IP address of a remote site manager for a specified array. The remote site manager has a list of all the servers that it can connect to that are located at the same site (the servers are listed in the RemoteServers list on the remote site manager). When this server comes up, it contacts the remote site manager and asks for a referral for a server that is in the specified array. The remote site manager returns the IP address of a server from that array. This server sets up connections with all the referred remote servers, and once every minute, checks to see if there are connections with all remote array servers. If for some reason one is not available, this server contacts the remote site manager for another referral.


Note You must add the allow new L3 remote servers 1 directive to the setupfile for both this server and the server that gets referred. If you do not add this directive, this server cannot establish a connection with the referred remote server.


default source ip <ip in dot notation> tport <minportno> - <maxportno> cport <portno>

Used to affect source packets if no specific information is provided in the individual mandatory e1000 interface entries.

ip—The default source IP address for an interface. This value is overridden by the mandatory e1000 interface entry. This IP address is not meaningful in a Layer 3 network. However, today it must have a non-zero value for the other values to be looked at.

tport—After a stream is started, a random port within the range specified is used as the source port for transport/streaming packets of the stream (assuming no specific port was selected for tport in the mandatory e1000 interface entries).

cport—The source port to use for cache-fill packets (assuming no specific port was selected for cport in the mandatory e1000 interface entries).


Note The default source IP is useful in a Layer 2 network. For Layer 3 networks, an IP address is required for each interface, so the value in the default source IP is superseded by the individual entries for the interfaces. However, the default source IP setting specifies other defaults (transport port and cache port). If you would like to specify a range of transport ports, then the default source IP could have a value of zero.



Note The default source IP can be used in conjunction with the mandatory e1000 interface entries. For example, the default source IP can be used to specify a range for the source transport port. However, the generation of a random port does not currently work on every stream start. Therefore, it is best not use this option.


bms address <ip> <port>

The IP address and port of the backoffice.

Identifying the Software Versions or Releases

The following sections describe the commands for identifying the software versions on the server.

Linux OS Version

To identify the software version of the Linux operating system (OS) on the CDSM, enter the following command:

# cat /proc/version or "uname -a"
Linux version 2.6.18-92.el5 (brewbuilder@ls20-bc2-13.build.redhat.com) (gcc version 
4.1.2 20071124 (Red Hat 4.1.2-41)) #1 SMP Tue Apr 29 13:16:15 EDT 2008

To identify the software version of the Linux OS on the Vault, Streamer, or ISV, enter the following commands:

# cat /proc/version
Linux version 2.6.18-53.el5.kernel.2_6_18.2008.10.07.01 (arroyoqa@build-svr) (gcc 
version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Mon Nov 17 18:21:51 PST 2008
# uname -a
Linux stm74 2.6.18-53.el5.kernel.2_6_18.2008.10.07.01 #1 SMP Mon Nov 17 18:21:51 PST 
2008 i686 i686 i386 GNU/Linux

CDS-Related Releases

The RTSP CDS software is a combination of an RTSP overlay, statsd software, and the CServer code. The following sections describe how to identify the software version of each.

RTSP Environment

To identify the software version of the CDS RTSP overlay image, enter the following command:

# cat /arroyo/image/tags
linux/linux              kernel-2_6_18-2009-01-15-01
snmp/statsd              statsd-2009-01-14-03
snmp/agent.531           snmpd-2009-01-16-01
cserver                  cserver-e013-2009-01-16-06
export                   export-e013-2008-10-06-01
avsdb                    avsdb-2009-01-06-01
cache2app                r_2_0v1-isa-e008-2009-01-15-01
isa                      r_2_0v1-isa-e008-2009-01-15-01
bss                      bss-r20-2009-01-15-01
avslauncher              avslauncher-2009-01-13-01
framework                frmwk-r20-2008-10-28-02
tavsdb                   tavsdb-2009-01-14-01
ui/http/httpd-2.2.9      asm-2008-09-21-01
ui/http/php-5.2.6        asm-2009-01-13-01
ui/http/www/R25          CDSM-1-15-2009
upscripts                upgrade-2009-1-8-01
packager                 1
flav_inst                1

statsd Program

To identify the software version of the statsd program, enter the following command:

# strings /home/stats/statsd |grep Rel
STATSD Release TOP_OF_TREE  (arroyoqa@build-svr) (gcc version 4.1.2 20070626 (Red Hat 
4.1.2-14)) #1-Nstatsd-2008-11-07-02 Mon Nov 17 18:34:15 PST 2008

CSserver Code

To identify the software version of the CServer on the Streamer, Vault, or ISV, enter the following command:

# strings avs_cserver.ko |grep CServer
Average setup time spent in CServer = 
AVS CServer Release #1-Ncserver-e013-2008-11-17-05 Mon Nov 17 18:54:01 PST 2008 
ENV_ISA_SR DEBUG
AVS CServer Information ENV_ISA_SR DEBUG (arroyoqa@build-svr) (gcc version 4.1.2 
20070626 (Red Hat 4.1.2-14)) #1-Ncserver-e013-2008-11-17-05 Mon Nov 17 18:54:01 PST 
2008

To view the CServer settings, status, and version, enter the following command:

# cat /proc/calypso/status/server_settings
AVS CServer Information ENV_ISA_SR PROD (arroyoqa@build-svr) (gcc version 4.1.2 
20070626 (Red Hat 4.1.2-14))
#1-Ncserver-e013-2009-01-20-03 Tue Jan 20 17:54:28 PST 2009

Server Settings:
    Server is operational
    Cache2App is operational
    TSCs Per Second is 2333447000

Network Settings:
    Running in L3 Network Mode
    Allow Jumbo Frames
    Transport/Stream Data Payload: 1316
    Cache/Fill Data Payload: 7680
    Cache/Fill Control Maximum Packet Size: 8048

Using ifstats to Monitor Traffic

The ifstats command shows real-time traffic on each Ethernet interface on the server.

# /home/stats/ifstats
ifstats  -   11:12:22 
============================================================           
Int#    R-Mbps    X-Mbps           R-Bytes           X-Bytes
 eth0        0         0          56760511         166307653
 eth1        0         0                 0                 0
 eth2        4       457        3439241508        3497139080
 eth3        4       457        3439172148        3099124288
 eth4        4       457        3441836680        2945489644
 eth5        4       472        3443060380        2736115618
 eth6        4       471        3438423816        2613199736
 eth7        5       464        3440066492        2419935662
 eth8        4       449        3439982812        2266582156
 eth9        4       465        3443251384        2164010982
eth10        5       465        3439982136        1915437726
eth11        4       464        3438935192         397577442
eth12        5       464        3440343164         300903930
eth13        4       465        3439540716        4454799830

Kernel Crash

The kernel debugger (KDB) provides information (in the form of a core dump file) when the server processing fails. For the server to enter KDB when the server has crashed, the /proc/sys/kernel/panic parameter must be zero. If the panic parameter is non-zero, the system reboots automatically without entering KDB.

In addition to KDB, there is a kdump service. The kdump service allows you to take a kernel dump of memory. The kdump service runs automatically if the server is configured to reboot automatically after a crash (which means the panic parameter is non-zero). The kdump service stores the kernel memory dump in the /var/arroyo/crash directory. After the kernel memory is dumped, the system reboots into the normal operating system.

If the server is configured to enter KDB (which means the panic parameter is zero), the server enters KDB mode. The kdump command allows you to take a kernel memory dump while the server is in KDB mode. The kdump command reboots the server into kdump mode, takes a kernel memory dump, and reboots the server into the normal operating system.

If a server has crashed after being started automatically from the /etc/rc.local directory, you need to boot in single-user mode. To boot in single-user mode, perform the following steps:


Step 1 Reboot the server.

Step 2 When a blue screen displays a list of Linux versions, press the E key to edit the kernel entry.

Step 3 Multiple lines are displayed. Use the Up Arrow and Down Arrow keys to highlight the second line. You may need to press the E key again to edit the line. A square cursor appears at the end of the line.

Step 4 Remove the 115200 from the console parameter (for example, console=ttySO,115200n8).

Step 5 Add the word "Single" or the letter "S" to the end of the line.

Step 6 Press Enter.

Step 7 Press the B key to boot the Linux kernel into single-user mode.

Step 8 Wait for the server to finish booting up.

Step 9 Edit the /etc/rc.local file and comment out the line /arroyo/test/vault/run.

Step 10 Reboot the server.


To view the contents of the core dump file from the Linux prompt, do the following:


Step 1 Run the GNU debugger (gdb), and specify the core file and binary file.

gdb --core=<core-file> <binary-file>

The core-file parameter is the core filename and the binary-file is the binary file that produced the core file.

Step 2 After the GNU debugger has started, enter the backtrace command, bt, at the gdb prompt and press Enter.

gdb> bt

The callback stack is displayed, which shows the history of the current function calls that were made at the time of the crash.


Disk Drive Issues

The disk drive order is irrelevant when reinserting disk drives after transporting a chassis, or transferring disk drives from one chassis to another.

To view the statistics of the internal boot drive, the disk drive that contains the software, enter the df -k command.

# df -k
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/hda1             10317828   3764936   6028776  39% /
/dev/hda2             20641788   1711372  17881776   9% /arroyo
/dev/hda3              8254272     32828   7802148   1% /arroyo/db
/dev/hda6             35641880   1185880  32645480   4% /arroyo/log
none                   1681200         0   1681200   0% /dev/shm

To view the statistics of a removable SATA or SCSI disk drive, use the following commands:

# cat /proc/calypso/status/streamer/diskinfo 
Disk Info: 
  Disks(12) Op(12)
  Storage: T(804G) A(21%) U(0)
  BW: (99%) w(1.35M/s) r(0/s)
  I/O Util: w(1:0%) e(0) a(0%)
Disk[ 1][67.0G] A[20%] B[11x]
Disk[ 2][67.0G] A[20%] B[0x]
Disk[ 3][67.0G] A[21%] B[0x]
Disk[ 4][66.5G] A[22%] B[0x]
Disk[ 5][67.0G] A[20%] B[0x]
Disk[ 6][67.0G] A[21%] B[0x]
Disk[ 7][67.0G] A[20%] B[0x]
Disk[ 8][67.0G] A[20%] B[0x]
Disk[ 9][67.0G] A[21%] B[0x]
Disk[10][67.0G] A[20%] B[0x]
Disk[11][67.0G] A[20%] B[0x]
Disk[12][67.0G] A[20%] B[0x]

CDSM GUI Disk Monitor Page Reports a Disk Warning

If the CDSM GUI Disk Monitor page reports a disk warning, check the disk drive status in the /arroyo/log/protocoltiming.log.<date> log file and the /var/log/debugmessages log file.

# grep drives /arroyo/log/protocoltiming.log.11132007

WARNING: 5 disk drives are non-operational
WARNING: 5 disk drives are non-operational
...
WARNING: 5 disk drives are non-operational
WARNING: 5 disk drives are non-operational



# grep disks /var/log/debugmessages

Nov 20 19:02:44 vault219 kernel: RAMDISK driver initialized: 16 RAM disks of 16384K 
size 4096 blocksize
Nov 20 19:03:34 vault219 kernel: Waiting for 2 disks to finish initializing
Nov 20 19:03:34 vault219 kernel: Waiting for 4 disks to finish initializing
Nov 20 19:03:35 vault219 kernel: Waiting for 3 disks to finish initializing
Nov 20 19:03:36 vault219 kernel: Waiting for 2 disks to finish initializing
Nov 20 19:03:36 vault219 kernel: Waiting for 1 disks to finish initializing
Nov 20 19:03:36 vault219 kernel: Waiting for 5 disks to finish initializing
Nov 20 19:03:42 vault219 kernel: Waiting for 6 disks to finish initializing
Nov 20 19:03:42 vault219 kernel: Waiting for 5 disks to finish initializing
Nov 20 19:03:43 vault219 kernel: Waiting for 4 disks to finish initializing
Nov 20 19:03:45 vault219 kernel: Waiting for 11 disks to finish initializing
Nov 20 19:03:46 vault219 kernel: Waiting for 10 disks to finish initializing
Nov 20 19:03:46 vault219 kernel: Waiting for 9 disks to finish initializing
Nov 20 19:03:46 vault219 kernel: Waiting for 8 disks to finish initializing
Nov 20 19:03:47 vault219 kernel: Waiting for 7 disks to finish initializing
Nov 20 19:03:47 vault219 kernel: Waiting for 6 disks to finish initializing
Nov 20 19:03:48 vault219 kernel: Waiting for 5 disks to finish initializing
Nov 20 19:03:48 vault219 kernel: Waiting for 4 disks to finish initializing
Nov 20 19:03:48 vault219 kernel: Waiting for 3 disks to finish initializing
Nov 20 19:03:48 vault219 kernel: Waiting for 2 disks to finish initializing
Nov 20 19:03:48 vault219 kernel: Waiting for 1 disks to finish initializing
Nov 20 19:03:50 vault219 kernel:   Total disk space = 24.0TB on 24 disk drives (Lost 
disks = 0)


Note Sometimes on the CDE400, the bus and host resets are used to reset the SATA driver because the Linux SATA driver, sats_mv.ko, does not provide a device reset vector. If the device is reset when there are no outstanding requests, warning messages are displayed on the console. These warning messages are informational and do not indicate a failure.


Memory Issues

To slow down the CDSM bootup to see the memory counter, do the following:


Step 1 Reboot the server.

Step 2 To enter the BIOS Setup Utility, press the Delete key on your keyboard when you see the following text prompt:

Press DEL to runSetup


Note In most cases, the Delete key is used to invoke the setup screen. There are a few cases where other keys are used, such as F1, F2, and so on.


Step 3 Use the Right Arrow key to navigate to the Boot menu.

Step 4 Choose the Boot Settings configuration option (Figure A-1).

Step 5 Choose Quick Boot and set it to Disabled.

Figure A-1 BIOS Setup Utility—Boot Settings

Step 6 Press F10 to save and exit the BIOS Setup Utility.


Network

The following commands are useful for checking your network configuration and activity.

To view the ARP table, enter the following command:

# arp -a
jetsam.v.com (111.0.110.151) at 00:00:0C:07:AC:00 [ether] on eth0
cds17-m1.v.com (111.0.210.170) at 00:30:48:58:5B:A1 [ether] on eth0
cds17-v1.v.com (111.0.210.171) at 00:30:48:31:53:B2 [ether] on eth0
? (111.0.210.175) at 00:30:48:32:0A:5A [ether] on eth0
cds17-s1.v.com (111.0.210.172) at 00:04:23:D8:89:44 [ether] on eth0
cds17-s1.v.com (111.0.210.172) at 00:04:23:D8:89:44 [ether] on eth0

To view the IP routing table, enter the following command:

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
111.0.210.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
111.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
0.0.0.0         111.0.210.1     0.0.0.0         UG        0 0          0 eth0

To view the CDS subnet table, enter the following command:

# cat /arroyo/test/SubnetTable
network 111.1.13.1 netmask 255.255.255.240 gateway 111.1.13.1 transport_source_ip 0


Note In Release 2.1, the local networks and their gateways are specified in the SubnetTable file. For backward compatibility, the local subnet and gateway in the RoutingTable are still supported and are used if the SubnetTable file does not exist. The Routing Table can still be used to specify static routes.


To view the CDS routing table, enter the following command:

# cat /arroyo/test/RoutingTable 
default gateway 111.1.13.1
network 111.1.13.1 netmask 255.255.255.240 gateway 0.0.0.0

To view the CDS remote server table, enter the following command:

# cat /arroyo/test/RemoteServers
remote server
id 141
ip 111.1.9.20
ip 111.1.9.21
ip 111.1.9.22
ip 111.1.9.23
ip 111.1.9.24
end remote server

remote server
id 143
ip 111.1.9.25
ip 111.1.9.26
end remote server

remote server
id 144
ip 111.1.9.27
ip 111.1.9.28
ip 111.1.9.29
ip 111.1.9.30
end remote server

Interface Information

To view basic interface information, use the ifconfig command.

# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:04:23:D8:9A:80
          inet addr:111.0.110.41  Bcast:111.0.110.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:13946269 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11594110 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3085199261 (2942.2 Mb)  TX bytes:1317620721 (1256.5 Mb)
          Interrupt:24 Base address:0x3000 Memory:dd240000-0

For detailed interface information, view the interface information file in the /proc/net/ directory.

# cat /proc/net/PRO_LAN_Adapters/eth0.info
Description                      Intel® Gigabit Ethernet Network Connections
Part_Number                      ffffff-0ff
Driver_Name                      igb
Driver_Version                   1.2.22-CDS
PCI_Vendor                       0x8086
PCI_Device_ID                    0x10a7
PCI_Subsystem_Vendor             0x15d9
PCI_Subsystem_ID                 0x10a7
PCI_Revision_ID                  0x02
PCI_Bus                          14
PCI_Slot                         0
PCI_Bus_Type                     UNKNOWN
PCI_Bus_Speed                    UNKNOWN
PCI_Bus_Width                    UNKNOWN
IRQ                              194
System_Device_Name               eth0
Current_HWaddr                   00:30:48:C3:26:9E
Permanent_HWaddr                 00:30:48:C3:26:9E

Link                             up
Speed                            1000
Duplex                           Full
State                            up

Rx_Packets                       406625
Tx_Packets                       135553
Rx_Bytes                         41539919
Tx_Bytes                         30390314
Rx_Errors                        0
Tx_Errors                        0
Rx_Dropped                       0
Tx_Dropped                       0
Multicast                        236747
Collisions                       0
Rx_Length_Errors                 0
Rx_Over_Errors                   0
Rx_CRC_Errors                    0
Rx_Frame_Errors                  0
Rx_FIFO_Errors                   0
Rx_Missed_Errors                 0
Tx_Aborted_Errors                0
Tx_Carrier_Errors                0
Tx_FIFO_Errors                   0
Tx_Heartbeat_Errors              0
Tx_Window_Errors                 0
Tx_Abort_Late_Coll               0
Tx_Deferred_Ok                   0
Tx_Single_Coll_Ok                0
Tx_Multi_Coll_Ok                 0
Rx_Long_Length_Errors            0
Rx_Short_Length_Errors           0
Rx_Align_Errors                  0
Rx_Flow_Control_XON              0
Rx_Flow_Control_XOFF             0
Tx_Flow_Control_XON              0
Tx_Flow_Control_XOFF             0
Rx_CSum_Offload_Good             406625
Rx_CSum_Offload_Errors           0

PHY_Media_Type                   Copper
PHY_Cable_Length                 Unknown Meters (+/- 20 Meters)
PHY_Extended_10Base_T_Distance   Unknown
PHY_Cable_Polarity               Normal
PHY_Disable_Polarity_Correction  Enabled
PHY_Idle_Errors                  0
PHY_Receive_Errors               0
PHY_MDI_X_Enabled                MDI
PHY_Local_Receiver_Status        OK
PHY_Remote_Receiver_Status       OK

Startup Issues

This section includes the following topics:

BIOS Settings—Operating System Hangs or Goes into KDB Mode

Serial Console Port Settings

Required Services Not Starting or Running Correctly

BIOS Settings—Operating System Hangs or Goes into KDB Mode

When a single bit error occurs in the memory of a server, it causes the Linux OS to lock up, which puts the server into kernel debugger (KDB) mode. This is because of the BIOS Error Correcting Code (ECC) Type being set incorrectly.

To correct the ECC Error Type setting in the BIOS Setup Utility, do the following.


Step 1 During the server bootup, press the Delete key to enter the BIOS Setup Utility.

Step 2 Navigate to the Advanced menu and choose Advanced Chipset Control.

Step 3 Choose ECC Error Type and change the setting to NMI (Figure A-2).

Figure A-2 BIOS Setup Utility—Advanced Chipset Control

Step 4 Press F10 to save and exit.


Serial Console Port Settings

The CDE servers ship with the following serial console settings: 115200 baud rate, no parity, 8 data bits, and 1 stop bit (115200-N-8-1). To verify the serial console settings, check the following:

BIOS Settings—Determines the speed during the very beginning of the boot process up to and including the GRUB menu. In the BIOS Setup Utility, check that the Console Redirection in the Advanced menu is set to VT100.

/etc/boot/menu.lst—Determines the speed after the kernel is loaded.

/etc/inittab—Determines the speed after the OS is loaded. Enter the following:

$ cat /etc/inittab | grep S0
S0:2345:respawn:/sbin/agetty ttyS0 115200 vt100

Required Services Not Starting or Running Correctly

From the CDSM GUI, view the Services page for each server by clicking Monitor > Server Level > Services. For more information, see the "Services Monitor" section on page 4-32. If the required services are not started, or they are not running correctly, check that there is two-way database connectivity between the Streamers and Vaults, and the CDSM.

$ netstat -an|grep 9999
tcp        0      0 0.0.0.0:9999                0.0.0.0:*           LISTEN
tcp        0      0 172.22.97.193:9999          172.22.97.197:56998 ESTABLISHED
tcp        0      0 172.22.97.193:34743         172.22.97.197:9999  ESTABLISHED
tcp        0      0 172.22.97.193:9999          172.22.97.192:50343 ESTABLISHED
tcp        0      0 172.22.97.193:39158         172.22.97.196:9999  ESTABLISHED
tcp        0      0 172.22.97.193:46030         172.22.97.192:9999  ESTABLISHED
tcp        0      0 172.22.97.193:9999          172.22.97.196:55780 ESTABLISHED
tcp        0      0 172.22.97.193:9999          172.22.97.191:50950 ESTABLISHED
tcp        0      0 172.22.97.193:60598         172.22.97.191:9999  ESTABLISHED
tcp        0      0 172.22.97.193:9999          172.22.97.194:37543 ESTABLISHED
tcp        0      0 172.22.97.193:56376         172.22.97.194:9999  ESTABLISHED

Two connections for each Vault and Streamer should be listed with a status of "ESTABLISHED."

If the connection states do not say "ESTABLISHED," check the configuration of /home/isa/.arroyorc file to make sure the settings are correct, then restart the database.

Log in to the server as isa and start the database.

$ arroyo start avsdb

Log into the server as root and start the statsd.

$ /home/stats/statsd

Check that the RTSP listener is running on the correct port.

$ arroyo status
$ netstat -an | grep 554

Management and Database Issues

This section includes the following topics:

System Health

Cannot Access the CDSM GUI

CDSM GUI Does Not Register the Vaults and Streamers

Database Monitoring

System Health

The colored boxes on the System Health Monitor page have the following meaning:

Green—All components are operating; occasionally click each check box to verify.

Yellow—Some components are not operational.

Red—All components have failed.

Cannot Access the CDSM GUI

If you cannot access the CDSM GUI, log in as root and verify that the Apache server is running on the CDSM.

# ps -aef | grep http
root      4023     1  0 Aug09 ?        00:01:44 /arroyo/www/bin/httpd
nobody    4033  4023  0 Aug09 ?        00:00:53 /arroyo/www/bin/httpd
nobody    4034  4023  0 Aug09 ?        00:00:53 /arroyo/www/bin/httpd
nobody    4035  4023  0 Aug09 ?        00:00:53 /arroyo/www/bin/httpd
nobody    4036  4023  0 Aug09 ?        00:00:53 /arroyo/www/bin/httpd
nobody    4037  4023  0 Aug09 ?        00:00:53 /arroyo/www/bin/httpd
nobody    4085  4023  0 Aug09 ?        00:00:52 /arroyo/www/bin/httpd
nobody    4086  4023  0 Aug09 ?        00:00:53 /arroyo/www/bin/httpd
nobody    4572  4023  0 Aug10 ?        00:00:52 /arroyo/www/bin/httpd
root     11598 30692  0 16:12 pts/0    00:00:00 grep http

If the Apache server is not running, restart the server.

# /arroyo/www/bin/apachectl start

CDSM GUI Does Not Register the Vaults and Streamers

If the CDSM GUI is not able to register that the Vaults and Streamers are part of the array or CDS, do the following:


Step 1 Log in to the Vault or Streamer as root.

Step 2 Verify two-way database connectivity with the CDSM.

# netstat -an | grep 9999

Step 3 Verify statsd is running.

# ps -aef | grep statsd

Step 4 Verify the correct version and permissions for /home/stats/svrinit or svrinit_15 are being used.

# ls -l /home/stats/

Step 5 On the Vault and the Streamer, initialize the CDS host in the database by using svrinit_15. Use the following options:

Option -i for the server physical IP address (eth0)

Option -s for the subnet mask of the network

Option -h for the host name

Option -d to deregister

For example, first deregister the CDS host using the -d option, then initialize the CDS host.

# svrinit_15 -i <ip_address> -s <subnet_mask> -h <host_name> -d
# svrinit_15 -i <ip_address> -s <subnet_mask> -h <host_name>

Step 6 If you still have trouble getting the tables initialized, then log in to the CDSM GUI with an engineering access user account and add the Streamer or Vault by clicking the Maintain > Software > System Configs page and using the Add New Server option.


Database Monitoring

To monitor a stuck database thread problem, use the following command:

netstat -an | grep 9999

Two connections for each Vault and Streamer should be listed with a status of "ESTABLISHED." If two-way connection does not exist, run db_shutdown on all servers including the CDSM, then start the database using the following commands:

# su - isa
$ arroyo start avsdb

If the database is stuck and db_shutdown does not take effect, use ps -ef |grep avsdb to query the process ID (PID), then use the kill -9 {pid} command to kill the avsdb process, and lastly restart the database.

Ingest Issues

This section includes the following ingest issues:

Ingest Interface

Bad Content

Network

Ingest Interface

This section includes the following topics on troubleshooting the ingest interface:

FSI Starts but Cannot Ingest Content

FSI Does Not Start

Content Not Ingesting

Asset Ingest Manager Restart Causes Endless Loop

FSI Starts but Cannot Ingest Content

Cause 1: FSI mode selected is incorrect.

Action 1: Check the FSI mode on the server by viewing the /home/isa/bss/log/fsi.log when FSI is started.

The FSI running mode should be what you set in /home/isa/bss/scripts/arroyo-site-env.sh; for example, set "FSI_DEPLOYMENT=1."

11/05/2008|19:04:50.035191|fsid.cc|444|366702208| Initializing fsi daemon...
11/05/2008|19:04:50.035470|fsid.cc|445|366702208| main(): Thread ID: 366702208
11/05/2008|19:04:50.035520|fsid.cc|447|366702208| Initializing FSI 
configuration...
11/05/2008|19:04:50.035654|DatabaseConnection.cc|62|366702208| Successfully 
connected database socket.
11/05/2008|19:04:50.048468|fsid.cc|459|366702208| Setting FSI running mode to "1" 
as specified by commandline.

FSI Does Not Start

Cause 1: Invalid port address.

cat fsi.err | grep  FSID
[fsid.cc:588] ERROR: FSID Port of 0 is an invalid port address

Action 1: Change the FSI port to a nonzero value (default=20004).

Using the CDSM GUI, choose Configure > Server Level > FSI Setup and enter a new port number in the FSI Server Port field. See the "Configuring FSI Setup" section on page 3-80 for more information.

Action 2: Log in to the server as isa and start the FSI service.

$ arroyo start fsi

Cause 2: Cannot open cache2app reported in fsi.log and fsi.err.

Action 1: Verify that the CSserver process is running

Log in to the server as root and enter the lsmod command to check if the avs_cserver is running. If the avs_cserver is not running, enter the rpm -qa | grep cserver command to see if it is installed.

If avs_cserver is installed, enter the /arroyo/test/run command to start avs_cserver.

If avs_cserver is not installed, call Cisco Technical Support.

Content Not Ingesting

Cause 1: Cannot connect to the FTP server (for example, Lysis Content Ingest Server).

Action 1: Check the network connectivity and user access.

Check the network connectivity, access control lists (ACLs), user access privileges, and so on for the FTP server.

Action 2: Verify the manual FTP request.

Verify that a manual FTP request to the FTP server from the user isa is working and list the content to be ingested.

Cause 2: Content is not correctly named in the ADI XML file.

Action 1: View the aim.log file.

cat aim.log | grep FTPP

11/23/2007|12:00:59.383279|AIMListenThread   |0144|06151|CALLBACK 
RECVD[FSICALLBACK 2251799813686769 400 FTPProcess.cc:328: Caught exception: 
FTPProcess::run_ftp(): FTP Failed: 
ftp://username:password@ftpserver:21//6663_3800p.mpg. Failed while waiting for a 
response to the retrieve command from ftp server.]

In this example, 6663_3800p.mpg does not exist on the FTP server. Make sure the FTP server content matches the content listed in the ADI XML file.

Cause 3: The Asset Ingest Manager (AIM) is not getting a message from the FTP server.

Action 1: Do the following:

1. Identify the master Vault by looking at the /sbin/ifconfig file and locating the eth0:1 interface.

2. On the master Vault, enter the tail -f aim.log command.

3. Retry the transfer from the FTP server.

4. If there is a connectivity issue, the aim.log is not updated.

5. Restart the AIM process by entering the arroyo stop aim command followed by the arroyo start aim command.

6. If necessary, restart the FTP server.

Asset Ingest Manager Restart Causes Endless Loop

Cause 1: AIM has orphaned threads.

Action 1: Check the aim.log for endless looping message.

12/12/2007|11:47:16.677035|AVSIngestMgrISA   |0178|01024|******************* AIMd starting*******************
12/12/2007|11:47:16.678530|pTCP              |0065|01024|Host info 10.212.17.12:5555
12/12/2007|11:47:16.684145|pRunMaster        |0035|01024|Current process is the master
12/12/2007|11:47:16.684267|pDomain           |0093|01024|UNIX Domain sock path [/tmp/isadb]
12/12/2007|11:47:16.684326|pSock             |0115|01024|Connected to /tmp/isadb
12/12/2007|11:47:16.684500|pDbConfigObject   |0257|01024|Db errcode[55]
12/12/2007|11:47:16.684597|AIMCache          |0164|01024|Trying to load dynamic libs.......
12/12/2007|11:47:16.686275|AIMCache          |0179|01024|Loaded [libAVSFSI_AIM.so]
12/12/2007|11:47:16.743879|AIMCache          |0179|01024|Loaded [libAVSVerimatrix_AIM.so]
12/12/2007|11:47:16.745238|AIMCache          |0179|01024|Loaded [libAVSTotalManage_AIM.so]
12/12/2007|11:47:16.745753|AIMCache          |0182|01024|Shared lib error[libAVSNGOD_AIM.so: cannot open 
shared object file: No such file or directory]
12/12/2007|11:47:16.745911|AIMCache          |0182|01024|Shared lib error[libAVSISA_AIM.so: cannot open 
shared object file: No such file or directory]
12/12/2007|11:47:16.746031|AIMCache          |0182|01024|Shared lib error[libAVSWideVine_AIM.so: cannot open 
shared object file: No such file or directory]
........

12/12/2007|15:55:20.321800|AIMDbManager      |0021|02051|AIM config change detected.....
12/12/2007|15:55:20.322220|AIMCache          |0316|02051|DB Tag[.0.0.0:20004/][]
12/12/2007|15:55:20.322670|AIMDbManager      |0054|02051|Config change from [http://0.0.0.0:20004/][]
12/12/2007|15:55:20.323040|AIMDbManager      |0055|02051|Config change to   [.0.0.0:20004/][]
12/12/2007|15:55:20.323430|AIMCache          |0316|02051|DB Tag[omeplace.here.there:8082/totalmanage/vod][]
12/12/2007|15:55:20.323960|AIMDbManager      |0054|02051|Config change from 
[http://someplace.here.there:8082/totalmanage/vod][]
12/12/2007|15:55:20.324330|AIMDbManager      |0055|02051|Config change to   
[omeplace.here.there:8082/totalmanage/vod][]
12/12/2007|15:55:20.324700|AIMCache          |0316|02051|DB Tag[nage/vod][]
12/12/2007|15:55:20.325100|AIMDbManager      |0054|02051|Config change from [/totalmanage/vod][5001]
12/12/2007|15:55:20.325500|AIMDbManager      |0055|02051|Config change to   [nage/vod][]
12/12/2007|15:56:20.443830|pDbConfigObject   |0257|02051|Db errcode[55]
12/12/2007|15:56:20.444500|AIMDbManager      |0021|02051|AIM config change detected.....
12/12/2007|15:56:20.445230|AIMCache          |0316|02051|DB Tag[0004/][]
12/12/2007|15:56:20.445690|AIMDbManager      |0054|02051|Config change from [.0.0.0:20004/][]
12/12/2007|15:56:20.446060|AIMDbManager      |0055|02051|Config change to   [0004/][]
12/12/2007|15:56:20.446460|AIMCache          |0316|02051|DB Tag[.here.there:8082/totalmanage/vod][]
12/12/2007|15:56:20.446900|AIMDbManager      |0054|02051|Config change from 
[omeplace.here.there:8082/totalmanage/vod][]
12/12/2007|15:56:20.447270|AIMDbManager      |0055|02051|Config change to   
[.here.there:8082/totalmanage/vod][]
12/12/2007|15:56:20.448310|AIMDbManager      |0054|02051|Config change from [nage/vod][]
12/12/2007|15:56:20.448700|AIMDbManager      |0055|02051|Config change to   [][]
12/12/2007|15:57:20.564550|pDbConfigObject   |0257|02051|Db errcode[55]

Action 1: Check for orphaned AIM threads during the AIM service being stopped.

Log in as isa and stop the AIM service, verify there are no orphaned AIM threads, and restart the AIM service.

$ arroyo stop aim

$ ps -aef | grep AVSIngestMgr 
	isa       7291  7243  0 20:59 pts/0    00:00:00 grep AVSIngestMgr

$ arroyo start aim

Cause 2: The master Vault AIM is in a degraded state.

Check the aim.log on the slave Vault to verify it is in a waiting state.

12/12/2007|14:57:08.954182|AVSIngestMgrISA   |0178|01024|******* AIMd starting****
12/12/2007|14:57:08.955726|pTCP	|0065|01024|Host info 10.212.17.12:5555
12/12/2007|14:57:08.961349|pTCP	|0065|01024|Host info *:5555
12/12/2007|14:57:08.961442|pRunMaster	|0029|01024|process is not the master, 
waiting on notification

Action 1: Cause a failover between the master and slave Vaults.

On the master Vault, stop the statsd and Ingest Manager (AIM) services, thereby causing Vault failover. The AIM service on the new master Vault should immediately start processing ingest requests.

On the old master Vault, log in as root and restart the statsd service, log in as isa and restart the AIM service. This Vault becomes the new standby and the AIM process is in waiting mode.

Bad Content

Variable bit rate (VBR) encoded content is not currently supported. See the "CDS Content Quality Guidelines" section for constant bit rate (CBR) guidelines.

Network

Ensure that the network maximum transmission unit (MTU) is appropriately set. If jumbo frames are enabled on the CDS, then the network must support jumbo frames. We recommend that the network support jumbo frames even when the jumbo frame option is disabled.

If a Layer 2 network is used for CDS, then appropriate MAC addresses (ARP entries) have to be configured on the switches and routers. Ensure that the CDS Vault and Streamer interfaces are in the same VLAN. If a Layer 3 network is used for the CDS, then ensure that the corresponding default gateways are correctly configured on CDS Vaults and Streamers for the various interfaces and Stream Groups.

Ensure that the content source (catcher, FTP server, and so on) is reachable from the Vaults or ISVs, and that manual content transfer using FTP works correctly.

For more information about the status of the network interfaces, network routing tables, ARP and so on see the "Network" section.

Content Processing Issues

This section includes the following content processing issues:

Listing Content

Content Mirroring

Verifying GOIDs

Trick-Mode Issues

Listing Content

To view the actual stored content versus what the database reports, enter the following commands:

$ su - isa
$ cd /arroyo/db
$ ./AVSDBUtil <<- EOF > db_content_list
> 1
> 2
> 0
> EOF
$ cat db_content_list

Content Mirroring

To enable content mirroring locally on one Vault, do the following:


Step 1 Modify the /home/isa/.arroyorc file by adding the following line:

cserver_opts "vault local copy count 2"

Step 2 Verify that the change has propagated to the /arroyo/test/vault/setupfile file.

The line "vault local copy count 2" should be added to the setupfile file.


Alternatively, enable local mirroring using the tunables. You can also use the tunables to verify the settings.

echo 2 > /proc/calypso/tunables/vaultlocalcopycount


Note Using the echo 2 command to enable local mirroring in the tunable file only changes the local copy count temporarily. The local copy count resets to its original value on reboot. To configure the local copy count permanently for any value other than 1, edit the /arroyo/test/vault/setupfile or use the CDSM GUI.


To enable content mirroring between two Vaults, do the following:


Step 1 In the CDSM GUI, choose Configure > Server Level > Server Setup. The Server Setup page is displayed.

For more information, see the "Configuring the Servers" section on page 3-58.

Step 2 From the Server IP drop-down list, choose the IP address of the server.

Step 3 From the Vault Mirror Copies drop-down list, choose 2.

Step 4 Click Submit.

Step 5 Verify the change has propagated by looking at /arroyo/test/vault/setupfile and /arroyo/log/protocoltiming.log.<date> files.

# grep mirror /arroyo/test/setupfile 
vault mirror copies 2

# grep LocalMirror /arroyo/log/protocoltiming.log.11202007
-LocalMirror Active=0:0 comp=0% obj=0.0/s read=0b/s write=0b/s copies=1
-LocalMirror Active=0:0 comp=0% obj=0.0/s read=0b/s write=0b/s copies=1


Verifying GOIDs

You cannot verify that the global object identifiers (GOIDs) among Vaults and Streamers are correct by comparing the total number of GOIDs on each server. There are actually multiple chains of GOIDs. If you list the GOID chains you can verify that the GOIDs are correct, because listing the GOIDs provides a summary at the end of the listing that reports any issues.

To list the GOIDs, enter the following command:

echo 2 > /proc/calypso/tunables/cm_logserverinfo

To list all GOID chains, enter the following command:

echo 4 > /proc/calypso/tunables/cm_logserverinfo

The /arroyo/log/serverinfo.log.<date> log file contains information about the GOIDs.


Note There is no need to identify and delete damaged or orphaned GOIDs. CServer repairs any damaged GOIDs. Orphaned GOIDs are deleted when the server reboots.


Trick-Mode Issues

Verify the trick-mode settings in the CDSM GUI and the Vault setupfile file.

From the CDSM GUI, choose Configure > System Level > Ingest Tuning to view the trick-mode settings.

To check the trick-mode setting in the setupfile on the Vault, enter the following command:

$ grep trick /arroyo/test/setupfile 
trickspeedsv2    4 10 32 -32 -10 -4 

Check the /arroyo/log/c2k.log.<date> log file and the session message logs during playout to verify that the trick-mode files are being streamed.

Cache-Fill Issues

This section covers the following cache-fill issues:

Rules for ISV Interoperability with Vaults and Streamers

Network

Rules for ISV Interoperability with Vaults and Streamers

The following rules apply for ISVs to interoperate with Vaults and Streamers:

An ISV can cache-fill both a colocated Streamer and a dedicated remote Streamer.

An ISV at one location cannot cache-fill a Streamer associated with an ISV at another location.

Two ISVs can mirror content with each other, but an ISV and a Vault cannot mirror content with each other.

A Vault cannot cache-fill an ISV.

Network


Note For more network troubleshooting methods, see the "Network" section.


Stream Stops Playing at the Same Place or Does Not Play at All

Cause 1: Jumbo frames are not supported or configured on the cache-fill network switch.

Check 1: Search the c2k.log file and the rtsp.log file for content read errors.

==> /arroyo/log//c2k.log.01152008 <==
15-Jan-2008 20:42:33 UTC :out:c2k_p_setcontentbundle: stream 3 localStreamHandlePtr 00000000 remoteServer 00000000 
15-Jan-2008 20:42:33 UTC :out:c2k_p_setcontentbundlecontinue: stream 3 localStreamHandle 0 
15-Jan-2008 20:42:33 UTC :out:c2k_p_setdestination: stream 3 localStreamHandle 0 ip 0xe0016401 port 10000 
15-Jan-2008 20:42:41 UTC :out:igate goid d346434b982851 finished read 0 length e3 lastbytes b4 retries 0 reqlen 0/e3 
15-Jan-2008 20:42:41 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw header 
15-Jan-2008 20:42:41 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw EOF record 
15-Jan-2008 20:42:44 UTC :out:igate goid d346434b982851 finished read 0 length e3 lastbytes b4 retries 1 reqlen 0/e3 
15-Jan-2008 20:42:44 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw header 
15-Jan-2008 20:42:44 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw EOF record 
15-Jan-2008 20:42:47 UTC :out:igate goid d346434b982851 finished read 0 length e3 lastbytes b4 retries 2 reqlen 0/e3 
15-Jan-2008 20:42:47 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw header 
15-Jan-2008 20:42:47 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw EOF record

==> rtsp.log <==
01/15/2008|21:43:03.585614|MyrioSession.cc|385|Successfully sent message to IPTV STB 
01/15/2008|21:43:03.585785|StreamImpl.cc|1980|***** Stream State (after Callback receipt) **** 
01/15/2008|21:43:03.585846|StreamImpl.cc|1981|Stream Position     :  = 1152089708 
01/15/2008|21:43:03.585899|StreamImpl.cc|1982|Current Operation   :  = 0 
01/15/2008|21:43:03.585957|StreamImpl.cc|1983|Operation Time (ms) :  = 1200429783585 
01/15/2008|21:43:03.586011|StreamImpl.cc|1984|Stream State        :  = stopped 
01/15/2008|21:43:03.586062|StreamImpl.cc|1985|************************************************ 
01/15/2008|21:43:03.586191|MyrioSession.cc|375|IPTV STB Message: 
ANNOUNCE * RTSP/1.0 
CSeq: 30539779 
Session: 13762563 
x-notice: 4400 "Error Reading Content Data" event-date=20080115T204303.586Z
01/15/2008|21:43:03.586259|MyrioSession.cc|385|Successfully sent message to IPTV STB

Check 2: Ping between the two devices.

Ping between the two devices on the cache-fill VLAN using a packet size greater than 1500 bytes.

Action 1: If the ping fails, verify that jumbo frames and cache-fill interfaces are configured correctly.

Verify that jumbo frames are enabled on the switch ports for the cache-fill VLAN, and verify that the cache-fill interfaces are configured correctly on the Streamers and Vaults. See the "Configuring the Servers" section on page 3-58 for information on configuring the cache-fill interfaces.

Streaming and Playout Issues

This section includes the following streaming and playout issues:

Listing of Streams

No Streaming

Stream Not Playing

Poor Video or Audio Quality

Listing of Streams

To monitor streams based on various criteria, go to the Stream Monitor page in the CDSM GUI by clicking Monitor > System Level > Stream Monitor. For more information, see the "Monitoring Stream Objects" section on page 4-13.

No Streaming

Some common causes for streaming problems are the following:

Server is in the process of being offloaded.

QAM device has no available bandwidth.

Tuning failure because of one of the following:

Error in the ARP table

QAM device is down

Network problem

Backoffice is out of synchronization with the CDS ContentStore, resulting in content not being found.

Stream Not Playing

The rtsp.log file has the entry, "error reading content data." This means that a callback was received from the CServer with a completion code of 3.

Cause 1: A piece of the content is missing.

In this case, a user can typically stream part of the content, but at some point the stream stops and the error message is returned in the ANNOUNCE message. The content needs to be validated at the CServer level.

Action 1: Set up a stream to play to a multicast address.

If this is successful, then there is a network issue, which is either a default gateway or unreachable remote client. You can verify whether it is successful by looking at the /home/stats/ifstats file.

Action 2: If ifstats information does not detect a problem, try streaming to another multicast IP address.

Repeat streaming to a multicast address with different content and, if possible, ingest known good content. Check the protocoltiming.log.<date> for damaged GOIDs by using the following command:

tail -f protocoltiming.log.<latest date> | grep Goids

Cause 2: There is a problem reaching the destination QAM device.

The CServer returns the same completion code, so the same error is returned in the announce message. In this case, the content does not stream at all. The play request and play response are separated by about 10 to 15 seconds, instead of the typical subsecond separation. This is because of the ARP timeout process the CServer is going through to reach the destination. After stream response fails, the CServer calls back with the completion code of 3, which causes the "error reading content data" message.

Action 1: Check that the interfaces involved in the streaming are up and operating at the correct speed.

Using the CDSM GUI, choose Monitor > Server Level > NIC Monitor, choose the IP address of the server, and verify the participating interfaces are up and operating at gigabit Ethernet speeds For more information, see the "NIC Monitor" section on page 4-28.

Action 2: Set up a stream to play to a multicast address.

If this is successful, then there is a network problem, which is either a default gateway or unreachable remote client. You can verify whether it is successful by looking at the /home/stats/ifstats information.

Action 3: If streaming to a multicast address is not successful, check that the Vaults can be reached.

Check the /arroyo/log/protocoltiming.log.<date> log file for the number of reachable remote servers. Additionally, if there is a cache-fill issue, you will see a large megabit value for the re-xmit buffer.

You can also check the /arroyo/log/c2k.log.<date> log file for any unreachable Vaults.

Cause 3: The c2k.log file reports "no streamer available, out of capacity."

The protocoltiming.log file reports "remote vaults 0 caches 0." This means the Streamer and Vault have lost connection with each other.

Action 1: Check the route configured on the servers.

Use the cat /arroyo/test/RoutingTable command to verify correct route table entries.

Check the routing table using the CDSM GUI by clicking Configure > Server Level > Route Tables. For more information, see the "Configuring the Route Table" section on page 3-66.

Action 2: Use the cat /arroyo/test/RemoteServers to check if the remote server is configured correctly.

Action 3: Check the routing table on the switch or router.

Poor Video or Audio Quality

This section includes the following issues that result in poor video or audio quality:

No Video Displayed

RTSP Cannot Start

Video Stops Playing

No Video Playing on the STB

CDS Is Streaming but No Video Is Playing on the STB

No Video Displayed

When content is streamed to a client device, if there is no video picture displayed on the client device and the audio is working fine, use the following troubleshooting methods:

Verify that the source is working properly and that the original content is of good quality.

Verify that the appropriate bit rates are being sent from the server using the following command on all Streamers:

/home/stats/ifstats

Verify that the content plays locally, and on a test client device (for example, a VLC client).

Test playing the content on an alternate player with an AVC plug-in.

Verify that the CDS is configured correctly.

Check the run script in the /arroyo/test/run directory. There is a tunable set for Telenet to stream null packets when the end of the stream is reached. This should be commented out or removed in a non-Telenet environment.

The interface that you are using for real-time ingests needs to be configured for the CServer. There are a couple of settings that define the interrupt for the real-time ingest interface and ensure that a single central processing unit (CPU) is responsible for receiving the packets for the ingest. Without these settings, packets can be out of order, which can cause problems with the video picture.

To fix this, use the cat /proc/interrupts command to display the interrupts and find the interrupt value associated with the interface you are using for ingest. After you know this value, add the following lines to the /arroyo/test/run script:

echo 1 > /proc/irq/<interrupt value>/smp_affinity
echo <interrupt value> > /proc/calypso/test/bypass_disable_irq

You can enter these lines at the Linux command line as well by doing so you do not have to reboot your system for them to take effect. Any content that you have previously ingested should be considered invalid.

RTSP Cannot Start

The owner and group permission of the RTSP binary file are incorrect. To correct this, enter the following commands:

# cd /home/isa/bss/bin
# chown root:root AVSRTSPServer
# chmod u+s AVSRTSPServer
# su - isa arroyo start rtsp

Video Stops Playing

If the video stops playing after a fixed interval (for example, five minutes), check to see that the interval the video played for matches the session inactivity timeout setting in the CDSM GUI Configure > Server Level > RTSP Setup page.

If the interval matches the setting, check the /home/isa/bss/log/rtsp.log file for the STB responses within the inactivity timeout interval.

This scenario could be caused by the STB possibly not responding, or the STB response may be getting lost in the network.

No Video Playing on the STB

Check the /home/isa/bss/log/rtsp.log file on the relevant Streamer. If you see a successful setup request and response, and do not see a play request for the same session, there is some issue with the STB communication to the RTSP server. The RTSP server should also be receiving a get_parameter request as a heartbeat to keep the session alive. If this does not happen, the session is torn down after the session inactivity timeout is exceeded, and a "session timed-out" message is sent.

Following is an example of the rtsp.log file showing the setup, response, teardown, and timeout messages.

SETUP rtsp://10.212.16.18:554/?AssetId=CAN36099.mpg RTSP/1.0^M CSeq: 10^M User-Agent: OpenTV VOD 1^M Transport:
MP2T/DVBC/QAM;unicast;client=168892769.6501124101;destination=172.23.68.2;client_port=1^M
 
06/13/2007|20:23:33.548631|RTSPTCPListener.cc|298|Valid RTSP request received, port  = 554
06/13/2007|20:23:33.548788|RTSPMsgHandler.cc|671|Current session count = 0
06/13/2007|20:23:33.548868|RTSPMsgHandler.cc|283|Created session id of:  = 1179649
06/13/2007|20:23:33.548942|StreamImpl.cc|111|Stream handle set to: = 1
06/13/2007|20:23:33.549200|StreamImpl.cc|1745|Stream setup - URL = CAN36099.mpg, downstream address = -1!0.0.0.0!172.23.68.2!1
06/13/2007|20:23:33.549814|StreamImpl.cc|545|DSADDR - IP: 172.23.68.2,  port: 1, node: -1
06/13/2007|20:23:33.550150|QuativeSession.cc|782|SETUP Response: RTSP/1.0 200 OK^M CSeq: 10^M Session:1179649; 
timeout=300^M Transport: MP2T/DVBC/QAM;unicast;client=168892769.6501124101; destination=172.23.68.2;client_port=1^M 
Location:rtsp://10.212.16.18:554^M
06/13/2007|20:23:33.550227|QuativeSession.cc|794|Successfully sent SETUP response.
06/13/2007|20:23:33.550377|PersistenceConnection.cc|137|Repository operation successfully completed
06/13/2007|20:23:37.313753|RTSPMsgHandler.cc|1161|In production mode, accepting incoming SETUP requests
06/13/2007|20:24:37.325864|RTSPMsgHandler.cc|1161|In production mode, accepting incoming SETUP requests
06/13/2007|20:25:37.338193|RTSPMsgHandler.cc|1161|In production mode, accepting incoming SETUP requests
06/13/2007|20:26:37.350250|RTSPMsgHandler.cc|1161|In production mode, accepting incoming SETUP requests
06/13/2007|20:27:37.362401|RTSPMsgHandler.cc|1161|In production mode, accepting incoming SETUP requests
06/13/2007|20:28:37.374832|RTSPMsgHandler.cc|1161|In production mode, accepting incoming SETUP requests
06/13/2007|20:29:13.075941|LivenessManager.cc|213|Session has timed-out, initiating teardown process.Session: 1179649
06/13/2007|20:29:13.076127|RTSPMsgHandler.cc|671|Current session count = 1
06/13/2007|20:29:13.076319|QuativeSession.cc|665|Message: ANNOUNCE * RTSP/1.0^M CSeq: 17956865^M Session: 1179649^M 
Notice:5402 Event-Date=20070613T182913.076Z "Client Session Terminated"^M

CDS Is Streaming but No Video Is Playing on the STB

Check the /home/isa/bss/log/rtsp.log file on the relevant Streamer for any of the following message flows:

Setup request is sent from Quative, followed by setup okay response returned from CDS.

Describe request is sent from the STB, followed by describe ok response and details returned from CDS.

Play request is sent from STB, followed by play ok response sent from CDS.

Teardown request is sent from STB, followed by teardown performed by CDS.

The possible causes for the above message flows are the following:

STB is not tuning to the correct frequency or program ID, or the session resource manager (SRM) is returning incorrect information.

SRM is providing the wrong edge QAM device to the CDS. In the rtsp.log file, check the IP address in the setup RTSP header.

Edge QAM device port mapping may be wrong. Check the edge QAM device configuration.

Radio frequency (RF) is not reaching the STB.

Following is an example of the rtsp.log file showing the request and response messages.

SETUP rtsp://87.231.193.114:554/?AssetId=CAN0000095932.mpg RTSP/1.0^M CSeq: 2^M User-Agent: OpenTV VOD 1^M Transport:
MP2T/DVBC/QAM;unicast;client=170027388.4312020202;destination=172.23.77.2;client_port=16^M 

11/29/2007|16:10:56.952766|RTSPTCPListener.cc|298|Valid RTSP request received, port  = 554
11/29/2007|16:10:56.952938|RTSPMsgHandler.cc|644|Current session count = 0
11/29/2007|16:10:56.953163|RTSPMsgHandler.cc|311|Created session id of:  = 30801921
11/29/2007|16:10:56.953250|StreamImpl.cc|111|Stream handle set to: = 30801921
11/29/2007|16:10:56.953562|StreamImpl.cc|1733|Stream setup - URL = CAN0000095932.mpg, downstream address =
-1!0.0.0.0!172.23.77.2!16 11/29/2007|16:10:56.953736|StreamImpl.cc|563|DSADDR - IP: 172.23.77.2,  port: 16, node: -1
11/29/2007|16:10:56.954630|QuativeSession.cc|786|SETUP Response: RTSP/1.0 200 OK^M CSeq: 2^M Session: 30801921;timeout=300^M 
Transport: MP2T/DVBC/QAM;unicast;client=170027388.4312020202;destination=172.23.77.2;client_port=16^M Location: 
rtsp://87.231.193.114:554^M
 
11/29/2007|16:10:56.954730|QuativeSession.cc|798|Successfully sent SETUP response.
11/29/2007|16:10:56.954832|PersistenceConnection.cc|137|Repository operation successfully completed
11/29/2007|16:10:57.195223|RTSPTCPListener.cc|155|Messages: DESCRIBE * RTSP/1.0^M CSeq: 90^M Session: 30801921^M 
Accept:application/sdp^M 
 
11/29/2007|16:10:57.195306|RTSPTCPListener.cc|298|Valid RTSP request received, port  = 554
11/29/2007|16:10:57.195443|RTSPMsgHandler.cc|644|Current session count = 1
11/29/2007|16:10:57.195711|ContentManager.cc|277|Calculated content duration:  = 3814142
11/29/2007|16:10:57.195834|RTSPQuativeProtocolHandler.cc|444|Describe response
11/29/2007|16:10:57.195911|QuativeSession.cc|684|Message: RTSP/1.0 200 OK^M CSeq: 90^M Session: 30801921^M 
Content-Type:application/sdp^M Content-Length: 170^M ^M v=0^M o=- 3405337857 3405337857 IN IP4 10.236.141.67^M s=RTSP 
Session^M t=0 0^M m=video 0 udp M2T^M c=IN IP4 0.0.0.0^M i=CAN0000095932.mpg^M a=type:vod^M a=range:npt=0.0-3814.142
11/29/2007|16:10:57.195975|QuativeSession.cc|694|Successfully sent message to STB
11/29/2007|16:10:57.377758|RTSPTCPListener.cc|155|Messages: PLAY * RTSP/1.0^M CSeq: 91^M Session: 30801921^M Scale: 1^M Range:npt=0-^M
11/29/2007|16:10:57.377850|RTSPTCPListener.cc|298|Valid RTSP request received, port  = 554
11/29/2007|16:10:57.377986|RTSPMsgHandler.cc|644|Current session count = 1
11/29/2007|16:10:57.378120|StreamImpl.cc|1121|play(*, from: 0 to: 2147483647)
11/29/2007|16:10:57.378179|StreamImpl.cc|588|handleContent(), url: *, play content now: 1, no flush: 0
11/29/2007|16:10:57.378235|StreamImpl.cc|697|handlePlay(), loop content: 0, play content now: 1, no flush: 0
11/29/2007|16:10:57.385722|RTSPQuativeProtocolHandler.cc|403|Play response
11/29/2007|16:10:57.385838|QuativeSession.cc|684|Message: RTSP/1.0 200 OK^M CSeq: 91^M Session: 30801921^M Scale: 1^M 
Range: npt=0.000-^M
 
11/29/2007|16:10:57.385895|PersistenceConnection.cc|137|Repository operation successfully completed
11/29/2007|16:10:57.385971|QuativeSession.cc|694|Successfully sent message to STB
11/29/2007|16:10:57.386033|PersistenceConnection.cc|137|Repository operation successfully completed
11/29/2007|16:10:58.456207|RTSPTCPListener.cc|155|Messages:PLAY * RTSP/1.0^M CSeq: 92^M Session:30801921^M Scale: 1.0^M

11/29/2007|16:10:58.456286|RTSPTCPListener.cc|298|Valid RTSP request received, port  = 554
11/29/2007|16:10:58.456450|RTSPMsgHandler.cc|644|Current session count = 1
11/29/2007|16:10:58.456595|StreamImpl.cc|1121|play(*, from: -2147483648 to: 2147483647)
11/29/2007|16:10:58.456654|StreamImpl.cc|588|handleContent(), url: *, play content now: 1, no flush: 0
11/29/2007|16:10:58.456711|StreamImpl.cc|697|handlePlay(), loop content: 0, play content now: 1, no flush: 0
11/29/2007|16:10:58.456997|StreamListener.cc|103|Call Handle: 4 Session Id:      30801921 Completion Code: 0 Error 
Code:      31 Position:        1523 Content Index:   0 Speed:           1 Play Server:     114
11/29/2007|16:10:58.457074|ModelMsg.cc|164|Starting the processing of a model msg.
11/29/2007|16:10:58.457142|RTSPMsgHandler.cc|923|Successful transition for call handle:  = 4
11/29/2007|16:10:58.457474|PersistenceConnection.cc|137|Repository operation successfully completed
11/29/2007|16:10:58.457535|PersistenceConnection.cc|137|Repository operation successfully completed
11/29/2007|16:10:58.457602|RTSPQuativeProtocolHandler.cc|403|Play response
11/29/2007|16:10:58.457704|QuativeSession.cc|684|Message: RTSP/1.0 200 OK^M CSeq: 92^M Session: 30801921^M Scale: 1^M 
Range: npt=1.523-^M ^M

11/29/2007|16:10:58.457760|QuativeSession.cc|694|Successfully sent message to STB
11/29/2007|16:11:02.546768|RTSPTCPListener.cc|155|Messages: TEARDOWN * RTSP/1.0^M CSeq: 93^M Session: 30801921^M ^M
 
11/29/2007|16:11:02.546866|RTSPTCPListener.cc|298|Valid RTSP request received, port  = 554
11/29/2007|16:11:02.547022|RTSPMsgHandler.cc|644|Current session count = 1
11/29/2007|16:11:02.547222|QuativeSession.cc|665|Message:ANNOUNCE * RTSP/1.0^M CSeq:2^M Session:30801921^M Notice: 2405 
Event-Date=20071129T151102.547Z "Session torn down: Requested by STB"^M

Session Messaging

This section includes the following topics on session messaging:

Log File Search Tips

Codes

Log File Search Tips

To search for general RTSP errors, use the following commands:

grep "RTSP/1.0 4" rtsp.log
grep "RTSP/1.0 5" rtsp.log

To search for content read errors, use the following command:

grep "Error Reading Content Data" rtsp.log

To search for stopped sessions, use the following command:

grep "stream_session: stopped" rtsp.log

To search for SETUP requests, use the following command:

grep "SETUP rtsp" rtsp.log

Use the CSeq header value in the rtsp.log file to match RTSP requests to responses.

Use the Session header value in the rtsp.log file to trace a single RTSP session from setup to teardown.

Codes

The ANNOUNCE method is a mechanism for RTSP servers to signal RTSP clients about start of stream or end of RTSP session events. An ANNOUNCE request must include a "CSeq" header and "Notice." Following are the Notice codes:

1103 Playout Stalled (from VOD server only)

1104 Playout Resumed (from VOD server only)

1500 New Scale (from VOD server only)

2101 EOS (end of stream) (from VOD server only)

2104 BOS (beginning of stream); can happen in case of rewind or reverse play (from VOD server only)

2401 Ticket Expired; the playout has stopped (from the MS server only).

4400 Error Reading Content Data; the playout has stopped (from VOD server only); from the Streamer to the client referring to a hole in the content or a delay in getting the content

5200 Server Resources Unavailable; the playout has stopped (from MS server only)

5402 Client Session Terminated; teardown has been initiated by the server, the session is closed (from MS server only)

5403 Server Shutting Down; the playout has stopped. If from MS server, the session is closed. If from VOD server, the client must issue a teardown of the session (first configuration only).

5502 Internal Server error; the playout has stopped. The client must issue a teardown of the session.

After receiving and interpreting a request message, the recipient responds with an RTSP response message. The status code returned in the RTSP response message must be in the range 100 to 599 (per RFC 2326). If the server returns another value, the client must treat the error according to the range value:

1xx: Informational—Request received, continuing process.

2xx: Success —Action was successfully received, understood, and accepted.

3xx: Redirection—Further action must be taken to complete the request.

4xx: Client Error—Request contains bad syntax or cannot be fulfilled.

5xx: Server Error—Server failed to fulfill an apparently valid request.

Table A-3 describes the RTSP status codes.

Table A-3 RTSP Status Codes 

State
Code
Description

OK

200

return "OK"

BAD_REQUEST

400

return "Bad Request"

FORBIDDEN

403

return "Forbidden"

NOT_FOUND

404

return "Not Found"

METHOD_NOT_ALLOWED

405

return "Method Not Allowed"

NOT_ACCEPTABLE

406

return "Not Acceptable"

REQUEST_TIMEOUT

408

return "Request Time-out"

UNSUPPORTED_MEDIA_TYPE

415

return "Unsupported Media Type"

INVALID_PARAMETER

451

return "Parameter Not Understood"

NO_BANDWIDTH

453

return "Not Enough Bandwidth"

SESSION_NOT_FOUND

454

return "Session Not Found"

INVALID_METHOD

455

return "Method Not Valid in This State"

INVALID_HEADER

456

return "Header Filed Not Valid for Resource"

INVALID_RANGE

457

return "Invalid Range"

UNSUPPORTED_TRANSPORT

461

return "Unsupported transport"

DESTINATION_UNREACHABLE

462

return "Destination unreachable"

DESTINATION_REQUIRED

463

return "Destination required" (nCUBE extension)

INTERNAL_SERVER_ERROR

500

return "Internal Server Error"

NOT_IMPLEMENTED

501

return "Not Implemented"

SERVICE_UNAVAILABLE

503

return "Service Unavailable"

UNSUPPORTED_VERSION

505

return "RTSP Version not supported"

UNSUPPORTED_OPTION

551

return "Option not supported"

NGOD Extensions for Session Setup Failure (SSF)

SSF_ASSET_NOT_FOUND

771

return "Server Setup Failed - Asset Not Found"

SSF_SOP_NOT_AVAILABLE

772

return "Server Setup Failed - SOP Not Available"

SSF_UNKNOWN_SOP_GROUP

773

return "Server Setup Failed - Unknown SOP Group"

SSF_UNKNOWN_SOP_NAMES

774

return "Server Setup Failed - Unknown SOP Names"

SSF_INSUFF_VOLUME_BW

775

return "Server Setup Failed - Insufficient Volume Bandwidth"

SSF_INSUFF_NETWORK_BW

776

return "Server Setup Failed - Insufficient Network Bandwidth"

SSF_INVALID_REQUEST

777

return "Server Setup Failed - Invalid Request"

SSF_INTERNAL_ERROR

778

return "Server Setup Failed - Internal Error"


Database Issues

This section covers the following database issues and troubleshooting methods:

Database Replication

Corruption Recovery

Database Replication

This section covers the following database issues:

CDSM GUI Does Not Report All the Ingested Content

Errors in Log File

Many Log Files

CDSM GUI Does Not Report All the Ingested Content

First, verify that the package has not already expired.

Second, check for index errors in the CDSM database logs, using the following command:

$ grep index /arroyo/log/avsdb.log.20071106

11-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -30976
11-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -30976
11-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -30976
11-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -30976
11-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -30976
11-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -30976

The example output indicates that the Vault and CDSM databases are not synchronized, possibly because of the server times not being synchronized, a network connectivity issue, a server failure, or some other similar issue.

For resolution, see the "Corruption Recovery" section.

Errors in Log File

If the error "FSI_ENV:XX:YY was not found in the database" occurs repeatedly in the logs (see the following example), there could be two causes for this issue.

03/22/2007|02:23:37.041319|../database/DbAlgorithms.h|142|2051| DEBUG_V: DbAlgorithms::get(__dataset="FSI_ENV:99:102")
03/22/2007|02:23:37.041508|../database/DbAlgorithms.h|152|2051| DEBUG_V: DbAlgorithms::get(__dataset=FSI_ENV:99:102): Received 1484 bytes 
from database socket.
03/22/2007|02:23:37.041606|../database/DbAlgorithms.h|175|2051| DbAlgorithms::get(__dataset=FSI_ENV:99:102): 'FSI_ENV:99:102' was not found 
in the database.

Cause 1: The databases are not synchronized.

If you are sure you entered an FSI configuration into the CDSM and the FSI is still giving this message, the configuration did not get replicated to the server where the FSI process is running.

Cause 2: The FSI configuration not existing in the database.

The FSI process can run without an existing database configuration by using its internal defaults.

Many Log Files

If one of the following conditions exist, it indicates that there were database replication errors:

There are many log files with a filename similar to log.00000XXXX in the /home/isa/Berkeley/DATADIR directory.

Database could not be started. See the "Services Monitor" section on page 4-32 for more information.

Bidirectional connections are lost between servers. See the "Required Services Not Starting or Running Correctly" section.

The following error message is listed in the /arroyo/log/avsdb-err.log.yyyyMMdd file:

tavsdb: unable to allocate memory for mutex; resize mutex region

# tail -f avsdb-err.log.20081111
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region

The /home/isa/Berkeley/DATADIR/REPLAY.db file increases to several GB.

$ ls -ltr
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002824
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002825
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002826
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002837
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002838
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002839
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002841
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002840
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002843
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002842
-rw-r--r-- 1 isa isa 5726769152 Nov 12 15:23 REPLAY.db

For resolution, see the "Corruption Recovery" section.

Corruption Recovery


Caution Escalate to tier-three support before making any intrusive database changes.

If the CDSM database is corrupted and the Vault database is not corrupted, do the following:


Step 1 As user root, stop the CDSM database.

# /usr/bin/db_shutdown

Step 2 Confirm that the database is shut down.

# ps -ef | grep avsdb 
isa 2646 1 0 Jan09 ? 00:14:50 /arroyo/db/avsdb
root 26088 26059 0 13:23 pts/1 00:00:00 grep avsdb 

Maker sure there is no avsdb process returned. If the avsdb hangs, use the process ID (2646 in the above example) with the kill command.

# kill -9 2646

Step 3 Delete all files in the /arroyo/db/DATADIR directory.

Step 4 As user root, stop the Vault database.

# /usr/bin/db_shutdown

Step 5 Confirm that the database is shut down.

# ps -ef | grep avsdb 

Step 6 Copy all files in /arroyo/db/DATADIR directory from the Vault to the CDSM.

Step 7 As user root, restart the Vault database.

# su - isa
# ps -ef | grep avsdb

Step 8 As user root, restart the CDSM database.

# su - isa
# ps -ef | grep avsdb

Step 9 Check the configuration on the CDSM and make sure no configuration parameters were lost.


Advanced Features and Applications

This section covers the Media Scheduler feature (live multicast ingest).

Live Multicast Ingest

Live multicast ingest is available as part of the Media Scheduler feature or the Real-Time Capture feature.

Ingest with Media Scheduler

Using Media Scheduler for live multicast ingest requires the following procedures:

1. Enable live ingest by setting both the Media Scheduler and the Ingest Manager to ON in the CDSM Setup. See the "Initializing the CDS and Activating the Optional Features" section on page 2-3 for more information.

2. Use the CDSM Input Channels page to configure the input channels. See the "Configuring Input Channels" section on page 3-24 for more information.

3. Upload channel schedules by importing the electronic program guide (EPG). See the "Uploading an EPG File" section on page 7-10 for more information.

Ingest without Media Scheduler

Using Real-Time Capture for live multicast ingest requires the following procedures:

1. Enable live ingest by configuring Ingest Manager ON and setting Real-Time Capture Type to Real-Time Capture (non-Media Scheduler) in the CDSM Setup page. Activate the Ingest Manager. Because the Ingest Manager is an optional feature, an activation key is required. See the "Initializing the CDS and Activating the Optional Features" section on page 2-3 for more information.

2. Use the CDSM CallSign Setup page to configure call signs with multicast IP addresses.

Ingest Troubleshooting

If the message "ERROR: Unable to login to the ftp location," is present in the /home/isa/bss/log/aim.log file, check the FTP server configured in the Ingest Manager by using the ps -ef | grep ftp command. If the FTP service is not running, enter the service vsftpd start command to start it.

Frequently Asked Questions

Many of the frequently asked questions (FAQs) responses were based on an ISV system, but guidelines can be easily extrapolated for a Vault and Streamer. This section covers the following topics:

Reliability and Availability

Serviceability and Manageability

Content

Other

Reliability and Availability

Q. How do I enable stream resiliency?

A. Log in to the CDSM with engineering access. The CDSM Setup page is displayed. For Stream Failover Support, choose "ON" and click Submit. For more information, see the "CDSM or VVIM Setup" section on page D-3.

Q. How do I check and make sure the database is running properly?

A. After starting the database, you should see two sockets (listening and non-listening) connecting to the database on each of the remote servers on port 9999. You can check them by using the netstat -an | grep 9999 command.

For example, the following output of the netstat command shows that the server (172.22.97.194) has both the listening and non-listening sockets binding on port 9999 to echo the four remote servers (172.22.97.192, 172.22.97.193, 172.22.97.195 and 172.22.97.191).

# netstat -an|grep 9999 
tcp 0 0 172.22.97.194:9999 172.22.97.195:48652 ESTABLISHED 
tcp 0 0 172.22.97.194:9999 172.22.97.191:42732 ESTABLISHED 
tcp 0 0 172.22.97.194:54563 172.22.97.195:9999 ESTABLISHED 
tcp 0 0 172.22.97.194:39342 172.22.97.191:9999 ESTABLISHED 
tcp 0 0 172.22.97.194:9999 172.22.97.192:40207 ESTABLISHED 
tcp 0 0 172.22.97.194:41815 172.22.97.192:9999 ESTABLISHED 
tcp 0 0 172.22.97.194:9999 172.22.97.193:33196 ESTABLISHED 
tcp 0 0 172.22.97.194:43269 172.22.97.193:9999 ESTABLISHED 
tcp 0 0 

If you can not see both listening and non-listening sockets binding on port 9999 for each of the remote servers, the database is not running properly. Check that you have the correct replication group members in your /home/isa/.arroyorc file.

Serviceability and Manageability

Q. How do I check the calypso server status?

A. Log in to the server as root and enter the cat /proc/calypso/status/server_settings command.

Q. How do I check central processing unit (CPU)?

A. Log in to the server as root and enter the cat /proc/cpuinfo command.

Q. How do I check the kernel network driver version?

A. Log in to the server as root and list the e1000.ko file to check the date and time it was created using the following command:

ls -l /lib/modules/<current running kernel name>/kernel/drivers/net/e1000/e1000.ko

The following example shows that the e1000.ko file is based on the kernel 2.5.18-53.el5.kernel.2_6_18.2009.01.08.01.

# ls -l /lib/modules/2.6.18-53.el5.kernel.2_6_18.2009.01.08.01/kernel/drivers/net/e1000/e1000.ko 
-rw-r--r-- 1 root root 2617502 Jan 8 18:13 
/lib/modules/2.6.18-53.el5.kernel.2_6_18.2009.01.08.01/kernel/drivers/net/e1000/e1000.ko 

Q. How do I stop, start, and, restart the Apache server on the CDSM?

A. Log in to the server as root and enter the following command:

# /arroyo/www/bin/apachectl stop
# /arroyo/www/bin/apachectl start
# /arroyo/www/bin/apachectl restart

Q. How do I check the Streamer static ARP table?

A. Log in to the server as root and enter the following command:

# cat /arroyo/test/ArpTable
ip 192.168.2.42 mac 000000000002
ip 192.168.2.43 mac 000000000002

Q. How do I view the ARP Table dump file?

# echo 1 > /proc/calypso/test/arp_dumpstate

Q. How do I recover the system from the kernel debugger (KDB) after a reboot?

A. If the server starts the KDB tool instead of rebooting, modify the /etc/grub.conf file as follows:

kdb=off panic=1 
kernel /boot/vmlinuz-2.4.32avs ro root=/dev/hda1 console=tty0 console=ttyS0,115200 
kdb=off panic=1

Q. What do I do if the KDB prompt is displayed when the server restarts after a failure?

A. Boot into single user-mode (see the "Kernel Crash" section).

Q. How do I identify any holes in the content?

A. Log in to the server as root and enter the following commands:

# echo 2 > /proc/calypso/tunables/cm_logserverinfo
# cat /arroyo/log/serverinfo.log.01132009

Look at the last two lines of output. If there are no holes, the last two lines should be the following:

BeingDeleted=0 HasHoles=0 CopyHoles=0 SectorHoles=0
Object Status Check Complete.

Q. How do I clear cached video blocks (data cache) on the Streamer?

A. Log in to the server as root and enter the echo 1 > /proc/calypso/test/clearcache command.

Q. How do I clear the data cache in memory?

A. Log in to the server as root and enter the echo 1 > /proc/calypso/test/clearmem command.


Note Make sure there are no streams running before you use this command. If there are streams, the data cache in memory is not cleared.


Q. How do I destroy all streams?

A. Log in to the server as root, stop the services, change to the database table directory, remove the session table, and restart the services.

# arroyo stop
# cd /home/isa/bss/database/DATADIR
# rm RTSP_SESSION.db, RTSP_ANNEX.db 
# arroyo start

All sessions are removed, and upon restarting the services, all streams that do not have an associated session are stopped.

Q. How do I delete an individual stream from the database?

A. Log in to the server as root and enter the following commands:

# su - isa 
$ cd bss/database
$ ./AVSDBUtil
	5 -> RTSP SESSIONS
	2 -> GET ALL SESSIONS
	1 -> DELETE SESSIONS
	Enter the Session Handle:

Q. How do I destroy all streams when none of the above methods work?

A. Log in to the server as root and enter the following commands:

[root@ssv3 root]# /usr/bin/db_shutdown
[root@ssv3 root]# ps -ef |grep avs

Wait for all avs processes to stop, then reboot the server.

[root@ssv3 root]# reboot

Q. How do I check the routing table and gateway?

A. Log in to the server as root and check the file /arroyo/test/RoutingTable.

# cat /arroyo/test/RoutingTable 
default gateway 192.169.131.250 
network 192.169.131.0 netmask 255.255.255.0 gateway 0.0.0.0 
default cache gateway 192.169.131.250 
local cache network 192.169.131.0 netmask 255.255.255.0 

Content

Q. How do I get information on a content stream that seems corrupted; for example, there is macroblocking, the stream stops and restarts, and so on?

A. Log in to the server as root and enter the following commands:

# echo 2 > /proc/calypso/tunables/cm_logserverinfo
# cat /arroyo/log/serverinfo.log.01132009

Check the last set of output lines to see the current content states.

Object Count=37708 LengthUnknown=0 
CouldNotRepair=0 IsDamaged=0 BeingRepaired=0 BeingCopied=0 
needCrcValidate=37708 isFragFlag=0 isFragd=0 Defrag=0 Smooth=0 
BeingFilled=0 OutOfService=0 NeedsISACheck=0 
BeingDeleted=0 HasHoles=0 CopyHoles=0 SectorHoles=0 
Object Status Check Complete. 

Q. How do I know if a content object has "holes"?

A. Log in to the server as root and view the /var/log/debugmessages. There is a message in the debug messages file about the GOID and the content holes.

Q. What do I do if content ingest is not working?

A. Log in to the server as root, enter the less /home/isa/ContentStore/server/GenericLog.log command, and review the log for anomalies.

Q. How do I delete ingests that are "stuck" in the active ingest state?

A. Log in to the server as root and enter the following commands:

# su - isa
$ ./arroyo/db/AVSDBUtil
log4cxx: No appender could be found for logger (cdscommon.db).
log4cxx: Please initialize the log4cxx system properly.
***************************************
       1: CONTENT        
       2: FSI LOG        
       3: FSI LPI        
       4: RTSP Config    
       5: RTSP SESSIONS  
       0: EXIT           

Enter [1/2/3/4/5] or 0? : 
***************************************

Choose the option 1 (CONTENT). In the next menu, choose option 1 (DELETE CONTENT). Enter the content ID of the "stuck" ingest, then choose the exit option for each menu until you are back at the Linux prompt.

Q. How do I manually ingest content from the command line?

A. Log in to the server as root and enter the following commands:

# su - isa
# cd ~/SDClient

Update the SDClient.cfg file with the local IP address.

# ./sdClient

Follow the SDClient menus.

Other

Q. How do I view the CServer code configuration file?

A. Log in to the server as root and enter the cat /arroyo/test/<server type>/setupfile command. The server type is one of the following: vault, streamer, or ssv.

# cat /arroyo/test/<server type>/setupfile
# CServer core configuration.  Changes to this file require
# a server reboot.

local 0 0 2 remote 0 0 2 fill 3 1 maxrate 900000 localip 0c0a80040
localip 0c0a80040
e1000 adapters: maxrate 965
e1000 0: streaming 1 fill 0
e1000 1: streaming 1 fill 0
e1000 3: streaming 0 fill 1

streamer 1 vault 1
serverid        64
groupid         64
maxpacketsize   1316
management      eth0
ingest          eth0
trickspeedsv2    10 0 0 0 0 0 0 0
ftpout if eth0 max utilization mbps 0 max sessions 0
fake cylindermap 1
test 4

Q. How do I know if a subsystem on a server is overloaded?

A. View the .arroyo.log.protocoltiming.log.<date> file. When you see the "COST REQUEST NO CAPACITY:" message, it means that the server is running out of capacity and it cannot accept new streaming requests.

Also, when you see a line in the /arroyo/log/c2k.log.<date> file that says the following:

01-May-2007 17:40:44 UTC :err:ServeStream::reserveStream: refused streamhandle 4 for 
goid a445c9780e7f8f due to its load 3750, current load 0

This entry typically means there are no stream ports linked. In the ten-second snapshot of the /arroyo/log/protocoltiming.log.<date> file, there is a line that shows load values for each of the major subsystems (LAN, memory, CPU, and so on). More than likely one of the subsystems is at 100, which is the subsystem that is having the problem.

Q. How do I enable debugging?

A. Log in to the server as root and enter the following commands:

# su - isa
# cd ~/StreamsDriver
# touch DEBUGGING_ON
# ~/IntegrationTest/debugging_on_off
#./stop_driver
#./run_driver

Q. How do I update the remote servers from /arroyo/test/RemoteServers?

A. Log in to the server as root and enter the following commands:

# echo 1 > /proc/calypso/test/readremoteservers

CDS Content Quality Guidelines

This section covers the following topics:

Supported Elementary Stream Types

Scrambling

Transport Bit Rate

Stream Length

Format Restrictions

Preferred Formats

Supported Elementary Stream Types

Video-only, audio-only (as well as audio streams with only a few or occasional video frames) and data-only streams are supported in addition to the customary multiplex of both audio and video.

Scrambling

The transport layer cannot be scrambled, meaning the transport header and any adaptation field must be in the clear. Streams whose Elementary Streams (ESs) are fully scrambled, including all start codes, are capable of being ingested and streamed, but are incapable of trick play.

For trick-play capability, the following cannot be scrambled:

Packetized Elementary Stream (PES) headers

Program Association Table (PAT) and Program Map Table (PMT)

Closed-caption data (if scrambled, the data is incorrectly included in tricks)

Transport Bit Rate

All transport streams are constant bit rate (CBR). Variable bit rate (VBR) is not supported. The maximum bit rate is 35 Mbps. There is no minimum bit rate. The ES video bit rate, as specified in the MPEG-2 sequence header, is ignored. The bit rates of individual ESs do not matter. The CBR for an individual ES (in particular the video) is not required. All that is required is that the aggregate transport stream be CBR.

Streams containing MPEG-2 or AVC video are expected to conform to the appropriate buffer models spelled out in ISO/IEC 13818-1 and 14496-10 .

Stream Length

All content must be at least one second in length. A content item must be under 12 hours in length or 81 GB, whichever comes first.

Format Restrictions

Following are the format restrictions for Advanced Video Coding (AVC), H.264, and MPEG-4:

Sequence Parameter Set (SPS) seq_parameter_set_id flag must be zero.

SPS pic_order_count_type flag must be zero.

SPS seq_scaling_matrix_present_flag must be zero.

SPS profile_idc flag must only be Baseline, Main, or High profile.

Preferred Formats

Using the following guidelines improves the performance of the system, the quality of the tricks, and the trick transitions.

1. All content should be encoded as a Single Program Transport Stream (SPTS) . If multiple programs must be included (for example, a Picture-in-Picture (PIP) stream), ensure that the "real" program is encoded with the lowest program number.

2. All content should follow the process ID (PID) numbering specified in the Content Encoding Profiles 2.0 Specification (MD-SP-VOD-CEP2.0-I02-070105), section 6.7.5. Regardless, the audio and video PIDs should be above 0x20.

3. All content should be preceded with a Program Association Table (PAT) and then a Program Map Table (PMT), and then a Program Clock Reference (PCR) before the first audio or video frame. Optionally, the discontinuity bit can be set.

4. All content should use the same PID for both PCR and video.

5. All content should begin with a closed Group Of Pictures (GOP) for MPEG-2 or with an Instantaneous Decoder Refresh (IDR) frame for AVC. This first frame is always accompanied by a sequence header for MPEG-2 or by an SPS for AVC.

6. To guarantee relatively smooth looking trick modes, the minimum I/IDR-frame frequency should be eight per second. If the minimum trick speed is 4x or less, the I/IDR-frame frequency should be at least two per second. In no case should two I/IDR frames be more than two seconds apart.

7. Each I-frame should be preceded by a sequence header and GOP header if any exist for an MPEG-2 video. Each I/IDR frame should be preceded by an SPS and Picture Parameter Set (PPS) for H.264 video.

8. Avoid mixing frame data from multiple video frames in the same transport packet. Specifically, no data belonging to the prior frame exists following the Packetized Elementary Stream (PES) packet header for the next frame. Breaking this rule may improve encoding efficiency slightly, but degrades the quality of the tricks on certain set-top boxes (STBs).

9. All content must be encoded as a single sequence, with no changes in horizontal or video resolutions, or changes in encoding parameters in the middle of the content.

10. The GOP size may be variable, but GOPs should generally not exceed two seconds. Using longer GOPs may improve encoding efficiency, but the quality of lower-speed tricks (3x, 4x) may suffer.

11. No more than four B-frames should be used between each pair of I-frames or P-frames.

12. There should be no continuity counter errors in the content.

13. There should be no discontinuities in the content, other than an optional one on the first PCR.

14. The accuracy requirements for PCRs, +/-five parts per million (5 ppm), as stated in ISO/IEC 13818-1, must be adhered to throughout the stream.

15. Audio and video are expected not to overflow the appropriate target buffer model specified.

16. A reasonable bit rate to use when encoding MPEG-2 standard definition (SD) video is 3.75 Mbps.

17. A reasonable bit rate to use when encoding MPEG-2 high definition (HD) video is 15 Mbps.

18. Appropriate bit rates for carriage of AVC are still being established, and while they are expected to be at least half the bit rates of MPEG-2, no specific recommendations can be offered.

19. There may be PIDs in the content that are not specified in the PMT. Such use is beyond the scope of this document.

20. All PATs and PMTs should be identical, with the same version number throughout.

21. The CDS support up to 30 Mbps MPEG-2 video encoding.

22. Content is filtered out if three occurrences of one-second synchronization lost are identified.

23. Content is filtered out if five seconds of null frames are identified.