Cisco TV CDS 2.0 ISA Software Configuration Guide
Troubleshooting
Downloads: This chapterpdf (PDF - 573.0KB) The complete bookPDF (PDF - 7.31MB) | Feedback

Troubleshooting

Table Of Contents

Troubleshooting

OpenStream Issues

CDS Server Cannot Register with OpenStream

OpenStream Reports Alert Messages

General Issues

File System

CDSM

Vault, Streamer, and ISV

Log Files

Linux Log Files

CServer Log Files

ISA Log Files

Server Configuration Files

Description of the .arroyorc Settings

Description of the setupfile Settings

Identifying the Software Versions or Releases

Linux OS Version

CDS-Related Releases

Using ifstats to Monitor Traffic

Kernel Crash

Disk Drive Issues

CDSM GUI Disk Monitor Page Reports a Disk Warning

Memory Issues

Network

No Output on the NSG

Vault Cannot Connect to FTP Server

Checking Network Configuration

Interface Information

Startup Issues

BIOS Settings—Operating System Hangs or Goes into KDB Mode

Serial Console Port Settings

Required Services Not Starting or Running Correctly

Management and Database Issues

System Health

Cannot Access the CDSM GUI

CDSM GUI Does Not Register the Vaults and Streamers

Database Monitoring

Ingest Issues

Ingest Interface

General Tips

Common Ingest Problems

CDS Is Not Registered to the Name Service

Restarting the ISA Services

Bad Content

Network

Content Processing Issues

Listing Content

Content Mirroring

Verifying GOIDs

Trick-Mode Issues

Name and Notify Services

CORBA Interface

Cache-Fill Issues

Rules for ISV Interoperability with Vaults and Streamers

Network

Stream Stops Playing at the Same Place or Does Not Play at All

Streaming and Playout Issues

Listing of Streams

No Streaming

Stream Not Playing

Tuning Failure

Restarting the ISA Services

Poor Video or Audio Quality

Session Messaging

Database Issues

Database Replication

CDSM GUI Does Not Report All the Ingested Content

Many Log Files

Corruption Recovery

Advanced Features and Applications

Live Multicast Ingest

Ingest with Media Scheduler

Ingest without Media Scheduler

Ingest Troubleshooting

Barker Stream

Frequently Asked Questions

Reliability and Availability

Serviceability and Manageability

Content

Other

CDS Content Quality Guidelines

Supported Elementary Stream Types

Scrambling

Transport Bit Rate

Stream Length

Format Restrictions

Preferred Formats


Troubleshooting


This appendix presents troubleshooting procedures for the CDS by showing the symptoms, probable causes, and recommended actions for a variety of issues. The topics covered in this appendix include:

OpenStream Issues

General Issues

Startup Issues

Management and Database Issues

Ingest Issues

Content Processing Issues

Cache-Fill Issues

Streaming and Playout Issues

Session Messaging

Database Issues

Advanced Features and Applications

Frequently Asked Questions

CDS Content Quality Guidelines

There are a variety of possible combinations of CDS topologies, backoffice environments, middleware, and so on. The engineers using this troubleshooting appendix is expected to know their system well enough that they can extrapolate the relevant troubleshooting guidelines. With all connectivity issues, physical integrity of cables and ports should be verified, as well as VLAN configuration if applicable.

All Linux commands described in this appendix require console access to the server, or Secure Shell (SSH) access to the server.


Caution Do not attempt to access the Linux command line unless you are familiar with the CDS, the Linux operating system, and have an understanding of the Linux command line.


Note It is important to verify at each step that the correct user account is being used. The root and isa user accounts are the only ones required to manipulate the files. The root user account uses the # as a prompt. The isa user account uses the $ as a prompt. We strongly recommend that you change these passwords as soon as possible using the passwd command.


OpenStream Issues

Following are the common connectivity problems with the OpenStream system:

CDS Server Cannot Register with OpenStream

OpenStream Reports Alert Messages

CDS Server Cannot Register with OpenStream

This issue could manifest in the following ways:

In the ISA Content Store Log (GenericLog.Log) file, which is viewable by logging in to the Linux operating system on the server, you do not see the following log entry:

CorbaEventSupplier.cc|158|Event publishing.........Done.

In the OpenStream GUI, go to System Link > System Monitor > Alerts > CORBA Status. The status icon is red next to at least one of the following:

ContentStoreFactory

VideoContentStoreFactory

StreamServiceFactory

In the ISA Content Store Log (GenericLog.Log) file, you see a log entry that states it could not find the ISA naming server.

Cause 1: Configuration mismatch between CDS and OpenStream.

The IP address, port, and ISA names configured on the CDS do not match those configured on the OpenStream system.

Action 1: Verify the CDS configuration.

Verify the following settings on the Streamer BMS page and Vault BMS page:

The IP address and port of the Content Svc Master are correct and match the settings in the OpenStream system.

The ContentStore name and Factory ID match the settings in the OpenStream system.

The IP address and port of the CORBA Name Service and CORBA Notify Service match those of the OpenStream system.

Cause 2: The ISA services are not running.

Action 1: Check the ISA services.

Check that the ISA services are running on the CDS server by going to the Monitor > Server Level > Services Monitor page. If they are not running, restart the services by going to the Maintain > Services > Services Restart page and restarting the ISA-OpenStream service.

OpenStream Reports Alert Messages

OpenStream reports the following alerts:

Content creation/provisioning failed alert

Content creation/provisioning failed for ContentStore

Package partial export alert

Package provision failed alert

Provision failed alert

Cause 1: CDS is unable to register with OpenStream

Action 1: See the "CDS Server Cannot Register with OpenStream" section.

Cause 2: The Content Store does not have the correct FTP port setting.

Action 1: Verify FTP server and client settings.

Verify the FTP server and client port settings are correct by going to the Configure > Array Level > Vault BMS page.

Cause 3: The package information is incorrect.

Action 1: Verify the URL, hostname, file path, filename, username, and password.

General Issues

This section describes the CDS file system, log files, configuration files, and general troubleshooting methods. This section includes the following:

File System

Log Files

Server Configuration Files

Identifying the Software Versions or Releases

Using ifstats to Monitor Traffic

Kernel Crash

Disk Drive Issues

Memory Issues

Network

File System

The CDSM file system differs from the file system on the other servers (Vault, Streamer, ISV).

CDSM

The CDSM has the following directory structures:

/arroyo/asmrpt—Contains comma-separated values (CSV) files that are created by extracting information from the database every 24 hours. These files are accessible through the Reports > Archived Data page. The asm_archiver job must be installed and added to the crontab for these files to be generated. For more information, see the "Archived Data" section on page 6-26.

/arroyo/db—Contains the database binaries, this roughly maps to the /home/isa/Berkeley directory on Streamers and Vaults.

/arroyo/db/DATADIR—Contains the database files and indexes.

/arroyo/image—The staging area for CDS software image files. This directory also includes backup directories when a software upgrade is performed on the server.

/arroyo/msa—Contains the Managed Services Architecture (MSA) logs that are created by extracting information from the database. The logs are processed by the iVAST MSA agent.

/arroyo/www—Contains the HTTP files for the CDSM GUI. The subdirectory, .../www/htdocs, contains the PHP files for the CDSM GUI.

/arroyo/www/modules—The link library for htdocs files.

/home/isa/—Contains configuration and log files.

Report Archiving

The CSV files are generated every 24 hours and are deleted when they are older than 30 days. The CSV files are stored in the /arroyo/asmrpt directory. In order for the CSV files to be generated, the report archiver needs to be installed and configured. The CSV files are accessible by going to the /arroyo/asmrpt directory, or by using an FTP client with the username "asmrpt" and the password "asmrpt."

Vault, Streamer, and ISV

The Vault, Streamer, and ISV have the following directory structures:

/arroyo/db

/arroyo/log

/arroyo/test/

/arroyo/archive

/home/isa

In addition to the above directories, the Vault, Streamer, and ISV have the following directories specific to the ISA environment.

/home/isa/Berkeley

/home/isa/ContentStore

/home/isa/IntegrationTest

/home/isa/Streaming

The log files are located in the /arroyo/log and /home/isa/* directories. A log file is automatically archived and moved to the /arroyo/archive directory when it reaches close to 100 MB in size.

Log Files

There are three types of log files in an ISA environment:

Linux Log Files

CServer Log Files

ISA Log Files

The CServer log files are automatically archived and moved to the /arroyo/archive directory when the disk storage reaches a certain level. The ISA log files are automatically archived and moved to the /home/isa/bss/log/archive directory whenever the log file reaches close to 40 MB in size. A total of nine revisions are kept of each log file, with the eight oldest being compressed and moved to the archive directory.

Linux Log Files

The Linux operating system has the following useful log files:

/var/log/debugmessages—Syslog messages

/var/log/messages—Includes useful bootup status messages

CServer Log Files

The CDS has the following useful log files:

/arroyo/log/c2k.log.<date>—This log has information about content read issues. The date extension for the log filename has the format of ddmmyyyy (for example, 01152008 is January 15, 2008). To increase the verbosity of this log file, use the following command:

# echo "6" > /proc/calypso/tunables/c2k_verbosedump

/arroyo/log/protocoltiming.log.<date>—Provides information about any network interface issues and any disk issues.

/arroyo/log/avsdb.log.<date>—Provides information about any database issues.

/arroyo/log/avsdb-err.log.<date>

/arroyo/log/statsd.log.<date>—Provides system statistics information.

/arroyo/log/stresstest.log.<date>—Provides CPU uptime information.

/root/avslauncher.log.<date>—Provides information about the startup of the avslauncher module.

Other CServer log files that may be useful are the following:

/arroyo.log/controlblocktiming.log.<date>

/arroyo.log/debug.log.<date>

/arroyo.log/decommissioned.log.<date>

/arroyo.log/deleted.log.<date>

/arroyo.log/executiontiming.log.<date>

/arroyo.log/objectRepair.log.<date>

/arroyo.log/serverinfo.log.<date>

/arroyo.log/streamevent.log.<date>

/arroyo.log/systemstats.log.<date>


Note The files with the extension <date> use the format mmddyyyy. The date is the Coordinated Universal Time (UTC) date.


CServer Error Codes

CServer error codes that appear in the c2k.log.<date> file do not necessarily mean an error has occurred. An actual error has "err" listed in the entry, as opposed to "out" or "ntc." Following is a list of important CServer error and status codes:

Error Codes

5—Completion of a task.

25—Insufficient resources.

Status Codes

0—Content is okay (cnOK).

1—Stream has ended (cnEnd).

2—Stream has been paused (cnPaused).

3—Error has occurred (cnError).

4—Next element is being processed (cnNextElement).

5—Live content has resumed (cnResumeLive).

6—Next content object is being processed (cnNextContent).

7—Next iteration is being processed (cnNextIteration).

9—There has been a failover (cnFailover) .

8—Stream has been destroyed (cnDestroyed).

ISA Log Files

The Linux user isa is the owner of the application files in an ISA environment. Logging in as isa starts the database. To change from the root login use the su - isa command.

The following log files provide information on content ingest and mirroring:

/home/isa/ContentStore/master/ContentStore.log

/home/isa/ContentStore/server/ContentStore.log

/home/isa/ContentStore/master/Cache2App.log

/home/isa/ContentStore/server/Cache2App.log

The following log files provide information on streaming:

/home/isa/Streaming/master/StreamService.log

/home/isa/Streaming/lscp_server/LSCPService.log

/home/isa/ResourceManager/server/OrphanStreams.log

The following log file provides services monitoring and registration to the NameService:

/home/isa/IntegrationTest/ns_log

Server Configuration Files

The server configuration settings are stored in the .arroyorc file and the setupfile file. This section describes the different parameters for each file.


Note This section is informational only. All changes to the configuration files should be accomplished through the initial configuration and CDSM GUI.


Description of the .arroyorc Settings

This section describes the different line entries of the .arroyorc file. The .arroyorc file is located in the /home/isa directory and is created during the initial configuration procedure outlined in the Cisco Content Delivery Engine 205/220/420 Hardware Installation Guide.

self

This number represents what type of server the CDE is:

0 = ISV (also known as SSV)

1 = Vault

2 = Streamer

3 = CDSM

groupid

All servers that are part of the same CDS system (managed by one CDSM) have the same group ID. This group ID should be unique across an enterprise. The purpose of the group ID is to allow servers in a group to recognize each other as belonging to the same group. If two server groups were on the same VLAN and they had the same group number they would conflict and cause issues. This is much more likely to be an issue in a lab environment with shared resources than an actual production deployment but this should still be managed.

serverid

Every server in the group has to have a unique ID ranging from 1 to 255. It is a good idea to use a standardized numbering solution; for example, all 1xx serverids are Streamers and all 2xx server IDs are Vaults.

vault

This parameter has the IP address of a Vault in the system. Each "vault" line represents an individual Vault. There may be multiple vault lines.

streamer

This is the IP address of a Streamer in the system. Each "streamer" line represents an individual Streamer. There may be multiple streamer lines.

controller

This is the IP address of the CDSM. There is only one controller line. This line is not needed in the file for the CDSM, but is used on Vaults and Streamers to point to the CDSM.

mirroring

This controls local mirroring, which is to say this determines the number of copies of a given piece of content that is stored locally.

partno

This allows the server to identify itself properly to the CDSM. The CDSM can then display the appropriate server graphic in the GUI and manage the appropriate number of disks, Ethernet ports, and so on.

mgmtif

The index of the management interface starting at eth0. Typically this remains 0.

ingestif

This parameter is only for Vaults. The index of the ingest interface starting at eth0. Typically this remains 0 but may have the value of 1 as well.

dbdomsock

This is the "file handle" where the applications address messages intended for the database.

dbnetport

This is the port number where the applications address messages intended for the database.

controlif

The index of the stream control interface starting at eth0. This is an optional configuration that is used when you want to separate the Setup and Control interface.

Description of the setupfile Settings

This section describes the different parameters of the setupfile file. The setupfile file is located in the /arroyo/test directory. Some values for the parameters in the setupfile file are set during the initial configuration (serverid, groupid, streamer vault), others are set by using the CDSM.


Note The localip # line entry has been deprecated. Ignore this line entry.


Required Settings

The following line entries are required in every setupfile file:

serverid #

An identifier that uniquely identifies the server within a group of servers identified by the group ID. See the "serverid" section for more information.

groupid #

An identifier that identifies the group of servers within the CDS. See the "groupid" section for more information.

streamer <0 or 1> vault <0 or 1>

To run the server as a Streamer, set streamer to 1, otherwise set streamer to 0. To run the server as a Vault, set vault to 1, otherwise set vault to 0. Setting both streamer and vault to 0 is not a valid option.

service address <ip in dot notation> setup <setup portno> control <control portno>

The service address is used to specify whether this server can assume the role of the Setup server, the Control server, or both the Setup and Control servers for the specified IP address. This parameter applies only to Streamers.

setup portno—A value of 0 means the server is not available to assume the role of the Setup server for the specified IP address. A value of 1 means to use the default port number 3300.

control portno—A value of 0 means the server is not available to assume the role of the Control server for the specified IP address. A value of 1 means to use the default port number 9000.

e1000 <index>: streaming <0 or 1> fill <0 or 1> ip <ip in dot notation> tport <transport portno> cport <cache portno> tgid <transport groupid>

The e1000 is used to configure the network interfaces for cache-fill and transport/streaming. Each "e1000" line represents an individual Ethernet port. Include one line per interface.

index—Refers to the interface index as known to the e1000 driver. In the case of servers with the Lindenhurst chipset, this matches one for one with the number for the eth# interface.

streaming—For transport/streaming. A value of 1 means this interface is used for streaming, otherwise set streaming to 0.

fill—For cache-fill. A value of 1 means this interface is used for cache-fill, otherwise set fill to 0.

ip—Each interface requires a source IP address. This assumes Layer 3 networks only.

tport—The transport port number used as the source in transporting (streaming) packets. A value of 0 means to use the default port number 1026 (unless affected by the optional default source IP entry).

cport—The cache port number used as the source in caching (fill) packets. A value of 0 means to use the default port number 48879 (unless affected by the optional default source IP entry).

tgid—The transport group ID for this interface. The transport group ID is used in conjunction with the TransportGroupIdTable file located in the /arroyo/test directory to determine which interface to use to transport the packet. This is based on packet's destination IP address or subnet. The default value is 0, which means this interface is available to any transport group. Any other value means the interface is dedicated to a particular transport group.

vault mirror copies <number of copies>

The Vault mirror copies is a numeric value representing the number of copies of each content to store on the Vaults.

Optional Settings

The following line entries are optional in the setupfile file:

management eth#

Specifies the interface used for management. The default is eth0.

ingest eth#

Specifies the interface used for live ingests (FTP push or UDP capture). By default, the management interface is used. This parameter is only applicable to Vaults.

e1000 adapters: maxrate <rate in Mbps>

Controls the maximum transmit bandwidth on this interface, either for streaming, for caching, or for both. The default is 975 Mbps.

ibg adapters

The maximum transmit bandwidth of the ibg adapters. The default is 975 Mbps.

disks #

Specifies the number of hard drives (disks) installed on a server. The default is 12 disks for a Streamer, and 24 disks for a Vault. If you have a server with 12 disks, you must add this entry and specify 12 disks; otherwise, warning messages stating disks are non-operational will be logged to the protocoltiming log file.

test #

Specifies the test mode of the server. The default is 4, which means to run the server in production mode.

cache_dscp #

Used to set the DSCP bits on cache-fill packets. The default value is 0.

cache_ecn #

Used to set the ECN bits on cache-fill packets. The default value is 0. This parameter should not be used.

transport_dscp #

Used to set the DSCP bits on transport/streaming packets. The default value is 0.

transport_ecn #

Used to set the ECN bits on transport/streaming packets. The default value is 0. This parameter should not be used.

trickspeedsv2 # # # # # # # #

Used to specify up to 8 speeds for generating trick objects during ingest. This parameter is only applicable to Vaults. The defaults are 5,-5,10,-10,32,-32. The highest speeds are -127 and 127. An entry larger than 127 defaults to 127. A value of 0 is ignored.

ftpout if eth# max utilization mbps # max sessions #

Used to specify which interface on the Vault is used for FTP out, the maximum bandwidth utilization for all active sessions (in Mbps), and the maximum number of simultaneous sessions allowed. By default the maximum sessions is 0, meaning that FTP out is not allowed. The default for bandwidth utilization is 0, which means unlimited usage. The default interface chosen is the management interface. For FTP out to function properly, the entire content must exist on the Vault.

arrayid #

Specifies the Streamer array this server belongs to. The array ID is used in conjunction with the StreamDestinationMap file located in the /arroyo/test directory to determine which play servers are available for selection. The selection is based on the packet's destination IP address or subnet. The default value is 0.

default source ip <ip in dot notation> tport <minportno> - <maxportno> cport <portno>

Used to affect source packets if no specific information is provided in the individual mandatory e1000 interface entries.

ip—The default source IP address for an interface. This value is overridden by the mandatory e1000 interface entry. This IP address is not meaningful in a Layer 3 network. However, today it must have a non-zero value for the other values to be looked at.

tport—Once a stream is started, a random port within the range specified is used as the source port for transport/streaming packets of the stream (assuming no specific port was selected for tport in the mandatory e1000 interface entries).

cport—The source port to use for cache-fill packets (assuming no specific port was selected for cport in the mandatory e1000 interface entries).


Note The default source IP is useful in a Layer 2 network. For Layer 3 networks, an IP address is required for each interface, so the value in the default source IP is superseded by the individual entries for the interfaces. However, the default source IP setting specifies other defaults (transport port and cache port). If you would like to specify a range of transport ports, then the default source IP could have a value of zero.



Note The default source IP can be used in conjunction with the mandatory e1000 interface entries. For example, the default source IP can be used to specify a range for the source transport port. However, the generation of a random port does not currently work on every stream start. Therefore, it is best not use this option.


bms address <ip> <port>

The IP address and port of the backoffice.

Identifying the Software Versions or Releases

The following sections describe the commands for identifying the software versions on the server.

Linux OS Version

To identify the software version of the Linux operating system (OS) on the CDSM, enter the following command:

# cat /proc/version or "uname -a"
Linux version 2.6.18-92.el5 (brewbuilder@ls20-bc2-13.build.redhat.com) (gcc version 
4.1.2 20071124 (Red Hat 4.1.2-41)) #1 SMP Tue Apr 29 13:16:15 EDT 2008

To identify the software version of the Linux OS on the Vault, Streamer, or ISV, enter the following commands:

# cat /proc/version
Linux version 2.6.18-53.el5.kernel.2_6_18.2008.10.07.01 (arroyoqa@build-svr) (gcc 
version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Mon Nov 17 18:21:51 PST 2008
# uname -a
Linux stm74 2.6.18-53.el5.kernel.2_6_18.2008.10.07.01 #1 SMP Mon Nov 17 18:21:51 PST 
2008 i686 i686 i386 GNU/Linux

CDS-Related Releases

The CDS software for ISA is a combination of the ISA overlay, statsd software, and the CServer code. The following sections describe how to identify the software version of each.

ISA Environment

To identify the software version of the CDS ISA overlay image, enter the following command:

# cat /home/isa/IntegrationTest/.build_tag 
** ISA Tag: r_2_0v1-isa-e008-2008-10-28-01
Build Time: Mon Nov 17 19:00:38 PST 2008

statsd Program

To identify the software version of the statsd program, enter the following command:

# strings /home/stats/statsd |grep Rel
STATSD Release TOP_OF_TREE  (arroyoqa@build-svr) (gcc version 4.1.2 20070626 (Red Hat 
4.1.2-14)) #1-Nstatsd-2008-11-07-02 Mon Nov 17 18:34:15 PST 2008

CSserver Code

To identify the software version of the CServer on the Streamer, Vault, or ISV, enter the following command:

# strings avs_cserver.ko |grep CServer
Average setup time spent in CServer = 
AVS CServer Release #1-Ncserver-e013-2008-11-17-05 Mon Nov 17 18:54:01 PST 2008 
ENV_ISA_SR DEBUG
AVS CServer Information ENV_ISA_SR DEBUG (arroyoqa@build-svr) (gcc version 4.1.2 
20070626 (Red Hat 4.1.2-14)) #1-Ncserver-e013-2008-11-17-05 Mon Nov 17 18:54:01 PST 
2008

To view the CServer settings, status, and version, enter the following command:

# cat /proc/calypso/status/server_settings
AVS CServer Information ENV_ISA_SR PROD (arroyoqa@build-svr) (gcc version 4.1.2 
20070626 (Red Hat 4.1.2-14))
#1-Ncserver-e013-2009-01-20-03 Tue Jan 20 17:54:28 PST 2009

Server Settings:
    Server is operational
    Cache2App is operational
    TSCs Per Second is 2333447000

Network Settings:
    Running in L3 Network Mode
    Allow Jumbo Frames
    Transport/Stream Data Payload: 1316
    Cache/Fill Data Payload: 7680
    Cache/Fill Control Maximum Packet Size: 8048

Using ifstats to Monitor Traffic

The ifstats command shows real-time traffic on each Ethernet interface on the server.

# /home/stats/ifstats

ifstats  -   11:12:22 
============================================================           
Int#    R-Mbps    X-Mbps           R-Bytes           X-Bytes
 eth0        0         0          56760511         166307653
 eth1        0         0                 0                 0
 eth2        4       457        3439241508        3497139080
 eth3        4       457        3439172148        3099124288
 eth4        4       457        3441836680        2945489644
 eth5        4       472        3443060380        2736115618
 eth6        4       471        3438423816        2613199736
 eth7        5       464        3440066492        2419935662
 eth8        4       449        3439982812        2266582156
 eth9        4       465        3443251384        2164010982
eth10        5       465        3439982136        1915437726
eth11        4       464        3438935192         397577442
eth12        5       464        3440343164         300903930
eth13        4       465        3439540716        4454799830

Kernel Crash

The kernel debugger (KDB) provides information (in the form of a core dump file) when the server processing fails.

If a server has crashed after being started automatically from the /etc/rc.local directory, you need to boot in single-user mode. To boot in single-user mode, perform the following steps:


Step 1 Reboot the server.

Step 2 When a blue screen displays a list of Linux versions, press the E key to edit the kernel entry.

Step 3 Multiple lines are displayed. Use the Up Arrow and Down Arrow keys to highlight the second line. You may need to press the E key again to edit the line. A square cursor appears at the end of the line.

Step 4 Remove the 115200 from the console parameter (for example, console=ttySO,115200n8).

Step 5 Add the word, "Single" or the letter "S" to the end of the line.

Step 6 Press Enter.

Step 7 Press the B key to boot the Linux kernel into single-user mode.

Step 8 Wait for the server to finish booting up.

Step 9 Edit the /etc/rc.local file and comment out the line /arroyo/test/vault/run.

Step 10 Reboot the server.


To view the contents of the core dump file from the Linux prompt, do the following:


Step 1 Run the GNU debugger (gdb), and specify the core file and binary file.

gdb --core=<core-file> <binary-file>

The core-file parameter is the core filename and the binary-file is the binary file that produced the core file.

Step 2 Once the GNU debugger has started, enter the backtrace command, bt, at the gdb prompt and press Enter.

gdb> bt

The callback stack is displayed, which shows the history of the current function calls that were made at the time of the crash.


Disk Drive Issues

When reinserting disk drives after transporting a chassis, or transferring disk drives from one chassis to another, the disk drive order is irrelevant.

To view the statistics of the internal boot drive, the disk drive that contains the software, enter the df -k command.

# df -k
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/hda1             10317828   3764936   6028776  39% /
/dev/hda2             20641788   1711372  17881776   9% /arroyo
/dev/hda3              8254272     32828   7802148   1% /arroyo/db
/dev/hda6             35641880   1185880  32645480   4% /arroyo/log
none                   1681200         0   1681200   0% /dev/shm

To view the statistics of a removable SATA or SCSI disk drive, use the following commands:

# cat /proc/calypso/status/streamer/diskinfo 
Disk Info: 
  Disks(12) Op(12)
  Storage: T(804G) A(21%) U(0)
  BW: (99%) w(1.35M/s) r(0/s)
  I/O Util: w(1:0%) e(0) a(0%)
Disk[ 1][67.0G] A[20%] B[11x]
Disk[ 2][67.0G] A[20%] B[0x]
Disk[ 3][67.0G] A[21%] B[0x]
Disk[ 4][66.5G] A[22%] B[0x]
Disk[ 5][67.0G] A[20%] B[0x]
Disk[ 6][67.0G] A[21%] B[0x]
Disk[ 7][67.0G] A[20%] B[0x]
Disk[ 8][67.0G] A[20%] B[0x]
Disk[ 9][67.0G] A[21%] B[0x]
Disk[10][67.0G] A[20%] B[0x]
Disk[11][67.0G] A[20%] B[0x]
Disk[12][67.0G] A[20%] B[0x]

CDSM GUI Disk Monitor Page Reports a Disk Warning

If the CDSM GUI Disk Monitor page reports a disk warning, check the disk drive status in the /arroyo/log/protocoltiming.log.<date> log file and the /var/log/debugmessages log file.

# grep drives /arroyo/log/protocoltiming.log.11132007

WARNING: 5 disk drives are non-operational
WARNING: 5 disk drives are non-operational
...
WARNING: 5 disk drives are non-operational
WARNING: 5 disk drives are non-operational



# grep disks /var/log/debugmessages

Nov 20 19:02:44 vault219 kernel: RAMDISK driver initialized: 16 RAM disks of 16384K 
size 4096 blocksize
Nov 20 19:03:34 vault219 kernel: Waiting for 2 disks to finish initializing
Nov 20 19:03:34 vault219 kernel: Waiting for 4 disks to finish initializing
Nov 20 19:03:35 vault219 kernel: Waiting for 3 disks to finish initializing
Nov 20 19:03:36 vault219 kernel: Waiting for 2 disks to finish initializing
Nov 20 19:03:36 vault219 kernel: Waiting for 1 disks to finish initializing
Nov 20 19:03:36 vault219 kernel: Waiting for 5 disks to finish initializing
Nov 20 19:03:42 vault219 kernel: Waiting for 6 disks to finish initializing
Nov 20 19:03:42 vault219 kernel: Waiting for 5 disks to finish initializing
Nov 20 19:03:43 vault219 kernel: Waiting for 4 disks to finish initializing
Nov 20 19:03:45 vault219 kernel: Waiting for 11 disks to finish initializing
Nov 20 19:03:46 vault219 kernel: Waiting for 10 disks to finish initializing
Nov 20 19:03:46 vault219 kernel: Waiting for 9 disks to finish initializing
Nov 20 19:03:46 vault219 kernel: Waiting for 8 disks to finish initializing
Nov 20 19:03:47 vault219 kernel: Waiting for 7 disks to finish initializing
Nov 20 19:03:47 vault219 kernel: Waiting for 6 disks to finish initializing
Nov 20 19:03:48 vault219 kernel: Waiting for 5 disks to finish initializing
Nov 20 19:03:48 vault219 kernel: Waiting for 4 disks to finish initializing
Nov 20 19:03:48 vault219 kernel: Waiting for 3 disks to finish initializing
Nov 20 19:03:48 vault219 kernel: Waiting for 2 disks to finish initializing
Nov 20 19:03:48 vault219 kernel: Waiting for 1 disks to finish initializing
Nov 20 19:03:50 vault219 kernel:   Total disk space = 24.0TB on 24 disk drives (Lost 
disks = 0)

Memory Issues

To slow down the CDSM bootup in order to see the memory counter, do the following:


Step 1 Reboot the server.

Step 2 To enter the BIOS Setup Utility, press the Delete key on your keyboard when you see the following text prompt:

Press DEL to runSetup


Note In most cases, the Delete key is used to invoke the setup screen. There are a few cases where other keys are used, such as F1, F2, and so on.


Step 3 Use the Right Arrow key to navigate to the Boot menu.

Step 4 Choose the Boot Settings configuration option (Figure A-1).

Step 5 Choose Quick Boot and set it to Disabled.

Figure A-1 BIOS Setup Utility—Boot Settings

Step 6 Press F10 to save and exit the BIOS Setup Utility.


Network

Following are the some common network problems:

No Output on the NSG

Vault Cannot Connect to FTP Server

Checking Network Configuration

No Output on the NSG

There is no output on the NSG when issuing a play command.

Cause 1: Cannot communicate with the QAM device or the QAM settings are incorrect.

Action 1: Verify the QAM configuration on the CDS.

Verify the IP address, and MAC address if applicable, of the QAM by going to the Configure > System Level > QAM Gateway page.

Vault Cannot Connect to FTP Server

In the ISA Content Store Log (GenericLog.Log) file, which is viewable by logging in to the Linux OS on the server, a log entry states it cannot connect to the FTP server.

Cause 1: Hostname is not resolving correctly.

Action 1: Verify the CDS configuration for DNS.

Verify the host table lookup by going to the Configure > System Level > Host Service page. The host table is used for DNS lookup before the DNS servers are queried.

Verify the DNS server settings by going to the Configure > System Level > DNS page.

Verify the DNS servers are reachable.

Checking Network Configuration

The following commands are useful for checking your network configuration and activity.

To view the ARP table, enter the following command:

# arp -a
jetsam.v.com (111.0.110.151) at 00:00:0C:07:AC:00 [ether] on eth0
cds17-m1.v.com (111.0.210.170) at 00:30:48:58:5B:A1 [ether] on eth0
cds17-v1.v.com (111.0.210.171) at 00:30:48:31:53:B2 [ether] on eth0
? (111.0.210.175) at 00:30:48:32:0A:5A [ether] on eth0
cds17-s1.v.com (111.0.210.172) at 00:04:23:D8:89:44 [ether] on eth0
cds17-s1.v.com (111.0.210.172) at 00:04:23:D8:89:44 [ether] on eth0

To view the IP routing table, enter the following command:

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
111.0.210.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
111.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
0.0.0.0         111.0.210.1     0.0.0.0         UG        0 0          0 eth0

To view the CDS routing table, enter the following command:

# cat /arroyo/test/RoutingTable 
default gateway 111.1.13.1
network 111.1.13.1 netmask 255.255.255.240 gateway 0.0.0.0

To view the CDS remote server table, enter the following command:

# cat /arroyo/test/RemoteServers
remote server
id 141
ip 111.1.9.20
ip 111.1.9.21
ip 111.1.9.22
ip 111.1.9.23
ip 111.1.9.24
end remote server

remote server
id 143
ip 111.1.9.25
ip 111.1.9.26
end remote server

remote server
id 144
ip 111.1.9.27
ip 111.1.9.28
ip 111.1.9.29
ip 111.1.9.30
end remote server

Interface Information

To view basic interface information, use the ifconfig command.

# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:04:23:D8:9A:80
          inet addr:111.0.110.41  Bcast:111.0.110.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:13946269 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11594110 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3085199261 (2942.2 Mb)  TX bytes:1317620721 (1256.5 Mb)
          Interrupt:24 Base address:0x3000 Memory:dd240000-0

For detailed interface information, view the interface information file in the /proc/net/ directory.

# cat /proc/net/PRO_LAN_Adapters/eth0.info
Description                      Intel® Gigabit Ethernet Network Connections
Part_Number                      ffffff-0ff
Driver_Name                      igb
Driver_Version                   1.2.22-CDS
PCI_Vendor                       0x8086
PCI_Device_ID                    0x10a7
PCI_Subsystem_Vendor             0x15d9
PCI_Subsystem_ID                 0x10a7
PCI_Revision_ID                  0x02
PCI_Bus                          14
PCI_Slot                         0
PCI_Bus_Type                     UNKNOWN
PCI_Bus_Speed                    UNKNOWN
PCI_Bus_Width                    UNKNOWN
IRQ                              194
System_Device_Name               eth0
Current_HWaddr                   00:30:48:C3:26:9E
Permanent_HWaddr                 00:30:48:C3:26:9E

Link                             up
Speed                            1000
Duplex                           Full
State                            up

Rx_Packets                       406625
Tx_Packets                       135553
Rx_Bytes                         41539919
Tx_Bytes                         30390314
Rx_Errors                        0
Tx_Errors                        0
Rx_Dropped                       0
Tx_Dropped                       0
Multicast                        236747
Collisions                       0
Rx_Length_Errors                 0
Rx_Over_Errors                   0
Rx_CRC_Errors                    0
Rx_Frame_Errors                  0
Rx_FIFO_Errors                   0
Rx_Missed_Errors                 0
Tx_Aborted_Errors                0
Tx_Carrier_Errors                0
Tx_FIFO_Errors                   0
Tx_Heartbeat_Errors              0
Tx_Window_Errors                 0
Tx_Abort_Late_Coll               0
Tx_Deferred_Ok                   0
Tx_Single_Coll_Ok                0
Tx_Multi_Coll_Ok                 0
Rx_Long_Length_Errors            0
Rx_Short_Length_Errors           0
Rx_Align_Errors                  0
Rx_Flow_Control_XON              0
Rx_Flow_Control_XOFF             0
Tx_Flow_Control_XON              0
Tx_Flow_Control_XOFF             0
Rx_CSum_Offload_Good             406625
Rx_CSum_Offload_Errors           0

PHY_Media_Type                   Copper
PHY_Cable_Length                 Unknown Meters (+/- 20 Meters)
PHY_Extended_10Base_T_Distance   Unknown
PHY_Cable_Polarity               Normal
PHY_Disable_Polarity_Correction  Enabled
PHY_Idle_Errors                  0
PHY_Receive_Errors               0
PHY_MDI_X_Enabled                MDI
PHY_Local_Receiver_Status        OK
PHY_Remote_Receiver_Status       OK

Startup Issues

This section includes the following topics:

BIOS Settings—Operating System Hangs or Goes into KDB Mode

Serial Console Port Settings

Required Services Not Starting or Running Correctly

BIOS Settings—Operating System Hangs or Goes into KDB Mode

When a single bit error occurs in the memory of a server, it causes the Linux OS to lock up, which puts the server into kernel debugger (KDB) mode. This is due to the BIOS Error Correcting Code (ECC) Type being set incorrectly.

To correct the ECC Error Type setting in the BIOS Setup Utility, do the following.


Step 1 During the server bootup, press the Delete key to enter the BIOS Setup Utility.

Step 2 Navigate to the Advanced menu and choose Advanced Chipset Control.

Step 3 Choose ECC Error Type and change the setting to NMI (Figure A-2).

Figure A-2 BIOS Setup Utility—Advanced Chipset Control

Step 4 Press F10 to save and exit.


Serial Console Port Settings

The CDE servers ship with the following serial console settings: 115200 baud rate, no parity, 8 data bits, and 1 stop bit (115200-N-8-1). To verify the serial console settings, check the following:

BIOS Settings—Determines the speed during the very beginning of the boot process up to and including the GRUB menu. In the BIOS Setup Utility, check that the Console Redirection in the Advanced menu is set to VT100.

/etc/boot/menu.lst—Determines the speed once the kernel is loaded.

/etc/inittab—Determines the speed once the OS is loaded. Enter the following:

$ cat /etc/inittab | grep S0
S0:2345:respawn:/sbin/agetty ttyS0 115200 vt100


Note The CDE100 may have the following serial console settings: 9600 baud rate, no parity, 8 data bits, and 1 stop bit (9600-N-8-1).


Required Services Not Starting or Running Correctly

From the CDSM GUI, view the Services page for each server by clicking Monitor > Server Level > Services. For more information, see the "Services Monitor" section on page 5-34. If the required services are not started, or they are not running correctly, check that there is two-way database connectivity between the Streamers and Vaults, and the CDSM.

$ netstat -an|grep 9999
tcp        0      0 0.0.0.0:9999                0.0.0.0:*           LISTEN
tcp        0      0 172.22.97.193:9999          172.22.97.197:56998 ESTABLISHED
tcp        0      0 172.22.97.193:34743         172.22.97.197:9999  ESTABLISHED
tcp        0      0 172.22.97.193:9999          172.22.97.192:50343 ESTABLISHED
tcp        0      0 172.22.97.193:39158         172.22.97.196:9999  ESTABLISHED
tcp        0      0 172.22.97.193:46030         172.22.97.192:9999  ESTABLISHED
tcp        0      0 172.22.97.193:9999          172.22.97.196:55780 ESTABLISHED
tcp        0      0 172.22.97.193:9999          172.22.97.191:50950 ESTABLISHED
tcp        0      0 172.22.97.193:60598         172.22.97.191:9999  ESTABLISHED
tcp        0      0 172.22.97.193:9999          172.22.97.194:37543 ESTABLISHED
tcp        0      0 172.22.97.193:56376         172.22.97.194:9999  ESTABLISHED

Two connections for each Vault and Streamer should be listed with a status of "ESTABLISHED."

If the connection states do not say "ESTABLISHED," check the configuration of /home/isa/.arroyorc file to make sure the settings are correct, then restart the database.

Log in to the server as isa and start the database.

$ arroyo start avsdb

Log into the server as root and start the statsd.

$ /home/stats/statsd

Check that the LSCP listener is running on the correct port.

$ arroyo status
$ netstat -an | grep 554

Management and Database Issues

This section includes the following topics:

System Health

Cannot Access the CDSM GUI

CDSM GUI Does Not Register the Vaults and Streamers

Database Monitoring

System Health

The colored boxes on the System Health Monitor page have the following meaning:

Green—All components are operating; occasionally click each check box to verify.

Yellow—Some components are not operational.

Red—All components have failed.

Cannot Access the CDSM GUI

If you cannot access the CDSM GUI, log in as root and verify that the Apache server is running on the CDSM.

# ps -aef | grep http
root      4023     1  0 Aug09 ?        00:01:44 /arroyo/www/bin/httpd
nobody    4033  4023  0 Aug09 ?        00:00:53 /arroyo/www/bin/httpd
nobody    4034  4023  0 Aug09 ?        00:00:53 /arroyo/www/bin/httpd
nobody    4035  4023  0 Aug09 ?        00:00:53 /arroyo/www/bin/httpd
nobody    4036  4023  0 Aug09 ?        00:00:53 /arroyo/www/bin/httpd
nobody    4037  4023  0 Aug09 ?        00:00:53 /arroyo/www/bin/httpd
nobody    4085  4023  0 Aug09 ?        00:00:52 /arroyo/www/bin/httpd
nobody    4086  4023  0 Aug09 ?        00:00:53 /arroyo/www/bin/httpd
nobody    4572  4023  0 Aug10 ?        00:00:52 /arroyo/www/bin/httpd
root     11598 30692  0 16:12 pts/0    00:00:00 grep http

If the Apache server is not running, restart the server.

# /arroyo/www/bin/apachectl start

CDSM GUI Does Not Register the Vaults and Streamers

If the CDSM GUI is not able to register that the Vaults and Streamers are part of the array or CDS, do the following:


Step 1 Log in to the Vault or Streamer as root.

Step 2 Verify two-way database connectivity with the CDSM.

# netstat -an | grep 9999

Step 3 Verify that statsd is running.

# ps -aef | grep statsd

Step 4 Verify the correct version and permissions for /home/stats/svrinit or svrinit_15 are being used.

# ls -l /home/stats/

Step 5 On the Vault and the Streamer, initialize the CDS host in the database by using svrinit_15. Use the following options:

Option -i for the server's physical IP address (eth0)

Option -s for the subnet mask of the network

Option -h for the hostname

Option -d to deregister

For example, first deregister the CDS host using the -d option, then initialize the CDS host.

# svrinit_15 -i <ip_address> -s <subnet_mask> -h <host_name> -d
# svrinit_15 -i <ip_address> -s <subnet_mask> -h <host_name>

Step 6 If you still have trouble getting the tables initialized, then log in to the CDSM GUI with an engineering access user account and add the Streamer or Vault by clicking the Maintain > Software > System Configs page and using the Add New Server option.


Database Monitoring

To monitor a stuck database thread problem, use the following command:

netstat -an | grep 9999

Two connections for each Vault and Streamer should be listed with a status of "ESTABLISHED." If two-way connection does not exist, run db_shutdown on all servers including the CDSM, then start the database using the following commands:

# su - isa
$ arroyo start avsdb

If the database is stuck and db_shutdown does not take effect, use ps -ef | grep avsdb to query the process ID (PID), then use the kill -9 {pid} command to kill the avsdb process, and lastly restart the database.

Ingest Issues

This section includes the following ingest issues:

Ingest Interface

Bad Content

Network

Ingest Interface

This section includes the following topics on troubleshooting the ingest interface:

General Tips

Common Ingest Problems

CDS Is Not Registered to the Name Service

Restarting the ISA Services

General Tips

Following are some general troubleshooting tips for ingest issues:

Use the ifstats command to monitor ingests.

Ping the FTP device, which is either a catcher or an FTP server.

Manually FTP to the catcher or FTP server.

Common Ingest Problems

Following are some possible causes of ingest issues:

FTP server is not reachable (server is down, routes are incorrect, network partition).

Incorrect URL for the content file.

ContentStore file system is full.

Check the /arroyo/log/protocoltiming.log.<date> log file.

Verify that the correct trickmode speeds are set.

CDS Is Not Registered to the Name Service

If the CDS is not registering with the CORBA name service, check the information from the following commands:

$ tail -f /home/isa/IntegrationTest/ns_log
$ /home/isa/IntegrationTest/show_calypso_services
$ /home/isa/IntegrationTest/list_all_content
$ /home/isa/IntegrationTest/list_all_streams

Restarting the ISA Services

If it is necessary to restart the ISA services, then enter the following commands:

$ /home/isa/IntegrationTest/stop_all
$ tail -f /home/isa/IntegrationTest/ns_log
$ /home/isa/IntegrationTest/run_isa

Bad Content

Variable bit rate (VBR) encoded content is not currently supported. Refer to the "CDS Content Quality Guidelines" section for constant bit rate (CBR) guidelines.

Network

Ensure that the network maximum transmission unit (MTU) is appropriately set. If jumbo frames are enabled on the CDS, then the network must support jumbo frames. We recommend that the network support jumbo frames even when the jumbo frame option is disabled.

If a Layer 2 network is used for CDS, then appropriate MAC addresses (ARP entries) have to be configured on the switches and routers. Ensure that the CDS Vault and Streamer interfaces are in the same VLAN. If a Layer 3 network is used for the CDS, then ensure that the corresponding default gateways are correctly configured on CDS Vaults and Streamers for the various interfaces and Stream Groups.

Ensure that the content source (catcher, FTP server, and so on) is reachable from the Vaults or ISVs, and that manual content transfer using FTP works correctly.

For more information about the status of the network interfaces, network routing tables, ARP and so on see the "Network" section.

Content Processing Issues

This section includes the following content processing issues:

Listing Content

Content Mirroring

Verifying GOIDs

Trick-Mode Issues

CORBA Interface

CORBA Interface

Listing Content

To view the actual stored content versus what the database reports, enter the following commands:

$ su - isa
$ cddb
$ ./dumpDB

The dumpDB command displays a menu that provides options for populating a list file with the selected information.

===============================
   ** DUMP (CDS) DB TABLES **
   ** Version: 2.0.0       **
===============================

 SELECT A TABLE TO DUMP TO FILE

 1: CONTENT(s)                 (ctnobj.lst)
 2: STREAM(s)                  (stmobj.lst)
 3: SERVICE GROUP(s) & TSID(s) (svcgrp.lst)
 4: QAM(s)                     (qam.lst)
 5: STREAM SVC LIST            (sslist.lst)
 6: LOAD(s)                    (load.lst)
 7: ISA ENV                    (isaenv.lst)
 8: MSA UNPROCESSED(s)         (unprocessed.lst)
 9: MSA PROCESSED(s)           (processed.lst)

 0: QUIT
===============================
 ENTER CHOICE [1-9,0]: 0
===============================

To view the list file (for example, ctnobj.1st), go to the /arroyo/db directory and use the cat command or the vi command.

Content Mirroring

To enable content mirroring locally on one Vault, do the following:


Step 1 Modify the /home/isa/.arroyorc file by adding the following line:

cserver_opts "vault local copy count 2"

Step 2 Verify that the change has propagated to the /arroyo/test/vault/setupfile file.

The line, "vault local copy count 2," should be added to the setupfile file.


Alternatively, enable local mirroring using the tunables. You can also use the tunables to verify the settings.

echo 2 > /proc/calypso/tunables/vaultlocalcopycount 


Note Using the echo 2 command to enable local mirroring in the tunable file, only changes the local copy count temporarily. The local copy count resets to its original value on reboot. To configure the local copy count permanently, for any value other than 1, edit the /arroyo/test/vault/setupfile or use the CDSM GUI.


To enable content mirroring between two Vaults, do the following:


Step 1 In the CDSM GUI, choose Configure > Server Level > Server Setup. The Server Setup page is displayed.

For more information, see the "Configuring the Servers" section on page 4-62.

Step 2 From the Server IP drop-down list, choose the IP address of the server.

Step 3 From the Vault Mirror Copies drop-down list, choose 2.

Step 4 Click Submit.

Step 5 Verify the change has propagated by looking at /arroyo/test/vault/setupfile and /arroyo/log/protocoltiming.log.<date> files.

# grep mirror /arroyo/test/setupfile 
vault mirror copies 2

# grep LocalMirror /arroyo/log/protocoltiming.log.11202007
-LocalMirror Active=0:0 comp=0% obj=0.0/s read=0b/s write=0b/s copies=1
-LocalMirror Active=0:0 comp=0% obj=0.0/s read=0b/s write=0b/s copies=1


Verifying GOIDs

You cannot verify that the global object identifiers (GOIDs) among Vaults and Streamers are correct by comparing the total number of GOIDs on each server. There are actually multiple chains of GOIDs. If you list the GOID chains you can verify that the GOIDs are correct, because listing the GOIDs provides a summary at the end of the listing that reports any issues.

To list the GOIDs, enter the following command:

echo 2 > /proc/calypso/tunables/cm_logserverinfo

To list all GOID chains, enter the following command:

echo 4 > /proc/calypso/tunables/cm_logserverinfo

The log file that contains information about the GOIDs is the /arroyo/log/serverinfo.log.<date> log file.

Trick-Mode Issues

Verify the trick-mode settings in the CDSM GUI and the Vault setupfile file.

From the CDSM GUI, choose Configure > System Level > Ingest Tuning to view the trick-mode settings.

To check the trick-mode setting in the setupfile on the Vault, enter the following command:

$ grep trick /arroyo/test/setupfile 
trickspeedsv2    4 10 32 -32 -10 -4 

Check the /arroyo/log/c2k.log.<date> log file and the session message logs during playout to verify that the trick-mode files are being streamed.

Name and Notify Services

Ensure that the Name Service IP address and port, and the Notify Service IP address and port are correct.

Verify that you are able to telnet to the IP address and port for the NameService (port 5000 or port 2000) and the IP address and port for the Notify Service (port 5005 or port 2010). The port numbers are based on whether they are on the same host (5000/5005 ports) or a different server (2000/2010 ports).

Use the following commands to verify that the ISA configuration is correct:

$ cat /home/isa/isa.cfg
$ cat /home/isa/IntegrationTest/.isa_env

CORBA Interface

Ensure that the CORBA-related parameters (including the Name and Notify services) are configured correctly. This includes the Content Service and CORBA Event Channel details such as the Content Store, Kind, Content Factory, Event Channel, Content Channel, Factory, and so on.

See the "CORBA Interface" section for more troubleshooting methods for CORBA. See the "Configuring the Streamer for BMS Connectivity" section on page 4-36 and the "Configuring the Vault for BMS Connectivity" section on page 4-40 for more information on configuring the CORBA-related parameters.

Cache-Fill Issues

This section covers the following cache-fill issues:

Rules for ISV Interoperability with Vaults and Streamers

Network

Rules for ISV Interoperability with Vaults and Streamers

The following rules apply for ISVs to interoperate with Vaults and Streamers:

An ISV can cache-fill both a colocated Streamer and a dedicated remote Streamer.

An ISV at one location cannot cache-fill a Streamer associated with an ISV at another location.

Two ISVs can mirror content with each other, but an ISV and a Vault cannot mirror content with each other.

A Vault cannot cache-fill an ISV.

Network


Note For more network troubleshooting methods, see the "Network" section.


Stream Stops Playing at the Same Place or Does Not Play at All

Cause 1: Jumbo frames are not supported or configured on the cache-fill network switch.

Check 1: Search the c2k.log file for content read errors.

==> /arroyo/log//c2k.log.01152008 <==
15-Jan-2008 20:42:33 UTC :out:c2k_p_setcontentbundle: stream 3 localStreamHandlePtr 00000000 remoteServer 00000000 
15-Jan-2008 20:42:33 UTC :out:c2k_p_setcontentbundlecontinue: stream 3 localStreamHandle 0 
15-Jan-2008 20:42:33 UTC :out:c2k_p_setdestination: stream 3 localStreamHandle 0 ip 0xe0016401 port 10000 
15-Jan-2008 20:42:41 UTC :out:igate goid d346434b982851 finished read 0 length e3 lastbytes b4 retries 0 reqlen 0/e3 
15-Jan-2008 20:42:41 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw header 
15-Jan-2008 20:42:41 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw EOF record 
15-Jan-2008 20:42:44 UTC :out:igate goid d346434b982851 finished read 0 length e3 lastbytes b4 retries 1 reqlen 0/e3 
15-Jan-2008 20:42:44 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw header 
15-Jan-2008 20:42:44 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw EOF record 
15-Jan-2008 20:42:47 UTC :out:igate goid d346434b982851 finished read 0 length e3 lastbytes b4 retries 2 reqlen 0/e3 
15-Jan-2008 20:42:47 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw header 
15-Jan-2008 20:42:47 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw EOF record

Check 2: Ping between the two devices.

Ping between the two devices on the cache-fill VLAN using a packet size greater than 1500 bytes.

Action 1: If the ping fails, verify that jumbo frames and cache-fill interfaces are configured correctly.

Verify that jumbo frames are enabled on the switch ports for the cache-fill VLAN, and verify that the cache-fill interfaces are configured correctly on the Streamers and Vaults. See the "Configuring the Servers" section on page 4-62 for information on configuring the cache-fill interfaces.

Streaming and Playout Issues

This section includes the following streaming and playout issues:

Listing of Streams

No Streaming

Poor Video or Audio Quality

Listing of Streams

To monitor streams based on various criteria, go to the Stream Monitor page in the CDSM GUI by clicking Monitor > System Level > Stream Monitor. For more information, see the "Stream Monitor" section on page 5-14.

No Streaming

Some common causes for streaming problems are the following:

Server is in the process of being offloaded.

QAM device has no available bandwidth.

Tuning failure as a result of one of the following:

Error in the ARP table

QAM device is down

Network problem

Backoffice is out of synchronization with the CDS ContentStore, resulting in content not being found.

Following are some general methods for troubleshooting stream and playout issues:

View the /arroyo/log/streamevent.log.<date> file for stream setup activity.

Check for GOIDs in the /arroyo/log/c2k.log.<date> file.

Check if the stream setup is using /home/isa/IntegrationTest/list_all_streams or /home/isa/IntegrationTest/list_a_stream.

View the /home/isa/Berkeley/dumpDB file and use the vi ctnobj.lst command to check that the Vaults are synchronized.

Check content integrity by looking at the objectStatus for damaged GOIDs.

Stream Not Playing

Cause 1: A piece of content is missing.

In this case, a user can typically stream part of the content, but at some point the stream stops and this error is returned in the ANNOUNCE message. The content needs to be validated at the CServer level.

Action 1: Set up a stream to play to a multicast address.

If this is successful, then there is a network issue, which is either a default gateway or unreachable remote client. You can verify whether it is successful by looking at the /home/stats/ifstats information.

Action 2: If ifstats information does not detect a problem, try streaming to another multicast IP address.

Repeat streaming to a multicast address with different content and, if possible, ingest known good content. Check the protocoltiming.log.<date> for damaged GOIDs by using the following command:

tail -f protocoltiming.log.<latest date> | grep Goids

Cause 2: There is a reachability problem to the destination QAM device.

The CServer returns the same completion code, so the same error is returned in the announce message. In this case, the content does not stream at all. The play request and play response are separated by about 10 to 15 seconds, instead of the typical subsecond separation. This is due to the ARP timeout process the CServer is going through to reach the destination. Once stream response fails, the CServer calls back with the completion code of 3, which causes the "error reading content data" message.

Action 1: Check that the interfaces involved in the streaming are up and operating at the correct speed.

Using the CDSM GUI, choose Monitor > Server Level > NIC Monitor, select the IP address of the server, and verify the participating interfaces are up and operating at gigabit Ethernet speeds For more information, see the "Configuring the Interfaces" section on page 4-59.

Action 2: Set up a stream to play to a multicast address.

If this is successful, then there is a network problem, which is either a default gateway or unreachable remote client. You can verify whether it is successful by looking at the /home/stats/ifstats information.

Action 3: If streaming to a multicast address is not successful, check that the Vaults can be reached.

Check the /arroyo/log/protocoltiming.log.<date> log file for the number of reachable remote servers. Additionally, if there is a cache-fill issue, you will see a large megabit value for the re-xmit buffer.

You can also check the /arroyo/log/c2k.log.<date> log file for any unreachable Vaults.

Tuning Failure

Cause 1: Fail to tune error message (Error 104).

A "fail to tune" error message (Error 104 in the /arroyo/log/streamevent.log file) occurs when the CDSM starts a stream, but the stream fails to reach its destination.

Action 1: To verify the ARP table, enter the arp -a command.

Action 2: Verify there are no network problems; for example, reachability, network partition, and so on.

Action 3: Look for timeout intervals.

Look for characteristic 10 to 30 second timeout intervals between the stream setup and the stream destroy event messages in the /arroyo/log/c2k.log.<date> file.

Restarting the ISA Services

If it is necessary to restart the ISA services, then enter the following commands:

$ /home/isa/IntegrationTest/stop_all
$ tail -f /home/isa/IntegrationTest/ns_log
$ /home/isa/IntegrationTest/run_isa

Poor Video or Audio Quality

When content is streamed to a client device, if there is no video picture displayed on the client device and the audio is working fine, use the following troubleshooting methods:

Verify that the source is working properly and that the original content is of good quality.

Verify that the appropriate bit rates are being sent from the server using the following command on all Streamers:

/home/stats/ifstats

Verify that the content plays locally, and on a test client device (for example, a VLC client).

Test playing the content on an alternate player with an AVC plug-in.

Verify that the CDS is configured correctly.

Check the run script in the /arroyo/test/run directory. There is a tunable set for Telenet to stream null packets when the end of the stream is reached. This should be commented out or removed in a non-Telenet environment.

The interface that you are using for real-time ingests needs to be configured for the CServer. There are a couple of settings that define the interrupt for the real-time ingest interface and ensure that a single central processing unit (CPU) is responsible for receiving the packets for the ingest. Without these settings, packets can be out of order, which can cause problems with the video picture.

To fix this, use the cat /proc/interrupts command to display the interrupts and find the interrupt value associated with the interface you are using for ingest. Once you know this value, add the following lines to the /arroyo/test/run script:

echo 1 > /proc/irq/<interrupt value>/smp_affinity
echo <interrupt value> > /proc/calypso/test/bypass_disable_irq

You can enter these lines at the Linux command line as well by doing so you do not have to reboot your system in order for them to take effect. Any content that you have previously ingested should be considered invalid.

See the "No Streaming" section for troubleshooting methods when there are streaming problems.

Session Messaging

All LSCP message requests and responses have the same format; which includes version, transaction ID, op code, status code, stream handle, and data. The LSCP messages are included in the /home/isa/Streaming/lscp_server/LSCPService.log file. Table A-1 describes the LSCP status codes.

Table A-1 LSCP Status Codes 

State
Code (hexadecimal)
Description

LSC_ OK

0

Success

LSC_BAD_REQUEST

10

Invalid request

LSC_BAD_STREAM

11

Invalid stream handle

LSC_WRONG_STATE

12

Wrong state

LSC_UNKNOWN

13

Unknown error

LSC_NO_PERMISSION

14

Client does not have permission for request

LSC_BAD_PARAM

15

Invalid parameter

LSC_NO_IMPLEMENT

16

Not implemented

LSC_NO_MEMORY

17

Dynamic memory allocation failure

LSC_IMP_LIMIT

18

Implementation limit exceeded

LSC_TRANSIENT

19

Transient error - reissue

LSC_NO_RESOURCES

1a

No resources

LSC_SERVER_ERROR

20

Server error

LSC_SERVER_FAILURE

21

Server has failed

LSC_BAD_SCALE

30

Incorrect scale value

LSC_BAD_START

31

Stream start time does not exist

LSC_BAD_STOP

32

Stream stop time does not exist

LSC_MPEG_DELIVERY

40

Unable to deliver MPEG stream

User-Defined

80-ff

User-defined


Database Issues

This section covers the following database issues and troubleshooting methods:

Database Replication

Corruption Recovery

Database Replication

This section covers the following database issues:

CDSM GUI Does Not Report All the Ingested Content

Many Log Files

CDSM GUI Does Not Report All the Ingested Content

First, verify that the package has not already expired.

Second, check for index errors in the CDSM database logs, using the following command:

$ grep index /arroyo/log/avsdb.log.20071106

11-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -30976
11-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -30976
11-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -30976
11-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -30976
11-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -30976
11-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -30976

The example output indicates that the Vault and CDSM databases are not synchronized, possibly due to the server times not being synchronized, a network connectivity issue, a server failure, or some other similar issue.

To check content among Vaults, use the /home/isa/Berkeley/dumpDB command.

For resolution, refer to the "Corruption Recovery" section.

Many Log Files

If one of the following conditions exists, it indicates that there were database replication errors:

Many log files with a filename similar to log.00000XXXX in the /home/isa/Berkeley/DATADIR directory.

Database could not be started. See the "Services Monitor" section on page 5-34 for more information.

Bidirectional connections are lost between servers. See the "Required Services Not Starting or Running Correctly" section.

The following error message is listed in the /arroyo/log/avsdb-err.log.yyyyMMdd file, "tavsdb: unable to allocate memory for mutex; resize mutex region."

# tail -f avsdb-err.log.20081111
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region
tavsdb: unable to allocate memory for mutex; resize mutex region

The /home/isa/Berkeley/DATADIR/REPLAY.db file increases to several GB in size.

$ ls -ltr
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002824
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002825
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002826
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002837
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002838
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002839
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002841
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002840
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002843
-rw-r----- 1 isa isa   10485760 Nov 11 17:46 log.0000002842
-rw-r--r-- 1 isa isa 5726769152 Nov 12 15:23 REPLAY.db

For resolution, refer to the "Corruption Recovery" section.

Corruption Recovery


Caution Escalate to tier-three support before making any intrusive database changes.

If the CDSM database is corrupted and the Vault database is not corrupted, do the following:


Step 1 As user root, stop the CDSM database.

# /usr/bin/db_shutdown

Step 2 Check to ensure the database is shut down.

# ps -ef | grep avsdb 
isa 2646 1 0 Jan09 ? 00:14:50 /home/isa/Berkeley/avsdb
root 26088 26059 0 13:23 pts/1 00:00:00 grep avsdb 

Make sure there is no avsdb process returned. If the avsdb hangs, use the process ID (2646 in the above example) with the kill command.

# kill -9 2646

Step 3 Delete all files in the /arroyo/db/DATADIR directory.

Step 4 As user root, stop the Vault database.

# /usr/bin/db_shutdown

Step 5 Check to ensure the database is shut down.

# ps -ef | grep avsdb 

Step 6 Copy all files in /arroyo/db/DATADIR directory from the Vault to the CDSM.

Step 7 As user root, restart the Vault database.

# su - isa
#ps -ef | grep avsdb

Step 8 As user root, restart the CDSM database.

# su - isa
#ps -ef | grep avsdb

Step 9 Check the configuration on the CDSM and make sure no configuration parameters were lost.


Advanced Features and Applications

This section covers the Media Scheduler feature (live multicast ingest) and the Barker Stream feature.

Live Multicast Ingest

Live multicast ingest is available as part of the Media Scheduler feature or the Real-Time Capture feature.

Ingest with Media Scheduler

Using Media Scheduler for live multicast ingest requires the following procedures:

1. Enable live ingest by setting both the Media Scheduler and the Ingest Manager to ON in the CDSM Setup. See the "Initializing the CDS and Activating the Optional Features" section on page 3-3 for more information.

2. Use the CDSM Input Channels page to configure the input channels. See the "Configuring Input Channels" section on page 4-30 for more information.

3. Upload channel schedules by importing the electronic program guide (EPG). See the "Uploading an EPG File" section on page 7-14 for more information.

Ingest without Media Scheduler

Using Real-Time Capture for live multicast ingest requires the following procedures:

1. Enable live ingest by configuring Ingest Manager ON and setting Real-Time Capture Type to Real-Time Capture (non-Media Scheduler) in the CDSM Setup page. Activate the Ingest Manager. Because the Ingest Manager is an optional feature, an activation key is required. See the "Initializing the CDS and Activating the Optional Features" section on page 3-3 for more information.

2. Use the CDSM CallSign Setup page to configure call signs with multicast IP addresses.

Ingest Troubleshooting

If the message "ERROR: Unable to login to the ftp location," is present in the /home/isa/bss/log/aim.log file, check the FTP server configured in the Ingest Manager by using the ps -ef | grep ftp command. If the FTP service is not running, enter the service vsftpd start command to start it.

Barker Stream

If you are having issues setting up a barker stream, use the following troubleshooting method. The barker stream runs only on the master Streamer. To identify the master Streamer, use the CDSM Monitor Services page to find the Streamer running the master stream service. See the "Services Monitor" section on page 5-34 for more information. Verify the following three items are present:

Valid content

Valid service group

Valid channel ID

For more information about barker streams, see the "Configuring Barker Streams" section on page 4-56.

Frequently Asked Questions

Many of the frequently asked questions (FAQs) responses were based on an ISV system, but guidelines can be easily extrapolated for a Vault and Streamer. This section covers the following topics:

Reliability and Availability

Serviceability and Manageability

Content

Other

Reliability and Availability

Q. How do I enable stream resiliency?

A. Log in to the CDSM with engineering access. The CDSM Setup page is displayed. For Stream Failover Support, select "ON" and click Submit. For more information see the "CDSM Setup" section on page D-2.

Q. How do I check and make sure the database is running properly?

A. After starting the database, you should see two sockets (listening and non-listening) connecting to the database on each of the remote servers on port 9999. You can check them by using the netstat -an | grep 9999 command.

For example, the following output of the netstat command shows that the server (172.22.97.194) has both the listening and non-listening sockets binding on port 9999 to echo the four remote servers (172.22.97.192, 172.22.97.193, 172.22.97.195 and 172.22.97.191).

# netstat -an|grep 9999 
tcp 0 0 172.22.97.194:9999 172.22.97.195:48652 ESTABLISHED 
tcp 0 0 172.22.97.194:9999 172.22.97.191:42732 ESTABLISHED 
tcp 0 0 172.22.97.194:54563 172.22.97.195:9999 ESTABLISHED 
tcp 0 0 172.22.97.194:39342 172.22.97.191:9999 ESTABLISHED 
tcp 0 0 172.22.97.194:9999 172.22.97.192:40207 ESTABLISHED 
tcp 0 0 172.22.97.194:41815 172.22.97.192:9999 ESTABLISHED 
tcp 0 0 172.22.97.194:9999 172.22.97.193:33196 ESTABLISHED 
tcp 0 0 172.22.97.194:43269 172.22.97.193:9999 ESTABLISHED 
tcp 0 0 

If you can not see both listening and non-listening sockets binding on port 9999 for each of the remote servers, the database is not running properly. Check that you have the correct replication group members in your /home/isa/.arroyorc file.

Serviceability and Manageability

Q. How do I check the calypso server status?

A. Log in to the server as root and enter the cat /proc/calypso/status/server_settings command.

Q. How do I check central processing unit (CPU)?

A. Log in to the server as root and enter the cat /proc/cpuinfo command.

Q. How do I check the ISA server status?

A. Log in to the server as root and enter the /home/isa/IntegrationTest/show_calypso_services command.

Q. How do I check the kernel network driver version?

A. Log in to the server as root and list the e1000.ko file to check the date and time it was created using the following command:

ls -l /lib/modules/<current running kernel name>/kernel/drivers/net/e1000/e1000.ko

The following example shows that the e1000.ko file is based on the kernel 2.5.18-53.el5.kernel.2_6_18.2009.01.08.01.

# ls -l /lib/modules/2.6.18-53.el5.kernel.2_6_18.2009.01.08.01/kernel/drivers/net/e1000/e1000.ko 
-rw-r--r-- 1 root root 2617502 Jan 8 18:13 
/lib/modules/2.6.18-53.el5.kernel.2_6_18.2009.01.08.01/kernel/drivers/net/e1000/e1000.ko 

Q. How do I stop, start, and, restart the Apache server on the CDSM?

A. Log in to the server as root and enter the following commands:

# /arroyo/www/bin/apachectl stop
# /arroyo/www/bin/apachectl start
# /arroyo/www/bin/apachectl restart

Q. How do I check the Streamer static ARP table?

A. Log in to the server as root and enter the following command:

# cat /arroyo/test/ArpTable
ip 192.168.2.42 mac 000000000002
ip 192.168.2.43 mac 000000000002

Q. How do I view the ARP Table dump file?

A. Log in to the server as root and enter the following command:

# echo 1 > /proc/calypso/test/arp_dumpstate

Q. How do I recover the system from the kernel debugger (KDB) after a reboot?

A. If the server starts the KDB tool instead of rebooting, modify the /etc/grub.conf file as follows:

kdb=off panic=1 
kernel /boot/vmlinuz-2.4.32avs ro root=/dev/hda1 console=tty0 console=ttyS0,115200 
kdb=off panic=1

Q. What do I do if the KDB prompt is displayed when the server restarts after a failure?

A. Boot into single user-mode (see the "Kernel Crash" section).

Q. How do I destroy an individual ISA stream?

A. Use the Delete button on the Stream Monitor page. See the "Stream Monitor" section on page 5-14 for more information. Alternatively, you can enter the following commands:

# su - isa
# cd ~/IntegrationTest
# ./list_all_streams
# cd ~/Streaming/client
# ./runclient
	2 -> Stream
	<name - as retrieved above>
	24 -> Destroy

If this does not work, for example, the stream just restarts, then the problem is not in the ISA subsystem.

Q. How do I identify any holes in the content?

A. Log in to the server as root and enter the following commands:

# echo 2 > /proc/calypso/tunables/cm_logserverinfo
# cat /arroyo/log/serverinfo.log.01132009

Look at the last two lines of output. If there are no holes, the last two lines should be the following:

BeingDeleted=0 HasHoles=0 CopyHoles=0 SectorHoles=0
Object Status Check Complete.

Q. How do I clear cached video blocks (data cache) on the Streamer?

A. Log in to the server as root and enter the echo 1 > /proc/calypso/test/clearcache command.

Q. How do I clear the data cache in memory?

Log in to the server as root and enter the echo 1 > /proc/calypso/test/clearmem command.


Note Make sure there are no streams running before you use this command. If there are streams, the data cache in memory is not cleared.


Q. How do I destroy all streams?

A. Log in to the server as root and enter the following commands:

# su - isa
# cd ~/IntegrationTest
# ./destroy_all_streams

All sessions are removed, and upon restarting the services, all streams that do not have an associated session are stopped.

Q. How do I delete an individual stream from the database?

A. Log in to the server as root and enter the following commands:

# su - isa 
# cd ~/Berkeley
# ./bsql
	2 -> Streamer Database
	1 -> Stream Object
	6 -> Get All
	3 -> Delete
	1 -> Name

After the colon for the Name command, name everything that was listed in the Get All command.

Q. How do I destroy all streams when none of the above methods work?

A. Log in to the server as root and enter the following commands:

[root@ssv3 root]# /usr/bin/db_shutdown
[root@ssv3 root]# ps -ef |grep avs

Wait for all avs processes to stop, then reboot the server.

[root@ssv3 root]# reboot

Q. How do I check the routing table and gateway?

A. Log in to the server as root and check the file /arroyo/test/RoutingTable.

# cat /arroyo/test/RoutingTable 
default gateway 192.169.131.250 
network 192.169.131.0 netmask 255.255.255.0 gateway 0.0.0.0 
default cache gateway 192.169.131.250 
local cache network 192.169.131.0 netmask 255.255.255.0 

Content

Q. How do I get information on a content stream that seems corrupted; for example, there is macroblocking, the stream stops and restarts, and so on?

A. Log in to the server as root and enter the following commands:

# echo 2 > /proc/calypso/tunables/cm_logserverinfo
# cat /arroyo/log/serverinfo.log.01132009

Check the last set of output lines to see the current content states.

Object Count=37708 LengthUnknown=0 
CouldNotRepair=0 IsDamaged=0 BeingRepaired=0 BeingCopied=0 
needCrcValidate=37708 isFragFlag=0 isFragd=0 Defrag=0 Smooth=0 
BeingFilled=0 OutOfService=0 NeedsISACheck=0 
BeingDeleted=0 HasHoles=0 CopyHoles=0 SectorHoles=0 
Object Status Check Complete. 

Q. How do I know if a content object has "holes"?

A. Log in to the server as root and view the /var/log/debugmessages. There is a message in the debug messages file regarding the GOID and the content holes.

Q. What do I do if content ingest is not working?

A. Log in to the server as root, enter the less /home/isa/ContentStore/server/GenericLog.log command, and review the log for anomalies.

Q. How do I delete ingests that are "stuck" in the active ingest state?

A. Log in to the Vault as root and enter the following commands:

# su - isa
# cd ~/Berkeley
# ./bsql

Choose the following options, one for each menu:

a. VAULT DATABASE

b. CONTENT OBJECT

c. DELETE

Enter the content ID of the "stuck" ingest, then choose the exit option for each menu until you are back at the Linux prompt.

Q. How do I manually ingest content from the command line?

A. Log in to the Vault as root and enter the following commands:

# su - isa
# cd ~/SDClient

Update the SDClient.cfg file with the local IP address.

# ./sdClient

Follow the SDClient menus.

Other

Q. How do I view the CServer code configuration file?

A. Log in to the server as root and enter the cat /arroyo/test/<server type>/setupfile command. The server type is one of the following: vault, streamer, or ssv.

# cat /arroyo/test/<server type>/setupfile
# CServer core configuration.  Changes to this file require
# a server reboot.

local 0 0 2 remote 0 0 2 fill 3 1 maxrate 900000 localip 0c0a80040
localip 0c0a80040
e1000 adapters: maxrate 965
e1000 0: streaming 1 fill 0
e1000 1: streaming 1 fill 0
e1000 3: streaming 0 fill 1

streamer 1 vault 1
serverid        64
groupid         64
maxpacketsize   1316
management      eth0
ingest          eth0
trickspeedsv2    10 0 0 0 0 0 0 0
ftpout if eth0 max utilization mbps 0 max sessions 0
fake cylindermap 1
test 4

Q. How do I know if a subsystem on a server is overloaded?

A. View the .arroyo.log.protocoltiming.log.<date> file. When you see the "COST REQUEST NO CAPACITY:" message, it means that the server is running out of capacity and it cannot accept new streaming requests.

Also, when you see a line in the /arroyo/log/c2k.log.<date> file that says the following:

01-May-2007 17:40:44 UTC :err:ServeStream::reserveStream: refused streamhandle 4 for 
goid a445c9780e7f8f due to its load 3750, current load 0

This entry typically means there are no stream ports linked. In the ten-second snapshot of the /arroyo/log/protocoltiming.log.<date> file, there is a line that shows load values for each of the major subsystems (LAN, memory, CPU, and so on). More than likely one of the subsystems is at 100, which is the subsystem that is having the problem.

Q. How do I enable debugging?

A. Log in to the server as root and enter the following commands:

# su - isa
# cd ~/StreamsDriver
# touch DEBUGGING_ON
# ~/IntegrationTest/debugging_on_off
#./stop_driver
#./run_driver

Q. How do I update the remote servers from /arroyo/test/RemoteServers?

A. Log in to the server as root and enter the following commands:

# echo 1 > /proc/calypso/test/readremoteservers

CDS Content Quality Guidelines

This section covers the following topics:

Supported Elementary Stream Types

Scrambling

Transport Bit Rate

Stream Length

Format Restrictions

Preferred Formats

Supported Elementary Stream Types

Video-only, audio-only (as well as audio streams with only a few or occasional video frames) and data-only streams are supported in addition to the customary multiplex of both audio and video.

Scrambling

The transport layer cannot be scrambled, meaning the transport header and any adaptation field must be in the clear. Streams whose Elementary Streams (ESs) are fully scrambled, including all start codes, are capable of being ingested and streamed, but are incapable of trick play.

For trick-play capability, the following cannot be scrambled:

Packetized Elementary Stream (PES) headers

Program Association Table (PAT) and Program Map Table (PMT)

Closed-caption data (if scrambled, the data is incorrectly included in tricks)

Transport Bit Rate

All transport streams are constant bit rate (CBR). Variable bit rate (VBR) is not supported. The maximum bit rate is 35 Mbps. There is no minimum bit rate. The ES video bit rate, as specified in the MPEG-2 sequence header, is ignored. The bit rates of individual ESs do not matter. The CBR for an individual ES (in particular the video) is not required. All that is required is that the aggregate transport stream be CBR.

Streams containing MPEG-2 or AVC video are expected to conform to the appropriate buffer models spelled out in ISO/IEC 13818-1and 14496-10 .

Stream Length

All content must be at least one second in length. A content item must be under 12 hours in length or 81 GB in size, whichever comes first.

Format Restrictions

Following are the format restrictions for Advanced Video Coding (AVC), H.264, and MPEG-4:

Sequence Parameter Set (SPS) seq_parameter_set_id flag must be zero.

SPS pic_order_count_type flag must be zero.

SPS seq_scaling_matrix_present_flag must be zero.

SPS profile_idc flag must only be Baseline, Main, or High profile.

Preferred Formats

Using the following guidelines improves the performance of the system, the quality of the tricks, and the trick transitions.

1. All content should be encoded as a Single Program Transport Stream (SPTS) . If multiple programs must be included (for example, a Picture-in-Picture [PIP] stream), ensure that the "real" program is encoded with the lowest program number.

2. All content should follow the process ID (PID) numbering specified in the Content Encoding Profiles 2.0 Specification (MD-SP-VOD-CEP2.0-I02-070105), section 6.7.5. Regardless, the audio and video PIDs should be above 0x20.

3. All content should be preceded with a Program Association Table (PAT) and then a Program Map Table (PMT), and then a Program Clock Reference (PCR) before the first audio or video frame. Optionally, the discontinuity bit can be set.

4. All content should use the same PID for both PCR and video.

5. All content should begin with a closed Group Of Pictures (GOP) for MPEG-2 or with an Instantaneous Decoder Refresh (IDR) frame for AVC. This first frame is always accompanied by a sequence header for MPEG-2 or by an SPS for AVC.

6. In order to guarantee relatively smooth looking trick modes, the minimum I/IDR-frame frequency should be eight per second. If the minimum trick speed is 4x or less, the I/IDR-frame frequency should be at least two per second. In no case should two I/IDR frames be more than two seconds apart.

7. Each I frame should be preceded by a sequence header and GOP header if any exist for an MPEG-2 video. Each I/IDR frame should be preceded by an SPS and Picture Parameter Set (PPS) for H.264 video.

8. Avoid mixing frame data from multiple video frames in the same transport packet. Specifically, no data belonging to the prior frame exists following the Packetized Elementary Stream (PES) packet header for the next frame. Breaking this rule may improve encoding efficiency slightly, but degrades the quality of the tricks on certain set-top boxes (STBs).

9. All content must be encoded as a single sequence, with no changes in horizontal or video resolutions, or changes in encoding parameters in the middle of the content.

10. The GOP size may be variable, but GOPs should generally not exceed two seconds in length. Using longer GOPs may improve encoding efficiency, but the quality of lower-speed tricks (3x, 4x) may suffer.

11. No more than four B-frames should be used between each pair of I-frames or P-frames.

12. There should be no continuity counter errors in the content.

13. There should be no discontinuities in the content, other than an optional one on the first PCR.

14. The accuracy requirements for PCRs, +/- five parts per million (5 ppm), as stated in ISO/IEC 13818-1 must be adhered to throughout the stream.

15. Audio and video are expected not to overflow the appropriate target buffer model specified.

16. A reasonable bit rate to use when encoding MPEG-2 standard definition (SD) video is 3.75 Mbps.

17. A reasonable bit rate to use when encoding MPEG-2 high definition (HD) video is 15 Mbps.

18. Appropriate bit rates for carriage of AVC are still being established, and while they are expected to be at least half the bit rates of MPEG-2, no specific recommendations can be offered.

19. There may be PIDs in the content that are not specified in the PMT. Such use is beyond the scope of this document.

20. All PATs and PMTs should be identical, with the same version number throughout.

21. The CDS support up to 30 Mbps MPEG-2 video encoding.

22. Content is filtered out if three occurrences of one-second synchronization lost is identified.

23. Content is filtered out if five seconds of null frames are identified.