Cisco VDS-TV 3.5 RTSP Configuration Guide

File System

The CDSM file system differs from the file system on the VDS servers (Vault, Streamer, Caching Node, ISV).

CDSM

The CDSM has the following directory structures:

• /arroyo/asmrpt—Contains comma-separated values (CSV) files that are created by extracting information from the database every 24 hours. These files are accessible through the Reports > Archived Data page. The asm_archiver job must be installed and added to the crontab for these files to be generated. For more information, see the “Archived Data” section.
• /arroyo/db—Contains the database binaries, this roughly maps to the /home/isa/Berkeley directory on Streamers and Vaults.
• /arroyo/db/DATADIR—Contains the database files and indexes.
• /arroyo/image—The staging area for VDS software image files. This directory also includes backup directories when a software upgrade is performed on the server.
• /arroyo/msa—Contains the Managed Services Architecture (MSA) logs that are created by extracting information from the database. The logs are processed by the iVAST MSA agent.
• /arroyo/www—Contains the HTTP files for the CDSM GUI. The subdirectory arroyo/www/htdocs, contains the PHP files for the CDSM GUI.
• /arroyo/www/modules—The link library for htdocs files.
• /home/isa/—Contains configuration files.

Report Archiving

The CSV files are generated every 24 hours and are deleted when they are older than 30 days. The CSV files are stored in the /arroyo/asmrpt directory. For the CSV files to be generated, the report archiver needs to be installed and configured. The CSV files are accessible by going to the /arroyo/asmrpt directory, or by using an FTP client with the username “asmrpt” and the password “asmrpt.”

Vault, Streamer, Caching Node, and ISV

The VDS servers (Vault, Streamer, Caching Node, and ISV) have the following directory structures:

• /arroyo/db
• /arroyo/log
• /arroyo/test/
• /arroyo/archive
• /home/isa

In addition to the above directories, the VDS servers have the following directories specific to the RTSP environment:

• /home/isa/bss
• /home/isa/bss/bin—Contains the VOD server binaries
• /home/isa/bss/database—Contains the database files
• /home/isa/bss/etc—Contains the configuration files for the binaries
• /home/isa/bss/lib—Contains the shared application libraries
• /home/isa/bss/run—Contains process ID (PID) files for running the application
• /home/isa/bss/scripts—Contains general scripts

Log Files

There are four types of log files in an RTSP environment:

• Linux Log Files
• CServer Log Files
• RTSP Log Files
• CDSM Log Files

The log files are rotated at least once a day and time stamps are added to the filenames. Some log files that grow rapidly are rotated more frequently (determined by file size); this rotation may happen up to once an hour. Most log files have the following suffix: .log.<YYYYMMDD.> The time zone for log rotation and filename suffixes is coordinated universal time (UTC).

The CServer log files are automatically archived and moved to the /arroyo/archive directory when the disk storage reaches a certain level. The RTSP log files are automatically archived and moved to the /arroyo/log archive directory whenever the FSI or RTSP process is restarted, or the log file reaches close to 50 MB. A total of nine revisions are kept of each log file, with the eight oldest being compressed and moved to the archive directory.

To change the log level or set the debug flags for the log files, use the Configure > System Level > Logging and Configure > Server Level > Logging pages. For more information, see the “Configuring the System Level Logging” section and the “Configuring the Server Level Logging” section.

The following log tools are available:

• loginfo —Provides information on each facility, associated log file, and debug flags. The loginfo tool can be run on any VDS server, including the CDSM. To view help on loginfo , enter the loginfo -h -v command.
• logconfig —Provides log configuration on CDSM. To view help on logconfig , enter the logconfig -h -v command.

Linux Log Files

The Linux operating system has the following useful log files:

• /var/log/debugmessages—Syslog messages
• /var/log/messages—Includes useful bootup status messages

CServer Log Files

The VDS has the following useful log files:

• /arroyo/log/c2k.log. <date>— This log has information about content read issues. The date extension for the log filename has the format of yyyymmdd (for example, 20090115 is January 15, 2009). To increase the verbosity of this log file, use the following command:

• /arroyo/log/protocoltiming.log. <date> —Provides information about any network interface issues and any disk issues.
• /arroyo/log/avsdb.log. <date> —Provides information about any database issues.
• /arroyo/log/statsd.log. <date>— Provides system statistics information.
• /arroyo/log/stresstest.log. <date> —Provides CPU uptime information.
• /root/avslauncher.log. <date> —Provides information about the startup of the avslauncher module.

Other CServer log files that may be useful are the following:

• /arroyo/log/controlblocktiming.log. <date>
• /arroyo/log/debug.log. <date>
• /arroyo/log/decommissioned.log. <date>
• /arroyo/log/deleted.log. <date>
• /arroyo/log/executiontiming.log. <date>
• /arroyo/log/objectRepair.log. <date>
• /arroyo/log/serverinfo.log. <date>
• /arroyo/log/streamevent.log. <date>
• /arroyo/log/systemstats.log. <date>

Note The files with the extension <date> use the format yyyymmdd. The date is the Coordinated Universal Time (UTC) date.

CServer Error Codes

CServer error codes that appear in the c2k.log. <date> file do not necessarily mean an error has occurred. An actual error has “err” listed in the entry, as opposed to “out” or “ntc.” Following is a list of important CServer error and status codes:

Error Codes

• 5—Completion of a task.
• 25—Insufficient resources.

Status Codes

• 0—Content is okay (cnOK).
• 1—Stream has ended (cnEnd).
• 2—Stream has been paused (cnPaused).
• 3—Error has occurred (cnError).
• 4—Next element is being processed (cnNextElement).
• 5—Live content has resumed (cnResumeLive).
• 6—Next content object is being processed (cnNextContent).
• 7—Next iteration is being processed (cnNextIteration).
• 9—There has been a failover (cnFailover) .
• 8—Stream has been destroyed (cnDestroyed).

Protocoltiming Warning Messages

Table A-1 describes some of the warning messages that might be seen in the Protocoltiming log.

RTSP Log Files

The Linux user isa is the owner of the application files in an RTSP environment. To change from the root, use the su – isa command.

All application log files are located in /arroyo/log directory. When a log file reaches around 50 MB in size, it is archived and moved to the archive directory. When the RTSP or FSI is restarted, all current log files are archived and moved to the archive directory. The latest archived log files remain uncompressed, while the rest are gzipped. The archive directory always maintains the ten latest archives of each log file. The older log files are deleted.

If the RTSP application or the FSI application does not start, check the following files for possible reasons:

• rtsp.err
• rtspmonitor.log
• fsi.err
• fsi.out

These files are located in the /arroyo//log/ directory.

The VDS RTSP environment has the following useful log files:

• /arroyo/log/fsi.log
• /arroyo//log/rtspmonitor.log
• /arroyo//log/aim.log
• /arroyo//log/library.log—Logs the cache2app library information

Log File—rtspmonitor.log

The rtspmonitor.log log file stores the processing state and associated session ID of each thread.

The thread state logging is controlled by the THREAD_MON_FREQ in the arroyo-env.sh script in the /home/isa/bss/scripts directory. The THREAD_MON_FREQ parameter is used as a time interval to print the thread state and the corresponding value, which could range from 0 to 65535. The default value for the THREAD_MON_FREQ parameter is 30. The logging can be turned off by setting THREAD_MON_FREQ to 0.

Note The THREAD_MON_FREQ parameter can be enabled from the CDSM. For more information about enabling the parameter, see Configuring the Server Level Logging section.

CDSM Log Files

The CDSM has the following logs:

• httpd.log.< yyyymmdd >—Apache error log
• httpd_access.log.< yyyymmdd >—Apache access log
• CDSM.log.< yyyymmdd >—CDSM GUI log
• CDSM-ws.log.< yyyymmdd >—Web Services log

All log files use UTC for the log entry time stamps and filenames. All four files are located in /arroyo/log directory.

The default log level for httpd.log is LOG_WARNING and the setting is preserved. The log level for httpd.log (facility is httpd) can be configured by using the /home/isa/logging/logconfig tool. The logconfig tool overwrites the LogLevel value in the httpd.conf file with the new value and the /arroyo/www/bin/apachectl restart command is issued.

The httpd_access.log (facility is httpd_access) is always on and the log level cannot be changed with the logconfig tool.

Following is an example of a log entry in the httpd_access.log file:

Following is an example of a log entry in the httpd.log file:

Server Configuration Files

The server configuration settings are stored in the .arroyorc file and the setupfile file. This section describes the different parameters for each file.

Note This section is informational only. All changes to the configuration files should be accomplished through the initial configuration and CDSM GUI.

Table A-2 lists the CDSM GUI ID names and maps them to the CServer names in the setupfile and .arroyorc files.

Description of the .arroyorc Settings

This section describes the different line entries of the .arroyorc file. The .arroyorc file is located in the /home/isa directory and is created during the initial configuration procedure outlined in the Cisco Content Delivery Engine 205/220/250/420 Hardware Installation Guide .

self

This number represents what type of server the CDE is:

• 0 = ISV (also known as SSV)
• 1 = Vault
• 2 = Streamer
• 3 = CDSM

groupid

All servers that are part of the same VDS system (managed by one CDSM) have the same group ID. This group ID should be unique across an enterprise. The purpose of the group ID is to allow servers in a group to recognize each other as belonging to the same group. If two server groups were on the same VLAN and they had the same group number, they would conflict and cause issues. This is much more likely to be an issue in a lab environment with shared resources than an actual production deployment, but this should still be managed.

serverid

Every server in the group has to have a unique ID ranging from 1 to 255. It is a good idea to use a standardized numbering solution; for example, all 1xx serverids are Streamers and all 2xx server IDs are Vaults.

vault

This parameter has the IP address of a Vault in the system. Each “vault” line represents an individual Vault. There may be multiple vault lines.

streamer

This is the IP address of a Streamer in the system. Each “streamer” line represents an individual Streamer. There may be multiple streamer lines.

controller

This is the IP address of the CDSM. There is only one controller line. This line is not needed in the file for the CDSM, but is used on Vaults and Streamers to point to the CDSM.

mirroring

This contro ls local mirr oring, which is to say this determines the number of copies of a given piece of content that is stored locally.

partno

This allows the server to identify itself properly to the CDSM. The CDSM can then display the appropriate server graphic in the GUI and manage the appropriate number of disks, Ethernet ports, and so on.

mgmtif

The index of the management interface starting at eth0. Typically, this remains 0.

ingestif

This parameter is only for Vaults. The index of the ingest interface starting at eth0. Typically, this remains 0 but may have the value of 1 as well.

dbdomsock

This is the “file handle” where the applications address messages are intended for the database.

dbnetport

This is the port number where the applications address messages are intended for the database.

controlif

The index of the stream control interface starting at eth0. This is an optional configuration that is used when you want to separate the Setup and Control interface.

Description of the setupfile Settings

This section describes the different parameters of the setupfile file. The setupfile file is located in the /arroyo/test directory. Some values for the parameters in the setupfile file are set during the initial configuration (serverid, groupid, streamer vault), others are set by using the CDSM.

Note The localip # line entry has been deprecated. Ignore this line entry.

Required Settings

The following line entries are required in every setupfile file:

serverid #

An identifier that uniquely identifies the server within a group of servers identified by the group ID. See the “serverid” section for more information.

groupid #

An identifier that identifies the group of servers within the VDS. See the “groupid” section for more information.

streamer <0 or 1> vault <0 or 1>

To run the server as a Streamer, set streamer to 1; otherwise, set streamer to 0. To run the server as a Vault, set vault to 1; otherwise, set vault to 0. Setting both streamer and vault to 0 is not a valid option.

service address <ip in dot notation> setup <setup portno> control <control portno>

The service address is used to specify whether this server can assume the role of the Setup server, the Control server, or both the Setup and Control servers for the specified IP address. This parameter applies only to Streamers.

• setup portno —A value of 0 means the server is not available to assume the role of the Setup server for the specified IP address. A value of 1 means to use the default port number 3300.
• control portno —A value of 0 means the server is not available to assume the role of the Control server for the specified IP address. A value of 1 means to use the default port number 9000.

e1000 <index>: streaming <0 or 1> fill <0 or 1> ip <ip in dot notation> tport <transport portno> cport <cache portno> tgid <transport groupid>

The e1000 is used to configure the network interfaces for cache-fill and transport/streaming. Each “e1000” line represents an individual Ethernet port. Include one line per interface.

• index—Refers to the interface index as known to the e1000 driver. In the case of servers with the Lindenhurst chipset, this matches one for one with the number for the eth# interface.
• streaming—For transport/streaming. A value of 1 means this interface is used for streaming; otherwise, set streaming to 0.
• fill—For cache fill. A value of 1 means this interface is used for cache fill. Otherwise, set fill to 0.
• ip—Each interface requires a source IP address. This assumes Layer 3 networks only.
• tport—The transport port number used as the source in transporting (streaming) packets. A value of 0 means to use the default port number 1026 (unless affected by the optional default source IP entry).
• cport—The cache port number used as the source in caching (fill) packets. A value of 0 means to use the default port number 48879 (unless affected by the optional default source IP entry).
• tgid—The transport group ID for this interface. The transport group ID is used in conjunction with the TransportGroupIdTable file located in the /arroyo/test directory to determine which interface to use to transport the packet. This is based on destination IP address or subnet of the packets. The default value is 0, which means this interface is available to any transport group. Any other value means the interface is dedicated to a particular transport group.

vault mirror copies <number of copies>

The Vault mirror copies is a numeric value representing the number of copies of each content to store on the Vaults.

Note The transport group ID (tgid) has been deprecated. Use the SubnetTable instead. See the “Network” section.

Optional Settings

The following line entries are optional in the setupfile file:

management eth #

Specifies the interface used for management. The default is eth0.

ingest eth #

Specifies the interface used for live ingests (FTP push or UDP capture). By default, the management interface is used. This parameter is applicable only to Vaults.

e1000 adapters: maxrate <rate in Mbps>

Controls the maximum transmit bandwidth on this interface, either for streaming, for caching, or for both. The default is 975 Mbps.

ibg adapters

The maximum transmit bandwidth of the ibg adapters. The default is 975 Mbps.

disks #

Specifies the number of hard drives (disks) installed on a server. The default is 12 disks for a Streamer, and 24 disks for a Vault. If you have a server with 12 disks, you must add this entry and specify 12 disks; otherwise, warning messages stating disks are non-operational are logged to the protocoltiming log file.

test #

Specifies the test mode of the server. The default is 4, which means to run the server in production mode.

cache_dscp #

Used to set the DSCP bits on cache-fill packets. The default value is 0.

cache_ecn #

Used to set the ECN bits on cache-fill packets. The default value is 0. This parameter should not be used.

transport_dscp #

Used to set the DSCP bits on transport/streaming packets. The default value is 0.

transport_ecn #

Used to set the ECN bits on transport/streaming packets. The default value is 0. This parameter should not be used.

trickspeedsv2 # # # # # # # #

Used to specify up to 8 speeds for generating trick objects during ingest. This parameter is applicable only to Vaults. The defaults are 5, –5, 10, –10, 32, and –32. The highest speeds are –127. An entry larger than 127 defaults to 127. A value of 0 is ignored.

ftpout if eth # max utilization mbps # max sessions #

Used to specify which interface on the Vault is used for FTP Out, the maximum bandwidth utilization for all active sessions (in Mbps), and the maximum number of simultaneous sessions allowed. By default the maximum sessions is 0, meaning that FTP Out is not allowed. The default for bandwidth utilization is 0, which means unlimited usage. The default interface selected is the management interface. For FTP Out to function properly, the entire content must exist on the Vault.

bypass_isacheck <0 or 1>

Used in the RTSP environments to bypass global object ID (GOID) checks at startup.

arrayid #

Specifies the array this server belongs to. The array ID is used in conjunction with the StreamDestinationMap file located in the /arroyo/test directory to determine which play servers are available for selection. The selection is based on the destination IP address or subnet of the packets. The default value is 0.

remote site manager <ip address> for arrayid #

Specifies the IP address of a remote site manager for a specified array. The remote site manager has a list of all the servers that it can connect to that are located at the same site (the servers are listed in the RemoteServers list on the remote site manager). When this server comes up, it contacts the remote site manager and asks for a referral for a server that is in the specified array. The remote site manager returns the IP address of a server from that array. This server sets up connections with all the referred remote servers, and once every minute, checks to see if there are connections with all remote array servers. If for some reason one is not available, this server contacts the remote site manager for another referral.

Note You must add the allow new L3 remote servers 1 directive to the setupfile for both this server and the server that gets referred. If you do not add this directive, this server cannot establish a connection with the referred remote server.

default source ip <ip in dot notation> tport <minportno> - <maxportno> cport <portno>

Used to affect source packets if no specific information is provided in the individual mandatory e1000 interface entries.

• ip—The default source IP address for an interface. This value is overridden by the mandatory e1000 interface entry. This IP address is not meaningful in a Layer 3 network. However, today it must have a non-zero value for the other values to be looked at.
• tport—After a stream is started, a random port within the range specified is used as the source port for transport/streaming packets of the stream (assuming no specific port was selected for tport in the mandatory e1000 interface entries).
• cport—The source port to use for cache-fill packets (assuming no specific port was selected for cport in the mandatory e1000 interface entries).

Note The default source IP is useful in a Layer 2 network. For Layer 3 networks, an IP address is required for each interface, so the value in the default source IP is superseded by the individual entries for the interfaces. However, the default source IP setting specifies other defaults (transport port and cache port). If you would like to specify a range of transport ports, then the default source IP could have a value of zero.

Note The default source IP can be used in conjunction with the mandatory e1000 interface entries. For example, the default source IP can be used to specify a range for the source transport port. However, the generation of a random port does not currently work on every stream start. Therefore, it is best not to use this option.

bms address <ip> <port>

The IP address and port of the back office.

Identifying the Software Versions or Releases

The following sections describe the commands for identifying the software versions on the server.

Linux OS Version

To identify the software version of the Linux operating system (OS) on the CDSM, enter the following command:

To identify the software version of the Linux OS on the Vault, Streamer, or ISV, enter the following commands:

VDS-Related Releases

The RTSP VDS software is a combination of an RTSP overlay, statsd software, and the CServer code. The following sections describe how to identify the software version of each.

RTSP Environment

To identify the software version of the VDS RTSP overlay image, enter the following command:

statsd Program

To identify the software version of the statsd program, enter the following command:

CSserver Code

To identify the software version of the CServer on the Streamer, Vault, or ISV, enter the following command:

To view the CServer settings, status, and version, enter the following command:

Using ifstats to Monitor Traffic

The ifstats command shows real-time traffic on each Ethernet interface on the server.

Kernel Crash

The kernel debugger (KDB) provides information (in the form of a core dump file) when the server processing fails. For the server to enter KDB when the server has crashed, the /proc/sys/kernel/panic parameter must be zero. If the panic parameter is non-zero, the system reboots automatically without entering KDB.

In addition to KDB, there is a kdump service. The kdump service allows you to take a kernel dump of memory. The kdump service runs automatically if the server is configured to reboot automatically after a crash (which means the panic parameter is non-zero). The kdump service stores the kernel memory dump in the /var/arroyo/crash directory. After the kernel memory is dumped, the system reboots into the normal operating system.

If the server is configured to enter KDB (which means the panic parameter is zero), the server enters KDB mode. The kdump command allows you to take a kernel memory dump while the server is in KDB mode. The kdump command reboots the server into kdump mode, takes a kernel memory dump, and reboots the server into the normal operating system.

If a server has crashed after being started automatically from the /etc/rc.local directory, you need to boot in single-user mode. To boot in single-user mode, perform the following steps:

Step 1 Reboot the server.

Step 2 When a blue screen displays a list of Linux versions, press the E key to edit the kernel entry.

Step 3 Multiple lines are displayed. Use the Up Arrow and Down Arrow keys to highlight the second line. You may need to press the E key again to edit the line. A square cursor appears at the end of the line.

Step 4 Remove the 115200 from the console parameter (for example, console=ttySO,115200n8).

Step 5 Add the word “Single” or the letter “S” to the end of the line.

Step 6 Press Enter .

Step 7 Press the B key to boot the Linux kernel into single-user mode.

Step 8 Wait for the server to finish booting up.

Step 9 Edit the /etc/rc.local file and comment out the line /arroyo/test/vault/run .

Step 10 Reboot the server.

To view the contents of the core dump file from the Linux prompt, do the following:

Step 1 Run the GNU debugger (gdb), and specify the core file and binary file.

The core-file parameter is the core filename and the binary-file is the binary file that produced the core file.

Step 2 After the GNU debugger has started, enter the backtrace command, bt , at the gdb prompt and press Enter .

The callback stack is displayed, which shows the history of the current function calls that were made at the time of the crash.

Disk Drive Issues

The disk drive order is irrelevant when reinserting disk drives after transporting a chassis, or transferring disk drives from one chassis to another.

To view the statistics of the internal boot drive, the disk drive that contains the software, enter the df -k command.

To view the statistics of a removable SATA or SCSI disk drive, use the following commands:

CDSM GUI Disk Monitor Page Reports a Disk Warning

If the CDSM GUI Disk Monitor page reports a disk warning, check the disk drive status in the /arroyo/log/protocoltiming.log.< date > log file and the /var/log/debugmessages log file.

Note Sometimes on the CDE400, the bus and host resets are used to reset the SATA driver because the Linux SATA driver, sats_mv.ko, does not provide a device reset vector. If the device is reset when there are no outstanding requests, warning messages are displayed on the console. These warning messages are informational and do not indicate a failure.

Memory Issues

To slow down the CDSM bootup to see the memory counter, do the following:

Step 1 Reboot the server.

Step 2 To enter the BIOS Setup Utility, press the Delete key on your keyboard when you see the following text prompt:

Note In most cases, the Delete key is used to invoke the setup screen. There are a few cases where other keys are used, such as F1, F2, and so on.

Step 3 Use the Right Arrow key to navigate to the Boot menu.

Step 4 Choose the Boot Settings configuration option (Figure A-1).

Step 5 Choose Quick Boot and set it to Disabled .

Figure A-1 BIOS Setup Utility—Boot Settings

Step 6 Press F10 to save and exit the BIOS Setup Utility.

Disabling USB Ports and Password-Protecting the BIOS

Disabling USB Ports and Password-Protecting the BIOS

The following procedure provides instructions on disabling the USB ports and password-protecting the BIOS. Disabling the USB ports and password-protecting the BIOS provides a way to secure the server from anyone outside the allowed administrative group from accessing the server through the USB port.

If the USB port is required to perform an operation (for example, a software upgrade), the operator can log in to the BIOS using the BIOS password and enable the USB port for the operation. After the operation is complete, the operator logs back in to the BIOS and disables the USB port.

To set the BIOS password and disable the USB ports, do the following:

Step 1 Reboot the server.

Step 2 To enter the BIOS Setup Utility on a VDS server (Vault, Caching Node, Streamer, or ISV), press the Delete key when you see the following text:

To enter the BIOS Setup Utility on a CDSM or VVIM, press F2 .

The BIOS menu is displayed with the Main tab selected.

Use the Right Arrow and Left Arrow keys to select a menu tab, and the Up Arrow and Down Arrow keys to select a menu item.

Step 3 To set the Administrator password, do the following:

a. Use the Right Arrow key to navigate to the Security menu. The Security options are displayed. The Administrator has read/write permission. The User has read-only permission.

b. Use the Down Arrow key to navigate to the Administrator Password option and press Enter . The Create New Password dialog box is displayed.

c. Enter the password.

To set a user password, use the D own Arrow key to highlight the User Password , press Enter , and enter the password in the Create New Password dialog box.

Step 4 To disable the USB ports, do the following:

a. Use the Left Arrow key to navigate to the Advanced menu. The Advanced options are displayed.

b. Use the Down Arrow key to navigate to the USB Configuration option and press Enter . The USB Configuration options are displayed.

c. Use the Down Arrow key to navigate to the USB Controller and press Enter . The USB Controller dialog box is displayed.

d. Use the Up Arrow and Down Arrow keys to highlight Disabled and press Enter . The USB Configuration options show Disabled for all USB options.

Step 5 To save your settings, press F10 . The Save & Reset confirmation dialog box is displayed.

Use the Right Arrow and Left Arrow keys to highlight Yes and press Enter .

After rebooting the server, to enter the BIOS Setup Utility, you are prompted to enter the BIOS password. To have read/write permission, enter the Administrator password. To have read-only permission, enter the User password.

To enable the USB ports, reboot the server, do the following:

Step 1 Reboot the server.

Step 2 Enter the BIOS Setup Utility on a VDS server (Vault, Caching Node, Streamer, or ISV) by pressing the Delete key when you see the following text:

To enter the BIOS Setup Utility on a CDSM or VVIM, press F2 .

The BIOS menu is displayed with the Main tab selected.

Step 3 Use the Left Arrow key to navigate to the Advanced menu. The Advanced options are displayed.

Step 4 Use the Down Arrow key to navigate to the USB Configuration option and press Enter . The USB Configuration options are displayed.

Step 5 Use the Down Arrow key to navigate to the USB Controller and press Enter . The USB Controller dialog box is displayed.

Step 6 Use the Up Arrow and Down Arrow keys to highlight Enabled and press Enter . The USB Configuration options show Enabled for all USB options.

Step 7 To save your settings, press F10 . The Save & Reset confirmation dialog box is displayed.

Use the Right Arrow and Left Arrow keys to highlight Yes and press Enter .

Network

The following commands are useful for checking your network configuration and activity.

To view the ARP table, enter the following command:

To view the IP routing table, enter the following command:

To view the VDS subnet table, enter the following command:

Note In Release 2.1, the local networks and their gateways are specified in the SubnetTable file. For backward compatibility, the local subnet and gateway in the RoutingTable are still supported and are used if the SubnetTable file does not exist. The Routing Table can still be used to specify static routes.

To view the VDS routing table, enter the following command:

To view the VDS remote server table, enter the following command:

Interface Information

To view basic interface information, use the ifconfig command.

For detailed interface information, view the interface information file in the /proc/net/ directory.

BIOS Settings—Operating System Hangs or Goes into KDB Mode

When a single bit error occurs in the memory of a server, it causes the Linux OS to lock up, which puts the server into kernel debugger (KDB) mode. This is because of the BIOS Error Correcting Code (ECC) Type being set incorrectly.

To correct the ECC Error Type setting in the BIOS Setup Utility, do the following.

Step 1 During the server bootup, press the Delete key to enter the BIOS Setup Utility.

Step 2 Navigate to the Advanced menu and choose Advanced Chipset Control.

Step 3 Choose ECC Error Type and change the setting to NMI (Figure A-2).

Figure A-2 BIOS Setup Utility—Advanced Chipset Control

Step 4 Press F10 to save and exit.

Serial Console Port Settings

The CDE servers ship with the following serial console settings: 115200 baud rate, no parity, 8 data bits, and 1 stop bit (115200-N-8-1). To verify the serial console settings, check the following:

• BIOS Settings—Determines the speed during the very beginning of the boot process up to and including the GRUB menu. In the BIOS Setup Utility, check that the Console Redirection in the Advanced menu is set to VT100.
• /etc/boot/menu.lst—Determines the speed after the kernel is loaded.
• /etc/inittab—Determines the speed after the OS is loaded. Enter the following:

Note The CDE100 may have the following serial console settings: 9600 baud rate, no parity, 8 data bits, and 1 stop bit (9600-N-8-1).

Required Services Not Starting or Running Correctly

From the CDSM GUI, view the Services page for each server by clicking Monitor > Server Level > Services . For more information, see the “Services Monitor” section. If the required services are not started, or they are not running correctly, check that there is two-way database connectivity between the Streamers and Vaults, and the CDSM.

Two connections for each Vault and Streamer should be listed with a status of “ESTABLISHED.”

If the connection states do not say “ESTABLISHED,” check the configuration of /home/isa/.arroyorc file to make sure the settings are correct, then restart the database.

Log in to the server as isa and start the database.

Log into the server as root and start the statsd.

Check that the RTSP listener is running on the correct port.

Process Monitoring with monit

The monit monitoring application is integrated into the arroyo script that is used for starting and stopping all FSI/RTSP related applications. When an application is started using the arroyo start < appname > command, the arroyo script not only starts the application but also instructs monit to start monitoring the application while it is running. In the event the application crashes, monit restarts the application. When an application is stopped using the arroyo stop <appname > command, not only does the arroyo script stop the application, but it also instructs monit to stop monitoring the application so that there is no race condition where the user is trying to stop the application using the arroyo stop < appname > command while monit keeps restarting it. To see if the monit daemon is running on the system, enter the ps -elf | grep monit command.

System Health

The colored boxes on the System Health Monitor page have the following meaning:

• Green—All components are operating; occasionally click each check box to verify.
• Yellow—Some components are not operational.
• Red—All components have failed.

Cannot Access the CDSM GUI

If you cannot access the CDSM GUI, log in as root and verify that the Apache server is running on the CDSM.

If the Apache server is not running, restart the server.

CDSM GUI Does Not Register the Vaults and Streamers

If the CDSM GUI is not able to register that the Vaults and Streamers are part of the array or VDS, do the following:

Step 1 Log in to the Vault or Streamer as root .

Step 2 Verify two-way database connectivity with the CDSM.

Step 3 Verify statsd is running.

Step 4 Verify the correct version and permissions for /home/stats/svrinit or svrinit_15 are being used.

Step 5 On the Vault and the Streamer, initialize the VDS host in the database by using svrinit_15. Use the following options:

• Option -i for the server physical IP address (eth0)
• Option -s for the subnet mask of the network
• Option -h for the host name
• Option -d to deregister

For example, first deregister the VDS host using the -d option, then initialize the VDS host.

Step 6 If you still have trouble getting the tables initialized, then log in to the CDSM GUI with an engineering access user account and add the Streamer or Vault by clicking the Maintain > Software > System Configs page and using the Add New Server option.

Database Monitoring

To monitor a stuck database thread problem, use the following command:

Two connections for each Vault and Streamer should be listed with a status of “ESTABLISHED.” If two-way connection does not exist, run db_shutdown on all servers including the CDSM, then start the database using the following commands:

If the database is stuck and db_shutdown does not take effect, use ps –ef |grep avsdb to query the process ID (PID), then use the kill -9 { pid } command to kill the avsdb process, and lastly restart the database.

Ingest Interface

This section includes the following topics on troubleshooting the ingest interface:

• FSI Starts but Cannot Ingest Content
• FSI Does Not Start
• Content Not Ingesting
• Asset Ingest Manager Restart Causes Endless Loop
• Asset Ingest Manager crashing

FSI Starts but Cannot Ingest Content

Cause 1: FSI mode selected is incorrect.

Action 1: Check the FSI mode on the server by viewing the /arroyo/log/fsi.log when FSI is started.

The FSI running mode should be what you set in /home/isa/bss/scripts/arroyo-site-env.sh; for example, set “FSI_DEPLOYMENT=1.”

FSI Does Not Start

Cause 1: Invalid port address.

Action 1: Change the FSI port to a nonzero value (default=20004).

Using the CDSM GUI, choose Configure > Server Level > FSI Setup and enter a new port number in the FSI Server Port field. See the “Configuring FSI Setup” section for more information.

Action 2: Log in to the server as isa and start the FSI service.

Cause 2: Cannot open cache2app reported in fsi.log and fsi.err.

Action 1: Verify that the CSserver process is running.

Log in to the server as root and enter the lsmod command to check if the avs_cserver is running. If the avs_cserver is not running, enter the rpm –qa | grep cserver command to see if it is installed.

• If avs_cserver is installed, enter the /arroyo/test/run command to start avs_cserver.
• If avs_cserver is not installed, call Cisco Technical Support.

Content Not Ingesting

Cause 1: Cannot connect to the FTP server (for example, Lysis Content Ingest Server).

Action 1: Check the network connectivity and user access.

Check the network connectivity, access control lists (ACLs), user access privileges, and so on for the FTP server.

Action 2: Verify the manual FTP request.

Verify that a manual FTP request to the FTP server from the user isa is working and list the content to be ingested.

Cause 2: Content is not correctly named in the ADI XML file.

Action 1: View the aim.log file.

In this example, 6663_3800p.mpg does not exist on the FTP server. Make sure the FTP server content matches the content listed in the ADI XML file.

Cause 3: The Asset Ingest Manager (AIM) is not getting a message from the FTP server.

Action 1: Do the following:

1. Identify the master Vault by looking at the /sbin/ifconfig file and locating the eth0:1 interface.

2. On the master Vault, enter the tail –f aim.log command.

3. Retry the transfer from the FTP server.

4. If there is a connectivity issue, the aim.log is not updated.

5. Restart the AIM process by entering the arroyo stop aim command followed by th e arroyo start aim command.

6. If necessary, restart the FTP server.

Cause 4: Asset Name larger than 64 bytes.

Action 1: Do the following:

1. Add a line ASSETNAME_64BYTE_TRUNCATION=1 in /home/isa/bss/etc/aim.conf file.

2. Restart AIM.

Cause 5: Cannot find the file in the FTP server.

Action 1: Verify the following:

While ingestion of a VOD asset, VDS-TV is provided with an FTP URL and the FTP URL path must be considered as a relative path instead of an absolute path.

For example,

Content URL : ftp://user:pass@ftphost/directorypath/filename.ts
After FTP login and transfer mode settings, the ingest system is expected to GET this file from directorypath/filename.ts and NOT /directorypath/filename.ts

Asset Ingest Manager Restart Causes Endless Loop

Cause 1: AIM has orphaned threads.

Action 1 : Check the aim.log for endless looping message.

Action 1: Check for orphaned AIM threads during the AIM service being stopped.

Log in as isa and stop the AIM service, verify there are no orphaned AIM threads, and restart the AIM service.

Cause 2: The master Vault AIM is in a degraded state.

Check the aim.log on the slave Vault to verify it is in a waiting state.

Action 1: Cause a failover between the master and slave Vaults.

On the master Vault, stop the statsd and Ingest Manager (AIM) services, thereby causing Vault failover. The AIM service on the new master Vault should immediately start processing ingest requests.

On the old master Vault, log in as root and restart the statsd service, log in as isa and restart the AIM service. This Vault becomes the new standby and the AIM process is in waiting mode.

Asset Ingest Manager crashing

Cause 1: AIM content path length greater than 128 bytes

Action 1: Verify that the maximum length of the absolute path location of an asset is 128 bytes.

Bad Content

Variable bit rate (VBR) encoded content is not currently supported. See the “VDS Content Quality Guidelines” section for constant bit rate (CBR) guidelines.

Network

Ensure that the network maximum transmission unit (MTU) is appropriately set. If jumbo frames are enabled on the VDS, then the network must support jumbo frames. We recommend that the network support jumbo frames even when the jumbo frame option is disabled.

If a Layer 2 network is used for VDS, then appropriate MAC addresses (ARP entries) have to be configured on the switches and routers. Ensure that the VDS Vault and Streamer interfaces are in the same VLAN. If a Layer 3 network is used for the VDS, then ensure that the corresponding default gateways are correctly configured on VDS Vaults and Streamers for the various interfaces and Stream Groups.

Ensure that the content source (catcher, FTP server, and so on) is reachable from the Vaults or ISVs, and that manual content transfer using FTP works correctly.

For more information about the status of the network interfaces, network routing tables, ARP and so on, see the “Network” section.

Listing Content

To view the actual stored content versus what the database reports, enter the following commands:

Orphan GOID mirroring during reboot of vaults

In steady state with no new contents being added or removed, if constant mirroring activity is seen during reboot of any group of vaults system then:

Enter the following command in all the vaults:

Content Mirroring

To enable content mirroring locally on one Vault, do the following:

Step 1 Modify the /home/isa/.arroyorc file by adding the following line:

Step 2 Verify that the change has propagated to the /arroyo/test/vault/setupfile file.

The line “vault local copy count 2” should be added to the setupfile file.

Alternatively, enable local mirroring using the tunables. You can also use the tunables to verify the settings.

Note Using the echo 2 command to enable local mirroring in the tunable file only changes the local copy count temporarily. The local copy count resets to its original value on reboot. To configure the local copy count permanently for any value other than 1, edit the /arroyo/test/vault/setupfile or use the CDSM GUI.

To enable content mirroring between two Vaults, do the following:

Step 1 In the CDSM GUI, choose Configure > Server Level > Server Setup . The Server Setup page is displayed.

For more information, see the “Configuring the Servers” section.

Step 2 From the Server IP drop-down list, choose the IP address of the server.

Step 3 From the Vault Mirror Copies drop-down list, choose 2 .

Step 4 Click Submit .

Step 5 Verify the change has propagated by looking at /arroyo/test/vault/setupfile and /arroyo/log/protocoltiming.log.< date > files.

Verifying GOIDs

You cannot verify that the global object identifiers (GOIDs) among Vaults and Streamers are correct by comparing the total number of GOIDs on each server. There are actually multiple chains of GOIDs. If you list the GOID chains you can verify that the GOIDs are correct, because listing the GOIDs provides a summary at the end of the listing that reports any issues.

To list the GOIDs, enter the following command:

To list all GOID chains, enter the following command:

The /arroyo/log/serverinfo.log.< date > log file contains information about the GOIDs.

Note There is no need to identify and delete damaged or orphaned GOIDs. CServer repairs any damaged GOIDs. Orphaned GOIDs are deleted when the server reboots.

Trick-Mode Issues

Verify the trick-mode settings in the CDSM GUI and the Vault setupfile file.

• From the CDSM GUI, choose Configure > System Level > Ingest Tuning to view the trick-mode settings.
• To check the trick-mode setting in the setupfile on the Vault, enter the following command:

Check the /arroyo/log/c2k.log.< date > log file and the session message logs during playout to verify that the trick-mode files are being streamed.

Tracking Cache-Fill Source

You can track whether or not a GOID for a stream is filling remotely or locally by enabling and tracking it in the fill.log. Streams can share the same GOID; in which case it is not possible to tell which stream is currently filling the data.

To track the cache-fill source of a stream, do the following:

Step 1 Find the stream that is playing in the c2k.log on the Streamer, along with the content that was requested (GOID number).

Step 2 Enable the fill.log on the Caching Nodes the Streamer is mapped to.

Step 3 On the Caching Nodes, use the tail command to follow the log and grep for the GOID.

Following is an example fill.log:

Burst and delay times are in microseconds. Bursts are sent immediately at a high rate. The delay time specifies when to start sending the data at rate, up to 30 seconds into the future.

If no Caching Nodes are reporting fill for the GOID, then the content is being filled from memory.

Step 4 Disable the fill logs on the Caching Nodes when finished.

Rules for ISV Interoperability with Vaults and Streamers

The following rules apply for ISVs to interoperate with Vaults and Streamers:

• An ISV can cache-fill both a colocated Streamer and a dedicated remote Streamer.
• An ISV at one location cannot cache-fill a Streamer associated with an ISV at another location.
• Two ISVs can mirror content with each other, but an ISV and a Vault cannot mirror content with each other.
• A Vault cannot cache-fill an ISV.

Network

Note For more network troubleshooting methods, see the “Network” section.

Stream Stops Playing at the Same Place or Does Not Play at All

Cause 1: Jumbo frames are not supported or configured on the cache-fill network switch.

Check 1: Search the c2k.log file and the rtsp.log file for content read errors.

Check 2: Ping between the two devices.

Ping between the two devices on the cache-fill VLAN using a packet size greater than 1500 bytes.

Action 1: If the ping fails, verify that jumbo frames and cache-fill interfaces are configured correctly.

Verify that jumbo frames are enabled on the switch ports for the cache-fill VLAN, and verify that the cache-fill interfaces are configured correctly on the Streamers and Vaults. See the “Configuring the Servers” section for information on configuring the cache-fill interfaces.

Listing of Streams

To monitor streams based on various criteria, go to the Stream Monitor page in the CDSM GUI by clicking Monitor > System Level > Stream Monitor . For more information, see the “Monitoring Stream Objects” section.

No Streaming

Some common causes for streaming problems are the following:

• Server is in the process of being offloaded.
• QAM device has no available bandwidth.
• Tuning failure because of one of the following:

– Error in the ARP table

– QAM device is down

– Network problem

• back office is out of synchronization with the VDS ContentStore, resulting in content not being found.

Stream Not Playing

The rtsp.log file has the entry, “error reading content data.” This means that a callback was received from the CServer with a completion code of 3.

Cause 1: A piece of the content is missing.

In this case, a user can typically stream part of the content, but at some point, the stream stops and the error message is returned in the ANNOUNCE message. The content needs to be validated at the CServer level.

Action 1: Set up a stream to play to a multicast address.

If this is successful, then there is a network issue, which is either a default gateway or unreachable remote client. You can verify whether it is successful by looking at the /home/stats/ifstats file.

Action 2: If ifstats information does not detect a problem, try streaming to another multicast IP address.

Repeat streaming to a multicast address with different content and, if possible, ingest known good content. Check the protocoltiming.log.< date > for damaged GOIDs by using the following command:

Cause 2: There is a problem reaching the destination QAM device.

The CServer returns the same completion code, so the same error is returned in the announce message. In this case, the content does not stream at all. The play request and play response are separated by about 10 to 15 seconds, instead of the typical subsecond separation. This is because of the ARP timeout process the CServer is going through to reach the destination. After stream response fails, the CServer calls back with the completion code of 3, which causes the “error reading content data” message.

Action 1: Check that the interfaces involved in the streaming are up and operating at the correct speed.

Using the CDSM GUI, choose Monitor > Server Level > NIC Monitor , choose the IP address of the server, and verify the participating interfaces are up and operating at gigabit Ethernet speeds. For more information, see the “NIC Monitor” section.

Action 2: Set up a stream to play to a multicast address.

If this is successful, then there is a network problem, which is either a default gateway or unreachable remote client. You can verify whether it is successful by looking at the /home/stats/ifstats information.

Action 3: If streaming to a multicast address is not successful, check that the Vaults can be reached.

Check the /arroyo/log/protocoltiming.log.< date > log file for the number of reachable remote servers. Additionally, if there is a cache-fill issue, you will see a large megabit value for the re-xmit buffer.

You can also check the /arroyo/log/c2k.log.< date > log file for any unreachable Vaults.

Cause 3: The c2k.log file reports “no streamer available, out of capacity.”

The protocoltiming.log file reports “remote vaults 0 caches 0.” This means the Streamer and Vault have lost connection with each other.

Action 1: Check the route configured on the servers.

Use the cat /arroyo/test/RoutingTable command to verify correct route table entries.

Check the routing table using the CDSM GUI by clicking Configure > Server Level > Route Tables . For more information, see the “Configuring the Route Table” section.

Action 2: Use the cat /arroyo/test/RemoteServers to check if the remote server is configured correctly.

Action 3: Check the routing table on the switch or router.

Poor Video or Audio Quality

This section includes the following issues that result in poor video or audio quality:

• No Video Displayed
• RTSP Cannot Start
• Video Stops Playing
• No Video Playing on the set-top
• VDS Is Streaming but No Video Is Playing on the set-top

No Video Displayed

When content is streamed to a client device, if there is no video picture displayed on the client device and the audio is working fine, use the following troubleshooting methods:

• Verify that the source is working properly and that the original content is of good quality.

– Verify that the appropriate bit rates are being sent from the server using the following command on all Streamers:

– Verify that the content plays locally, and on a test client device (for example, a VLC client).

– Test playing the content on an alternate player with an AVC plug-in.

• Verify that the VDS is configured correctly.

– Check the run script in the /arroyo/test/run directory. There is a tunable set for Telenet to stream null packets when the end of the stream is reached. This should be commented out or removed in a non-Telenet environment.

– The interface that you are using for real-time ingests needs to be configured for the CServer. There are a couple of settings that define the interrupt for the real-time ingest interface and ensure that a single central processing unit (CPU) is responsible for receiving the packets for the ingest. Without these settings, packets can be out of order, which can cause problems with the video picture.

To fix this, use the cat /proc/interrupts command to display the interrupts and find the interrupt value associated with the interface you are using for ingest. After you know this value, add the following lines to the /arroyo/test/run script:

You can enter these lines at the Linux command line as well. By doing so you do not have to reboot your system for them to take effect. Any content that you have previously ingested should be considered invalid.

RTSP Cannot Start

The owner and group permission of the RTSP binary file are incorrect. To correct this, enter the following commands:

Video Stops Playing

If the video stops playing after a fixed interval (for example, five minutes), check to see that the interval the video played for matches the session inactivity timeout setting in the CDSM GUI Configure > Server Level > RTSP Setup page.

If the interval matches the setting, check the /arroyo/log/rtsp.log file for the set-top responses within the inactivity timeout interval.

This scenario could be caused by the set-top possibly not responding, or the set-top response may be getting lost in the network.

No Video Playing on the set-top

Check the /arroyo/log/rtsp.log file on the relevant Streamer. If you see a successful setup request and response, and do not see a play request for the same session, there is some issue with the set-top communication to the RTSP server. The RTSP server should also be receiving a get_parameter request as a heartbeat to keep the session alive. If this does not happen, the session is torn down after the session inactivity timeout is exceeded, and a “session timed-out” message is sent.

Following is an example of the rtsp.log file showing the setup, response, teardown, and timeout messages.

VDS Is Streaming but No Video Is Playing on the set-top

Check the /arroyo/log/rtsp.log file on the relevant Streamer for any of the following message flows:

• Setup request is sent from Quative, followed by setup okay response returned from VDS.
• Describe request is sent from the set-top, followed by describe ok response and details returned from VDS.
• Play request is sent from set-top, followed by play ok response sent from VDS.
• Teardown request is sent from set-top, followed by teardown performed by VDS.

The possible causes for the above message flows are the following:

• set-top is not tuning to the correct frequency or program ID, or the session resource manager (SRM) is returning incorrect information.
• SRM is providing the wrong edge QAM device to the VDS. In the rtsp.log file, check the IP address in the setup RTSP header.
• Edge QAM device port mapping may be wrong. Check the edge QAM device configuration.
• Radio frequency (RF) is not reaching the set-top.

Following is an example of the rtsp.log file showing the request and response messages.

Log File Search Tips

To search for general RTSP errors, use the following commands:

To search for content read errors, use the following command:

To search for stopped sessions, use the following command:

To search for SETUP requests, use the following command:

Use the CSeq header value in the rtsp.log file to match RTSP requests to responses.

Use the Session header value in the rtsp.log file to trace a single RTSP session from setup to teardown.

Codes

The ANNOUNCE method is a mechanism for RTSP servers to signal RTSP clients about start of stream or end of RTSP session events. An ANNOUNCE request must include a “CSeq” header and “Notice.” Following are the Notice codes:

• 1103 Playout Stalled (from VOD server only)
• 1104 Playout Resumed (from VOD server only)
• 1500 New Scale (from VOD server only)
• 2101 EOS (end of stream) (from VOD server only)
• 2104 BOS (beginning of stream); can happen in case of rewind or reverse play (from VOD server only)
• 2401 Ticket Expired; the playout has stopped (from the MS server only).
• 4400 Error Reading Content Data; the playout has stopped (from VOD server only); from the Streamer to the client referring to a hole in the content or a delay in getting the content
• 5200 Server Resources Unavailable; the playout has stopped (from MS server only)
• 5402 Client Session Terminated; teardown has been initiated by the server, the session is closed (from MS server only)
• 5403 Server Shutting Down; the playout has stopped. If from MS server, the session is closed. If from VOD server, the client must issue a teardown of the session (first configuration only).
• 5502 Internal Server error; the playout has stopped. The client must issue a teardown of the session.

After receiving and interpreting a request message, the recipient responds with an RTSP response message. The status code returned in the RTSP response message must be in the range 100 to 599 (per RFC 2326). If the server returns another value, the client must treat the error according to the range value:

• 1xx: Informational—Request received, continuing process.
• 2xx: Success —Action was successfully received, understood, and accepted.
• 3xx: Redirection—Further action must be taken to complete the request.
• 4xx: Client Error—Request contains bad syntax or cannot be fulfilled.
• 5xx: Server Error—Server failed to fulfill an apparently valid request.

Table A-3 describes the RTSP status codes.

Database Replication

This section covers the following database issues:

• CDSM GUI Does Not Report All the Ingested Content
• Errors in Log File
• Many Log Files

CDSM GUI Does Not Report All the Ingested Content

First, verify that the package has not already expired.

Second, check for index errors in the CDSM database logs, using the following command:

The example output indicates that the Vault and CDSM databases are not synchronized, possibly because of the server times not being synchronized, a network connectivity issue, a server failure, or some other similar issue.

For resolution, see the “Corruption Recovery” section.

Errors in Log File

If the error “ FSI_ENV:XX:YY was not found in the database” occurs repeatedly in the logs (see the following example), there could be two causes for this issue.

Cause 1: The databases are not synchronized.

If you are sure you entered an FSI configuration into the CDSM and the FSI is still giving this message, the configuration did not get replicated to the server where the FSI process is running.

Cause 2: The FSI configuration not existing in the database.

The FSI process can run without an existing database configuration by using its internal defaults.

Many Log Files

If one of the following conditions exist, it indicates that there were database replication errors:

• Database could not be started. See the “Services Monitor” section for more information.
• Bidirectional connections are lost between servers. See the “Required Services Not Starting or Running Correctly” section.
• The following error message is listed in the /arroyo/log/avsdb-err.log. yyyyMMdd file:

• The /home/isa/Berkeley/DATADIR/REPLAY.db file increases to several GB.

For resolution, see the “Corruption Recovery” section.

Corruption Recovery

If the CDSM database is corrupted and the Vault database is not corrupted, do the following:

Step 1 As user root , stop the CDSM database.

Step 2 Confirm that the database is shut down.

Make sure there is no avsdb process returned. If the avsdb hangs, use the process ID (2646 in the above example) with the kill command.

Step 3 Delete all files in the /arroyo/db/DATADIR directory.

Step 4 As user root , stop the Vault database.

Step 5 Confirm that the database is shut down.

Step 6 Copy all files in /arroyo/db/DATADIR directory from the Vault to the CDSM.

Step 7 As user root , restart the Vault database.

Step 8 As user root , restart the CDSM database.

Step 9 Check the configuration on the CDSM and make sure no configuration parameters were lost.

Live Multicast Ingest

Live multicast ingest is available as part of the Media Scheduler feature or the Real-Time Capture feature.

Ingest with Media Scheduler

Using Media Scheduler for live multicast ingest requires the following procedures:

1. Enable live ingest by setting both the Media Scheduler and the Ingest Manager to ON in the CDSM Setup. See the “Initializing the VDS and Activating the Optional Features” section for more information.

2. Use the CDSM Input Channels page to configure the input channels. See the “Configuring Input Channels” section for more information.

3. Upload channel schedules by importing the electronic program guide (EPG). See the “Uploading an EPG File” section for more information.

Ingest without Media Scheduler

Using Real-Time Capture for live multicast ingest requires the following procedures:

1. Enable live ingest by configuring Ingest Manager ON and setting Real-Time Capture Type to Real-Time Capture (non-Media Scheduler) in the CDSM Setup page. Activate the Ingest Manager. Because the Ingest Manager is an optional feature, an activation key is required. See the “Initializing the VDS and Activating the Optional Features” section for more information.

2. Use the CDSM CallSign Setup page to configure call signs with multicast IP addresses.

Ingest Troubleshooting

If the message “ERROR: Unable to login to the ftp location,” is present in the /arroyo/log/aim.log file, check the FTP server configured in the Ingest Manager by using the ps -ef | grep ftp command. If the FTP service is not running, enter the service vsftpd start command to start it.

Ingest Troubleshooting with SSM enabled

For a live recording with real time capture type and source specific multicast support enabled, while recording if there is a primary source (feed) problem recording vault will failover to secondary source(if available). Default timeout for this source failover is 30 seconds.

This value can be adjusted with a tunable /proc/calypso/tunables/recording_nodatatimeout up to 10 seconds.

Reliability and Availability

Q. How do I enable stream resiliency?

Log in to the CDSM with engineering access. The CDSM Setup page is displayed. For Stream Failover Support, choose “ON” and click Submit . For more information, see the “CDSM or VVIM Setup” section.

Q. How do I check and make sure the database is running properly?

After starting the database, you should see two sockets (listening and non-listening) connecting to the database on each of the remote servers on port 9999. You can check them by using the netstat -an | grep 9999 command.

For example, the following output of the netstat command shows that the server (172.22.97.194) has both the listening and non-listening sockets binding on port 9999 to echo the four remote servers (172.22.97.192, 172.22.97.193, 172.22.97.195 and 172.22.97.191).

If you can not see both listening and non-listening sockets binding on port 9999 for each of the remote servers, the database is not running properly. Check that you have the correct replication group members in your /home/isa/.arroyorc file.

Serviceability and Manageability

Q. How do I check the calypso server status?

Log in to the server as root and enter the cat /proc/calypso/status/server_settings command.

Q. How do I check central processing unit (CPU)?

Log in to the server as root and enter the cat /proc/cpuinfo command.

Q. How do I check the kernel network driver version?

Log in to the server as root and list the e1000.ko file to check the date and time it was created using the following command:

The following example shows that the e1000.ko file is based on the kernel 2.5.18-53.el5.kernel.2_6_18.2009.01.08.01.

Q. How do I stop, start, and, restart the Apache server on the CDSM?

Log in to the server as root and enter the following command:

Q. How do I check the Streamer static ARP table?

Log in to the server as root and enter the following command:

Q. How do I view the ARP Table dump file?

# echo 1 > /proc/calypso/test/arp_dumpstate

Q. How do I recover the system from the kernel debugger (KDB) after a reboot?

If the server starts the KDB tool instead of rebooting, modify the /etc/grub.conf file as follows:

Q. What do I do if the KDB prompt is displayed when the server restarts after a failure?

Boot into single user-mode (see the “Kernel Crash” section).

Q. How do I identify any holes in the content?

Log in to the server as root and enter the following commands:

Look at the last two lines of output. If there are no holes, the last two lines should be the following:

Q. How do I clear cached video blocks (data cache) on the Streamer?

Log in to the server as root and enter the echo 1 > /proc/calypso/test/clearcache command.

Q. How do I clear the data cache in memory?

Log in to the server as root and enter the e cho 1 > /proc/calypso/test/clearmem command.

Note Make sure there are no streams running before you use this command. If there are streams, the data cache in memory is not cleared.

Q. How do I destroy all streams?

Log in to the server as root , stop the services, change to the database table directory, remove the session table, and restart the services.

All sessions are removed, and upon restarting the services, all streams that do not have an associated session are stopped.

Q. How do I delete an individual stream from the database?

Log in to the server as root and enter the following commands:

Q. How do I destroy all streams when none of the above methods work?

Log in to the server as root and enter the following commands:

Wait for all avs processes to stop, then reboot the server.

Q. How do I check the routing table and gateway?

Log in to the server as root and check the file /arroyo/test/RoutingTable.

Content

Q. How do I get information on a content stream that seems corrupted; for example, there is macroblocking, the stream stops and restarts, and so on?

Log in to the server as root and enter the following commands:

Check the last set of output lines to see the current content states.

Q. How do I know if a content object has “holes”?

Log in to the server as root and view the /var/log/debugmessages. There is a message in the debug messages file about the GOID and the content holes.

Q. How do I delete ingests that are “stuck” in the active ingest state?

Log in to the server as root and enter the following commands:

Choose the option 1 (CONTENT). In the next menu, choose option 1 (DELETE CONTENT). Enter the content ID of the “stuck” ingest, then choose the exit option for each menu until you are back at the Linux prompt.

Q. How do I manually ingest content from the command line?

Log in to the server as root and enter the following commands:

Update the SDClient.cfg file with the local IP address.

Follow the SDClient menus.

Other

Q. How do I view the CServer code configuration file?

Log in to the server as root and enter the cat /arroyo/test/<server type>/setupfile command. The server type is one of the following: vault, streamer, or ssv.

Q. How do I know if a subsystem on a server is overloaded?

View the .arroyo.log.protocoltiming.log.< date > file. When you see the “COST REQUEST NO CAPACITY:” message, it means that the server is running out of capacity and it cannot accept new streaming requests.

Also, when you see a line in the /arroyo/log/c2k.log.< date > file that says the following:

This entry typically means there are no stream ports linked. In the ten-second snapshot of the /arroyo/log/protocoltiming.log.< date> file, there is a line that shows load values for each of the major subsystems (LAN, memory, CPU, and so on). More than likely one of the subsystems is at 100, which is the subsystem that is having the problem.

Q. How do I enable debugging?

Log in to the server as root and enter the following commands:

Q. How do I update the remote servers from /arroyo/test/RemoteServers?

Log in to the server as root and enter the following commands:

Supported Elementary Stream Types

Video-only, audio-only (as well as audio streams with only a few or occasional video frames) and data-only streams are supported in addition to the customary multiplex of both audio and video.

Scrambling

The transport layer cannot be scrambled, meaning the transport header and any adaptation field must be in the clear. Streams whose Elementary Streams (ESs) are fully scrambled, including all start codes, are capable of being ingested and streamed, but are incapable of trick play.

For trick-play capability, the following cannot be scrambled:

• Packetized Elementary Stream (PES) headers
• Program Association Table (PAT) and Program Map Table (PMT)
• Closed-caption data (if scrambled, the data is incorrectly included in tricks)

Transport Bit Rate

All transport streams are constant bit rate (CBR). Variable bit rate (VBR) is not supported. The maximum bit rate is 30 Mbps. There is no minimum bit rate. The ES video bit rate, as specified in the MPEG-2 sequence header, is ignored. The bit rate of an individual ES is not significant. The aggregate transport stream must have a constant bit rate. An individual ES (particularly, the video ES), however, does not have this bit rate requirement.

Streams containing MPEG-2 or AVC video are expected to conform to the appropriate buffer models spelled out in ISO/IEC 13818-1 and 14496-10.

Stream Length

All content must be at least one second in length. If ingested as a single item, the content must be under 12 hours in duration or under 30 GB in size, whichever occurs first. If the content is chunked, the overall size of the content must be under 120 GB, while there is no restriction on the duration of the content.

Format Restrictions

Following are the format restrictions for Advanced Video Coding (AVC), H.264, and MPEG-4:

• Sequence Parameter Set (SPS) seq_parameter_set_id flag must be zero.
• SPS pic_order_count_type flag must be zero.
• SPS seq_scaling_matrix_present_flag must be zero.
• SPS profile_idc flag must only be Baseline, Main, or High profile.

Preferred Formats

Using the following guidelines improves the performance of the system, the quality of the tricks, and the trick transitions.

1. All content should be encoded as a Single Program Transport Stream (SPTS). If multiple programs must be included (for example, a Picture-in-Picture (PIP) stream), ensure that the “real” program is encoded with the lowest program number.

2. All content should follow the process ID (PID) numbering specified in the Content Encoding Profiles 2.0 Specification (MD-SP-VOD-CEP2.0-I02-070105), section 6.7.5. Regardless, the audio and video PIDs should be above 0x20.

3. All content should be preceded with a Program Association Table (PAT) and then a Program Map Table (PMT), and then a Program Clock Reference (PCR) before the first audio or video frame. Optionally, the discontinuity bit can be set.

4. All content should use the same PID for both PCR and video.

5. All content should begin with a closed Group Of Pictures (GOP) for MPEG-2 or with an Instantaneous Decoder Refresh (IDR) frame for AVC. This first frame is always accompanied by a sequence header for MPEG-2 or by an SPS for AVC.

6. To guarantee relatively smooth looking trick modes, the minimum I/IDR-frame frequency should be eight per second. If the minimum trick speed is 4x or less, the I/IDR-frame frequency should be at least two per second. In no case should two I/IDR frames be more than two seconds apart.

7. Each I-frame should be preceded by a sequence header and GOP header if any exist for an MPEG-2 video. Each I/IDR frame should be preceded by an SPS and Picture Parameter Set (PPS) for H.264 video.

8. Avoid mixing frame data from multiple video frames in the same transport packet. Specifically, no data belonging to the prior frame exists following the Packetized Elementary Stream (PES) packet header for the next frame. Breaking this rule may improve encoding efficiency slightly, but degrades the quality of the tricks on certain set-tops.

9. All content must be encoded as a single sequence, with no changes in horizontal or video resolutions, or changes in encoding parameters in the middle of the content.

10. The GOP size may be variable, but GOPs should generally not exceed two seconds. Using longer GOPs may improve encoding efficiency, but the quality of lower-speed tricks (3x, 4x) may suffer.

11. No more than four B-frames should be used between each pair of I-frames or P-frames.

12. There should be no continuity counter errors in the content.

13. There should be no discontinuities in the content, other than an optional one on the first PCR.

14. The accuracy requirements for PCRs, +/–five parts per million (5 ppm), as stated in ISO/IEC 13818-1, must be adhered to throughout the stream.

15. Audio and video are expected not to overflow the appropriate target buffer model specified.

16. A reasonable bit rate to use when encoding MPEG-2 standard definition (SD) video is 3.75 Mbps.

17. A reasonable bit rate to use when encoding MPEG-2 high definition (HD) video is 15 Mbps.

18. Appropriate bit rates for carriage of AVC are still being established, and while they are expected to be at least half the bit rates of MPEG-2, no specific recommendations can be offered.

19. There may be PIDs in the content that are not specified in the PMT. Such use is beyond the scope of this document.

20. All PATs and PMTs should be identical, with the same version number throughout.

21. The VDS support up to 30 Mbps MPEG-2 video encoding.

22. Content is filtered out if three occurrences of one-second synchronization lost are identified.

23. Content is filtered out if five seconds of null frames are identified.