-
Cisco TV CDS 3.0 ISA Software Configuration Guide
-
Preface
-
Product Overview
-
Network Design
-
Getting Started
-
Configuring the CDS
-
System Monitoring
-
System Reporting
-
System Maintenance
-
Troubleshooting
-
Creating Bulk Configuration Files
-
BMS Communication
-
SNMP MIB and Trap Information
-
Using the TV CDS Streamer Application Monitoring Tool
-
Engineering Access Level Pages
-
Software Licensing Information
-
Feedback
Table Of Contents
CDS Server Cannot Register with OpenStream
OpenStream Reports Alert Messages
General Information and Issues
Vault, Streamer, Caching Node, and ISV
Description of the .arroyorc Settings
Description of the setupfile Settings
Identifying the Software Versions or Releases
Using ifstats to Monitor Traffic
CDSM GUI Disk Monitor Page Reports a Disk Warning
Disabling USB Ports and Password-Protecting the BIOS
Vault Cannot Connect to FTP Server
Checking Network Configuration
BIOS Settings—Operating System Hangs or Goes into KDB Mode
Required Services Not Starting or Running Correctly
Management and Database Issues
CDSM GUI Does Not Register the Vaults and Streamers
CDS Is Not Registered to the Name Service
Rules for ISV Interoperability with Vaults and Streamers
Stream Stops Playing at the Same Place or Does Not Play at All
CDSM GUI Does Not Report All the Ingested Content
Advanced Features and Applications
Ingest without Media Scheduler
Serviceability and Manageability
CDS Content Quality Guidelines
Supported Elementary Stream Types
Troubleshooting
This appendix presents troubleshooting procedures for the CDS by showing the symptoms, probable causes, and recommended actions for a variety of issues. The topics covered in this appendix include:
•
General Information and Issues
•
•
•
There are a variety of possible combinations of CDS topologies, backoffice environments, middleware, and so on. The engineers using this troubleshooting appendix are expected to know their system well enough that they can extrapolate the relevant troubleshooting guidelines. With all connectivity issues, physical integrity of cables and ports should be verified, as well as VLAN configuration if applicable.
All Linux commands described in this appendix require console access to the server, or Secure Shell (SSH) access to the server.
Caution
Note
OpenStream Issues
Following are the common connectivity problems with the OpenStream system:
•
•
CDS Server Cannot Register with OpenStream
This issue could manifest in the following ways:
•
CorbaEventSupplier.cc|158|Event publishing.........Done.•
–
–
–
•
Cause 1: Configuration mismatch between CDS and OpenStream.
The IP address, port, and ISA names configured on the CDS do not match those configured on the OpenStream system.
Action 1: Verify the CDS configuration.
Verify the following settings on the Streamer BMS page and Vault BMS page:
•
•
•
Cause 2: The ISA services are not running.
Action 1: Check the ISA services.
Check that the ISA services are running on the CDS server by going to the Monitor > Server Level > Services Monitor page. If they are not running, restart the services by going to the Maintain > Services > Services Restart page and restarting the ISA-OpenStream service.
OpenStream Reports Alert Messages
OpenStream reports the following alerts:
•
•
•
•
•
Cause 1: CDS is unable to register with OpenStream
Action 1: See the "CDS Server Cannot Register with OpenStream" section.
Cause 2: The Content Store does not have the correct FTP port setting.
Action 1: Verify FTP server and client settings.
Verify the FTP server and client port settings are correct by going to the Configure > Array Level > Vault BMS page.
Cause 3: The package information is incorrect.
Action 1: Verify the URL, hostname, file path, filename, username, and password.
General Information and Issues
This section describes the CDS file system, log files, configuration files, and general troubleshooting methods. This section includes the following:
•
•
•
File System
The CDSM file system differs from the file system on the CDS servers (Vault, Streamer, Caching Node, ISV).
CDSM
The CDSM has the following directory structures:
•
•
•
•
•
•
•
•
Report Archiving
The CSV files are generated every 24 hours and are deleted when they are older than 30 days. The CSV files are stored in the /arroyo/asmrpt directory. For the CSV files to be generated, the report archiver needs to be installed and configured. The CSV files are accessible by going to the /arroyo/asmrpt directory, or by using an FTP client with the username "asmrpt" and the password "asmrpt."
Vault, Streamer, Caching Node, and ISV
The CDS servers (Vault, Streamer, Caching Node, and ISV) have the following directory structures:
•
•
•
•
•
In addition to the above directories, the CDS servers have the following directories specific to the ISA environment:
•
•
•
•
The log files are located in the /arroyo/log directory. A log file is automatically archived and moved to the /arroyo/archive directory when it reaches close to 100 MB in size.
Log Files
There are three types of log files in an ISA environment:
The log files are rotated at least once a day and time stamps are added to the filenames. Some log files that grow rapidly are rotated more frequently (determined by file size); this rotation may happen up to once an hour. Most log files have the following suffix: .log.<YYYYMMDD.> The time zone for log rotation and filename suffixes is coordinated universal time (UTC).
The CServer log files are automatically archived and moved to the /arroyo/archive directory when the disk storage reaches a certain level. The ISA log files are automatically archived and moved to the /home/isa/bss/log/archive directory whenever the log file reaches close to 40 MB. A total of nine revisions are kept of each log file, with the eight oldest being compressed and moved to the archive directory.
To change the log level or set the debug flags for the log files, use the Configure > System Level > Logging and Configure > Server Level > Logging pages. For more information, see the "Configuring the System Level Logging" sectionand the "Configuring the Server Level Logging" section.
The following log tools are available:
•
•
Linux Log Files
The Linux operating system has the following useful log files:
•
•
CServer Log Files
The CDS has the following useful log files:
•
# echo "6" > /proc/calypso/tunables/c2k_verbosedump•
•
•
•
•
•
Other CServer log files that may be useful are the following:
•
•
•
•
•
•
•
•
•
Note
CServer Error Codes
CServer error codes that appear in the c2k.log.<date> file do not necessarily mean an error has occurred. An actual error has "err" listed in the entry, as opposed to "out" or "ntc." Following is a list of important CServer error and status codes:
Error Codes
•
•
Status Codes
•
•
•
•
•
•
•
•
•
•
Protocoltiming Warning Messages
Table A-1 describes some of the warning messages that might be seen in the Protocoltiming log.
ISA Log Files
The Linux user isa is the owner of the application files in an ISA environment. Logging in as isa starts the database. To change from the root login use the su - isa command.
The following log files provide information on content ingest and mirroring:
•
•
The following log files provide information on streaming:
•
•
•
The following log file provides services monitoring and registration to the NameService:
•
CDSM Log Files
The CDSM has the following logs:
•
•
•
•
All log files use UTC for the log entry time stamps and filenames. All four files are located in /arroyo/log directory.
The default log level for httpd.log is LOG_WARNING and the setting is preserved. The log level for httpd.log (facility is httpd) can be configured by using the /home/isa/logging/logconfig tool. The logconfig tool overwrites the LogLevel value in the httpd.conf file with the new value and the /arroyo/www/bin/apachectl restart command is issued.
The httpd_access.log (facility is httpd_access) is always on and the log level cannot be changed with the logconfig tool.
Following is an example of a log entry in the httpd_access.log file:
02-09-2011 15:44:09.937115 UTC vqe-dev-29 161.44.183.124 - - [02/Sep/2011:08:44:09 -0700] "POST /includes/configGrpSubmitAjax.php HTTP/1.1" 200 12Following is an example of a log entry in the httpd.log file:
2-09-2011 15:45:07 UTC vqe-dev-29 [notice] Apache/2.2.9 (Unix) PHP/5.2.6 configured -- resuming normal operationsServer Configuration Files
The server configuration settings are stored in the .arroyorc file and the setupfile file. This section describes the different parameters for each file.
Note
Table A-2 lists the CDSM GUI ID names and maps them to the CServer names in the setupfile and .arroyorc files.
Description of the .arroyorc Settings
This section describes the different line entries of the .arroyorc file. The .arroyorc file is located in the /home/isa directory and is created during the initial configuration procedure outlined in the Cisco Content Delivery Engine 205/220/250/420 Hardware Installation Guide.
self
This number represents what type of server the CDE is:
•
•
•
•
groupid
All servers that are part of the same CDS system (managed by one CDSM) have the same group ID. This group ID should be unique across an enterprise. The purpose of the group ID is to allow servers in a group to recognize each other as belonging to the same group. If two server groups were on the same VLAN and they had the same group number, they would conflict and cause issues. This is much more likely to be an issue in a lab environment with shared resources than an actual production deployment, but this should still be managed.
serverid
Every server in the group has to have a unique ID ranging from 1 to 255. It is a good idea to use a standardized numbering solution; for example, all 1xx serverids are Streamers and all 2xx server IDs are Vaults.
vault
This parameter has the IP address of a Vault in the system. Each "vault" line represents an individual Vault. There may be multiple vault lines.
streamer
This is the IP address of a Streamer in the system. Each "streamer" line represents an individual Streamer. There may be multiple streamer lines.
controller
This is the IP address of the CDSM. There is only one controller line. This line is not needed in the file for the CDSM, but is used on Vaults and Streamers to point to the CDSM.
mirroring
This controls local mirroring, which is to say this determines the number of copies of a given piece of content that is stored locally.
partno
This allows the server to identify itself properly to the CDSM. The CDSM can then display the appropriate server graphic in the GUI and manage the appropriate number of disks, Ethernet ports, and so on.
mgmtif
The index of the management interface starting at eth0. Typically, this remains 0.
ingestif
This parameter is only for Vaults. The index of the ingest interface starting at eth0. Typically this remains 0 but may have the value of 1 as well.
dbdomsock
This is the "file handle" where the applications address messages are intended for the database.
dbnetport
This is the port number where the applications address messages are intended for the database.
controlif
The index of the stream control interface starting at eth0. This is an optional configuration that is used when you want to separate the Setup and Control interface.
Description of the setupfile Settings
This section describes the different parameters of the setupfile file. The setupfile file is located in the /arroyo/test directory. Some values for the parameters in the setupfile file are set during the initial configuration (serverid, groupid, streamer vault), others are set by using the CDSM.
Note
Required Settings
The following line entries are required in every setupfile file:
serverid #
An identifier that uniquely identifies the server within a group of servers identified by the group ID. See the "serverid" section for more information.
groupid #
An identifier that identifies the group of servers within the CDS. See the "groupid" section for more information.
streamer <0 or 1> vault <0 or 1>
To run the server as a Streamer, set streamer to 1; otherwise, set streamer to 0. To run the server as a Vault, set vault to 1; otherwise, set vault to 0. Setting both streamer and vault to 0 is not a valid option.
service address <ip in dot notation> setup <setup portno> control <control portno>
The service address is used to specify whether this server can assume the role of the Setup server, the Control server, or both the Setup and Control servers for the specified IP address. This parameter applies only to Streamers.
•
•
e1000 <index>: streaming <0 or 1> fill <0 or 1> ip <ip in dot notation> tport <transport portno> cport <cache portno> tgid <transport groupid>
The e1000 is used to configure the network interfaces for cache-fill and transport/streaming. Each "e1000" line represents an individual Ethernet port. Include one line per interface.
•
•
•
•
•
•
•
vault mirror copies <number of copies>
The Vault mirror copies is a numeric value representing the number of copies of each content to store on the Vaults.
Note
Optional Settings
The following line entries are optional in the setupfile file:
management eth #
Specifies the interface used for management. The default is eth0.
ingest eth #
Specifies the interface used for live ingests (FTP push or UDP capture). By default, the management interface is used. This parameter is applicable only to Vaults.
e1000 adapters: maxrate <rate in Mbps>
Controls the maximum transmit bandwidth on this interface, either for streaming, for caching, or for both. The default is 975 Mbps.
ibg adapters
The maximum transmit bandwidth of the ibg adapters. The default is 975 Mbps.
disks #
Specifies the number of hard drives (disks) installed on a server. The default is 12 disks for a Streamer, and 24 disks for a Vault. If you have a server with 12 disks, you must add this entry and specify 12 disks; otherwise, warning messages stating disks are non-operational are logged to the protocoltiming log file.
test #
Specifies the test mode of the server. The default is 4, which means to run the server in production mode.
cache_dscp #
Used to set the DSCP bits on cache-fill packets. The default value is 0.
cache_ecn #
Used to set the ECN bits on cache-fill packets. The default value is 0. This parameter should not be used.
transport_dscp #
Used to set the DSCP bits on transport/streaming packets. The default value is 0.
transport_ecn #
Used to set the ECN bits on transport/streaming packets. The default value is 0. This parameter should not be used.
trickspeedsv2 # # # # # # # #
Used to specify up to 8 speeds for generating trick objects during ingest. This parameter is applicable only to Vaults. The defaults are 5, -5, 10, -10, 32, and -32. The highest speeds are -127 and 127. An entry larger than 127 defaults to 127. A value of 0 is ignored.
ftpout if eth # max utilization mbps # max sessions #
Used to specify which interface on the Vault is used for FTP Out, the maximum bandwidth utilization for all active sessions (in Mbps), and the maximum number of simultaneous sessions allowed. By default, the maximum sessions is 0, meaning that FTP Out is not allowed. The default for bandwidth utilization is 0, which means unlimited usage. The default interface chosen is the management interface. For FTP Out to function properly, the entire content must exist on the Vault.
arrayid #
Specifies the array this server belongs to. The array ID is used in conjunction with the StreamDestinationMap file located in the /arroyo/test directory to determine which play servers are available for selection. This selection is based on the IP address or subnet of the destination of the packet. The default value is 0.
remote site manager <ip address> for arrayid #
Specifies the IP address of a remote site manager for a specified array. The remote site manager has a list of all the servers that it can connect to that are located at the same site (the servers are listed in the RemoteServers list) on the remote site manager). When this server comes up, it contacts the remote site manager and asks for a referral for a server that is in the specified array. The remote site manager returns the IP address of a server from that array. This server sets up connections with all the referred remote servers, and once every minute, checks to see if there are connections with all remote array servers. If for some reason one is not available, this server contacts the remote site manager for another referral.
Note
default source ip <ip in dot notation> tport <minportno> - <maxportno> cport <portno>
Used to affect source packets if no specific information is provided in the individual mandatory e1000 interface entries.
•
•
•
Note
Note
bms address <ip> <port>
The IP address and port of the backoffice.
Identifying the Software Versions or Releases
The following sections describe the commands for identifying the software versions on the server.
Linux OS Version
To identify the software version of the Linux operating system (OS) on the CDSM, enter the following command:
# cat /proc/version or "uname -a"Linux version 2.6.18-92.el5 (brewbuilder@ls20-bc2-13.build.redhat.com) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-41)) #1 SMP Tue Apr 29 13:16:15 EDT 2008To identify the software version of the Linux OS on the Vault, Streamer, or ISV, enter the following commands:
# cat /proc/versionLinux version 2.6.18-53.el5.kernel.2_6_18.2008.10.07.01 (arroyoqa@build-svr) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Mon Nov 17 18:21:51 PST 2008# uname -aLinux stm74 2.6.18-53.el5.kernel.2_6_18.2008.10.07.01 #1 SMP Mon Nov 17 18:21:51 PST 2008 i686 i686 i386 GNU/LinuxCDS-Related Releases
The CDS software for ISA is a combination of the ISA overlay, statsd software, and the CServer code. The following sections describe how to identify the software version of each.
ISA Environment
To identify the software version of the CDS ISA overlay image, enter the following command:
# cat /home/isa/IntegrationTest/.build_tag** ISA Tag: r_2_0v1-isa-e008-2008-10-28-01Build Time: Mon Nov 17 19:00:38 PST 2008statsd Program
To identify the software version of the statsd program, enter the following command:
# strings /home/stats/statsd |grep RelSTATSD Release TOP_OF_TREE (arroyoqa@build-svr) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1-Nstatsd-2008-11-07-02 Mon Nov 17 18:34:15 PST 2008CSserver Code
To identify the software version of the CServer on the Streamer, Vault, or ISV, enter the following command:
# strings avs_cserver.ko |grep CServerAverage setup time spent in CServer =AVS CServer Release #1-Ncserver-e013-2008-11-17-05 Mon Nov 17 18:54:01 PST 2008 ENV_ISA_SR DEBUGAVS CServer Information ENV_ISA_SR DEBUG (arroyoqa@build-svr) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1-Ncserver-e013-2008-11-17-05 Mon Nov 17 18:54:01 PST 2008To view the CServer settings, status, and version, enter the following command:
# cat /proc/calypso/status/server_settingsAVS CServer Information ENV_ISA_SR PROD (arroyoqa@build-svr) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14))#1-Ncserver-e013-2009-01-20-03 Tue Jan 20 17:54:28 PST 2009Server Settings:Server is operationalCache2App is operationalTSCs Per Second is 2333447000Network Settings:Running in L3 Network ModeAllow Jumbo FramesTransport/Stream Data Payload: 1316Cache/Fill Data Payload: 7680Cache/Fill Control Maximum Packet Size: 8048Using ifstats to Monitor Traffic
The ifstats command shows real-time traffic on each Ethernet interface on the server.
# /home/stats/ifstatsifstats - 11:12:22============================================================Int# R-Mbps X-Mbps R-Bytes X-Byteseth0 0 0 56760511 166307653eth1 0 0 0 0eth2 4 457 3439241508 3497139080eth3 4 457 3439172148 3099124288eth4 4 457 3441836680 2945489644eth5 4 472 3443060380 2736115618eth6 4 471 3438423816 2613199736eth7 5 464 3440066492 2419935662eth8 4 449 3439982812 2266582156eth9 4 465 3443251384 2164010982eth10 5 465 3439982136 1915437726eth11 4 464 3438935192 397577442eth12 5 464 3440343164 300903930eth13 4 465 3439540716 4454799830Kernel Crash
The kernel debugger (KDB) provides information (in the form of a core dump file) when the server processing fails. For the server to enter KDB when the server has crashed, the /proc/sys/kernel/panic parameter must be zero. If the panic parameter is non-zero, the system reboots automatically without entering KDB.
In addition to KDB, there is a kdump service. The kdump service allows you to take a kernel dump of memory. The kdump service runs automatically if the server is configured to reboot automatically after a crash (which means the panic parameter is non-zero). The kdump service stores the kernel memory dump in the /var/arroyo/crash directory. After the kernel memory is dumped, the system reboots into the normal operating system.
If the server is configured to enter KDB (which means the panic parameter is zero), the server enters KDB mode. The kdump command allows you to take a kernel memory dump while the server is in KDB mode. The kdump command reboots the server into kdump mode, takes a kernel memory dump, and reboots the server into the normal operating system.
If a server has crashed after being started automatically from the /etc/rc.local directory, you need to boot in single-user mode. To boot in single-user mode, perform the following steps:
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7
Step 8
Step 9
/arroyo/test/vault/run.Step 10
To view the contents of the core dump file from the Linux prompt, do the following:
Step 1
gdb --core=<core-file> <binary-file>The core-file parameter is the core filename and the binary-file is the binary file that produced the core file.
Step 2
gdb> btThe callback stack is displayed, which shows the history of the current function calls that were made at the time of the crash.
Disk Drive Issues
The disk drive order is irrelevant when reinserting disk drives after transporting a chassis, or transferring disk drives from one chassis to another.
To view the statistics of the internal boot drive, the disk drive that contains the software, enter the df -k command.
# df -kFilesystem 1k-blocks Used Available Use% Mounted on/dev/hda1 10317828 3764936 6028776 39% //dev/hda2 20641788 1711372 17881776 9% /arroyo/dev/hda3 8254272 32828 7802148 1% /arroyo/db/dev/hda6 35641880 1185880 32645480 4% /arroyo/lognone 1681200 0 1681200 0% /dev/shmTo view the statistics of a removable SATA or SCSI disk drive, use the following commands:
# cat /proc/calypso/status/streamer/diskinfoDisk Info:Disks(12) Op(12)Storage: T(804G) A(21%) U(0)BW: (99%) w(1.35M/s) r(0/s)I/O Util: w(1:0%) e(0) a(0%)Disk[ 1][67.0G] A[20%] B[11x]Disk[ 2][67.0G] A[20%] B[0x]Disk[ 3][67.0G] A[21%] B[0x]Disk[ 4][66.5G] A[22%] B[0x]Disk[ 5][67.0G] A[20%] B[0x]Disk[ 6][67.0G] A[21%] B[0x]Disk[ 7][67.0G] A[20%] B[0x]Disk[ 8][67.0G] A[20%] B[0x]Disk[ 9][67.0G] A[21%] B[0x]Disk[10][67.0G] A[20%] B[0x]Disk[11][67.0G] A[20%] B[0x]Disk[12][67.0G] A[20%] B[0x]CDSM GUI Disk Monitor Page Reports a Disk Warning
If the CDSM GUI Disk Monitor page reports a disk warning, check the disk drive status in the /arroyo/log/protocoltiming.log.<date> log file and the /var/log/debugmessages log file.
# grep drives /arroyo/log/protocoltiming.log.11132007WARNING: 5 disk drives are non-operationalWARNING: 5 disk drives are non-operational...WARNING: 5 disk drives are non-operationalWARNING: 5 disk drives are non-operational# grep disks /var/log/debugmessagesNov 20 19:02:44 vault219 kernel: RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksizeNov 20 19:03:34 vault219 kernel: Waiting for 2 disks to finish initializingNov 20 19:03:34 vault219 kernel: Waiting for 4 disks to finish initializingNov 20 19:03:35 vault219 kernel: Waiting for 3 disks to finish initializingNov 20 19:03:36 vault219 kernel: Waiting for 2 disks to finish initializingNov 20 19:03:36 vault219 kernel: Waiting for 1 disks to finish initializingNov 20 19:03:36 vault219 kernel: Waiting for 5 disks to finish initializingNov 20 19:03:42 vault219 kernel: Waiting for 6 disks to finish initializingNov 20 19:03:42 vault219 kernel: Waiting for 5 disks to finish initializingNov 20 19:03:43 vault219 kernel: Waiting for 4 disks to finish initializingNov 20 19:03:45 vault219 kernel: Waiting for 11 disks to finish initializingNov 20 19:03:46 vault219 kernel: Waiting for 10 disks to finish initializingNov 20 19:03:46 vault219 kernel: Waiting for 9 disks to finish initializingNov 20 19:03:46 vault219 kernel: Waiting for 8 disks to finish initializingNov 20 19:03:47 vault219 kernel: Waiting for 7 disks to finish initializingNov 20 19:03:47 vault219 kernel: Waiting for 6 disks to finish initializingNov 20 19:03:48 vault219 kernel: Waiting for 5 disks to finish initializingNov 20 19:03:48 vault219 kernel: Waiting for 4 disks to finish initializingNov 20 19:03:48 vault219 kernel: Waiting for 3 disks to finish initializingNov 20 19:03:48 vault219 kernel: Waiting for 2 disks to finish initializingNov 20 19:03:48 vault219 kernel: Waiting for 1 disks to finish initializingNov 20 19:03:50 vault219 kernel: Total disk space = 24.0TB on 24 disk drives (Lost disks = 0)
Note
Memory Issues
To slow down the CDSM bootup to see the memory counter, do the following:
Step 1
Step 2
Press DEL to runSetup
Note
Step 3
Step 4
Step 5
Figure A-1 BIOS Setup Utility—Boot Settings
Step 6
Disabling USB Ports and Password-Protecting the BIOS
Disabling USB Ports and Password-Protecting the BIOS
The following procedure provides instructions on disabling the USB ports and password-protecting the BIOS. Disabling the USB ports and password-protecting the BIOS provides a way to secure the server from anyone outside the allowed administrative group from accessing the server through the USB port.
If the USB port is required to perform an operation (for example, a software upgrade), the operator can log in to the BIOS using the BIOS password and enable the USB port for the operation. After the operation is complete, the operator logs back in to the BIOS and disables the USB port.
To set the BIOS password and disable the USB ports, do the following:
Step 1
Step 2
Press DEL to runSetupTo enter the BIOS Setup Utility on a CDSM or VVIM, press F2.
The BIOS menu is displayed with the Main tab selected.
Use the Right Arrow and Left Arrow keys to select a menu tab, and the Up Arrow and Down Arrow keys to select a menu item.
Step 3
a.
b.
c.
To set a user password, use the Down Arrow key to highlight the User Password, press Enter, and enter the password in the Create New Password dialog box.
Step 4
a.
b.
c.
d.
Step 5
Use the Right Arrow and Left Arrow keys to highlight Yes and press Enter.
After rebooting the server, to enter the BIOS Setup Utility, you are prompted to enter the BIOS password. To have read/write permission, enter the Administrator password. To have read-only permission, enter the User password.
To enable the USB ports, reboot the server, do the following:
Step 1
Step 2
Press DEL to runSetupTo enter the BIOS Setup Utility on a CDSM or VVIM, press F2.
The BIOS menu is displayed with the Main tab selected.
Step 3
Step 4
Step 5
Step 6
Step 7
Use the Right Arrow and Left Arrow keys to highlight Yes and press Enter.
Network
Following are the some common network problems:
•
•
No Output on the NSG
There is no output on the NSG when sending a play command.
Cause 1: Cannot communicate with the QAM device or the QAM settings are incorrect.
Action 1: Verify the QAM configuration on the CDS.
Verify the IP address, and MAC address if applicable, of the QAM by going to the Configure > System Level > QAM Gateway page.
Vault Cannot Connect to FTP Server
In the ISA Content Store Log file, which is viewable by logging in to the Linux OS on the server, a log entry states it cannot connect to the FTP server.
Cause 1: Hostname is not resolving correctly.
Action 1: Verify the CDS configuration for DNS.
•
•
•
Checking Network Configuration
The following commands are useful for checking your network configuration and activity.
To view the ARP table, enter the following command:
# arp -ajetsam.v.com (111.0.110.151) at 00:00:0C:07:AC:00 [ether] on eth0cds17-m1.v.com (111.0.210.170) at 00:30:48:58:5B:A1 [ether] on eth0cds17-v1.v.com (111.0.210.171) at 00:30:48:31:53:B2 [ether] on eth0? (111.0.210.175) at 00:30:48:32:0A:5A [ether] on eth0cds17-s1.v.com (111.0.210.172) at 00:04:23:D8:89:44 [ether] on eth0cds17-s1.v.com (111.0.210.172) at 00:04:23:D8:89:44 [ether] on eth0To view the IP routing table, enter the following command:
# netstat -rnKernel IP routing tableDestination Gateway Genmask Flags MSS Window irtt Iface111.0.210.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0111.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo0.0.0.0 111.0.210.1 0.0.0.0 UG 0 0 0 eth0To view the CDS subnet table, enter the following command:
# cat /arroyo/test/SubnetTablenetwork 111.1.13.1 netmask 255.255.255.240 gateway 111.1.13.1 transport_source_ip 0
Note
To view the CDS routing table, enter the following command:
# cat /arroyo/test/RoutingTabledefault gateway 111.1.13.1network 111.1.13.1 netmask 255.255.255.240 gateway 0.0.0.0To view the CDS remote server table, enter the following command:
# cat /arroyo/test/RemoteServersremote serverid 141ip 111.1.9.20ip 111.1.9.21ip 111.1.9.22ip 111.1.9.23ip 111.1.9.24end remote serverremote serverid 143ip 111.1.9.25ip 111.1.9.26end remote serverremote serverid 144ip 111.1.9.27ip 111.1.9.28ip 111.1.9.29ip 111.1.9.30end remote serverInterface Information
To view basic interface information, use the ifconfig command.
# ifconfig eth0eth0 Link encap:Ethernet HWaddr 00:04:23:D8:9A:80inet addr:111.0.110.41 Bcast:111.0.110.255 Mask:255.255.255.0UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1RX packets:13946269 errors:0 dropped:0 overruns:0 frame:0TX packets:11594110 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:3085199261 (2942.2 Mb) TX bytes:1317620721 (1256.5 Mb)Interrupt:24 Base address:0x3000 Memory:dd240000-0For detailed interface information, view the interface information file in the /proc/net/ directory.
# cat /proc/net/PRO_LAN_Adapters/eth0.infoDescription Intel® Gigabit Ethernet Network ConnectionsPart_Number ffffff-0ffDriver_Name igbDriver_Version 1.2.22-CDSPCI_Vendor 0x8086PCI_Device_ID 0x10a7PCI_Subsystem_Vendor 0x15d9PCI_Subsystem_ID 0x10a7PCI_Revision_ID 0x02PCI_Bus 14PCI_Slot 0PCI_Bus_Type UNKNOWNPCI_Bus_Speed UNKNOWNPCI_Bus_Width UNKNOWNIRQ 194System_Device_Name eth0Current_HWaddr 00:30:48:C3:26:9EPermanent_HWaddr 00:30:48:C3:26:9ELink upSpeed 1000Duplex FullState upRx_Packets 406625Tx_Packets 135553Rx_Bytes 41539919Tx_Bytes 30390314Rx_Errors 0Tx_Errors 0Rx_Dropped 0Tx_Dropped 0Multicast 236747Collisions 0Rx_Length_Errors 0Rx_Over_Errors 0Rx_CRC_Errors 0Rx_Frame_Errors 0Rx_FIFO_Errors 0Rx_Missed_Errors 0Tx_Aborted_Errors 0Tx_Carrier_Errors 0Tx_FIFO_Errors 0Tx_Heartbeat_Errors 0Tx_Window_Errors 0Tx_Abort_Late_Coll 0Tx_Deferred_Ok 0Tx_Single_Coll_Ok 0Tx_Multi_Coll_Ok 0Rx_Long_Length_Errors 0Rx_Short_Length_Errors 0Rx_Align_Errors 0Rx_Flow_Control_XON 0Rx_Flow_Control_XOFF 0Tx_Flow_Control_XON 0Tx_Flow_Control_XOFF 0Rx_CSum_Offload_Good 406625Rx_CSum_Offload_Errors 0PHY_Media_Type CopperPHY_Cable_Length Unknown Meters (+/- 20 Meters)PHY_Extended_10Base_T_Distance UnknownPHY_Cable_Polarity NormalPHY_Disable_Polarity_Correction EnabledPHY_Idle_Errors 0PHY_Receive_Errors 0PHY_MDI_X_Enabled MDIPHY_Local_Receiver_Status OKPHY_Remote_Receiver_Status OKStartup Issues
This section includes the following topics:
•
•
BIOS Settings—Operating System Hangs or Goes into KDB Mode
When a single bit error occurs in the memory of a server, it causes the Linux OS to lock up, which puts the server into kernel debugger (KDB) mode. This is because of the BIOS Error Correcting Code (ECC) Type being set incorrectly.
To correct the ECC Error Type setting in the BIOS Setup Utility, do the following.
Step 1
Step 2
Step 3
Figure A-2 BIOS Setup Utility—Advanced Chipset Control
Step 4
Serial Console Port Settings
The CDE servers ship with the following serial console settings: 115200 baud rate, no parity, 8 data bits, and 1 stop bit (115200-N-8-1). To verify the serial console settings, check the following:
•
•
•
$ cat /etc/inittab | grep S0S0:2345:respawn:/sbin/agetty ttyS0 115200 vt100
Note
Required Services Not Starting or Running Correctly
From the CDSM GUI, view the Services page for each server by clicking Monitor > Server Level > Services. For more information, see the "Services Monitor" section. If the required services are not started, or they are not running correctly, check that there is two-way database connectivity between the Streamers and Vaults, and the CDSM.
$ netstat -an|grep 9999tcp 0 0 0.0.0.0:9999 0.0.0.0:* LISTENtcp 0 0 172.22.97.193:9999 172.22.97.197:56998 ESTABLISHEDtcp 0 0 172.22.97.193:34743 172.22.97.197:9999 ESTABLISHEDtcp 0 0 172.22.97.193:9999 172.22.97.192:50343 ESTABLISHEDtcp 0 0 172.22.97.193:39158 172.22.97.196:9999 ESTABLISHEDtcp 0 0 172.22.97.193:46030 172.22.97.192:9999 ESTABLISHEDtcp 0 0 172.22.97.193:9999 172.22.97.196:55780 ESTABLISHEDtcp 0 0 172.22.97.193:9999 172.22.97.191:50950 ESTABLISHEDtcp 0 0 172.22.97.193:60598 172.22.97.191:9999 ESTABLISHEDtcp 0 0 172.22.97.193:9999 172.22.97.194:37543 ESTABLISHEDtcp 0 0 172.22.97.193:56376 172.22.97.194:9999 ESTABLISHEDTwo connections for each Vault and Streamer should be listed with a status of "ESTABLISHED."
If the connection states do not say "ESTABLISHED," check the configuration of /home/isa/.arroyorc file to make sure the settings are correct, then restart the database.
Log in to the server as isa and start the database.
$ arroyo start avsdbLog into the server as root and start the statsd.
$ /home/stats/statsdCheck that the LSCP listener is running on the correct port.
$ arroyo status$ netstat -an | grep 554Management and Database Issues
This section includes the following topics:
•
System Health
The colored boxes on the System Health Monitor page have the following meaning:
•
•
•
Cannot Access the CDSM GUI
If you cannot access the CDSM GUI, log in as root and verify that the Apache server is running on the CDSM.
# ps -aef | grep httproot 4023 1 0 Aug09 ? 00:01:44 /arroyo/www/bin/httpdnobody 4033 4023 0 Aug09 ? 00:00:53 /arroyo/www/bin/httpdnobody 4034 4023 0 Aug09 ? 00:00:53 /arroyo/www/bin/httpdnobody 4035 4023 0 Aug09 ? 00:00:53 /arroyo/www/bin/httpdnobody 4036 4023 0 Aug09 ? 00:00:53 /arroyo/www/bin/httpdnobody 4037 4023 0 Aug09 ? 00:00:53 /arroyo/www/bin/httpdnobody 4085 4023 0 Aug09 ? 00:00:52 /arroyo/www/bin/httpdnobody 4086 4023 0 Aug09 ? 00:00:53 /arroyo/www/bin/httpdnobody 4572 4023 0 Aug10 ? 00:00:52 /arroyo/www/bin/httpdroot 11598 30692 0 16:12 pts/0 00:00:00 grep httpIf the Apache server is not running, restart the server.
# /arroyo/www/bin/apachectl startCDSM GUI Does Not Register the Vaults and Streamers
If the CDSM GUI is not able to register that the Vaults and Streamers are part of the array or CDS, do the following:
Step 1
Step 2
# netstat -an | grep 9999Step 3
# ps -aef | grep statsdStep 4
# ls -l /home/stats/Step 5
•
•
•
•
For example, first deregister the CDS host using the -d option, then initialize the CDS host.
# svrinit_15 -i <ip_address> -s <subnet_mask> -h <host_name> -d# svrinit_15 -i <ip_address> -s <subnet_mask> -h <host_name>Step 6
Database Monitoring
To monitor a stuck database thread problem, use the following command:
netstat -an | grep 9999Two connections for each Vault and Streamer should be listed with a status of "ESTABLISHED." If two-way connection does not exist, run db_shutdown on all servers including the CDSM, then start the database using the following commands:
# su - isa$ arroyo start avsdbIf the database is stuck and db_shutdown does not take effect, use ps -ef | grep avsdb to query the process ID (PID), then use the kill -9 {pid} command to kill the avsdb process, and lastly restart the database.
Ingest Issues
This section includes the following ingest issues:
Ingest Interface
This section includes the following topics on troubleshooting the ingest interface:
•
General Tips
Following are some general troubleshooting tips for ingest issues:
•
•
•
Common Ingest Problems
Following are some possible causes of ingest issues:
•
•
•
–
–
CDS Is Not Registered to the Name Service
If the CDS is not registering with the CORBA name service, check the information from the following commands:
$ tail -f /arroyo/log/ns_log$ /home/isa/IntegrationTest/show_calypso_services$ /home/isa/IntegrationTest/list_all_content$ /home/isa/IntegrationTest/list_all_streamsRestarting the ISA Services
If it is necessary to restart the ISA services, then enter the following commands:
$ /home/isa/IntegrationTest/stop_all$ tail -f /arroyo/log/ns_log$ /home/isa/IntegrationTest/run_isaBad Content
Variable bit rate (VBR) encoded content is not currently supported. See the "CDS Content Quality Guidelines" section for constant bit rate (CBR) guidelines.
Network
Ensure that the network maximum transmission unit (MTU) is appropriately set. If jumbo frames are enabled on the CDS, then the network must support jumbo frames. We recommend that the network support jumbo frames even when the jumbo frame option is disabled.
If a Layer 2 network is used for CDS, then appropriate MAC addresses (ARP entries) have to be configured on the switches and routers. Ensure that the CDS Vault and Streamer interfaces are in the same VLAN. If a Layer 3 network is used for the CDS, then ensure that the corresponding default gateways are correctly configured on CDS Vaults and Streamers for the various interfaces and Stream Groups.
Ensure that the content source (catcher, FTP server, and so on) is reachable from the Vaults or ISVs, and that manual content transfer using FTP works correctly.
For more information about the status of the network interfaces, network routing tables, ARP and so on, see the "Network" section.
Content Processing Issues
This section includes the following content processing issues:
Listing Content
To view the actual stored content versus what the database reports, enter the following commands:
$ su - isa$ cddb$ ./dumpDBThe dumpDB command displays a menu that provides options for populating a list file with the selected information.
===============================** DUMP (CDS) DB TABLES **** Version: 2.0.0 **===============================SELECT A TABLE TO DUMP TO FILE1: CONTENT(s) (ctnobj.lst)2: STREAM(s) (stmobj.lst)3: SERVICE GROUP(s) & TSID(s) (svcgrp.lst)4: QAM(s) (qam.lst)5: STREAM SVC LIST (sslist.lst)6: LOAD(s) (load.lst)7: ISA ENV (isaenv.lst)8: MSA UNPROCESSED(s) (unprocessed.lst)9: MSA PROCESSED(s) (processed.lst)0: QUIT===============================ENTER CHOICE [1-9,0]: 0===============================To view the list file (for example, ctnobj.1st), go to the /arroyo/db directory and use the cat command or the vi command.
Content Mirroring
To enable content mirroring locally on one Vault, do the following:
Step 1
cserver_opts "vault local copy count 2"Step 2
The line "vault local copy count 2" should be added to the setupfile file.
Alternatively, enable local mirroring using the tunables. You can also use the tunables to verify the settings.
echo 2 > /proc/calypso/tunables/vaultlocalcopycount
Note
To enable content mirroring between two Vaults, do the following:
Step 1
For more information, see the "Configuring the Servers" section.
Step 2
Step 3
Step 4
Step 5
# grep mirror /arroyo/test/setupfilevault mirror copies 2# grep LocalMirror /arroyo/log/protocoltiming.log.11202007-LocalMirror Active=0:0 comp=0% obj=0.0/s read=0b/s write=0b/s copies=1-LocalMirror Active=0:0 comp=0% obj=0.0/s read=0b/s write=0b/s copies=1
Verifying GOIDs
You cannot verify that the global object identifiers (GOIDs) among Vaults and Streamers are correct by comparing the total number of GOIDs on each server. There are actually multiple chains of GOIDs. If you list the GOID chains you can verify that the GOIDs are correct, because listing the GOIDs provides a summary at the end of the listing that reports any issues.
To list the GOIDs, enter the following command:
echo 2 > /proc/calypso/tunables/cm_logserverinfoTo list all GOID chains, enter the following command:
echo 4 > /proc/calypso/tunables/cm_logserverinfoThe /arroyo/log/serverinfo.log.<date> log file contains information about the GOIDs.
Note
Trick-Mode Issues
Verify the trick-mode settings in the CDSM GUI and the Vault setupfile file.
•
•
$ grep trick /arroyo/test/setupfiletrickspeedsv2 4 10 32 -32 -10 -4Check the /arroyo/log/c2k.log.<date> log file and the session message logs during playout to verify that the trick-mode files are being streamed.
Name and Notify Services
Ensure that the Name Service IP address and port, and the Notify Service IP address and port are correct.
•
•
$ cat /home/isa/isa.cfg$ cat /home/isa/IntegrationTest/.isa_envCORBA Interface
Ensure that the CORBA-related parameters (including the Name and Notify services) are configured correctly. This includes the Content Service and CORBA Event Channel details such as the Content Store, Kind, Content Factory, Event Channel, Content Channel, Factory, and so on.
See the "CORBA Interface" section for more troubleshooting methods for CORBA. See the "Configuring the Streamer for BMS Connectivity" section and the "Configuring the Vault for BMS Connectivity" section for more information on configuring the CORBA-related parameters.
Cache-Fill Issues
This section covers the following cache-fill issues:
•
Tracking Cache-Fill Source
You can track whether or not a GOID for a stream is filling remotely or locally by enabling and tracking it in the fill.log. Streams can share the same GOID; in which case it is not possible to tell which stream is currently filling the data.
To track the cache-fill source of a stream, do the following:
Step 1
Step 2
echo 1 >/proc/calypso/tunables/enableFillLogStep 3
Following is an example fill.log:
<omitted content>..18:30:23 44 DISK 000814a4132455c4 0000c1f7 to 00014e3e 0ea6 FINISHED 0x0000f55818:30:24 4 NET 000864b26ab0a076 3fde3a14 to 3fe86299 3a98 TRUNCATE 0x3fdeb83c18:30:24 4 NET 000864b26ab0a076 3fde3a14 to 40000000 3a98 FINISHED 0x3fdeb83418:30:24 4 NET 000864b26ab0a076 3fde3a14 to 3fdeb83c 3a98 CANCEL 0x3fdeb93e18:30:25 44 DISK 000884b7c94042f4 3ff3730d to 3ffca5d0 3a98 CANCEL 0x3ff3fcb518:30:25 43 DISK 000884b7c94042f4 3ff3730d to 3ffca5d0 3a98 FINISHED 0x3ff3fcb518:30:27 43 DISK 000814a4132455c4 0001fadb to 0002d79d 0ea6 START delay 1437618:30:28 43 DISK 000854b26ab11667 3ffd2b92 to 3ffd4198 0ea6 FINISHED 0x3ffd4198<omitted content>...START - fill started from DISK or NETFINISHED - fill finishedCANCEL - fill cancelledTRUCATE - fill truncated to new ending sector offsetBurst and delay times are in microseconds. Bursts are sent immediately at a high rate. The delay time specifies when to start sending the data at rate, up to 30 seconds into the future.
If no Caching Nodes are reporting fill for the GOID, then the content is being filled from memory.
Step 4
echo 0 >/proc/calypso/tunables/enableFillLog
Rules for ISV Interoperability with Vaults and Streamers
The following rules apply for ISVs to interoperate with Vaults and Streamers:
•
•
•
•
Network
Note
Stream Stops Playing at the Same Place or Does Not Play at All
Cause 1: Jumbo frames are not supported or configured on the cache-fill network switch.
Check 1: Search the c2k.log file for content read errors.
==> /arroyo/log//c2k.log.01152008 <==15-Jan-2008 20:42:33 UTC :out:c2k_p_setcontentbundle: stream 3 localStreamHandlePtr 00000000 remoteServer 00000000 15-Jan-2008 20:42:33 UTC :out:c2k_p_setcontentbundlecontinue: stream 3 localStreamHandle 0 15-Jan-2008 20:42:33 UTC :out:c2k_p_setdestination: stream 3 localStreamHandle 0 ip 0xe0016401 port 10000 15-Jan-2008 20:42:41 UTC :out:igate goid d346434b982851 finished read 0 length e3 lastbytes b4 retries 0 reqlen 0/e3 15-Jan-2008 20:42:41 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw header 15-Jan-2008 20:42:41 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw EOF record 15-Jan-2008 20:42:44 UTC :out:igate goid d346434b982851 finished read 0 length e3 lastbytes b4 retries 1 reqlen 0/e3 15-Jan-2008 20:42:44 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw header 15-Jan-2008 20:42:44 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw EOF record 15-Jan-2008 20:42:47 UTC :out:igate goid d346434b982851 finished read 0 length e3 lastbytes b4 retries 2 reqlen 0/e3 15-Jan-2008 20:42:47 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw header 15-Jan-2008 20:42:47 UTC :err:IGate::ReadClose(goid 0): ERROR: Never saw EOF recordCheck 2: Ping between the two devices.
Ping between the two devices on the cache-fill VLAN using a packet size greater than 1500 bytes.
Action 1: If the ping fails, verify that jumbo frames and cache-fill interfaces are configured correctly.
Verify that jumbo frames are enabled on the switch ports for the cache-fill VLAN, and verify that the cache-fill interfaces are configured correctly on the Streamers and Vaults. See the "Configuring the Servers" section for information on configuring the cache-fill interfaces.
Streaming and Playout Issues
This section includes the following streaming and playout issues:
Listing of Streams
To monitor streams based on various criteria, go to the Stream Monitor page in the CDSM GUI by clicking Monitor > System Level > Stream Monitor. For more information, see the "Stream Monitor" section.
No Streaming
Some common causes for streaming problems are the following:
•
•
•
–
–
–
•
Following are some general methods for troubleshooting stream and playout issues:
•
•
•
•
•
Stream Not Playing
Cause 1: A piece of content is missing.
In this case, a user can typically stream part of the content, but at some point the stream stops and this error is returned in the ANNOUNCE message. The content needs to be validated at the CServer level.
Action 1: Set up a stream to play to a multicast address.
If this is successful, then there is a network issue, which is either a default gateway or unreachable remote client. You can verify whether it is successful by looking at the /home/stats/ifstats information.
Action 2: If ifstats information does not detect a problem, try streaming to another multicast IP address.
Repeat streaming to a multicast address with different content and, if possible, ingest known good content. Check the protocoltiming.log.<date> for damaged GOIDs by using the following command:
tail -f protocoltiming.log.<latest date> | grep GoidsCause 2: There is a problem reaching the destination QAM device.
The CServer returns the same completion code, so the same error is returned in the announce message. In this case, the content does not stream at all. The play request and play response are separated by about 10 to 15 seconds, instead of the typical subsecond separation. This is because of the ARP timeout process the CServer is going through to reach the destination. After stream response fails, the CServer calls back with the completion code of 3, which causes the "error reading content data" message.
Action 1: Check that the interfaces involved in the streaming are up and operating at the correct speed.
Using the CDSM GUI, choose Monitor > Server Level > NIC Monitor, choose the IP address of the server, and verify the participating interfaces are up and operating at gigabit Ethernet speeds. For more information, see the "Configuring the Interfaces" section.
Action 2: Set up a stream to play to a multicast address.
If this is successful, then there is a network problem, which is either a default gateway or unreachable remote client. You can verify whether it is successful by looking at the /home/stats/ifstats information.
Action 3: If streaming to a multicast address is not successful, check that the Vaults can be reached.
Check the /arroyo/log/protocoltiming.log.<date> log file for the number of reachable remote servers. Additionally, if there is a cache-fill issue, you will see a large megabit value for the re-xmit buffer.
You can also check the /arroyo/log/c2k.log.<date> log file for any unreachable Vaults.
Tuning Failure
Cause 1: Fail to tune error message (Error 104).
A "fail to tune" error message (Error 104 in the /arroyo/log/streamevent.log file) occurs when the CDSM starts a stream, but the stream fails to reach its destination.
Action 1: To verify the ARP table, enter the arp -a command.
Action 2: Verify there are no network problems; for example, reachability, network partition, and so on.
Action 3: Look for timeout intervals.
Look for characteristic 10 to 30 second timeout intervals between the stream setup and the stream destroy event messages in the /arroyo/log/c2k.log.<date> file.
Restarting the ISA Services
If it is necessary to restart the ISA services, then enter the following commands:
$ /home/isa/IntegrationTest/stop_all$ tail -f /arroyo/log/ns_log$ /home/isa/IntegrationTest/run_isaPoor Video or Audio Quality
When content is streamed to a client device, if there is no video picture displayed on the client device and the audio is working fine, use the following troubleshooting methods:
•
–
/home/stats/ifstats–
–
•
–
–
To fix this, use the cat /proc/interrupts command to display the interrupts and find the interrupt value associated with the interface you are using for ingest. After you know this value, add the following lines to the /arroyo/test/run script:
echo 1 > /proc/irq/<interrupt value>/smp_affinityecho <interrupt value> > /proc/calypso/test/bypass_disable_irqYou can enter these lines at the Linux command line as well. By doing so you do not have to reboot your system for them to take effect. Any content that you have previously ingested should be considered invalid.
See the "No Streaming" section for troubleshooting methods when there are streaming problems.
Session Messaging
All LSCP message requests and responses have the same format, which includes version, transaction ID, op code, status code, stream handle, and data. The LSCP messages are included in the /arroyo/log/LSCPService.log file. Table A-3 describes the LSCP status codes.
Database Issues
This section covers the following database issues and troubleshooting methods:
Database Replication
This section covers the following database issues:
•
CDSM GUI Does Not Report All the Ingested Content
First, verify that the package has not already expired.
Second, check for index errors in the CDSM database logs, using the following command:
$ grep index /arroyo/log/avsdb.log.2007110611-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -3097611-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -3097611-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -3097611-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -3097611-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -3097611-06-2007 07:54:22PM:db_error DB_SECONDARY_BAD:Secondary index inconsistent with primary -30976The example output indicates that the Vault and CDSM databases are not synchronized, possibly because of the server times not being synchronized, a network connectivity issue, a server failure, or some other similar issue.
To check content among Vaults, use the /home/isa/Berkeley/dumpDB command.
For resolution, see the "Corruption Recovery" section.
Many Log Files
If one of the following conditions exists, it indicates that there were database replication errors:
•
•
•
•
tavsdb: unable to allocate memory for mutex; resize mutex region# tail -f avsdb-err.log.20081111tavsdb: unable to allocate memory for mutex; resize mutex regiontavsdb: unable to allocate memory for mutex; resize mutex regiontavsdb: unable to allocate memory for mutex; resize mutex regiontavsdb: unable to allocate memory for mutex; resize mutex regiontavsdb: unable to allocate memory for mutex; resize mutex regiontavsdb: unable to allocate memory for mutex; resize mutex regiontavsdb: unable to allocate memory for mutex; resize mutex regiontavsdb: unable to allocate memory for mutex; resize mutex regiontavsdb: unable to allocate memory for mutex; resize mutex regiontavsdb: unable to allocate memory for mutex; resize mutex region•
$ ls -ltr-rw-r----- 1 isa isa 10485760 Nov 11 17:46 log.0000002824-rw-r----- 1 isa isa 10485760 Nov 11 17:46 log.0000002825-rw-r----- 1 isa isa 10485760 Nov 11 17:46 log.0000002826-rw-r----- 1 isa isa 10485760 Nov 11 17:46 log.0000002837-rw-r----- 1 isa isa 10485760 Nov 11 17:46 log.0000002838-rw-r----- 1 isa isa 10485760 Nov 11 17:46 log.0000002839-rw-r----- 1 isa isa 10485760 Nov 11 17:46 log.0000002841-rw-r----- 1 isa isa 10485760 Nov 11 17:46 log.0000002840-rw-r----- 1 isa isa 10485760 Nov 11 17:46 log.0000002843-rw-r----- 1 isa isa 10485760 Nov 11 17:46 log.0000002842-rw-r--r-- 1 isa isa 5726769152 Nov 12 15:23 REPLAY.dbFor resolution, see the "Corruption Recovery" section.
Corruption Recovery
Caution
If the CDSM database is corrupted and the Vault database is not corrupted, do the following:
Step 1
# /usr/bin/db_shutdownStep 2
# ps -ef | grep avsdbisa 2646 1 0 Jan09 ? 00:14:50 /home/isa/Berkeley/avsdbroot 26088 26059 0 13:23 pts/1 00:00:00 grep avsdbMake sure there is no avsdb process returned. If the avsdb hangs, use the process ID (2646 in the above example) with the kill command.
# kill -9 2646Step 3
Step 4
# /usr/bin/db_shutdownStep 5
# ps -ef | grep avsdbStep 6
Step 7
# su - isa# ps -ef | grep avsdbStep 8
# su - isa# ps -ef | grep avsdbStep 9
Advanced Features and Applications
This section covers the Media Scheduler feature (live multicast ingest) and the Barker Stream feature.
Live Multicast Ingest
Live multicast ingest is available as part of the Media Scheduler feature or the Real-Time Capture feature.
Ingest with Media Scheduler
Using Media Scheduler for live multicast ingest requires the following procedures:
1.
2.
3.
Ingest without Media Scheduler
Using Real-Time Capture for live multicast ingest requires the following procedures:
1.
2.
Ingest Troubleshooting
If the message "ERROR: Unable to login to the ftp location," is present in the /arroyo/log/aim.log file, check the FTP server configured in the Ingest Manager by using the ps -ef | grep ftp command. If the FTP service is not running, enter the service vsftpd start command to start it.
Barker Stream
If you are having issues setting up a barker stream, use the following troubleshooting method. The barker stream runs only on the master Streamer. To identify the master Streamer, use the CDSM Monitor Services page to find the Streamer running the master stream service. See the "Services Monitor" section for more information. Verify the following three items are present:
•
•
•
For more information about barker streams, see the "Configuring Barker Streams" section.
Frequently Asked Questions
Many of the frequently asked questions (FAQs) responses were based on an ISV system, but guidelines can be easily extrapolated for a Vault and Streamer. This section covers the following topics:
•
Reliability and Availability
Q.
A.
Q.
A.
For example, the following output of the netstat command shows that the server (172.22.97.194) has both the listening and non-listening sockets binding on port 9999 to echo the four remote servers (172.22.97.192, 172.22.97.193, 172.22.97.195 and 172.22.97.191).
# netstat -an|grep 9999tcp 0 0 172.22.97.194:9999 172.22.97.195:48652 ESTABLISHEDtcp 0 0 172.22.97.194:9999 172.22.97.191:42732 ESTABLISHEDtcp 0 0 172.22.97.194:54563 172.22.97.195:9999 ESTABLISHEDtcp 0 0 172.22.97.194:39342 172.22.97.191:9999 ESTABLISHEDtcp 0 0 172.22.97.194:9999 172.22.97.192:40207 ESTABLISHEDtcp 0 0 172.22.97.194:41815 172.22.97.192:9999 ESTABLISHEDtcp 0 0 172.22.97.194:9999 172.22.97.193:33196 ESTABLISHEDtcp 0 0 172.22.97.194:43269 172.22.97.193:9999 ESTABLISHEDtcp 0 0If you can not see both listening and non-listening sockets binding on port 9999 for each of the remote servers, the database is not running properly. Check that you have the correct replication group members in your /home/isa/.arroyorc file.
Serviceability and Manageability
Q.
A.
Q.
A.
Q.
A.
Q.
A.
ls -l /lib/modules/<current running kernel name>/kernel/drivers/net/e1000/e1000.koThe following example shows that the e1000.ko file is based on the kernel 2.5.18-53.el5.kernel.2_6_18.2009.01.08.01.
# ls -l /lib/modules/2.6.18-53.el5.kernel.2_6_18.2009.01.08.01/kernel/drivers/net/e1000/e1000.ko-rw-r--r-- 1 root root 2617502 Jan 8 18:13 /lib/modules/2.6.18-53.el5.kernel.2_6_18.2009.01.08.01/kernel/drivers/net/e1000/e1000.koQ.
A.
# /arroyo/www/bin/apachectl stop# /arroyo/www/bin/apachectl start# /arroyo/www/bin/apachectl restartQ.
A.
# cat /arroyo/test/ArpTableip 192.168.2.42 mac 000000000002ip 192.168.2.43 mac 000000000002Q.
A.
# echo 1 > /proc/calypso/test/arp_dumpstateQ.
A.
kdb=off panic=1kernel /boot/vmlinuz-2.4.32avs ro root=/dev/hda1 console=tty0 console=ttyS0,115200 kdb=off panic=1Q.
A.
Q.
A.
# su - isa# cd ~/IntegrationTest# ./list_all_streams# cd ~/Streaming/client# ./run_client2 -> Stream<name - as retrieved above>24 -> DestroyIf this does not work, for example, the stream just restarts, then the problem is not in the ISA subsystem.
Q.
A.
# echo 2 > /proc/calypso/tunables/cm_logserverinfo# cat /arroyo/log/serverinfo.log.01132009Look at the last two lines of output. If there are no holes, the last two lines should be the following:
BeingDeleted=0 HasHoles=0 CopyHoles=0 SectorHoles=0Object Status Check Complete.Q.
A.
Q.
Log in to the server as root and enter the echo 1 > /proc/calypso/test/clearmem command.
Note
Q.
A.
# su - isa# cd ~/IntegrationTest# ./destroy_all_streamsAll sessions are removed, and upon restarting the services, all streams that do not have an associated session are stopped.
Q.
A.
# su - isa# cd ~/Berkeley# ./bsql2 -> Streamer Database1 -> Stream Object6 -> Get All3 -> Delete1 -> NameAfter the colon for the Name command, name everything that was listed in the Get All command.
Q.
A.
[root@ssv3 root]# /usr/bin/db_shutdown[root@ssv3 root]# ps -ef |grep avsWait for all avs processes to stop, then reboot the server.
[root@ssv3 root]# rebootQ.
A.
# cat /arroyo/test/RoutingTabledefault gateway 192.169.131.250network 192.169.131.0 netmask 255.255.255.0 gateway 0.0.0.0default cache gateway 192.169.131.250local cache network 192.169.131.0 netmask 255.255.255.0Content
Q.
A.
# echo 2 > /proc/calypso/tunables/cm_logserverinfo# cat /arroyo/log/serverinfo.log.01132009Check the last set of output lines to see the current content states.
Object Count=37708 LengthUnknown=0CouldNotRepair=0 IsDamaged=0 BeingRepaired=0 BeingCopied=0needCrcValidate=37708 isFragFlag=0 isFragd=0 Defrag=0 Smooth=0BeingFilled=0 OutOfService=0 NeedsISACheck=0BeingDeleted=0 HasHoles=0 CopyHoles=0 SectorHoles=0Object Status Check Complete.Q.
A.
Q.
A.
# su - isa# cd ~/Berkeley# ./bsqlChoose the following options, one for each menu:
a.
b.
c.
Enter the content ID of the "stuck" ingest, then choose the exit option for each menu until you are back at the Linux prompt.
Q.
A.
# su - isa# cd ~/SDClientUpdate the SDClient.cfg file with the local IP address.
# ./sdClientFollow the SDClient menus.
Other
Q.
A.
# cat /arroyo/test/<server type>/setupfile# CServer core configuration. Changes to this file require# a server reboot.local 0 0 2 remote 0 0 2 fill 3 1 maxrate 900000 localip 0c0a80040localip 0c0a80040e1000 adapters: maxrate 965e1000 0: streaming 1 fill 0e1000 1: streaming 1 fill 0e1000 3: streaming 0 fill 1streamer 1 vault 1serverid 64groupid 64maxpacketsize 1316management eth0ingest eth0trickspeedsv2 10 0 0 0 0 0 0 0ftpout if eth0 max utilization mbps 0 max sessions 0fake cylindermap 1test 4Q.
A.
Also, when you see a line in the /arroyo/log/c2k.log.<date> file that says the following:
01-May-2007 17:40:44 UTC :err:ServeStream::reserveStream: refused streamhandle 4 for goid a445c9780e7f8f due to its load 3750, current load 0This entry typically means there are no stream ports linked. In the ten-second snapshot of the /arroyo/log/protocoltiming.log.<date> file, there is a line that shows load values for each of the major subsystems (LAN, memory, CPU, and so on). More than likely one of the subsystems is at 100, which is the subsystem that is having the problem.
Q.
A.
# su - isa# cd ~/StreamsDriver# touch DEBUGGING_ON# ~/IntegrationTest/debugging_on_off#./stop_driver#./run_driverQ.
A.
# echo 1 > /proc/calypso/test/readremoteserversCDS Content Quality Guidelines
This section covers the following topics:
•
Supported Elementary Stream Types
Video-only, audio-only (as well as audio streams with only a few or occasional video frames) and data-only streams are supported in addition to the customary multiplex of both audio and video.
Scrambling
The transport layer cannot be scrambled, meaning the transport header and any adaptation field must be in the clear. Streams whose Elementary Streams (ESs) are fully scrambled, including all start codes, are capable of being ingested and streamed, but are incapable of trick play.
For trick-play capability, the following cannot be scrambled:
•
•
•
Transport Bit Rate
All transport streams are constant bit rate (CBR). Variable bit rate (VBR) is not supported. The maximum bit rate is 35 Mbps. There is no minimum bit rate. The ES video bit rate, as specified in the MPEG-2 sequence header, is ignored. The bit rates of individual ESs do not matter. The CBR for an individual ES (in particular the video) is not required. All that is required is that the aggregate transport stream be CBR.
Streams containing MPEG-2 or AVC video are expected to conform to the appropriate buffer models spelled out in ISO/IEC 13818-1 and 14496-10.
Stream Length
All content must be at least one second in length. A content item must be under 12 hours in length or 15 GB, whichever comes first.
Format Restrictions
Following are the format restrictions for Advanced Video Coding (AVC), H.264, and MPEG-4:
•
•
•
•
Preferred Formats
Using the following guidelines improves the performance of the system, the quality of the tricks, and the trick transitions.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
