Table Of Contents
Release Notes for Cisco TV CDS 2.4.1
These release notes cover Cisco TV CDS Release 2.4.1.Revised: November 2010 OL-23699-01
The following information is in the release notes:
Release 2.4.1 of the Cisco TV CDS introduces the CDE250 hardware models 2S5 and 2S6.
The following enhancement has been added in Release 2.4.1:
Enclosure support—when a failed drive is displayed on CDSM, the corresponding drive's red light is displayed so it is easily identified.
Cisco TV CDS 2.4.1 Software Release supports the following environments and associated backoffice integrations:
–ISA with STUN handshake
The Cisco TV CDS Release 2.4.1 runs on the CDE250-2S5, CDE250-2S6 and CDE110 (if the CDE110 is used as a CDSM to configure, manage and monitor the stream domain with CDE250-2S5 and CDE250-2S6). See the Cisco Content Delivery Engine110 Hardware Installation Guide, and the Cisco Content Delivery Engine 205/220/420 Hardware Installation Guide.
Release 2.4.1 running on CDE250 interoperates with 2.2.x and 2.3.x running on CDE220 and CDE420 within a cluster.
Release 2.4.1 does not support the CDEs with the ServerWorks chipset. All CDEs with the ServerWorks chipset need to be replaced with the CDEs with the Lindenhurst chipset or the Next Generation Appliances (CDE110 and CDE250) before upgrading to Release 2.4.1.
See the "Related Documentation" section for links to the documentation online.
Table 1 shows all the platforms and software releases that have been tested with the CDE250-2S6 and the Cisco TV CDS 2.4.1 Software Release.
Limitations and Restrictions
Cisco TV CDS 2.4.1 Software Release has the following limitations and restrictions:
•This release can only be run on a CDE250-2S5 and CDE250-2S6.
•When adding one or more CDE250-2S5/CDE250-2S6 modules to a deployment cluster, the CDE110 for CDSM also needs to run with release 2.4.1.
•The fill bandwidth limit is 2000 SD streams when using Standard Ethernet Frames; it is 5000 SD streams when using Jumbo Ethernet Frames.
•When a CDE250 fails, there must be enough capacity on other streamers to which streams can failover; otherwise, they will be dropped.
This release contains the following open caveats:
In a configuration with streamers, caches, and vaults, when a large number of streams (100+) are started concurrently, the stream start rate is significantly slower when the data is not cached on the streamers or caches (when the data is filled from the vault). Also, it is worse for newly ingested content.
When the desired content isn't located on the streamer or cache server, the overhead to locate and fill the data is significantly higher with newly ingested content. Additionally, work needs to be done such as mapping the PAID to a GOID value. When multiple concurrent streams are started, this appears to become a bottleneck causing slower stream starts.
The best way to avoid this problem is to have the popular content located in the cache servers, which should happen naturally through normal use.
After streams have been playing awhile on a heavily loaded streamer, "out of capacity" errors are observed in the protocol timing log and streams are kicked off (moved to another streamer if available). If another streamer is not available, the stream may be dropped.
Because a Merced streamer using Standard Frames has a fill bandwidth limitation of around 2000 SD streams, if only part of the content is cached on the streamer, the streamer can have increasing resource needs as the stream play progresses. When this happens concurrently across thousands of streams, the streamer can get in a state where the 2000 SD fill bandwidth limit is no longer sufficient to meet the needs of the playing streams. When this happens, some streams will be failed over to other available streamers that have sufficient resourced to handle the load.
1. Run with jumbo frames between the streamer and the caches/vaults. When using jumbo frames, the fill bandwidth limit is increased to 5000 SD streams. This is sufficient for most loads.
2. Run with additional streamers in the array so that if a streamer gets bottle necked trying to fill content from a remote server, streams can be relocated to other streamers in the array.
When a CDE250 running a high load of streams fails, under some conditions some of the streams may not failover to a different streamer.
When streams failover to a new streamer, if the new streamer doesn't have enough available resources to host the stream, the stream cannot start playing on the new streamer.
For example, if 8000 streams are playing 1000 different programs, and the stream starts are spread across a 30 minute interval, each of the streams will be playing at a different place. The first streams started will fill the data from a remote cache/vault; later streams will be able to get the data from memory or from local disk at the streamer. Then if the streamer fails and the streams are relocated to another streamer, the new streamer may not have local copies of the desired data and all 8000 streams could need to fill the data from a remote cache/vault. And in addition to the regular stream data, the IGate data for the streams will also need to be filled. Because of the limited fill bandwidth on a CDE250 (2000 SD streams when using standard ethernet frames and 5000 SD streams when using jumbo ethernet frames) the new streamer may run out of fill bandwidth.
–Using jumbo frames on the network between the streamers and the caches/vaults greatly improves the fill bandwidth available to the streamers and helps avoid out-of-bandwidth problems.
–Having more streamers available to handle the failover load in the event of a streamer failure reduces the amount of work that any one streamer will need to do and reduces the probability that any one streamer will run out of resources when it tries to play the failover streams.
The following caveats have been resolved for Cisco TV CDS Release 2.4.1. Not all resolved issues are mentioned here. The following list highlights caveats associated with customer deployment scenarios.
Error in CDSM /arroyo/www/logs/error_log:
"/arroyo/www/htdocs/scripts" not found or unable to stat, referer
Create symbolic link in /arroyo/www/htdocs
ln -s /arroyo/www/htdocs/scripts /arroyo/www/htdocs/scripts
New DSCP markings get configured in the /arroyo/test/cache/setupfile and /arroyo/test/vault/setupfile. The values are selected from the Configure > Server Level > Server Setup page, under Assured Forwarding Class. There is no way to keep these settings out of the setupfile.
The VVIM configuration under normal operation is adding these lines. If the network is not configured to handle this marking, the packets will be dropped by the network.
While running cdsconfig on a streamer in a CDSM Domain ( Split Domain using HTTP as cache fill protocol), it doesn't accept a Server ID greater than 1000.
Run cdsconfig on a streamer in a CDSM Domain ( Split Domain using HTTP as cache fill protocol).
All servers on the platform get unregistered from their respective groups (stream/vault/cache/ssv).
The script that was implemented in 2.2 CDSM to automatically remove orphaned servers that remained attached to stream/vault/cache/ssv groups after being deleted was getting called with incorrect vars when customers had automated scripts logging into the CDSM. These automated scripts did not pass the correct parameters to the clean up function, and the result was that servers that were not orphans were incorrectly removed from groups.
The RemoteServers contains all streamers in a deployment, not simply the streamers that are part of the group.
This is normal.
RTSP requests are not being processed and/or are being processed very slowly (long latency between request and response).
CDS TV 2.3.x - RTSP service is incorrectly stopping threads when it tries to clean up requests for sessions that have been torn-down. As threads are stopped, this reduces the available thread pool which will increase latency times and eventually after all threads are stopped, RTSP requests will not be processed.
Upgrading to Release 2.4.1
For new installation and upgrade procedures, see the Cisco TV CDS 2.2 Installation, Upgrade, and Maintenance Guide.
The CDE250-2S6 streamer is only supported in the CDS-TV 2.4.1 release. So the CDSM has to be upgraded to the 2.4.1 image to configure the 2S6 streamer through the CDSM. Download the 2.4.1 image from the Cisco ftp download server.
The upgrade usually starts from the slave CDSM, then the master CDSM if CDSM Redundancy is enabled. If CDSM Redundancy is enabled, statsd runs on each CDSM. If CDSM Redundancy is disabled and only one CDSM is in service, statsd does not run on that CDSM.
Perform the following steps to upgrade the CDS-TV Software on the CDSM:
Step 1 Log in to the slave CDSM server as root.
Step 2 Shut down the avsdb process by entering the following command:# db_shutdown
a. Make sure "pgrep avsdb" returns no process ID (PID).
b. Make sure "netstat -an | grep 9999" returns nothing.
Step 3 Shut down the statsd process. Make sure "pgrep statsd" returns no process ID (PID).
Step 4 Backup the CDSM by entering the following command:[root@cdsm19 ~]# /home/upgrade/2_2/preupgradeStarting Backup of configuration and database filesChecking that all processes are stopped on the systemChecking that cserver is not running on the systemStarting Backup of files to: /root/cdsm19Backup of files completed.Creating Tarball of backed up filesTarball of backed up files created sucessfully!! IMPORTANT : Make sure that the directory /arroyo/backup is created on the machine to back up !!IP Address of machine to backup configuration/database files?: firstname.lastname@example.org's password:cdsm19.tgz 100% 3233KB 3.2MB/s 00:00Tarball uploaded to 172.22.102.21Scripts executed successfully !!!Please reboot the server and run the script 'upgrade' when the server comes back up.
Note The Remote Server 172.22.102.21 will have the cdsm19.tgz file in the /arroyo/backup directory.
[root@cdsm21 backup]# lscdsm19.tgz[root@cdsm21 backup]# pwd/arroyo/backup[root@cdsm21 backup]#
Step 5 Run the cdsinstall script to upgrade to the ISO image from your current running version to release 2.4. by entering the following command:[root@cdsm19 ~]# ./cdsinstall CDS-TV-PROD.isoSelect Deployment Type (ctrl-c to quit):1) CDSM1CDSM Selected[Omitted]
Step 6 Check the /arroyo/image/tags to verify the 2.4.1 ISO image is installed:[root@mc_cdsm_ccdn 2_2]# cat /arroyo/image/tagscdstv-2.4.1-b1
Step 7 Reboot the CDSM with /etc/rc.local and follow the statements if you want to automatically start avsdb and the Apache Server:#!/bin/sh## This script will be executed *after* all the other init scripts.# You can put your own initialization stuff in here if you don't# want to do the full Sys V style init stuff.touch /var/lock/subsys/local# Lines below this one modified by cdsflavconfig (ISA):su - isa -c "cd /home/isa"sleep 30/arroyo/www/bin/apachectl startsleep 30# statsd process should be running with CDSM Redundancy enabled only/home/stats/statsd -i <virtual-ip> -s <network-mask> -d <eth0>
Note avsdb, statsd (applicable if CDSM Redundancy is enabled), and the Apache Server are running if a CDSM is in a working state.
Step 8 Repeat Steps 1 to 7 to upgrade the master CDSM if CDSM Redundancy is enabled.
Adding a CDE250-2S6 Streamer
Setup the new 2S6 Streamer
After you install the 2S6 streamer into your rack, please make sure that it has the serial console and the management port (eth0) connected properly. You will need the serial console and the management network to configure the 2S6 streamer.
Once making sure the 2S6 streamer has cabling properly, power on the streamer to boot it up. Then start to configure it to your management network from the serial console.
Note The baud rate of the serial console has been set to 9600. Please make sure the serial port on the Terminal Server side has the baud rate set to 9600 as well.
The steps to configure the network for the 2S6 streamer are as:
1. Edit /etc/sysconfig/network to change the hostname and the gateway.
2. Edit /etc/syscofig/network-scripts/ifcfg-eth0 to change the IP address and the netmask.
3. Edit /etc/hosts to change the entry of the IP and hostname map.
4. Reboot the streamer to take your changes.
You need to edit the /etc/sysconfig/network file to modify the hostname and the gateway to match your need.[root@CCDN-2S6-1 ~]# cat /etc/sysconfig/networkNETWORKING=yesHOSTNAME=CCDN-2S6-1GATEWAY=172.22.102.1GATEWAYDEV=eth0
To change the IP address and the netmask, you need to edit the ifcfg-eth0 file and use the IP address and netmask you will have assigned for this streamer.[root@CCDN-2S6-1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0DEVICE=eth0BOOTPROTO=staticIPADDR=172.22.102.158NETMASK=255.255.255.0ONBOOT=yes
Edit the /etc/hosts file to change the IP address and the hostname to match yours.[root@CCDN-2S6-1 ~]# cat /etc/hosts# Do not remove the following line, or various programs# that require network functionality will fail.# Lines below this one modified by cdsnetconfig127.0.0.1 localhost.localdomain localhost172.22.102.158 CCDN-2S6-1
Initially Configure the CDE250-2S6
In this section, you will run cdsconfig to configure the 2S6 streamer and make it ready to be added into your working VOD cluster.
The cdsconfig script will configure its hostname, IP, group ID and server ID as well as create /home/isa/.arroyorc file with replication group member information. It will also create the configure files for the RTSP application.
To initially configure the CDE250-2S6 streamer, do the following:
Step 1 Log in to the 2S6 streamer as root.
Step 2 Run the cdsconfig script. The script displays prompts with the default values in brackets. If a default value is correct, press Enter to accept the value; otherwise, enter the correct value and press Enter.[rootCDE250-2S6-1 ~]# cdsconfigATTENTION!!!If a new image is installed on this server, a reboot is required before running cdsconfig.If a reboot is already performed, please continue.Otherwise, please exit and execute cdsconfig after rebooting the serverDo you want to continue ? (yes/no) [y]:Enter management interface [eth0]:Please ensure an IP address and netmask are configured formanagement interface eth0:Select an option or an interface to re-configure/disable:1. eth0 ip:XXX.XX.XXX.XXX mask:255.255.255.255 bcast:XXX.XX.XXX.XXX2. Configure another interface3. DoneChoice :Backing up old scripts in /etc/sysconfig/network-scriptsWriting new ifcfg-ethX scriptsEnter a hostname [CDE250-2S6-1]:Enter the number of the eth interface that connects to the gateway :Enter the default gateway IP address [XXX.XX.XXX.X]:Backing up /etc/sysconfig/networkWriting new /etc/sysconfig/networkBacking up /etc/hostsWriting new /etc/hostsShutting down interface eth0: [OK]Shutting down loopback interface: [OK]PCI: Enabling device 0000:01:00.0 (0000 -> 0003)PCI: Enabling device 0000:01:00.1 (0000 -> 0003)Restarting network services, this may take a minute:Shutting down loopback interface: [OK]Bringing up loopback interface: [OK]Bringing up interface eth0: [OK]Network services restarted; may take a few seconds to establish connectivityReboot for hostname changes to take effectNetwork configuration completePlease choose your platform from the following list of valid platforms:1. CDE250-2S52. CDE250-2S63. CDE250-2G3Choice: 2Please select a device role:1. streamerChoice: 1Is this Server going to get added to a Stream Domain in a CDN Split Domain Management Environment ? (yes/no) [y]: yesIs this Stream Domain going to use CCP as Cache Fill Protocol? (yes/no) [y]: noPlease enter a Group ID(Array ID): 666Please enter a server ID : 158Enter Stream Control interface (Hit 'Enter' to skip):Writing new configuration to /home/isa/.arroyorcNo existing replication group information foundDo you want to configure replication group members now? (yes/no) [y]: yesThere are currently no replication group members.Do you want to add another replication group member? (yes/no) [y]: yesSelect a role for the new replication group member. Select 'exit' to exit this menu:1. ssv2. vault3. streamer4. cache5. controller6. exitChoice: 3Enter an IP address for new streamer: XXX.XX.XXX.100Current replication group members:streamer XXX.XX.XXX.100Do you want to add another replication group member? (yes/no) [n]: yesSelect a role for the new replication group member. Select 'exit' to exit this menu:1. ssv2. vault3. streamer4. cache5. controller6. exitChoice: 3Enter an IP address for new streamer: XXX.XX.XXX.101Current replication group members:streamer XXX.XX.XXX.100streamer XXX.XX.XXX.101Do you want to add another replication group member? (yes/no) [n]: yesSelect a role for the new replication group member. Select 'exit' to exit this menu:1. ssv2. vault3. streamer4. cache5. controller6. exitChoice: 3Enter an IP address for new streamer: XXX.XX.XXX.102Current replication group members:streamer XXX.XX.XXX.100streamer XXX.XX.XXX.101streamer XXX.XX.XXX.102Do you want to add another replication group member? (yes/no) [n]: yesSelect a role for the new replication group member. Select 'exit' to exit this menu:1. ssv2. vault3. streamer4. cache5. controller6. exitChoice: 3Enter an IP address for new streamer: XXX.XX.XXX.102Current replication group members:streamer XXX.XX.XXX.100streamer XXX.XX.XXX.101streamer XXX.XX.XXX.102streamer XXX.XX.XXX.103Do you want to add another replication group member? (yes/no) [n]: yesSelect a role for the new replication group member. Select 'exit' to exit this menu:1. ssv2. vault3. streamer4. cache5. controller6. exitChoice: 3Enter an IP address for new streamer: XXX.XX.XXX.104Current replication group members:streamer XXX.XX.XXX.100streamer XXX.XX.XXX.101streamer XXX.XX.XXX.102streamer XXX.XX.XXX.103streamer XXX.XX.XXX.104Do you want to add another replication group member? (yes/no) [n]: yesSelect a role for the new replication group member. Select 'exit' to exit this menu:1. ssv2. vault3. streamer4. cache5. controller6. exitChoice: 5Enter an IP address for new controller: XXX.XX.XXX.200Current replication group members:. streamer XXX.XX.XXX.100streamer XXX.XX.XXX.101streamer XXX.XX.XXX.102streamer XXX.XX.XXX.103streamer XXX.XX.XXX.104. controller XXX.XX.XXX.200Do you want to add another replication group member? (yes/no) [n]: noConfiguring RTSP ecosystemIs this an NGOD deployment? (yes/no): yesChoose NPT Syntax:1. NGOD2. NGOD_SC3. StandardChoice [NGOD]: 3Writing /home/isa/bss/scripts/arroyo-env.shWriting /home/isa/bss/scripts/arroyo-site-env.shSetting attributes for AVSRTSPServerDo you want to enable Redirect Server ? (yes/no) [y]: noIs this node getting added to an existing deployment ? (yes/no) [y]: yesmv: cannot stat `/arroyo/db/start_db': No such file or directoryStarting database sync...
Step 3 If you see the message "Database Unavailable Cannot Set Things UP!" on the output of the cdsconfig command, the database was not ready when cdsconfig executed the svrinit_15 command. So, you need to wait till the database is ready and enter the svrinit_15 command manually to seed the database with a server map record so that CDSM can configure, manage and monitor the server.
To make sure the database up running properly, verify that two-way TCP connections exist for the database on this server to its remote servers by entering the following command:# netstat -an|grep 9999tcp 0 0 0.0.0.0:9999 0.0.0.0:* LISTENtcp 0 0 XXX.XX.XXX.XXX:54625 XXX.XX.XXX.XXX:9999 ESTABLISHEDtcp 0 0 XXX.XX.XXX.XXX:9999 XXX.XX.XXX.XXX8:51958 ESTABLISHEDtcp 0 0 XXX.XX.XXX.XXX:9999 XXX.XX.XXX.XXX:56122 ESTABLISHEDtcp 0 0 XXX.XX.XXX.XXX:45778 XXX.XX.XXX.XXX:9999 ESTABLISHED
After the database up and running correctly, enter the following svrinit_15 command to initialize the database tables for this server:# /home/stats/svrinit_15 -h <hostname> -i <IP-address> -s <netmask> -gw <gateway-IP> eth0
Step 4 Restart the statsd process.
The cdsconfig script starts the database and statsd processes. But the statsd process starts without the netmask. Restart the statsd with the netmask by entering the following command:Root# /home/stats/statsd -i XXX.XX.XXX.XXX -s 255.255.255.255 -d eth0
Configuring a CDE250-2S6 Streamer through the CDSM
This section provides the procedure for configuring a CDE250-2S6 Streamer through the CDSM.
Reset the RTSP Deployment Type
Step 1 Login to the CDSM with the user id arroyo and password arr0y021.
Step 2 Go to Maintain -> CDSM Setup page and change the RTSP Deployment Type to RTSP from NGOD, then submit.
Step 3 Change the RTSP Deployment Type back to NGOD and submit. Now you can configure the new CDE250- 2S6 Streamer for the NGOD environment.
Note When you are adding a new streamer into the CCDN cluster, CDSM configures it as a pure RTSP streamer if you do not switch the RTSP Deployment from NGOD to RTSP and then switch back to NGOD.
Configure the 2S6 streamer into the exist stream group
Step 1 Configure Network Interfaces on the 2S6 streamer by going to the Configure -> Server Level -> Interface Setup page, selecting the IP of the 2S6 streamer from the pull down menu and pressing Display.
Step 2 The Interface Setup page for this streamer appears. Fill out the information for the ports based your requirement and then submit the page.
Note The port eth2-5 are the four 1Gb copper NICs on the Quad card and the port eth6-9 are the four 10Gb fiber NICs.
Note Use the 1GE copper NICs on the 2S6 streamer although they work for the cache/stream network. These ports are not yet verified for streaming with a large load. We cannot guarantee the number of streams you could get from the four 1GE NICs at this time.
Step 3 Configure the Server Setup for the 2S6 streamer.
For example, you can configure the stream group to use two SOPs. Each SOP has its own subnetwork, so when adding the 2S6 streamer, assigned all eight NICs as follows:
–eth2-3(1GE) and eth6-7(10GE) are assigned for subnetwork 1 for SOP1.
–eth4-5(1GE) and eth8-9(10GE) are assigned for subnetwork 2 for SOP2.
You can see the assignment from the following Server Setup page.
Step 4 Configure Routing Table for the 2S6 streamer.
Step 5 Add the streamer to the exist stream group.
The following sample adds the streamer 172.22.102.158 to stream group strmg1. The first figure is before adding in the streamer and the second figure is shows after the streamer was added.
Step 6 Resubmit the Stream Group to Cache Group Map.
Step 7 Resubmit the Control/Setup IP page to push the configure files.
Step 8 Configure the RTSP Server on RTSP Setup page.
You will need to configure the RTSP Server with filling out the RTSP Setup page.
Step 9 Resubmit Stream Destination on QAM Gateway page.
Refer to the following documents for additional information about the Cisco TV CDS 2.4:
•Cisco TV CDS 2.2 ISA Software Configuration Guide
•Cisco TV CDS 2.2 RTSP Software Configuration Guide
Note Both the Cisco TV CDS 2.3 ISA Software Configuration Guide and the Cisco TV CDS 2.3 RTSP Software Configuration Guide are available through the CDSM GUI.
•Cisco TV CDS 2.1-2.2 API Guide
•Cisco TV CDS 2.2 Installation, Upgrade, and Maintenance Guide
•Cisco Content Delivery System 2.x Documentation Roadmap
•Cisco Content Delivery Engine 205/220/250/420 Hardware Installation Guide
Note This document will be provided by your Customer Support representative.
•Cisco Content Delivery Engine 110 Hardware Installation Guide
•Regulatory Compliance and Safety Information for Cisco Content Delivery Engines
•Adding CDE250-2S5/2S6 Streamer Procedure (EDCS-884951)
Note This document will be provided by your Customer Support representative.
The entire CDS software documentation suite is available on Cisco.com at:
The entire CDS hardware documentation suite is available on Cisco.com at:
Obtaining Documentation and Submitting a Service Request
For information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What's New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at:
Subscribe to the What's New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free service and Cisco currently supports RSS version 2.0.
CCDE, CCENT, CCSI, Cisco Eos, Cisco Explorer, Cisco HealthPresence, Cisco IronPort, the Cisco logo, Cisco Nurse Connect, Cisco Pulse, Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1005R)
Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.
© 2010 Cisco Systems, Inc. All rights reserved.