Cisco on Cisco

Streaming Video Case Study: How Cisco Designed and Deployed IPTV for Global Communications


IPTV architecture reduces travel expenses, improves employee communication, and uses WAN to transport video efficiently.
BACKGROUND

Cisco IP/TV broadcasts are one of several ways that Cisco Systems® delivers multimedia content, some of the others being virtual classrooms and video on demand (VoD). In general, Cisco uses Cisco IP/TV broadcasts for providing high-quality, live, or prerecorded information to large audiences of more than 50 people. In contrast the virtual classroom solution works best for audiences of fewer than 50 that require frequent live interaction, demos, audience polling, or question-and-answer sessions. VoD offers a low-cost means of providing individual users with pre-packaged information at any time, from any networked location, without the live interaction provided by Cisco IP/TV broadcasts or the virtual classroom. At Cisco, participants who miss a live Cisco IP/TV broadcast or virtual classroom session can usually view a recording of it within 24 hours using VoD.

CHALLENGE

In the fast-paced networking industry, giving employees timely information about new market opportunities and products can make a critical difference in capturing early market share. In its early days, Cisco held in-person meetings for product launches and important company updates. In 1995, to cut travel expenses and communicate directly with more people, Cisco began traditional video broadcasting, renting a TV studio and movie theaters around the country. After the broadcasts, the company would ship VHS tapes (and later CD-ROMs) to offices for the convenience of employees who missed the live broadcasts.

Figure 1. Simultaneous IP Multicast and Unicast Video Streams for Internal and External Audiences

Click on Image to Enlarge popup

“While traditional video proved more cost-effective than sending thousands of employees to instructor-led courses or flying a team of presenters to multiple sites for a product launch, it still cost Cisco approximately $200,000 per event, effectively limiting us to one to two events per quarter,” says Mike Mitchell, director of Cisco Media Network.

In 1997 Cisco began to supplement traditional video broadcasting with IP video streaming to certain locations. The solution: Cisco IP/TV software from Precept Software, a company that Cisco later acquired. Cisco IP/TV software delivers full-screen, full-motion video to desktop PCs over private IP networks and the public Internet. “We still rented the TV studio but no longer needed to rent theaters in those regions that received the Cisco IP/TV stream, which reduced costs,” says Mitchell. In 1999 Cisco decided to deploy Cisco IP/TV technology everywhere, with the goal of improving productivity by eliminating travel time, strengthening the corporate culture by communicating directly and more frequently with more people, and reducing travel costs.

To deliver live broadcasts across its diverse global network, the company faced several technical challenges. One was finding the best way to deliver the video stream at the highest bit rate that each individual office could handle without overwhelming lower-bandwidth WAN links in the enterprise. This challenge was especially daunting in parts of the world where high-bandwidth connections are costly, including Europe, the Middle East, and Africa; Asia and the Pacific region; and South America. Another challenge was to automate the process of configuring thousands of Cisco routers across the enterprise for live broadcasts, a time-consuming task if performed manually. Yet another challenge, which emerged later, was to deliver the live broadcasts not only to Cisco employees in offices with WAN connectivity, but also to customers, partners, and teleworkers.

SOLUTION

Cisco solved the technical challenges of live broadcasts with a combination of multicast and unicast technologies. Using Cisco IP/TV products and the Cisco Application and Content Networking System (ACNS) Software, Cisco simultaneously produces IP Multicast video streams for employees connected to the Cisco network and unicast video streams for telecommuters and external audiences (Figure 1).

Table 1 shows the variety of video streams that Cisco delivers to audiences with different operating systems and connection types:

Table 1. Video Stream Offered to Each Audience
Audience Transmission Technology Encoder Bit Rate
Cisco employees with Windows desktops and high-bandwidth WAN connections either using IP satellite or terrestrial connections IP Multicast Cisco IP/TV
Broadcast Server
900 kbps
Cisco employees with Windows desktops and low-bit-rate, terrestrial WAN connections IP Multicast Cisco IP/TV Broadcast Server 100 kbps
Cisco employees with UNIX/Linux workstations and WAN connections IP Multicast RealMedia 300 kbps
Customer and partners over Internet and VPN IP Unicast RealMedia 300, 100, 56, and 14 kbps
Cisco teleworkers with Windows desktops over VPN IP Unicast Windows Media 100, 56, and 14 kbps


IP Multicast over the WAN
“In many companies, when changes occur, the president tells the vice presidents, who tell the directors, who tell the managers, who tell individual contributors. This serial process creates delays, and each human ‘hop’ can dilute or distort the message. Using Cisco IP/TV broadcasts, Cisco can disseminate consistent information to its employees, on a timely basis, enterprisewide.”

Cisco uses IP Multicast to deliver live broadcasts to employees using Windows PCs or UNIX/Linux workstations that are directly connected to the network. Whereas IP Unicast sends one stream per client—quickly devouring bandwidth as audience size increases—IP Multicast provides a single stream that is available to all desktops on the WAN. This preserves network bandwidth and facilitates scalability because a given broadcast uses the same amount of bandwidth on WAN links no matter how large the audience.

In the diverse Cisco global enterprise network, remote offices have WAN links with varying bandwidths, and Cisco employees use different platforms. Therefore, Cisco generates three video streams per broadcast. Employees with Windows desktops and high-bandwidth connections receive a 900-kbps MPEG-1 video stream; those with Windows desktops and low-bandwidth connections receive a 100-kbps MPEG-4 video stream; and those with UNIX/Linux systems receive a 300-kbps RealMedia video stream. Each stream is produced by a separate Cisco IP/TV Broadcast Server that receives the audio and video signals directly from the camera.

Directing each stream to the appropriate WAN links required careful planning at the outset. “If we were to send out a single 900-kbps stream of IP Multicast traffic, we’d have problems delivering it over, say, a single 1500-kbps T1 WAN link,” says Mike Anderson, network engineer. To solve the problem, the team assigned specific bit rates to ranges of IP Multicast addresses. They used the address ranges to build standard boundary access control lists (ACLs) that could be applied to WAN links of specific bandwidth to ensure that high-bit-rate IP Multicast wouldn’t flow across low-bandwidth WAN links. These administrative scopes are also used to determine the group address that should be allocated to someone responsible for an application or server sending IP Multicast traffic. For example, a Cisco IP/TV server sending a 900-Kbps MPEG-1 video stream would get a group address in a different administrative scope from a multicast music-on-hold server sending out a G.711 64-Kbps audio stream. The result: A group address from the administrative scope reserved for high-quality video cannot be passed over a low-bandwidth WAN link that is unable to handle the traffic. “Administrative scopes are a key enabler for scaling multicast,” says Anderson.

The Cisco IP/TV solution includes the following components:

  • Cisco IP/TV Content Manager— Residing at headquarters, the Cisco IP/TV Content Manager is the policy manager that communicates scheduling information and desired video encoding formats to the Cisco IP/TV broadcast servers. It also generates a program listing and handles network and device configuration and management. Administrators can monitor the usage and quality of streams delivered throughout the multicast network using the Cisco IP/TV StreamWatch monitoring tool. StreamWatch shows both the status and number of users.
  • Cisco IP/TV broadcast servers— Residing at regional offices, Cisco IP/TV broadcast servers encode and multicast video streams according to the directions received from the Cisco IP/TV Content Manager. The source of the video stream can be video cameras, VCRs, DVDs, satellite and cable feeds, or prerecorded Windows media, AVI, MP3, and MPEG files.

Figure 2. IP Satellite Network Service Augments the Corporate WAN for Bandwidth-Intensive Video Files

Click on Image to Enlarge popup

IP Multicast over an IP Satellite Network Service

High-speed WAN connections are expensive in certain regions where Cisco operates, including Europe, South America, and certain parts of North America. In these locations, the IP satellite network augments the WAN (Figure 2). “A one-way satellite service is attractive because it can deliver content to all offices in its footprint at a fraction of the cost of providing equivalent terrestrial bandwidth,” says Anderson. Cisco frequently re-evaluates its transport options as the cost of higher-speed lines changes.

IP Unicast for Customers and Partners

When Cisco decided to extend select Cisco IP/TV broadcasts to customers and partners across the Internet in 2000, the company deployed an IP Unicast solution to complement its IP Multicast solution. Unicast streams are sent from a single source to a single destination.

The video stream is sent from the camera to an encoder—Cisco uses several from members of the Cisco Ecosystem Partner Program—for encoding at different bit rates. From there, the stream is sent to the Cisco External Media Hub, located at Cisco headquarters in San Jose, California (Figure 3). The Cisco External Media Hub comprises Windows Media and RealMedia streaming video servers and acts as the egress points for streams sent to external audience members. Rather than sending the stream directly to external audience members, which would unnecessarily increase traffic on the Cisco network, the Cisco External Media Hub sends the stream to a global Internet service provider, which then directs the stream to the viewer from the closest server.

Figure 3. IP Unicast Streaming Video for Customers and Partners Connected Through the Internet

Click on Image to Enlarge popup

Windows Media IP Unicast for Remote Employees

Cisco also provides live unicast streaming video for employees who access the corporate intranet through an IP-VPN connection. Remote employees connect to a local VPN concentrator through the Internet and are directed to the nearest streaming Cisco ACNS content engine. The current Cisco content delivery network (CDN) provides unicast pull-splitting technology (Figure 4). This enables the Cisco ACNS to pull a live stream from the Windows Media or RealMedia encoder and then provide that stream to locally connected clients. Most Cisco ACNS content engines also provide other services, including caching of Web and VoD traffic.

IP Multicast for Employees with UNIX and Linux Workstations

In 2002 Cisco also began using RealMedia encoding from RealNetworks over IP Multicast to deliver live broadcasts to Cisco employees who use UNIX and Linux workstations. “It’s a key tenet of Cisco culture to use the best tool to serve your clients instead of being overly attached to any one technology,” says Dan Stolt, IT ACNS program manager. “By combining Windows and RealMedia via multicast and unicast, we’re serving all our clients—internal and external, and Windows users as well as UNIX/Linux users—in the best way available.”

Automated Router Configuration

Cisco takes precautions to ensure that all routers worldwide are configured consistently. “A consistent, accurate configuration across the entire global network is essential for reliable content delivery and for successfully adding new technologies,” says Anderson.

Manually implementing new technologies on the thousands of routers in the Cisco enterprise would require many worker-hours while engineers executed repetitive configuration commands. Therefore, Cisco automated a great deal of the network configuration using custom scripts or the autoconfig Cisco network management tool that maintains consistency across all routers. Anderson’s group used automated Practical Extraction and Reporting Language (PERL) scripts to configure multicast commands at the interface level. Recently, for example, Cisco implemented Source Specific Multicast (SSM), which is an extension of Protocol Independent Multicast (PIM). SSM simplifies the multicast setup process, providing more stability or reliability to the multicast sessions. "The team created PERL scripts to deploy the SSM and Internet Gateway Management Protocol Cisco IOS® Software configuration commands that the regional network engineers needed to execute," says Anderson. Cisco also created a script that tracked the adoption rate of SSM globally so that each region knew its exact status. “The entire project took about three months, and most of that time was devoted to coordination and management,” says Anderson. “Manual migration would have taken about twice as long.”

Figure 4. Unicast Streaming Video for Remote Employees Connecting Through the IP VPN Link

Click on Image to Enlarge popup

Experience of Cisco IP/TV Users

Cisco informs employees, partners, and customers of upcoming live events through e-mail, electronic newsletters, and Websites. A full schedule is also posted on the Cisco broadcast calendar found on the Cisco Media Network site. Users can click a link in the publication to begin viewing the broadcast at the specified time.

By opening the Cisco IP/TV Viewer on their desktops, employees can search daily, weekly, or monthly programming information, watch the program, or select a stream. The image size of the video is 320 by 240 pixels. Employees can test their reception before the event because a program or test stream is always being broadcast. Employees who work from a remote location can use a standard Web browser to view broadcasts, which in this case are transmitted by unicast.

Employees don’t need to select the video stream with the correct bit rate for their circumstances. Rather, the solution determines the appropriate bit rate by mapping the employee’s IP address to a city, which corresponds to a certain bit rate.

Employees who miss a live event can view an on-demand version of the broadcast, often available just 24 hours later.

RESULTS

On average, Cisco hosts nearly 50 live streaming broadcasts for about 11,250 viewers per month. The average number of viewers per broadcast is 250; the largest audience has been about 5000 Cisco employees, for a company meeting. With frequent live communications over its IP network, Cisco benefits in the following ways.

  • Productivity gains— Suppose 400 Cisco employees watch a Cisco IP/TV broadcast of a new product introduction. Without the Cisco IP/TV solution, about 10 percent of those people would have traveled to see the broadcast live, losing productivity during the travel time. “With the Cisco IP/TV solution, not only do those employees save the expense of travel, they’re also able to receive and act on information more quickly,” says Mitchell. The productivity gains are especially valuable for the salesforce because the travel time they save can be spent with customers.
  • Maintenance of corporate culture— For geographically dispersed companies such as Cisco, maintaining a consistent corporate culture presents a challenge, especially during times of rapid change, such as acquisitions. “In many companies, when changes occur, the president tells the vice presidents, who tell the directors, who tell the managers, who tell individual contributors,” says Mitchell. “This serial process creates delays, and each human ‘hop’ can dilute or distort the message. Using Cisco IP/TV broadcasts, Cisco can disseminate consistent information on a timely basis, across the enterprise.”
  • Travel avoidance and other cost savings— “Cisco saves tens of millions of dollars directly by using the Cisco IP/TV solution and other multimedia content solutions to avoid travel,” says Mitchell. “Indirectly, we are saving millions more every year in productivity improvements.”

Since switching to an IP-based video system, Cisco has seen the burdened cost per broadcast drop from about $200,000 to $8,000. In addition, all Cisco live streaming videos are now automatically captured and archived in the onsite studio while programs are streaming to the audience. Immediately after the broadcasts, these captured video assets are converted to VoD format. This capability offers tremendous cost savings by eliminating the estimated $2000-per-hour cost of video production once spent on an outside vendor to re-encode a video into VoD format. It also eliminates the previous week-long wait while vendors transferred videotaped versions of Cisco programs to VoD.

“Broadcasting live events via Cisco IP/TV technology is the least expensive method,” says Mitchell. “Nobody told us, ‘Use the IP network because this is Cisco.’ Rather, I was directed to find the most cost-effective way to deliver live content to the most people, and Cisco IP/TV technology emerged as the clear choice.”

More frequent education on more topics—Because the costs of video production have dropped so dramatically, Cisco has expanded the scope of content from product introductions to business unit “all-hands” meetings and company meetings. Forty percent of broadcasts pertain to new product introductions for the salesforce, another 40 percent involve corporate communications, and 20 percent focus on formal training, including technology training and human resources training.

LESSONS LEARNED

Mitchell notes that it is important to build a strong partnership between a company’s IT and business staffs. “Without your IT team, any message produced by the business may never reach the intended audience. By the same token, it’s important for the IT team to build the capacity to accommodate the business requirements. At Cisco, we say, ‘If the business does it without IT, nobody can use it. If IT does it without the business, nobody will use it.’ ” Anderson concurs: “To develop and maintain a reliable, end-to-end solution, it’s absolutely critical that engineers and content producers cooperate closely. At Cisco, IP/TV technology matters to IP Multicast engineers because it’s the number-one IP Multicast application. And IP Multicast is important to the Cisco IP/TV group because without it, the Cisco IP/TV solution couldn’t scale globally. The ultimate beneficiary of the collaboration between both groups is the end user.

NEXT STEPS

Cisco intends to incorporate the features of its Cisco IP/TV solution into the Cisco ACNS solution after version 5.1 is released. By consolidating the infrastructures for live video and VoD, Cisco IT will reduce both capital expenditure and administrative costs associated with configuration and maintenance. The Cisco ACNS solution includes a device called the Cisco Content Distribution Manager (CDM), which pushes device configurations to up to 2000 Cisco ACNS devices at a time. “Currently, we dedicate two high-level, full-time system administrators to the Windows 2000 servers for Windows Media and RealMedia,” says Stolt. “After we consolidate both of them on the ACNS solution, a contractor with lesser skills will spend a fraction of that time ensuring that configurations are pushed out. That will free our administrators to focus on other projects.”

Cisco also plans to replace both Windows Media unicast and RealMedia multicast with MPEG-4, the new International Organization for Standardization format for high quality at low bandwidths. “With MPEG-4 we’ll be able to send out a single, standards-based stream instead of two streams, and anyone with a standard browser will be able to view live broadcasts,” says Stolt.

To use its own experiences with Cisco IP/TV technology for the benefit of its customers, Cisco has partnered with third-party content authoring and management vendors to offer customers a bundled solution. For more information about this solution, go to www.cisco.com/go/video.