Carrying audio conferencing on corporate network reduces costs for web conferences.
For years Cisco employees have increasingly used online web conferences in place of in-person meetings and audio-only conference calls. This growth in web conferences at Cisco was due to several factors:
Company policies for reducing costly travel
Employees working more frequently from home, remote locations, and / or mobile
Growing numbers of global teams, with members working together from around the world
Participants' increasing comfort with video and online meetings
Availability of Cisco WebEx® features for sharing video, desktop applications, and documents in a web conference
Easy access to Cisco WebEx from multiple desktop platforms and mobile devices
Ability for non-Cisco employees (e.g., partners, customers, and prospects) to attend online meetings / web conferences
WebEx usage at Cisco started at more than one million audio minutes per month in 2007. This usage very quickly reached nearly three million audio minutes per month within the first year of deployment. Because this growth showed no sign of slowing down, Cisco IT grew more concerned about the costs of the audio portion of these meetings.
When it was first deployed inside Cisco, the WebEx service carried these growing numbers of call minutes over the toll public switched telephone network (PSTN) (see Figure 1).
Figure 1. Audio Call Flow in the Cisco WebEx Legacy Audio Solution
As Cisco global WebEx adoption grew (see Figure 2), so did their voice call costs. In 2012, about 25 percent of all Cisco WebEx voice traffic was international. Phone calls using traditional service provider services (e.g., audio teleconference bridges) from around the world range greatly in price, from a few cents to a few dollars per minute. Our mix of domestic and international voice calls cost Cisco an estimated 2.7 cents per minute, which brought early PSTN charges to almost US$1 million per year. Cisco had also invested in a global IP backbone for voice, video, and multimedia services, which were not being leveraged to a great extent with these user workflows.
Figure 2. WebEx Meeting Growth at Cisco
The unpredictable and growing cost of WebEx audio at Cisco was significant, and difficult to estimate for our budget planning. Even worse was the possibility that growing costs would cause user groups inside Cisco to stop using advanced collaboration tools. If costs forced employees to revert to limited voice conferencing capabilities, it would slow their teams' productivity and their ability to collaborate on a global scale. To preserve the movement toward greater collaboration, and at the same time reduce and stabilize collaboration costs, Cisco IT needed to reduce the cost of WebEx audio calls and leverage existing investments in a cloud and unified communication architectures.
Cisco IT implemented a simple and scalable network design that grew to handle, by early 2012, more than 125 million WebEx audio minutes per month (Cisco is likely to exceed 1.5 billion audio conferencing minutes in 2012). The new design incorporates the WebEx audio traffic into the more cost-effective call routing across the company WAN. Rather than using PSTN trunking, it delivers the audio portion of a WebEx conference to the Cisco WebEx servers over Session Initiation Protocol (SIP) trunks from the Cisco IP telephony network. Calls carried via IP across the internal WAN and over the SIP trunks are significantly less expensive than calls carried across the PSTN, because Cisco has already paid the expense for the WAN infrastructure, and little if any additional bandwidth needed to be provisioned to carry the additional WebEx voice traffic. This network design is now available commercially to WebEx customers as Cisco WebEx Cloud Connected Audio (see Figure 3).
In this design, the voice portion for web conference sessions travels over the customers' existing IP telephony network, not the PSTN. The design moves all on-net calls to the WAN, and uses the existing WAN gateways to connect guest and off-net PSTN users, which substantially reduces WebEx audio conferencing costs.
How It Works
To carry these audio conference calls on the Cisco IP telephony network instead of the PSTN, Cisco IT deployed SIP trunks connecting the Cisco IP telephony network to the WebEx collaboration cloud. These SIP trunks transport the conference call traffic between the WebEx collaboration cloud and the Cisco WAN, which already carries all internal IP telephony traffic in the most cost-effective manner possible. The Cisco WAN then connects to individual participants at global Cisco sites, or to local or remote PSTN WAN gateways and then over the PSTN, depending on each participant's location.
Today, the infrastructure connecting WebEx conferencing with the Cisco WAN encompasses four major elements (see Figure 4).
Dedicated trunks to the WebEx cloud. Cisco IT deployed a pair of 10 Gigabit Ethernet links between the WebEx service provider gateway in Mountain View and the Cisco WAN gateways in San Jose, California, and another pair of Gigabit Ethernet links between the WebEx service provider gateway in London and the Cisco WAN gateway near London.
Cisco Unified Communications Manager (UCM) clusters. A single eight-subscriber cluster running Cisco UCM Session Manager Edition (SME), located in San Jose, California and Richardson, Texas, is dedicated to conferencing traffic. This cluster acts as the head-end system for switching calls between the different network destinations and call types (i.e., audio, video). "By using the Cisco UCM Session Manager Edition to manage the user calls, we can easily accommodate new WebEx features without changing the backend network infrastructure," says Kees Gerritsen, unified communications design engineer, Cisco IT. "Fewer changes means Cisco IT can maintain stability in the PSTN and SIP infrastructure that we use for conferencing. It also means that we can provide our users with the latest version of WebEx, with the latest features, without going through major upgrades to the infrastructure."
Cisco 3845 Integrated Services Routers (ISRs). A total of 32 ISRs handle as many as 10,000 concurrent user calls for the 33,000 WebEx conferences held by Cisco employees with each other, and with our partners and customers each business day. (Other ISRs, such as the Cisco 3900 series, would also work; but Cisco IT deployed the 3845s before the 3900s were available.) Together with the Cisco UCM configuration and sufficient bandwidth in the SIP trunks, this deployment is designed to handle the thousands of join and exit transactions for conferences during the peak time periods. "Your call-processing system really needs to be balanced as a whole in order to handle these traffic spikes, which typically occur at the beginning of each hour throughout the day," says Gerritsen.
Cisco Session Border Controller (SBC) on the Cisco Unified Border Element (UBE). The Cisco ISRs run Session Border Controller software, acting as a Cisco Unified Border Element. This UBE manages call signaling on the SIP trunk that connects the Cisco UCM SME and the WebEx audio bridge. All audio calls for a conference, whether they originate from on-net or off-net phones, are routed by the Cisco UCM SME systems, using the SBC, to the WebEx cloud over the SIP trunk. The Cisco UCM SME system also hosts the conferencing access numbers, and all callbacks to users from WebEx are made on the Cisco network via the SBC.
Costs of Cloud Connected Audio
Cisco IT worked with a service provider to install and maintain the two pairs of SIP trunks that connect to the WebEx service. Cisco IT maintains the UBE gateways at the Cisco sites and covers the cost of the Cisco UCM conferencing cluster. In addition, WebEx charges a per-port price, to cover the cost of bridging the voice calls. Last, although most of the calls travel most of their distance across the Cisco WAN, Cisco IT maintains voice gateways in major hub locations around the world to connect to off-net callers and pays the local telephony charges in each country for each outbound call. This off-net calling has always been a function of the Cisco IP telephony network, but migrating to the Cloud Connected Audio design added extra voice traffic onto the WAN and its gateways.
User Access to WebEx Services
Participants, whether on-net employees calling from within the WAN or off-net employees or guests calling from the PSTN, can join the audio portion of a WebEx conference in one of three ways:
As an inbound call to the conference bridge, by dialing a local or toll-free number from any device, including PSTN, mobile, and voice over IP (VoIP) Internet devices.
As an outbound call, by joining the web conference first from a PC, mobile smartphone, or tablet, then using the callback feature to join the audio portion on a separate telephone. In this case, the user can specify any telephone number, whether on-net to a Cisco phone number or off-net to the user's mobile or landline phone. Cisco IT selected this option to make it easier for employees to connect to WebEx meetings and easily get on board and adopt the service. Today about 85 percent of WebEx connections to Cisco meetings are via dial-back.
As a VoIP call directly from the WebEx client, by selecting "connect using Internet" directly from the client itself (see Figure 5). All voice packets are carried as part of the IP data stream, without special voice quality of service (QoS) support but without requiring any traffic to be carried over a separate voice network.
Figure 5. WebEx Connection Screen
Reducing Costs through Call Routing
For off-net callbacks, Cisco IT uses existing carrier terminations with Tail End Hop Off (TEHO) routing where permitted by local regulations (see Cisco IT case study on TEHO). Calls are carried on the WAN as close as possible to the destination endpoint, and then hop off at a more cost-effective voice gateway to complete the last few miles across the PSTN. In countries where Cisco cannot use TEHO (e.g., India, Pakistan, and most of the Middle East), Cisco IT manages costs by routing the audio calls to take advantage of the lowest long-distance charges available in each world region.
The Cisco IT voice team reviews WebEx usage data monthly to verify that network capacity and QoS configurations are adequate to serve the call demand. This review also includes an analysis of call detail records (CDRs) from the Cisco UCM, as well as WebEx-provided data. The team also runs a script in the Cisco network management system to check that router configurations are set correctly for the traffic levels.
As usage grows, Cisco IT can increase capacity to serve more WebEx conferences by adding more Cisco UBE routers, adding PSTN capacity where needed for specific locations, verifying that QoS configurations are adequate, and increasing the bandwidth on the SIP trunk to the WebEx cloud.
The Cisco IT help desk, called the Global Technical Response Center, provides first-level support to users, and the WebEx team provides second-level support. Outages and other problems that affect the WebEx conferencing services are handled jointly by the WebEx and Cisco IT voice and network teams.
To reduce support calls, Cisco IT asks users to watch an online training video about hosting a WebEx session. Additional user information about the WebEx Conferencing service is posted on a wiki portal (part of our internal Integrated Workforce Experience powered by WebEx Social) for user self-support.
By routing audio calls based on the WebEx Cloud Connected Audio design, Cisco has gained significant financial and operational benefits, and improved the overall user experience.
Cost savings and cost avoidance. Cisco IT has significantly reduced the PSTN charges associated with WebEx by carrying all on-net calls over the Cisco network, and carrying as many of the off-net calls using low-cost TEHO routing. Although Cisco IT continues to maintain toll-free and local phone numbers for dial-in access, employees and other users connected to the WAN are actively urged to use the WebEx callback feature over the WAN to reduce PSTN costs. As of May 2012, only 14 percent of the audio minutes for Cisco's WebEx sessions incurred toll charges, because most of the audio calls were routed over the Cisco network.If Cisco had paid for this voice traffic over the PSTN (assuming an average global PSTN rate of 2.7 cents per minute), the costs could have reached US$32.4 million per year.1
For outbound calls to non-Cisco endpoints in India and the Middle East (where TEHO is not allowed), Cisco is able to reduce costs by routing the calls via a service provider in Europe that offers low per-minute long-distance charges.
Reliable, scalable network design to support very high conference usage. Based on usage data in early 2012, the Cisco network carries more than 1.2 billion minutes of WebEx audio calls each year. Creating a separate Cisco UCM SME infrastructure helps to support the network reliability and scalability necessary to serve this demand.
Infrastructure reuse. Cisco IT gained cost and time savings in the migration to WebEx by reusing the global WAN network infrastructure already in place and used for our unified communications voice and video services. Cisco IT has also been able to reduce the number of costly PSTN connections dedicated to audio conferences in each corporate location.
Fewer support cases. With the availability of online support resources, Cisco IT has received fewer user support cases for the WebEx service than for the previous audio conferencing service.
Improved User Experience. Cisco employees are able to take advantage of a collaborative meeting tool that enables them to share voice, video, and data with each other and with partners and customers, without having to worry about the added cost of voice calls. By providing a high-quality meeting and collaboration tool for all Cisco global employees, regardless of location, unconnected to any conference room location, Cisco IT has helped our employees become a more global and more mobile workforce. Our employees value this extra flexibility, and various studies of our employees who work from home or in flexible work environments show improved productivity and improved user satisfaction. These and other studies show that this mobility increases our ability to attract and retain employees, especially Generation Y employees who have become used to greater mobility and flexibility in their lives. And Cisco values the increased business resilience that comes from a workforce that is not tied to physical locations. Employees are capable of working from home or any number of Internet-connected locations when our office buildings close due to snow, storms, or disasters; and can be highly connected and productive during emergencies or pandemics.
Cisco customers who plan a migration from an existing audio conferencing solution to Cisco WebEx can gain insights from the lessons learned by Cisco IT during its own transition.
Expect high call volumes. WebEx conference call volumes have increased steadily since the service began. As of May 2012, the internal Cisco WebEx service supported an average of more than 700,000 meetings per month. Additionally, of the users who have WebEx accounts, 85 percent have actually hosted meetings, indicating the strong popularity of the service. Many employees are involved in multiple WebEx meetings each day, either as hosts or participants; at Cisco, we are averaging over 1.6 meetings per person per day. Indeed, WebEx conferences may replace some direct calls and in-person meetings for convenience and because people like knowing they will be able to reach each other at a confirmed time, without having to book a conference room.
Monitor QoS prioritization. At the peak levels of 10,000 simultaneous calls, WebEx represents 800 Mb of traffic on the Cisco network, which reaches the limits of QoS definitions on certain WAN links. The 3845 hardware has reached its limited capacity per ISR. Cisco IT plans to resolve this issue by migrating the network routers to Cisco ASR 1000 Series Aggregation Services Routers that can handle peaks of 10,000 simultaneous streams per ASR 1004 RP2
Plan for user transition. Cisco already had a robust audio conferencing service in place, and needed to support our user population as we migrate from it to WebEx. Cisco IT put in place an interactive voice response (IVR) system to simplify user access to both conferencing services until the transition is complete. With the IVR, the same set of toll-free and local phone numbers were used for all conference access, allowing users to choose either service when the IVR answers the inbound call. This solution was seen as less expensive and less confusing than establishing a separate set of access numbers for the new conferencing service. To encourage WAN-connected employees to avoid expensive PSTN toll-free numbers and to use the callback feature across the WAN, Cisco IT set the toll-free access lines to block calls that originate in area codes where a large number of WAN-connected employees are located.
Cisco IT will consider expanding its WebEx support to transport video from employees' Cisco video IP phones as part of the audio stream (e.g., Cisco Unified 9971 phones, and desktop video endpoints). Currently, employees can only use video on their PCs in a WebEx conference; that video is carried in the data portion of the user's session. This video support, carrying the video portion across the WAN (and across the CCA ports), will increase WAN bandwidth usage on the Cisco network and may require an increase in network capacity in selected locations. (Today WebEx CCA service is audio-only.)
Cisco IT also plans to transition from the Cisco 3845 Integrated Services Routers currently used in the audio infrastructure to Cisco ASR 1000 Series routers to gain the scalability to serve more WebEx calls and reduce the number of routers.
To read additional Cisco IT case studies on a variety of business solutions, visit Cisco on Cisco: Inside Cisco IT
This publication describes how Cisco has benefited from the deployment of its own products. Many factors may have contributed to the results and benefits described; Cisco does not guarantee comparable results elsewhere.
CISCO PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Some jurisdictions do not allow disclaimer of express or implied warranties, therefore this disclaimer may not apply to you.
1Note that many assumptions are behind this number. Before Cisco IT had reached a $42M per year spend, Cisco IT might have:
1Chosen to limit collaboration traffic to reduce this cost.
1Investigated other carrier options that may have been lower than the average PSTN cost.
1Required WebEx users to use the integrated VoIP "connect using Internet" feature for international voice (with potential quality issues from remote locations).
1Carrying the audio portion over the Cisco WAN via SIP trunk provided better service, and was significantly easier and less expensive than the alternatives considered.