Cisco on Cisco
MeetingPlace and Webex Case Study: How Cisco Integrated WebEx and MeetingPlace
Cisco IT built a scalable infrastructure to support 1.4 billion conferencing minutes annually.
(PDF - 312 KB)
Cisco IT Case Study / Collaboration / Cisco WebEx Meeting Center and Cisco Unified MeetingPlace: Cisco has used two different desktop collaboration tools. Cisco WebExâ„¢, delivered using the software as a service (SaaS) model, makes it easy to collaborate with people outside the firewall, including customers and partners, without prescheduling. Cisco Unified MeetingPlace®, hosted on Cisco's premises, avoids carrier toll charges and takes advantage of the quality of service (QoS) in the Cisco network to deliver high-quality audio and video. This case study explains the business value of both tools at Cisco and explains how Cisco IT designed its infrastructure to support 100,000 users and 250 million conferencing minutes each month. Cisco customers can draw on Cisco IT's real-world experience in this area to plan their own architectures for rich-media conferencing.
Cisco regards network-based collaboration as a powerful strategy for productivity, innovation, sales, and cost savings. The company began using Cisco® Unified MeetingPlace in 2004 and added Cisco WebEx Meeting Center in mid-2008. From December 2007 to December 2008, Cisco employees conducted three million remote meetings with 1.4 billion minutes.
“For many Cisco employees, the work day begins and ends with a meeting,” says Nur Quadir, IT program manager, Cisco. The main benefits of desktop collaboration for Cisco include:
- Avoiding the time, cost, and environmental impact of driving or flying to meetings
- Building teams with experts worldwide
- Accelerating decision-making by making it easier for employees to meet and share information
- Being able to engage the most qualified Cisco expert in any global location to meet with a customer
Product Sales Specialist, Cisco
See the"Results" section for more detailed benefits.
Cisco IT needed to help ensure that the infrastructure could support continued growth (Figure 1). "Traffic demands grew by 60 percent in 2009," says Mwiza Munyandamutsa, IT service manager, Cisco. In the last quarter of 2009, Cisco employees participated in 180 million minutes of voice conferencing and 220 minutes of web conferencing.
Factors contributing to growth of conferencing at Cisco include:
- Travel-reduction policies. Cisco requests that employees use remote meeting solutions whenever possible, to reduce costs and minimize the company's carbon footprint. Many people who previously were unable to work together because of distance now meet routinely."Virtual meetings have become the norm at Cisco," says Bailey Szeto, director of IT strategy and architecture, Cisco.
- Growing comfort with video in online meetings. Increasingly, Cisco employees who host internal meetings begin by asking participants to turn on their video cameras. Depending on the nature of the meeting, the host might switch over to a presentation after introductions, but many participants keep the video panel open so that they can see other team members during the session. "Collaboration succeeds best when people feel like they know each other, and video contributes to that feeling," says Szeto.
- Collaborating on documents during in-person meetings. Look inside a Cisco conference room, and you are likely to see employees typing on their laptops. "People use Cisco WebEx during face-to-face meetings so that they can share documents without a projector and everyone can collaborate in real time," says Munyandamutsa.
- Augmenting Cisco TelePresence meetings. People who are not near a Cisco TelePresence™ conference room use Cisco WebEx and Cisco Unified MeetingPlace to participate, although without the same video quality. Even Cisco employees who are meeting with Cisco TelePresence sometimes use Cisco WebEx on their laptops so that they can collaborate on documents during the session.
Working from home. Approximately 70 percent of Cisco employees telecommute at least one day a week, including Munyandamutsa. "The combination of conferencing and instant messaging lets me fully participate in meetings," she says.
Figure 1. Cisco Employees Use Cisco Unified MeetingPlace and Cisco WebEx
Cisco IT needed to build a conferencing infrastructure that would:
- Support continued growth of conferencing
- Minimize infrastructure and operational costs
- Deliver an outstanding user experience
Different groups at Cisco tended to use either Cisco WebEx or Cisco Unified MeetingPlace predominantly, depending on their business need:
- Cisco Unified MeetingPlace, an on-premise solution. Employees often used Cisco Unified MeetingPlace when participants would be connecting with different types of standards-based video endpoints, including desktop clients and room systems. "Cisco has invested in an end-to-end Cisco network, with the QoS and traffic engineering to deliver a fantastic audio and video experience," Szeto says. Employees frequently used Cisco Unified MeetingPlace for internal calls because Cisco incurred no toll charges for the audio portion of calls, which traveled over the Cisco WAN. With Cisco Unified MeetingPlace, costs are fixed and predictable, limited to the internal costs of supporting the servers and underlying IP network.
- Cisco WebEx, a SaaS solution.Employees tend to use Cisco WebEx for web conferences, and when meeting with people outside of Cisco, such as customers and partners. Cisco WebEx supports webcam-quality video. People can join Cisco WebEx sessions with standards-based video endpoints when WebEx is integrated with Cisco Unified MeetingPlace or Cisco Unified Videoconferencing.
When Cisco used both tools, conference participants dialed the same number whether they were joining a Cisco Unified MeetingPlace or Cisco WebEx conference. Then Cisco Unified IP Interactive Voice Response (IP IVR) prompted them to press 1 for MeetingPlace or 3 for WebEx. (See the "Cisco WebEx Architecture" section for details on how Cisco IT deployed this option.)
Figure 2. Cisco Unified MeetingPlace Conferencing Infrastructure
Cisco IT deployed 14 Cisco Unified MeetingPlace clusters in four locations: San Jose, California; Research Triangle Park, North Carolina; Amsterdam; and Hong Kong (Figure 2). Cisco IT used a 1:1 architecture to provide failover and redundancy. The results of this deployment are described in other Cisco on Cisco case studies: “Global Web Conferencing Deployment” and “Enterprise Web Conferencing.” Meetings were hosted on the server nearest the person who scheduled the meeting. Cisco's Call Center Application Team wrote code to set up meeting IDs with leading digits that identify the hosting server. This setup enabled someone in San Jose, for example, to invite participants from global locations. Attendees always dialed their regional phone number for access, and their call was routed over the Cisco WAN to the appropriate server.
Cisco originally used the same Cisco Unified Communications Manager cluster for IP telephony traffic and conferencing traffic for the San Jose and Research Triangle Park campuses. Cisco already had a 19-node supercluster in San Jose, the maximum size supported. When planning to introduce Cisco Unified MeetingPlace, Cisco IT decided to build a dedicated conferencing system, to offload conferencing traffic from the primary campus communications systems.
To estimate voice traffic demands, Cisco IT considers calls per second as well as protocol overhead. Cisco uses H.323 and computer telephony integration (CTI) protocols and terminates Cisco WebEx calls using a Skinny Call Control Protocol (SCCP) Media Termination Point. "The protocols we use impose a high cost on the processing power of Cisco Unified Communications Manager," says Mike Howell, member of IT technical staff, Cisco.
Therefore, to help ensure that conference participants can connect even at the busiest times, Cisco IT built a 19-node Cisco Unified Communications Manager cluster dedicated to conferencing traffic. Currently, half of the conferencing servers are hosted in San Jose and the other half in Richardson, Texas. Each site hosts the following equipment:
- Cisco Unified Communications Manager servers
- Cisco Unified IP IVR servers
- Cisco 3845 Routers acting as public switched telephone network (PSTN) gateways for the following types of traffic:
- Long-distance outbound
- Toll-free inbound
- Local access, inbound and outbound for San Jose and Richardson only
As of mid-2009, Cisco had 65,000 Cisco WebEx Meeting Center users. Until 2009, Cisco WebEx was hosted in one location, Mountain View, California. Because of the critical nature of conferencing at Cisco, in 2009, Cisco IT added another WebEx data center in London, with a mirrored design. "Users don't need to know which server they are connecting to," says Quadir. "Eventually we will have a global Cisco WebEx cloud that consists of multiple sites."
The voice portion of Cisco WebEx sessions travels to the Cisco WebEx data center over Cisco’s WAN instead of the PSTN. How Cisco IT accomplished this is described in another case study, "WebEx Integration with IP Telephony." Sending the voice portion of WebEx conferences over the Cisco voice network eliminated long-distance charges for Cisco participants working in a Cisco corporate office or using an IP phone at their home. When participants respond to the IVR prompt by indicating that they are joining a Cisco WebEx conference, the call is routed from the PSTN or IP phone over the Cisco corporate IP backbone, by way of a gatekeeper trunk. When the call reaches Cisco’s backbone, a Cisco Unified Border Element converts the call to Session Initiation Protocol (SIP) and transfers it to the WebEx audio mixers in Mountain View.
Figure 3.Calls per Second Peak at the Beginning of the Hour and 30 Minutes Later
To make it easy to join conferences, Cisco IT provided a single phone number for either type of conference, Cisco WebEx or Cisco Unified MeetingPlace. Cisco Unified Contact Center Express prompted callers to enter 1 to join Cisco Unified MeetingPlace conferences and 3 for Cisco WebEx conferences. Therefore, the Cisco Unified IP Interactive Voice Response (IVR) servers had the potential to be a point of congestion, especially at the beginning of the hour and 30 minutes later, when most meetings begin (Figure 3).
Cisco Unified IP IVR servers are designed to handle four to six new calls per second, sustained. If the server receives more than eight to ten call requests per second, for even a very short sustained period, calls may be dropped, which is unacceptable to Cisco. "Cisco Unified IP IVR servers can handle a spike of short duration," Howell says."It's when the spike is sustained that trouble occurs, as it is at the beginning of each hour. We don't want to oversubscribe."
At peak times, Cisco receives approximately 40 conferencing calls per second, which requires the capacity of seven IVR servers. To accommodate expected growth in conferencing, Cisco IT decided to build the largest possible cluster, with 16 Cisco IP IVR servers. "We didn't have a way to anticipate the number of users, calls per second, or minutes of usage, and contraction is always easier than expansion," says Howell.
When Cisco IT used the same Cisco Unified Communications Manager supercluster for ordinary IP telephony traffic and conferencing traffic, all traffic traveled over the same inbound trunks. After Cisco IT built a separate conferencing cluster, the company shifted some of the trunks to the new conferencing cluster. Cisco IT gradually added more trunks as the number of conferencing users increased from 35,000 to 100,000.
Initially, Cisco did not need inbound trunks for Cisco WebEx sessions, because all WebEx voice traffic traveled over service provider trunks. Cisco paid per-minute usage charges. Later, Cisco IT modified the service for its own internal use so that the voice portion of Cisco WebEx Meeting Center conferences traveled over Cisco's existing IP telephony network, avoiding toll charges.
When WAN bandwidth is available, outbound calls travel across the WAN to the junction nearest the user, reducing PSTN phone charges. If a customer in Europe joins a conference hosted in San Jose, for example, the outbound call to the customer travels across the Cisco WAN all the way to one of Cisco IT's larger hub office gateways, and only then hops off to the PSTN.
The increasing popularity of Cisco WebEx has increased the ratio of outbound to inbound trunks. At Cisco, 30 to 40 percent of conferencing participants select the option to have the system call them (over an outbound trunk) rather than dialing into the system themselves (over an inbound trunk).
Outbound calling is different if local lines are saturated. For example, in Cisco's Switzerland office, 100 teleworkers join monthly Cisco WebEx conferences. This traffic exceeds the capacity of the office's E1 line. Cisco IT performed a cost analysis that showed it was more cost effective to call the employees from the server in the United States than to purchase another E1 line that would only be used one day a month. Today, the first employees to connect to the Cisco WebEx session are called from the local server in Switzerland. After the bandwidth is used up, the conferencing cluster routes subsequent outbound calls to users through the supercluster in San Jose or Research Triangle Park.
Figure 1.Active Call Report, Showing Activity by Second
Cisco IT continuously monitors trunk capacity for each service: long-distance, toll-free, and local. The team uses custom-developed Perl and Excel scripts to pull information from call detail records (CDRs), such as calls per second, active calls, and busy-hour call attempts."“We use the information to rebalance inbound and outbound trunks to not oversubscribe the Cisco Unified Communications servers, IVR servers, and WAN capacity," says Bill Lowers, IT engineer, Cisco. Cisco IT regularly reviews the reports to see how calls arrive and exit the system. “We check to make sure there are no routing loops, and to identify anomalous behavior that might indicate hacker activity," says Lowers. Figure 4 shows a report on usage by second.
Cisco employees had a similar experience with Cisco Unified MeetingPlace and Cisco WebEx:
- Scheduling meetings. Employees scheduled both types of conferences using Microsoft Outlook, by clicking a button (Cisco WebEx) or selecting a tab (Cisco Unified MeetingPlace).
- Joining meetings. Employees and others dialed the same number regardless of which type of conference they were joining. They were prompted to enter 1 to join a Cisco Unified MeetingPlace conference and 3 for a Cisco WebEx conference. Cisco used to invite people to join meetings by dialing a toll-free 800 number. Now people in the local area code can only connect if they dial the local access number, saving US$415,000 monthly.
- Recording. Both Cisco WebEx and Cisco Unified MeetingPlace Enterprise can record audio, video, presentations, and interactive elements such as polling. "Recordings are very useful for employees who cannot attend meetings because of time zone differences," says Szeto.
In the final quarter of Cisco's 2009 fiscal year (ending July 25, 2009), Cisco employees logged 51.5 million minutes on Cisco WebEx Meeting Center and 152.9 million minutes on Cisco Unified MeetingPlace. Benefits of desktop collaboration at Cisco include business process transformation and cost savings through travel avoidance.
Cisco employees have transformed business processes by collaborating over the network instead of in person, when appropriate.
Cisco sales teams take advantage of Cisco WebEx to replace certain in-person meetings, saving travel time. Cisco account managers in Canada, for example, typically drive 2.5 to 3 hours one way to meet with a client. To regain this time, the salesforce purchased webcams for their customers. “A US$50 investment enabled us to reclaim an entire day for our sales teams," says Chuck Churchill, IT senior manager for infrastructure planning, Cisco.
Similarly, the Cisco Illinois commercial sales team credits Cisco WebEx with helping a small group cover a very large geography, manage thousands of small and medium-sized business accounts, and coach and motivate hundreds of partner account managers. "With Cisco WebEx, we are interacting with customers and partners three or four times as often, reducing travel-related expenses, and improving job satisfaction by communicating with our families with video when we're working late," says Rick Sexton, product sales specialist, Cisco. The team's unified communications specialist, who lives in another state, uses Cisco WebEx Meeting Center to conduct several customer briefings each week, more than he could manage if he had to travel. He also uses Cisco WebEx to conduct multimedia pre-briefings with the account team, partner, and customer to develop a relevant Customer Briefing Center agenda.
In 2009, Cisco used Cisco WebEx to train partners on how government and private sector customers can use economic stimulus funds for network infrastructure improvements. Approximately 800 partners joined the session. “If our trainers had flown across the country to meet in person with customers in different regions, the training would have taken a month or more," Szeto says.
With a combination of Cisco WebEx, Cisco Unified MeetingPlace, and Cisco TelePresence, Cisco's North American sales organization has reduced air travel expenses by more than 50 percent, avoided more than 27 million cubic meters of carbon emissions between October 2006 and January 2009, and reclaimed hundreds of hours of employee time to invest in product innovation and customer support.
In May 2009, Cisco conducted its three-day Senior Leadership Offsite (SLO) completely virtually, saving US$2.8 million in travel and hotel expenses. More than 3100 Cisco leaders attended the SLO using Cisco WebEx Event Center, Cisco TV, Cisco TelePresence, and the SLO virtual environment. Compared to 2008, the cost per person decreased from $2800 to $680, with no difference in the quality or quantity of speakers and training sessions. SLO hosted multiple live breakout sessions and leadership development sessions, including General Session Executive Q&A chats, a unique opportunity for leaders to pose questions directly to the company's top executives.
Cisco's sales teams report that conferencing solutions help them close deals faster and get the right subject-matter expert in front of the customer, regardless of location.
John Chambers, Cisco CEO, announced in the third-quarter fiscal-year-2009 financial results announcement: “We "have permanently cut our travel budget from $750 million run rate per year to $350 million run rate per year." Much of this was enabled by employees' use of Telepresence, but they have also been meeting far more frequently using MeetingPlace and Webex to enable that cost savings.
Cisco employees participated in more than three million meetings between December 2007 and December 2008 using Cisco Unified MeetingPlace and Cisco WebEx. The company has not collected statistics directly connecting the use of these tools with travel avoidance, but conservative estimates suggest significant cost savings. For example, 19 percent of Cisco TelePresence meetings replace travel for Cisco employees. If only 1 percent of Cisco Unified MeetingPlace and Cisco WebEx meetings replaced an in-person meeting requiring a trip that would have cost US$1000, then Cisco saved at least $30 million in 2008 alone from desktop collaboration.
Similarly, if only 10 percent of remote meetings enabled employees to avoid driving to the office or a customer or partner location, and if each roundtrip would have required two hours, then Cisco employees saved 600,000 commute hours in 2008 alone. This savings is equivalent to reclaiming the annual work time of 300 full-time employees.
The Cisco IT team shares the following lessons learned with other organizations seeking to plan an enterprise architecture for Cisco WebEx and Cisco Unified MeetingPlace:
- Provide training. Cisco initially did not plan to provide training on Cisco WebEx, because the interface is so intuitive. However, Cisco employees were accustomed to Cisco Unified MeetingPlace, which has a different interface of sharing control of the screen or designating alternate hosts. More than two-thirds of help-desk calls were from employees confused about the differences, so Cisco now gives new employees a 10-minute training on conferencing tools.
- Give employees guidelines for conducting effective virtual meetings."As an example, we have discovered that meetings are more effective if all participants connect over the network," says Szeto. "It doesn't work as well if some people are physically together and others are not." Also, employees might need time to become accustomed to video. Some Cisco groups are considering requiring video in certain types of meetings because it tends to make participants more attentive.
- Expect a portion of travel savings to shift to telephone bills. "Desktop collaboration can significantly reduce travel time and costs," says Churchill. "When you invite customers and partners to meetings, you'll pay their connection costs. Include this in telephony budgets, and be aware that the corporate return on investment is huge."
- Size the infrastructure for peak volume. Cisco IT doubled the IVR cluster size to support an audio menu that callers use to select Cisco Unified MeetingPlace or Cisco WebEx connections. Sizing is important because thousands of people dial in at the top of the hour and bottom of the hour.
- Provision the correct ratio of inbound and outbound trunks. As Cisco increased its use of Cisco WebEx, Cisco IT increased the ratio of outbound to inbound trunks because external users tend to select the option for Cisco WebEx to call them.
Cisco IT has already migrated to an integrated Cisco Unified MeetingPlace and Cisco WebEx solution. Over time, the company will no longer need its IVR infrastructure to prompt callers to indicate the type of conference they are joining. Cisco is also enhancing its conferencing software to eliminate the need for Media Termination Points, which will simplify the conferencing infrastructure.