CiscoLive Europe is Cisco's annual premier education and training conference for IT, networking, and communications professionals in Europe. CiscoLive Europe 2010 was held at Barcelona from January 25 to January 28, 2010, with a total attendance of 3,148.
CiscoLive, an annual one-week event, posed unique challenges for network operators. Unlike a corporate network, which takes years to build, network buildup and teardown for this event took just a few days, even though the scale of the network was comparable to that of a midsize company. The network needed to support all the product demonstrations from the show floor, wireless access for all attendees, and streaming of all speaking sessions. It also provided various advanced services, including Cisco
™, video surveillance, and unified communications. Since the network's availability, reliability, and performance were so crucial to the success of the event, the operations staff needed to monitor network availability efficiently and to gain insight into network performance, such as the health of the network, the types of traffic traversing through the network, and service quality.
Cisco Unified Communications Management Suite (UCMS) was used to manage unified communications deployment at CiscoLive 2010. Cisco Unified Operations Manager (UOM) was used to monitor the operational status of all components of the Cisco Unified Communications system, including Cisco Unified Communications Managers, voice gateways, and IP Phones, as well as the underlying Cisco routers and switches to proactively identify and help resolve issues before users noticed them. Service quality monitoring was achieved by using Cisco Network Analysis Module (NAM) with Cisco Unified Service Monitor (USM). Cisco NAM provided visibility into the quality of voice calls based on voice signaling protocols as well as RTP stream monitoring. The metrics, including Mean Opinion Score (MOS), packet loss, jitter, and seconds of severe concealment, were fed into USM. The combination provided real-time monitoring of voice quality, detailed reports, and notification if the quality dropped below a user-defined threshold.
Cisco Unified Service Statistics Manager (USSM) was used to get a longer-term comparison of key metrics such as call volume, overall call quality distribution, IP Service-Level Agreements (SLAs), and the usage level of trunks to allow the event staff to anticipate any performance and capacity issues. USSM was able to report on the overall quality and availability of the network for the entire week.
UCMS solution deployment started one day prior to the show opening. UOM, USM, and USSM were deployed on a single dual-core server on the mangement VLAN of the core network. The Unified Communications components were discovered automatically by UOM as they were added to the core network and subsequently monitored on the topology map for health and performance.
The SVC-NAM-2 service module running NAM Software version 4.1(1) was installed in the core Catalyst
® 6500 Switch and added as a Quality of Service (QoS) data source for USM. The installation and setup was quick and easy because of the one central component.
Usage and Benefits
Health and Performance Monitoring with Cisco Unified Operations Manager
In the Network Operations Center (NOC), where a team of administrators monitored the network, UOM was used as a central monitor console for all Unified Communications components.
Monitoring Highlights in NOC
• The UOM Alerts and Events Dashboard was extensively used to obtain notifications on Unified Communications network issues. The dashboard provided navigation to more detailed fault information such as service name, time of occurrence, and recommended actions to speed up troubleshooting. At one point in time during the show, the gateway was unregistered due to an intermittent network connectivity issue. An alert from UOM allowed the network administrator to detect the problem quickly and minimize the outage (Figure 1).
Figure 1. UOM Detecting a Gateway Outage
• The UOM Service-Level View (SLV) was used as a real-time monitoring dashboard to obtain an integrated and unified view of the operational status of the entire Unified Communications deployment. Changes in the status of the network entities were reflected in color-coded changes in the corresponding icons in the SLV. Whenever possible, network administration personnel would glance over the SLV to make sure no critical event had been overlooked. In Figure 2 the red circles with exclamation marks denote critical events.
Figure 2. UOM Service Level View
• Diagnostic tests were set up in UOM for proactively detecting availability, performance, and service degradation issues. Synthetic tests, such as the phone registration test and dial tone test, were created to continuously check the Unified Communications Managers and IP Phones. The tests were running every minute of every day during the event to make sure that end-user services were available without any disruption.
• UOM performance graphs were used to get visibility into critical performance metrics of each element, such as resource usage (CPU, memory, media Digital Signal Processor [DSP] resources), active calls, trunk usage, and others. There were two E1 circuits installed on the voice gateway with one E1 circuit actively registered and utilized to route the calls over the Public Switched Telephone Network (PSTN). In Figure 3, the trunk utilization graph from UOM worked flawlessly, indicating no trunk capacity issue throughout the event.
Figure 3. UOM Trunk Utilization Graph
• There were approximately 45 IP Phones set up in the lobby to enable attendees to stay connected. It was critical to make sure that all phones were functional. The IP Phone status display in UOM was used to track the phones that experienced the outages. The detailed information about a phone's switch, such as the switch port, was used to help to locate the malfunctioning phone and troubleshoot the problem very quickly.
Service Quality Monitoring with Cisco Unified Service Monitor
USM was used for real-time voice quality monitoring. The Cisco NAM was configured to monitor the voice VLAN and calculate on a real-time basis the voice quality metrics, such as MOS, jitter, and packet loss, which were collected by USM for detailed voice quality reporting and alerting. Using USM and NAM together provided network administrators with the ability to quickly measure and establish a MOS baseline for the network. This was particularly useful to validate that there were no significant voice quality issues in the network during the event. As seen in Figure 4, about 64 percent of the calls were of excellent quality, 34 percent of good quality.
Figure 4. Call Quality Distribution Report
Specific occurrences of voice quality issues during initial network setup were quickly detected and resolved using USM events generated when MOS thresholds where crossed and reported to UOM (see Figure 5). The network administrator quickly acted upon these alerts by navigating into the USM stream correlation reports for further troubleshooting. The stream correlation reports align metrics from Cisco Network Analysis Module and call detail records from Cisco Unified Communications Manager to help administrators identify network segments that have a lower-quality user experience. This way, they can determine whether voice quality degradation was caused by the network or by the carrier.
Figure 5. Service Quality Alert in UOM
Performance Analysis with Cisco Unified Service Statistics Manager
USSM uses the raw data collected by UOM and USM to consolidate performance statistics from a variety of devices, applications, and systems. USSM provided a performance analysis window, reporting no major performance and capacity issues during the five-day event. USSM allowed administrators to gain insight into a variety of key performance metrics, such as call volume, call quality, Communications Manager resource utilization, gateway utilization, and IP SLAs, on a daily basis. Figure 6 displays the daily resource utilization statistics for Communications Manager.
Figure 6. USSM Daily Resource Utilization Report for Communications Manager
Cisco used its own management products in the Cisco Unified Communications Management Suite to make sure that the CiscoLive Barcelona 2010 event was a success. The management solution provided significant diagnostic time savings and high network reliability by providing real-time monitoring for the Unified Communications network at CiscoLive Barcelona 2010.
Cisco Unified Communications Management Suite provided business benefits:
• Customer Satisfaction: Most of the attendees and vendors pay to be in the show and expect the highest performance network from Cisco. The extensive monitoring of all the Cisco Unified Communications elements by Cisco Unified Communications Management Suite empowered the event organizers to maximize the performance and availability of the Unified Communications network.
• Proactive Monitoring: Cisco Unified Communications Management Suite proactively monitors the Unified Communications components to notify the administrators of problems before they happen.
• Improved Mean Time to Recovery (MTTR): UCMS allowed the administrators to perform targeted troubleshooting, to isolate problems quickly, and to reduce the downtime.