CiscoWorks LAN Management Solution and Cisco Network Analysis Module
CiscoLive is Cisco's annual IT and communications conference. The conference features Networkers, the premier education and training program, and other special programs designed to meet the diverse needs of today's IT and communications professional, from increasing technical proficiency to understanding the impact of technology on business. CiscoLive 2009 was held at the Moscone Center in San Francisco from June 27th to July 2nd. Total registration topped 10,300, with customers and partners accounting for more than 6200 of that total. Some 4300 people from 28 countries chose to attend virtually through CiscoLive Virtual.
CiscoLive poses unique challenges for the network operators. Unlike a corporate network that takes years to build, the network buildup and teardown of this event took days, even though the scale of the network is comparable to that of a mid-sized company. There are about 250 network devices deployed in total, including four 7202 core routers uplinked to the ISP, one Cisco Catalyst® 6500 Switch as core switch, one wireless controller, 140 distribution and access switches, and more than 90 wireless access points. Quick deployment and configuration of these devices to optimal settings in a relatively short time period are critical to the network operators, and to the success of the show.
CiscoLive is an annual one-week event. The network availability, reliability, and performance are crucial to the success of the event. The network is required to support all the product demonstrations from the show floor, wireless access for all attendees, and streaming of all speaking sessions, techtorials, breakouts, and certification testing. On top of the network, various advanced services are being run, including Cisco Telepresence™, video surveillance, IPv6, Media Net, and various other technologies. With so many different activities and demands on the network, the operations staff needed to efficiently monitor network availability and gain insight into network performance such as health of the network, types of traffic traversing through the network, response times for the applications, and viruses.
CiscoWorks LAN Management Solution (LMS) and Cisco® Network Analysis Module (NAM) were used to manage the network infrastructure of CiscoLive 2009. CiscoWorks LMS was used to monitor the overall network, manage the devices, and enable Cisco® technologies. Cisco® NAM 2204 Appliance focused on monitoring the performance of the network.
Initial planning and staging of the network were done inside a Cisco lab. Most equipment used in the event had been deployed in various Cisco trade shows. The equipment was tested extensively to help ensure that no hardware defect existed from wear-and-tear, which was quite a labor-intensive effort. LMS proved very efficient in hardware verification. For example, the gigabit interface converter (GBIC) connectors on the Cisco Catalyst Switch are prone to damages and need to be manually tested one by one. Using CiscoView, the graphical device manager inside the LMS solution bundle, the contractors were able to visualize the switches and routers, view defects, and identify which GBICs needed to be replaced. This capability saved contractors time with the setup.
Figure 1. Faulty GBICs Showing as "Unknown GBIC Type" in CiscoView
Two weeks prior to the show, the equipment was moved to the Moscone Center in San Francisco. A small team of three started building the core network to connect the three buildings of the Moscone Center and two hotels. Two LMS servers were deployed on the core network to serve as a redundant backup to each other. The network devices were discovered automatically by LMS as they were added to the core network, and subsequently monitored on the topology map for health and utilization.
Figure 2. Network Topology
Five days prior to the show opening, a 15-person team was called onsite to deploy the edge devices and wireless access points in all the locations. It took two days for the network to stabilize and get into production. A Cisco NAM 2204 appliance was connected to the core Cisco Catalyst 6500 Switch, and a Switched Port Analyzer (SPAN) session was configured on the switch to monitor the performance of the network. The NAM setup took less than 10 minutes.
Network in Production: Operation Manageability and Proactive Automation
Monitoring in the NOC Room
In the Network Operations Center (NOC), where the network was monitored by a team of administrators, the CiscoWorks LMS Portal was projected on the big screen to display the network view (see Figure 3). To show what's happening on the network, an integrated view was created on the LMS Portal, including integration with Cisco Wireless Control System (WCS) and NAM to offer the user one portal to monitor all the activities on the network, including the core network, wireless, and network performance.
Figure 3. Unified Network View on the LMS Portal
• N-hop view was composed to show the core network devices in color-coded fashion, reflecting the health of the devices. The device icons were refreshed with small red icons if there was any fault.
• One iFrame portlet was added to the Network View and pointed to the WCS server to pull out the critical status of wireless network. Another iFrame portlet was pointed to the NAM server to monitor the performance of the network.
• A Cisco® Health and Utilization Monitor (HUM) was used to poll the devices every one minute for availability. The status was reported through the "Device Availability" portlet on the Network View. Cisco HUM also reported top-10 devices with the highest CPU utilization or memory utilization.
• "High Severity Alerts" from the Fault Manager highlighted the critical alarms on the network.
• To get detailed insights of the real-time health of the network, three syslog portlets were added to the Network View, including "Top-N Syslog Sender," "Syslog Messages" (customizable by category), and "Syslog Alerts" (all severity 0 ~2 messages).
The network monitoring goes beyond the NOC. To assist the network administrators on the go, notifications of critical events, such as device down and CPU hogging, were sent to administrators by the LMS server based on Thresholding. Using smart cell phones like iPhone, the users were also able to access LMS portlets to view critical stats without logging into a computer.
Figure 4. Monitoring on the Go from iPhone Network Optimization
To optimize network configurations and proactively reduce the possibility of network problems, LMS checks the Layer 2 discrepancies and best practice deviations of the network based on Cisco's best practice recommendations for configuration, which some users dub as "CCIE in a Box". This feature was leveraged by the administrator to resolve issues in the network before problems happen. In Figures 5 and 6, the LMS server was used not only to report the discrepancies and best practice deviations but also to fix the problems right on the report.
Figure 5. LMS Reports Best Practices Deviation
Figure 6. Fix Discrepancies of the Network from the Report
Troubleshoot Connectivity Issues
The User Tracking function of the LMS made it efficient to troubleshoot end-station issues. One vendor at the show hooked up a rogue Dynamic Host Configuration Protocol (DHCP) server to the network and was handing out rogue IP addresses. User Tracking quickly detected the problem, allowing administrators to quickly locate the port connected to the rogue server and isolate the problem.
Figure 7. Detecting and Locating Rogue Hosts
Making CiscoLive a Green Event
As part of Cisco's Green Initiative, the organizers of CiscoLive 2009 vowed to make the event a green event. To achieve this goal, the wireless access points and IP phones in the public areas were shut down during off hours to conserve energy and at the same time provide better security. This step was taken by deploying Embedded Event Manager (EEM) scripts using LMS to disable Power over Ethernet (PoE) ports at 9 pm every night and bring back the PoE ports at 7am the next morning just before the attendees arrived.
Figure 8. Deploy EEM Scripts for Power Conservation Compliance
Using the Compliance Management of LMS, the administrators were able to set up configuration policies and enforce compliance on the devices. For example, one configuration policy was to direct all the syslog messages to the LMS server as a central syslog collector. Once this policy was created as the baseline template, the operators were able to run the policy check and correct the noncompliance in real time.
Figure 9. Check and Deploy Compliance Policies
Network Performance Monitoring
The Cisco NAM Appliance worked flawlessly, reporting no major congestions and no packet drops throughout the five-day event. During the event, there were a variety of protocols and applications seen by the NAM, such as HTTP, IP Encapsulating Security Protocol (ESP), IP Security IPsec, web, and email. The majority of the traffic was HTTP, IPsec, Control and Provisioning of Wireless Access Points (CAPWAP), HTTPS, and IPsec-Network Address Translation (NAT) as shown in Figure 10. Figure 11 displays the applications traversing the CiscoLive network in more detail.
Figure 10. NAM Overview
Figure 11. Application Group in NAM
Cisco NAM displayed attendees accessing CiscoLive, Facebook, Google, and other web pages. Figure 12 displays the top 10 URLs accessed during the event.
Figure 12. Top 10 URLs Detected by NAM
After the Show
After the six days, the show came to an end. It took two days to tear down the network and pack up for shipping. To improve productivity and eliminate errors, the administrators used the custom inventory reports generated by LMS as a checklist for inventory checking. These reports focus on the information needed for asset management, such as device types and serial numbers, and give the operators accurate information to help prepare for the next big show. Other powerful inventory reports were also provided to the organizers. The End of Sale/End of Life (EoS/EoL) reports track not only the hardware down to the module level but also the Cisco IOS® software versions that have reached either EoS or EoL. The Product Security Incidence Responses Team (PSIRT) tracks the security vulnerabilities of the devices. These reports were taken by the organizer back to the lab to further optimize the network in preparation for the next Cisco event.
Cisco used its own manageability products CiscoWorks LMS and the Cisco Network Analysis Module to help ensure that the 2009 CiscoLive event was a success. By utilizing CiscoWorks LMS and Cisco NAM, the event organizers were able to smoothly set up, monitor, and maintain the conference. The products provided significant time savings and reduced the overall operating expenses by simplifying the initial setup, proactively monitoring and conserving power.
CiscoWorks LMS and Cisco NAM provided business benefits:
• Customer satisfaction: The products helped ensure network connectivity and high performance for the show. Most of the attendees and vendors pay to be in the show and expect nothing less than the highest performance network from Cisco.
• Proactive monitoring: LMS server proactively monitors the network and devices to notify the administrators of problems before they even happen.
• Improved Mean Time to Recovery (MTTR): LMS enabled the administrators to perform targeted troubleshooting, quickly isolate problems, and reduce the downtime
• Network performance: Cisco Network Management solutions helped ensure flawless execution of the network. In several events, Cisco CEO John Chambers visited the show floor to check out the latest technologies. The network segment involved was optimized and watched closely to be certain there was enough bandwidth and no glitches in the network.
• Operational optimization: LMS was used to optimize the network configurations by reducing discrepancies and best practice deviations, upgrade the device to the latest software, and help ensure compliance based on configuration policies.
• Green event:Using the latest Embedded Event Manager (EEM) technology, LMS deployed scripts to the switches to manage the power consumption of Power over Ethernet (PoE) ports. The IP phones and wireless access points in public areas were shut down during off hours to conserve power and help ensure better security. Those devices were automatically brought up early next morning.