Cisco on Cisco
Network Systems Case Study: How Cisco IT Optimizes Network to Support Operational Excellence
The Network Optimization Service saves time every week for Cisco IT and helps to balance innovation with operational excellence.
The Cisco Network Optimization Service helps customers optimize their network infrastructure to support business objectives such as improving collaboration and accelerating time to market.
Director of IS, Cisco
Customers that subscribe to the service are assigned a team of professionals who advise on preparing the core routing and switching network and advanced technologies to support new business processes, applications, and technologies. The service can include assessments, strategic reviews, and ongoing network support.
Anytime, anywhere access is the foundation for productivity and collaboration at Cisco. To support ubiquitous access, Cisco IT has built a borderless network, whose main tenets are a highly available core network with quality of service (QoS), mobility, and security. With these characteristics, the network has become a platform for collaboration tools such as Cisco TelePresence, Cisco WebEx, and Cisco Unified MeetingPlace.
Previously, Cisco IT had to devote significant resources to help ensure that the network delivered the needed availability and performance. This effort required the following activities:
- Certifying new switch and router hardware and software before introducing them into production
- Evaluating different design approaches to select the one best suited to the unique business needs and network environment of Cisco
- Periodically conducting inventory of the company's nearly 12,000 routers and switches to plan changes to hardware, software, and modules
As the Cisco network grew, these activities took more and more time, diverting IT resources from more strategic projects. Sometimes the IT department simply did not have the resources to fully complete certification of new hardware or software before a deployment deadline. This situation increased the risk of downtime from an unknown issue.
"There was a time when Cisco IT had an imbalance between operational excellence and innovation, "says Craig Williams, director, IS, Cisco. "We wanted a trusted advisor to provide operational oversight so that we could focus on business transformation"
In late 2007, Cisco IT engaged Cisco Services to deliver the Network Optimization Service. Cisco Services assigned Cisco IT a team of professionals who would continually optimize the Cisco network to provide the availability and performance needed to meet business objectives. Cisco IT currently takes advantage of the following Network Optimization Service activities.
Cisco IT asks the Network Optimization Service team to evaluate design options and make a recommendation. For example, Cisco IT engaged the team to provide input on two different Cisco firewall solutions. "Our Network Optimization Service team modeled the two designs in their labs and tested performance in different failover scenarios," says Wilson Ng, IT engineer, Cisco. "From experience with other customers, they were able to alert us to potential issues in our environment relating to routing protocols and timers." Based on guidance from Cisco Services, Cisco IT decided to use the Cisco Adaptive Security Appliance.
When issues occur, Cisco IT escalates them to the Cisco High-Touch Technical Support (HTTS) service. Cisco IT also refers Priority 1 and Priority 2 implementation and operational issues to the Network Optimization Service team for faster resolution. "From my perspective, the most important benefit of NOS is giving us an escalation path that we use in conjunction with HTTS," says Henry Ku, IT architect, Cisco. "For high-priority issues, we involve Cisco Advanced Services in the discussions with HTTS. After the immediate problem is solved, we can work with our Network Optimization Service engineers to discover the root causes and help ensure it won't happen again."
New code for the Cisco IOS Software used within Cisco is released up to four times a year. "Our Network Optimization Service team provides ongoing support for tracking the correct Cisco IOS versions," says John Moe, IT architect, Cisco. "In the last 12 months we engaged Cisco Services to test five trains and recommend one. Their knowledge of other companies' experience with different versions of the Cisco IOS Software saves us a lot of time in selecting the version that's best for our environment."
When one new version of Cisco IOS Software was released, the Network Optimization Service team knew that the multicast behavior in one version of the Cisco IOS we were considering would not work in our environment because of an unusual requirement. "Without the service, we would have spent 40 to 60 hours trying to find the issue, and then started the testing process all over again with another version," says Moe. "By proactively notifying us of potential issues, the Network Optimization Service reduces false starts. And in this case it freed up the equivalent of full-time employee for one to one-and-one-half weeks."
The Cisco IP network is growing rapidly to support more users, more applications, and greater use of collaboration. The Network Optimization Service team is helping Cisco IT scale the implementation model. For example, if the Cisco IT team plans to upgrade code on 1000 devices, they begin with a pilot of 30 devices in global locations. If no problems emerge after three weeks, they proceed with general deployment. Previously, Cisco IT did not have documented acceptance criteria or automated processes for the pilot phase. "To collect pilot data, we had to manually log in to view hardware configuration or CPU utilization," says Ku. "Now Cisco Services is helping us automate the process and become more proactive."
Cisco IT uses the internally developed Enterprise Management (EMAN) software tool to conduct monthly inventory on the nearly 12,000 devices in the Cisco global network. But although EMAN shows hardware and software on the network, it does not report the modules used in each device. "Our team spent 250 man-hours using scripts to retrieve device module information over the network and parse the data into the desired output format," says Moe. "We realized that this process would not scale to meet our ongoing requirements for timely inventory data."
Now the Network Optimization Service team uses Cisco Network Collector, an automated network discovery and inventory tool, to provide a detailed network inventory that includes modules. The tool performs daily checks to confirm it can reach each device, and periodically checks inventory and configuration. "The technology audit required no incremental investment or training, and freed up hundreds of man hours," Moe says.
Cisco IT once asked Cisco Services to diagnose and fix an issue related to content switching. "Finding and certifying the fix internally would have required four to five weeks, because our resources wouldn't be able to work on it full-time," Ng says. "Cisco Advanced Services completed the job in just two weeks. We saved time and potentially avoided more downtime by resolving the issue more quickly."
In the Cisco data center, Cisco Services professionals test new Cisco hardware and software before Cisco IT deploys it for production, as part of the Data Center Networking (DCN) Optimization Service. "Our data center team is very busy making the transition to Data Center 3.0 technologies, because we don't have the resources to perform tests ourselves," says Ng. "Having our Cisco Services team test a new version of the NX-OS saves a lot of time."
"We have integrated the Network Optimization Service team into our major IT programs, tapped into their knowledge, and adopted the best practices they've developed from their other customers' experience," says Williams. "Our operational metrics are the highest they have ever been, and we're enabling more innovative project endeavors than ever before. The Network Optimization Service has helped us balance operational excellence and innovation."
Cisco's return on investment from the Network Optimization Service includes time savings for network optimization, troubleshooting activities, and downtime avoidance. "If we didn't hire the resources to perform the certifications, it's very likely we would have experienced downtime, which has a high cost in terms of productivity and customer service," says Moe.
As one example of time and productivity savings, Cisco IT engineers previously spent five minutes auditing each device during pilots of new switches and routers, every day for three weeks For a pilot with 30 devices, this added up to 52.5 hours. "If the resources weren't available, we might not find out about a code issue until a user reported a problem," says Ku. "Our philosophy is to be proactive rather than reactive." Now the Network Optimization Service team uses an automated tool to perform daily network audits during pilots, supplemented with alerts through Cisco Remote Management Services. Cisco IT took advantage of this service when it introduced Cisco Catalyst 3750E Switches, Cisco Catalyst 6500 Switches with Virtual Switching Service, and Cisco 1000 Series Aggregation Services Routers.
The Cisco Network Optimization Service team serves as trusted advisors, collaborating with Cisco IT in weekly meetings. "Our Network Optimization Service engineers have become trusted advisors to our IT staff," says John Manville, vice president of IT Network and Data Center Services, Cisco. "We count on their tools, best practices, and customer experiences to aid in our IT processes from architecture and design to operations."
The contribution from Cisco Advanced Services is recognized at all levels of Cisco's IT organization. "It's a two-way collaboration," says Ng. "We like to try out new ideas with our team and receive validation on new approaches. In turn, they share their experiences with their other customers." Ku adds, "Having our own team of Cisco Services professionals gives us continuity and a high comfort level."
The Network Optimization Service continually expands to include newly introduced Cisco solutions and technologies. Cisco IT will take advantage of these new offerings as it deploys advanced technologies. Cisco customers benefit from the engagement, because Cisco Services shares Cisco IT's experiences when it delivers the Network Optimization Service to other customers.