Cisco on Cisco
Routing and Switching Case Study: How Cisco IT Systematically Upgrades the Network to Support Future Technologies
Proactive program continually refreshes core IT infrastructure to accelerate new technology deployments.
In 2001, Cisco began deferring capital expenditures for refreshing its production IP network to free budget resources for time-sensitive projects such as compliance with the Sarbanes-Oxley Act and Oracle 11i applications. The price of that decision became evident in 2005, when Cisco realized it would have to wait to deploy several new technologies until the infrastructure was upgraded. "Our infrastructure could not support a couple of projects that were important to the business, such as Power over Ethernet [PoE] for our wireless LANs and Network Admission Control [NAC] for our desktop network," says Nicole Shaw, project manager for the Cisco IT Fleet Management Program. Before deploying the new technologies, Cisco IT would need to upgrade 1100 desktop access layer devices at Cisco global offices, a project that would take almost two years. "This convinced Cisco that we needed to keep our infrastructure up-to-date so that we could more quickly deploy network technologies that can create operational efficiencies," says Shaw.
Cisco responded by establishing the IT Fleet Management Program. Its goal is to enable the company to be an early adopter of technologies by upgrading the core IT networking infrastructure in alignment with business factors, new opportunities for innovation, and architectural and product roadmaps. The program was implemented in four steps.
The program team first identified the various triggers for infrastructure upgrades at Cisco:
Operational - A product has been identified for end of life or is aging and lacks the availability of newer products. This trigger is the primary one for refreshing the Cisco network infrastructure.
Innovation - Certain emerging network products or standards have the potential to increase Cisco employee productivity or otherwise provide a competitive advantage.
IT demand - Certain IT technologies that Cisco wants to adopt require new infrastructure capabilities. Newer Cisco wireless access points require IEEE PoE cards in Cisco Catalyst® switches, for example, and server virtualization requires updated supervisor cards in the Cisco Catalyst 6500 Series Switches.
Showcase value - If Cisco introduces a new technology that applies to its own business, the company wants to be an early adopter so that it can share business benefits and lessons learned with customers.
Next, Cisco conducted a thorough inventory of network infrastructure devices. To facilitate the inventory, the company first segmented its global network infrastructure into categories called Places in the Network (PINs), as defined in the Cisco Service-Oriented Network Architecture (SONA). Segmenting the network by PIN is useful because Cisco produces reference designs for each PIN that enable the full capabilities of the Cisco SONA framework. Table 1 shows PINs in the Cisco global network.
|Table 1. Location and Descriptions of Cisco PINs|
|Core||Aggregates multiple network PIN distribution-layer devices at a single location, typically a campus.|
|Data center||Provides corporate application and service resources.|
|Demilitarized zone (DMZ)||Spans and includes the corporate firewalls and the service provider gateways.|
|Lab||Development and testing environment that is not associated with the production network but has access to it.|
|Campus user LAN||Serves desktop users in a location containing multiple network PINs.|
|Extranet||Provides network connections to business partners, not including business-to-business and application service provider environments.|
|Out-of-band||Provides special out-of-band management and administrative access to host and network equipment management interfaces of devices located in other PINS. Traffic to the out-of-band network may travel in-band|
|Remote office||Network locations that connect through the WAN and contain only a user LAN infrastructure.|
|Teleworker||Refers to remote workers who connect to the Cisco network over a variety of access technologies, usually VPN.|
|WAN interconnect||Refers to site-to-site networking, including Cisco global WAN backbone, remote office transport, WAN aggregation, and the campus interconnect to the WAN.|
To conduct the inventory, Cisco used the Cisco Network Analysis Toolkit (NATkit), a Web-based tool that lists Cisco Catalyst devices and their configurations. Shaw notes that Cisco customers also can use Cisco NATkit, which Cisco provides as part of its Advanced Services contract.
Cisco IT Fleet Management Program Manager
After establishing the PINs, the Cisco IT Fleet Management Program team developed a plan to refresh every device in the global network based on business and technology triggers. The devices in the plan included infrastructure devices as well as Cisco Unified Communications products such as IP phones. Most devices had not been replaced since 2000. As part of the plan, Cisco considered how each PIN would be affected by the three main triggers for change: operational concerns, IT demand, and showcase value. The team also considered product end of life, both for chassis and their power supplies and modules. The Cisco Catalyst 6500 chassis is replaced less frequently than its components, for example, which Cisco continually upgrades as technology advances.
The plan was established with input from the network R&D community and industry standards groups as well as the following Cisco individuals and organizations:
Subject matter experts within the Cisco Intelligent Network Services (INS) technologies group - These individuals recommend infrastructure upgrades based on product end of life, and also coordinate with their counterparts in Cisco business units to gain early awareness of planned product enhancements. Using this information, the team built technology roadmaps that determine when each PIN will be refreshed.
Cisco Workplace Resources - The Cisco IT Fleet Management Program aligns its plan with the Cisco real estate strategy, making adjustments as buildings are scheduled to house more or fewer employees.
Cisco Remarketing - If the Cisco equipment replaced during a refresh cycle is deemed unusable, Cisco Remarketing disposes of it in an environmentally responsible and secure manner. If it is usable, Cisco Remarketing refurbishes it and offers it through the remarketing chain to customers seeking used equipment.
Cisco IT Infrastructure Enterprise Asset Management Group - This group is creating a repository of all infrastructure assets, which Cisco IT Fleet Management plans to use to gain real-time visibility into inventory.
The Cisco internal communications group - Consistent communications help to ensure that Cisco network users receive accurate and timely information about the effect of network changes.
Cisco Infrastructure Governance Organization - This group oversees portfolio management for Cisco IT infrastructure, sets priorities for infrastructure projects, and helps create a program budget. The Cisco Finance organization helps the Cisco IT Fleet Management Program team adhere to the budget.
The Cisco IT Fleet Management Program team needed to secure management's commitment to a large, ongoing investment to keep the entire network infrastructure up-to-date. The basic funding equation for infrastructure readiness follows:
Total infrastructure investment ÷ Average lifetime = Average annual investment
"All levels of management agreed that keeping the production network up-to-date made sense—particularly when it was determined that deferred upgrades would cause a seven-quarter delay for two important IT initiatives," says Shaw.
The Cisco IT Fleet Management Program has achieved its major objective: ensuring that the need for infrastructure upgrades does not postpone the company's ability to adopt new technologies with the potential to benefit the company. "We have met or exceeded our aggressive upgrade schedules," says Shaw.
Previously, whenever Cisco IT wanted to deploy a new technology, the staff had to first determine whether the current infrastructure could support the product technology. "The fleet management program helps ensure that the network infrastructure is always up-to-date so that we can deploy new technologies quickly and efficiently," says Anderson. "And in some cases, the ability to deploy certain technologies in 6 months rather than 16 months, because the infrastructure is current, can translate to productivity gains or operational savings." Upgrading the network proactively instead of reactively also enables Cisco to predict capital expenditures.
The Fleet Management team notes that it plays a behind-the-scenes role. "The network infrastructure does not earn us accolades from customers or employees," says Anderson. "However, the infrastructure enables the new network technologies and services that increase our competitiveness by improving operational efficiency or service."
The Fleet Management team measures its success based on deployment in the targeted timeframe. The team also constantly seeks out opportunities to deploy lower-cost products when they will meet the business need as well as higher-cost products. "For example, we deploy high-end Cisco Catalyst switches in small offices that we project to grow," says Shaw. "If the office does not grow as anticipated, during the next refresh cycle we will replace it with an appropriately sized switch."
The team is currently upgrading the PINs listed in Table 2:
|Table 2. PINs Being Upgraded|
|Access layer replacement||Campus user LAN||Replace Cisco Catalyst 6000 Series Switches with new switches populated with two Supervisor 32 cards and seven PoE blades|
|Global data center switch replacement||Data center||Replace data center access switches connecting to data center hosts, servers, router gateways, and server switches. Provide service modules such as load balancing, content switching, and Secure Sockets Layer (SSL) acceleration|
|Global core and distribution switch replacement||Core||Replace all core and distribution switches globally with new switches populated with Supervisor 720 cards|
|IP telephony||Campus user LAN, Remote office||Upgrade to Cisco Unified CallManager 5.0, which is based on the Linux operating system and provides increased security and double-byte character support for Asian languages|
|Routing and switching||DMZ||Implement current platform for Internet edge routers|
The Cisco IT Fleet Management Program incorporates lessons learned from each infrastructure upgrade project. "Whenever we change a product on the network, we document what went well and what needed improvement, and apply lessons learned to the next project," says Anderson. "The goal is that each successive project will be more cost-effective and require fewer resources."
The Cisco IT Fleet Management team shares the following lessons for other companies developing fleet management strategies:
Be sure to conduct a thorough inventory and do not underestimate the time needed. "It is critical to know what you have, and the factors that necessitated replacement," says Shaw.
Realize that deployment resources can be a limiting factor. Upgrading the network infrastructure is a manual process that requires onsite personnel.
When developing the plan, consider future plans as well as today's immediate triggers for change. If the team knows about impending product changes six months instead of one month ahead of time, for example, it can more efficiently plan the necessary upgrades.
Know when you can be flexible. Changes such as new product introductions, new buildings, and building closes can require changes to the schedule. "We have a limited budget, so we need to consider which of several competing projects will deliver the greatest return on investment," says Anderson. "By making sure that we fully understand the business ramifications of deferring an upgrade, we can recommend the PINs whose refresh requirements we can defer for a quarter or two, or undertake at a slower pace."
Coordinate with other business groups within the company. The team attributes its ability to stay ahead of business needs to close alignment with the company's business strategy, real estate strategy, and technology strategy.
Consult standards bodies such as the IEEE and organizations such as IETF for early awareness of technology directions.
"We constantly learn more about how to make our deployments more efficient," says Shaw. "One of the questions we are considering now is: Should we always align a project based on the device to be deployed, or should we also consider the physical location? For example, if we need to visit a field sales office to upgrade a targeted device, it might be more efficient use of resources to upgrade other devices in the office as well—even if the office is not scheduled for a refresh for a few more months."
The Cisco IT Fleet Management Program will manage the new infrastructure projects listed in Table 3 in fiscal year 2007:
|Table 3. New Infrastructure Projects|
|Data center||Campus data center access switches (large)
Campus data center distribution gateway
Campus data center service switches
|DMZ||DMZ Internet access gateway
Internet edge router
|Remote office||Cisco Unified CallManager
Cisco Unified IP Phones
Cisco Unity Unified Messaging
Cisco wireless access points
Cisco wireless LAN controllers
|Campus user LAN||User LAN access switches (large)
User LAN access switches (small)
|Core||Campus core and distribution switches|
|Teleworker||Remote access VPN management gateway
Remote access VPN concentrator
|Data center||Campus data center access switches (large)
Campus data center distribution gateway
Campus data center service switches
"It is critical for our competitiveness that we be able to roll out technologies as quickly as possible, to showcase them for our customers," says Anderson. "The Cisco IT Fleet Management Program facilitates rapid rollout by keeping the network infrastructure updated."