Cisco on Cisco

The Cisco IT Amsterdam data center hosts all Cisco IT production applications and some development and staging environments for Europe, the Middle East, and Africa (EMEA). Many critical applications such as order entry and accounts receivable for Cisco products and services sold outside the United States are hosted in the data center because Cisco SystemsŪ may be liable for U.S. taxes on these orders if quoting, pricing, and invoicing are processed on servers within the United States.

In total, the data center has more than 120 Windows 2000 systems and more than 100 UNIX systems, including 24 development systems, more than 100 production application systems, and more than 30 production database instances. As a strategy, Cisco IT uses UNIX systems to host the applications with the highest demand for throughput and input/output, and the Windows 2000 platforms support the midlevel to low-level applications and off-the-shelf Windows-based products.

The different service levels supported in the data center include four top-priority (Priority 1) systems and more than 100 high-priority (Priority 2-3) systems. The main applications supported are enterprise resource planning (ERP), multiple Oracle databases, Web services, and Exchange 2000, which will soon be migrating to Exchange 2003 for all 6000 or more EMEA- based Cisco employees.

The team supporting this environment, the EMEA hosting team, consists of 14 staff members, partly outsourced through a vendor providing managed services. Additionally, the team has one full-time consultant (for Network Appliance storage filers) and some part-time consultants who handle EMC and HP storage equipment.

Currently, the Amsterdam data center storage environment has a raw data capacity of 60 terabytes (TB). Today, most of this is part of a storage area network (SAN), but a few years ago it was almost all direct attached storage (DAS).

During 2001 and 2002, the hosting team faced some challenges because of the dramatic demand in data growth and the limitations of the early DAS model. DAS was rarely used efficiently, and the cost of storage on servers that were never used continued to increase with each server purchase. Another problem is that DAS architecture is inflexible. In one example, Steve Moon, storage manager and system administrator for the Amsterdam hosting team, says, "Because of distance restrictions with DAS, there used to be an implementation rule that no server would get installed unless it contained its own storage of more than 60 gigabytes. So systems under 60 gigabytes could not take advantage of feature-rich storage frame systems. This obviously greatly reduced our flexibility to react to changes and kept us from using shared storage. Moreover, tools for storage inventory, capacity planning, and storage provisioning in such a large and fragmented DAS environment were lacking."

Instead, the EMEA hosting team had to do all these tasks manually, which consumed valuable time from the core staff resources. Jeroen Sourbron, IT project manager, says, "To prepare for the coming years, you need to know the current status and how to define and measure it. You need to be able to track your progress and answer questions like 'What is your average utilization rate?' and 'What is the average duration for provisioning and inventory?' These questions had to be answered, but we could not answer them without investing valuable time from the core team resources. Inventory was more or less an ad-hoc spreadsheet exercise, and provisioning could often take one employee working full time."

To overcome this challenge, the hosting team started the migration toward a storage model based on SANs rather than DAS. They set up different SAN islands aligned with the major business applications to replicate the earlier and familiar DAS environment. Then they selected a mix of Brocade and McDATA Fibre Channel switches within the different islands. This environment supported more than 20 TB of data around mid-2002. This migration to SAN islands increased storage use and made provisioning new storage much simpler and faster.

However, physical limitations of the SAN islands model were soon reached as storage demands from the business increased. Floor space and port capacity on the individual Fibre Channel switches were limited and nearly exhausted. The team was in need of a radical change that would overcome these challenges and prepare for the coming years.

In early 2003, the team started planning a migration from individual SAN islands based on smaller Fiber Channel switches to a single shared SAN service based on the larger Cisco MDS 9509 Multilayer Director Switch. The Cisco MDS switch, they reasoned, has the industry-highest port density, supports virtual SANs (VSANs) on the same switch, and offers enhanced features that greatly reduce complexity and increase operational efficiency.

The migration toward one physical SAN fabric with logical separation was planned, staged, and implemented by the core resources within the hosting team. The hosting team was able to free the resources needed for this migration by out-tasking context-related work that consisted of routine operational and repetitive tasks. Bob Stemmerick, IT manager of the hosting team, says, "We out-tasked routine operational tasks such as managed services to an outside services group, governed by strict service reporting, service level agreements, and a flexible billing model. This has allowed Cisco hosting team members to refocus their priorities on core Cisco and client-facing tasks and projects, which obviously included the Cisco MDS implementation. Besides, there is no value in a senior systems administrator racking his own servers or performing tape management duties over deploying our SAN fabric using Cisco MDS switches. That's what we want our staff to be working on: innovation, high-value chain, low-touch activities. Finally, the benefit of out-tasking versus outsourcing is that the work gets done, but we, Cisco IT, are responsible and accountable and retain full control of the tasks, at a lower cost."

Moon mapped out the strategy for the Cisco MDS architecture (Figure 1). A core switch-edge switch design (using a Cisco MDS 9509 Multilayer Director Switch and a Cisco MDS 9216 Multilayer Fabric Switch) was chosen based on total cost of ownership (TCO) and flexibility to adapt to changes. The Cisco Advanced Services team was also involved early to provide its input and approval on the milestones throughout planning and design. Cisco IT reaped the benefits of the Advanced Services team's expertise and as a result has increased the confidence level in the deployment. The integration of an IP SAN environment into the existing Fibre Channel environment is also expected to benefit from the Cisco Advanced Services team. Sourbron notes: "The experience and assistance from the Advanced Services resources in Amsterdam has decreased our deployment time from weeks to a couple of days."

The SAN contains two distinct paths. The two core switches/fabrics are not interconnected in a meshed configuration. However, Cisco IT plans to connect all SANs in each data center into a single SAN to create a single storage service shared by all business groups. And plans are being considered to connect SANs between data centers as well. Seven VSANs reside on each core, and their VSAN IDs are selected from a global Cisco block of reserved IDs. This will help ensure that future interconnection among these cores will not result in a VSAN zone merge, which could occur if two VSANs are given the same VSAN ID and then connected.

Tape libraries are connected to the edge Cisco MDS 9216 switches and on a separate VSAN to other disk-based storage VSANs. A port channel is used to aggregate backup traffic and to lower the TCO for the SAN ports. All storage is attached to the core Cisco MDS 9509 switches and is connected to 16-port Fibre Channel line cards (which support 16 Fiber Channel links of 2 Mbps each) for higher performance.

All Inter-Switch Links (ISLs) are deployed on 16-port line cards for higher performance and are paired to reduce the likelihood of topology change if a link drops. All hosts are connected to 32-port Fibre Channel line cards, except for the NetBackup media servers. All Priority 1 hosts are connected to pairs of S-32 line cards on each core switch, which means that each host is multipathed to four line cards.

After planning, product availability, and physical installation of the equipment during the course of 2003, the migration of existing systems onto the Cisco MDS SAN fabric started in October 2003. In total, 10 Cisco MDS switches were installed. Currently, 225 Fiber Channel ports are in use with 20 ISLs or trunk ports with a capacity to grow to up to 560 available Fibre Channel ports and 16 Gigabit Ethernet ports.

The migration was completed in four months. This was a major achievement, especially considering that the team couldn't make changes to the data center during freeze periods, which accounted for 30 percent of the overall time. New application servers and storage systems that were needed for expansion were already connected to the core Cisco MDS switches at that point.

The migration was unusually smooth. During the migration period, only one trouble report was received, which is rare in such a critical environment. Moon says, "There were no unplanned outages, although some remediation had to be done because of existing multipathing configuration. Overall, only 10 percent of the systems migrated required planned downtime, within an average duration of 30 minutes. With the Exchange migration, there was, for example, no planned downtime, no reboot necessary. Over the total migration period, 90 percent of the time has been spent before the migration, so basically in the planning stage."

Marcus Chambers, storage sales operations director, says, "I am very excited that we have had a very straightforward migration of all our applications within the data center across to 100 percent MDS environment. We not only managed to do this with zero unplanned downtime but also are already seeing cost savings from being able to consolidate all disparate SAN islands into one virtual SAN. I am very confident moving forward we will see increased savings by the management tools and quality of service through utilizing the intelligence and management software within our intelligent Cisco MDS switch."

With the migration barely finished, benefits of reduced risk and increased productivity and cost savings were immediately apparent.

• The overall risks have been reduced greatly through the use of VSANs, which enhance the overall protection level of the environment. Moon adds, "Not only is the whole security environment easier to manage; you also avoid the risk of corruption. Any damage that could occur within one VSAN is not communicated across the SAN to other VSANs." The profitability of the data center increases by offering investment protection to maximize capital expenditures. The high port density of the Cisco MDS 9509 switch and the possibility of aggregated uplinks allow for expansion of storage subsystems.
• Productivity has increased through more efficient execution of the daily storage management tasks. The Amsterdam data center has nearly tripled in capacity over the past 18 months, from 21 TB raw storage to 60 TB, and the staffing has remained the same, namely one full-time Cisco IT employee. Another advantage is that building the SAN using only the Cisco MDS platform allows the administrator a single management view for all storage subsystems, including the backup library and all connected application hosts.
• One of the benefits of this single management platform is its advanced troubleshooting. As Moon says, "Of the few issues we encounter, most arise during setup time of a new server or application. For systems already running, the Cisco MDS switch offers advanced troubleshooting features that allow me very quickly to look at the frame level. Low-level analysis of the frames is possible. I would need advanced licensing from Brocade or McDATA to do this same kind of low-level analysis, but this is by default enabled on the MDS." Also, the inventory of systems can be done at the touch of a button, which greatly reduces the manual effort that used to be required and makes application growth control more efficient.
• The speed of reaction to unexpected business changes has increased. Moon adds, "When sudden requests come up, I don't have to worry about putting systems in a specific spot to be within reach of the necessary storage, and the physical planning in terms of floor space is easier. Now it is easier to configure an individual storage disk logical unit number (LUN) on the consolidated environment with the host systems localized."

Three major evolutions are possible because of this migration to one multiprotocol-supporting consolidated environment. Cisco IT in EMEA is investigating each of these options for future improvements.

Integration of IP SAN using Small Computer System Interface over IP (iSCSI) - For the mid- to low-level applications, mostly based on Windows 2000, the use of an 8-port IP storage services module would allow Cisco IT to integrate an iSCSI-based environment with the Cisco Catalyst® 6500 Series Switch. This integration would greatly reduce the TCO for this environment. An EMEA-specific study has been conducted stating that enabling a host with Fibre Channel is more expensive than enabling the same host with iSCSI. Using iSCSI saves from US$2000 to $3200 per host. The break-even point to allow for payback of the investment in IP SAN is reached when 38 new hosts are connected to the IP SAN environment.

The test environment for IP SAN has been set up to establish a benchmark blueprint design that can be used globally to provide high availability and security.

• Implementation of EMC Control Center Software - The implementation of EMC Control Center software as storage resource management (SRM) software would have inherent benefits. By implementing SRM on top of the consolidated environment, storage provisioning would be even more efficient because everything could be managed through a single view.
• Interconnecting data centers to create a virtual data center - The architecture allows for flexibility with the extension and connection to other data centers within Cisco to build them together into a virtual data center. This "virtual" data center makes manageability of business continuity more efficient. This interconnection and preparation for future expansion is made possible by connecting SAN gateway switches (for example, Cisco MDS 9506 multilayer director switches using the same services module within the Cisco MDS for Fibre Channel Interface Protocol as the remote connectivity layer).