Cisco on Cisco
Storage Networking Case Study: How Cisco Migrated to a SAN Environment in Small European Data Center
Cisco Multilayer Director Switch improves manageability, speeds deployment, and reduces costs in smaller data center.
Cisco IT Case Study/Storage Networking/Data Center SAN in Europe: This case study describes Cisco IT’s internal deployment of the Cisco MDS 9509 Multilayer Director Switch within the Cisco network, a leading-edge enterprise environment that is one of the largest and most complex in the world. Cisco customers can draw upon Cisco IT's real-world experience to help support similar enterprise needs.
- The Cisco IT Amsterdam data center hosts all production applications and some development and staging environments for the EMEA region
- Dramatic demand in data growth and limitations of direct-attached storage
- Migration from individual SAN islands to a single, shared SAN service based on the Cisco MDS 9000 Family storage networking switches
- Planning and design assistance from the Cisco Advanced Services team
- Reduced risk for networked storage
- Increased productivity for storage management tasks
- Easier access to stored data
- Ability to add storage easily and quickly
- Lower total cost of ownership (TCO)
- Faster reaction to business changes
- Advanced switch troubleshooting
- Visibility of all storage devices
- Integrate SAN using iSCSI
- Implement EMC Control Center software
- Create a virtual data center through interconnection of multiple physical centers
As of 2003, the Amsterdam data center had more than 120 Windows 2000 systems and more than 100 UNIX systems, including 24 development systems, more than 100 production application systems, and more than 30 production database instances. As a strategy, Cisco IT uses UNIX systems to host the applications with the highest demand for throughput and input/output, and the Windows 2000 platforms support the mid-level to low-level applications and Windows-based products.
The different service levels supported in the data center include four top-priority (Priority 1) systems and more than 100 high-priority (Priority 2–3) systems. The main applications supported are enterprise resource planning (ERP), multiple Oracle databases, Web services, and Exchange 2000, which will soon migrate to Exchange 2003 for all Cisco employees in the EMEA region.
The EMEA hosting team that supports this environment consists of 14 staff members, partly outsourced through a vendor that provides managed services. The team also has one full-time consultant (for Network Appliance storage filers) and some part-time consultants who handle EMC and HP storage equipment.
Storage Sales, Operations Director, Cisco
As of 2003, the Amsterdam data center storage environment had a raw data capacity of 60 terabytes (TB) in a storage area network (SAN). Previously, the data center used direct-attached storage (DAS).
During 2001 and 2002, the hosting team faced a dramatic demand in data growth and limitations of the early DAS model. DAS was rarely used efficiently, and the cost of storage on servers that were never used continued to increase with each server purchase. Also problematic was that DAS architecture is inflexible. For example, says Steve Moon, storage manager and system administrator for the Amsterdam hosting team, “Because of distance restrictions with DAS, there used to be an implementation rule that no server would get installed unless it contained its own storage of more than 60 gigabytes. So systems under 60 gigabytes could not take advantage of feature-rich storage frame systems. This limitation obviously greatly reduced our flexibility to react to changes and kept us from using shared storage. Moreover, tools for storage inventory, capacity planning, and storage provisioning in such a large and fragmented DAS environment were lacking.”
Instead, the EMEA hosting team had to manually perform the inventory, planning, and provisioning tasks, which consumed the valuable time of the core staff. Jeroen Sourbron, IT project manager, says, “To prepare for the coming years, you need to know the current status and how to define and measure it. You need to be able to track your progress and answer questions like ‘What is your average utilization rate?’ and ‘What is the average duration for provisioning and inventory?’ These questions had to be answered, but we could not answer them without investing valuable time from the core team resources. Inventory was more or less an ad-hoc spreadsheet exercise, and provisioning could often take one employee working full time.”
To overcome these challenges, the hosting team started the migration toward a storage model based on SANs rather than DAS. They set up different SAN islands aligned with the major business applications to replicate the earlier and familiar DAS environment. Then they selected a mix of Brocade and McDATA Fibre Channel switches within the different islands. This environment supported more than 20 TB of data around mid-2002 and increased storage use and made provisioning new storage much simpler and faster.
However, physical limitations of the SAN islands model were soon reached as storage demands from the business increased. Floor space and port capacity on the individual Fibre Channel switches were limited and nearly exhausted. The team needed radical change to overcome these challenges and prepare storage resources for the coming years.
In early 2003, the EMEA hosting team began planning a migration from individual SAN islands based on smaller Fiber Channel switches to a single, shared SAN service based on the larger Cisco MDS 9509 Multilayer Director Switch. The Cisco MDS switch, they reasoned, had the high port density, support for virtual SANs (VSANs) on the same switch, and enhanced features that greatly reduce complexity and increase operational efficiency.
Core employees within the hosting team planned, staged, and implemented the migration toward one physical SAN fabric. The hosting team was able to free the employees needed for the migration by outsourcing routine operational and repetitive tasks. “We out-tasked routine operational tasks such as managed services to an outside services group, governed by strict service reporting, service level agreements, and a flexible billing model,” says Bob Stemmerick, IT manager of the hosting team. “This decision allowed Cisco hosting team members to refocus their priorities on core tasks and projects, which included the Cisco MDS implementation. Also, there is no value in a senior systems administrator racking his own servers or performing tape management duties instead of deploying our SAN fabric using Cisco MDS switches. We want our staff to work on innovation, high value chain, low-touch activities. The benefit of out-tasking versus outsourcing is that the work gets done, but Cisco IT is responsible
and accountable, and we retain full control of the tasks at a lower cost.”
Moon mapped out the strategy for the Cisco MDS architecture (Figure 1). A core switch-edge switch design, using a Cisco MDS 9509 Multilayer Director Switch and a Cisco MDS 9216 Multilayer Fabric Switch, was chosen based on total cost of ownership (TCO) and flexibility to adapt to changes.
Early in the migration, the Cisco Advanced Services team also provided input and approval on the milestones throughout planning and design. Cisco IT reaped the benefits of the Advanced Services team’s expertise and as a result has increased confidence level in the deployment. Integrating an IP SAN environment into the existing Fibre Channel environment is also expected to benefit from the involvement of the Cisco Advanced Services team. “The experience and assistance from the Advanced Services resources in Amsterdam has decreased our deployment time from weeks to a couple of days,” says Sourbron.
Figure 1. Shared SAN Environment in Cisco Amsterdam Data Center
The SAN contains two distinct paths; the two core switches/fabrics are not interconnected in a meshed configuration. However, Cisco IT plans to connect all SANs in each data center into a single SAN to create a single storage service shared by all business groups. Other plans to connect SANs between data centers are also under consideration. Seven VSANs reside on each core, and their VSAN IDs are selected from a global Cisco block of reserved IDs. This method will help ensure that future interconnection among these cores will not result in a VSAN zone merge, which could occur if two VSANs are given the same VSAN ID and then connected.
Tape libraries are connected to the edge Cisco MDS 9216 switches and on a separate VSAN to other disk-based storage VSANs. A port channel aggregates backup traffic and lowers total cost of ownership (TCO) for the SAN ports. All storage is attached to the core Cisco MDS 9509 switches through 16-port Fibre Channel switching modules (which support 16 Fiber Channel links of 2 Mbps each) for higher performance.
All Inter-Switch Links (ISLs) are deployed on 16-port line cards for higher performance and are paired to reduce the likelihood of topology change if a link drops. All hosts are connected to Cisco MDS 9000 Family 32-port Fibre Channel switching modules, except for the NetBackup media servers. All Priority 1 hosts are connected to pairs of S-32 line cards on each core switch, which means that each host is multipathed to four line cards.
After planning, product availability, and physical installation of the equipment during 2003, the migration of existing systems onto the Cisco MDS SAN fabric started in October 2003. In total, 10 Cisco MDS switches were installed with 225 Fibre Channel ports in use and 20 ISLs or trunk ports. This deployment has the capacity to grow to up to 560 available Fibre Channel ports and 16 Gigabit Ethernet ports.
- Cisco MDS 9216 Multilayer Fabric Switch
- Cisco MDS 9216 Multilayer Fabric Switch
- Cisco MDS 9000 Family Fibre Channel Switching Modules
- Cisco MDS 9000 Storage Services Modules
The migration was completed in four months, a major achievement, especially considering that the team could not make changes to the data center during freeze periods, which accounted for 30 percent of the overall time. New application servers and storage systems that were needed for expansion were already connected to the core Cisco MDS switches at that point.
The migration was unusually smooth. During the migration period, only one trouble report was received, which is rare in such a critical environment. “There were no unplanned outages, although some remediation had to be done because of existing multipathing configuration,” says Moon. “Overall, only 10 percent of the systems migrated required planned downtime, with an average duration of 30 minutes. With the Exchange migration, there was, for example, no planned downtime, no reboot necessary. Over the total migration period, 90 percent of the time has been spent before the migration, in the planning stage.”
Marcus Chambers, storage sales operations director, says, “I am very excited that we have had such a straightforward migration of all our applications within the data center to a 100 percent Cisco MDS environment. We not only managed to do this with zero unplanned downtime, but also are already seeing cost savings from being able to consolidate all disparate SAN islands into one virtual SAN. I am confident that we will continue to see increased savings by the management tools and quality of service because of the intelligence and management software in our Cisco MDS switch.”
With the migration barely completed, benefits of reduced risk and increased productivity and cost savings were immediately apparent.
Overall risks of storage networking have been reduced greatly by using VSANs which enhance the overall protection level of the environment. Moon adds, “Not only is the whole security environment easier to manage; you also avoid the risk of corruption. Any damage that could occur within one VSAN is not communicated across the SAN to other VSANs.”
The profitability of the data center is increased by offering investment protection to maximize capital expenditures. The high port density of the Cisco MDS 9509 switch and the possibility of aggregated uplinks allow for expansion of storage subsystems.
Cisco Storage Manager and System Administrator
Productivity has increased through more efficient execution of daily storage management tasks. The Amsterdam data center nearly tripled in capacity from 2002-2003, from 21 TB raw storage to 60 TB, and staffing has remained the same, namely one full-time Cisco IT employee. Another advantage is that building the SAN using only the Cisco MDS platform allows the administrator a single management view for all storage subsystems, including the backup library and all connected application hosts.
One benefit of this single management platform is its advanced troubleshooting. As Moon says, “Of the few issues we encounter, most arise during setup time of a new server or application. For systems already running, the Cisco MDS switch offers advanced troubleshooting features that allow me very quickly to look at the frame level. Low-level analysis of the frames is possible. I would need advanced licensing from Brocade or McDATA to do this same kind of low-level analysis, but this is enabled by default on the Cisco MDS.” Also, the inventory of systems can be done at the touch of a button, which greatly reduces the manual effort that was previously required and makes application growth control more efficient.
The speed of reaction to unexpected business changes has increased. "When sudden requests come up, I do not have to worry about putting systems in a specific spot to be within reach of the necessary storage, and the physical planning in terms of floor space is easier. Now it is easier to configure an individual storage disk LUN [logical unit number] on the consolidated environment with the host systems localized," says Moon.
Three major evolutions are possible because of the migration to a single, multiprotocol storage environment. Cisco IT in the EMEA region is investigating each of these options for future improvements.
Integration of IP SAN using Small Computer System Interface over IP (iSCSI). Formid-level to low-level applications, mostly based on Windows 2000, using Cisco MDS 9000 Series 8-port IP Storage Services Module will allow Cisco IT to integrate an iSCSI-based environment with Cisco Catalyst® 6500 Series Switches. This integration would greatly reduce the TCO for this environment. A study in the EMEA region has indicated that enabling a host with Fibre Channel is more expensive than enabling the same host with iSCSI. Using iSCSI saves from US$2000 to $3200 per host. The break-even point to allow for payback of the investment in an IP SAN is reached when 38 new hosts are connected to the IP SAN environment.
The test environment for IP SAN has been set up to establish a benchmark blueprint design that can be used globally to provide high storage availability and security.
Implementation of EMC Control Center Software. Implementation of EMC Control Center software as storage resource management (SRM) software would have inherent benefits. By implementing SRM on top of the consolidated environment, storage provisioning would be even more efficient because everything could be managed through a single view.
Interconnecting data centers to create a virtual data center. The architecture allows for flexibility with the extension and connection of other Cisco data centers to create a “virtual” data center that would make manageability of business continuity more efficient. This interconnection and preparation for future data center expansion is made possible by connecting SAN gateway switches. For example, Cisco MDS 9506 Multilayer Director Switches could use the same services module within the Cisco MDS for Fibre Channel Interface Protocol as the remote connectivity layer.