Cisco on Cisco
Data Center Case Study: How Cisco IT Virtualizes Data Center Application Servers
Deploying virtualized servers produces significant cost savings, lowers demand for data center resources, and reduces server deployment time.
(PDF - 285 KB)
A large enterprise IT department might manage hundreds of corporate applications, running on thousands of servers. In many cases, each application or instance requires a dedicated server, even if the application uses only a small portion of the serve's available physical CPU or memory capacity. The result? Overall computing resources are not being used efficiently. Meanwhile, IT departments are facing rapid growth in the number of servers that must be purchased, deployed, monitored, and maintained - along with associated resources such as electrical power, cooling, rack space, cabling, and support personnel.
Cisco IT has faced these conditions in its data centers, with nearly 4000 applications running on more than 11,000 servers as of early 2007. What's more, this large number of installed servers was growing at a rate of 15 percent per year. To support this many servers, Cisco IT needed to address the following challenges:
- High costs of server hardware, cabling, and installation support
- Limited space in existing data centers, with little to no available space for data center expansion in Cisco's existing campuses
- Growing demand for expensive electrical power and air conditioning in the data centers. Backup power, provided by uninterruptible power supply (UPS) units and diesel generators, creates concerns about environmental impacts.
- Increasingly longer delays (12 weeks and more) for deploying new servers
Server Virtualization Program Manager, Cisco IT
"As our data center resources became more constrained, it took more planning time to find where the space and power and cooling were available to install a new server," says Mike Matthews, Cisco IT program manager for server virtualization.
To overcome these data center challenges, Cisco IT chose to deploy virtualized servers and related Cisco technologies for server networking and data center management. Instead of the
- Support 4000 applications running on 15,000 physical and virtual servers
- Meet server growth rate of 15% per year
- Reduce costs and demand for power, cooling, space, and cabling in data centers
- Virtual servers using VMware
- Related Cisco technologies for server networking and data center management
- US$19 million in cost savings and cost avoidance
- Reduced need for data center space
- Faster deployment of new servers
- Increased productivity for IT personnel
- Improved application security and stability
- Choose suitable applications and servers
- Create appropriate management practices
- Prepare internal teams for the migration
- Gain a clear understanding of vendor support
- Consider a “proof of concept” project
- Consolidate data centers in a new facility
- Deploy virtualized data center provisioning solution
Virtualized servers allow Cisco IT to solve a critical issue: the time required to provision new servers. "If we cannot provision servers within committed timeframes, that affects development projects and increases cycle time and costs. These costs rise especially when newly hired development resources remain idle waiting for compute services," says Ken Bulkin, senior IT manager of the Cisco Service-Oriented Data Center (SODC) team. The SODC team is a strategic program within Cisco IT, established to deliver IT innovation and productivity enhancements through technologies for virtualizing and orchestrating computing resources.
"Provisioning delays caused us to become aggressive in deploying virtualized servers, even for applications where stated support did not yet exist. We tested these applications and assumed a measured support risk in order to leverage the agility virtualization introduced. We were confident that these applications would operate correctly, but also provided a plan for backing out the installation to a physical server if required," says Bulkin. "In our first 1500 virtualized server deployments, we only needed to take that step once."
Cisco IT currently uses the VMware Infrastructure 3 as the foundation for server virtualization in the data center. VMware supports the creation of virtualized servers, each potentially using multiple CPUs and multiple gigabytes of memory. The number of CPUs and memory can be modified easily as applications grow, and Cisco IT can relocate virtualized servers between physical servers to accommodate an application’s changing demands for computing resources, to handle an incident, or to conduct planned maintenance.
Server virtualization also allows multiple operating systems to be installed on a single physical server. Each application runs on a standard, dedicated operating system instance (for example., Microsoft Windows or Linux). Only the physical server resources are shared among the operating systems or applications. This design increases overall use of the physical hardware without sacrificing application availability, reliability, or integrity.
In the production environment, Cisco IT has deployed VMware in server farms, which group four, eight, or twelve physical servers. The server farms also provide flexibility for distributing traffic loads as well as redundancy if a physical server fails. Cisco IT maintains separate VMware environments for application testing by developers.
The VMware farms are managed to provide the flexibility, reliability, and security required by Cisco's mission-critical applications. "We continually monitor the performance of the servers and farms to prevent issues that will negatively affect application performance," says Matthews. "In cases where one server experiences a failure or high utilization, we will distribute or relocate the affected virtualized machines to other physical servers in the farm. We can make these changes while the servers remain online, with no service interruption to the relocated virtualized machines."
A variety of Cisco technologies support virtualized servers within the overall operation and management of the data centers (Figure 1). These technologies include routing, switching, load balancing, content switching, and storage-area networks (SANs) for connecting servers, storage devices, and other data center systems across the Cisco network. Cisco also uses an internal management system for managing devices on the Cisco network, including the virtualized servers.
Figure 1. Virtualized servers in a Cisco data center are accessible from the Cisco network and can interact with pooled storage for greater computing efficiency.
Server virtualization is also a key component of the Cisco IT SODC model, which allows computing, network, and storage resources to be dynamically partitioned, provisioned, and assigned to different applications through an intelligent network fabric. "We expect to manage the majority of our compute infrastructure needs end to end from servers to network services and storage, whether these resources are virtualized or not," says Bulkin. For more on Cisco IT's migration to the SODC model, see the case study at http://www.cisco.com/web/about/ciscoitatwork/data_center/dc_consolidation.html.
Cisco IT determined that slightly more than half of its existing servers were viable candidates for reconfiguration as virtualized servers. Based on this assessment, Cisco IT identified the goals of virtualizing 50 percent of existing servers and 75 percent of newly deployed servers. As Cisco IT continues working toward these goals, the deployment of virtualized servers has yielded significant results.
Cost savings due to fewer physical servers. In early 2007, Cisco IT had deployed more than 1500 virtualized servers, which produced cost avoidance and savings valued at a cumulative total of nearly US$10 million (calculated as of mid-2007). Approximately 70 percent of the virtualized servers were new deployments that avoided the purchase of new physical servers and associated hardware and cabling. The remaining 30 percent of deployments were existing physical servers that were reconfigured as virtualized servers to support more applications.
Cisco IT estimates that a virtualized server costs approximately US$2,000 to deploy, compared to US$7,000 for a standard physical server with two CPUs. Fewer physical servers also lowers costs for operation, management, maintenance, and support.
Reduced need for data center space. A typical physical server supports between 10 to 20 virtualized servers running applications. With fewer physical servers to install, Cisco IT can slow the demand for data center space even as the number of deployed server instances continues to grow. Fewer physical servers also equates to lower demand for power, air conditioning, and backup resources in data centers. These reductions will have a significant financial impact because power costs are predicted to reach 30 percent of the Cisco IT budget by 2011.
Faster deployment of new servers. With fewer physical servers to deploy, Cisco IT can now fulfill a new server request within a standard target of three days, but when necessary, within a few hours. Bulkin says, "We now have physical servers in the data center that are already provisioned to support new virtualized hosts for applications, which means that we can quickly fulfill new server requests from development teams."
Additionally, Matthews says, "When virtualized servers can be installed rapidly, development projects can be completed very quickly, which allows Cisco IT to complete more internal requests for application enhancements, accelerate application development, and benefit from general agility in our computing infrastructure."
Increased productivity for IT personnel. Cisco IT expects to manage more than 15,000 servers (virtualized and physical) by 2009. Data center support teams can be more productive because virtualized servers are much simpler to deploy and provision than physical servers.
Improved application security and stability. Applications experience fewer, and shorter, client-impacting outages caused by hardware failure or system maintenance activities. Applications security is enhanced with secure virtualized LANs (VLANs), improved monitoring, and an encapsulated operating system.
Cisco IT has gained several insights for other enterprises that are considering deploying virtualized servers.
Choose applications and servers that are suitable for virtualization. Not all applications are suited to deployment on virtualized servers, and not all servers are suitable candidates for virtualization. Applications that might not allow operation on a virtualized server include those that use specialized devices or that require VLAN or server load-balancing support. Physical server factors that do not allow virtualization include requirements for more than two CPUs, significant requirements for memory and disk space, and servers that already have high CPU utilization levels.
Create appropriate management practices. "Virtualized servers are prompting us to look at capacity management differently and at various levels of granularity," says Bulkin. "We continue to monitor the traditional factors of CPU, memory, and disk I/O. We are also starting to look at the number of virtualized machines running on a physical server and the associated criticality of the services running on that physical server."
"Introducing a new technology often comes with the need to reinvent our processes," Bulkin says. "For example, as a best practice we want to help ensure that all mission-critical applications are not running on the same physical server."
Prepare internal teams for the migration effort. Application developers will need to be educated about the benefits of server virtualization and be closely involved in the migration activity. The IT team can expect to address developers' concerns about the migration time and effort, the risk of application disruption, and the testing requirements. "Although we have tried to make the migration to virtualized servers as simple as possible, it still requires a lot of work and close coordination with the application teams," says Matthews.
Gain a clear understanding of vendor support. Vendors might not fully support virtualization of their applications, and this can influence which applications run on virtualized servers and in what form. "In some cases, we may run only the development and testing environment for the application on virtualized servers, or only part of the application," says Matthews. "We also test how the application will perform on a virtualized server before changing its production deployment."
Bulkin says, "Your willingness to assume a greater level of risk in terms of internal support for applications will be a key factor in determining how much your IT organization will adopt virtualization. In addition, you may need to work more closely with key software vendors to confirm their position on virtualization as a supportable technology."
Senior IT Manager,
Cisco SODC Team
Consider a "proof of concept" project. Before virtualizing production servers, the IT team might implement a test project to verify the migration process and application performance on a virtualized server. This proof of concept can also identify any application issues that may need resolution with the vendor, as well as proper configuration of storage and network connections and services. For help in troubleshooting problems that arise during application migration to virtualized servers, sufficient physical server capacity should be maintained to recreate the application environment and to rule out virtualization as a potential root cause of the problem.
Plan for the same number of server instances. Deploying virtualized servers does not necessarily result in a reduced level of server management. "In our case, virtualization did not reduce the number of server instances that must be managed," says Bulkin. "However, the agility of the virtual environment and our ability to provision servers rapidly allowed us to appropriately scale our support staff with the increase in server instances."
Cisco IT has been preparing for two major data center developments that will begin in late July 2007 and continue through 2008. The first is consolidation of all Cisco North America production data centers into a new facility located in Richardson, Texas, which will provide ample space for future expansion. The second is deployment of the virtualized data center provisioning software solution, which will greatly enhance how Cisco IT manages servers, storage, and other I/O resources to create virtual computing services end to end in the data center.