A major shift is occurring in the hosting of applications across a wide variety of enterprise and service provider data centers. These shifts are accelerating because of the growing facility and administration costs associated with managing hundreds to thousands of servers, and a reversal in thinking that more is better. Moreover, the application groups are becoming more demanding with faster time to productivity requirements and the need for the IT infrastructure groups to activate servers in hours rather than days and months. These requirements call for new approaches to server hosting architectures and the operational models that support them.
Server virtualization is an important new hosting technology that is gaining increased market adoption. This technology abstracts the applications and operating system from the underlying server hardware. This abstraction, commonly referred to as a hypervisor, allows for server consolidation because one physical server can reliably host multiple independent virtual machines. It also offers the ability to transparently move applications from one physical server to another based on virtual server load-balancing policies. This hosting model addresses many of the growing cost concerns today, as well as offering greater agility when deploying new applications.
A large number of space- and cost-constrained data centers have already benefited from this technology. Many customers are consolidating Intel and AMD microprocessor-based servers with hypervisors and have achieved on average at least a 10-to-1 server consolidation of applications running in virtualized machines. This has allowed them to reclaim rack space, cooling, and power and in many cases has deferred the need for a new data center.
Given the early success of these hosting changes from physical to virtual, server hosting managers are aggressively looking at expanding server virtualization with applications that require a rich set of networking services. These networking services typically include tightly controlled broadcast domains (VLANs), Internet security (firewalls), application optimization (server load balancing), Secure Sockets Layer (SSL) offload, quality of services, and service-level monitoring. Moreover, as server hosting managers add more load-balancing automation into these virtualized server environments, they will require more "on-demand" physical server capacity.
These growth and application expansion requirements will necessitate greater integration with the networking and storage hardware provisioning and operation teams because the real-time dependency on the services these groups manage will increase. The current manual approach in requesting these services, including weeks of planning, phone calls, and provisioning approval processes, will not scale, based on the real-time nature of how applications need to be hosted and scaled in minutes and how virtual machines can be moved in real time between servers.
The coupling of these network and storage resources and the ability to change these services in real time to align with new virtual machine deployments or moves require a dynamic, orchestrated provisioning approach. Specifically, as virtual machines move across server pools, based on load balancing policy changes, the end-to-end networking services must be configured to maintain a consistent networking environment on a per virtual machine basis. A failure to ensure the consistency of networking environment could prevent a migration from completing or result in a loss of the virtual machine's network security context. In addition, communications could be lost and performance could be greatly degraded. Bottom line, this mobility of virtual machines must be orchestrated with the networking services.
This white paper extensively reviews many of the server virtualization hosting benefits offered by VMware, the added value of Cisco
® network services in scaling virtual machines when hosting business critical and Internet-facing applications, and a real-time integrated provisioning approach between VMware's VirtualCenter management platform and the recently released Cisco VFrame Data Center (VFrame DC) service orchestration platform.
The Changing Landscape of the Data Center
The face of the data center is changing. Accelerated by pressure from rising energy prices, mounting real estate costs, and increasing global competition, enterprises are quickly adopting virtualization technologies. For example, with VMware Virtual Infrastructure 3 (VI3), server hosting managers can consolidate multiple services onto autonomous virtual machines running on a single physical server chassis and dynamically redistribute those services across a virtual infrastructure. Virtualization gives organizations the ability to quickly host applications with higher availability and greater efficiency, and at a lower cost than in traditional data centers. Using VMware's virtualization technology, organizations can deploy or upgrade services in minutes or hours-not days or weeks. IT organizations can mitigate the disruptive and costly effects of change management by migrating services from one physical host to another with no downtime. Some enterprises have scaled their virtualization environments to host thousands of services on virtual machines. In fact, many organizations have adopted VMware's virtualization technology as their business standard and best practice for the deployment and hosting of application services.
Virtualization Accelerates Provisioning Model Changes
The power of virtualization technology, such as VMware VI3 and the positive results IT organizations have experienced with this technology, has led to a commensurate rise in the level of customer expectations. Application and server hosting managers expect to deploy new applications with greater server platform mobility, flexibility, and agility. For many of these applications there is a growing need for network services, including secured Layer 2 partitions; access control filters; application optimization; Web caching; and a host of other performance, availability, and security-enhancing features. This need is placing increased pressure on the networking provisioning teams to configure these services within hours, as well as offering the ability to dynamically adapt these services in real time when a virtual machine moves from one server to another.
Unfortunately, traditional network provisioning approaches do not meet the needs of these new server hosting models. The static model of coordinating the services on a spreadsheet between departments during the design stages and the need to call each group prior to any operation, do not serve today's new dynamic server virtualization approaches. This old school approach often takes days or weeks to implement, whereas a virtual machine can be operational in hours.
Clearly there is a growing chasm between today's manual service planning and provisioning approach and the growing momentum of real-time, adaptive hosting with lights-out policy-based provisioning changes. In this chasm, either the network security configuration of the virtual machine may be compromised as the server hosting department may choose to simplify the topology to minimize complexity, or the powerful real time mobility aspects of virtual machine mobility is greatly reduced as configuration requests are manually coordinated and these can take days before changes go into effect.
A manual, loosely defined approach to cross-organizational infrastructure planning and deployment has become impractical. Many data center managers are enthusiastically promoting the industry's market-leading virtualization technology, VMware VI3, as their company-standard hosting platform. Yet, having to configure unique network attributes such as firewall rules, access control lists, broadcast domains, tightly managed VLAN memberships, and virtual IP address settings within, for example, a server load balancer takes time, and a great deal of interdepartmental coordination. The organizational inertia created by the traditional "siloed" approach to data center management is leading organizations to reassess the efficiency of their cross-functional communications and processes.
Oversimplifying the End-to-End Infrastructure for Virtual Machine Mobility
Virtual machine mobility, or VMotion, is a critical benefit of VMware VI3. Using VMotion, administrators can migrate a running virtual machine from one physical VMware ESX Server to another without any disruption of the running applications or operating system. VMotion eliminates the lock between the physical host and the operating system or applications. A set of VMware ESX Servers effectively becomes a "cluster" or "resource pool", with virtual machines moving from host to host. This non-disruptive mobility has compelling implications for data center maintenance and load balancing. VMotion allows IT administrators to migrate all running virtual machines off a physical host, creating a window in which to perform system maintenance on that host without affecting service availability.
Using VMware's Distributed Resource Scheduler (DRS) in conjunction with the VMware VirtualCenter Management Server, administrators and business stakeholders can optimize and load-balance virtualized resources. By defining a set of policies and triggers, DRS can automatically redistribute running virtual machine workloads across a defined set of virtualization hosts. These automated migrations can be triggered from within VirtualCenter or through VMware's management application programming interface (API).
The creation of a VMotion domain with an ESX resource pool requires compatibility of all participating virtualization hosts. Each host must have access to the same storage area network (SAN)-based shared storage, as well as the same set of network VLANs, among other attributes. Data center managers implementing VI3 must account for these network and storage dependencies when designing their architectures, especially when they move virtual machines from one server to another. The target server must have access to the same services for the application being migrated to function the same. Adding granular services on a per-virtual machine or per-physical server basis adds complexities, while at the same time improving performance, security, and scalability.
Although VI3 and VMotion operate well in functionally rich network and storage topologies, administrators who are either unfamiliar with virtualization or unable to quickly make changes when they move a virtual machine may oversimplify their network and storage topologies to minimize the perceived complexity and present a manageable view of their infrastructure. This simplification may place IT in the position of having to compromise the desired architecture. These decisions, led by expediency, can threaten adherence to specific performance, availability, and security metrics of a negotiated service-level agreement (SLA).
Dynamic Service Orchestration Platforms
These changing hosting demands are leading the industry to create a new class of provisioning platforms. These platforms must be designed to meet the demands of the new, dynamic nature of service provisioning, where application services are mobile across server platforms, and the storage and network configurations on which they depend are subject to frequent changes. These new provisioning platforms must support multiple layers of data center infrastructure as they are orchestrating the delivery of service instances, not simply managing the configuration of a single Layer 2 switching domain. And these new provisioning platforms must integrate with external policy tools, which make decisions on where to host virtual machines. This requires an extensible two-way API.
These new provisioning systems are known as "dynamic service orchestration platforms". The platforms are "dynamic" in the sense that they can make configuration changes in real time, with a high degree of granularity, and can react to, or anticipate the needs of, the services they are supporting. The term "orchestration" refers to the ability of the platform to manage the configuration requirements of multiple, disparate technologies that constitute a "service" definition. And the term "service" refers to a set of underlying physical and virtualized infrastructure resources that are grouped together both physically and logically based on the requirements of the application being hosted, end to end.
The configuration requirements of a service often span multiple layers of technology, including physical servers, virtual machines, networking chassis, virtual networks, virtual firewalls, and storage infrastructure. Points of configuration can include Open Systems Interconnection (OSI) Layer 2 and Layer 3 Ethernet services, SAN boot capabilities and storage logical unit number (LUN) provisioning, as well as network content load-balancing services. The service definition may include high-level functions such as the creation of a "fast-failover" network topology or high-availability storage using multipathing access in "virtual switch" mode.
The dynamic service orchestration platforms themselves are typically offered as an appliance with a secondary appliance for configuration database backup and high-availability recovery if the primary fails. These provisioning appliances perform a multitude of functions, including the discovery of the underlying fabrics and available network services. These services are abstracted as a set of configurable resources and are grouped together through service design templates. Specific resource filters can be added for ensuring that specific applications are hosted on customer-defined hardware. In the case of VI3, customers can choose to host ESX on a filtered set of servers picked from a general server resource pool.
Fundamental to all these functions is the need to configure many different devices concurrently, based on real-time change events. These platforms must be robust, with the ability to quickly and reliably make configuration changes.
In view of these requirements, Cisco has developed a dynamic service orchestration platform and is aggressively integrating this platform with Virtual Infrastructure 3. This product is known as Cisco VFrame Data Center and is currently shipping with many of the features described in this document.
Dynamic Boot and Provisioning of Stateless Servers
One of the more notable functions of Cisco VFrame Data Center as an end-to-end provisioning platform is its ability to create stateless servers, in which the server no longer maintains the boot image on local internal or directly attached disks (DASs). Instead the server boot is accomplished by directing the server at boot level through the network to arbitrarily defined boot LUNs (SAN) or NAS boot files (network attached storage). This approach removes the one-to-one binding of whatever is on the local boot disk, thus creating a generic pool of servers that can be reconfigured within minutes and making server pools far more flexible. Server hosting managers, based on policies, can re-image a server running Microsoft Windows, with one application, to Vmware ESX Server and run 10 virtual machines, or the reverse thereof.
This emerging approach to centralized bare-metal server boot shifts the focus from the server as a fixed asset to one where the network, through I/O remappings, allows for dynamic server reallocations. Enterprises can now change servers in real time based on the need for more capacity, service recovery, data center hotspot zone changes, or time-of-day maintenance updating. In this manner, organizations can minimize service interruptions, normalize maintenance windows, and increase availability.
Integration of Dynamic Service Provisioning with Virtual Servers
Cisco VFrame Data Center adds significant value in a server virtualization environment. VFrame DC will add bare-metal ESX Server capacity on demand for the case where all the servers within a virtual machine domain pool are reaching full usage. In addition, Cisco VFrame DC will orchestrate the dynamic reconfiguration of network and storage services to support the dynamic nature of virtualized environments.
Overall, Cisco VFrame DC will integrate with VMware Virtual Infrastructure 3, at both the physical server and infrastructure virtualization layers. Based upon predefined policies, these platforms can perform such services as:
• Add "on-demand" capacity by booting an additional server from a preallocated pool
• Configure network and storage services for both the ESX Server and its guest virtual machines
• Reconfigure these dependencies as VMotion migrations occur
• Monitor and adjust network services to assure conformance to defined SLAs
• Automate maintenance updates as organizations undertake software upgrades
Full End-to-End Services View
Cisco VFrame DC can also present a holistic view of a service, tracking all downstream dependencies defined within a service definition template. This holistic view is valuable because it provides a "single pain of glass" for troubleshooting all the configurations that comprise end-to-end services for a specific application. If the service manager encounters a problem, Cisco VFrame DC can verify the state of all downstream dependencies and quickly pinpoint the cause. Using native authorization functions, Cisco VFrame DC can organize and assign service owners and relative levels of access. These service views, service templates, and roles-based access controls offer a better approach to simplify cross-organization coordination and maximize agility.
It is important to differentiate between Cisco VFrame DC and server virtualization technologies, such as VMware Virtual Infrastructure 3. Virtualization infrastructures consist of the hypervisor platforms, the guest virtual machines that run on the hypervisor, and the virtualization management platform that manages the configuration and mobility of the hypervisor and virtual machines. The server hypervisor (such as VMware ESX Server) segments the server into virtual machines, creating discrete partitions with virtualized, dedicated memory, CPU, and I/O hardware for each. The virtualization management platform (such as VMware VirtualCenter) controls the configuration of the virtualization host, its internal networking and storage configuration, and the resource allocation and internal connectivity of the virtual machines. However, VI3 has limited visibility into all the downstream infrastructure services on which it depends.
Cisco VFrame Data Center manages the external dependencies VI3 does not address. It orchestrates the configuration of all the services downstream from the hypervisor, including:
• Network connectivity between the physical host and the adjacent physical switch
• Mapping of the Layer 3 logical address ranges to content load balancers
• Fabric settings between the physical host bus adapter (HBA) and the SAN switch
• Configuration of external security devices, including physical firewalls and intrusion prevention appliances
• Boot of the runtime bare metal server operating system from centralized storage
• Virtualized configuration view of all the downstream virtual machine services
Cisco VFrame Data Center represents the next logical step in the evolution of provisioning platforms. This platform allows organizations to manage services "end to end" as a comprehensive set of objects and dependencies spanning the entirety of their virtualization, storage, network, and security infrastructure. This new approach to provisioning offers enterprises and service providers the agility they now need to manage their increasingly dynamic data centers, minimizing the inertia created by interdepartmental dependencies and facilitating capacity management and service mobility to meet changes in their environment.
Cisco VFrame Data Center Feature Overview
Cisco VFrame Data Center exemplifies a dynamic service orchestration platform that allows the coordinated provisioning and reuse of physical and virtualized compute, storage, and network resources from shared pools in support of application services. Developed to meet the needs of the dynamic data center, VFrame takes advantage of the power of "stateless servers" to deliver on-demand, utility computing resources.
Cisco believes that "bare-metal" provisioning is a powerful approach to managing server capacity, maintenance, and upgrades, enabling service automation using intelligent network capabilities. Cisco VFrame Data Center can instantiate Microsoft Windows systems, Linux systems, and VMware ESX Servers from a pool of bare-metal computing resources. The stateless server approach offers organizations the flexibility to allocate resources dynamically to meet changing demands. (For benefits of server pooling, refer to the Cisco VFrame Data Center product bulletin at
Cisco VFrame Data Center Integration with VMware Virtual Infrastructure
Integration of Cisco VFrame Data Center with VMware VI3 helps data center managers enhance the resilience, responsiveness, and efficiency of service infrastructure to meet dynamic application hosting requirements.
"Enhancing business resilience and agility means that data center managers can optimize architectures that were once static and inflexible by implementing virtual infrastructure. VMware and Cisco are working together to improve and simplify the provisioning models and approaches to scale both VMware VI3 and Cisco's networking services, which are the foundation of an agile and highly available data center."
-Brian Byun, Vice President , Global Partners and Solutions
This tight integration is realized through offerings such as ESX Server capacity automation, VI3 I/O conformity, downstream application and network services, and VFrame DC API integration with VMware VirtualCenter (Table 1).
Table 1. Cisco VFrame Data Center Integration with VMware Virtual Infrastructure
ESX Server Capacity Automation
Automates server installation (not image builds) based on capacity or high-availability requirements such as peak demand periods or server outages
ESX Server I/O Conformity
Provides design conformity that helps ensure best practices regarding SLAs and security
Downstream Application and Network Services
Offers service definitions that help ensure the right levels of security and performance, and offer a way for multiple organizations to coordinate the creation of service definitions
API Integration with Virtual Center
Helps enable bidirectional communication between Cisco VFrame DC and VMware VirtualCenter to improve performance, simplify virtual machine migration, and facilitate system maintenance
These integration offerings are described in more detail in the following sections.
ESX Server Capacity Automation
To complement the booting of bare-metal servers with runtime operating system images, Cisco VFrame Data Center will support the loading of VMware ESX Server, to provide additional virtualization resource capacity. This boot function is event-directed and can be triggered by internal VFrame DC events or through events received through its API. Managed by the Cisco VFrame DC policy engine, the provisioning process adheres to the requirements defined by the service definition template. This template specifies the logic sequence (domino effect) of provisioning steps, helping ensure that all network and storage configuration requirements are met to enable the service for immediate availability. VFrame DC policies can be created to automate the addition of capacity to accommodate peaks in usage, or shifts to a high-availability architecture at critical periods when a service outage could be particularly dangerous.
Cisco VFrame Data Center supports the deployment of VMware ESX Server onto bare-metal hardware. VFrame DC offers data center managers the ability to use their chosen server imaging tools to build a "gold" VMware ESX Server image. Using VFrame DC, administrators can replicate that image onto the LUNs from which they will remotely boot an operational ESX Server image on physical hardware. When the VMware ESX Server boot images are distributed, managers can dynamically allocate bare-metal Intel and AMD based servers on which to load those images to add additional virtualization capacity as needed. However, VFrame DC also orchestrates the configuration of the downstream network and storage that the newly deployed server will require.
Cisco VFrame Data Center can dynamically add VI3 capacity, and it can do so intelligently. The Cisco VFrame Data Center resource manager maintains a database of server hardware configuration attributes, and it can intelligently match the hardware requirements of a given deployable image with the members of the utility computing pool that meets those requirements. Only servers that match the filter criteria defined within VFrame DC can be designated as candidates for instantiation. For example, if the server hosting manager specifies that automated ESX Server deployments can take place only on systems with two CPU sockets with dual-core CPUs and 4 gigabytes of RAM, VFrame DC boots only servers that match those minimum system requirements.
VI3 I/O Conformity
Cisco VFrame Data Center offers an intelligent, automated resource discovery process that can map server connectivity across both the Ethernet network and the Fibre Channel fabric. This discovery function analyzes all I/O of the physical host, capturing information including the number of network interface cards (NICs) and HBAs, as well as how those interfaces map to the access layer and Fibre Channel switches. This discovery and I/O mapping concurrently spans all services being accessed by the profiled system. The profiling process is performed by the VFrame DC high-availability appliance without disruption of targeted server activity. All the data generated by the discovery process is then populated to a backend database that can be exported for use in other environments.
The comprehensive visibility of Cisco VFrame Data Center into the I/O requirements of a profiled server, in combination with its ability to reconfigure the I/O configuration at boot time, provides a powerful platform for integration with VMware VI3. By using VFrame DC to allocate additional virtualization server capacity, data center managers can help ensure the consistency of all ESX Server instances. Using the policy profile and resource manager definitions in VFrame DC, administrators can be sure that each ESX Server meets strict configuration guidelines. Each new host will have the required number of network and storage interfaces and each of those interfaces will be configured with the appropriate VLAN, virtual SAN (VSAN), and storage zone settings. Provisioning additional servers in this manner assures the immediate availability of the newly added capacity and provides certainty that each instance will adhere to best practices, meet SLA requirements, and follow security guidelines, no matter how many times the process is repeated.
Orchestration of Downstream Application and Network Services
For virtualization infrastructures today, the data center network should be designed and provisioned with the services required to enhance the scalability, security, and availability of virtual machine deployments. The guiding principle for network design is to help ensure that applications residing on a virtual machine can achieve the same service levels as if they were hosted on a physical server.
Cisco VFrame Data Center helps ensure that application and security services can be deployed in a dynamic and consistent way to meet the defined SLAs of the virtualized infrastructure.
In coordination with VMware VirtualCenter, Cisco VFrame DC offers each application service running on different virtual machines the appropriate network security and application services so that all virtual machine-hosted services are protected and optimized as they are deployed.
API Integration with VirtualCenter
Cisco VFrame DC has been developed with a Simple Object Access Protocol (SOAP)-based Extensible Markup Language (XML) API that can be integrated with many northbound- and southbound-facing data center operation tools. Support includes integration with VMware VirtualCenter using the Virtual Infrastructure SDK (VI SDK). Specifically, Cisco VFrame Data Center has a well-defined API that can notify VirtualCenter when a new ESX Server is instantiated remotely through the VFrame DC and is ready to host virtual machines. When VFrame DC has completed the booting of the newly provisioned ESX Server, it will hand the newly provisioned ESX Server to VirtualCenter and allow VirtualCenter to make autonomous decisions regarding which virtual machines should be run on the newly provisioned ESX Server instance.
Additionally, VirtualCenter will notify VFrame DC regarding the need to provision additional ESX Server capacity. Cisco VFrame DC will be able to quickly bring up another ESX Server and add it to the VI3 resource pool through its dynamic remote boot and resource manager function. This boot takes no more than 5 minutes and is based on the time it takes to load ESX Server physical hosts across the network. This capability is useful in resolving any physical server outages VirtualCenter detects. If VirtualCenter detects the failure of a physical server, it will automatically restart the virtual machines from the failed host on the remaining ESX Servers in the cluster and notify VFrame DC to instantiate a new ESX Server to replace the resource capacity.
The converse is true as well. If either Cisco VFrame DC or VMware VirtualCenter detects idle server capacity above a threshold defined by the system administrators, both systems will communicate to coordinate the removal of the underused ESX Server from the resource pool (after triggering a VMotion migration of all virtual machines on that server to other ESX Servers in the resource pool). This removal also includes the removal of downstream configuration parameters by Cisco VFrame DC. This removal has to be fully coordinated to ensure all virtual machines are migrated prior to removing any server.
Finally, this bidirectional communication between Cisco VFrame DC and VMware VirtualCenter can be used effectively to perform ESX Server 2.5.0-to-ESX Server 3.0 migrations, or to perform ESX Server maintenance migration from one version level to another. This level of integration becomes critical when dozens of VMware ESX Servers need to be upgraded and the upgrades require a reboot. In this case server hosting managers first migrate the running virtual machines from the server they need to upgrade (easily done by placing the ESX Server(s) in maintenance mode, a step that triggers the migration automatically).
These migrations can be enhanced by Cisco VFrame DC, because VFrame can check to ensure that all ESX Servers hosting the migrated virtual machines have the required I/O connection as the back-channel migration transport (based on the best practice recommendation from VMware). When the virtual machines are migrated successfully, VirtualCenter can signal Cisco VFrame DC to install a new ESX Server build. When the new install is completed, VFrame DC can direct VirtualCenter to reintegrate the upgraded ESX Server into the appropriate resource pool and VirtualCenter can migrate the original virtual machines back to the upgraded server using VMotion.
Using VMware VI3, organizations can quickly deploy additional virtual machines on their VI3 infrastructure to add service capacity as needed. Using VMware VMotion and the VMware Distributed Resource Scheduler, data center managers can optimize the distribution of virtual machine workloads across their ESX Server resource pools. And, as enterprises and service providers expand their VI3 deployments into the hundreds or thousands of ESX Servers, with thousands of virtual machines, managers can use Cisco VFrame Data Center to simplify and automate the provisioning process.
Cisco VFrame DC offers a rich middleware platform to orchestrate the provisioning of VMware ESX Server in a quick, easy, consistent, and repeatable manner. Cisco VFrame DC will integrate with VMware VI3 to automate the provisioning of stateless, network-based ESX Server images onto a utility pool of Intel and AMD processor based servers. Using service definition templates to capture end-to-end requirements, this integration will help orchestrate the delivery of all the network and storage services on which ESX Servers depend. Through rich API integration, VFrame DC can dynamically adjust the network and storage configurations in response to the changes in the virtual infrastructure and add additional ESX Server capacity to adjust to changes in demand for a service.
The powerful integration of Cisco VFrame Data Center and VMware Virtual Infrastructure 3 can optimize the performance, availability, agility, and efficiency of service infrastructure to meet the evolving demands of the dynamic data center.