Industry trends indicate a vast data center transformation toward shared infrastructures. Enterprise customers are moving away from silos of information and moving toward shared infrastructures to virtualized environments and eventually to the cloud to increase agility, operational efficiency, optimal resource utilization, and reduced costs.
FlexPod® is a pretested data center solution built on a flexible, scalable, shared infrastructure from Cisco Unified Computing System servers with Cisco Nexus switches and NetApp® unified storage systems running NetApp Data ONTAP®. FlexPod components are integrated and standardized to help you eliminate the guesswork and achieve timely, repeatable, consistent deployments. FlexPod has been optimized with a variety of mixed application workloads and design configurations in various environments such as virtual desktop infrastructure and secure multi-tenancy environments.
Customers should accelerate their transition to the cloud with the FlexPod data center solution, which integrates disparate compute, storage, and network components into a single architecture that scales to fit a variety of virtualized and nonvirtualized customer environments. But with the increased complexity and extreme performance requirements, FlexPod with Cisco UCS can reduce business risk, increase data center efficiencies, and protect current investments, while scaling for future growth.
Key points of FlexPod
• Single platform from industry leaders in networking, computing, and storage.
• Pretested, validated solution platform to reduce risk and increase efficiencies.
• Flexible IT architecture for today's needs, yet scales for future growth.
• Cooperative support model for efficient and streamlined resolution.
VMware vSphere built on FlexPod includes NetApp storage, Cisco networking, the Cisco Unified Computing System (Cisco UCS), and VMware virtualization software in a single package. This solution is deployed and tested on the defined set of hardware and software. For more information on VMware vSphere built on FlexPod architecture, see http://www.netapp.com/us/technology/flexpod/.
Target Audience
This document is intended to assist solution architects, project managers, infrastructure managers, sales engineers, field engineers, and consultants in planning, designing, and deploying Oracle Database11g R2 GRID Infrastructure with RAC option hosted on VMware virtualization solutions in a FlexPod environment. This document assumes that the reader has an architectural understanding of the Cisco Unified Computing System, VMware, Oracle Database 11gR2 GRID Infrastructure, Oracle Real Application Cluster Database, NetApp storage systems, and related software.
Purpose of this Guide
This Oracle Database 11g R2 GRID Infrastructure with RAC option on FlexPod using VMware demonstrates how enterprises can apply best practices to deploy Oracle Database 11g R2 GRID Infrastructure with RAC option using VMware vSphere, VMware vCenter, Cisco Unified Computing System, Cisco Nexus family switches, and NetApp FAS storage.
This design validation was completed using typical OLTP workloads in two different scenarios.
1. Deployment of four node Oracle Database 11g R2 GRID Infrastructure with RAC option in a virtualized environment.
2. Scaling and consolidation study of multiple, two-node Oracle Database 11g R2 GRID Infrastructure with RAC option in a virtualized environment.
Business Needs
Business applications are moving into the consolidated compute, network, and storage environment. FlexPod using VMware helps to reduce costs and complexity of every component of a traditional Oracle Database 11g R2 GRID Infrastructure with RAC option deployment. The complexity of integration management is reduced while maintaining the multiple Oracle Database 11g R2 GRID Infrastructure with RAC option.
The following are the business needs for Oracle Database 11g R2 GRID Infrastructure with RAC option deployment in a virtualized environment:
• Consolidating multiple Oracle Database 11g R2 GRID Infrastructure with RAC option on the same physical server
• Increasing the DBA's productivity by using NetApp technologies such as NetApp Snapshot™, NetApp FlexClone®, and so on.
• Reducing Oracle licensing costs by consolidating multiple Oracle Database 11g R2 GRID Infrastructure with RAC option on the same physical server.
• Saving costs, power and lab space by reducing the number of physical servers.
• By enabling a global virtualization policy and it is not necessary to have bare metal servers for Oracle Database 11g R2 GRID Infrastructure with RAC option.
• Taking advantage of vSphere management policies.
• A balanced configuration that yields predictable purchasing guidelines at the compute, network and storage tiers for a given workload.
Solution Overview
Oracle Database 11g R2 GRID Infrastructure with RAC option on FlexPod and VMware using Cisco VM-FEX and Oracle Direct NFS Client
This solution provides an end-to-end architecture with Cisco Unified Computing System, VMware, Oracle, and NetApp technologies that demonstrates the implementation of Oracle Database 11g R2 GRID Infrastructure with RAC option on FlexPod and VMware using Cisco VM-FEX (virtual machine fabric extender) and highlights the advantage of using Cisco VICs (Virtual Interface Card) and Oracle Direct NFS Client.
Furthermore this solution demonstrates the scalability and consolidation of multiple Oracle Database 11g R2 GRID Infrastructure with RAC option on FlexPod and VMware using Oracle Direct NFS Client.
Key points of the solution:
• Each virtual machine will have a dedicated Cisco dynamic virtual interface port.
• Each ESXi will boot using iSCSI target.
• All virtual machine traffic is sent directly to the dedicated interface on the switch. · VMware ESXi 5.x as the hypervisor for deploying VMs.
• Cisco software-based Pass Through switch in the hypervisor.
• Hosting multiple Oracle Database 11g R2 GRID Infrastructures with RAC option on same physical server.
• Reducing Oracle licensing costs by consolidating multiple databases on same physical server.
• Saving costs, power and lab space by reducing number of physical servers.
• Allow to implement a consistent virtualization policy across data center by eliminating need to manage bare metal servers running Oracle Databases.
The following are the components used for the design and deployment of Oracle Database 11g R2 GRID Infrastructure with RAC option:
• Oracle Database 11g R2 GRID Infrastructure with RAC option
• Cisco Unified Computing System server platform
• VMware vSphere 5 virtualization platform
• Cisco Passthrough switching for the virtual servers
• Data Center Business Advantage Architecture
• LAN architectures
• NetApp storage components
• NetApp OnCommand® System Manager 2.0
Figure 1. Solution Architecture
Technology Overview
Cisco Unified Computing System
The Cisco Unified Computing System here after called as Cisco UCS is a next-generation data center platform that unites computing, network, storage access, and virtualization into a single cohesive system which is controlled and managed centrally.
The main components of the Cisco UCS are:
• Compute-The system is based on an entirely new class of computing system that incorporates blade servers based on Intel Xeon® E5-2600 Series Processors. The Cisco UCS blade servers offer the patented Cisco Extended Memory Technology to support applications with large datasets and allow more virtual machines per server.
• Network-The system is integrated onto a low-latency, lossless, 80-Gbps unified network fabric. This network foundation consolidates LANs, SANs, and high-performance computing networks which are separate networks today. The unified fabric lowers costs by reducing the number of network adapters, switches, and cables, and by decreasing the power and cooling requirements.
• Virtualization-The system unleashes the full potential of virtualization by enhancing the scalability, performance, and operational control of virtual environments. Cisco security, policy enforcement, and diagnostic features are now extended into virtualized environments to better support changing business and IT requirements.
• Storage access-The system provides consolidated access to both storage area network (SAN) and network-attached storage (NAS) over the unified fabric. By unifying storage access, Cisco UCS can access storage over Ethernet, Fibre Channel, Fibre Channel over Ethernet (FCoE), and iSCSI. This provides customers with the options for setting storage access and investment protection. Additionally, server administrators can preassign storage-access policies for system connectivity to storage resources, thereby simplifying storage connectivity and management for increased productivity.
• Management-The system uniquely integrates all system components which enable the entire solution to be managed as a single entity by the Cisco UCS Manager. The Cisco UCS Manager has an intuitive graphical user interface (GUI), a command-line interface (CLI), and a robust application programming interface (API) to manage all system configuration and operations.
The Cisco UCS is designed to deliver:
• A reduced Total Cost of Ownership (TCO), increased Return on Investment (ROI) and increased business agility.
• Increased IT staff productivity through just-in-time provisioning and mobility support.
• A cohesive, integrated system which unifies the technology in the data center. The system is managed, serviced and tested as a whole.
• Scalability through a design for hundreds of discrete servers and thousands of virtual machines and the capability to scale I/O bandwidth to match demand.
• Industry standards supported by a partner ecosystem of industry leaders.
Figure 2. Cisco UCS Components
Cisco UCS Blade Chassis
The Cisco UCS 5100 Series Blade Server Chassis is a crucial building block of the Cisco Unified Computing System, delivering a scalable and flexible blade server chassis.
The Cisco UCS 5108 Blade Server Chassis is six rack units (6RU) high and can mount in an industry-standard 19-inch rack. A single chassis can house up to eight half-width Cisco UCS B-Series Blade Servers and can accommodate both half-width and full-width blade form factors.
Four single-phase, hot-swappable power supplies are accessible from the front of the chassis. These power supplies are 92 percent efficient and can be configured to support non-redundant, N+ 1 redundant and grid-redundant configurations. The rear of the chassis contains eight hot-swappable fans, four power connectors (one per power supply), and two I/O bays for Cisco UCS 2208 XP Fabric Extenders.
A passive mid-plane provides up to40 Gbps of I/O bandwidth per server slot and up to 80 Gbps of I/O bandwidth for two slots. The chassis is capable of supporting future 80 Gigabit Ethernet standards.
Figure 3. Cisco Blade Server Chassis (Front, Rear and Populated with Blades View)
Cisco UCS B200 M3 Blade Server
The Cisco UCS B200 M3 Blade Server is a half-width, two-socket blade server. The system uses two Intel Xeon® E5-2600 Series Processors, up to 384 GB of DDR3 memory, two optional hot-swappable small form factor (SFF) serial attached SCSI (SAS) disk drives, and two VIC adaptors that provides up to 80 Gbps of I/O throughput. The server balances simplicity, performance, and density for production-level virtualization and other mainstream data center workloads.
Figure 4. Cisco UCS B200 M3 Blade Server
Cisco UCS Virtual Interface Card 1240
A Cisco innovation, the Cisco UCS VIC 1240 is a four-port 10 Gigabit Ethernet, FCoE-capable modular LAN on motherboard (mLOM) designed exclusively for the M3 generation of Cisco UCS B-Series Blade Servers. When used in combination with an optional port expander, the Cisco UCS VIC 1240 capabilities can be expanded to eight ports of 10 Gigabit Ethernet.
Cisco UCS Virtual Interface Card 1280
A Cisco innovation, the Cisco UCS VIC 1280 is an eight-port 10 Gigabit Ethernet, FCoE-capable mezzanine card designed exclusively for Cisco UCS B-Series Blade Servers.
The Cisco UCS VIC 1240 and 1280 enable a policy-based, stateless, agile server infrastructure that can present up to 256 PCI Express (PCIe) standards-compliant interfaces to the host that can be dynamically configured as either NICs or HBAs. In addition, the Cisco UCS VIC 1280 supports Cisco Data Center VM-FEX technology, which extends the Cisco UCS fabric interconnect ports to virtual machines, simplifying server virtualization deployment.
Cisco Data Center VM-FEX Virtualization Support and Virtualization Adapter
With Cisco Data Center VM-FEX, virtual machines have virtual links that allow them to be managed in the same way as physical links. Virtual links can be centrally configured and managed without the complexity of traditional systems, which interpose multiple switching layers in virtualized environments. I/O configurations and network profiles move along with virtual machines, helping increase security and efficiency while reducing complexity. Cisco Data Center VM-FEX helps improve performance and reduce network interface card (NIC) infrastructure.
The Virtual Interface Card provides the first implementation of the Cisco VM-FEX technology. The VM-FEX technology eliminates the virtual switch within the hypervisor by providing individual virtual machine virtual ports on the physical network switch. Virtual machine I/O is sent directly to the upstream physical network switch, in this case, the Cisco UCS 6200 Series Fabric Interconnect, which takes full responsibility for virtual machine switching and policy enforcement.
In a VMware environment, the VIC presents itself as three distinct device types: a Fiber Channel interface, a standard Ethernet interface, and a special Dynamic Ethernet interface. The Fiber Channel and Ethernet interfaces are consumed by standard VMware vmkernel components and provide standard capabilities. The Dynamic Ethernet interfaces are not visible to vmkernel layers and are preserved as raw PCIe devices.
Using the Cisco vDS VMware plug-in and Cisco VM-FEX technology, the VIC provides a solution that is capable of discovering the Dynamic Ethernet interfaces and registering all of them as uplink interfaces for internal consumption of the vDS. The vDS component on each host discovers the number of uplink interfaces that it has and presents a switch to the virtual machines running on the host as shown in the Figure 10. All traffic from an interface on a virtual machine is sent to the corresponding port of the vDS switch. The traffic is mapped immediately to a unique Dynamic Ethernet interface presented by the VIC. This vDS implementation guarantees the 1:1 relationship with a virtual machine interface and an uplink port. The Dynamic Ethernet interface selected is a precise proxy for the virtual machine's interface.
The Dynamic Ethernet interface presented by the VIC has a corresponding virtual port on the upstream network switch.
Figure 5. Each Virtual Machine Interface Showing its Own Virtual Port on a Physical Switch
Cisco UCS Manager running on the Cisco UCS Fabric Interconnect works in conjunction with the VMware vCenter software to coordinate the creation and movement of virtual machines. Port profiles are used to describe the virtual machine interface attributes such as VLAN, port security, rate limiting, and QoS marking. Port profiles are managed and configured by network administrators using Cisco UCS Manager. To facilitate integration with the VMware vCenter, Cisco UCS Manager pushes the catalog of port profiles into VMware vCenter, where they are represented as distinct port groups. This integration allows the virtual machine administrators to simply select from a menu of port profiles as they create virtual machines. When a virtual machine is created or moved to a different host, it communicates its port group to the Virtual Interface Card. The VIC gets the port profile corresponding to the requested profile from the Cisco UCS Manager and the virtual port on the Fabric Interconnect switch is configured according to the attributes defined in the port profile.
Cisco VM-FEX technology addresses the concerns raised by server virtualization and virtual networking and provides the following benefits:
• Unified virtual and physical networking: Cisco VM-FEX technology consolidates the virtual network and physical network into a single switching point that has a single management point. Using Cisco VM-FEX technology, the number of network management points can be reduced by an order of magnitude.
• Consistent performance and feature availability: All traffic is controlled at the physical switch, leading to consistent treatment for all network traffic, virtual or physical. Each virtual machine interface is coupled with a unique interface on the physical switch, which allows precise decisions to be made related to the scheduling of and operations on traffic flow from and to a virtual machine.
• Reduced broadcast domains: The virtual machine's identity and positioning information is known to the physical switch, so the network configuration can be precise and specific to the port in question.
Modes of Operations for VM-FEX technology
Cisco VM-FEX technology supports virtual machine interfaces that run in the following modes:
• Emulated mode
The hypervisor emulates a NIC (also referred to as a back-end emulated device) to replicate the hardware it virtualizes for the guest virtual machine. The emulated device presents descriptors, for read and write, and interrupts to the guest virtual machine just as a real hardware NIC device would. One such NIC device that VMware ESXi emulates is the vmxnet3 device. The guest OS in turn instantiates a device driver for the emulated NIC. All the resources of the emulated devices' host interface are mapped to the address space of the guest OS.
• PCIe Pass-Through or VMDirectPath mode
Virtual Interface Card uses PCIe standards-compliant IOMMU technology from Intel and VMware's VMDriectPath technology to implement PCIe Pass-Through across the hypervisor layer and eliminate the associated I/O overhead. The Pass-Through mode can be requested in the port profile associated with the interface using the "high-performance" attribute
These devices provide a single point for connectivity and management for the entire system. Typically deployed as an active-active pair, the system's fabric interconnects integrate all components into a single, highly-available management domain controlled by Cisco UCS Manager. The fabric interconnects manage all I/O efficiently and securely at a single point, resulting in deterministic I/O latency regardless of a server or virtual machine's topological location in the system.
Cisco UCS 6200 Series Fabric Interconnects support the system's 80-Gbps unified fabric with low-latency, lossless, cut-through switching that supports IP, storage, and management traffic using a single set of cables. The fabric interconnects feature virtual interfaces that terminate both physical and virtual connections equivalently, establishing a virtualization-aware environment in which blade, rack servers, and virtual machines are interconnected using the same mechanisms. The Cisco UCS 6248UP is a 1-RU fabric interconnect that features up to 48 universal ports that can support 80 Gigabit Ethernet, Fibre Channel over Ethernet, or native Fibre Channel connectivity. The Cisco UCS 6296UP packs 96 universal ports into only two rack units.
Figure 6. Cisco UCS 6248UP 20-Port Fabric
UCS 6248UP Rear
UCS 6248UP Front
Cisco UCS 2200 Series Fabric Extenders
These zero-management, low-cost, low-power consuming devices distribute the system's connectivity and management planes into rack and blade chassis to scale the system without complexity. Designed never to lose a packet, Cisco fabric extenders eliminate the need for top-of-rack switches and blade-server-resident Ethernet and Fibre Channel switches and management modules, dramatically reducing infrastructure cost per server.
Cisco UCS 2208XP Fabric Extenders bring the unified fabric and management planes into Cisco UCS 5108 Blade Server Chassis. Typically deployed in pairs, each device brings up to 80 Gbps of bandwidth to the blade server chassis, for a total of up to 160 Gbps across up to eight servers. Each half-width blade has access to up to 80 Gbps of bandwidth.
Figure 7. Cisco UCS 2208 XP
Cisco UCS Manager
Cisco UCS Manager is an embedded, unified manager that provides a single point of management for Cisco UCS. Cisco UCS Manager can be accessed through an intuitive GUI, a command-line interface (CLI), or the comprehensive open XML API. It manages the physical assets of the server and storage and LAN connectivity, and it is designed to simplify the management of virtual network connections through integration with several major hypervisor vendors. It provides IT departments with the flexibility to allow people to manage the system as a whole, or to assign specific management functions to individuals based on their roles as managers of server, storage, or network hardware assets. It simplifies operations by automatically discovering all the components available on the system and enabling a stateless model for resource use.
The elements managed by Cisco UCS Manager include:
• BIOS firmware and settings, including server universal user ID (UUID) and boot order
• Converged network adapter (CNA) firmware and settings, including MAC addresses and worldwide names (WWNs) and SAN boot settings
• Virtual port groups used by virtual machines, using Cisco Data Center VM-FEX technology
• Interconnect configuration, including uplink and downlink definitions, MAC address and WWN pinning, VLANs, VSANs, quality of service (QoS), bandwidth allocations, Cisco Data Center VM-FEX settings, and EtherChannels to upstream LAN switches
Cisco UCS is designed from the start to be programmable and self-integrating. A server's entire hardware stack, ranging from server firmware and settings to network profiles, is configured through model-based management. With Cisco virtual interface cards (VICs), even the number and type of I/O interfaces is programmed dynamically, making every server ready to power any workload at any time.
With model-based management, administrators manipulate a model of a desired system configuration and associate a model's service profile with hardware resources, and the system configures itself to match the model. This automation accelerates provisioning and workload migration with accurate and rapid scalability. The result is increased IT staff productivity, improved compliance, and reduced risk of failures due to inconsistent configurations.
Cisco FEX Technology reduces the number of system components that need to be purchased, configured, managed, and maintained by condensing three network layers into one. It eliminates both blade server and hypervisor-based switches by connecting fabric interconnect ports directly to individual blade servers and virtual machines. Virtual networks are now managed exactly the same way that physical networks are, but enable massive scalability. This approach represents a radical simplification compared to traditional systems, reducing capital expenditures (capex) and operating expenses (opex) while increasing business agility, simplifying and accelerating deployment, and improving performance
UCS Service Profiles
Programmatically Deploying Server Resources
Cisco UCS Manager provides centralized management capabilities, creates a unified management domain, and serves as the central nervous system of the Cisco UCS. Cisco UCS Manager is embedded device management software that manages the system from end-to-end as a single logical entity through an intuitive GUI, CLI, or XML API. Cisco UCS Manager implements role- and policy-based management using service profiles and templates. This construct improves IT productivity and business agility. Now infrastructure can be provisioned in minutes instead of days, shifting IT's focus from maintenance to strategic initiatives.
Dynamic Provisioning with Service Profiles
Cisco UCS resources are abstract in the sense that their identity, I/O configuration, MAC addresses and WWNs, firmware versions, BIOS boot order, and network attributes (including QoS settings, ACLs, pin groups, and threshold policies) all are programmable using a just-in-time deployment model. The manager stores this identity, connectivity, and configuration information in service profiles that reside on the Cisco UCS 6200 Series Fabric Interconnect. A service profile can be applied to any blade server to provision it with the characteristics required to support a specific software stack. A service profile allows server and network definitions to move within the management domain, enabling flexibility in the use of system resources. Service profile templates allow different classes of resources to be defined and applied to a number of resources, each with its own unique identities assigned from predetermined pools.
Service Profiles and Templates
A service profile contains configuration information about the server hardware, interfaces, fabric connectivity, and server and network identity. The Cisco UCS Manager provisions servers utilizing service profiles. The UCS Manager implements a role-based and policy-based management focused on service profiles and templates. A service profile can be applied to any blade server to provision it with the characteristics required to support a specific software stack. A service profile allows server and network definitions to move within the management domain, enabling flexibility in the use of system resources.
Service profile templates are stored in the Cisco UCS 6200 Series Fabric Interconnects for reuse by server, network, and storage administrators. Service profile templates consist of server requirements and the associated LAN and SAN connectivity. Service profile templates allow different classes of resources to be defined and applied to a number of resources, each with its own unique identities assigned from predetermined pools.
The UCS Manager can deploy the service profile on any physical server at any time. When a service profile is deployed to a server, the Cisco UCS Manager automatically configures the server, adapters, Fabric Extenders, and Fabric Interconnects to match the configuration specified in the service profile. A service profile template parameterizes the UIDs that differentiate between server instances.
This automation of device configuration reduces the number of manual steps required to configure servers, Network Interface Cards (NICs), Host Bus Adapters (HBAs), and LAN and SAN switches. The Figure 8 shows the Service profile which contains abstracted server state information, creating an environment to store unique information about a server.
Figure 8. Service Profile
Cisco Nexus 5548UP Switch
The Cisco Nexus 5548UP is a 1RU 1 Gigabit and 10 Gigabit Ethernet switch offering up to 960 gigabits per second throughput and scaling up to 48 ports. It offers 32 1/10 Gigabit Ethernet fixed enhanced Small Form-Factor Pluggable (SFP+) Ethernet/FCoE or 1/2/4/8-Gbps native FC unified ports and three expansion slots. These slots have a combination of Ethernet/FCoE and native FC ports.
Figure 9. Cisco Nexus 5548UP switch
VMware vSphere 5.0 Architecture Overview
VMware ESXi is an enterprise-level computer virtualization solution. VMware ESXi is a production-proven virtualization layer that runs on physical servers that abstract processor, memory, storage, and networking resources to be provisioned to multiple virtual machines.
In the VMware ESXi architecture, shown in Figure 10, the VMware Virtualization Kernel (VMkernel) is augmented by a management partition known as the console operating system or service console. The primary purpose of the console operating system is to provide a management interface with the host. Various VMware management agents are deployed in the console operating system, along with other infrastructure service agents (for example, name service, time service, and logging agents). Furthermore, individual administrative users can log in to the console operating system to run configuration and diagnostic commands and scripts.
Figure 10. VMware ESXi 5.0 Architecture
Virtualization using VMware ESXi provides an abstraction layer that decouples the physical hardware from the operating system to deliver greater IT resource utilization and flexibility. Virtualization allows multiple virtual machines with heterogeneous operating systems (for example, RHEL, Microsoft Windows 2008 Server and SuSE Linux) and applications to run in isolation side by side on the same physical machine. A virtual machine is the representation of a physical machine by software. It has its own set of virtual hardware (RAM, CPU, NICs, hard disks, etc.) on which an operating system and applications are loaded. The operating system sees a consistent, normalized set of hardware regardless of the actual physical hardware components. VMware virtual machines contain advanced hardware features such as 64-bit computing and virtual symmetric multiprocessing. Figure 11 shows server virtualization with VMware ESXi in which virtual machines directly access the network through Cisco Data Center VM-FEX.
Figure 11. VMware ESXi 5.0 with Cisco Data Center VM-FEX
NetApp Storage Technologies and Benefits
Data ONTAP is the fundamental NetApp software platform that runs on all NetApp storage systems. Data ONTAP is a highly optimized, scalable operating system that supports mixed NAS and SAN environments and a range of protocols, including Fibre Channel, iSCSI, FCoE, NFS, and CIFS. The platform includes the Write Anywhere File Layout (WAFL®) file system and storage virtualization capabilities. By leveraging the Data ONTAP platform, the NetApp Unified Storage Architecture offers the flexibility to manage, support, and scale to different business environments by using a common knowledge base and tools. This architecture enables users to collect, distribute, and manage data from all locations and applications at the same time. This allows the investment to scale by standardizing processes, cutting management time, and increasing availability. Figure 12 shows the various NetApp Unified Storage Architecture platforms.
The NetApp storage hardware platform used in this solution is the FAS3270A. The FAS3200 series is an excellent platform for primary and secondary storage for an Oracle Database 11g R2 GRID Infrastructure deployment.
A number of NetApp tools and enhancements are available to augment the storage platform. These tools assist in deployment, backup, recovery, replication, management, and data protection. This solution makes use of a subset of these tools and enhancements.
RAID-DP
RAID-DP® is NetApp's implementation of double-parity RAID 6, which is an extension of NetApp's original Data ONTAP WAFL® RAID 4 design. Unlike other RAID technologies, RAID-DP provides the ability to achieve a higher level of data protection without any performance impact, while consuming a minimal amount of storage. For more information on RAID-DP, see http://www.netapp.com/us/products/platform-os/raid-dp.html.
FlexVol
NetApp FlexVol® technology allows storage to be provisioned just like traditional storage. However, the storage is not consumed until the data is written (just-in-time storage).
NetApp OnCommand System Manager 2.0
System Manager is a powerful management tool for NetApp storage that allows administrators to manage a single NetApp storage system as well as clusters, quickly and easily.
Some of the benefits of the System Manager Tool are:
• Easy to install
• Easy to manage from a Web browser
• Does not require storage expertise
• Increases storage productivity and response time
• Cost effective
• Leverages storage efficiency features such as thin provisioning and compression
Snapshot
NetApp Snapshot technology provides zero-cost, near-instantaneous backup and point-in-time copies of the volume or LUN by preserving the Data ONTAP WAFL consistency points.
Creating Snapshot copies incurs minimal performance effect because data is never moved, as it is with other copy-out technologies. The cost for Snapshot copies is at the rate of block-level changes and not 100% for each backup, as it is with mirror copies. Using Snapshot can result in savings in storage cost for backup and restore purposes and opens up a number of efficient data management possibilities.
NetApp's Strategy for Storage Efficiency
NetApp's strategy for storage efficiency is based on the built-in foundation of storage virtualization and unified storage provided by its core Data ONTAP operating system and the WAFL file system. Unlike its competitors' technologies, NetApp's technologies for its FAS and V-Series product line have storage efficiency built into their core. Customers who already have other vendors' storage systems and disk shelves can still leverage all the storage saving features that come with the NetApp FAS system simply by using the NetApp V-Series product line. This is in alignment with NetApp's philosophy of storage efficiency, because customers can continue to use their existing third-party storage infrastructure and disk shelves, yet save more by leveraging NetApp's storage-efficient technologies.
Oracle Database 11g R2 GRID Infrastructure with the RAC Option
Oracle Database 11g Release 2 provides the foundation for IT to successfully deliver more information with higher quality of service, reduce the risk of change within IT, and make more efficient use of IT budgets.
Oracle Database 11g R2 Enterprise Edition provides industry-leading performance, scalability, security, and reliability on a choice of clustered or single-servers with a wide range of options to meet user needs.
Grid computing relieves users from concerns about where data resides and which computer processes their requests. Users request information or computation and have it delivered - as much as they want, whenever they want. For the DBA, the grid is about resource allocation, information sharing, and high availability. Oracle Database with Real Application Clusters and Oracle Clusterware provide the infrastructure for your database grid. Automatic Storage Management provides the infrastructure for a storage grid. Oracle Enterprise Manager Grid Control provides you with holistic management of your grid.
Oracle Database 11g Direct NFS Client
Direct NFS client is an Oracle developed, integrated, and optimized client that runs in user space rather than within the operating system kernel. This architecture provides for enhanced scalability and performance over traditional NFS v3 clients. Unlike traditional NFS implementations, Oracle supports asynchronous I/O across all operating system environments with Direct NFS client. In addition, performance and scalability are dramatically improved with its automatic link aggregation feature. This allows the client to scale across as many as four individual network pathways with the added benefit of improved resiliency when Network connectivity is occasionally compromised. It also allows Direct NFS client to achieve near block level Performance. For more information on Direct NFS Client comparison to block protocols, see: http://media.netapp.com/documents/tr-3700.pdf.
Design Topology
This section presents physical and logical high-level design considerations for Cisco UCS networking and computing with VMware ESXi virtualization on NetApp storage for Oracle Database 11g R2 GRID Infrastructure with RAc option deployments.
Table1 shows the Hardware and Software used for this experiment.
Table 1. Software and Hardware used for Oracle Database 11g R2 GRID Infrastructure with RAC Option Deployment
Vendor
Name
Version
Description
Cisco
Cisco 6248UP
UCSM 2.0(2r)
Cisco UCS 6200 UP Series Fabric Interconnects
Cisco
Cisco UCS Chassis
5108
Chassis
Cisco
Cisco UCS IOM
2208
IO Module
Cisco
Nexus 5548up Switch
NX-OS
Nexus Switch 5500 series
Cisco
UCS Blade Server
B200 M3
Half width Blade server (Database Server)
Cisco
Cisco UCS VIC Adaptor
1240 and 1280
Virtual Interface Card
Cisco
UCS Blade Server
B200 M2
Half width Blade server for workload
VMware
ESXi 5.0
5.0
Hypervisor
VMware
vCenter Server
5.0
VMware Management
RedHat
RHEL 5.8 64-bit
5.8 64-bit
Operating System
Oracle
Oracle 11g R2 GRID
11.2.0.3
GRID Infrastructure
Oracle
Oracle 11g R2 Database
11.2.0.3
Database
NetApp
FAS 3270 controller
Data ONTAP 8.0.1
NetApp storage controller FC, FCoE, Ethernet
NetApp
DS 4243
600GB, 15k RPM
Shelves
SAS drives
Cisco UCS and iSCSI/NFS Storage Network
This section explains Cisco UCS iSCSI networking and computing design considerations when deploying Oracle Database 11g R2 GRID Infrastructure with RAC option in a VMware ESXi environment. In this design, the iSCSI and NFS traffic is isolated from the regular management and application data network using the same Cisco UCS infrastructure by defining logical VLAN networks to provide better data security. This design also reduces opex and capex compared to a topology in which a separate dedicated physical switch is deployed to handle iSCSI traffic.
Figure 13 presents a detailed view of the physical topology, identifying the various levels of the architecture and some of the main components of Cisco UCS in an iSCSI and NFS network design.
Figure 13. Cisco UCS Component in iSCSI and NFS Network Design
As shown in Figure 13, a pair of Cisco UCS 6248UP fabric interconnects carries both storage and network traffic from the blades with the help of Cisco Nexus 5548UP switch. Both the fabric interconnect and the Cisco Nexus 5548UP switch are clustered with the peer link between them to provide high availability. Six virtual PortChannels (vPCs) are configured to provide public network, private network and storage access paths for the blades to northbound switches. Each vPC has VLANs created for application network data, iSCSI storage data, and management data paths. There is also a dedicated VLAN for VMware vMotion data traffic for VMware ESXi Server.
Oracle Database 11g R2 Data Network and Storage Network vPC Mapping
Cisco Nexus 5548UP vPC configurations with the vPC domains and corresponding vPC names and IDs for Oracle Database Servers is as shown in the Table 2. To provide Layer 2 and Layer 3 switching, a pair of Cisco Nexus 5548UP Switches with upstream switching are deployed, providing high availability in the event of failure to Cisco UCS to handle management, application, and Network storage data traffic. In the Cisco Nexus 5548UP switch topology, a single vPC feature is enabled to provide high availability, faster convergence in the event of a failure, and greater throughput.
Table 2. vPC Mapping
vPC Domain
vPC Name
vPC ID
100
vPC-Public1
101
100
vPC-Public2
102
100
vPC-Private1
103
100
vPC-Private2
104
100
vPC-Storage1
105
100
vPC-Storage2
106
In the vPC design table, a single vPC domain, Domain 100, is created across Cisco Nexus 5548UP member switches to define vPCs to carry specific network traffic. This topology defines six vPCs with IDs 101 through 106. vPC IDs 101,102,103 and 104 are defined for traffic from Cisco UCS fabric interconnects, and vPC IDs 105 and 106 are defined for traffic to NetApp storage. These vPCs are managed within the Cisco Nexus 5548UP, which connects Cisco UCS fabric interconnects and the NetApp storage system.
When configuring the Cisco Nexus 5548UP with vPCs, be sure that the status for all vPCs is "Up" for connected Ethernet ports by running the commands shown in Figure 14 from the CLI on the Cisco Nexus 5548UP Switch.
Figure 14. PortChannel Status on Cisco Nexus 5548UP
Table 3 shows the vPC configuration details for Cisco UCS 6248UP Fabric Interconnects A and B with the required vPC IDs, VLAN IDs, and Ethernet uplink ports for a Oracle Database Server data network design.
Table 3. Fabric Interconnects A and B (Oracle Database Server Data Network)
vPC Name
vPC ID
LAN Uplink Ports
VLAN ID
vPC-Public 1
101
Fabric Interconnect A (Eth 1/17 and 1/18)
108 (management), Public Access, Virtual IP, SCAN IP
192 (iSCSI Boot, NFS Storage)
vPC-Public 2
102
Fabric Interconnect B (Eth 1/17 and 1/18)
108 (management), Public Access, Virtual IP, SCAN IP
192 (iSCSI Boot, NFS Storage)
vPC-Private1
103
Fabric Interconnect A (Eth 1/31 and 1/32)
191(Private Interconnect)
Allows Multicast IP for Private Interconnect
vPC-Private2
104
Fabric Interconnect B (Eth 1/31 and 1/32)
191(Private Interconnect)
Allows Multicast IP for Private Interconnect
On Cisco UCS Fabric Interconnect A, Ethernet uplink ports 17 and 18 are connected to Cisco Nexus 5548UP-1 (port 7) and Cisco Nexus 5548UP-2 (port 7), which are part of vPC ID 101 and have access to VLAN IDs 108 and 192. The same configuration is replicated for vPC ID 102 on Fabric interconnect B, with ports 17 and 18 connected to port 8 of Cisco Nexus 5548UP-1(port 8) and Cisco Nexus 5548UP-2 (port 8). For the private interconnect (Oracle Cluster ware heartbeat), we have configured two vPC ID (103 and 104) using Ethernet uplink port 1/31 and 1/32 of Fabric Interconnect.
After configuring Cisco UCS 6248UP Fabric Interconnects A and B with vPCs, make sure that the status of all the PortChannels are showing "Enabled," as shown in the Cisco UCS Manager screen in Figure 15.
Figure 15. Uplink Interfaces and PortChannel Status on Fabric Interconnect
On the Cisco Nexus 5548UP Switch, a separate vPC is created to access NetApp shared storage for iSCSI boot as well as NFS data access. The vPC is created with the vPC name and corresponding vPC ID and required VLAN IDs, as shown in Table 4.
Table 4. vPC on Nexus 5548UP for NetApp Storage Access
vPC Name
iSCSI Ports (Controllers A and B)
vPC ID
VLAN ID
vPC-Storage 1
e1b and e1c (Controller A)
105
192
vPC-Storage 2
e1b and e1c (Controller B)
106
192
On NetApp Storage Controller A, Ethernet 10-Gbps port e1b is connected to Cisco Nexus 5548UP-1 (port 1), and Ethernet port e1c is connected to Cisco Nexus 5548UP-2 (port 1), which are part of vPC-Storage1 with vPC ID 105 that allows traffic from VLAN ID 192. On NetApp Storage Controller B, Ethernet 10-Gbps port e1b is connected to Cisco Nexus 5548UP-1 (port 2), and Ethernet port e1c is connected to Cisco Nexus 5548UP-2 (port 2), which are part of vPC-Storage2 with vPC ID 106 that allows traffic from VLAN ID 192.
Cisco UCS Manager LAN Pin Group
Cisco UCS uses LAN pin groups to pin Ethernet traffic from a vNIC on a server to an uplink Ethernet port or port channel on the fabric interconnect. You can use this pinning to manage the distribution of traffic from the servers.
To configure pinning for a server, you must include the LAN pin group in a vNIC policy. The vNIC policy is then included in the service profile assigned to that server. All traffic from the vNIC travels through the I/O module to the specified uplink Ethernet port.
Following steps needs to perform from the Cisco UCS Manager to create LAN Pin Group.
Step 1. In the Navigation pane, click the LAN tab.
Step 2. On the LAN tab, expand LAN > LAN Cloud.
Step 3. Right-click LAN Pin Groups and select Create LAN Pin Group.
Step 4. In the Create LAN Pin Group dialog box, enter a unique name and description for the pin group.
Step 5. To pin traffic for fabric interconnect A, do the following in the Targets area:
a. Check the Fabric Interconnect A check box.
b. Click the drop-down arrow on the Interface field and navigate through the tree-style browser to select the port or port channel you want to associate with the pin group.
Step 6. To pin traffic for fabric interconnect B, do the following in the Targets area:
a. Check the Fabric Interconnect B check box.
b. Click the drop-down arrow on the Interface field and navigate through the tree-style browser to select the port or port channel you want to associate with the pin group.
Step 7. Click OK.
Figure 16 shows the LAN Pin Groups created in this experiment.
Figure 16. Cisco UCS Manager LAN PIN Group
Cisco UCS Manager Quality-of-Service System and Policy
Cisco UCS uses IEEE Data Center Bridging (DCB) to handle all traffic within Cisco UCS. This industry-standard enhancement to Ethernet divides the bandwidth of the Ethernet pipe into eight virtual lanes. System classes determine how the DCB bandwidth in these virtual lanes is allocated across the entire Cisco UCS platform.
Each system class reserves a specific segment of the bandwidth for a specific type of traffic, providing an assured level of traffic management even in an oversubscribed system. For example, you can configure the Fibre Channel Priority system class to determine the percentage of DCB bandwidth allocated to FCoE traffic.
Table 5 describes the system classes.
Table 5. System Classes
System Class
Description
• Platinum Priority
• Gold Priority
• Silver Priority
• Bronze Priority
These classes set the quality of service (QoS) for all servers that include one of these system classes in the QoS definition in the service profile associated with the server. Each of these system classes manages one lane of traffic. All properties of these system classes are available for you to assign custom settings and policies.
Best-Effort Priority
This class sets the QoS for the lane that is reserved for basic Ethernet traffic. Some properties of this system class are preset and cannot be modified. For example, this class has a drop policy to allow it to drop data packets if required.
Fibre Channel Priority
This class sets the QoS for the lane that is reserved for FCoE traffic. Some properties of this system class are preset and cannot be modified. For example, this class has a no-drop policy to help ensure that it never drops data packets.
QoS policies assign a system class to the outgoing traffic for a vNIC or virtual HBA (vHBA). You must include a QoS policy in a vNIC policy and then include that policy in a service profile to configure the vNIC.
To provide efficient network utilization and bandwidth control in a Oracle Database 11g R2 GRID Infrastructure with RAC option environment on VMware ESXi over a NFS network, QoS system classes and corresponding policies are defined for network traffic generated by NFS storage, VMware vMotion, Oracle Database 11g R2 GRID and the guest virtual machine management network in Cisco UCS are explained as follows:
• Oracle Clusterware heartbeat requires high bandwidth and a fast response for cache fusion and to flow interconnect traffic. To meet this requirement, a RAC_HB QoS policy is created and defined with the Platinum class with the highest weight (bandwidth) and a maximum transmission unit (MTU) of 9000.
• NFS storage traffic requires reasonable bandwidth and a fast response time to access Oracle database files (datafiles and redolog files) stored in the shared storage. To meet this requirement, an OracleDB QoS policy is created with the Gold class with the second highest weight (bandwidth) and an MTU of 9000.
• iSCSI boot for ESXi kernel requires dedicated network bandwidth to boot the ESXi host from the NetApp storage. To meet this requirement, iSCSI_Jumbo QoS policy is created and is defined with the Silver class and with the third highest weight (bandwidth) and an MTU of 9000 to handle jumbo VMkernel packets from vNICs (static) in the service profiles in which the VMware ESXi host is installed, which is a part of the VMware ESXi host-based iSCSI environment.
• To handle VMware ESXi host and guest virtual machine network traffic for management and operations that have lower bandwidth requirements, the Best-Effort QoS class with the least weight (bandwidth) is defined on Cisco UCS.
Note: To apply QoS across the entire system, from Cisco UCS to the upstream switches (Cisco Nexus 5548UP Switches), you need to configure similar QoS class and policy types with the right class-of-service (CoS) values that match the Cisco UCS QoS classes.
Table 6 shows each QoS policy name with the corresponding priority, weight, and MTU value. These values are applied to static and dynamic vNICs in the Oracle Database 11g R2 GRID Infrastructure deployment environment.
In this experiment we used different QoS policy as mentioned in table 6. However default policy with "Best Effort" priority can be used for deploying Oracle RAC on FlexPod.
Figure 17 shows Cisco UCS QoS system class and QoS policy configurations defined for application on static and dynamic vNICs for accessing an Oracle Database Server.
Figure 17. Cisco UCS QoS System Class and QoS Policy Configuration Window
Figure18 shows how the class priorities are applied to the named QoS policies in Cisco UCS Manager.
Figure 18. Cisco UCS QoS System Class and QoS Policy Configuration Window
Table 7 shows Cisco UCS and Cisco Nexus 5548UP switch QoS mapping, with Cisco UCS QoS policy configuration values matched with Cisco Nexus 5548UP switch QoS policy values to achieve end-to-end QoS.
On the Cisco Nexus 5548UP switch, a single policy type map is defined with multiple class types, with Cisco UCS QoS matching configuration values that are applied on the global system level.
Table 7. Cisco UCS and Cisco Nexus 5548UP QoS Mapping
This section discusses the NetApp storage layout design considerations when deploying an Oracle Database 11g R2 GRID Infrastructure with RAC option on a VMware ESXi hypervisor on Cisco UCS in an NFS environment.
Figure 19 shows a high-level storage design overview of a NetApp FAS3270 HA storage system.
Figure 19. Design Overview of a NetApp High-Availability Storage
The NetApp aggregation layer provides a large virtualized pool of storage capacity and disk IOPS to be used on demand by all the virtual machines hosted on the aggregation layer. The aggregation-layer sizing is based on the storage requirements for Oracle Database to meet the storage capacity, performance, and Snapshot backup requirements of the assumed typical OLTP workload. When sizing your environment, you need to perform the necessary planning to determine the exact storage configuration to meet your individual requirements. Aggregation layer 0 (Aggr0) is defined for hosting the root NetApp flexible volumes (FlexVol volumes), which use the NetApp Data ONTAP operating system to handle NetApp storage configurations. For detailed NetApp storage command options, see.http://support.netapp.com/documentation/docweb/index.html?productID=40378
Table 8 shows the NetApp storage layout with volumes and LUNs created for various purposes.
Table 8. NetApp Storage Layout with Volumes and LUNs
NetApp Storage Layout
Aggregation and NetApp Controller
NetApp FlexVol Volume
Flexible LUN
Comments
Aggr1 on Controller A
Boot_OS_VOL
ESXi_OS_LUN
iSCSI boot LUN for VMware ESXi host for node 1 and node 2 of the failover cluster with Cisco UCS B200M3 blade server
Aggr1 on Controller A
OCR_VOTE_VOL
Used to store OCR and voting disk using NFS
Aggr1 on Controller A
DB_VOL
Used to store datafiles, spfiles, and the copy of control files.
Aggr1 on Controller A
LOG_VOL
Used to store redo log files and copy of control files.
Aggr1 on Controller B
Boot_OS_VOL
ESXi_OS_LUN
iSCSI boot LUN for VMware ESXi host for node 3 and node 4 of the failover cluster with Cisco UCS B200M3 blade server
Aggr1 on Controller B
OCR_VOTE_VOL
Used to store OCR and voting disk using NFS
Aggr1 on Controller B
DB_VOL
Used to store datafiles, spfiles, and the copy of control files.
Aggr1 on Controller B
LOG_VOL
Used to store redo log files and the copy of control files.
Use the following commands to configure the NetApp storage systems to implement the storage layout design described here.
NetApp FAS3270HA (Controller A)
1. Create Aggr1 with a RAID group size of 16, 55 disks, and RAID_DP redundancy for hosting NetApp FlexVol volumes and LUNs, as shown in Table 8.
2. Create NetApp FlexVol volumes on Aggr1 for hosting iSCSI LUNs and database volumes, as shown in Table 8. These volumes are exposed to VMware ESXi host and guest virtual machines.
FAS3270HA-Controller A> vol create Boot_OS_VOL aggr1 200g
FAS3270HA-Controller A> vol DB_VOL aggr1 1024g
FAS3270HA-Controller A> vol create LOG_VOL aggr1 500g
FAS3270HA-Controller A> vol create OCR_VOTE_VOL aggr1 10g
3. Repeat steps 1 and 2 on Controller B after the creation of FlexVol volumes on Controller A.
4. Create LUNs on NetApp FlexVol volumes for iSCSI boot of ESXi host.
FAS3270HA-Controller A> lun create -s 150g -t vmware /vol/Boot_OS_VOL/ESXi_OS_LUN
5. Repeat step 4 on Controller B after the creation of the LUN on Controller A.
6. Create an initiator group (igroup) for mapping the VMware ESXi host boot LUN. FAS3270HA-Controller A> igroup create -I -t vmware iSCSI-ESXi -Boot iqn.2012-01.com.vmware:ESXi
7. Repeat step 6 on Controller B after the creation of the igroup on Controller A.
8. Map the LUNs to specific igroups, to access the VMware ESXi host boot.
9. Repeat step 8 on Controller B once you complete it on Controller A.
NFS exports all the flexible volumes (data volumes, redo log volumes, and OCR and voting disk volumes) from both Controller A and Controller B, providing read/write access to the root user of all hosts created in the previous steps.
NetApp Multimode Virtual Interfaces
The NetApp multimode virtual interface (VIF) feature is enabled on NetApp storage systems on 10 Gigabit Ethernet ports (e1a and e1b) for configuring the iSCSI target through which OS boot LUNs are exposed over the iSCSI protocol to host iSCSI initiators (VMware ESXi host and guest virtual machines).
Also, in this experiment we used the same VIF to access all flexible volumes created to store Oracle Database files that use using the NFS protocol.
The following are the NetApp CLI commands for configuring the multilevel dynamic VIF on NetApp FAS3270HA (Controllers A and B) cluster storage systems.
NetApp FAS3270HA (Controller A)
FAS3270HA-Controller A> iscsi start
FAS3270HA-Controller A > ifgrp create lacp iscsiA
FAS3270HA-Controller A > ifgrp add iscsiA e1a e1b
FAS3270HA-Controller A > ifconfig iscsiA mtusize 9000 192.191.1.2 netmask 255.255.255.0 partner iscsiB up
NetApp FAS3270HA (Controller B)
FAS3270HA-Controller B> iscsi start
FAS3270HA-Controller B > ifgrp create lacp iscsiA
FAS3270HA-Controller B > ifgrp add iscsiA e1a e1b
FAS3270HA-Controller B > ifconfig iscsiB mtusize 9000 192.191.1.4 netmask 255.255.255.0 partner iscsiA up
Make sure that the MTU is set to 9000 and that jumbo frames are enabled on the Cisco UCS static and dynamic vNICs and on the upstream Cisco Nexus 5548UP switches.
Figure 20 shows the virtual interface "iSCSI-A" created with the MTU size set to 9000 and the trunk mode set to multiple, using two 10 Gigabit Ethernet ports (e1a and e1b) on NetApp storage Controller A. Repeat the creation the virtual interface on NetApp Controller B using the settings shown in Figure 20.
Figure 20. Virtual Interface (VIF) on NetApp Storage
The following commands show how to configure the class of service (CoS) on Nexus 5548 for untagged packets originating from NetApp storage on the port channels.
This section describes the Cisco UCS service profile design for deploying the VMware ESXi host OS booting from the NetApp shared iSCSI target on the Cisco UCS B-Series server. In this deployment, the Cisco UCS VIC 1280 adaptor is used for iSCSI boot of the VMware ESXi OS from the NetApp iSCSI target.
1. Create service profiles, for example (Chassis1_Slot5_B200M3, Chassis1_Slot6_B200M3, Chassis1_Slot7_B200M3 and Chassis1_Slot8_B200M3), then associate them with Cisco B200M3 blades having the Cisco UCS VIC 1280 adaptor to install VMware ESXi 5.0 from the iSCSI target on NetApp FAS3270. Figure 21 shows the list of newly created service profiles.
Figure 21. Created Service Profiles
2. On the Service Profiles tab for each newly created service profile, create two more static vNICs, iSCSI1 and iSCSI2, on Fabric A and Fabric B, respectively, with the MTU value set to 9000, and network VLAN access to VLAN ID 192 (iSCSI storage), apart from vNIC (Public and Public2) as shown in Figure 22.
Figure 22. Created Static vNICs on Fabric Interconnects
3. Static vNICs (Public and Public2) would be used to access VMware ESXi for management purpose with the appropriate vLAN ID 108, whereas static vNICs (iSCSI1 and iSCSI2) would be used to do iSCSI boot of ESXi OS from NetApp Storage with the appropriate VLAN ID 192. Static vNICs (Public and Public2) will provide uplinks to the VMware ESXi vSwitch and Cisco Data Center VM-FEX DVS.
4. On the desired service profile, create two iSCSI vNICs, iscsi_Boot_1 and iscsi_Boot_2, which are required to access the NetApp storage iSCSI target during booting process to load the VMware ESXi operating system over the iSCSI network. Make sure that the iSCSI vNIC, iscsi_Boot_1 is overlaid on static vNIC iSCSI1, and that iscsi_Boot_2 is overlaid on static vNIC iSCSI2, as shown in Figure 23.
Figure 23. iSCSI vNICs Overlaid on Static vNICs
For the Cisco UCS VIC 1280 adaptor, make sure that the MAC address is marked "Derived" and that the correct VLAN ID (192) is selected to access the NetApp iSCSI target during VMware ESXi iSCSI bootup.
5. In Cisco UCS Manager, create a new iSCSI boot policy, Boot_iSCSI, with iSCSI vNIC iscsi_Boot_1 as a primary path and iscsi_Boot_2 as a secondary path to provide redundancy for VMware ESXi host iSCSI boot in case of software or hardware faults. Figure 24 shows the boot policy configuration.
Figure 24. New iSCSI Boot Policy in Cisco UCS Manager
6. After the iSCSI Boot policy is created, choose a newly created boot order policy for the desired service profile. For the chosen service profile on the Cisco UCS Manager Boot Order tab, assign iscsi_Boot_1 as the primary iSCSI vNIC and iscsi_Boot_2 as the secondary iSCSI vNIC with VMware ESXi iSCSI boot parameters as shown in Table 9 and Figure 25.
Table 9. iSCSI Boot Parameters
iSCSI vNIC Name
iSCSI Initiator iSCSI Qualified Name (IQN)
Initiator IP Address Policy
Initiator IP Address
iSCSI Target IQN
iSCSI Port
iSCSI Target IP Address
LUN ID
Iscsi_Boot_1
iqn.2012-01.com.vmware.ESX5i
Static
192.191.1.20
iqn.1992-08.com.netapp.sn:1574126331
3260
192.191.1.2
1
Iscsi_Boot_2
iqn.2012-01.com.vmware.ESX5i
Static
192.191.1.21
iqn.1992-08.com.netapp.sn:1574126331
3260
192.191.1.2
1
Figure 25. Setting iSCSI Boot Parameters
7. Associate the service profile with the desired blade (Cisco UCS B200M3 in this case). On Cisco UCS Manager in the associated service profile, launch the keyboard, video, and mouse (KVM) console. Through the virtual media interface, click on add image to map the VMware ESXi 5.0 ISO image from your staging area as shown in Figure 26.
Figure 26. KVM Console and Mapping Virtual Media
8. Click Reset to boot the server and install the operating system on the NetApp iSCSI boot LUN exposed over the iSCSI network, as shown in Figure 27.
Figure 27. NetApp iSCSI LUN Exposed During Booting Up Server
9. After the completion of the VMware ESXi 5.0 OS installation and VMware ESXi 5.0 boot, on the VMware ESXi console press the F2 key to configure the management network. Under the Network Adapters option, select the Ethernet port vmnic1 and vmnic2 as it mapped with static vNICs Public and Public2 (compare the mac address of static vNIC Public and Public2 in Cisco UCS Manager service profile) as uplinks for the default VMware ESXi vSwitch named vSwitch0 as shown in Figure 28.
Figure 28. Configuring Network Management
10. Under the IP Configuration option, enter the management IP address which is on VLAN 108 associated with the VMkernel management port group. Note that by default the IP address is set to the iSCSI vNIC IP address (VLAN ID 192). Figure 29 shows the management IP address configuration details.
Figure 29. Management IP Configuration Details
With these steps, VMware ESXi 5.0 installation is completed with the iSCSI boot LUN configured on the NetApp FAS3270.
Cisco Data Center VM-FEX Distributed Virtual Switch (DVS) Configuration
This section describes the configuration of Cisco Data Center VM-FEX DVS and the benefits of using it in a guest VM for the Oracle Database 11g R2 GRID Infrastructure with RAC option deployment and database access from NetApp storage using the NFS protocol.
Cisco Data Center VM-FEX is a software DVS that can be used in the VMware ESXi environment to provide better visibility and manageability and allow hypervisor VMware VMDirectPath I/O as well as PassThrough I/O, which provides wire-speed 10-Gbps capability to the guest virtual machine while running I/O-intensive applications.
Cisco Data Center VM-FEX significantly reduces the number of network management points, enabling both physical and virtual network traffic to be treated in a consistent policy-based way.
The Cisco Data Center VM-FEX software extends the Cisco fabric extender technology to the virtual machine with the following capabilities:
• Each virtual machine has a dedicated interface on the parent switch.
• All virtual machine traffic is sent directly to the dedicated interface on the switch.
• The software-based switch in the hypervisor is bypassed or pass through.
The following section provides a high-level overview of Ethernet network infrastructure configuration for a Oracle Database 11g R2 GRID Infrastructure deployment on Cisco UCS, the VMware ESXi 5.0 hypervisor, and a RHEL 5.8 64-bit guest virtual machine.
Figure 30 shows the physical and logical architecture of the Cisco UCS Manager virtual machine-specific configuration and VMware ESXi and Cisco VM-FEX DVS configuration to deploy Oracle 11g R2 GRID Infrastructure on RHEL 5.8 64-bit guest virtual machine.
Figure 30. Physical and Logical Architecture of Cisco UCS Manager
Cisco UCS Manager Virtual Machine Port Profile Design
This section describes the Cisco Data Center VM-FEX port profile in Cisco UCS manager that is required to deploy an Oracle 11g R2 GRID Infrastructure network layout on the RHEL 5.8 64-bit guest virtual machine accessing database (datafiles, redo logfiles, control files, spfiles, OCR and voting disk) stored on NetApp storage using the NFS protocol.
The Cisco UCS Manager port profile for Cisco Data Center VM-FEX provides network properties and settings (VLAN ID, QoS, VMware VMDirectPath, and so on) to apply on the Cisco UCS dynamic vNIC VIFs that are exposed to the VMware ESXi hypervisor through the VMware vCenter server. These dynamic vNICs are attached to the guest virtual machine (RHEL5.8) running ESXi 5.0 to access the NetApp NFS storage.
The following steps describe the Cisco Data Center VM-FEX port profile design process in Cisco UCS Manager on the VM tab:
1. To manage and configure the both VMware ESXi host and guest virtual machines, define the Cisco UCS virtual machine port profile Public with VLAN ID 108, a 64-port maximum, and a QoS policy with the Best Effort class for management traffic on dynamic vNICs assigned to the guest virtual machine.
2. Define the port profile Private for dynamic vNICs through which Oracle GRID nodes communicate each other also this network is used for cache fusion hosted on the guest virtual machine. This port profile is configured with VLAN ID 191, a 64-port maximum, and a QoS policy of RAC-HB with the Platinum class.
3. Oracle Database is accessed by using the NFS protocol. To provide traffic isolation for better security and bandwidth, define the port profile Storage with VLAN ID as 192 and a QoS policy of OracleDB with the Gold class.
Table 10 provides the Cisco Data Center VM-FEX port profile Cisco UCS design VLAN ID, QoS policy, maximum port count.
Table 10. Cisco Data Center VM-FEX Port Profile Properties in Cisco UCS Manager
Cisco UCS: Cisco Data Center VM-FEX Port Profile
Port-Profile Properties
Public
QoS policy: Public
Network control policy: Default
Maximum ports: 64
VLAN ID: 108
Private
QoS policy: RAC-HB
Network control policy: Default
Maximum ports: 64
VLAN ID: 191
Storage
QoS policy: OracleDB
Network control policy: Default
Maximum ports: 64
VLAN ID: 192
Figure 31 shows the newly created Cisco Data Center VM-FEX port profiles on the Cisco UCS Manager
Figure 31. Cisco Data Center VM-FEX Port Profiles
Details of the properties assigned to each of the newly created port profiles can be verified by selecting Port Profiles on the Cisco UCS Manger VM tab, as shown in Figure 32.
Figure 32. Port Profile Properties in Cisco UCS Manager
Cisco UCS Dynamic vNICs in Service Profile
This section explains dynamic vNIC network design with the Cisco UCS service profile to deploy a Oracle Database 11g R2 GRID Infrastructure in guest virtual machine running RHEL 5.8 64-bit with VMware ESXi 5.0 host using a Cisco Data Center VM-FEX DVS. A goal of this design is to achieve high I/O throughput and high availability.
Configure the service profile created in previous steps (Chassis1_Slot5_B200M3, Chassis1_Slot6_B200M3, Chassis1_Slot7_B200M3 and Chassis1_Slot8_B200M3), with the dynamic vNIC connection policy with a required number of vNICs, which are exposed to the VMware ESXi host to connect management VMware VMkernel network adapters (vmnic1 and vmnic2 are part of the VMware ESXi vSwitch0). However, these vNICs will be migrated later to the Cisco Data Center VM-FEX DVS.
Figure 33 shows the configuration details for static and dynamic vNICs created for the specific service profiles.
Figure 33. Configuration Details for Static and Dynamic vNICs
Table 11 shows the properties of static and dynamic vNICs created for the service profile for deploying Oracle Database 11g R2 GRID Infrastructure on guest virtual machine RHEL 5.8 on VMware ESXi 5.0 in a Cisco Data Center VM-FEX environment.
Table 11. Cisco UCS Manager Service Profile's Network Design
Figure 34 shows VMware ESXi vSwitch configuration after the migration is complete.
The two uplink ports, vmnic4 and vminc6, of the iSCSI Boot port group of iScsiBootPG switch should be left undisturbed. Altering these settings can affect VMware ESXi bootup through the iSCSI LUNs.
Following steps are used to create the Guest VMs to deploy Oracle 11g R2 GRID Infrastructure.
1. Login to VMware vCenter Server and select the ESXi server.
2. Click on the "Create a New virtual machine" to create guest VM.
3. Select appropriate number of vCPU and size of Memory for the Guest VM.
Make sure to add four network adaptors in each Guest VM (one for public access, two for private interconnect, and one for storage access through NFS). Also make sure to select appropriate VM-FEX DVS switch for each individual network adaptor.
Figure 35 shows the guest virtual machine with the network adapter settings as explained in the above steps.
Install the guest operating system RHEL 5.8 64-bit with all required rpms for Oracle Database 11g R2 GRID Infrastructure in each Guest VM. Also provide the IP address for each network adaptor in each Guest VM with appropriate setting of MTU size (for Public Access MTU is set to 1500 and for both private interconnect and storage access MTU is set to 9000) as shown in Figure 36.
Figure 36. Configuring Each Network Interface on the Guest Virtual Machine
To support end-to-end jumbo frame (MTU 9000) to carry NFS traffic from the RHEL 5.8 guest virtual machine, Cisco UCS, Cisco Data Center VM-FEX, and NetApp storage, perform the following steps:
b. Configure MTU 9000 in the Jumbo field on the appropriate network interfaces in the guest virtual machine.
c. Configure NetApp VIFs to enable the MTU 9000 value.
d. Configure the Policy-map in Nexus 5548UP and set the system jumbo mtu size to 9000
Complete the following steps in order to configure the Policy-map in Nexus 5548UP:
switch(config)#policy-map type network-qos jumbo
switch(config-pmap-nq)#class type network-qos class-default
switch(config-pmap-c-nq)#mtu 9216
switch(config-pmap-c-nq)#exit
switch(config-pmap-nq)#exit
switch(config)#system qos
switch(config-sys-qos)#service-policy type network-qos jumbo
VMware ESXi vCenter provides a single pane to view all the guest virtual machine dynamic vNICs that are part of the Cisco Data Center VM-FEX DVS. The pane also shows the port ID, link connectivity status, and Cisco UCS port profile information applied, with the MAC address, MTU, and IP address configured in the RHEL 5.8 64-bit guest virtual machine, as shown in Figure 37.
Figure 37. Link Connectivity Status and Cisco UCS Port Profile Information with MAC Address
Oracle Database 11g R2 GRID infrastructure with RAC option Deployment
This section describes the deployment of Oracle Database 11g R2 GRID Infrastructure with RAC option in a virtualized environment. After installation of guest Operating system RHEL 5.8 64-bit in each RAC node, verify all the required RPMs are installed as part of guest OS installation which is required for Oracle GRID Installation.
Use the following Oracle document for pre-installation tasks, such as setting up the kernel parameters, RPM packages, user creation, etc.
Follow the steps to complete the Oracle Database 11g R2 GRID Infrastructure with RAC option Installation. For this experiment we have created the user, group, directory structure, kernel parameters and user limits as shown through the steps below. You can resize the kernel parameter, user limits and rename the directory structure, user name, groups as per your business requirement.
Step 1. Create required oracle users and groups in each RAC nodes.
groupadd -g 1000 oinstall
groupadd -g 1200 dba
useradd -u 2000 -g oinstall -G dba grid
passwd grid
useradd -u 1100 -g oinstall -G dba oracle
passwd oracle
Step 2. Create local directories on each RAC nodes and give the ownership of these directories to newly created users in Step 1.
"grid" user owns the Oracle GRID installation whereas "oracle" user owns the Oracle Database Installation. In this experiment we used local directory for GRID Installation and Database binary Installation. However these binaries can be installed in shared directory on NFS volumes.
Following table 12 shows the mapping of NFS Volumes with newly created directory in each RAC node
Table 12. Table 12. Local Mount Points and NetApp NFS Volumes
Local Directory on Guest OS
NetApp NFS Volumes
Owner
Purpose
/u01/app/11.2.0/grid
NA
grid
Oracle GRID binary installation
/u01/app/oracle
NA
oracle
Oracle Database binary installation
/data_A
/vol/RAC1_DB
oracle
Datafiles and control files
/data_B
/vol/RAC1_DB_B
oracle
Datafiles and control files
/log_A
/vol/RAC1_LOG
oracle
Redo log files and control files
/log_B
/vol/RAC1_LOG_B
oracle
Redo log files and control files
/ocrvote
/vol/OCR_VOTE1
grid
OCR and voting disks
Step 3. Edit /etc/fstab file in each RAC node and add the entry for all volumes and its corresponding local directories created in the above steps with the proper mount options as shown in Figure 38.
Figure 38. Shows the mount options used to mount NFS volumes
Note: Note An rsize and wsize of 65536 is supported by NFS v3 and used in this configuration to improve performance.
Step 4. Mount all the local directories created to store database, ocr file and voting disks, once you complete the editing of /etc/fstab in each RAC node using root user.
node1# mount /ocrvote
node1# mount /data_A
node1# mount /data_B
node1# mount /log_A
node1# mount /log_B
Give ownership of mounted directories to appropriate user.
Step 5. Configure the private and public NICs with the appropriate IP addresses.
Step 6. Identify the virtual IP addresses and SCAN IPs and have them setup in DNS per Oracle's recommendation. Alternatively, you can update the /etc/hosts file with all the details (private, public, SCAN and virtual IP) if you do not have DNS services available
Step 7. Create raw devices under /ocrvote local directories as follows.
Login as "grid" user from any one node and create the following raw files
Step 8. Configure the ssh option (with no password) for the Oracle user and grid user.
For more information about ssh configuration, refer to the Oracle installation documentation.
Step 9. Configure the "/etc/sysctl.conf" file by adding shared memory and semaphore parameters required for Oracle GRID Installation. Also configure "/etc/security/limits.conf" file by adding user limits for oracle and grid users.
You are now ready to install Oracle Database 11g R2 GRID Infrastructure with RAC option and the database.
Installing Oracle Database 11g R2 GRID Infrastructure with RAC option and the Database
It is not within the scope of this document to include the specifics of an Oracle RAC installation; you should refer to the Oracle installation documentation for specific installation instructions for your environment.
To install Oracle, follow these steps:
Step 1. Download the Oracle Database 11g Release 2 Grid Infrastructure (11.2.0.3.0) and Oracle Database 11g Release 2 (11.2.0.3.0) for Linux x86-64.
During the grid installation it will ask to place cluster registry (OCR) files and voting disks files. Select the option shown in Figure 39 and click next.
Figure 39. Oracle ASM to Store OCR and Voting Disks
Now you can provide disk group name and change discovery path as "/ocrvote/* "would discover all the raw devices created in earlier steps, as shown in Figure 40.
Figure 40. ASM Disk Group and ASM Disks
Step 3. Install Oracle Database 11g Release 2 Database "Software Only"; do not create the database after Oracle GRID Installation as oracle user. See Real Application Clusters Installation Guide for Linux and UNIX for detailed installation instructions.
Step 4. Run the dbca tool as oracle user to create the database. Make sure to place the datafiles, redo logs and control files in proper directory/ paths as created in above steps.
Step 5. Configure Direct NFS client.
For improved NFS performance, Oracle recommends using the Direct NFS Client shipped with Oracle 11g. The direct NFS client looks for NFS details in the following locations:
1. $ORACLE_HOME/dbs/oranfstab
2. /etc/oranfstab
3. /etc/mtab
Since the NFS mount point details were defined in the "/etc/fstab", and therefore the "/etc/mtab" file also, there is no need to configure any extra connection details. When setting up your NFS mounts, reference the Oracle documentation for guidance on what types of data can/cannot be accessed via Direct NFS Client. For the client to work we need to switch the libodm11.so library for the libnfsodm11.so library, as shown below.
srvctl stop database -d rac1db
cd $ORACLE_HOME/lib
mv libodm11.so libodm11.so_stub
ln -s libnfsodm11.so libodm11.so
srvctl start database -d rac1db
With the configuration complete, you can see the direct NFS client usage via the following views:
Note: Note The Direct NFS Client supports direct I/O and asynchronous I/O by default.
Workloads and Database Configuration
A typical OLTP workload generated by benchmark factory for both the configurations as mentioned in design validation of purpose of document session.
Benchmark Factory is a database performance and code scalability testing tool that simulates users and transactions on the database and replays production workload in non-production environments. A benchmark is a performance test of hardware and/or software on a system-under-test. Benchmark Factory provides the option of using industry standard benchmarks during the load testing process. Benchmarks measure system peak performance when performing typical operations.
The two scenarios tested in this experiment are:
1. Deployment of four node Oracle Database 11g R2 GRID Infrastructure with RAC option in a virtualized environment.
2. Scaling and consolidation study of a two-node Oracle Database 11g R2 GRID Infrastructure with RAC option in a virtualized environment.
Deployment of four node Oracle Database 11g R2 GRID Infrastructure with RAC option
In this experiment we concentrate on implementation of four node Oracle Database 11g R2 GRID with RAC option on FlexPod using VM-FEX and Cisco VICs. We used four fully loaded B200 M3 (Cisco Blade servers) and created one guest VM on each blade to setup four node Oracle Database 11g R2 with RAC option. Each ESXi OS is iSCSI boot from a NetApp Storage and NFS protocol is used to access data volumes and guest OS volume from NetApp Storage. RHEL 5.8 64-bit Operating System is installed in each guest VM.
Following table 13 shows the configuration of each Guest VM.
Table 13. Guest VM Configuration
Guest VM Component
Details
Description
CPU
16 vCPU
16 dynamic virtual CPU
Memory
128 GB
Physical memory
NIC1
Public Access
MTU Size 1500, for Management and Public Access
NIC2
Private Interconnect1
Private Interconnect configured for HAIP, MTU SIZE 9000
NIC3
Private Interconnect2
Private Interconnect configured for HAIP, MTU Size 9000
NIC4
Storage access
Database access through NFS, MTU size 9000
SWAP Space
64 GB
Swap Space
Following Figure 41 shows the VM details in each Physical blade server and Oracle Cluster setup.
Figure 41. Guest VM in Each Blade Server
Oracle Database Binary and Oracle Database 11g R2 GRID Infrastructure with RAC option are installed in the local storage of the each guest VM; however, these binaries can be installed in shared NFS storage. The database's datafile, redo log files, and CRS and voting disks are stored in NetApp storage and are accessed through NFS.
We configured Oracle Direct NFS client after installation of database binary and database creation.
Following table14 shows the configuration of directory structure created in of each Guest VM to store Database as well as Database Binaries.
Table 14. Guest VM Local Directory Configuration.
Guest VM Directory
Details
Description
/u01/app/oracle
Locally Created
Database Binary
/u01/app/11.2.0/
Locally created
GRID Infrastructure
/ocrvote
Mounted with NFS Volume
OCR file and voting disk
/data_A
Mounted with NFS Volume from Controller A
Datafiles and copy of control file
/data_B
Mounted with NFS Volume from Controller B
Datafiles and copy of control file
/log_A
Mounted with NFS Volume from Controller A
Redo log files and copy of control file
/log_B
Mounted with NFS Volume from Controller B
Redo log files and copy of control file
Following table 15 shows the storage configuration required to store 1 TB database.
Table 15. Storage Design for Database
Aggregate Name
Storage Controller
Volume Name
Size
Description
Oracle_Aggr
Controller A
/vol/Data_Vol
500 GB
Datafiles and copy of control file
Oracle_Aggr
Controller A
/vol/Log_Vol
200 GB
Redo logs and copy of control file
Oracle_Aggr
Controller A
/vol/ocrvote
20 GB
OCR file and voting disk
Oracle_Aggr_B
Controller B
/vol/Data_Vol_B
500 GB
Datafiles and copy of control file
Oracle_Aggr_B
Controller B
/vol/Log_Vol_B
200 GB
Redo Logs and copy of control file
In this experiment we created one TB size of database using Benchmark Factory for Database on four node clustered Instances. TPCC workload ran for 24 hours with 1800 concurrent user load to verify no data corruption and it sustained the workload for continues 24 hours without any failure or data corruptions. We collected three data points based on storage utilization (Mainly CPU and Disks) as well as Server Utilization with the number of IOPS. The IOPS were captured from storage, which is generated from both Flash Cache and disks.
Figure 42 shows the ESXi Server CPU Utilization along with transaction per second for three different user load.
Figure 42. CPU and TPS on ESXi Host
Figure 42 shows the storage CPU, storage disk utilization, and storage IOPS for three different user loads of 1500, 1800, and 2100. As mentioned previously, these IOPS are the combined IOPS from Flash Cache and physical disks.
Figure 43. Storage CPU, Storage Disk and IOPS
Scaling and consolidation study of two-node Oracle Database 11g R2 GRID Infrastructure with RAC option
In this experiment we concentrate on scaling and consolidation of multiple of two-node Oracle Database 11g R2 with RAC option on FlexPod using VM-FEX and Cisco VICs. We used two fully loaded B200 M3 (Cisco Blade servers).
There are three different types of consolidation scenario of Oracle 11g R2 Database with RAC option can be implemented, however we used following scenario 1 in this experiment and created one guest VM on each blade to setup a two-node Oracle Database 11g R2 with RAC option. After loading the database on newly setup oracle clustered database, we added one after another VM on each of the ESXi server to setup individual two-node Oracle Database 11g R2 with RAC option.
Scenario 1: Setting up two-node Oracle cluster by using one guest VM from each ESXi server and two instances of a database is running on the same Oracle cluster. Setup one after another Oracle cluster and create one database on each Oracle Cluster as explained in following Figure 44.
Figure 44 shows the scenario 1 Guest VM details in each physical blade server as well as the multiple Oracle Cluster configurations.
Figure 44. Scenario 1 Guest VM in Each Blade Server and Various Oracle Cluster Configurations
Scenario 2: Setting up one single largest Oracle cluster by using all the Guest VMs from both of the ESXi Server. Create each database on two nodes by selecting one guest VM from each ESXi Server as explained in following Figure 45.
Figure 45 shows the scenario 2 Oracle 11g R2 Database with RAC consolidation with Guest VM Configuration.
Scenario 3: Create one large VM in each ESXi Server. Setup a two-node Oracle cluster by using large Guest VM from both of the ESXi Server. Create each database on two-node Oracle Cluster as explained in following Figure 46.
Following Figure 46 shows the scenario 3 Oracle 11g R2 Database with RAC consolidation with guest VM configuration.
In this experiment we used scenario 1 for consolidating multiple Oracle Clustered database. We were able to consolidate five 500GB size of database with their independent Oracle cluster, considering optimal utilization of CPU, memory and network in each ESXi server, as well as storage
Each ESXi OS is an iSCSI boot from the NetApp storage. The NFS protocol is used to access the data volumes and guest OS volume from NetApp storage. RHEL 5.8 64-bit is installed in each guest VM.
Following table 16 shows the configuration of each Guest VM.
Table 16. Guest VM Configuration
Guest VM Component
Details
Description
CPU
4 vCPU
4 dynamic virtual CPUs
Memory
32 GB
Physical memory
NIC1
Public Access
MTU size 1500, for Management and Public Access
NIC2
Private Interconnect1
Private Interconnect configured for HAIP, MTU SIZE 9000
NIC3
Private Interconnect2
Private Interconnect configured for HAIP, MTU Size 9000
NIC4
Storage access
Database access through NFS, MTU size 9000
SWAP Space
32 GB
Swap Space
In this experiment we used scenarios 1; however, the other two scenarios can be implemented as per customer requirement. Oracle Database Binary and Oracle GRID Infrastructure are installed in local storage of the each guest VM; however, these Binaries can be installed in a shared NFS storage. Database's datafile, redo log files and crs, voting disks are stored in NetApp storage and access through NFS.
Following table17 shows the configuration of directory structure created in of each Guest VM to store Database as well as Binaries.
Table 17. Guest VM Local Directory Configuration for each Oracle Cluster
Guest VM Directory
Details
Description
/u01/app/oracle
Locally Created
Database Binary
/u01/app/11.2.0/
Locally created
GRID Infrastructure
/ocrvote
Mounted with NFS Volume
OCR file and voting disk
/data_A
Mounted with NFS Volume from Controller A
Datafiles and copy of control file
/data_B
Mounted with NFS Volume from Controller B
Datafiles and copy of control file
/log_A
Mounted with NFS Volume from Controller A
Redo log files and copy of control file
/log_B
Mounted with NFS Volume from Controller B
Redo log files and copy of control file
500GB size of database created individually for each Oracle Cluster through Benchmark Factory for Database.
Following table 18 shows the storage configuration required for each Oracle Cluster to store 500GB size of database.
Table 18. Storage Configuration for Database
Aggregate Name
Storage Controller
Volume Name
Size
Description
Oracle_Aggr
Controller A
/vol/Data_Vol
250 GB
Datafiles and copy of control file
Oracle_Aggr
Controller A
/vol/Log_Vol
100 GB
Redo logs and copy of control file
Oracle_Aggr
Controller A
/vol/ocrvote
20 GB
OCR file and voting disk
Oracle_Aggr_B
Controller B
/vol/Data_Vol_B
250 GB
Datafiles and copy of control file
Oracle_Aggr_B
Controller B
/vol/Log_Vol_B
100 GB
Redo logs and copy of control file
In this experiment we created one 500 GB size of database using Benchmark Factory for Database on each two-node clustered Instances. Each Database was carrying a 300 users issuing transaction at a pre-defined rate and the same transaction rate was observed when five Oracle Clustered databases are deployed.
We verified liner scaling of Oracle cluster node. After adding one Oracle RAC database after another, we found linear utilization of ESXi host as well as NetApp Storage. We were able to consolidate five of 500 GB databases, individually running on their own Oracle cluster considering optimal use of ESXi Server and Storage.
Figure 47 shows the linear scaling of ESXi Server CPU after adding one after another Oracle RAC Database along with the transaction per second. In this experiment we found the NFS response time below 3 ms.
Figure 47. ESXi Host CPU and TPS
Figure 48 shows Storage CPU and Storage Disk utilization along with the Transaction per second generated from the storage after adding one after another Oracle RAC database.
Figure 48. ESXi Host CPU and TPS
Hardware Components used for Oracle 11g R2 GRID Infrastructure with RAC Option.
Table 19. Hardware Components Used in the Deployment
Server Details
Storage Details
Four Cisco UCS Blade Servers B200 M3
NetApp FAS3270
CPU Model: Intel Xeon E5-2690
Protocol License: NFS, iSCSI
Memory: 256GB
Network: 10Gbps Ethernet and iSCSI
Network: VIC Adaptor with 80 Gbps bandwidth
Flash Cache: 2 x 500GB
Server Role: ESXi Server hosting guest VM for Oracle Database 11g R2 GRID with RAC option
Total Number of Disk Drives and Model-125 SAS 15K rpm
Summary
FlexPod which constitutes various technologies mainly, Cisco Unified Computing System, VMware vSphere 5.0 and NetApp Storage Technologies together to form a highly reliable, robust and a virtualized solution for Oracle Database 11g R2 GRID Infrastructure.
Oracle Database 11g R2 GRID Infrastructure with RAC option consolidation using Virtualization technology has tremendous benefits for optimizing data center resources. FlexPod provides an ideal platform for Oracle RAC database consolidation. Cisco's unique technologies such as Cisco Extended Memory Technology, Cisco Virtual Interface Card and Cisco VM-FEX Technology enables an exceptional level of consolidation that stands out in the industry for this class of platform, thereby resulting in significantly improved ROI and lower TCO for IT managers.
This White Paper introduces four node Oracle Database 11g R2 GRID Infrastructure with RAC option validation on FlexPod in a virtualize environment and the two-node Oracle Database 11g R2 GRID Infrastructure with RAC option consolidation on FlexPod in a virtualized environment. We found the linear scaling of resource utilization both for storage as well as server while consolidating multiple Oracle RAC databases. We were able to consolidate five Oracle RAC databases of 500GB each size in a pair of B200M3 servers.
In this experiment we used scenario 1 of a two-node Oracle Database 11g R2 GRID Infrastructure with RAC option scaling and consolidation study, however other two scenarios can be used based on customer requirement.
The Cisco® Server Fabric Switch enables utility computing by dramatically simplifying the data center architecture. It creates a unified, "wire-once" fabric that aggregates I/O and server resources. With the unified fabric, instead of servers having many cables coming out of them, the server switch connects every server with a single high-bandwidth, low-latency network cable (two cables for redundancy).
Aggregating the server's I/O resources saves significant capital expense. Consolidating resources over the unified fabric eliminates costs of underutilized Fibre Channel HBAs and NICs as well as associated cabling complexity. Instead of being designed to accommodate bandwidth peaks using a dedicated switch port for each host, a data center can share remote Fibre Channel and Gigabit Ethernet ports, enabling network designs based on average load across multiple servers. This can save up to 50 percent of the cost of the I/O associated with a server. Also, by eliminating multiple adapters and local storage by introducing a single high-bandwidth, low-latency connection, the size of the server is driven only by CPU and memory requirements. This often results in a reduction in the size and cost of the server as well as in its space, power, and cooling needs, resulting in immediate return-on-investment savings of up to 50 percent.
Bill of Material
The Table 20 gives details of the components used in the Document.
UCS 6248UP Rear
UCS 6248UP Front
