Guest

Design Zone for Data Centers

Virtualizing Oracle JD Edwards on Cisco Unified Computing System

  • Viewing Options

  • PDF (11.2 MB)
  • Feedback

Table Of Contents

About the Authors

Acknowledgment

About Cisco Validated Design (CVD) Program

Virtualizing Oracle JD Edwards on Cisco Unified Computing System

Executive Summary

Purpose of this Guide

Business Needs

Audience

Solution Overview

Technology Overview

Cisco Unified Computing System Overview

Main Differentiating Technologies

Unified Fabric

Embedded Multirole Management

Cisco Data Center VM-FEX Virtualization Support and Virtualization Adapter

Dynamic Provisioning with Service Profiles

Cisco UCS Manager

Cisco Unified Computing System Components

Cisco UCS Fabric Interconnects

Cisco UCS Virtual Interface Card 1240

Cisco UCS Virtual Interface Card 1280

Cisco Nexus 5500 Series Switch

Cisco UCS Service Profiles

Programmatically Deploying Server Resources

Dynamic Provisioning with Service Profiles

Service Profiles and Templates

UCS Central

EMC VNX Storage Family

EMC VNX Storage Platforms

FAST Cache Technology

FAST VP

EMC PowerPath

Oracle VM

Oracle VM Architecture

Paravirtualized VM (PVM)

Oracle JD Edwards EnterpriseOne

HTML Server

Enterprise Server

Database Server

Deployment Server

Server Manager

Batch Server

Design Considerations for Oracle JD Edwards EnterpriseOne Implementation on Cisco UCS with Oracle VM

Scalable Architecture Using Cisco UCS Servers

Boot from SAN

Sizing Guidelines for Oracle JD Edwards EnterpriseOne

Oracle JDE HTML Server

Oracle JDE Enterprise Server

Oracle JDE Database Server

Oracle JD Edwards EnterpriseOne Deployment Architecture

Infrastructure Setup

Cisco Nexus Switch Configuration

First Time Setup of Cisco Nexus 5548 UP Switch

Enabling Features and Global Configuration

Creating VSAN and Adding FC Interfaces

VLAN Configuration

Virtual Port Channel (vPC) Configuration

Setting Up Jumbo Frames on Cisco Nexus 5500

Cisco Unified Computing System Configuration

Validate Installed Firmware

Chassis Discovery Policy

Enabling Network Components

Creating MAC Address Pools

Creating WWPN Pools

Creating WWNN Pools

Creating UUID Suffix Pools

Creating VLANs

Creating Uplink Ports Channels

Creating VSANs

SAN Boot Policy Configuration

Modifying Service Profile for Boot Policy

BIOS Policy

UCS Service Profile Configuration

Create vNIC Templates

Create vHBA Templates

Creating Service Profile Templates

Creating Service Profile from the Template and Associating it to Cisco UCS Blade Server

Creation of Zoneset and Zones on Cisco Nexus 5548UP Switch

Configuring Connectivity for EMC VNX Storage

Creation of Storage Pools/ RAID Groups

Oracle Virtual Server (OVS) Configuration

Oracle Virtual Server Installation

OVM Manager Installation

Oracle VM Configuration

Oracle Linux Installation

Oracle RAC Setup

Oracle Grid Installation

Oracle Database Installation

Oracle RAC Post Installation

Oracle JD Edwards EnterpriseOne Installation

Pre-Requisites

General Install Requirements

Oracle JD Edwards EnterpriseOne Specific Install Requirements

JDE Install Port Numbers

Oracle JDE Deployment Server Install

Tools Upgrade on the Deployment Server

Install Planner ESU

Enterprise Server Install

DB Server Install

Installation Plan

Installation Workbench

Change Assistant

Baseline ESU Install

UL2 Install

HTML Server Install

Configuring the Cluster

Oracle HTTP Server Installation

Install the Oracle HTTP server on the JDE HTML server.

JDE User Creation

Oneworld Client Install

Full Package Build

Summary

Oracle JD Edwards EnterpriseOne Performance and Scalability

Workload Description

Test Methodology

Interactive Workload Mix

Interactive workload Scaling

User Response Time

CPU Utilization

Memory Utilization

I/O Performance

Individual UBEs

Only Batch Execution

Best Practices & Tuning Recommendations

System Configuration

Jumbo Frames

TCP Parameter Configuration

Oracle RAC Configuration

WebLogic Server Configuration

Oracle JD Edwards Enterprise Server Configuration

Conclusion

Bill of Materials

Appendix A - Workload Mix for Batch and Interactive Test

Appendix B - Reference Documents

Appendix C - Reference Links


Virtualizing Oracle JD Edwards on Cisco Unified Computing System
Oracle JD Edwards EnterpriseOne on Cisco UCS with Oracle VM 3.1.1 and EMC VNX 5300
Last Updated: January 2, 2014

Building Architectures to Solve Business Problems

About the Authors

Anil Dhiman, Technical Marketing Engineer, SAVBU, Cisco Systems

Anil Dhiman is a Technical Marketing Engineer with Server Access Virtualization Business Unit (SAVBU) at Cisco. Anil has over 12 years of experience in benchmarking and performance analysis of large multi-tier systems, such as Foreign Exchange products. Anil specializes in optimizing and tuning of applications deployed on J2EE application servers and has delivered world record numbers for SPECjbb2005 and SPECjbb2013 benchmarks on Cisco Unified Computing Systems. Anil has worked as a performance engineer for Oracle IAS team. Prior to joining Cisco, Anil worked as a performance engineering architect with Symphony Services.

Acknowledgment

For their support and contribution to the design, validation, and creation of the Cisco Validated Design, I would like to thank:

Vadiraja Bhatt- Cisco

Ramakrishna Nishtala- Cisco

Niranjan Mohapatra- Cisco

John McAbel- Cisco

Sindhu Sudhir- Cisco

Rini Kuriakose- Cisco

Anthony Schifano- Oracle

Kathy Sharp- EMC

About Cisco Validated Design (CVD) Program


The CVD program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information visit:

http://www.cisco.com/go/designzone

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at http://www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1005R)

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

© 2013 Cisco Systems, Inc. All rights reserved.

Virtualizing Oracle JD Edwards on Cisco Unified Computing System


Executive Summary

Oracle JD Edwards is a suite of products from Oracle that caters to the Enterprise Resource Planning (ERP) requirement of an Organization. Oracle has three flag ship ERP Applications, Oracle E-Business Suite, PeopleSoft, and JD Edwards. ERP Applications have been thriving and improving the productivity of Organizations for a couple of decades now. But with the increased complexity and extreme performance requirements, customers are constantly looking for better Infrastructure to host and run these Applications.

This document presents a differentiated solution using Oracle Virtualized Environment that validates Oracle JD Edwards EnterpriseOne environment on Cisco Unified Computing System (UCS) with Oracle VM 3.1.1, Oracle Linux 5.8, Oracle 11gR2 RAC Database, Cisco UCS Blade Servers, Cisco Nexus 5548UP Switches, Cisco UCS Management System and EMC VNX5300. Cisco Oracle Competency Center tested, validated, benchmarked and has showcased the Oracle JD Edwards ERP Application using Oracle Day in the Life (DIL) Kit.

Purpose of this Guide

This design guide demonstrates the best practices for deployment of Oracle JD Edwards EnterpriseOne on Oracle Virtualized Environment. It elaborates on step by step procedure to deploy Cisco UCS, Cisco Nexus family of switches, EMC VNX 5300 and Oracle JD Edwards application on Oracle VM.

The design was validated by measuring the performance and scalability of Oracle JD Edwards. This was achieved by executing Oracle JD Edwards Day in the Life (DIL) Kit on Cisco UCS Platform by benchmarking various Application workloads using Load Runner tool. Oracle JDE DIL kit comprises of interactive Application workloads and batch workload, Universal Batch Engine Processes (UBEs). The interactive Application Users were validated and benchmarked by scaling from 1000 to 7,000 concurrent users. Achieving sub second response times for various JDE Application workloads with a large variation of interactive apps and UBEs, clearly demonstrate the suitability of Cisco UCS for small to large Oracle JD Edwards EnterpriseOne and help customers to make an informed decision on choosing Cisco UCS for their virtualized Oracle JD Edwards implementation.

Business Needs

Customers constantly look value for money, when they transition from one platform to another or migrate from proprietary systems to commodity hardware platforms; they endeavor to improve operational efficiency and optimal resource utilization.

Other important aspects are Management and Maintenance; ERP Applications are business critical Applications and need to be up and running without down time. Ease of Maintenance and efficient management with minimal staff and reduced budgets are pushing infrastructure managers to balance uptime and ROI.

Organizations have steadily adopted virtualization technologies in recent years to reduce the size, cost, and complexity of their infrastructure. Virtualization technologies let IT departments run multiple operating systems and applications on the same physical server, thereby allowing them to consolidate server hardware.

Server sprawl, old technologies that consume precious real estate space and power with increase in the cooling requirement have pushed customers to look for innovative technologies that can address some of these challenges.

Audience

The target audience for this guide include sales engineers, field consultants, professional services staff, IT managers, partner engineering staff, and customers who want to deploy Oracle JDE on Cisco UCS with EMC VNX5300 and Oracle VM.

This design guide is intended to assist Solution Architects, JDE Project Manager, Infrastructure Manager, Sales Engineers, Field Engineers and Consultants in planning, designing, and deployment of Oracle JD Edwards hosted on Cisco UCS environment. This document assumes that the reader has an architectural understanding of the Cisco UCS servers, Nexus 5548UP Switch, Oracle JD Edwards, EMC VNX5300, Oracle VM and related software.

Solution Overview

The solution in the design guide demonstrates the deployment of Oracle JD Edwards EnterpriseOne on Cisco UCS. The Oracle JD Edwards solution architecture is designed to run on multiple platforms and multiple databases. In this deployment, the Oracle JD Edwards Enterprise One (JDE E1) Release 9.0.2 was deployed on Oracle VM 3.1.1 with Oracle Linux 5.8 as the OS for Guest VMs. The JDE E1 database was hosted on two-node Oracle 11gR2, and the JDE HTML server ran on Oracle Weblogic Server Release 10.3.6. The design validates the use of Oracle RAC database for mission critical applications such as JDE E1 to provide high availability and scalability.

The deployment and testing was conducted in a Cisco® test and development environment to measure the performance of Oracle's JDE E1 Release 9.0.2 on Oracle VM with Cisco UCS. The JDE E1 DIL kit was deployed as a test workload which is a suite of 17 test scripts that exercise representative transactions of the most popular JDE E1 applications, including SCM, SRM, HCM, CRM, and Financial Management. This complex mix of applications simulate workloads to more closely reflect the customer environments.

The solution describes the following aspects of Oracle JD Edwards deployment on Cisco UCS:

Sizing and Design guidelines for Oracle JD Edwards E1 using JDE E1 DIL kit for both interactive and batch processes.

Configuring Cisco UCS for Oracle JD Edwards

Configuring Fabric Interconnect

Configuring Cisco UCS Blades

Configuring EMC VNX5300 storage system for Oracle JD Edwards

Configuring the storage and creating LUNs

Associating LUNs with the hosts

Configuring Oracle VM environment

Installation of Oracle Virtual Server (OVS) and Oracle VM Manager

Installation and Configuration of Para-Virtualized Guest (PVMs) with Oracle Linux 5.8

Installation and Configuration of two-node Oracle RAC 11.2.0.3 on Oracle Virtualized environment

Installing and configuring JDE E1 release 9.02 with Tool update 8.98.4.6

Provisioning the required server resource

Installing and Configuring JDE HTML Server, JDE Enterprise Server and Oracle 11gR2 RAC on Oracle Linux 5.8

Performance characterization of JD Edwards E1 Day in the life Kit (DIL Kit)

Performance and Scaling analysis for JDE E1 interactive Apps

Performance Analysis of JDE batch processes (UBEs)

Best Practices and Tuning Recommendations to deploy Oracle JD Edward son Cisco UCS

Figure 1 illustrates the components of JD Edwards E1 using Cisco UCS Servers.

Figure 1 Deployment Overview of JD Edwards EnterpriseOne on Cisco UCS with EMC VNX5300

Technology Overview

Cisco Unified Computing System Overview

Cisco UCS is a set of pre-integrated data center components, including Cisco Blade Servers, Adapters, Fabric Interconnects, and Fabric Extenders, that are integrated within a common embedded management system. This approach results in far fewer system components and much better manageability, operation efficiency, and more flexibility than comparable data center platforms.

Main Differentiating Technologies

The main differentiating technologies described here make Cisco UCS unique and give it advantages over competing offerings. The technologies presented here are high level, and the discussions do not include the technologies (such as Fibre Channel over Ethernet [FCoE]) that support these high-level elements.

Unified Fabric

Unified fabric can dramatically reduce the number of network adapters, Cisco Blade Server Switches, cables, and management touch points by passing all network traffic to parent Cisco Fabric Interconnects, where it can be prioritized, processed, and managed centrally. This approach improves performance, agility, and efficiency and dramatically reduces the number of devices that need to be powered, cooled, secured, and managed.

Embedded Multirole Management

Cisco UCS Manager is a centralized management application that is embedded on the fabric switch. Cisco UCS Manager controls all Cisco UCS elements within a single redundant management domain. These elements include all aspects of system configuration and operation, eliminating the need to use multiple, separate element managers for each system component. Massive reduction in the number of management modules, consoles and in the proliferation of agents resident on all the hardware (which must be separately managed and updated) are important deliverables of Cisco UCS. Cisco UCS Manager, using role-based access and visibility, helps enable cross-function communication efficiency, promoting collaboration between data center roles for increased productivity.

Cisco Data Center VM-FEX Virtualization Support and Virtualization Adapter

With Cisco data center VM-FEX, virtual machines have virtual links that allow them to be managed in the same way as physical links. Virtual links can be centrally configured and managed without the complexity of traditional systems, which interpose multiple switching layers in virtualized environments. I/O configurations and network profiles move along with virtual machines, helping increase security and efficiency while reducing complexity. Cisco data center VM-FEX helps improve performance and reduce Network Interface Card (NIC) infrastructure.

Dynamic Provisioning with Service Profiles

Cisco UCS Manager delivers service profiles, which contain abstracted server-state information, creating an environment in which everything unique about a server is stored in the fabric, and the physical server is simply another resource to be assigned. Cisco UCS Manager implements role-based and policy-based management focused on service profiles and templates. These mechanisms fully provision one or many servers and their network connectivity in minutes, rather than hours or days.

Cisco UCS Manager

Cisco UCS Manager (UCSM) is an embedded, unified manager that provides a single point of management for Cisco UCS. Cisco UCS Manager can be accessed through an intuitive GUI, a command-line interface (CLI), or the comprehensive open XML API. It manages the physical assets of the server and storage - LAN connectivity and it is designed to simplify the management of virtual network connections through integration with several major hypervisor vendors. It provides IT departments with the flexibility to allow people to manage the system as a whole, or to assign specific management functions to individuals based on their roles as managers of server or network hardware assets. It simplifies operations by automatically discovering all the components available on the system and enabling a stateless model for resource use. Every instance of Cisco UCS Manager and all of the components managed by it form a domain. Multiple Domains can be managed through Cisco UCS Central which integrates with Cisco UCS Manager, and utilizes it to provide global configuration capabilities for pools, policies, and firmware. Cisco UCS Central software manages multiple, globally distributed Cisco UCS domains with thousands of servers from a single pane.

The elements managed by Cisco UCS Manager include:

Cisco UCS Integrated Management Controller (IMC) firmware

RAID controller firmware and settings

BIOS firmware and settings, including server Universal User ID (UUID) and boot order

Converged Network Adapter (CNA) firmware and settings, including MAC addresses and WorldWide Names (WWNs) and SAN boot settings

Virtual port groups used by virtual machines, using Cisco data center VM-FEX technology

Interconnect configuration, including uplink and downlink definitions, MAC address and WWN pinning, VLANs, VSANs, Quality of Service (QoS), bandwidth allocations, Cisco Data Center VM-FEX settings, and EtherChannels to upstream LAN switches

Cisco Unified Computing System Components

Figure 2 shows the Cisco Unified Computing System (UCS) components.

Figure 2 Cisco UCS Components

Cisco UCS is designed from the start to be programmable and self-integrating. A server's entire hardware stack, ranging from server firmware and settings to network profiles, is configured through model-based management. With Cisco virtual interface cards (VICs), even the number and type of I/O interfaces is programmed dynamically, making every server ready to power any workload at any time.

With model-based management, administrators manipulate a model of a desired system configuration and associate a model's service profile with hardware resources, and the system configures itself to match the model. This automation accelerates provisioning and workload migration with accurate and rapid scalability. The result is increased IT staff productivity, improved compliance, and reduced risk of failures due to inconsistent configurations.

Cisco FEX Technology reduces the number of system components that need to be purchased, configured, managed, and maintained by condensing three network layers into one. It eliminates both blade server and hypervisor-based switches by connecting Cisco UCS Fabric Interconnect ports directly to individual blade servers and virtual machines. Virtual networks are now managed exactly the same way that physical networks are, but enable massive scalability. This approach represents a radical simplification compared to traditional systems, reducing capital expenditures (CapEx) and operating expenses (OpEx) while increasing business agility, simplifying and accelerating deployment, and improving performance.

Cisco UCS Fabric Interconnects

Cisco UCS Fabric Interconnects create a unified network fabric throughout Cisco UCS. They provide uniform access to both networks and storage, eliminating the barriers to deployment of a fully virtualized environment based on a flexible, programmable pool of resources. Cisco Fabric Interconnects comprise a family of line-rate, low-latency, lossless 10 Gigabit Ethernet, IEEE Data Center Bridging (DCB), and FCoE interconnect switches. Based on the same switching technology as the Cisco Nexus® 5000 Series Switches, Cisco UCS 6200 Series Fabric Interconnects provide additional features and management capabilities that make them the central nervous system of Cisco UCS. The Cisco UCS Manager software runs inside the Cisco UCS Fabric Interconnects. The Cisco UCS 6200 Series Fabric Interconnects expand the Cisco UCS networking portfolio and offer higher capacity, higher port density, and lower power consumption. These interconnects provide the management and communication backbone for the Cisco UCS B-Series Blade Servers and Cisco UCS Blade Server Chassis. All chassis and all blades that are attached to interconnects are part of a single, highly available management domain. By supporting unified fabric, the Cisco UCS 6200 Series provides the flexibility to support LAN and SAN connectivity for all blades within its domain at configuration time. Typically deployed in redundant pairs, Cisco UCS Fabric Interconnects provide uniform access to both networks and storage, facilitating a fully virtualized environment.

The Cisco UCS Fabric Interconnect portfolio currently consists of the Cisco UCS 6248UP Fabric Interconnects deployed in redundant pair.

Figure 3 Cisco UCS 6248UP 48-Port Fabric Interconnect

The Cisco UCS 6248UP 48-Port Fabric Interconnect is a one-rack-unit (1RU) 10 Gigabit Ethernet, IEEE DCB, and FCoE interconnect providing more than 1-terabit-per-second (Tbps) throughput with low latency. It has 32 fixed ports of Fibre Channel, 10 Gigabit Ethernet, IEEE DCB, and FCoE Enhanced Small Form-Factor Pluggable (SFP+) ports.

One expansion module slot can provide up to 16 additional Fibre Channel, 10 Gigabit Ethernet, IEEE DCB, and FCoE SFP+ ports.

Cisco UCS Virtual Interface Card 1240

A Cisco innovation, the Cisco UCS VIC 1240 is a four-port 10 Gigabit Ethernet, FCoE-capable modular LAN on motherboard (mLOM) designed exclusively for the M3 generation of Cisco UCS B-Series Blade Servers. When used in combination with an optional port expander, the Cisco UCS VIC 1240 capabilities can be expanded to eight ports of 10 Gigabit Ethernet.

Cisco UCS Virtual Interface Card 1280

A Cisco innovation, the Cisco UCS VIC 1280 is an eight-port 10 Gigabit Ethernet, FCoE-capable mezzanine card designed exclusively for Cisco UCS B-Series Blade Servers.

The Cisco UCS VIC 1240 and 1280 enable a policy-based, stateless, agile server infrastructure that can present up to 256 PCI Express (PCIe) standards-compliant interfaces to the host that can be dynamically configured as either NICs or HBAs. In addition, the Cisco UCS VIC 1280 supports Cisco Data Center VM-FEX technology, which extends the Cisco UCS Fabric Interconnect ports to virtual machines, simplifying server virtualization deployment.

Figure 4 Cisco UCS 5100 Series Blade Server Chassis

The Cisco UCS 5108 Blade Server Chassis is a 6RU blade chassis that accepts up to eight half-width Cisco UCS B-Series Blade Servers or up to four full-width Cisco UCS B-Series Blade Servers, or a combination of the two. The Cisco UCS 5108 can accept four redundant power supplies with automatic load sharing and failover and two Cisco UCS 2100 or 2200 Series Fabric Extenders. The chassis is managed by Cisco UCS chassis management controllers, which are mounted in the Cisco UCS Fabric Extenders and work in conjunction with Cisco UCS Manager to control the chassis and its components.

A single Cisco UCS managed domain can theoretically scale to up to 40 individual chassis and 320 blade servers. At this time, Cisco UCS supports up to 20 individual chassis and 160 blade servers.

Basing the I/O infrastructure on a 10-Gbps unified network fabric allows Cisco UCS to have a streamlined chassis with a simple yet comprehensive set of I/O options. The result is a chassis that has only five basic components:

The physical chassis with passive midplane and active environmental monitoring circuitry

Four power supply bays with power entry in the rear and hot-swappable power supply units accessible from the front panel

Eight hot-swappable fan trays, each with two fans

Two Cisco Fabric Extender slots accessible from the back panel

Eight half width Cisco Blade Server slots accessible from the front panel

Figure 5 Cisco UCS B200 M3 Blade Servers

The Cisco UCS B200 M3 Blade Server delivers performance, versatility, and density without compromise. It addresses the broadest set of workloads, from IT and web infrastructure to distributed databases. Building on the success of the Cisco UCS B200 M2 Blade Server, the enterprise-class Cisco UCS B200 M3 Blade Server further extends the capabilities of the Cisco UCS portfolio in a half-width blade form factor. The Cisco UCS B200 M3 harnesses the power of the latest Intel Xeon processor E5-2600 product family, with up to 384 GB of RAM (using 16-GB DIMMs), two disk drives, and up to dual 4x 10 Gigabit Ethernet throughput. In addition, Cisco UCS has the architectural advantage of not having to power and cool excess switches in each blade chassis. With a larger power budget per blade server, Cisco can design uncompromising expandability and capabilities in its blade servers, as evidenced by the new Cisco UCS B200 M3, with its leading memory slot and drive capacity.

Cisco Nexus 5500 Series Switch

The Cisco Nexus 5500 Series Switch is designed for data center environments with cut-through technology that enables consistent, low-latency Ethernet solutions with front-to-back or back-to-front cooling and data ports in the rear, bringing switching into close proximity with servers and making cable runs short and simple. The switch series is highly serviceable, with redundant, hot-pluggable power supplies and fan modules. It uses data-center-class Cisco NX-OS Software for high reliability and ease of management.

Cisco Nexus 5500 series switches provide higher density, lower latency, and multilayer services. The Cisco Nexus 5500 series switch is well suited for enterprise-class data center server access layer deployments across a diverse set of physical, virtual, storage-access, and high-performance computing (HPC) data center environments.

Figure 6 Cisco Nexus 5548UP

The Cisco Nexus 5548UP is a 1RU 10 Gigabit Ethernet (10 GE), Fibre Channel (FC), and Fibre Channel over Ethernet (FCoE) switch offering up to 960 Gbps of throughput and up to 48 ports. The switch has 32 unified ports and one expansion slot supporting modules with 10 Gigabit Ethernet and FCoE ports or connectivity to Fibre Channel SANs with 8/4/2/1 Gbps Fibre Channel switch ports, or both.

Cisco UCS Service Profiles

Programmatically Deploying Server Resources

Cisco UCS Manager provides centralized management capabilities, creates a unified management domain, and serves as the central nervous system of the Cisco UCS. Cisco UCS Manager is embedded device management software that manages the system from end-to-end as a single logical entity through an intuitive GUI, CLI, or XML API. Cisco UCS Manager implements role- and policy-based management using service profiles and templates. This construct improves IT productivity and business agility. Now infrastructure can be provisioned in minutes instead of days, shifting IT's focus from maintenance to strategic initiatives.

Dynamic Provisioning with Service Profiles

Cisco UCS resources are abstract in the sense that their identity, I/O configuration, MAC addresses and WWNs, firmware versions, BIOS boot order, and network attributes (including QoS settings, ACLs, pin groups, and threshold policies) all are programmable using a just-in-time deployment model. The manager stores this identity, connectivity, and configuration information in service profiles that reside on the Cisco UCS 6200 Series Fabric Interconnect. A service profile can be applied to any blade server to provision it with the characteristics required to support a specific software stack. A service profile allows server and network definitions to move within the management domain, enabling flexibility in the use of system resources. Service profile templates allow different classes of resources to be defined and applied to a number of resources, each with its own unique identities assigned from predetermined pools.

Service Profiles and Templates

A service profile contains configuration information about the server hardware, interfaces, fabric connectivity, and server and network identity. The Cisco UCS Manager provisions servers utilizing service profiles. The Cisco UCS Manager implements a role-based and policy-based management focused on service profiles and templates. A service profile can be applied to any blade server to provision it with the characteristics required to support a specific software stack. A service profile allows server and network definitions to move within the management domain, enabling flexibility in the use of system resources.

Service profile templates are stored in the Cisco UCS 6200 Series Fabric Interconnects for reuse by server, network, and storage administrators. Service profile templates consist of server requirements and the associated LAN and SAN connectivity. Service profile templates allow different classes of resources to be defined and applied to a number of resources, each with its own unique identities assigned from predetermined pools.

The Cisco UCS Manager can deploy the service profile on any physical server at any time. When a service profile is deployed to a server, the Cisco UCS Manager automatically configures the server, adapters, Fabric Extenders, and Fabric Interconnects to match the configuration specified in the service profile. A service profile template parameterizes the UIDs that differentiate between server instances.

This automation of device configuration reduces the number of manual steps required to configure servers, Network Interface Cards (NICs), Host Bus Adapters (HBAs), and LAN and SAN switches. The Figure 9 shows the Service profile which contains abstracted server state information, creating an environment to store unique information about a server.

Figure 7 Service Profile

UCS Central

Cisco UCS® Manager provides a single point of management for an entire Cisco Unified Computing System™ (Cisco UCS) domain of up to 160 servers and associated infrastructure. Cisco UCS Manager uses a policy-based approach to provision servers by applying a desired configuration to physical infrastructure. Using Cisco UCS service profiles, administrators can reproduce existing physical environments, including the I/O configuration, firmware, and settings. The configuration is applied quickly, accurately, and automatically, improving business agility. A role-based access control (RBAC) model helps ensure security of the system configurations.

Cisco UCS Central Software extends the simplicity and agility of managing a single Cisco UCS domain across multiple Cisco UCS domains. Cisco UCS Central Software allows companies to easily work on a global scale, putting computing capacity close to users while managing infrastructure with centrally defined policies. Cisco UCS Central Software makes it easy to help ensure global policy compliance, with subject-matter experts choosing the resource pools and policies that need to be enforced globally or managed locally. With a simple drag-and-drop operation, Cisco UCS service profiles can be moved between geographies to enable fast deployment of infrastructure, when and where it is needed, to support business workloads. Cisco UCS Central Software does not replace Cisco UCS Manager, which is the basic engine for managing a Cisco UCS domain. It builds on the capabilities provided by Cisco UCS Manager and requires it to be in place to effect changes in individual domains.

Figure 8 Cisco UCS Central Software Architecture

Features and Benefits of Cisco UCS Central

Cisco UCS Central Software enables global management of many Cisco UCS domains, making staff more efficient and effective. It gives Cisco UCS administrators a high-level view and management of all, or groups of, Cisco UCS domains with:

Centralized Inventory of all Cisco UCS components for a definitive view of the entire infrastructure and simplified integration with current Information Technology Infrastructure Library (ITIL) processes

Centralized, policy-based firmware upgrades that can be applied globally or selectively through automated schedules or as business workloads demand

Global ID pooling to eliminate identifier conflicts

Global administrative policies that enable both global and local management of the Cisco UCS domains

An XML API, building on the Cisco UCS Manager XML API for easy integration into higher-level data center management frameworks

EMC VNX Storage Family

The EMC VNX family of storage systems represents EMC's next generation of unified storage, optimized for virtual environments, while offering a cost effective choice for deploying mission-critical enterprise applications such as Oracle JD EDwards. The massive virtualization and consolidation trends with servers demand a new storage technology that is dynamic and scalable. The EMC VNX series meets these requirements and offers several software and hardware features for optimally deploying enterprise applications such. The EMC VNX family is shown in Figure 9.

Figure 9 The EMC VNX Family of Unified Storage Platforms

A key distinction of the VNX Series is support for both block and file-based external storage access over a variety of access protocols, including Fibre Channel (FC), iSCSI, FCoE, NFS, and CIFS network shared file access. Furthermore, data stored in one of these systems, whether accessed as block or file-based storage objects, is managed uniformly via Unisphere®, a web-based interface window. For more information on Unisphere see the White Paper titled, Introducing EMC Unisphere: A Common Midrange Element Manager.

EMC VNX Storage Platforms

The EMC VNX Series continues the EMC tradition of providing some of the highest data reliability and availability in the industry. Apart from this they also include in their design a boost in performance and bandwidth to address the sustained data access bandwidth rates. The new system design has also placed heavy emphasis on storage efficiencies and density, as well as crucial green storage factors, such as a smaller data center footprint, lower power consumption, and improvements in power reporting.

The VNX has many features that help improve availability. Data protection is heightened by the lack of any single point of failure from the network to the actual disk drive in which the data is stored. The data resides on the VNX for block storage system, which delivers data availability, protection, and performance. The VNX uses RAID technology to protect the data at the drive level. All data paths to and from the network are redundant.

The basic design principle for the VNX Series storage platform includes the VNX for file front end and the VNX for block hardware for the storage processors on the back end. The control flow is handled by the storage processors in block-only systems and the control station in the file-enabled systems. The VNX OE for block software has been designed to ensure the I/O is well balanced between the two SPs. At the time of provisioning, the odd number LUNs are owned by one SP and even number LUNs are owned by the other. This results in LUNs being evenly distributed between the two SPs. If a failover occurs, LUNs would trespass over to the alternate path/SP. EMC PowerPath® restores the "default": path once the error condition is recovered. This brings the LUNs back to a balanced state between the SPs. For more information on EMC VNX Series, see:

http://www.emc.com/collateral/hardware/data-sheets/h8520-vnx-family-ds.pdf

Key efficiency features available with the EMC VNX Series include FAST Cache and FAST VP.

FAST Cache Technology

FAST Cache is a storage performance optimization feature that provides immediate access to frequently accessed data. In traditional storage arrays, the DRAM caches are too small to maintain the hot data for a long period of time. Very few storage arrays give an option to non-disruptively expand DRAM cache, even if they support DRAM cache expansion. FAST Cache extends the available cache to customers by up to 2 TB using enterprise Flash drives. FAST Cache tracks the data temperature at 64 KB granularity and copies hot data to the Flash drives once its temperature reaches a certain threshold. After a data chunk gets copied to FAST Cache, the subsequent accesses to that chunk of data will be served at Flash latencies. Eventually, when the data temperature cools down, the data chunks get evicted from FAST Cache and will be replaced by newer hot data. FAST Cache uses a simple Least Recently Used (LRU) mechanism to evict the data chunks.

FAST Cache is built on the premise that the overall applications' latencies can improve when most frequently accessed data is maintained on a relatively smaller sized, but faster storage medium, like Flash drives. FAST Cache identifies the most frequently accessed data which is temporary and copies it to the flash drives automatically and non-disruptively. The data movement is completely transparent to applications, thereby making this technology application-agnostic and management-free. For example, FAST Cache can be enabled or disabled on any storage pool simply by selecting/clearing the "FAST Cache" storage pool property in advanced settings.

FAST Cache can be selectively enabled on a few or all storage pools within a storage array, depending on application performance requirements and SLAs.

There are several distinctions to EMC FAST Cache:

It can be configured in read /write mode, which allows the data to be maintained on a faster medium for longer periods, irrespective of application read-to-write mix and data re-write rate.

FAST Cache is created on a persistent medium like Flash drives, which can be accessed by both the storage processors. In the event of a storage processor failure, the surviving storage processor can simply reload the cache rather than repopulating it from scratch. This can be done by observing the data access patterns again, which is a differentiator.

Enabling FAST Cache is completely non-disruptive. It is as simple as selecting the Flash drives that are part of FAST Cache and does not require any array disruption or downtime.

Since FAST Cache is created on external Flash drives, adding FAST Cache will not consume any extra PCI-E slots inside the storage processor.

Figure 10 EMC FAST Cache

Additional information on EMC Fast Cache is documented in the white paper titled EMC FAST Cache - A Detailed Review which is available at:

http://www.emc.com/collateral/software/white-papers/h8046-clariion-celerra-unified-fast-cache-wp.pdf

FAST VP

VNX FAST VP is a policy-based auto-tiering solution for enterprise applications. FAST VP operates at a granularity of 1 GB, referred to as a "slice." The goal of FAST VP is to efficiently utilize storage tiers to lower customers' TCO by tiering colder slices of data to high-capacity drives, such as NL-SAS, and to increase performance by keeping hotter slices of data on performance drives, such as Flash drives. This occurs automatically and transparently to the host environment. High locality of data is important to realize the benefits of FAST VP. When FAST VP relocates data, it will move the entire slice to the new storage tier. To successfully identify and move the correct slices, FAST VP automatically collects and analyzes statistics prior to relocating data. Customers can initiate the relocation of slices manually or automatically by using a configurable, automated scheduler that can be accessed from the Unisphere management tool. The multi-tiered storage pool allows FAST VP to fully utilize all the storage tiers: Flash, SAS, and NL-SAS. The creation of a storage pool allows for the aggregation of multiple RAID groups, using different storage tiers, into one object. The LUNs created out of the storage pool can be either thickly or thinly provisioned. These "pool LUNs" are no longer bound to a single storage tier. Instead, they can be spread across different storage tiers within the same storage pool. If you create a storage pool with one tier (Flash, SAS, or NL-SAS), then FAST VP has no impact on the performance of the system. To operate FAST VP, you need at least two tiers.

Additional information on EMC FAST VP for Unified Storage is documented in the white paper titled EMC FAST VP for Unified Storage System - A Detailed Review, see:

http://www.emc.com/collateral/software/white-papers/h8058-fast-vp-unified-storage-wp.pdf

FAST Cache and FAST VP are offered in a FAST Suite package as part of the VNX Total Efficiency Pack. This pack includes the FAST Suite which automatically optimizes for the highest system performance and lowest storage cost simultaneously. In addition, this pack includes the Security and Compliance Suite which keeps data safe from changes, deletions, and malicious activity. For additional information on this Total Efficiency Pack as well as other offerings such as the Total Protection Pack, see:

http://www.emc.com/collateral/software/data-sheet/h8509-vnx-software-suites-ds.pdf

EMC PowerPath

EMC PowerPath is host-based software that provides automated data path management and load-balancing capabilities for heterogeneous server, network, and storage deployed in physical and virtual environments. A critical IT challenge is being able to provide predictable, consistent application availability and performance across a diverse collection of platforms. PowerPath is designed to address those challenges, helping IT meet service-level agreements and scale-out mission-critical applications.

This software supports up to 32 paths from multiple HBAs (iSCSI TCI/IP Offload Engines [TOEs] or FCoE CNAs) to multiple storage ports when the multipathing license is applied. Without the multipathing license, PowerPath will use only a single port of one adapter (PowerPath SE). In this mode, the single active port can be zoned to a maximum of two storage ports. This configuration provides storage port failover only, not host-based load balancing or host-based failover. It is supported, but not recommended, if the customer wants true I/O load balancing at the host and also HBA failover.

PowerPath balances the I/O load on a host-by-host basis. It maintains statistics on all I/O for all paths. For each I/O request, PowerPath intelligently chooses the most under utilized path available. The available under utilized path is chosen based on statistics and heuristics, and the load-balancing and failover policy in effect.

In addition to the load balancing capability, PowerPath also automates path failover and recovery for high availability. If a path fails, I/O is redirected to another viable path within the path set. This redirection is transparent to the application, which is not aware of the error on the initial path. This avoids sending I/O errors to the application. Important features of PowerPath include standardized path management, optimized load balancing, and automated I/O path failover and recovery.

For more information on Powerpath, see:

http://www.emc.com/collateral/software/data-sheet/l751-powerpath-ve-multipathing-ds.pdf

Oracle VM

Oracle VM is a platform that provides a fully equipped environment with all the latest benefits of virtualization technology. Oracle VM enables you to deploy operating systems and application software within a supported virtualization environment. Oracle VM is a Xen-based hypervisor that runs at nearly bare-metal speeds.

Oracle VM Architecture

Figure 11 elaborates on the Oracle VM architecture. The Oracle VM architecture has three important components:

Figure 11 Oracle VM Architecture

Oracle VM Manager—Provides the user interface, which is a standard ADF (Application Development Framework) web application, to manage Oracle VM Servers. Manages virtual machine lifecycle, including creating virtual machines from installation media or from a virtual machine template, deleting, powering off, uploading, deployment and live migration of virtual machines. Manages resources, including ISO files, virtual machine templates and sharable hard disks.

Oracle VM Server—A self-contained virtualization environment designed to provide a lightweight, secure, server-based platform for running virtual machines. Oracle VM Server is based upon an updated version of the underlying Xen hypervisor technology, and includes Oracle VM Agent.

Oracle VM Agent—Installed with Oracle VM Server. It communicates with Oracle VM Manager for management of virtual machines.

Advantage of using Oracle VM for Oracle JD Edwards EnterpriseOne with Oracle RAC Database

Oracle's virtualization technologies are an excellent delivery vehicle for Independent Software Vendors (ISV's) looking for a simple, easy-to-install and easy-to-support application delivery solution.

Oracle now offers a highly available, grid-ready virtualization solution for your data center, combining all the benefits of a fully virtualized environment.

The combination of Oracle VM and Oracle JD Edwards EnterpriseOne enables a better server consolidation (bare-metal deployments with under utilized CPU resources can often benefit from consolidation with other workloads using server virtualization) sub-capacity licensing, and rapid provisioning.

Oracle RAC on Oracle VM also supports the creation of non-production virtual clusters on a single physical server for production demos and test/dev environments. This deployment combination permits dynamic changes to pre-configured database resources for agile responses to changing service level requirements common in consolidated environments.

Moreover, Oracle VM is the only software based virtualization solution that is fully supported and certified for both Oracle JD Edwards and Oracle real Application Clusters.

There are several reasons why customers may want to run Oracle JD Edwards EnterpriseOne in an Oracle VM environment. Some of the more common ones are:

Server Consolidation: Oracle JD Edwards EnterpriseOne under utilized CPU resources or variable CPU utilization can often benefit from consolidation with other workloads using server virtualization. A typical use case for this scenario would be the consolidation of several Oracle JD Edwards EnterpriseOne components, such as multiple Batch & interactive Servers running on individual hardware into virtualized environment through multiple VMs hosted on less number of servers.

Sub-capacity licensing: The current Oracle licensing model requires the Oracle RAC database to be licensed for all CPUs on each server in the cluster. Several times customers wish to use only a subset of the CPUs on the server for a particular Oracle RAC database. Oracle VM can be configured in such way that it is recognized as a hard partition. Hard partitions allow customers to only license those CPUs used by the partition instead of licensing all CPUs on the physical server. More information on sub-capacity licensing using hard partitioning can be found in the Oracle partitioning paper. For more information on using hard partitioning with Oracle VM refer to the "Hard Partitioning with Oracle VM" white paper.

Create a virtual cluster: Oracle VM enables the creation of a virtual cluster on a single physical server. This use case is particularly interesting for product demos, educational settings, and test environments. This configuration should never be used to run production Oracle JD Edwards environment. The following are valid deployments for this use case:

Test/ development cluster

Demo cluster

Education cluster

Rapid Provisioning: The provisioning time of a new application consists of the server (physical or virtual) deployment time, and the software install and configuration time. Oracle VM can help reduce the deployment time for both of these components. Oracle VM supports the ability to create deployment templates. These templates can then be used to rapidly provision new systems.

Oracle VM Templates for JD Edwards EnterpriseOne enable rapid implementation of your JD Edwards EnterpriseOne system. With these pre configured, pretested templates of the JD Edwards EnterpriseOne enterprise, database, and HTML servers, you can have your system up and running in just a few days. Oracle VM Templates contain the complete technology stack necessary to run the server: operating system, database, web server, and JD Edwards EnterpriseOne tools and applications.

Paravirtualized VM (PVM)

Guest virtual machines running on Oracle VM server can be configured in para-virtualized mode. In this mode the kernel of the guest operating system is modified to distinguish that it is running on a hypervisor instead of on the bare metal hardware. As a result, I/O actions and system clock timers in particular are handled more efficiently, as compared with non-paravirtualized systems where I/O hardware and timers have to be emulated in the operating system. Oracle VM supports PV kernels for Oracle Linux and Red Hat Enterprise Linux, offering better performance and scalability.

Oracle JD Edwards EnterpriseOne

Oracle JD Edwards is the ERP solution of choice for many small and medium-sized businesses (SMB). JDE E1 offers an attractive combination of a large number of easy-to-deploy and easy-to-use ERP applications across multiple industries. These applications include Supply Chain Management (SCM), Human Capital Management (HCM), Supplier Relationship Management (SRM), Financials, and Customer Relationship Management (CRM). The various components of Oracle JD Edwards EnterpriseOne are shown in Figure 12.

Figure 12 JD Edwards EnterpriseOne Components

HTML Server

HTML Server is the interface of the JDE to the outside world. It allows the JDE ERP users to connect to their Applications using their Browsers via the Web server. It is one of the Tiers of the standard three-tier JDE Architecture. HTML Server is just not an interface, it has logic and runs Web Services which processes some of the data and only the result set is sent through the WAN to the end users.

Enterprise Server

Enterprise Server hosts the JDE Applications that execute all the basic functions of the JDE ERP systems, like running the Transaction processing service, Batch Services, data replication, security and the entire time stamp and distributed processing happens at this layer. Multiple enterprise servers can be added for scalability, especially when we need to apply Electronic Software Updates (ESU) to one server while the other is online.

Database Server

Database Server in a JDE environment is used to host the data. It simply is a data repository and is not used to process JDE logic. The JDE Database server can run many supported databases such as, Oracle, SQL Server, DB2 or Access. Since this server does not run any Applications as mentioned, the only licensing that is required for this server is the database license; hence the server should be sized correctly. If this server has excess capacity the UBE's can be run on this server to improve their performance.

Deployment Server

Deployment server essentially is a centralized software (C Code) repository for deploying software packages to all the servers and workstations that are part of the Cisco JDE solution. Although the Deployment server is not a business critical server, it is very important to note that it is a critical piece of the JDE Architecture, without which the Installation, upgrade, development or modification of packages (codes) or reports would become impossible.

Server Manager

The Server Manager is a key JDE software component that helps customers deploy the latest JDE tools software onto various JDE Servers that are registered with the server manager. The server manager is web based and enables life cycle management of JDE products like the Enterprise server and HTML server via a web based console. It has in-built abilities for configuration management and it maintains an audit history of changes made to the components and configuration of various JDE Server software.

Batch Server

Batch processes (UBE) are background processes requiring no operator intervention or interactivity. One of most important batch process in JDE is MRP Process. Batch Process can be scheduled using a Process Scheduler which runs in the Batch Server. JDE customers running a high volume of reports often split the load on their Enterprise server such that they have one or more Batch servers which handle the high volume reporting (UBE) loads, thereby freeing up their enterprise server to handle interactive user loads more efficiently. This leads to better interactive application and UBE performance due to the expanded scaling provided by the additional hardware.

Design Considerations for Oracle JD Edwards EnterpriseOne Implementation on Cisco UCS with Oracle VM

The design document provides best practices for designing Oracle JD Edwards EnterpriseOne virtualized environment with Cisco UCS, EMC VNX with Oracle VM. There are several factors which need to be considered vis-à-vis JD Edwards HTML Server, JDE E1 application server for interactive and batch (UBE) processes and most importantly the scalability, ease of deployment and maintenance of hardware installed for JD Edwards deployment.

Scalable Architecture Using Cisco UCS Servers

An obvious immediate benefit with Cisco is a single trusted vendor providing all the components needed for a JD Edwards EnterpriseOne deployment with the ability to provide scalable platform, dynamic provisioning, failover with minimal downtimes and reliability.

Some of the capabilities offered by Cisco UCS, which complement the scalable architecture include the following:

Dynamic provisioning and service profiles—Cisco UCS Manager supports service profiles, which contain abstracted server states, creating a stateless environment. It implements role-based and policy-based management focused on service profiles and templates. These mechanisms fully provision one or many servers and their network connectivity in minutes, rather than hours or days. This can be very valuable in a virtualized JD Edwards EnterpriseOne environment, wherein new VMs may need to be provisioned on short notice, or even whole new farm for specific development activities.

Cisco Unified Fabric and Fabric Interconnects—The Cisco Unified Fabric leads to a dramatic reduction in network adapters, blade-server switches, and cabling by passing all network and storage traffic over one cable to the parent Cisco Fabric Interconnects, where it can be processed and managed centrally. This improves performance and reduces the number of devices that need to be powered, cooled, secured, and managed. The Cisco UCS 6200 series Fabric Interconnects offer key features and benefits, including:

High performance Unified Fabric with line-rate, low-latency, lossless 10 Gigabit Ethernet, and Fibre Channel over Ethernet (FCoE).

Centralized unified management with Cisco UCS Manager Software.

Virtual machine optimized services with the support for VN-Link technologies.

Unified Fabric for Oracle JD Edwards deployment helps to use same fabric path for both Ethernet and FC traffic. This provides multiple storage options using either block or file level storage. 10 Gigabit loseless Ethernet connectivity provides high throughput between Oracle Virtual Servers, thus helping to reduce long running UBE execution time such as MRP.

To accurately design JD Edwards EnterpriseOne on any hardware configuration, we need to understand the characteristics of each tier in JDE deployment vis-à-vis CPU, memory and I/O operations. For instance JDE Enterprise Server for Interactive is both CPU and memory intensive, but is low on disk utilization, whereas, the database server is more memory and disk intensive rather than CPU utilization. Some of the important characteristics to design JD Edwards EnterpriseOne on Cisco UCS Server are listed in Table 1:

Table 1 Oracle JD Edwards EnterpriseOne Design Considerations

Server Type
CPU
Memory
Disk I/O
Comments

JDE HTML Server

Medium

Medium

Low

Multiple JVMs run across single HTML Server and due to high GC activity each JVM require intensive CPU processing cycles.Multiple JVMs configured on a single server with optimum heap size reduces the Full GC pause time for each JVM when compared with a single JVM with large heap size. Guest OS for HTML Server can be configured through virtual disk which is carved out of a Storage Repository with a RAID Pool with low disk giving lesser IOPs.

This is also a reason we chose a storage repository with RAID 5 and 10 SAS disks configured for HTML Server as well as Enterprise Server deployment. The Guest VM for HTML Server is configured with 8 vCPUs and 64 GB of physical memory

JDE Enterprise Server for Interactive apps

High

Medium

Low

JDE Enterprise Server for interactive is CPU intensive and requires very less IOPs throughout.

We chose a Guest OS with 10 vCPUs and 64 GB of physical memory. The Virtual Disk is carved out of a common repository used both for HTML and Enterprise Server VMs.

JDE Enterprise Server for Batch

High

Low

Low

JDE Enterprise server for batch is CPU intensive but is low on memory and Disk utilization. Presently we have chosen to use configuration for batch server VM similar to JDE Enterprise Server for Apps.

Database Server

High

High

High

Database Server has high memory and Disk utilization (esp. for UBE processes)

The Oracle RAC nodes are configures with 10 vcPUs and 120 GB of memory each. The DATA and LOG files are configured on physical disk attached directly to the Guest VM. This allows lower IO latency for both DATA and LOG drives.

Deployment Server and Server Manager

Low

Low

Low

Deployment Server is utilized to build and deploy packages for JDE deployment. Server Manager is used for Monitoring JDE run time for example, monitoring Daemon process.

Thus, UCS B200 M3 with two Intel E5-2609 (4 core) processors and just 32 GB of physical memory was deployed.

The utilization of this server is maximized by deploying both Server Manager and Deployment Server on a single physical server.


Boot from SAN

Boot from SAN is a critical feature which helps to maximize the benefits of Cisco UCS stateless computing, which does not require a static binding between a physical server and the OS/ hypervisor hosted on that server. The hypervisor is installed on a SAN LUN and is booted using the service profile. When the service profile is moved to another server, the server policy and the PWWN of the HBAs also moves along with the service profile. Thereby, the new server takes the identity of the old server and looks identical to the old server.

The following are the benefits of boot from SAN:

Reduce Server Footprint—Boot from SAN eliminates the need for each server to have its own direct-attached disk (internal disk) which is a potential point of failure. The following are the advantages of diskless servers:

Require less physical space

Require less power

Require fewer hardware components

Less expensive

Disaster Recovery—Boot information and production data stored on a local SAN can be replicated to another SAN at a remote disaster recovery site. When server functionality at the primary site goes down in the event of a disaster, the remote site can take over with a minimal downtime.

Recovery from server failures—Recovery from server failures is simplified in a SAN environment. Data can be quickly recovered with the help of server snapshots, and mirrors of a failed server in a SAN environment. This greatly reduces the time required for server recovery.

High Availability—A typical data center is highly redundant in nature with redundant paths, redundant disks and redundant storage controllers. The operating system images are stored on SAN disks which eliminates potential problems caused due to mechanical failure of a local disk.

Rapid Re-deployment—Businesses that experience temporary high production workloads can take advantage of SAN technologies to clone the boot image and distribute the image to multiple servers for rapid deployment. Such servers may only need to be in production for hours or days and can be readily removed when the production need has been met. Highly efficient deployment of boot images makes temporary server usage highly cost effective.

Centralized Image Management—When operating system images are stored on SAN disks, all upgrades and fixes can be managed at a centralized location. Servers can readily access changes made to disks in a storage array.

With boot from SAN, the hypervisor image resides on the SAN and the server communicates with the SAN through a Host Bus Adapter (HBA). The HBA BIOS contain instructions that enable the server to find the boot disk. After Power On-Self Test (POST), the server hardware component fetches the designated boot device in the hardware BOIS settings. Once the hardware detects the boot device, it follows the regular boot process.

Sizing Guidelines for Oracle JD Edwards EnterpriseOne

Sizing ERP deployments on a virtualized environment is a complex, and setting it up correctly depends on the input provided by the customers on how they intend to use the ERP system, and what their priorities are in terms of end user as well as corporate expectations.

Some of the common questions related to ERP sizing are: the number of concurrent interactive users using the system, total number of ERP end users, the kind of applications that the end users will access as well as number of reports and type of reports generated during peak activity can help size the system for optimal performance. Analyzing the demands and its characteristics, the JDE system is expected to handle during different time periods in the fiscal year is necessary to do a proper sizing.

The Oracle JD Edwards EnterpriseOne configuration used in the present deployment is geared to handle a very high workload of end users running heavy SRM interactive applications as well as a high number of batch processes. Each of the HTML Server, Enterprise Server and Database nodes are load balanced with an active-active setup. Each Oracle Virtual Server is hosting an instance of HTML Server, Enterprise Server and Oracle RAC node, thus providing failover during maintenance or failure of Oracle Virtual Server nodes.

The following sections briefly describe the sizing aspects of each tier of the three tier Oracle JD Edwards EnterpriseOne deployment architecture.

Oracle JDE HTML Server

The Oracle JDE HTML server, serves end user interactive application requests from JDE users. The JDE HTML server loads the application forms and requests services from the JDE Enterprise server for application processing based on form input. Some very lightweight application logic also runs on the JDE HTML Server. Client requests do result in significant load on the JDE HTML servers since the JDE HTML servers make and manage database as well as network connections. The JDE HTML server's utilization of CPU and memory depends heavily on the number of interactive users using the server. Disk utilization is not a major factor in the sizing of the JDE HTML Server.

Typically, on the Oracle Linux Server, the number of interactive users per JVM when deployed on an Oracle VM should be capped to around 150 to 200 interactive users for optimal performance. This allows minimizing the heap size for each of the JVM thus reduces Full GC pause time. It is recommended to having multiple JVMs load balanced through Oracle HTTP Server, thus allowing reduced GC pause time and high availability of application. Due to high number of JVMs, typically an HTML server is configured with medium memory and medium computing power.

Oracle JDE Enterprise Server

The Oracle JDE Enterprise Server acts as the central point for serving requests for application logic. The Oracle JDE EnterpriseOne clients make requests for application processing. Depending upon the JDE environment used as well as user preferences, the input data is then processed and returned back to the client. The Call Object kernels running on the JDE Enterprise server are delegated the responsibility of processing end user application processing requests and the Security kernel handles the responsibility of ensuring authentication of the end users. The application processing is CPU intensive and the CPU frequency and number of cores available to the Enterprise server plays a large part affecting the performance and throughput of the system. As the number of interactive users requests grow, the memory requirements of the JDE Enterprise server also increases. This is also true for the batch (UBE) reports that the JDE Enterprise server processes.

The typical sizing recommendation for number of users per call object kernel on Oracle VM would be between 8 - 12 users/call object kernels and about 1 security kernel for every 50 interactive users. The in memory cache usage of call object kernels increase with user load so it is typical for the memory usage of individual call objects to increase with increase in user loads.

Oracle JDE Database Server

The JDE Database Server services the data requests made by both the JDE Enterprise and JDE HTML Servers. The JDE Database Server sizing depends on the type of reports being processed as well as the interactive user loads. Some JDE reports can be very Disk I/O intensive and depending on the kind of reports being processed, careful consideration needs to be given to disk layout. If the Oracle Database Server database has ample memory available to it and the memory is utilized to cache data it can benefit application performance by reducing disk I/O operations. The JDE Database server typically benefits from having faster disk and high memory allocation. The choice of Oracle RAC database for Oracle JD Edwards EnterpriseOne environment helps to provide high availability, increased transaction response time for concurrent UBE processes and enhanced scalability for JD Edwards EnterpriseOne online users. Moreover, the DATA and LOG disk must be configured as a physical disk to Guest OS configured for Oracle RAC nodes. In the present deployment, we have used virtual machines with 10 vCPU and 124 GB of physical memory for each of the Oracle RAC nodes.

Oracle JD Edwards EnterpriseOne Deployment Architecture

The deployment architecture of Oracle JD Edwards EnterpriseOne on is shown in Figure 13.

Figure 13 Deployment Architecture of JD Edwards EnterpriseOne on Oracle VM

The configuration for each of the Oracle JD Edward's Guest OS deployed on Oracle VM is shown in Figure 14.

Figure 14 Guest OS Configuration of Oracle JD Edwards EnterpriseOne on Oracle VM

The configuration presented in this document is based on the following main configuration components.

Table 2 Configuration Components

Oracle VM Servers

2x1 Cisco UCS B200M3 Blade Server equipped with two Intel Xeon E5-2690 2.9GHz Processors and 256GB of physical memory.

Oracle VM3.1.1 (Build 819)

JDE HTML Server 1 and 2

Each VM is configured with 8 vCPUs and 64 GB of physical memory

Oracle WebLogic 10.3.6 on Oracle Linux 5.8 (uek kernel)

JD Edwards HTML server code release 8.98.4.10 was deployed

JDE E1 Enterprise Server (interactive apps)

Guest OS is configured with 10 vCPUs and 64 GB of physical memory

JDE E1 Release 9.0, Update 2, with Tools Release 8.98.4.10, deployed on Oracle Linux 5.8 (uek kernel)

JDE E1 Enterprise Server (batch/ UBEs)

Guest OS is configured with 10 vCPUs and 64 GB of physical memory

JDE E1 Release 9.0, Update 2, with Tools Release 8.98.4.10, deployed on Oracle Linux 5.8 (uek kernel)

JDE Database Server (node1 and node2)

Each VM is configured with 10 vCPUs and 120 GB of physical memory

Oracle RAC 11.2.0.3

Deployment Server and Server Manager

Cisco UCS B200 M2 Blade Server equipped with two 4-core Intel Xeon X5620 2.4-GHz processors and configured with 24 GB of physical memory

Microsoft Windows 2008 R2 Enterprise Edition

Storage

EMC VNX 5300

Operating System (64 bit)

Oracle VM 3.1.1 for Oracle VM Host

Guest OS are configured with Oracle Linux 5.8 (uek kernel)

Test Client

LoadRunner 9.5 on Microsoft Windows 2003 Server


Infrastructure Setup

This section elaborates the infrastructure setup details used to deploy Oracle JD Edwards EnterpriseOne. The high-level workflow to configure the system is detailed in Figure 15.

Figure 15 Workflow to Setup Infrastructure for Oracle JD Edwards EnterpriseOne

Cisco Nexus Switch Configuration

This section elaborates on the procedure to configure Cisco Nexus 5548UP Switches. Some of the important configurations detailed in the section are:

First Time Setup of Cisco Nexus 5548UP Switch

Enabling Features and Global Configuration

Creating VSAN and Adding FC Interfaces

VLAN configuration

Virtual Port Channel (vPC) configuration

First Time Setup of Cisco Nexus 5548 UP Switch

The NX-OS setup should automatically start on initial boot and connection to the serial or console port of the switch. Enter the following commands to configure the Cisco Nexus 5548UP Switch:

1. Enter yes to enforce secure password standards: yes

2. Enter the password for the administrator (adminuser): <xxxxx>

3. Enter the password a second time to commit the password; <xxxxx>

4. Enter yes to enter the basic configuration dialog: yes

5. Create another login account (yes/no) [n]: Enter

6. Configure read-only SNMP community string (yes/no) [n]: Enter

7. Configure read-write SNMP community string (yes/no) [n]: Enter

8. Enter the switch name: Nexus5K-A Enter

9. Continue with out-of-band (mgmt0) management configuration? (yes/no) [y]: Enter

10. Mgmt0 IPv4 address: <Nexus A mgmt0 IP> Enter

11. Mgmt0 IPv4 netmask: <Nexus A mgmt0 netmask> Enter

12. Configure the default gateway? (yes/no) [y]: Enter

13. IPv4 address of the default gateway: <Nexus A mgmt0 gateway> Enter

14. Enable the telnet service? (yes/no) [n]: Enter

15. Enable the ssh service? (yes/no) [y]: Enter

16. Type of ssh key you would like to generate (dsa/rsa):rsa

17. Number of key bits <768-2048> :1024 Enter

18. Configure the ntp server? (yes/no) [y]: n Enter

19. NTP server IPv4 address: <NTP Server IP> Enter

20. Enter basic FC configurations (yes/no) [n]: Enter

21. Would you like to edit the configuration? (yes/no) [n]: Enter


Note Be sure to review the configuration summary before enabling it.


22. Use this configuration and save it? (yes/no) [y]: Enter


Note Configuration may be continued from the console or by using SSH. To use SSH, connect to the mgmt0 address of Nexus A.



Note Log in as user admin with the password previously entered.


Follow similar to configure the Cisco Nexus 5548 Switch B (Nexus5k-B).

Enabling Features and Global Configuration

To enable appropriate Nexus 5548 switch licensing, follow these steps on Cisco Nexus 5548 A - (Nexus5k-A), and Cisco Nexus 5548 B - (Nexus5k-B) separately:

1. Type "config t" to enter into the global configuration mode.

2. Type "feature lacp".

3. Type "feature fcoe".

4. Type "feature npiv".

5. Type "feature vpc".

6. Type "feature fport-channel-trunk".


Note FCoE feature needs to be enabled first before enabling npiv.


Verification: The figure given below, lists the enabled features on Cisco Nexus 5548 (show feature | include enabled").

Figure 16 Features Enabled in Cisco Nexus 5548

Creating VSAN and Adding FC Interfaces

To create VSAN and add FC interfaces, follow these steps on Cisco Nexus 5548 A - (Nexus5k-A), and Cisco Nexus 5548 B - (Nexus5k -B) separately:

1. Type "config t" to enter into the global configuration mode.

2. Type "vsan database".

3. Type "vsan 20 name JDE".

4. Type "vsan 20 interface fc1/29-32".

5. Type "y" on the "Traffic on fc1/29 may be impacted. Do you want to continue? (y/n) [n]".

6. Similarly type "y" for fc1/29 to fc1/32 interfaces.

Verification: The command sh vsan membership should lists ports fc1/29-32 under "vsan 20".

To configure the ports 29-32 as FC ports, follow these steps on Cisco Nexus 5548 A - (Nexus5k-A), and Cisco Nexus 5548 B - (Nexus5k-B) separately:

1. Type "config t" to enter into the global configuration mode.

2. Type "slot 1".

3. Type "interface fc 1/29-32".

4. Type "switchport mode F".

5. Type "no shut".

Follow the steps in the previous section to configure the Cisco Nexus Switch B.

Verification: The command "show interface brief" should list these interfaces as FC (Admin Mode "F").

VLAN Configuration

A VLAN is a group of end stations in a switched network that is logically segmented by function, project team, or application, irrespective of the physical locations of the users. VLANs have the same attributes as physical LANs, but the end stations can be grouped even if they are not physically located on the same LAN segment.

In this design the VLANs configured are listed in Table 3.

Table 3 VLAN Configuration

VLAN Information
VLAN Purpose
VLAN ID

VLAN0613

For external traffic

613

VLAN0760

For traffic between HTML Server, E1 Server and DB Server

760

VLAN0191

For Oracle Clusterware Interconnect Traffic

191


To configure the VLANs, follow these steps:

1. Type "config t" to enter into the global configuration mode.

2. From the global configuration mode, type "vlan613" and press "Enter".

3. Type <<name>> to enter a descriptive name for the VLAN.

4. Type "vlan760".

5. Type <<name>>.

6. Type "vlan191".

7. Type <<name>>.

8. Type "Interface ethernet1/17-19" (make sure to choose the Ethernet interfaces where Fabric Interconnects are connected).

9. Type "switchport mode trunk".

10. Type "switchport trunk allowed vlan 613,760, 191".

11. Type "exit"

Verification: The command show vlan should list the vlans and interfaces assigned to it. Or, the command show run interface <interface name> should show the configuration for a given interface or port channel. The figure given below, lists the executed command:

Nexus5k-A# sh vlan
 
 
VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Po1, Eth1/7, Eth1/8, Eth1/9
                                                Eth1/10, Eth1/11, Eth1/14
                                                Eth1/15, Eth1/16, Eth1/17
                                                Eth1/18, Eth1/19, Eth1/20
                                                Eth1/21, Eth1/22, Eth1/23
                                                Eth1/24, Eth1/25, Eth1/26
                                                Eth1/31, Eth1/32
191  Private                          active  Eth1/17, Eth1/19
613  VLAN0613                    active  Eth1/17, Eth1/19, Eth1/32
760  Public                           active  Eth1/17, Eth1/19, Eth1/32
 
 
 
 

Note Interface Eth1/32 is configured for outside traffic.


Virtual Port Channel (vPC) Configuration

A virtual port channel (vPC) allows links that are physically connected to two different Cisco Nexus 5000 Series devices to appear as a single port channel by a third device.

Cisco Nexus 5548UP vPC configurations with the vPC domains and corresponding vPC names and IDs for the present setup is listed in Table 4 and Table 5. To provide Layer 2 and Layer 3 switching, a pair of Cisco Nexus 5548UP Switches with upstream switching are deployed, providing high availability in the event of failure to Cisco UCS to handle management, application, and Network storage data traffic. In the Cisco Nexus 5548UP switch topology, a single vPC feature is enabled to provide high availability, faster convergence in the event of a failure, and greater throughput.

Table 4 vPC Mapping for Cisco Nexus 5548A

vPC Domain
vPC Name
vPC ID
LAN UpLink Ports
Connected Components

100

vPC-Public1

110

FI A - eth1/9

N5KA - eth1/7

FI-A and N5kA provide vPC for all public and storage network traffic

100

vPC-Public2

111

FI B- eth1/9

N5KA - eth1/8

FI-B and N5kA provide vPC for all public network traffic


Table 5 vPC Mapping for Cisco Nexus 5548B

vPC Domain
vPC Name
vPC ID
LAN UpLink Ports
Connected Components

100

vPC-Public1

110

FI A - eth1/10

N5KB - eth1/7

FI-A and N5kB provide vPC for all public & Storage network traffic

100

vPC-Public2

111

FI B- eth1/10

N5KB - eth1/8

FI-B and N5kB provide vPC for all public and Storage network traffic



Note Configuration of ports are listed in Table 4 and Table 5 are for demonstration. Customers can configure ports as per their environment and availability. Follow the network diagram (Figure 13)


In the vPC design table, a single vPC domain, Domain 100, is created across Cisco Nexus 5548UP member switches to define vPCs to carry specific network traffic. This topology defines two vPCs. These vPCs are managed within the Cisco Nexus 5548UP, which connects Cisco UCS Fabric Interconnects and the EMC storage system.

To create vPC, follow these steps for both Cisco Nexus 5548 A and Cisco Nexus 5548 B separately with configuration details as per Table 4 and Table 5.

1. In the global configuration mode, type "vpc domain 100".

2. Type "role priority 1000".

3. Type "peer-keepalive destination 10.x.x.x". (This IP is the Nexus 5548 B Management IP)

4. Type "int port-channel 100".

5. Type "switchport mode trunk".

6. Type "switchport trunk allowed vlan all".

7. Type "vpc peer-link".

8. Type "int ethernet 1/3" (peer link port).

9. Type "switchport mode trunk".

10. Type "switchport trunk allowed vlan all".

11. Type "channel-group 100 mode active".

12. Type "Exit".


Note vPC domain creation command from Step 1 to 12 has to be executed only in one Nexus 5548 UP.


13. Type "int port-channel 110". (Public & Storage Network Traffic)

14. Type "switchport mode trunk".

15. Type "switchport trunk allowed vlan191, 760, 613".

16. Type "vpc 110".

17. Type "int ethernet 1/7".

18. Type "channel-group 110 mode active".

19. Type "switchport mode trunk".

20. Type "int port-channel 111". (Public & Storage Network Traffic)

21. Type "switchport mode trunk".

22. Type "switchport trunk allowed vlan191, 760, 613".

23. Type "vpc 111".

24. Type "int ethernet 1/8".

25. Type "channel-group 111 mode active".

26. Type "switchport mode trunk".

Verification: When configuring the Cisco Nexus 5548UP with vPCs, make sure that the status for all vPCs is "Up" for connected Ethernet ports by running the commands (show vpc) from the CLI on the Cisco Nexus 5548UP Switch.

Figure 17 vPC Status on Cisco Nexus 5548UP A and Nexus 5548UP B

For more information on vPC configuration for Cisco Nexus 5548UP switches, see:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/configuration_guide_c.07-543563.html

Setting Up Jumbo Frames on Cisco Nexus 5500

Jumbo frames with an mtu size 9000 have to be setup on Cisco Nexus 5500. Oracle Interconnect traffic under normal conditions does not go to the northbound switch like the Cisco Nexus 5500, as all the private vNICs are configured in Fabric B. However if there is a partial link or IOM failure, the private interconnect traffic has to go to the immediate northbound switch (Nexus 5500 in our case) to reach Fabric B.

Follow these commands to configure Jumbo Frames on Cisco Nexus 5500 Fabric A Switch:

Nexus5k-A# conf terminal 
Enter configuration commands, one per line.  End with CNTL/Z.
Nexus5k-A(config)# class-map type network-qos class-platinum
Nexus5k-A(config-cmap-nq)# exit
Nexus5k-A(config)# policy-map type network-qos jumbo
Nexus5k-A(config-pmap-nq)# class type network-qos class-default
Nexus5k-A(config-pmap-nq-c)# mtu 9216
Nexus5k-A(config-pmap-nq-c)# multicast-optimize 
Nexus5k-A(config-pmap-nq-c)# exit
Nexus5k-A(config-pmap-nq)# system qos
Nexus5k-A(config-sys-qos)# service-policy type network-qos jumbo
Nexus5k-A(config-sys-qos)# exit
Nexus5k-A(config)# copy running-config startup-config 
[########################################] 100%
Nexus5k-A(config)#

Similarly, enable Jumbo Frames on Nexus 5500 Fabric B Switch.

Cisco Unified Computing System Configuration

This section details the Cisco UCS configuration that is performed as part of the infrastructure build for deployment of Oracle JD Edwards EnterpriseOne. The racking, power, and installation steps can be referred from the Installation Guide, as it is beyond the scope of this document. For more details on each step refer:

Cisco UCS 5108 Server Chassis Installation Guide:

http://www.cisco.com/en/US/docs/unified_computing/ucs/hw/chassis/install/install.html

Cisco Unified Computing System CLI Configuration Guide:

http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/cli/config/guide/2.1/b_UCSM_CLI_Configuration_Guide_2_1_chapter_010.html

Cisco UCS Manager GUI Configuration Guide: http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.1/b_UCSM_GUI_Configuration_Guide_2_1.html

Validate Installed Firmware

To login to Cisco UCS Manager, follow these steps:

1. Open the Web browser with the Cisco UCS 6248UP Fabric Interconnect cluster address.

2. Click Launch to download the Cisco UCS Manager software.

3. You might be prompted to accept security certificates; accept as necessary.

4. In the login page, enter "admin" for both username and password text boxes.

5. Click Login to access the Cisco UCS Manager software.

6. Click Equipment tab and then Installed Firmware tab in the right pane.

7. Verify the Firmware installed. The firmware during the deployment was 2.1(1a).

For more information on Firmware Management, see: http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/firmware-mgmt/gui/2.1/b_GUI_Firmware_Management_21_chapter_0100.html

Verification: The Installed Firmware should be displayed as 2.1(1a) as shown in Figure 18.

Figure 18 Verifying the Installed Firmware Version

Chassis Discovery Policy

To edit the chassis discovery policy, follow these steps:

1. Navigate to the Equipment tab in the left pane of the UCS Manager.

2. In the right pane, click the Policies tab.

3. Under Policies, click Global Policies, change the Chassis Discovery Policy to 4-link.

4. Click Save Changes in the bottom right corner.

Verification: The chassis discovery policy configured to 8-link is displayed in Figure 19.

Figure 19 Displaying the Chassis Discovery Policy Configuration

Enabling Network Components

To enable Fibre Channel, servers, and uplink ports, follow these steps:

1. Select the Equipment tab on the top left of the UCS Manager window.

2. Select Equipment > Fabric Interconnects > Fabric Interconnect A (primary) > Fixed Module.

3. Expand the Unconfigured Ethernet Ports section.

4. Select ports 1-8 that are connected to the UCS chassis and right-click on them and select Configure as Server Port.

5. Click Yes to confirm, and then click OK to continue.

6. Select ports 9 and 10. These ports are connected to the Cisco Nexus 5548 switches for NFS traffic. Right-click on them and select Configure as Uplink Port

7. Click Yes to confirm, and then click OK to continue.

8. Select ports 11 and 12. These ports are connected to the Cisco Nexus 5548 switches. Right-click on them and select Configure as Uplink Port.

9. Click Yes to confirm, and then click OK to continue.

10. On the FI Expansion Module configure port 15 and port 16 as Fiber Channel.

11. Select Equipment > Fabric Interconnects > Fabric Interconnect A (primary).

12. Right click, and select Set FC End-Host Mode to put the Fabric Interconnect in Fiber Channel Switching Mode.

13. Click Yes to confirm.

14. A message displays stating that the "Fiber Channel End-Host Mode has been set and the switch will reboot". Click OK to continue. Wait until the UCS Manager is available again and log back into the interface.

15. Re-execute Step 2 to 14 for Fabric Interconnect B.

Verification: Check if all configured links show their status as "up" as shown in Figure 20 for Fabric Interconnect A. This can also be verified on the Cisco Nexus switch side by running "show int status" and all the ports connected to the Cisco UCS Fabric Interconnects are shown as "up".


Note The FC ports are shown as enabled with VSAN default ID "1". Once we create VSAN id for the present deployment (vsan20 as created in Nexus 5548 FC Configuration steps), we would re-enable the FC port, with VAN16.


Figure 20 Enabling the Server Ports on Cisco UCS Fabric Interconnect

Creating MAC Address Pools

To create MAC Address pools, follow these steps:

1. Select the LAN tab on the left of the UCS Manager window.

2. Under Pools > root.


Note Two MAC address pools will be created, one each for Fabric A and Fabric B.


3. Right-click on MAC Pools under the root organization and select Create MAC Pool to create the MAC address pool for Fabric A.

4. Enter "MAC-JDE-FIA" for the name of the MAC pool for Fabric A.

5. Enter a description of the MAC pool in the Description text box (optional).

6. Click Next to continue.

7. Click Add to add the MAC address pool.

8. Specify a starting MAC address for Fabric A.


Note The default is fine, but it is recommended to change the pool address as per the deployment and also to differentiate between MAC address for Fabric A and Fabric B. Presently we have configured it as (DE:25:B5:0A:00:00).


9. Specify the size as "24" for the MAC address pool for Fabric A. (MAC address Size "24" is just for demonstration and not a hard limit for MAC address Pool)

10. Click OK.

11. Click Finish.

12. A pop-up message box appears, click OK to save changes.

13. Right-click on MAC Pools under the root organization and choose Create MAC Pool to create the MAC address pool for Fabric B.

14. Enter "MAC-JDE-FIB" for the name of the MAC pool for Fabric B.

15. Enter a description of the MAC pool in the Description text box (optional).

16. Click Next to continue.

17. Click Add to add the MAC address pool.

18. Specify a starting MAC address for Fabric B.


Note The default is fine, but it is recommended to change the pool address as per the deployment and also to differentiate between MAC address for Fabric A and Fabric B. Presently we have configure it as (DE:25:B5:0B:00:00)


19. Specify the size as "24" for the MAC address pool for Fabric B.

20. Click OK.

21. Click Finish.

22. A pop-up message box appears; click OK to save changes and exit.

Verification: Select LAN tab > Pools > root. Select MAC Pools and it expands to show the MAC pools created. On the right pane, details of the MAC pools are displayed as shown in Figure 21.

Figure 21 MAC Pool Details

Creating WWPN Pools

To create WWPN pools, follow these steps:

1. Select the SAN at the top left of the UCS Manager window.

2. Select WWPN Pools > root.


Note Two WWPN pools will be created, one each for Fabric A and Fabric B.


3. Right-click on WWPNPools, and select Create WWPN Pool.

4. Enter "ovm_wwpn_A" as the name for the WWPN pool for Fabric A.

5. Enter a description of the WWPN pool in the Description text box (optional).

6. Click Next.

7. Click Add to add a block of WWPNs.

8. Enter "20:00:ED:25:B5:A0:00:00" as the starting WWPN in the block for Fabric A.


Note It is recommended to change the WWPN prefix as per the deployment. This would help in identifying WWPNs initiated from Fabric A or Fabric B.


9. Set the size of the WWPN block to "24". (WWPN Pool Size "24" is just for demonstration and not a hard limit for WWPN Pool)

10. Click OK to continue.

11. Click Finish to create the WWPN pool.

12. Click OK to save changes.

13. Right-click on the WWPN Pools and select Create WWPN Pool.

14. Enter "ovm_wwpn_B" as the name for the WWPN pool for Fabric B.

15. Enter a description of the WWPN pool in the description text box. This is optional; you can choose to omit the description.

16. Click Next.

17. Click Add to add a block of WWPNs.

18. Enter "20:00:ED:25:B5:B0:00:00" as the starting WWPN in the block for Fabric B.

19. Set the size of the WWPN block to "24".

20. Click OK to continue.

21. Click Finish to create the WWPN pool.

22. Click OK to save changes and exit.

Verification: As displayed in Figure 22, the new name with the 24 block size.

Figure 22 WWPN Pool Details

Creating WWNN Pools

To create WWNN pools, follow these steps:

1. Select the SAN tab at the top left of the UCS Manager window.

2. Select Pools > root.

3. Right-click on WWNN Pools and click Create WWNN Pool.

4. Enter "OVM_JDE_WWNN" as the name of the WWNN pool.

5. Enter a description of the WWNN pool in the Description text box (optional).

6. Click Next to continue.

7. A pop-up window "Add WWN Blocks" appears; click Add button at the bottom of the page.

8. A pop-up window "Create WWN Blocks" appears; set the size of the WWNN block to "24".

9. Click OK to continue.

10. Click Finish.

11. Click OK to save changes and exit.

Verification: The new name with the 24 block size displays in the right panel when WWNN pools is selected on the left panel, as shown in Figure 23. (WWNN Pool Size "24" is just for demonstration and not a hard limit for WWNN Pool).

Figure 23 WWNN Pool Details

Creating UUID Suffix Pools

To create UUID suffix pools, follow these steps:

1. Select the Servers tab on the top left of the UCS Manager window.

2. Select Pools > root.

3. Right-click on "UUID Suffix Pools" and select Create UUID Suffix Pool.

4. Enter the name the UUID suffix pool as "OVMU_UID".

5. Enter a description of the UUID suffix pool in the Description text box (optional).

6. Prefix is set to "derived" by default. Do not change the default setting.

7. Click Next to continue.

8. A pop-up window "Add UUID Blocks" appears. Click Add button at the bottom of the window to add a block of UUID suffixes.

9. The "Form" field will be in default setting. Do not change the "From" field.

10. Set the size of the UUID suffix pool to "24".

11. Click OK to continue.

12. Click Finish to create the UUID suffix pool.

13. Click OK to save changes and exit.

Verification: Ensure that the UUID suffix pools created are displayed as shown in Figure 24.

Figure 24 UUID Suffix Pool Details

Creating VLANs

To create VLANs, follow these steps:

1. Select the LAN tab on the left of the UCS Manager window.


Note Three VLANs will be created -Management Traffic, Data traffic and Oracle RAC database inter-node private traffic as listed in Table 3.


2. Right-click on the "VLANs" in the tree, and click Create VLAN(s).

3. Enter <<name>> as the name of the VLAN (for example, 760). This name will be used for traffic management.

4. Keep the option "Common/Global" selected for the scope of the VLAN.

5. Enter a VLAN ID for the management VLAN. Keep the sharing type as "none".

6. Similarly create other three VLANs.

Verification: Select LAN tab > LAN Cloud > VLANs. Open VLANs and all of the created VLANs are displayed. The right pane gives the details of all individual VLANs as shown in Figure 25.

Figure 25 Details of Created VLANs

Creating Uplink Ports Channels

To create uplink port channels to Cisco Nexus 5548 switches, follow these steps:

1. Select the LAN tab on the left of the UCS Manager window.

2. Expand the "Fabric A" tree.

3. Right-click on the "Port Channels" and click Create Port Channel.

4. Enter "110" as the unique ID of the port channel.

5. Enter "Public-Private1" as the name of the port channel.

6. Click Next.

7. Select ports 1/9 and 1/10 to be added to the port channel.

8. Click >> to add the ports to the Port Channel.

9. Click Finish to create the port channel.

10. A pop-up message box appears, click OK to continue.

11. In the left pane, choose the newly created port channel.

12. In the right pane under "Actions", choose Enable Port Channel option.

13. In the pop-up box, click Yes, and then click OK to save changes.

14. Expand the "Fabric B" tree.

15. Right-click on the "Port Channels" and click Create Port Channel.

16. Enter "111" as the unique ID of the port channel.

17. Enter "Public_Private2" as the name of the port channel.

18. Click Next.

19. Select ports 1/9 and 1/10 to be added to the Port Channel.

20. Click >> to add the ports to the Port Channel.

21. Click Finish to create the port channel.

22. A pop-up message box appears, click OK to continue.

23. In the left pane, click the newly created port channel.

24. In the right pane under "Actions", choose Enable Port Channel option.

25. In the pop-up box, click Yes, and then click OK to save changes.

Verification: Select LAN tab > LAN Cloud. On the Right Pane, select the LAN Uplinks and expand the Port channels listed as shown in Figure 26.


Note In order for the Fabric Interconnect Port Channels to get enabled, we have already configured vPC in "Cisco Nexus Switch Configuration" section.


Figure 26 Details of Port Channels

Creating VSANs

All the servers under JD Edwards EnterpriseOne deployment on UCS with EMC VNX 5300 are booted from SAN. The following section elaborates on VSAN creation which we configured under Cisco Nexus Switch Configuration configuration

To create VSANs, follow these steps:

1. Select the SAN tab at the top left of the UCS Manager window.

2. Expand the SAN cloud tree.

3. Right-click on the "VSANs" and click Create VSAN.

4. Enter "OVM_VSAN" as the VSAN name

5. Enter "20" as the VSAN ID.

6. Enter "20" as the FCoE VLAN ID.

7. Click OK to create the VSANs.

Verification: Select SAN tab >SAN Cloud >VSANs on the left panel. The right panel displays the created VSANs as shown in Figure 27.

Figure 27 Details of VSANs

SAN Boot Policy Configuration

In the present deployment, the UCS Servers are booted from SAN (EMC VNX5300). With boot from SAN, the OS image resides on the SAN and the server communicates with the SAN through a host bus adapter (HBA). The HBAs BIOS contain the instructions that enable the server to find the boot disk. After power on self test (POST), the server hardware component fetches the boot device that is designated as the boot device in the hardware BOIS settings. When the hardware detects the boot device, it follows the regular boot process.

Modifying Service Profile for Boot Policy

In this setup, vhba0 and vhba1 are used both for SAN Boot and JD Edwards EnterpriseOne Installation. Storage SAN WWPN ports will be connected in the boot policy as shown:

To modify the Service Profile for boot policy, follow these steps:

1. Login to the UCS Manager. Click Servers tab > Policies > Boot Policies and the click Add. A pop-up window "Create Boot Policy" appears.

2. Enter the name as "JDE_OVM_BootPol" in the "Name" text box and in the Description text box enter "for JD Edwards" and ensure that the check box "Reboot on Boot Order Change" is checked.

3. Add the first target as CD-ROM, as this will enable you to install Operating System through KVM Console.

4. Click Add SAN Boot on the vHBAs section; in the "Add SAN Boot" pop-up window, type "vHBA0" and select the type as Primary and click OK. This will be the SAN Primary Target.

5. Click Add SAN Boot Target to add a target to the SAN Boot Primary in the vHBAs window. In the "Add SAN Boot Target" pop-up window, type "0" in the "Boot Target LUN". Enter "50:06:01:60:3e:a0:05:47" in the "Boot Target WWPN" and select the type as "Primary" and then click OK.

6. To add another target to the SAN Boot Primary, click Add to add another SAN Boot Target in the vHBAs window; in the "Add SAN Boot Target" pop-up box, type "0" in the Boot Target LUN; type "50:06:01:68:3e:a0:05:47" in the Boot Target WWPN and ensure that the type selected is ""Secondary" is already selected and greyed out.and click OK.


Note These WWPNs are from storage SPB0 / SPA0 ports (please refer Figure 31: San Zoning Configuration).


7. Similarly for the SAN Secondary, Click Add SAN Boot in the vHBAs window; in the "Add SAN Boot" pop-up window, type "vHBA1" and select the type as "Secondary" and then click OK.

8. Click Add SAN Boot Target to add a target to the SAN Boot Secondary (vHBA1) in the vHBAs window. In the "Add SAN Boot Target" pop-up window, type "0" in the "Boot Target LUN". Enter "50:06:01:61:3E:A0:05:47" in the "Boot Target WWPN" and select the type as "Primary" and then click OK.

9. To add another target to the SAN Boot Secondary, click Add to add another SAN Boot Target in the vHBAs window; in the "Add SAN Boot Target" pop-up box, type "0" in the Boot Target LUN; type "50:06:01:69:3E:A0:05:47" in the Boot Target WWPN and ensure that the type selected is "Secondary" which would have already been greyed out and click OK.

10. Click Save Changes to save all the settings. The Boot Policy window in UCS Manager is as shown in Figure 28.

Figure 28 Boot Policy Service Profile


Note During OVM Installation, we would change the boot policy so that only one path is available during the boot time. Also make sure that only one path is available before the installation process. Once the installation is complete, this may be reverted back to its full paths.


BIOS Policy

BIOS policy is one of the features of Cisco UCS Service Profiles which enable users to incorporate similar BIOS settings all across the deployed servers. It helps to create application specific BIOS setting, For instance in a virtualized environment we would enable VT for direct IO wherein in the non-virtualized environment we would leave that as default. This helps a consistent configuration and the Administrators do not need to interrupt the boot process on each server to alter the BIOS setting. This helps faster deployment as users do not have to use the F2 option to change BIOS of hundreds of servers deployed in a Data Center. The BIOS policy configured for JDE deployment is shown in Figure 29.

Figure 29 BIOS Setting for CPU Performance

Figure 30 BIOS setting for Physical Memory Performance


Note Disable I/O MMU Virtualization Settings (VT-d). See: http://docs.oracle.com/cd/E27300_01/E27307/html/vmrns-bugs.html


UCS Service Profile Configuration

An important aspect of configuring a physical server in a Cisco UCS 5108 chassis is to develop a service profile through Cisco UCS Manager. Service profile is an extension of the virtual machine abstraction applied to physical servers. The definition has been expanded to include elements of the environment that span the entire data center, encapsulating the server identity (LAN and SAN addressing, I/O configurations, firmware versions, boot order, network VLAN, physical port, and quality-of-service [QoS] policies) in logical "service profiles" that can be dynamically created and associated with any physical server in the system within minutes rather than hours or days. The association of service profiles with physical servers is performed as a simple, single operation. It enables migration of identities between servers in the environment without requiring any physical configuration changes and facilitates rapid bare metal provisioning of replacements for failed servers.

Service profiles can be created in several ways:

Manually—Create a new service profile using the Cisco UCS Manager GUI.

From a Template—Create a service profile from a template.

By Cloning—Cloning a service profile creates a replica of a service profile. Cloning is equivalent to creating a template from the service policy and then creating a service policy from that template to associate with a server.

In the present scenario we created a Service profile updating template and thereafter instantiated service profile through the template. A service profile template parameterizes the UIDs that differentiate one instance of an otherwise identical server from another. Templates can be categorized into two types:

Initial Template—The initial template is used to create a new server from a service profile with UIDs, but after the server is deployed, there is no linkage between the server and the template, so changes to the template will not propagate to the server, and all changes to items defined by the template must be made individually to each server deployed with the initial template.

Updating Template—An updating template maintains a link between the template and the deployed servers, and changes to the template (most likely to be firmware revisions) cascade to the servers deployed with that template on a schedule determined by the administrator.

Service profiles, templates, and other management data is stored in high-speed persistent storage on the Cisco UCS Fabric Interconnects, with mirroring between fault-tolerant pairs of Fabric Interconnects.

Create vNIC Templates

To configure the vNIC template, follow these steps:

1. Click the LAN tab.

2. Filter on policies and select vNIC templates. Three templates are created; Public Network template, Private Network template and VM Management Network template. The Private network is for the internal Heart Beat and message transfers between Oracle Nodes while Public network for JD Edwards inter-tier communication. The Management Network is used for Oracle VM Management and VM migration.

3. Define Name and Description of vNIC template.

4. Pin Public and Management vNIC template to FI-A and Private network vNIC template to FI-B.

5. Enable Failover for each of the Fabric option.

6. Select VLANs; Public for Public vNIC Template, Private for Private vNIC template and Management VLAN for Oracle VM Management Network.

7. Select MAC Pool that was created in the previous steps.

8. Define MTU. For Oracle Private Network we would use 9000 as the MTU size whereas MTU size of 1500 is used for Management & Public Network. Figure 31, Figure 32, and Figure 33 show the created vNIC templates.

Figure 31 vNIC Template for Public Network

Figure 32 vNIC Template for Private Network

Figure 33 vNIC Template for Management Network

Create vHBA Templates

To create a HBA template, follow these steps:

1. Click the SAN tab.

2. Filter out policies, right-click the vHBA templates and choose Create vHBA Template.

3. Create two Templates, one each for Fabric A and Fabric B.

4. Select OVM_VSAN, as created in creating VAN step. (This is configured with VAN ID 20)

5. Select WWPN pool as ovm_wwpn_A for Fabric A vHBA template and ovm_wwpn_B for Fabric B template

6. The options selected are shown in Figure 34.

Figure 34 vHBA Template

Creating Service Profile Templates

To create service profile templates in UCSM, follow these steps:

1. Click the Servers tab at the top left of the UCS Manager window.

2. Choose Service Profile Templates > root. Choose Create Service Profile Template under the Actions tab.

3. The Create Service Profile Template window appears.

4. Identify the Service Profile Template section

a. Enter the name of the service profile template as "SPTemplate-JDE".

b. Select the type as "Updating Template". In the event of change in server configuration, updating template helps to update all the service profiles attached to updating template.

c. In the UUID section, select "OVMU_UID" as the UUID pool.

d. Click Next to continue to the next section.

5. Network Section

a. Keep default for the Local Storage option.

b. Select the option "Expert" for the field "How would you like to configure LAN connectivity".

c. In the WWNN Assignment field, select "OVM_JDE_WWNN".

d. Select "use vNIC template".

e. Define vNIC name as vNIC_Public and select vNIC template as vNIC_Public_FIA

f. Click OK.

g. Similarly create vNIC for Private and Management Network, with appropriate vNIC template as created in previous step.

6. Storage Section

a. Keep default for the Local Storage option.

b. Select the option "Expert" for the field "How would you like to configure SAN connectivity".

c. In the WWNN Assignment field, select "OVM_JDE_WWNN".

d. Click Add.

e. Define vHBA name as vHBA0.

f. Select "use vHBA template".

g. Select vHBA template vHBA0.

h. Select Adaptor Policy as Linux.

i. Click OK.

j. Repeat the above steps for vHBA1 with vHBA template as vHBA-B.

7. Zoning section

a. Leave as default and move the next section

8. vNIC/vHBA Placement section

a. Restore the default setting as "Let System Perform Placement" in the Select Placement field.

b. Ensure that all the vHBAs are created.

c. Click Next to continue.

9. Server Boot Order section

a. Select JDE_Boot_Pol as created in the previous steps.

10. Select default settings for Maintenance Policy and click Next.

11. In Server Assignment section, create a Server Pool (JDE_ServerPool) for Oracle JD Edwards deployment and click Next.

A server pool contains a set of servers. These servers typically share the same characteristics. Those characteristics can be their location in the chassis, or an attribute such as server type, amount of memory, local storage, type of CPU, or local drive configuration. You can manually assign a server to a server pool, or use server pool policies and server pool policy qualifications to automate the assignment.

If your system implements multi-tenancy through organizations, you can designate one or more server pools to be used by a specific organization. For example, a pool that includes all servers with two CPUs could be assigned to the Marketing organization, while all servers with 64 GB memory could be assigned to the Finance organization.

For Oracle JD Edwards deployment, this feature can help in assigning servers to a Server Pool with specific CPU and memory for batch workload or for interactive workload. With the help of UCS XML API we can easily integrate Oracle Enterprise Manager (OEM) and provide automatic failover during the failure of server assigned to specific Server Pool. For instance, in the event of a batch server failure assigned to a Batch server pool, OEM can initiate a call to UCS XML scripts and a server from Spare Server Pool can move to a Batch Server Pool and can be automatically assigned to the Service Profile of Failed Batch Server.

12. Under Operational Policies, select the BIOS Policy (JDE_BIOS) created under BIOS Policy section.

13. Click Finish to complete the creation of Service profile template. The Figure 35 shows the created Service Profile Template.

Figure 35 Service Profile Template

Creating Service Profile from the Template and Associating it to Cisco UCS Blade Server

To create a service profile from the template and associating it to a blade, follow these steps:

1. Select the Servers tab at the top left of the UCS Manager window.

2. Select Service Profile Templates > root > Sub-Organizations > Service Template JD Edwards Template.

3. Click Create Service Profiles From Template in the Actions tab of the right pane of the window.

4. Enter "JDE-OVM-" in the Naming Prefix text box and the number as "2".

5. Click OK to create service Profile. This would create two service Profiles each named as JDE-OVM-1 & JDE-OVM-2 respectively.

6. Once the Service Profile is created, it automatically associates to the available server in the Server Pool.

Figure 36 Successful Association of the Service Profile

Creation of Zoneset and Zones on Cisco Nexus 5548UP Switch

Once the server association to a Service Profile is done, we can see the published WWPNs of vhab0 and vhba1 on Cisco Nexus 5548 A, and 5548 B respectively for that specific server. This section details the steps to configure zoneset and zones on each of the Nexus 5548 switch for server Oracle VM Similar the same steps need to be followed for second server on the Oracle JD Edwards deployment with Oracle VM on Cisco Nexus 5548 A, and 5548 B.

Follow these steps to configure zoneset and zones on Nexus 5548UP Switches:

1. Identify Storage and Server WWPNs

 
 
Nexus5k-A# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Nexus5k-A(config)# sh flogi database
--------------------------------------------------------------------------------
INTERFACE        VSAN    FCID           PORT NAME               NODE NAME
--------------------------------------------------------------------------------
fc2/1            20    0x7801ef  50:06:01:60:3e:a0:05:47 50:06:01:60:be:a0:05:47
fc2/2            20    0x7800ef  50:06:01:68:3e:a0:05:47 50:06:01:60:be:a0:05:47
fc2/3            20    0x780002  20:4d:54:7f:ee:56:ca:00 20:14:54:7f:ee:56:ca:01
fc2/3            20    0x780005  20:00:00:25:b5:de:0a:0e 20:00:00:25:b5:de:de:0e
fc2/4            20    0x780000  20:4e:54:7f:ee:56:ca:00 20:14:54:7f:ee:56:ca:01
fc2/5            16    0x6d0000  50:0a:09:83:9d:93:40:7f 50:0a:09:80:8d:93:40:7f
fc2/6            16    0x6d0001  50:0a:09:84:8d:93:40:7f 50:0a:09:80:8d:93:40:7f
fc2/7            16    0x6d0002  20:4f:54:7f:ee:56:ca:00 20:10:54:7f:ee:56:ca:01
fc2/7            16    0x6d0003  20:00:00:25:b5:de:aa:0e 20:de:00:25:b5:00:00:0e
fc2/7            16    0x6d0005  20:00:00:25:b5:de:aa:0a 20:de:00:25:b5:00:00:0a
fc2/8            16    0x6d0004  20:00:00:25:b5:de:aa:0c 20:de:00:25:b5:00:00:0c
fc2/8            16    0x6d0006  20:00:00:25:b5:de:aa:09 20:de:00:25:b5:00:00:09
fc2/8            16    0x6d000a  20:50:54:7f:ee:56:ca:00 20:10:54:7f:ee:56:ca:01
fc2/8            16    0x6d000b  20:00:00:25:b5:de:aa:0f 20:de:00:25:b5:00:00:0f
 
 
Total number of flogi = 14.

Note The storage and JDE-OVM servers and EMC VNX controller WWPNs and are marked in bold.


2. Create zone and zoneset

Nexus5k-A(config)# zone name jde-ovm1-vhba0 vsan 20
Nexus5k-A(config-zone)# member pwwn 50:06:01:60:3e:a0:05:47
Nexus5k-A(config-zone)# member pwwn 50:06:01:68:3e:a0:05:47
Nexus5k-A(config-zone)# member pwwn 20:00:00:25:b5:de:0a:0e
Nexus5k-A(config-zone)# zoneset name jde-ovm-n5kA vsan 20
Nexus5k-A(config-zoneset)# member jde-ovm1-vhba0
Nexus5k-A(config-zoneset)# zoneset activate name jde-ovm-n5kA vsan 20
WARNING: You are trying to activate zoneset jde-ovm-n5kA, which is different from 
current active zoneset jdeovm-n5ka. Do you want to continue? (y/n) [n] y
Zoneset activation initiated. check zone status
Nexus5k-A(config)# sh zoneset active vsan 20
zoneset name jde-ovm-n5kA vsan 20
  zone name jde-ovm1-vhba0 vsan 20
  * fcid 0x7801ef [pwwn 50:06:01:60:3e:a0:05:47]
  * fcid 0x7800ef [pwwn 50:06:01:68:3e:a0:05:47]
  * fcid 0x780005 [pwwn 20:00:00:25:b5:de:0a:0e]
Nexus5k-A(config)# copy r s
[########################################] 100%
Copy complete, now saving to disk (please wait)

Similar configuration has to be done on Nexus 5548 B for vhba1 of JDE-OVM-1 Service Profile. Once we configure zones and zoneset on Nexus 5548 B we would see following configuration.

Nexus5k-B(config)# sh zoneset active vsan 20
zoneset name jde-ovm1-n5kB vsan 20
  zone name jde-ovm1-vhba1 vsan 20
  * fcid 0xdf01ef [pwwn 50:06:01:61:3e:a0:05:47]
  * fcid 0xdf00ef [pwwn 50:06:01:69:3e:a0:05:47]
  * fcid 0xdf0005 [pwwn 20:00:00:25:b5:de:b0:0e]

Configuring Connectivity for EMC VNX Storage

The present JD Edwards deployment leverages EMC VNX5300 storage, connected to the Cisco UCS system. It leverages the EMC FAST Cache as well as EMC FAST Virtual Pool capability.

As Edwards Enterprise One Server has very frequent and random access to the database; the Oracle RAC data files would require very fast read/write operations. For such access patterns a storage pool with Flash disks also known as Solid State Disks (SSD) are chosen with RAID level 5. SSD disks provide faster response time for random access IO patterns. The Data LUN would be configured as Physical disk to the Oracle RAC Guest VMs Oracle RAC log files) involves sequential write intensive operations, hence SAS drives with RAID level 10 is chosen. The log LUN would be configured as physical disk to the Oracle RAC Guest VMs Oracle RAC ASM luns is configured with RAID level 10 and attached as physical disk to Oracle RAC guest VMs. For boot LUNs, RAID level 5 is chosen for reliability. All the boot luns would be configured as Virtual Disk to each of the Guest VMs. The installation directory for HTML, JDE E1 and Database resides on RAID5, using multiple SAS drives. For Backing up LUNs, NL-SAS drives are chosen for their high capacity and lower cost characteristics that align with the backup model.

Figure 37 illustrates the disk layout configured for Oracle JD Edwards on Cisco UCS with Oracle VM.

Figure 37 Disk layout

LUNs carved from the EMC VNX storage forming the disk layout for the Oracle JD Edwards EnterpriseOne deployment are shown in Table 6.

Table 6 LUN Configuration for JD Edwards EnterpriseOne Deployment

Name
RAID Type
User Capacity (GB)
Host Information
Description

Jde-ovm1-os-lun

RAID 5

40

jde-ovm1

Boot LUN for OVM Server1

Jde-ovm2-os

RAID 5

40

jde-ovm2

Boot LUN for OVM Server2

Cluster-lun

RAID 5

30

jde-ovm1/ jde-ovm2

Cluster LUN shared across OVM Servers for OVM Cluster Pool

Ovm1-data

RAID 5

550

jde-ovm1/ jde-ovm2

OVM1 repository to carve out OS and data luns for Guest VMs

Ovm2-data

RAID 5

550

jde-ovm2/ jde-ovm2

OVM1 repository to carve out OS and data luns for Guest VMs

Orcl-asm-lun

RAID 10

100

jde-ovm1/ jde-ovm2

Physical disk to carve out asm for Oracle RAC VMs

Orcl-data-A

RAID 5

400

jde-ovm1/ jde-ovm2

Physical disk for Oracle Data files for Oracle RAC VMs

Orcl-data-B

RAID 5

400

jde-ovm1/ jde-ovm2

Physical disk for Oracle Data files for Oracle RAC VMs

Orcl-log-A

RAID 10

200

jde-ovm1/ jde-ovm2

Physical disk for Oracle log files for Oracle RAC VMs

Orcl-log-B

RAID 10

200

jde-ovm1/ jde-ovm2

Physical disk for Oracle log files for Oracle RAC VMs



Note Luns except for OVM Server boot luns are shared across multiple Host. This is a requirement to successfully share OVM repositories and execute VM migration.


Creation of Storage Pools/ RAID Groups

To create storage pools/RAID groups, follow these steps:

1. Login to the EMC Unisphere to create storage pools.

2. To create Storage Pool, click Storage >Storage Configuration > storage pools > Pools tab and the click Create. The "create storage pool" pop-up window appears.

a. Ensure that the Storage Pool type is "Pool".

b. Enter an appropriate name for the storage pool name in the text box.

c. Select appropriate RAID group from the drop-down list.

d. Select the required disks from the disk selection popup window and the click OK.

3. To create LUNs from the storage pool, right-click on the desired storage pool. A pop window "Create LUN" appears. In the General tab of Create LUN pop-up box.

4. Click General tab of the Create LUN window. Enter the required LUN size in the LUN properties text box.

5. Enter the name for the LUN in the "LUN Name" text box.

6. Ensure that the Database LUNs (SQL Server data) are selected as "Highest Available Tier" and Application LUNs are selected as "Lowest Available Tier" in the Tiering Policy.

7. To associate LUNs to the host, Navigate to Hosts > Storage Group and the click Create, A pop-up window "Create Storage Group" appears.

a. Enter an appropriate name in the "Storage Group Name" text box; click OK and then click Yes to confirm. Click LUNs tab, a pop-up window "Storage Group properties" appears.

b. Select the LUN from the respective SPA / SPB and click Add in the "Available LUNs" to add the selected LUNs. In the "Show LUN" drop-down list select the option "All" instead of "Not in other storage groups". The host would be attached to Storage Group once we complete the Nexus 5548 zoning configuration.


Note The Host ID which is typically 0 for the first LUN attached to the storage group and this Host Id should match with UCS Manager Service Profile > Create Boot Policy > LUN ID for SAN boot.


Figure 38 Host ID on Storage Group

Since the zones are configured in the Cisco Nexus switches with the Host HBA WWPNs, they will appear in the EMC VNX5300 - "Host connectivity status".

To establish the Host connectivity at the EMC VNX5300 array, follow these steps:

1. Login to the EMC Unisphere, click Hosts > Connectivity Status under "Host management" on the right side of the window. A pop-up window "Connectivity Status" appears.

2. Under the Host Initiators Tab, the vHBA WWPNs of the associated blade is available.

3. vHBA0 and vHBA1of the blade would appear first time before any OS install on that blade. After successful OS installation.

4. When one of the HBA initiator is selected, click Register. A pop-up window "Register Initiator Record" appears.

5. Select the "Initiator Type" as "CLARiiON Open" and Failover mode as "Active-Active mode(ALUA)-failovermode 4". Define the hostname (JDEOVM1) and IP address which would be allocated to the JD Edwards OVM Server 1.

6. Similarly register the other vhba1 WWPN with the same host and IP address. Select same Failover Mode as "Active-Active mode(ALUA)-failovermode 41".

To associate the LUN for this blade, associate LUN to the already created Storage group. For the JD Edwards HTML Server, the storage group created is "html1-install-lun". To do this, click Hosts > Storage Groups (html1-install-lun) > Connect Hosts; this will open Storage group properties pop-up box > Hosts tab. Locate the "JDEOVM1" from the available hosts section and Click OK. Associating host to the Storage Groups (JDEOVM1) is shown in Figure 39.

Figure 39 Storage Group - Host Association

Oracle Virtual Server (OVS) Configuration

It is not within the scope of this document to include the specifics of Oracle Virtual Server (OVS) Installation. You should refer to the Oracle installation documentation for specific installation instructions for your Environment.

Oracle Virtual Server Installation

Some of the important steps during Oracle Virtual Server installation are:

1. Before we install OVS from SAN, we need to edit the Boot Policy of OVM Service Profile, such that only single SAN path is visible during installation.

2. Figure 40 shows the Boot Policy specifically used during OVS Installation.

Figure 40 Single FC Path for OVS Installation

3. Attach OVS ISO to KVM virtual media, as shown in Figure 41.


Note Ensure to use OVS build 3.1.1.819 or later. Contact Oracle support to download the same.


Figure 41 OVS ISO Attached as Virtual Media to KVM Console

4. Make sure that a single Storage path is visible during OVS installation. The WWPN of storage login, should be similar to WWPN we defined in JDE_OVM_INSTALL boot policy. Figure 42 displays single storage path.

Figure 42 Single FC Path for OS Installation

5. Select the EMC LUN configured during Storage Group creation.

Figure 43 EMC Boot LUN

6. Select Allow Boot from multipath device.

Figure 44 Boot from Multipath Device

7. Configure the management Ethernet interface with appropriate ip. Make sure to verify the displayed MAC address with the vNIC Mgmt Ethernet interface created on OVM Service Profile.

Figure 45 vNIC Mgmt Configured for Management IP

8. Configure Host Name and install OVS Server.

Figure 46 OVS Installation Status

Similar steps need to be executed to install the second Oracle Virtual Server.

Some of the important steps executed post OVS installation are:

1. Reconfigure the Boot Policy for OVM Service Profile with all the four paths to EMC.

Figure 47 Reconfigure Boot Policy

2. Update fnic driver.

 [root@jde-ovm1 ~]# modinfo fnic
filename:       
/lib/modules/2.6.39-200.37.1.el5uek/kernel/drivers/scsi/fnic/fnic.ko
version:        1.5.0.1
license:        GPL v2
author:         Abhijeet Joglekar <abjoglek@cisco.com>, Joseph R. Eykholt 
<jeykholt@cisco.com>
description:    Cisco FCoE HBA Driver
srcversion:     60031A0CDAEE010E0A0B25A
alias:          pci:v00001137d00000045sv*sd*bc*sc*i*
depends:        libfcoe,libfc,scsi_transport_fc
vermagic:       2.6.39-200.37.1.el5uek SMP mod_unload modversions
parm:           fnic_log_level:bit mask of fnic logging levels (int)
 
 
[root@jde-ovm1 ~]# mkdir /test
[root@jde-ovm1 ~]# mount -o loop,ro dd-fnic-1.5.0.18-oracle-ovs-3.1.1.iso /test
[root@jde-ovm1 ~]# cd /test
 
 
[root@jde-ovm1 test]# rpm -ivh --nodeps 
/test/rpms/x86_64/kmod-fnic-1.5.0.18-el6uek_ovm_3.1.1.x86_64.rpm
Preparing...                ########################################### [100%]
   1:kmod-fnic              ########################################### [100%]
WARNING: The initrd image will contain multipath boot from san
 
 
[root@jde-ovm1 2.6.39-200.37.1.el5uek]# uname -a
Linux jde-ovm1 2.6.39-200.37.1.el5uek #1 SMP Fri Feb 8 03:53:53 PST 2013 x86_64 
x86_64 x86_64 GNU/Linux
 
 
[root@jde-ovm1 /]# cd lib/modules/2.6.39-200.37.1.el5uek/kernel/drivers/scsi/fnic/
 
 
[root@jde-ovm1 fnic]# cp  /lib/modules/2.6.39-200.1.1.el5uek/extra/fnic/fnic.ko .
cp: overwrite `./fnic.ko'? y
[root@jde-ovm1 fnic]# modprobe fnic
[root@jde-ovm1 fnic]# modinfo fnic
filename:       
/lib/modules/2.6.39-200.37.1.el5uek/kernel/drivers/scsi/fnic/fnic.ko
version:        1.5.0.18
license:        GPL v2
author:         Abhijeet Joglekar <abjoglek@cisco.com>, Joseph R. Eykholt 
<jeykholt@cisco.com>
description:    Cisco FCoE HBA Driver
srcversion:     24F8E443F0EEDBDF4802F20
alias:          pci:v00001137d00000045sv*sd*bc*sc*i*
depends:        libfc,libfcoe,scsi_transport_fc
vermagic:       2.6.39-200.1.1.el5uek SMP mod_unload modversions
parm:           fnic_log_level:bit mask of fnic logging levels (int)
parm:           fnic_trace_max_pages:Total allocated memory pages for fnic trace 
buffer (uint)
 
 

3. Edit /etc/multipath.conf for EMC specific multipath device.

 
 
        device {
                vendor                  "EMC"
                product                 "SYMMETRIX"
                getuid_callout          "/sbin/scsi_id -g -u -ppre-spc3-83 -s 
/block/%n"
                features                "0"
                hardware_handler        "0"
                path_grouping_policy    multibus
                rr_weight               uniform
                rr_min_io               8
                path_checker            tur
       }
 
 

4. Verify that all the FC paths are visible on OVS Server

 
 
[root@jde-ovm1 ~]# multipath -l
3600601604c3029009a6d5f53d7d4e211 dm-0 DGC,VRAID
size=40G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 0:0:0:0  sda  8:0    active ready running
  |- 0:0:1:0  sdo  8:224  active ready running
  |- 1:0:0:0  sdac 65:192 active ready running
  `- 1:0:1:0  sdaq 66:160 active ready running
3600601604c302900cc4b23b27bd5e211 dm-1 DGC,VRAID
size=550G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 0:0:0:1  sdb  8:16   active ready running
  |- 0:0:1:1  sdp  8:240  active ready running
  |- 1:0:0:1  sdad 65:208 active ready running
  `- 1:0:1:1  sdar 66:176 active ready running
 
 

5. As power path is not installed, following commands are issued to scan new luns added to OVS Host.

 
 
    # ls /sys/class/fc_host
    host0  host1
 
 
    # echo "1" > /sys/class/fc_host/host0/issue_lip
    # echo "- - -" > /sys/class/scsi_host/host0/scan
    # echo "1" > /sys/class/fc_host/host1/issue_lip
    # echo "- - -" > /sys/class/scsi_host/host1/scan
 
 

6. Once post-installation steps are executed we would proceed to Oracle VM Manager Installation.

OVM Manager Installation

Oracle VM Manager is installed as a production level; this is the preferred installation type, with options for selecting Oracle SE or EE database as the location for the Oracle VM Manager repository. Prior to installing Oracle VM Manager, ensure that Oracle SE Database is installed. Please follow the steps below to successfully install OVM Manager.

 
 
[root@ovmmanager ovmmanager]# ./runInstaller.sh
 
 
Oracle VM Manager Release 3.1.1 Installer
 
 
Oracle VM Manager Installer log file:
 /tmp/install-2013-06-10-164951.log
 
 
Please select an installation type:
   1: Demo
   2: Production
   3: Uninstall
   4: Help
 
 
   Select Number (1-4): 2
 
 
Starting production installation ...
 
 
 
 
Verifying installation prerequisites ...
 
 
Oracle Database Repository
==========================
Use an existing Oracle database
Enter the Oracle Database hostname [localhost]: ovmmanager
Enter the Oracle Database System ID (SID) [XE]: orcl
Enter the Oracle Database SYSTEM password:
 
 
Enter the Oracle Database listener port [1521]: 1521
Enter the Oracle VM Manager database schema [ovs]: ovs1
Enter the Oracle VM Manager database schema password:
 
 
Invalid password.
Passwords need to be between 8 and 16 characters in length.
Passwords must contain at least 1 lower case and 1 upper case letter.
Passwords must contain at least 1 numeric value.
Enter the Oracle VM Manager database schema password:
 
 
Enter the Oracle VM Manager database schema password (confirm):
 
 
 
 
Oracle Weblogic Server 11g
==========================
Enter the Oracle WebLogic Server 11g user [weblogic]:
Enter the Oracle WebLogic Server 11g user password:
 
 
Enter the Oracle WebLogic Server 11g user password (confirm):
 
 
Passwords do not match
Enter the Oracle WebLogic Server 11g user password:
 
 
Enter the Oracle WebLogic Server 11g user password (confirm):
 
 
 
 
Oracle VM Manager application
=============================
Enter the username for the Oracle VM Manager administration user [admin]:
Enter the admin user password:
 
 
Enter the admin user password (confirm):
 
 
 
 
Verifying configuration ...
 
 
Start installing the configured components:
   1: Continue
   2: Abort
 
 
   Select Number (1-2): 1
 
 
Step 1 of 9 : Database ...
Installing Database ...
Database installation skipped ...
 
 
Step 2 of 9 : Java ...
Installing Java ...
 
 
Step 3 of 9 : Database Schema ...
Creating database schema 'ovs1' ...
 
 
Step 4 of 9 : WebLogic ...
Retrieving Oracle WebLogic Server 11g ...
Installing Oracle WebLogic Server 11g ...
 
 
Step 5 of 9 : ADF ...
Retrieving Oracle Application Development Framework (ADF) ...
Unzipping Oracle ADF ...
Installing Oracle ADF ...
Installing Oracle ADF Patch...
 
 
Step 6 of 9 : Oracle VM  ...
Retrieving Oracle VM Manager Application ...
Extracting Oracle VM Manager Application ...
Installing Oracle VM Manager Core ...
 
 
Step 7 of 9 : Domain creation ...
Creating Oracle WebLogic Server domain ...
Starting Oracle WebLogic Server 11g ...
Configuring data source 'OVMDS' ...
Creating Oracle VM Manager user 'admin' ...
 
 
Step 8 of 9 : Deploy ...
Deploying Oracle VM Manager Core container ...
Deploying Oracle VM Manager UI Console ...
Deploying Oracle VM Manager Help ...
Enabling HTTPS ...
Granting ovm-admin role to user 'admin' ...
 
 
Step 9 of 9 : Oracle VM Manager Shell ...
Retrieving Oracle VM Manager Shell & API ...
Extracting Oracle VM Manager Shell & API ...
Installing Oracle VM Manager Shell & API ...
 
 
Retrieving Oracle VM Manager Upgrade tool ...
Extracting Oracle VM Manager Upgrade tool ...
Installing Oracle VM Manager Upgrade tool ...
Copying Oracle VM Manager shell to '/usr/bin/ovm_shell.sh' ...
Installing ovm_admin.sh in '/u01/app/oracle/ovm-manager-3/bin' ...
Installing ovm_upgrade.sh in '/u01/app/oracle/ovm-manager-3/bin' ...
Enabling Oracle VM Manager service ...
Shutting down Oracle VM Manager instance ...
Restarting Oracle VM Manager instance ...
Waiting 15 seconds for the application to initialize ...
Oracle VM Manager is running ...
Oracle VM Manager installed.
 
 
Please wait while WebLogic configures the applications... This can take up to 5 
minutes.
 
 
Installation Summary
--------------------
Database configuration:
  Database host name          : ovmmanager
  Database instance name (SID): orcl
  Database listener port      : 1521
  Application Express port    : None
  Oracle VM Manager schema    : ovs1
 
 
Weblogic Server configuration:
  Administration username     : weblogic
 
 
Oracle VM Manager configuration:
  Username                    : admin
  Core management port        : 54321
  UUID                        : 0004fb00000100000a5c59c7f7487ffe
 
 
 
 
Passwords:
There are no default passwords for any users. The passwords to use for Oracle VM 
Manager, Oracle Database 11g XE, and Oracle WebLogic Server have been set by you 
during this installation. In the case of a default install, all passwords are the 
same.
 
 
Oracle VM Manager UI:
  http://ovmmanager:7001/ovm/console
  https://ovmmanager:7002/ovm/console
Log in with the user 'admin', and the password you set during the installation.
 
 
Please note that you need to install tightvnc-java on this computer to access a 
virtual machine's console.
 
 
For more information about Oracle Virtualization, please visit:
  http://www.oracle.com/virtualization/
 
 
Oracle VM Manager installation complete.
 
 
Please remove configuration file /tmp/ovm_configzFYrq_.

Post OVM Manager installation, apply Oracle VM Manager 3.1.1 Patch Update (Build 365) [ID 1530546.1]. This would help in resolution of time out issues on creation of Oracle VM 3.1.1.

Oracle VM Configuration

Some of the important steps to configure Oracle VM environment are shown in Figure 48.

Figure 48 Oracle VM Configuration Steps

7. Discover Oracle VM Servers. The servers will be listed as Unassigned Server under Servers and VMs tab.

Figure 49 Unassigned Servers

8. Configure VM Network as shown in Figure 50.

Figure 50 Network Configuration

9. Create Server Pool with the cluster LUN as the repository.

Figure 51 Cluster Pool

10. Create Storage Repository for each of the data luns configured for OVM Server. Ensure that the luns are shared across each of the Storage Group configured in EMC UniSphere.

Figure 52 Storage Repository

11. Create VMs for each of the Oracle JDE tiers as shown in Figure 53. In accordance with Oracle recommendations, PVM Guest VMs are created. We would create two Guest VMS for each of the Oracle RAC nodes, two VMS for JDE App Servers and two VMs for JDE HTML Servers.

Figure 53 Guest VMs Details

Figure 54 Guest VMs

12. Ensure that Oracle RAC node VMs are configured with Private network nics, as shown in Figure 50.

13. JDE HTML and App Servers are configured with Virtual Disk, whereas the Oracle RAC Guest VMs are configured with Virtual Disk for OS and DB binary installation and physical disk for ASM, DATA and LOG luns. Each of these configurations is shown in Figure 55.

Figure 55 Disk Configuration for JDE Application Server

Figure 56 Disk Configuration for Oracle RAC Nodes

Once the Guest VMs are created we would proceed to Installation of Oracle Linux 5.8 on each of the VMs.

Oracle Linux Installation

Some of the important steps during Oracle Linux installation are:

1. Configure HTTP server location for PVM Guest Installation.

Figure 57 http Setup for PVM Installation

2. Ensure Text Mode is selected for PVM installation.

Figure 58 Text Mode Selection

3. Select Virtual Disk for OS installation.

Figure 59 Virtual Disk Selection

4. Once OS is installed Reboot the VM and proceed to post installation steps.

Figure 60 Confirm OS Installation

Some of the important steps executed post Oracle Linux installation are:

1. Edit private network vNIC of Oracle RAC nodes with MTU=9000

2. Configure private and public IPs for Oracle RAC node VMs and public IPs JDE App and THML Servers.

3. Edit Network, to add IP configuration for VLAN 760 (Public) and VLAN191 (for Private Interconnect traffic) to the respective ethernet interfaces.

4. Once the IP is updated, we would apply iptables patch to the all the Guest VMs. At high concurrent users we see following error messages on the /var/log/messages logs.

kernel: nf_conntrack: table full, dropping packet.
kernel: nf_conntrack: table full, dropping packet.
 
 

Download package iptables-1.3.5-9.2.0.2.el5_8.x86_64 from URL= https://linux.oracle.com => Home > Channels > Channel: Oracle Linux 5 Update 10 Patch (x86_64)>Packages.

Update the rpm with rpm -Uvh iptables-1.3.5-9.2.0.2.el5_8.x86_64.rpm.

Apply the above patch to the all the Guest VMs with Oracle Linux 5.8 (uek kernel).

Once we complete the post-installation steps we would proceed to Oracle RAC setup on Guest VMs.

Oracle RAC Setup

This section describes the deployment of Oracle Database 11g R2 GRID Infrastructure with RAC option for Oracle JD Edwards EnterpriseOne deployment on Oracle Virtualized environment. After installation of Oracle Linux 5.8 on each RAC node, verify that all the required rpms are installed as part of OS installation which is required for Oracle GRID Installation.

As a best practice recommended from Oracle, ready-to-go Oracle VM Templates for Oracle RAC can be downloaded from Oracle Software Delivery Cloud for faster deployment.

Use the following Oracle document for pre-installation tasks, such as setting up the kernel parameters, RPM packages, user creation, and so on.

(http://download.oracle.com/docs/cd/E11882_01/install.112/e10812/prelinux.htm#BABHJHCJ)

Follow the steps to complete the Oracle Database 11g R2 GRID Infrastructure with RAC option Installation. For this solution the user, group, directory structure, kernel parameters and user limits can be created as shown in the steps below. One can resize the kernel parameter, user limits and rename the directory structure, user name, groups as per the business requirement.

1. Create required oracle users and groups in each RAC nodes.

groupadd -g 1000 oinstall
groupadd -g 1200 dba
useradd -u 2000 -g oinstall -G dba grid
passwd grid
useradd -u 1100 -g oinstall -G dba oracle
passwd oracle
 
 

Note "grid" user owns the Oracle GRID installation whereas "oracle" user owns the Oracle Database Installation. In the present setup, we used OS LUN directory for GRID Installation and Database binary Installation.


2. Identify the virtual IP addresses and SCAN IPs and have them setup in DNS per Oracle's recommendation. Alternatively, you can update the /etc/hosts file with all the details (private, public, SCAN and virtual IP) if you do not have DNS services available

3. List all the physical disk mounted on Oracle RAC VM nodes. Physical disk are shared across both the Oracle VM nodes.

Disk /dev/xvdb: 429.4 GB, 429496729600 bytes - orcl-data-a
Disk /dev/xvdc: 214.7 GB, 214748364800 bytes- orcl-log-a
Disk /dev/xvdd: 107.3 GB, 107374182400 bytes - orcl-asm
Disk /dev/xvde: 429.4 GB, 429496729600 bytes - orcl-data-b
Disk /dev/xvdf: 214.7 GB, 214748364800 bytes - orcl-log-b
Disk /dev/xvdg: 107.3 GB  -  orcl-asm1
 
 

4. Create partition on each of the physical disk. This has to be executed only in one of the Oracle RAC nodes. Following is the listing for each of the partitions.

[root@vm-db-node1 ~]# fdisk -l
 
 
Disk /dev/xvda: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
 
    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *           1          13      104391   83  Linux
/dev/xvda2              14       39162   314464342+  8e  Linux LVM
 
 
Disk /dev/dm-0: 216.7 GB, 216795185152 bytes
255 heads, 63 sectors/track, 26357 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
 
Disk /dev/dm-0 doesn't contain a valid partition table
 
 
Disk /dev/dm-1: 105.1 GB, 105193144320 bytes
255 heads, 63 sectors/track, 12789 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
 
Disk /dev/dm-1 doesn't contain a valid partition table
 
 
Disk /dev/xvdb: 429.4 GB, 429496729600 bytes
255 heads, 63 sectors/track, 52216 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
 
    Device Boot      Start         End      Blocks   Id  System
/dev/xvdb1               1       52216   419424988+  83  Linux
 
 
Disk /dev/xvdc: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
 
    Device Boot      Start         End      Blocks   Id  System
/dev/xvdc1               1       26108   209712478+  83  Linux
 
 
Disk /dev/xvdd: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
 
    Device Boot      Start         End      Blocks   Id  System
/dev/xvdd1               1        1305    10482381   83  Linux
/dev/xvdd2            1306        2710    11285662+  83  Linux
/dev/xvdd3            2711        4015    10482412+  83  Linux
 
 
Disk /dev/xvde: 429.4 GB, 429496729600 bytes
255 heads, 63 sectors/track, 52216 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
 
    Device Boot      Start         End      Blocks   Id  System
/dev/xvde1               1       52216   419424988+  83  Linux
 
 
Disk /dev/xvdf: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
 
    Device Boot      Start         End      Blocks   Id  System
/dev/xvdf1               1       26108   209712478+  83  Linux
 
 
Disk /dev/xvdg: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
 
    Device Boot      Start         End      Blocks   Id  System
/dev/xvdg1               1        1305    10482381   83  Linux
/dev/xvdg2            1306        2710    11285662+  83  Linux
 
 

Note For OCR voting disk, we have created 5 partitions 10G each, we had to allocated 2 luns of 100G as we can only have 4 partitions (primary or extended) on each physical. Hence, we created three 10G partitions on /dev/xvdd and two partitions on /dev/xvdg


5. Configure OracleASM.

[root@vm-db-node1 /]# /etc/rc.d/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
 
 
Default user to own the driver interface [grid]:
Default group to own the driver interface [grid]: oinstall
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
 
 

6. Create OCR voting disks.

[root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk ocrvote1 /dev/xvdd1
Marking disk "ocrvote1" as an ASM disk:                    [  OK  ]
[root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk ocrvote2 /dev/xvdd2
Marking disk "ocrvote2" as an ASM disk:                    [  OK  ]
[root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk ocrvote3 /dev/xvdd3
Marking disk "ocrvote3" as an ASM disk:                    [  OK  ]
[root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk ocrvote4 /dev/xvdg1
Marking disk "ocrvote4" as an ASM disk:                    [  OK  ]
[root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk ocrvote5 /dev/xvdg2
Marking disk "ocrvote5" as an ASM disk:                    [  OK  ]
 
 
Create data and log disk
 
 
[root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk data_A /dev/xvdb1
Marking disk "data_A" as an ASM disk:                      [  OK  ]
[root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk data_B /dev/xvde1
Marking disk "data_B" as an ASM disk:                      [  OK  ]
[root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk log_A /dev/xvdc1
Marking disk "log_A" as an ASM disk:                       [  OK  ]
[root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk log_B /dev/xvdf1
Marking disk "log_B" as an ASM disk:                       
 
 

7. List ASM disks.

[root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm listdisks
DATA_A
DATA_B
LOG_A
LOG_B
OCRVOTE1
OCRVOTE2
OCRVOTE3
OCRVOTE4
OCRVOTE5
 
 

8. Scan ASM disk on second Oracle Node.

[root@vm-db-node2 ~]# /etc/rc.d/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@vm-db-node2 ~]# /etc/rc.d/init.d/oracleasm listdisks
 
 
DATA_A
DATA_B
LOG_A
LOG_B
OCRVOTE1
OCRVOTE2
OCRVOTE3
OCRVOTE4
OCRVOTE5
 
 

Once ASM disks are configured on the both the Oracle RAC node VMs, we would proceed to Oracle Grid Installation.

Oracle Grid Installation

Some of the important steps to configure Oracle Grid Infrastructure are:

1. Attach VNC session on one the Oracle RAC VM as grid user and execute Grid installer.

Figure 61 Execute Grid Installer

2. Click the Skip Software Updates radio button.

3. Click the Install and configure Oracle Grid Infrastructure for a cluster radio button.

Figure 62 Select Grid Installation

4. Select Advance Installation and move on to Grid Plug and Play Information.

Figure 63 Scan and Cluster Name

5. Define Oracle RAC cluster node2 information.

Figure 64 Add Cluster Node Information

6. Establish ssh connectivity for both the Oracle RAC nodes.

Figure 65 SSH Connectivity

7. Specify Network Interface Usage.

Figure 66 Specify Interface Type

8. Select Oracle ASM as the storage Options.

Figure 67 Storage Options

9. Scan ASM disk and create OCR disk group.

Figure 68 ASM Disk Group

Figure 69 Select Disks

10. Specify ASM password, and move on to specify Installation location.

Figure 70 Specify Installation Location

11. Verify all the pre-requisite check are completed successfully and verify the installation Summary.

Figure 71 Installation Summary

12. Continue to Install Oracle Grid Infrastructure.

13. Execute root scripts on each of the Oracle RAC node VMs.

Figure 72 Execute Configuration Script

14. Once root scripts are executed, we will get the Installation Complete window.

Figure 73 Installed Product - Progress and Status

Once Oracle RAC Grid infrastructure is installed, proceed to Oracle Database Installation.

For more information on Oracle RAC Grid, see Grid Infrastructure Installation Guide for Linux:

http://www.oracle.com/pls/db112/to_toc?pathname=install.112/e10812/toc.htm

Oracle Database Installation

Some of the important steps to configure Oracle Grid Infrastructure are:

1. Attach VNC session on one of the Oracle RAC VM as oracle user and execute database installer.

Figure 74 Execute Database Installer

2. Click the Install database software only radio button. We can configure the database once we have the DB binaries

Figure 75 Select Install Database Software

3. Select both the nodes on Oracle RAC database installation.

Figure 76 Select Nodes

4. Configure ssh connectivity.

Figure 77 Configure SSH Connectivity

5. Once the installation location and connectivity between Oracle RAC nodes is verified, we can see them in the Installation Summary window.

Figure 78 Installation Summary

6. Execute root scripts on each of the Oracle RAC nodes

Figure 79 Execute Root Scripts

7. Verify that the Database installation is completed successfully.

Figure 80 Installation Successful

For complete details on Oracle RAC database installation, see Real Application Clusters Installation Guide for Linux and UNIX for detailed installation instructions (http://www.oracle.com/pls/db112/to_toc?pathname=install.112/e10813/toc.htm)

Oracle RAC Post Installation

After successfully installation Grid Infrastructure and Oracle Database binaries we would proceed to configure of Oracle RAC instance. This involves creation of ASM disk group for Oracle Data & Oracle Logs through asmca and configuration of DB instance through dbca.

Following are some of the important Oracle RAC Post installation steps:

1. VNC as grid user and execute asmca. Select Oracle Data path for creation of DATA disk group.

Figure 81 Select Oracle Data path for Creation of DATA Disk Group

2. Create Disk Group.

Figure 82 Create Disk Group

3. Similarly create Oracle LOG disk group.

Figure 83 Create Oracle LOG Disk Group

4. Now VNC with Oracle User and execute dbca Select +DATA and +LOG as path for Oracle Data and Oracle Log files.

5. Once dbca is executed successfully we would see instances of Oracle RAC on each of the VM.

Figure 84 Oracle RAC Instance on Each VM

Oracle JD Edwards EnterpriseOne Installation

The Installation of Oracle JD Edwards (JDE) 9.0.2 suite on Oracle VM with Oracle Linux 5.8 and Oracle RAC 11.2.0.3 as the RDBMS is described in the sections below:

Pre-Requisites

See the latest JDE MTRs for the most up-to-date information regarding the pre-requisites for your install http://docs.oracle.com/cd/E24902_01/nav/installation.htm.

All the required JDE software were downloaded from Oracle eDelivery and Oracle UpdateCenter

Network connectivity between all the machines involved.

General Install Requirements

Following are generic pre-requisites before installing the Oracle JDE Enterprise Server.

Make sure the Disk space is sufficient enough for the installation, for detailed space requirements, see: https://support.oracle.com/epmos/faces/DocumentDisplay?id=747323.1

Database Server Software has to be installed in JDE Database Server.

Database client software has to be installed in other JDE Servers like Application Server and Deployment Server.

Make sure enough temporary disk space is available for the installers and wizards.

Oracle JD Edwards EnterpriseOne Specific Install Requirements

The below mentioned are some of the requirements to be considered for JD Edwards EnterpriseOne installation


Note It is strongly recommended that installation be performed by running installers using the `run as administrator' option.


Table 7 JD Edwards EnterpriseOne Installation Requirements

Server Type
Install Requirements

JDE Deployment Server

Installation of JDE Deployment server for JDE version 9.0

Installation of Application update 2 for 9.0 JDE applications

Installation of JDE Deployment server tools version 8.98.4.10

Installation of JDE ServerManager for 8.98.4.6

Installation of Microsoft VisualStudio 2008 SP1

Installation of Microsoft Windows SDK v6.0A

Visual Studio 2005sp1 runtime libraries should be installed on deployment server.

JDE Enterprise Server

Installation of JDE Enterprise server tools version 8.98.4.10

Installation of Oracle Linux 5.8

JDE Database Server

Installation of JDE 9.0 databases in Database Server machine.

Installation of Oracle Linux 5.8

Oracle 11.2.0.3 RAC DB

JDE HTML Server

Installation of Oracle Linux 5.8

Installation of JDK or jrockit-jdk1.6.0_29-R28.2.2-4.1.0 in Web Server machine before doing the Web logic server installation.

Installation of Oracle Weblogic 10.3.6 in web server machine.

Installation of Oracle HTTP server

Installation of JDE HTML server 8.98.4.10


JDE Install Port Numbers

Table 8 shows the port numbers that are used for WLS, JDE, HTTP ports, and so on.

Table 8 Port numbers

Port
Port Number

WLS

7503-7581

JDE Enterprise Server

9700

HTTP

7777

SQLServer

14501


The steps followed to install Oracle JD Edwards on Cisco UCS are shown in Figure 85:

Figure 85 Oracle JD Edwards EnterpriseOne Installation Workflow

Oracle JDE Deployment Server Install

The Deployment server is used as a repository of JD Edwards installation and upgrade software and data artifacts. This section shows the steps for installing the JD Edwards deployment server. The installation steps shown here forth are specific to the JD Edwards 9.0.2 application suite used in conjunction with JD Edwards tools release 8.98.4.10.

To Install the JDE Deployment Server, follow these steps:

1. Download the Deployment Server binaries from Oracle eDelivery into a directory and extract the zip files in place using tools like Winzip or 7zip.

Run the steps below, as referenced from Oracle solution ID: Document 1310036.1

Follow the below mentioned steps to download the installer for the database and the installer for the .NET Framework from Microsoft. In addition, you must follow these directions to download a program from Oracle that runs the two Microsoft installers, passing them parameters that EnterpriseOne needs to run properly.


Note Referenced from Oracle solution ID: Document 1310036.1


2. To download the Microsoft SQL Server Express 2005 SP3 Installer on the Deployment server, follow these steps:

a. Go to the Microsoft Download Center: http://www.microsoft.com/downloads

b. In the search field near the top of the screen, enter SQL Server 2005 Express Edition SP3 and click Search.

c. Click the link SQL Server 2005 Express Edition SP3.

d. Next to the file called SQLEXPR.EXE, click Download.

e. Save the file to your Deployment Server in this location: <dep_svr_install_dir>\OneWorld Client Install\ThirdParty\SSE

3. The .NET Framework contains new Windows files that applications such as SSE can use. Oracle highly recommends that you install at least version 4.0 of the Microsoft .NET Framework. For this procedure, you should download the installer to the Deployment Server as described below.

a. Go to the Microsoft Download Center: http://www.microsoft.com/downloads

b. In the menu bar at the top of the screen, click Downloads A-Z and type N and NET

c. Click the link Microsoft .NET Framework 4 (Web Installer).

d. Next to the file called dotNetFx40_Full_setup.exe, click Download.

4. Save the file to your deployment server in this location: <dep_svr_install_dir>\OneWorld Client Install\ThirdParty\SSE

5. The DotNetSSESetup.exe program runs the .NET Framework and SSE installers. Locate and download the EnterpriseOne DotNetSSESetup.exe and related file called settings.ini from E-Delivery using this part number and description:

V24818-01 JD Edwards EnterpriseOne Tools 8.98.4.2 - Microsoft SQL Server 2005 Express SP3 Local Database Installer for Deployment Server and Development Client

Place the SSE 2005 SP3 installer SQLEXPR.exe and the .NET Framework 4 installer dotNetFx40_Full_setup.exe onto your Deployment Server in this directory: <dep_svr_install_dir>\OneWorld Client Install\ThirdParty\SSE

6. Edit the settings.ini file in this directory: <dep_svr_install_dir>\OneWorld Client Install\ThirdParty\SSE The settings.ini file contains settings for installing the .NET Framework and SSE. For completeness, these settings include those for .NET Framework 2.0/SSE 2005 prior to SP3 and for .NET Framework 4.0/SSE 2005 SP3.

7. In the settings.ini file, uncomment the settings for the set of installers that you will be using, and comment out (by adding a semicolon at the start of the line) the settings for the set of installers that you will not be using. Note: Only one set must be uncommented, and only one set must be commented out.

8. Save the settings.ini file.

9. Run the DotNetSSESetup.exe file as administrator.

10. After the .NET Framework is installed successfully, download the appropriate SQL Server JDBC driver for SQL server 2005 SP3 from MSDN.

11. Place the SQL Server JDBC drivers in a folder named JDBC.

12. Now execute the RunInstaller in Admin mode from the deployment server disk1 folder that was unzipped in Step 1.

13. Choose the directory for installing deployment server as well as the directory which contains the SQLServer driver.

14. Install the Deployment Server Installer and then Install the Deployment Server. After the successful Installation proceed to next installation.

Figure 86 JD Edwards Enterprise One Deployment Server Installation Progress

15. Install Microsoft Visual Studio C Runtime libraries using the vcredist.exe (download the relevant platform from msdn) for VisualStudio 2005 SP1.

16. Run the vcredist.exe as administrator. Click Yes to accept the license agreement, and proceed to Install the Microsoft Visual Studio C Runtime Libraries.

Figure 87 Installing the Visual Studio C Runtime libraries

Figure 88 Accepting the License Agreement Prior to Install

Tools Upgrade on the Deployment Server

After the deployment server install is completed successfully, a tools upgrade to 8.98.4.10 is performed. To upgrade the tools release, follow these steps:

1. Download the appropriate tools release (8.98.4.10 deployment server in this case) from update center and run the InstallManager.exe as an administrator. Choose Workstation Install, which encapsulates most installs on client workstations, to get options regarding available tools releases.

2. Install the chosen tools release for the deployment server (8.98.4.10 in this case).

Figure 89 Installing the JD Edwards Enterprise One Workstation

3. Click Next to proceed with the Installation, and click the Tools Release radio button to define the setup type. Click Finish to complete the tools upgrade.

Figure 90 Selecting Options to Proceed Installation

Figure 91 Selecting the Tools Upgrade Setup Type

Install Planner ESU

To download and install the Install Planner ESU, follow these steps:

1. Download the appropriate planner ESU from Oracle update center onto the deployment server and unzip the contents. Run the executable as an administrator. The planner is installed using the EnterpriseOne client workstation installation program.

Figure 92 Sign in to the JD Edwards EnterpriseOne Client Install Using the JDE User

Figure 93 Selecting the Setup Type

2. After the Planner has installed successfully, run the Special Instructions for the planner. Open windows cmd prompt in administrator mode.

3. In the cmd prompt, change the directory to the scripts directory of the planner ESU.

Figure 94 Changing the Directory to Scripts

4. Run specialInstrs.bat command in the command prompt.

Figure 95 Running the command SpecialInstrs

5. Choose the SQL Server Express option and type S in the command prompt.

Figure 96 Selecting the SQL Server Express

6. Run the R98403XB XJDE0002 report, which copies control records from the shipped XML database into the local planner databases, since ESUs are shipped with an XML database, starting with Apps 9.0.

Figure 97 Copying the Control Records

7. Edit jdbj.ini in the OC4J webclient deployment on the deployment server (C:\JDEdwards\E900\SYSTEM\OC4J\j2ee\home\applications\webclient.ear\webclient\WEB-INF\classes).

8. Comment as 400 and Oracle drivers with #.

Figure 98 Commenting the AS400 and Oracle Drivers

9. In the Connection Pooling Settings, increase the connectionTimeout in jdbj.ini to 3600000.

Figure 99 Incrementing the Connection Timeout

10. Run report R98403XB, which copies control records from the shipped XML database into the local planner databases. Open ActiveConsole in administrator mode.

11. Sign in using the JDE user credentials using Environment-JDEPLAN, Role-ALL.

12. Type BV in fastpath text boxes, to open the batch versions.

13. Type in R98403XB in Batch Application text box and click Search.

Figure 100 Searching for the Batch Application

14. Select the Copy Control Tables to planner environment batch version.

Figure 101 Selecting the Batch Application Version

15. Set the processing options for the selected version of the UBE selected. Choose Copy Control Tables to planner environment > Row > Processing Options to launch the Processing Option window.

Figure 102 Setting the Processing Options

16. Enter the path to the planner data folder, which contains the planner ESU folder and enter target environment name. Click OK to proceed.

Figure 103 Defining the Path to the Planner Data Folder

17. Check the Data Selection checkbox, and click Submit.

Figure 104 Submitting the Data Selection

18. Click the On Screen radio button to define the report output destination.

Figure 105 Selecting the Report Destination

19. After the report completes, verify it for any errors. The copies completed successfully above. This concludes the JD Edwards Deployment Server Installation.

Figure 106 Completion of the Report download

Enterprise Server Install

In this setup, the JDE Enterprise server and JDE database server are on different machines. The platform pack software pertaining to these two JDE servers has to be installed separately onto these two machines. Download the relevant platform pack software (part id V12449-01 -> V12452-01) from Oracle eDelivery and extract its contents by unzipping the zip files. Please note that all zip files must be extracted in place within a folder as sub-folders disk1, disk2 etc.

To install the enterprise server, follow these steps:

1. After extracting the zip files, go to subfolder disk1, open a xWindows session, cd the disk1 directory and run install.sh.

Figure 107 Running the install.sh command

2. Launch the InstallShield Wizard. Click Next.

Figure 108 Proceeding the JDE EnterpriseOne UNIX Platform Pack

3. Choose the directory where the JDE Enterprise server binaries are to be installed.

Figure 109 Defining the Directory

4. Click the Install EnterpriseOne radio button. Click Next.

Figure 110 Selecting the Install option

5. Click the Custom radio button, to define the installation type.

Figure 111 Selecting the Installation type

6. Check the JD Edwards EnterpriseOne Enterprise Server checkbox, and all the components. Click Next.


Note The DB components are not chosen since DB is installed on a different server.


Figure 112 Selecting the Features to be Installed

7. Type the Database Server Name and Database Type in the text boxes. Select the Database creation options. Use the default password. Click Next.

Figure 113 Defining the DB Attributes

8. Set the ini for the appropriate number of users. Click the Yes radio button. Click Next.

Figure 114 Defining the concurrent user number

9. Enter the Oracle DB SID and the name of the UNIX account for Oracle DB. Click Next.

Figure 115 Naming the Database Instance

10. Verify the selections used for installing the enterprise Server in the summary page. Click Next.

Figure 116 Summary of the Enterprise Server features

11. Click Finish.

DB Server Install

The PlatformPack binaries used for the Enterprise server are copied onto the DB server.

Figure 117 Launching the JD Edwards EnterpriseOne UNIX Platform Pack

1. Choose the directory where the JDE Enterprise server binaries are to be installed. Click Next.

Figure 118 Defining the Directory

2. Click the Install EnterpriseOne radio button. Click Next.

Figure 119 Defining the Install Option

3. Click the Custom install radio button. Click Next.

Figure 120 Defining the Installation type

4. Check the JD Edwards EnterpriseOne Database Server checkbox, and all the components. Click Next.

Figure 121 Selecting the DB Server features

5. Type the Database Server Name and Database Type in the text boxes. Select the Database creation options. Use the default password. Click Next.

Figure 122 Defining the DB Options

6. Enter the UNIX owner and group owning the oracle install, connect string, sysadmin user and password in the corresponding text boxes. Define the directory.

Figure 123 Defining the Database Instance References

7. Click the Yes radio button to install the database. Click Next.

Figure 124 Verifying the Creation of new database

8. Verify the selections made in the summary page. Click Next to complete Installation. Click Finish to exit.

Figure 125 Summary of the Database Server Features

The DB platform pack is installed successfully.

Installation Plan

To Install the Installation Plan, follow these steps:

1. Open JDE admin console on Deployment Server by using run as admin. Login as JDE user in JDEPLAN environment.

2. Type GH961 in the Fast Path text box, to go to the Installation Plan.

3. Right-click Custom Installation Plan. Choose Prompt For > Values.

4. In the Processing Options Window type 2 in the Prompt Mode text box(2 for verbose mode). Click OK.

Figure 126 Defining the Verbose Mode

5. Click the Process Mode tab. Type 1 in the Installation text box. Click OK.

Figure 127 Defining the Process Mode

6. Click the Default tab. Type 1 as prompt value for all default options at runtime. Click OK.

Figure 128 Defining the Default Mode Values

7. Click the Status Change tab. Type the values in the text boxes, Mark as complete must always be 60:60; Disbale:70; Enable:10. Click OK.

Figure 129 Defining the Detail Status

8. Click the Completion tab. Type 2 (run automatically) in finalize and validate plan text boxes. Click OK.

Figure 130 Defining the Process after Completion

9. Click the Replication tab. Type 1 to prompt for options at runtime in all text boxes. Click OK.

Figure 131 Defining the Values for Replication

10. Click the Packages tab. Type 1 for prompt for pushing at runtime in the text box. Click OK.

Figure 132 Defining the Package Push Options

11. Click Add, to add the custom installation plan.

Figure 133 Adding the Custom Installation Plan

12. Enter name, description, and Status in the text boxes. Click the Install radio button.

Figure 134 Define the Name and Description for the Plan

13. After adding the plan, define a location that would be associated with the Plan. Click OK in the Location message box.

Figure 135 Adding the Location

14. Type the location properties, and click OK.

Figure 136 Defining the Location Parameters

15. For various JDE servers type the Machine Name, Description, Release, and primary User in the text boxes. Define the Deployment Server Share Path.

Figure 137 Defining the JDE Servers Details

16. Type Enterprise Server installation path and description. Type the name of deployment server.

Figure 138 Defining the Installation Path and Server Details

17. Click OK to go through DataSource revisions so as to confirm default data source information.

Figure 139 Confirming the Data Source

18. Verify the installation information for the HTML server in the HTML tab. Input the ports, server url and installation path. Type the Deployment Server Name.

19. Verify Database specific installation plan information as database resides on separate Server. Click OK to define Data Server properties.

Figure 140 Defining the Data Server on Separate Server

20. Type Datasource type, Platform, Database Server Name and Database Name and ID.

21. Verify the Data Dictionary data source configuration values- platform, server name and datasource type for all the datasources.

Figure 141 Verify all Data Sources

22. Check the Default Environments, and Default Data Load checkboxes to configure valid environments and load relevant environment data.

Figure 142 Defining the Environment Data

23. Click No in the Location message box, since we are not adding another location.

Figure 143 Not Defining Other Location

24. Information Prompt message box confirms the plan is finalized.

Figure 144 Completion of Plan

25. Click the On Screen radio button to choose the report destination. Click OK.

Figure 145 Running the Planner Validation Report

26. Run the Planner Validation Report, R9840B, and validate all records.

Installation Workbench

To install the JDE Workbench, follow these steps:

1. Execute the custom plan created previously.

2. Sign into JDEPLAN on Deployment Server, fastpath to GH961.

3. Choose Installation workbench > Prompt For > Values. The Processing Options window opens.

4. In the Process tab, type1 for unattended workbench mode, and 60 in the Plan Detail Status text box. Click OK.

Figure 146 Defining the Workbench Mode

5. Search for the available plan Status.

Figure 147 Search the Plans Available

6. Select the custom plan that was created in the previous section.

Figure 148 Selecting the Plan

7. For unattended workbench mode, all workbenches are completed without any intervention no task breaks was set.

Figure 149 Completion of the Installation Plan

8. After the Installation Plan is completed, the status changes from Validated to Installed.

Figure 150 Verifying the Plan Status

Change Assistant

To download and install the Change assistant, follow these steps:

1. Download change assistant from Oracle updateCenter onto the Deployment Server.

2. Install the change assistant on the Deployment Server.

Figure 151 Downloading the Change Assistant

Baseline ESU Install

To Install the Baseline ESU, follow these steps:

1. On the Deployment Server, sign into the change assistant using Oracle support credentials.

2. Choose Search for Packages > JD Edwards > Electronic Software Update 9.0 baseline ESUs. Choose the unattended mode and click OK.

Figure 152 Selecting the Unattended Mode

3. Wait for the ESUs to be downloaded and get applied. After the Baseline ESUs are applied the summary window appears.

Figure 153 Downloading the Baseline ESU

4. Summary of the ESU Downloaded

Figure 154 Summary of the ESU Downloaded


Note Apply the special instructions for the baseline ESUs. Some special instructions pertaining to localizations for specific countries might not be relevant for the install, so it is essential to read through the special instructions.


UL2 Install

The Oracle-JDE DIL kit, which best represents a standard customer workload for JD Edwards applications, needs the JDE application level to be at 9.0 update level 2, so UL2 was applied to the installed base application.

To install the UL2, follow these steps:

1. Download The UL2 file from eDelivery and extracted via winzip. Run the RunInstall.exe using run as administrator.

2. Click the 9.0 Update 2 Installation link in the JD Edwards Install Manager window.

Figure 155 Clicking the UL2 Update Installation

3. The Client Workstation Setup window launches. Click Next.

Figure 156 Proceeding with the UL2 Installation

4. Click Next, after disk check completes. Click Finish.

5. Click OK in the Installation message box.

Figure 157 Successful Installation of the UL2

6. Go to work with software update after signing into Active console on deployment server and apply UL2 to the chosen environments.

Figure 158 Defining the Software Update Parameters

Figure 159 Select the Environment (DV900 was chosen)

Figure 160 Uncheck the Create OMW Project and Package Assembly

Figure 161 Completion of the Software Update

HTML Server Install

Download the Weblogic (WLS) 10.3.6 binaries from Oracle eDelivery, and install the WLS Sever on the HTML server. To install Oracle Weblogic Server 10.3.6, see, http://docs.oracle.com/cd/E21764_01/doc.1111/e14142.pdf.

After the installation of WLS, perform the steps described in this section to create a cluster on WLS and deploy JDE Html server on it. A cluster defines groups of Weblogic Server servers that work together to improve scalability and reliability.

Create a domain. Configure the admin server port to 7501 as shown in above link in Appendix B.

Configuring the Cluster

To configure a cluster, follow these steps:

1. Sign into the admin console using the user and password configured during WLS install.

Figure 162 Weblogic Admin Console Page

2. Choose Environment > Cluster > Lock and Edit > New. Type Cluster E1C1 in the Name field.

Figure 163 Naming the Cluster

3. Click OK, to finish creating the cluster.

Figure 164 Summary of the Clusters Created

4. Click Activate changes, and then choose Machines to configure the machine that needs to be configured in the current WebLogic Server domain.

Figure 165 Create a new Machine entry

Figure 166 Summary of Machines Added

5. Type the Cluster Node name and other properties for the new Server. Click Next.

Figure 167 Defining Properties for the New Server

6. Select the machine, cluster, from dropdown menu.

Figure 168 Configuring the General Features

7. Sign into ServerManager, and select create a managed instance of type HTML server. Use the port configured in previous step.

Figure 169 Defining the Instance Properties

8. Type the configuration items for the JDV900 environment. Click Continue.

Figure 170 Validating the Configuration Items

9. Click Create Instance.

Figure 171 Completing the Creation of the Managed Instance

Oracle HTTP Server Installation

Install the Oracle HTTP server on the JDE HTML server.

To install Oracle JRF and Oracle HTTP server, see:

ADF/JRF: http://www.oracle.com/technetwork/developer-tools/adf/documentation/index.html

Oracle HTTP Server: http://docs.oracle.com/cd/E23943_01/doc.1111/e14260/overview.htm

JDE User Creation

To create the JDE user sign into the DEP900 environment on the deployment server, and follow these steps:

1. Type the Name and other credentials to login to Oracle JDE.

Figure 172 Logging into the Deployment Server

2. Fastpath to P980001.

Figure 173 Loading the P980001

3. Click Add, to add users.

Figure 174 Adding the Users

4. Type the JDE user and set the password. Data Source is default.

5. To verify the user created, search for the user.

Figure 175 Searching for the User Created

Figure 176 Verifying the User

6. Fastpath to P98OWSEC.

Figure 177 Loading the P98OWSEC

7. Search for JDE user created in previous steps.

Figure 178 Verifying the User

8. Perform configuration for system user mapping and password management.

Figure 179 Setting the Password Management Options

Figure 180 Modifying the Password Change Frequency

Oneworld Client Install

Use the shared location for client installation binaries on deployment server to install the client.

To Install the Oneworld Client, follow these steps:

1. Right-click InstallManager, and select Run as administrator. This launches the Oracle JD Edwards Install Manager.

Figure 181 Running the OneWorld Install Files

2. Click the Workstation Install link to launch the client.

Figure 182 Clicking Workstation Installation Link

3. Click Next and proceed with the installation.

Figure 183 Proceeding with the Installation

4. Choose the DV900FA (Development full package) DV900FA package to install.

Figure 184 Selecting the Development Full Package

Full Package Build

To Build the Full package, follow these steps:

1. Login to Oracle JDE and open the Active Console

2. Type GH9083 and Fastpath to the Package and Deployment Tools menu.

Figure 185 Loading the GH9083

3. Select the Package Assembly from the Package and Deployment Tools menu.

Figure 186 Selecting the Package Assembly

4. Click Add to create new package assembly.

Figure 187 Defining the new Package Assembly

5. The Package Assembly Director opens. Click Next.

Figure 188 Launching the Package Assembly Director

6. Type package name, description and pathcode. Click the Express radio button.

Figure 189 Defining the Package Information

7. Verify the Package Assembly properties. Click End.

Figure 190 Summary of the Package Features

8. Select the package and click Activate. Click Define Build.

Figure 191 Selecting the Package Build

9. Click Next.

Figure 192 Proceeding to Package Building

10. Check the Client, Server, and Share Specs checkboxes in the Build Location area.

Figure 193 Defining the Package Build Location

11. Click Next to get a list of available Enterprise servers. Select the Enterprise server.

Figure 194 Selecting Enterprise Servers

12. Verify the Server specifications. Click End.

Figure 195 Server Build Specifications

13. Click Sequence.

Figure 196 Adding the Package Sequence

14. Click Active/Inactive.

Figure 197 Activating the Package Build

15. Click Submit Build.

Figure 198 Submitting the Package Build Defined

16. Click On Screen radio button to print report to screen.

Figure 199 Selecting the Report Output Destination


Note The full package build usually takes between 4-6 hours to finish. Deploy the successfully built package on the Enterprise server


Summary

The above sections described the approach taken for installing JD Edwards 9.0.2 on a Oracle Virtualized environment with two-node Oracle RAC database. The benchmarking effort required the use of a weblogic cluster with an Oracle Http server front end to load balance the users among the various cluster nodes. It is very important to check Oracle support documents for information regarding the latest support statements in the various JDE MTRs as well as to check for recently released patches.

Oracle JD Edwards EnterpriseOne Performance and Scalability

Workload Description

The Oracle JD Edwards Day in the life (DIL) kit is an attempt to capture how a typical customer interacts with the JDE system during the course of a typical day. The DIL kit accomplishes this with a set of scripts for 17 interactive applications as well as a set of JDE reports (UBEs) which processes a specific set of data which is part of the DIL database. Due to the availability of this standard set of scripts and UBEs, various hardware vendors, including Cisco have endeavored to characterize JDE implementations on their hardware platform to deliver a value proposition for prospective customers.

The DIL kit interactive application workload skews more towards SRM applications which feature prominently in the application workloads used by the large JDE customer base in the mid-scale manufacturing industry segment. The UBE workload is also representative of the type of reports that would be run by customers in this segment, though it does incorporate reports that cater to a larger audience of customers.

The DIL workload incorporates a good mix of applications ranging from multiple line items Sales Order and Purchase Order entries, coupled with light weight applications like supplier ledger enquiry. Similarly, the UBEs range from long running MRP processing and General Ledger Post reports to the short running Company constants and Business unit reports.

The LoadRunner scripts for the JDE interactive applications that the DIL kit incorporates measure the response times for certain key, representative transactions and these are incorporated in this whitepaper. The UBE performance is measured in terms of the total time taken to generate the report, as measured by timings recorded in the JDE logs for those UBEs.

Test Methodology

The interactive and batch version of the JDE E1 DIL kit was run to capture the end-user response time variation and Batch execution rate with important system characteristics such as CPU, memory, and I/O across the servers in the test system. All four components of the JDE E1 deployment—HTML server, Enterprise Server for interactive user, Enterprise Server for Batch and Oracle database server—were monitored through nmon Linux monitoring tool. EMC nar files were used to measure the total IOPs and latency on EMC VNX5300.

Test scenarios included a broad range of JDE applications ensuring that they closely mimic how a potential customer would utilize Oracle JD Edwards EnterpriseOne. The documented response times and the best practices to deploy Oracle JDE E1 server would give customers a good indication on how they could expect such a configuration to perform when deployed in their production environments.

Cisco endeavored to truly stress the hardware configuration, as well as provide customers with scenarios which provide a mix of interactive and batch processes running concurrently. Cisco devised various scenarios to test and record the impact of running a mixed batch workload would have on the interactive performance of JDE applications, since batch processes are typically resource hungry, thereby impacting the responsiveness of JDE interactive applications.

The elaborate test scenarios executed for JDE deployment on Cisco UCS are as follows:

Interactive Scaling: Scaling of JDE interactive users from 1000 to 7,000 concurrent users

Individual UBEs: Execution of individual long running UBEs on JDE E1 Server for batch/UBE processes

Only Batch: Execution of batch/UBE processes on JDE Enterprise Server Virtual Machine without interactive apps

Interactive Workload Mix

The JDE E1 DIL kit is a set of 17 scripts that include Oracle SCM, SRM, HCM, CRM, and Financials applications.

Table 9 shows the transaction mix used for the JD Edwards EnterpriseOne interactive test with the JDE E1 DIL kit.

Table 9 Workload Mix

Oracle Application
Percentage Weight

Financial Management System

20

Supplier Relationship Management

24

Supply Chain Management

49

Customer Relationship Management

5

Human Capital Management

2

 

100%


Interactive workload Scaling

The test scenario for Interactive Scaling was executed to determine the variation in end-user response time in the event of increasing interactive users from 1000 to 7,000 concurrent users. The system resource utilization such as CPU, memory and disk IO was captured across all the three tiers i.e. HTML Server, JDE E1 Server and Oracle Database server. For 2500 concurrent interactive users, a single HTML Server VM and E1 server VM with two-node Oracle RAC VMs was utilized. For users beyond 2500 concurrent users two HTML VMs and two E1 servers VMs with a common two-node Oracle RAC VM was utilized. This was deployed to enable high interactive user scaling as well as demonstrate failover setup for HTML and E1 server.

Oracle's JDE E1 DIL Kit Illustrating Weighted Average Response Time for Interactive UsersShows the weighted average response time for 1000 to 7,000 interactive users.

Figure 200 Oracle's JDE E1 DIL Kit Weighted Average Response Time for Interactive Users

As illustrated in Figure 200, JDE E1 deployment on a Cisco UCS blade server infrastructure on Oracle Virtualized environment scales exceptionally well, with an almost flat response time of around 0.14 second for 1000 to 7,000 concurrent users. It elaborates how well Oracle JD Edwards EnterpriseOne can be consolidated on just two Cisco UCS B200 M3 servers, with peak of 7000 users.

User Response Time

User response time was captured at the LoadRunner Controller for all 17 interactive JDE E1 DIL test scripts. The five important JD Edwards EnterpriseOne applications measured using the JDE E1 DIL Kit were:

Financial Management System (FMS)

Supplier Relationship Management (SRM)

Supply Chain Management (SCM)

Customer Relationship Management (CRM)

Human Capital Management (HCM)

The transaction mix for these applications is detailed in the Workload Mix section of this document.

Oracle's JDE E1 DIL Kit Weighted Average Response Time, shows the weighted average response time for all 17 JDE E1 DIL kit scripts and for the five JD Edwards applications.

Figure 201 Oracle's JDE E1 DIL Kit Weighted Average Response Time

As detailed in Figure 201, the average response time of any of the JD Edwards application for interactive user tests was around 0.6 seconds during the scalability from 1000 to 7,000 concurrent users.

CPU Utilization

Each of the tiers; HTML server,Enterprise Server and two-node Oracle RAC database was deployed on separate Oracle VMs deployed on just two Cisco UCS B200 M3 Blade Servers, equipped with two-socket Intel Xeon E5-2690 processors. Figure 202 illustrates the average CPU utilization across the 3-Tier JD Edwards Technology Stack.

Figure 202 JDE E1 CPU Utilization

Observations:

The average CPU utilization for each of the Oracle Virtual Servers was around 70 to 75% for a peak of 7000 interactive users.

Each of the JD Edwards EnterpriseOne VMs with 10 vCPUs each was just around 35% for a peak of 7000 users.

The average CPU utilization for each of the Oracle RAC nodes VMs deployed with 10 vCPUs each was at around 40%.

The average CPU utilization recorded on each of HTML servers VMs deployed with 8 vCPUs each was around 70%.

CPU utilization across all tiers gradually increased, reflecting the linear scalability of the workload.

Memory Utilization

Memory utilization across each of the Oracle Guest VMs for the test with 1000 to 7,000 Interactive users across the 3-Tier JD Edwards technology stack is illustrated JDE E1 Memory Utilization.

Each of the JDE OVM Servers was deployed on Cisco UCS B200 M3 server equipped with 256 GB of memory.

Figure 203 JDE E1 Memory Utilization

Observations:

Each of JDE HTML Server VM was configured with 64 GB of memory. The memory utilization for each of the VM was at around 55 GB. This was due to the fact that for 3500 concurrent users on each VM, around 20 JVM instances with heap size of 2 GB each were configured in Oracle Weblogic. These instances were load balanced through Oracle HTTP Server which was installed on the same HTML Server VM.

Each of JDE EnterpriseOne Server VM was configured with 64 GB of memory. The memory utilization for each of the VM was at around 42 GB.

Oracle RAC database nodes was configured with 120 GB of physical memory. The peak utilization was at around 88 GB for total users 3500 users on each of the VM.

I/O Performance

The EMC VNX5300 was configured as the storage system for each of the three components of the JDE E1 deployment: HTML server, JDE E1 server, and Oracle RAC database server. The Cisco UCS servers were booted from the SAN (EMC VNX5300), which allows unleashing of the full capabilities of Cisco UCS statelessness. Cisco UCS stateless configuration allows migration of Cisco UCS service profiles from a failed physical server to a standby server.

JDE E1 Average IOPs on VNX5300 illustrates total disk I/O performance captured with the help of EMC NAR files, during the scalability test for 1000 to 7000 JDE E1 interactive users.

Figure 204 JDE E1 Average IOPs on VNX5300

Observations:

The number of I/O operations per second (IOPS) generated on EMC VNX5300 scaled linearly, reflecting the gradual increase in the user count.

The response time observed from the EMC generated NAR files was less than 5 ms for the duration of the test.

The VNX5300 is capable of handling a higher number of IOPS than reflected in this graph. The IOPS shown are a result of what was driven by this JDE DIL kit workload

Individual UBEs

Batch processing is another critical activity in Oracle JD Edwards EnterpriseOne environment and it was important to test and determine the execution time for long running individual UBE processes. This helps in proper sizing of batch servers for a Oracle JD Edwards deployment. In real-world JD Edwards deployment, several long running UBEs are run after business hours, and they must complete within a fixed duration of time since they are end of day reports and should not spill over to the following day affecting interactive users. The performance characteristics of these UBEs are summarized in Table 10, with a brief description of what the UBE provides and what dataset they operate on. These long running reports ran on a fixed set of data with a standard set of processing options against the JDE DIL database, as described below.

Table 10 Long Running UBE Execution Time

UBE
Time Taken (mm:ss)
Description

R43500

17:25

This Purchase Order Print UBE processed records in F4311 table (Purchase Order Detail File) with a status code of 280 in one business unit. 255,471 records were processed using this data range

R3483

15:20

This MRP UBE processed 50,000 records in F4102 using one business unit. This is a night only process.

R31410

26:45

The Work Order Processing UBE acts on the document invoice numbers in the Work Order Master File, F4801 and 28,751 records processed with the data selection

R09801

7:05

This General Ledger post UBE acts on records located in the Batch Control Records table, F0011 the data selection used a batch status "A" and batch type "G." and processed 990,099 records.

R31802a

24:30

The Manufacturing Account Journal UBE acted on document invoice numbers in the Work Order Master File, F4801 with a status code of "95." and processed 1,501 records for our data selection criterion.


Only Batch Execution

There are customers who run only JDE Batch processing for many business functions during a dedicated window and it was imperative on Cisco to test and provide enough information for such customers to make an informed decision regarding their deployment of JDE on Cisco UCS using Oracle Virtualization. For such customers a test scenario was configured wherein, high volume of short running UBEs as well as 3 long running UBEs were executed, and the impact of running this test scenario was measured in terms of CPU and memory consumed on the Enterprise server, Database server and each of the two Oracle Virtual Servers.

The test successfully achieved 320 UBEs/minute. The average IOPS measure on the EMC VNX5300 was around 3200. It would be a good strategy for real-world JD Edwards EnterpriseOne customers, to schedule very high volume of UBEs during those non-peak hours when minimal interactive users are logged in to the JDE system

Resource Utilization for Only Batch Execution, illustrates the CPU and memory utilization on JDE Enterprise Server and two-node Oracle RAC database during the execution of Only Batch Processes.

Figure 205 Resource Utilization for Only Batch Execution

Best Practices & Tuning Recommendations

Oracle JD Edwards EnterpriseOne deployed on Cisco UCS with EMC Storage system with Oracle VM was configured for small to medium scale ERP deployment. The benchmark of JDE DIL Kit demonstrated exceptional performance for JDE interactive users and JDE Batch processes. The subsequent sections elaborate on the tuning parameters and best practices incorporated across hardware and software stack.

System Configuration

All the tests were executed using two Cisco UCS B200 M3 servers. Each server was equipped with two E5-2690 processor and 256 GB of physical memory. Six Guest VMs were carved on the two B200 M3, with two instance each of HTML Server, JDE EnterpriseOne Server and Oracle RAC node.

Both of the Cisco UCS Blade Servers were attached to a BIOS policy. BIOS policy is one of the features of UCS Service Profiles which enable users to incorporate similar BIOS settings all across the deployed servers. This helps a consistent configuration and the Administrators do not need to interrupt the boot process on each server to alter the BIOS setting. The BIOS policy configured for Oracle JDE deployment is shown in Figure 206.

Figure 206 BIOS setting for CPU Performance

Figure 207 BIOS setting for Indirect IO

Figure 208 BIOS Setting for Physical Memory

Disable I/O MMU Virtualization Settings (VT-d). See,

http://docs.oracle.com/cd/E27300_01/E27307/html/vmrns-bugs.html

Jumbo Frames

Jumbo frames are supported in Oracle VM 3.1.1, this enables utilization of 10G ports for Oracle RAC interconnect traffic. Please ensure to use Oracle VM 3.1.1 Build 819 and above to successfully configure jumbo frames Oracle RAC private network.

Configuration of jumbo frames is demonstrated in the Infrastructure Setup Section of this document.

TCP Parameter Configuration

Default TCP parameters in most Linux VM distributions are conservative, and are tuned to handle 100Mb/s or 1Gb/s port speeds, and result in buffer sizes that are too small for 10Gb networks. Modifying these values can lead to significant performance gain in a 10Gbs VM network.

Buffer sizes should be adjusted to permit the maximum number of bytes in transit and prevent traffic throttling. The following values can be set on the virtual machines as well as dom0. Edit /etc/sysctl.conf for the below mentioned changes. These can be changed both on virtual machines as well as dom0. (See, Oracle VM 3: 10GbE Network Performance Tuning whitepaper

net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.core.netdev_max_backlog = 300000
net.ipv4.tcp_moderate_rcvbuf =1

Additionally, for Oracle VM Server for x86 prior to version 3.2, netfilter should be turned off on bridge devices. This is done automatically in OVM version 3.2 and later.

net.bridge.bridge-nf-call-iptables=0
net.bridge.bridge-nf-call-arptables=0
net.bridge.bridge-nf-call-ip6tables=0

Oracle RAC Configuration

Several Settings were changed on Oracle RAC cluster VMs to support the high load that the RDBMS was handling. Some of the important tuning parameters are:

The data files where configured on RIAD5 with 7 SSDs on EMC VNX5300. This provided low read and write latency thus helping lower response time for JD Edwards EnterpriseOne interactive Apps.

Virtual machines may run in one of two main modes, paravirtualized (PVM) or hardware virtualized (HVM). In paravirtualized mode, the kernel of the guest operating system is recompiled to be made aware of the virtual environment. This allows the paravirtualized guest to run at near native speed, since most memory, disk and network accesses are optimized for maximum performance. Oracle recommends creating paravirtualized virtual machines if possible, as the performance of a paravirtualized virtual machine is superior to that of a hardware virtualized guest.

Each of the data and log LUN were attached as physical disk to each of the Oracle RAC VM nodes. This provides better IO performance for both data and log disks.

To understand the memory required for high interactive and batch workloads, awr report was analyzed and the SGA target and PGA target was set to 36GB and 16GB respectively.

In general, JDE Edwards online user consumes around 150 to 200 process per 100 concurrent interactive users. For high interactive user test, the process limit was increased to 10,000.

For more details on Disk layout for Oracle RAC database, please refer Table 11. LUN Configuration for JD Edwards EnterpriseOne deployment

Some of the important Oracle RAC configuration parameters are detailed below:

Table 11 Oracle RAC Configuration parameters

NAME
VALUE

memory_max_target

0

memory_target

0

parallel_servers_target

512

pga_aggregate_target

16 GB

sga_target

36 GB

processes

10000

session_cached_cursors

4000

sessions

30030


Each of the Oracle RAC VM resided on a separate B200M3 server (Ref: Figure 14. Guest OS configuration of Oracle JD Edwards EnterpriseOne on OVM). Oracle RAC VM was configured with 10 vCPU and 120 GB of physical memory. There is a significant impact to the database memory consumption with dedicated connections that can be avoided by using shared server connections. (Ref: Doc ID 1188240.1 on https://support.oracle.com. In order to scale to 7000 users and with limited available memory for Oracle RAC VM, the present Oracle RAC setup was configured in shared connection mode.

WebLogic Server Configuration

The JRockit Java Virtual Machine (JVM) was used along with WebLogic 10.3.6. A vertical cluster of up to 20 JVMs was created and Oracle HTTP Server was used to load balance the load among the various nodes of the vertical cluster

Some of the important configuration details for the WebLogic Server are:

For optimal performance, about 150 to 200 JDE interactive users were hosted per cluster node/JVM.

The minimum and maximum heap size for each node was set to 2 GB.

The garbage collection policy was set to gencon since the pattern of object creation and destruction on the JDE HTML server indicated that a large number of short lived objects were created and destroyed frequently.

The nursery size was set to 512 MB.

The number of `gcthreads' was set to 6, java flight recorder was switched off for formal runs, and a minimum TLA size of 4k was chosen, with the preferred size being 1024 kb

Oracle JD Edwards Enterprise Server Configuration

The JD Edwards tools release 8.98.4.10 was used with JD Edwards application release 9.0.2. The number of interactive users per callobject peaked at around 9/callobject kernel.

Some of the important configuration settings for JDE initialization files are detailed below

JDE.ini

Kernel configurations:

- Security kernels 70

- Call Object kernels 400

- Workflow kernels 30

- Metadata kernels 1

[JDENET]

- maxNetProcesses=60

- maxNetConnections=8000

- maxKernelProcesses=1000

- maxNumSocketMsgQueue=1000

- maxIPCQueueMsgs=600

- maxLenInlineData=4096

- maxLenFixedData=16384

- maxFixedDataPackets=2000

- internalQueueTimeOut=90

[JDEIPC]

- maxNumberOfResources=4000

- maxNumberOfSemaphores=2000

- startIPCKeyValue=6001

- avgResourceNameLength=40

- avgHandles=200

- hashBucketSize=53

- maxMsgqMsgBytes=5096

- maxMsgqEntries=1024

- maxMsgqBytes=65536

- msgQueueDelayTimeMillis=40

jdbj.ini

JDBj-CONNECTION POOL

- minConnection=5

- maxConnection=800

- poolGrowth=5

- initialConnection=25

- maxSize=500

jas.ini

OWWEB

- MAXUser=500

- OWVirtualThreadPoolSize=800

JDENET

- maxPoolSize=500

Conclusion

This CVD demonstrates how Cisco UCS servers on Oracle VM along with the latest EMC VNX storage technologies, form a highly reliable, robust solution for Oracle JDE Edwards implementation. 

Enterprise Resource Planning (ERP) has been around for many decades and has provided agile IT practices to the Business. Organizations that have used the ERP Packages have immensely benefitted by streamlining their back end processes to improve management and improve ROI.

ERP being a business critical Application, takes a long time to implement and test; there is always a concern to move to newer technologies or experiment with the advanced features that are available today. Since this is a Business Critical Application, one of the most important concerns is predictability; will it work for us, how and at what cost?

Cisco has invested considerable time and effort to test, validate and characterize Oracle JD Edwards on Cisco UCS Platform using EMC VNX Storage with Oracle VM, thus providing comprehensive scalable architecture and best practices. By leveraging the best practices and expertise in our extensive Oracle JDE benchmark activity, customers can confidently deploy and consolidate JD Edwards EnterpriseOne on Cisco UCS platform with Oracle VM virtualization and EMC VNX storage family.

Cisco Oracle Competency Center has provided considerable information in this document by testing and characterizing the Cisco UCS on Oracle Virtual environment using the Oracle JD Edwards EnterpriseOne software stack. With the scalability demonstrated by the test results, Cisco is confident that these astounding results will prove that Cisco UCS, Oracle VM and EMC VNX storage is a solid fit for any customer considering virtualization of Oracle JD Edwards EnterpriseOne.

Bill of Materials

Table 12 and Table 13 give details of all the hardware and software components used in the CVD.

Table 12 Component Description

Description
Part #

Cisco Unified Computing System

N20-Z0001

Cisco UCS 5108 Blade Server Chassis

N20-C6508

Cisco UCS 6248UP 1RU Fabric Int/No PSU/32 UP/ 12p LICfans/no SFP+

UCS-FI-6248UP

Cisco UCS 2208XP Fabric Extender/ 8 external 10Gb ports

UCS-IOM-2208XP

Cisco UCS B200 M3 Blade Server w/o CPU, mem, HDD, mLOM/mezz

UCSB-B200-M3-CH

Intel Xeon E5-2690CPUs (2.9 GHz & 8 Cores)

UCS-CPU-E5-2690=

16GB DDR3-1600-MHz RDIMM/PC3-12800/dual rank/1.35v

UCS-MR-1X162RY-A=

8GB DDR3-1600-MHz RDIMM/PC3-12800/dual rank/1.35v

UCS-MR-1X082RY-A=

Cisco Nexus 5548UP

N5K-C5548UP-FA

Cisco Nexus 5548UP Storage Protocols Services License

N5548P-SSK9

10GBASE-SR SFP Module

SFP-10G-SR

10GBASE-CU SFP+ Cable 3 Meter

SFP-H10GB-CU3M

8 Gbps Fibre Channel LW SFP+, LC

DS-SFP-FC8G-LW=

EMC VNX Storage System

VNX5300


Table 13 Software Details

Platform
Software Type
Name
Value

UCS 6248

Management

UCSN

2.1(1a)

UCS 6248

OS

NX-OS

2.1(1a)

Nexus 5548 up

OS

NX-OS

5.0(3)N2 (1)

Blade servers

Virtualization Technology

Oracle VM 3.1.1, Build 819 and above

Oracle VM 3.1.1 (2.6.39-200.37.1.el5uek)

Guest OS

OS

Oracle Linux 5.8 (2.6.32-300.10.1.el5uek)

OL 5.8

Virtualization

Virtualization Technology

Oracle VM 3.1.1, Build 819 and above

Oracle VM 3.1.1 (2.6.39-200.37.1.el5uek)

Database

Database

Oracle RAC

11.2.0.3

Application

Application Software

Oracle JD Edwards 9.02 with Tools 8.98.4.10

9.02/ 8.98.4.10


Appendix A - Workload Mix for Batch and Interactive Test

Table 14 Batch Workload Mix

UBE name
Description
Long/short

R03b31

Activity Log Report

Short

R03b155

A/R Summary Analysis

Short

r0004p

UDC Records Types Print

Short

r0006p

Business Unit Report

Short

r0008p

Date Patterns Report

Short

r0010p

Company Constants Report

Short

r0012p1

AAI Report

Short

r0014

Payment terms Report

Short

r0018p

Tax Detail Report

Short

r01402w

Who's Who Report

Short

r41542

Item Ledger As Of Record Generation

Short

r42072

Price Category Print

Short

r41411

Select Items Cost Count

Short

R31410

Work Order Processing

Long

R43500

Purchase Order Print

Long

R3483

MRP report

Long


Table 15 JD Edwards Enterprise One Interactive Transactions

Transaction Name
Description
Virtual Users per transaction for 500 interactive user workload

H03B102E_OK

Apply Receipts

50

H0411I_1_FIND

Supplier Ledger inquiry

50

H051141E_Row_OK

Daily Time Receipt

10

H17500E_Find

Case Management Add

25

H31114U_OK

Work Order Completion

15

H3411AE_Post_OK

MRP Messages (WO Orders)

10

H3411BE_Post_OK

MRP Messages (OP Orders)

10

H3411CE_Post_OK

MRP Messages (OT Orders)

10

H4113E_OK

Inventory Transfer

25

H42101E_Submit_Close

Sales Order Entry - 10 Line Items

125

H42101U_SubmitClose

Sales Order Update

25

H4310E_Post_OK

Purchase Order Entry - 25 Line Items

100

H4312U_OK

Purchase Order Receipts

10

H4314U_Row_OK

Voucher Match

10

H4915AU_Find

Ship Confirmation - Approval only

15

H4915CE_Find

Ship Confirmation - Confirm/Ship only

5

H4915CU_Find

Ship Confirmation - Confirm and Change Entry

5


Appendix B - Reference Documents

Oracle VM 3.1.1 User Guide

JD Edwards MTRs for Windows client, enterprise servers server and database server

Oracle-JD Edwards 9.0.2 Release notes

JD Edwards EnterpriseOne Applications Release 9.0 Installation Guide for Oracle On Linux

JD Edwards EnterpriseOne 8.98.3 Clustering Best Practices with Oracle WebLogic Server

Appendix C - Reference Links

The racking, power and installation of the chassis are described in the Install Guide:

http://www.cisco.com/en/US/docs/unified_computing/ucs/hw/chassis/install/ucs5108_install.html

Cisco Unified Computing System CLI Configuration Guide: http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/cli/config/guide/2.1/b_UCSM_CLI_Configuration_Guide_2_1_chapter_010.html

Cisco UCS Manager GUI configuration Guide:

http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.1/b_UCSM_GUI_Configuration_Guide_2_1.html

VP for Unified Storage System - A Detailed Review, see:

http://www.emc.com/collateral/software/white-papers/h8058-fast-vp-unified-storage-wp.pdf

EMC FAST Cache - A Detailed Review, see:

http://www.emc.com/collateral/software/white-papers/h8046-clariion-celerra-unified-fast-cache-wp.pdf

EMC FAST VP for Unified Storage System - A Detailed Review, see:

http://www.emc.com/collateral/software/white-papers/h8058-fast-vp-unified-storage-wp.pdf

Additional information on the VNX Series, see:

http://www.emc.com/collateral/hardware/data-sheets/h8520-vnx-family-ds.pdf