Virtualizing Oracle JD Edwards on Cisco Unified Computing System
Available Languages
Table Of Contents
About Cisco Validated Design (CVD) Program
Virtualizing Oracle JD Edwards on Cisco Unified Computing System
Cisco Unified Computing System Overview
Main Differentiating Technologies
Cisco Data Center VM-FEX Virtualization Support and Virtualization Adapter
Dynamic Provisioning with Service Profiles
Cisco Unified Computing System Components
Cisco UCS Fabric Interconnects
Cisco UCS Virtual Interface Card 1240
Cisco UCS Virtual Interface Card 1280
Cisco Nexus 5500 Series Switch
Programmatically Deploying Server Resources
Dynamic Provisioning with Service Profiles
Service Profiles and Templates
Oracle JD Edwards EnterpriseOne
Design Considerations for Oracle JD Edwards EnterpriseOne Implementation on Cisco UCS with Oracle VM
Scalable Architecture Using Cisco UCS Servers
Sizing Guidelines for Oracle JD Edwards EnterpriseOne
Oracle JD Edwards EnterpriseOne Deployment Architecture
Cisco Nexus Switch Configuration
First Time Setup of Cisco Nexus 5548 UP Switch
Enabling Features and Global Configuration
Creating VSAN and Adding FC Interfaces
Virtual Port Channel (vPC) Configuration
Setting Up Jumbo Frames on Cisco Nexus 5500
Cisco Unified Computing System Configuration
Creating Uplink Ports Channels
Modifying Service Profile for Boot Policy
UCS Service Profile Configuration
Creating Service Profile Templates
Creating Service Profile from the Template and Associating it to Cisco UCS Blade Server
Creation of Zoneset and Zones on Cisco Nexus 5548UP Switch
Configuring Connectivity for EMC VNX Storage
Creation of Storage Pools/ RAID Groups
Oracle Virtual Server (OVS) Configuration
Oracle Virtual Server Installation
Oracle JD Edwards EnterpriseOne Installation
Oracle JD Edwards EnterpriseOne Specific Install Requirements
Oracle JDE Deployment Server Install
Tools Upgrade on the Deployment Server
Oracle HTTP Server Installation
Install the Oracle HTTP server on the JDE HTML server.
Oracle JD Edwards EnterpriseOne Performance and Scalability
Best Practices & Tuning Recommendations
Oracle JD Edwards Enterprise Server Configuration
Appendix A - Workload Mix for Batch and Interactive Test
Appendix B - Reference Documents
Virtualizing Oracle JD Edwards on Cisco Unified Computing SystemOracle JD Edwards EnterpriseOne on Cisco UCS with Oracle VM 3.1.1 and EMC VNX 5300Last Updated: January 2, 2014
Building Architectures to Solve Business Problems
About the Authors
Anil Dhiman, Technical Marketing Engineer, SAVBU, Cisco SystemsAnil Dhiman is a Technical Marketing Engineer with Server Access Virtualization Business Unit (SAVBU) at Cisco. Anil has over 12 years of experience in benchmarking and performance analysis of large multi-tier systems, such as Foreign Exchange products. Anil specializes in optimizing and tuning of applications deployed on J2EE application servers and has delivered world record numbers for SPECjbb2005 and SPECjbb2013 benchmarks on Cisco Unified Computing Systems. Anil has worked as a performance engineer for Oracle IAS team. Prior to joining Cisco, Anil worked as a performance engineering architect with Symphony Services.
Acknowledgment
For their support and contribution to the design, validation, and creation of the Cisco Validated Design, I would like to thank:
•
Vadiraja Bhatt- Cisco
•
•
•
•
•
•
•
About Cisco Validated Design (CVD) Program
The CVD program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information visit:
http://www.cisco.com/go/designzone
ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at http://www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1005R)
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.
© 2013 Cisco Systems, Inc. All rights reserved.
Virtualizing Oracle JD Edwards on Cisco Unified Computing System
Executive Summary
Oracle JD Edwards is a suite of products from Oracle that caters to the Enterprise Resource Planning (ERP) requirement of an Organization. Oracle has three flag ship ERP Applications, Oracle E-Business Suite, PeopleSoft, and JD Edwards. ERP Applications have been thriving and improving the productivity of Organizations for a couple of decades now. But with the increased complexity and extreme performance requirements, customers are constantly looking for better Infrastructure to host and run these Applications.
This document presents a differentiated solution using Oracle Virtualized Environment that validates Oracle JD Edwards EnterpriseOne environment on Cisco Unified Computing System (UCS) with Oracle VM 3.1.1, Oracle Linux 5.8, Oracle 11gR2 RAC Database, Cisco UCS Blade Servers, Cisco Nexus 5548UP Switches, Cisco UCS Management System and EMC VNX5300. Cisco Oracle Competency Center tested, validated, benchmarked and has showcased the Oracle JD Edwards ERP Application using Oracle Day in the Life (DIL) Kit.
Purpose of this Guide
This design guide demonstrates the best practices for deployment of Oracle JD Edwards EnterpriseOne on Oracle Virtualized Environment. It elaborates on step by step procedure to deploy Cisco UCS, Cisco Nexus family of switches, EMC VNX 5300 and Oracle JD Edwards application on Oracle VM.
The design was validated by measuring the performance and scalability of Oracle JD Edwards. This was achieved by executing Oracle JD Edwards Day in the Life (DIL) Kit on Cisco UCS Platform by benchmarking various Application workloads using Load Runner tool. Oracle JDE DIL kit comprises of interactive Application workloads and batch workload, Universal Batch Engine Processes (UBEs). The interactive Application Users were validated and benchmarked by scaling from 1000 to 7,000 concurrent users. Achieving sub second response times for various JDE Application workloads with a large variation of interactive apps and UBEs, clearly demonstrate the suitability of Cisco UCS for small to large Oracle JD Edwards EnterpriseOne and help customers to make an informed decision on choosing Cisco UCS for their virtualized Oracle JD Edwards implementation.
Business Needs
Customers constantly look value for money, when they transition from one platform to another or migrate from proprietary systems to commodity hardware platforms; they endeavor to improve operational efficiency and optimal resource utilization.
Other important aspects are Management and Maintenance; ERP Applications are business critical Applications and need to be up and running without down time. Ease of Maintenance and efficient management with minimal staff and reduced budgets are pushing infrastructure managers to balance uptime and ROI.
Organizations have steadily adopted virtualization technologies in recent years to reduce the size, cost, and complexity of their infrastructure. Virtualization technologies let IT departments run multiple operating systems and applications on the same physical server, thereby allowing them to consolidate server hardware.
Server sprawl, old technologies that consume precious real estate space and power with increase in the cooling requirement have pushed customers to look for innovative technologies that can address some of these challenges.
Audience
The target audience for this guide include sales engineers, field consultants, professional services staff, IT managers, partner engineering staff, and customers who want to deploy Oracle JDE on Cisco UCS with EMC VNX5300 and Oracle VM.
This design guide is intended to assist Solution Architects, JDE Project Manager, Infrastructure Manager, Sales Engineers, Field Engineers and Consultants in planning, designing, and deployment of Oracle JD Edwards hosted on Cisco UCS environment. This document assumes that the reader has an architectural understanding of the Cisco UCS servers, Nexus 5548UP Switch, Oracle JD Edwards, EMC VNX5300, Oracle VM and related software.
Solution Overview
The solution in the design guide demonstrates the deployment of Oracle JD Edwards EnterpriseOne on Cisco UCS. The Oracle JD Edwards solution architecture is designed to run on multiple platforms and multiple databases. In this deployment, the Oracle JD Edwards Enterprise One (JDE E1) Release 9.0.2 was deployed on Oracle VM 3.1.1 with Oracle Linux 5.8 as the OS for Guest VMs. The JDE E1 database was hosted on two-node Oracle 11gR2, and the JDE HTML server ran on Oracle Weblogic Server Release 10.3.6. The design validates the use of Oracle RAC database for mission critical applications such as JDE E1 to provide high availability and scalability.
The deployment and testing was conducted in a Cisco® test and development environment to measure the performance of Oracle's JDE E1 Release 9.0.2 on Oracle VM with Cisco UCS. The JDE E1 DIL kit was deployed as a test workload which is a suite of 17 test scripts that exercise representative transactions of the most popular JDE E1 applications, including SCM, SRM, HCM, CRM, and Financial Management. This complex mix of applications simulate workloads to more closely reflect the customer environments.
The solution describes the following aspects of Oracle JD Edwards deployment on Cisco UCS:
•
•
–
–
•
–
–
•
–
–
•
•
–
–
•
–
–
•
Figure 1 illustrates the components of JD Edwards E1 using Cisco UCS Servers.
Figure 1 Deployment Overview of JD Edwards EnterpriseOne on Cisco UCS with EMC VNX5300
Technology Overview
Cisco Unified Computing System Overview
Cisco UCS is a set of pre-integrated data center components, including Cisco Blade Servers, Adapters, Fabric Interconnects, and Fabric Extenders, that are integrated within a common embedded management system. This approach results in far fewer system components and much better manageability, operation efficiency, and more flexibility than comparable data center platforms.
Main Differentiating Technologies
The main differentiating technologies described here make Cisco UCS unique and give it advantages over competing offerings. The technologies presented here are high level, and the discussions do not include the technologies (such as Fibre Channel over Ethernet [FCoE]) that support these high-level elements.
Unified Fabric
Unified fabric can dramatically reduce the number of network adapters, Cisco Blade Server Switches, cables, and management touch points by passing all network traffic to parent Cisco Fabric Interconnects, where it can be prioritized, processed, and managed centrally. This approach improves performance, agility, and efficiency and dramatically reduces the number of devices that need to be powered, cooled, secured, and managed.
Embedded Multirole Management
Cisco UCS Manager is a centralized management application that is embedded on the fabric switch. Cisco UCS Manager controls all Cisco UCS elements within a single redundant management domain. These elements include all aspects of system configuration and operation, eliminating the need to use multiple, separate element managers for each system component. Massive reduction in the number of management modules, consoles and in the proliferation of agents resident on all the hardware (which must be separately managed and updated) are important deliverables of Cisco UCS. Cisco UCS Manager, using role-based access and visibility, helps enable cross-function communication efficiency, promoting collaboration between data center roles for increased productivity.
Cisco Data Center VM-FEX Virtualization Support and Virtualization Adapter
With Cisco data center VM-FEX, virtual machines have virtual links that allow them to be managed in the same way as physical links. Virtual links can be centrally configured and managed without the complexity of traditional systems, which interpose multiple switching layers in virtualized environments. I/O configurations and network profiles move along with virtual machines, helping increase security and efficiency while reducing complexity. Cisco data center VM-FEX helps improve performance and reduce Network Interface Card (NIC) infrastructure.
Dynamic Provisioning with Service Profiles
Cisco UCS Manager delivers service profiles, which contain abstracted server-state information, creating an environment in which everything unique about a server is stored in the fabric, and the physical server is simply another resource to be assigned. Cisco UCS Manager implements role-based and policy-based management focused on service profiles and templates. These mechanisms fully provision one or many servers and their network connectivity in minutes, rather than hours or days.
Cisco UCS Manager
Cisco UCS Manager (UCSM) is an embedded, unified manager that provides a single point of management for Cisco UCS. Cisco UCS Manager can be accessed through an intuitive GUI, a command-line interface (CLI), or the comprehensive open XML API. It manages the physical assets of the server and storage - LAN connectivity and it is designed to simplify the management of virtual network connections through integration with several major hypervisor vendors. It provides IT departments with the flexibility to allow people to manage the system as a whole, or to assign specific management functions to individuals based on their roles as managers of server or network hardware assets. It simplifies operations by automatically discovering all the components available on the system and enabling a stateless model for resource use. Every instance of Cisco UCS Manager and all of the components managed by it form a domain. Multiple Domains can be managed through Cisco UCS Central which integrates with Cisco UCS Manager, and utilizes it to provide global configuration capabilities for pools, policies, and firmware. Cisco UCS Central software manages multiple, globally distributed Cisco UCS domains with thousands of servers from a single pane.
The elements managed by Cisco UCS Manager include:
•
•
•
•
•
•
Cisco Unified Computing System Components
Figure 2 shows the Cisco Unified Computing System (UCS) components.
Figure 2 Cisco UCS Components
Cisco UCS is designed from the start to be programmable and self-integrating. A server's entire hardware stack, ranging from server firmware and settings to network profiles, is configured through model-based management. With Cisco virtual interface cards (VICs), even the number and type of I/O interfaces is programmed dynamically, making every server ready to power any workload at any time.
With model-based management, administrators manipulate a model of a desired system configuration and associate a model's service profile with hardware resources, and the system configures itself to match the model. This automation accelerates provisioning and workload migration with accurate and rapid scalability. The result is increased IT staff productivity, improved compliance, and reduced risk of failures due to inconsistent configurations.
Cisco FEX Technology reduces the number of system components that need to be purchased, configured, managed, and maintained by condensing three network layers into one. It eliminates both blade server and hypervisor-based switches by connecting Cisco UCS Fabric Interconnect ports directly to individual blade servers and virtual machines. Virtual networks are now managed exactly the same way that physical networks are, but enable massive scalability. This approach represents a radical simplification compared to traditional systems, reducing capital expenditures (CapEx) and operating expenses (OpEx) while increasing business agility, simplifying and accelerating deployment, and improving performance.
Cisco UCS Fabric Interconnects
Cisco UCS Fabric Interconnects create a unified network fabric throughout Cisco UCS. They provide uniform access to both networks and storage, eliminating the barriers to deployment of a fully virtualized environment based on a flexible, programmable pool of resources. Cisco Fabric Interconnects comprise a family of line-rate, low-latency, lossless 10 Gigabit Ethernet, IEEE Data Center Bridging (DCB), and FCoE interconnect switches. Based on the same switching technology as the Cisco Nexus® 5000 Series Switches, Cisco UCS 6200 Series Fabric Interconnects provide additional features and management capabilities that make them the central nervous system of Cisco UCS. The Cisco UCS Manager software runs inside the Cisco UCS Fabric Interconnects. The Cisco UCS 6200 Series Fabric Interconnects expand the Cisco UCS networking portfolio and offer higher capacity, higher port density, and lower power consumption. These interconnects provide the management and communication backbone for the Cisco UCS B-Series Blade Servers and Cisco UCS Blade Server Chassis. All chassis and all blades that are attached to interconnects are part of a single, highly available management domain. By supporting unified fabric, the Cisco UCS 6200 Series provides the flexibility to support LAN and SAN connectivity for all blades within its domain at configuration time. Typically deployed in redundant pairs, Cisco UCS Fabric Interconnects provide uniform access to both networks and storage, facilitating a fully virtualized environment.
The Cisco UCS Fabric Interconnect portfolio currently consists of the Cisco UCS 6248UP Fabric Interconnects deployed in redundant pair.
Figure 3 Cisco UCS 6248UP 48-Port Fabric Interconnect
The Cisco UCS 6248UP 48-Port Fabric Interconnect is a one-rack-unit (1RU) 10 Gigabit Ethernet, IEEE DCB, and FCoE interconnect providing more than 1-terabit-per-second (Tbps) throughput with low latency. It has 32 fixed ports of Fibre Channel, 10 Gigabit Ethernet, IEEE DCB, and FCoE Enhanced Small Form-Factor Pluggable (SFP+) ports.
One expansion module slot can provide up to 16 additional Fibre Channel, 10 Gigabit Ethernet, IEEE DCB, and FCoE SFP+ ports.
Cisco UCS Virtual Interface Card 1240
A Cisco innovation, the Cisco UCS VIC 1240 is a four-port 10 Gigabit Ethernet, FCoE-capable modular LAN on motherboard (mLOM) designed exclusively for the M3 generation of Cisco UCS B-Series Blade Servers. When used in combination with an optional port expander, the Cisco UCS VIC 1240 capabilities can be expanded to eight ports of 10 Gigabit Ethernet.
Cisco UCS Virtual Interface Card 1280
A Cisco innovation, the Cisco UCS VIC 1280 is an eight-port 10 Gigabit Ethernet, FCoE-capable mezzanine card designed exclusively for Cisco UCS B-Series Blade Servers.
The Cisco UCS VIC 1240 and 1280 enable a policy-based, stateless, agile server infrastructure that can present up to 256 PCI Express (PCIe) standards-compliant interfaces to the host that can be dynamically configured as either NICs or HBAs. In addition, the Cisco UCS VIC 1280 supports Cisco Data Center VM-FEX technology, which extends the Cisco UCS Fabric Interconnect ports to virtual machines, simplifying server virtualization deployment.
Figure 4 Cisco UCS 5100 Series Blade Server Chassis
The Cisco UCS 5108 Blade Server Chassis is a 6RU blade chassis that accepts up to eight half-width Cisco UCS B-Series Blade Servers or up to four full-width Cisco UCS B-Series Blade Servers, or a combination of the two. The Cisco UCS 5108 can accept four redundant power supplies with automatic load sharing and failover and two Cisco UCS 2100 or 2200 Series Fabric Extenders. The chassis is managed by Cisco UCS chassis management controllers, which are mounted in the Cisco UCS Fabric Extenders and work in conjunction with Cisco UCS Manager to control the chassis and its components.
A single Cisco UCS managed domain can theoretically scale to up to 40 individual chassis and 320 blade servers. At this time, Cisco UCS supports up to 20 individual chassis and 160 blade servers.
Basing the I/O infrastructure on a 10-Gbps unified network fabric allows Cisco UCS to have a streamlined chassis with a simple yet comprehensive set of I/O options. The result is a chassis that has only five basic components:
•
•
•
•
•
Figure 5 Cisco UCS B200 M3 Blade Servers
The Cisco UCS B200 M3 Blade Server delivers performance, versatility, and density without compromise. It addresses the broadest set of workloads, from IT and web infrastructure to distributed databases. Building on the success of the Cisco UCS B200 M2 Blade Server, the enterprise-class Cisco UCS B200 M3 Blade Server further extends the capabilities of the Cisco UCS portfolio in a half-width blade form factor. The Cisco UCS B200 M3 harnesses the power of the latest Intel Xeon processor E5-2600 product family, with up to 384 GB of RAM (using 16-GB DIMMs), two disk drives, and up to dual 4x 10 Gigabit Ethernet throughput. In addition, Cisco UCS has the architectural advantage of not having to power and cool excess switches in each blade chassis. With a larger power budget per blade server, Cisco can design uncompromising expandability and capabilities in its blade servers, as evidenced by the new Cisco UCS B200 M3, with its leading memory slot and drive capacity.
Cisco Nexus 5500 Series Switch
The Cisco Nexus 5500 Series Switch is designed for data center environments with cut-through technology that enables consistent, low-latency Ethernet solutions with front-to-back or back-to-front cooling and data ports in the rear, bringing switching into close proximity with servers and making cable runs short and simple. The switch series is highly serviceable, with redundant, hot-pluggable power supplies and fan modules. It uses data-center-class Cisco NX-OS Software for high reliability and ease of management.
Cisco Nexus 5500 series switches provide higher density, lower latency, and multilayer services. The Cisco Nexus 5500 series switch is well suited for enterprise-class data center server access layer deployments across a diverse set of physical, virtual, storage-access, and high-performance computing (HPC) data center environments.
Figure 6 Cisco Nexus 5548UP
The Cisco Nexus 5548UP is a 1RU 10 Gigabit Ethernet (10 GE), Fibre Channel (FC), and Fibre Channel over Ethernet (FCoE) switch offering up to 960 Gbps of throughput and up to 48 ports. The switch has 32 unified ports and one expansion slot supporting modules with 10 Gigabit Ethernet and FCoE ports or connectivity to Fibre Channel SANs with 8/4/2/1 Gbps Fibre Channel switch ports, or both.
Cisco UCS Service Profiles
Programmatically Deploying Server Resources
Cisco UCS Manager provides centralized management capabilities, creates a unified management domain, and serves as the central nervous system of the Cisco UCS. Cisco UCS Manager is embedded device management software that manages the system from end-to-end as a single logical entity through an intuitive GUI, CLI, or XML API. Cisco UCS Manager implements role- and policy-based management using service profiles and templates. This construct improves IT productivity and business agility. Now infrastructure can be provisioned in minutes instead of days, shifting IT's focus from maintenance to strategic initiatives.
Dynamic Provisioning with Service Profiles
Cisco UCS resources are abstract in the sense that their identity, I/O configuration, MAC addresses and WWNs, firmware versions, BIOS boot order, and network attributes (including QoS settings, ACLs, pin groups, and threshold policies) all are programmable using a just-in-time deployment model. The manager stores this identity, connectivity, and configuration information in service profiles that reside on the Cisco UCS 6200 Series Fabric Interconnect. A service profile can be applied to any blade server to provision it with the characteristics required to support a specific software stack. A service profile allows server and network definitions to move within the management domain, enabling flexibility in the use of system resources. Service profile templates allow different classes of resources to be defined and applied to a number of resources, each with its own unique identities assigned from predetermined pools.
Service Profiles and Templates
A service profile contains configuration information about the server hardware, interfaces, fabric connectivity, and server and network identity. The Cisco UCS Manager provisions servers utilizing service profiles. The Cisco UCS Manager implements a role-based and policy-based management focused on service profiles and templates. A service profile can be applied to any blade server to provision it with the characteristics required to support a specific software stack. A service profile allows server and network definitions to move within the management domain, enabling flexibility in the use of system resources.
Service profile templates are stored in the Cisco UCS 6200 Series Fabric Interconnects for reuse by server, network, and storage administrators. Service profile templates consist of server requirements and the associated LAN and SAN connectivity. Service profile templates allow different classes of resources to be defined and applied to a number of resources, each with its own unique identities assigned from predetermined pools.
The Cisco UCS Manager can deploy the service profile on any physical server at any time. When a service profile is deployed to a server, the Cisco UCS Manager automatically configures the server, adapters, Fabric Extenders, and Fabric Interconnects to match the configuration specified in the service profile. A service profile template parameterizes the UIDs that differentiate between server instances.
This automation of device configuration reduces the number of manual steps required to configure servers, Network Interface Cards (NICs), Host Bus Adapters (HBAs), and LAN and SAN switches. The Figure 9 shows the Service profile which contains abstracted server state information, creating an environment to store unique information about a server.
Figure 7 Service Profile
UCS Central
Cisco UCS® Manager provides a single point of management for an entire Cisco Unified Computing System™ (Cisco UCS) domain of up to 160 servers and associated infrastructure. Cisco UCS Manager uses a policy-based approach to provision servers by applying a desired configuration to physical infrastructure. Using Cisco UCS service profiles, administrators can reproduce existing physical environments, including the I/O configuration, firmware, and settings. The configuration is applied quickly, accurately, and automatically, improving business agility. A role-based access control (RBAC) model helps ensure security of the system configurations.
Cisco UCS Central Software extends the simplicity and agility of managing a single Cisco UCS domain across multiple Cisco UCS domains. Cisco UCS Central Software allows companies to easily work on a global scale, putting computing capacity close to users while managing infrastructure with centrally defined policies. Cisco UCS Central Software makes it easy to help ensure global policy compliance, with subject-matter experts choosing the resource pools and policies that need to be enforced globally or managed locally. With a simple drag-and-drop operation, Cisco UCS service profiles can be moved between geographies to enable fast deployment of infrastructure, when and where it is needed, to support business workloads. Cisco UCS Central Software does not replace Cisco UCS Manager, which is the basic engine for managing a Cisco UCS domain. It builds on the capabilities provided by Cisco UCS Manager and requires it to be in place to effect changes in individual domains.
Figure 8 Cisco UCS Central Software Architecture
Features and Benefits of Cisco UCS Central
Cisco UCS Central Software enables global management of many Cisco UCS domains, making staff more efficient and effective. It gives Cisco UCS administrators a high-level view and management of all, or groups of, Cisco UCS domains with:
•
•
•
•
•
EMC VNX Storage Family
The EMC VNX family of storage systems represents EMC's next generation of unified storage, optimized for virtual environments, while offering a cost effective choice for deploying mission-critical enterprise applications such as Oracle JD EDwards. The massive virtualization and consolidation trends with servers demand a new storage technology that is dynamic and scalable. The EMC VNX series meets these requirements and offers several software and hardware features for optimally deploying enterprise applications such. The EMC VNX family is shown in Figure 9.
Figure 9 The EMC VNX Family of Unified Storage Platforms
A key distinction of the VNX Series is support for both block and file-based external storage access over a variety of access protocols, including Fibre Channel (FC), iSCSI, FCoE, NFS, and CIFS network shared file access. Furthermore, data stored in one of these systems, whether accessed as block or file-based storage objects, is managed uniformly via Unisphere®, a web-based interface window. For more information on Unisphere see the White Paper titled, Introducing EMC Unisphere: A Common Midrange Element Manager.
EMC VNX Storage Platforms
The EMC VNX Series continues the EMC tradition of providing some of the highest data reliability and availability in the industry. Apart from this they also include in their design a boost in performance and bandwidth to address the sustained data access bandwidth rates. The new system design has also placed heavy emphasis on storage efficiencies and density, as well as crucial green storage factors, such as a smaller data center footprint, lower power consumption, and improvements in power reporting.
The VNX has many features that help improve availability. Data protection is heightened by the lack of any single point of failure from the network to the actual disk drive in which the data is stored. The data resides on the VNX for block storage system, which delivers data availability, protection, and performance. The VNX uses RAID technology to protect the data at the drive level. All data paths to and from the network are redundant.
The basic design principle for the VNX Series storage platform includes the VNX for file front end and the VNX for block hardware for the storage processors on the back end. The control flow is handled by the storage processors in block-only systems and the control station in the file-enabled systems. The VNX OE for block software has been designed to ensure the I/O is well balanced between the two SPs. At the time of provisioning, the odd number LUNs are owned by one SP and even number LUNs are owned by the other. This results in LUNs being evenly distributed between the two SPs. If a failover occurs, LUNs would trespass over to the alternate path/SP. EMC PowerPath® restores the "default": path once the error condition is recovered. This brings the LUNs back to a balanced state between the SPs. For more information on EMC VNX Series, see:
http://www.emc.com/collateral/hardware/data-sheets/h8520-vnx-family-ds.pdf
Key efficiency features available with the EMC VNX Series include FAST Cache and FAST VP.
FAST Cache Technology
FAST Cache is a storage performance optimization feature that provides immediate access to frequently accessed data. In traditional storage arrays, the DRAM caches are too small to maintain the hot data for a long period of time. Very few storage arrays give an option to non-disruptively expand DRAM cache, even if they support DRAM cache expansion. FAST Cache extends the available cache to customers by up to 2 TB using enterprise Flash drives. FAST Cache tracks the data temperature at 64 KB granularity and copies hot data to the Flash drives once its temperature reaches a certain threshold. After a data chunk gets copied to FAST Cache, the subsequent accesses to that chunk of data will be served at Flash latencies. Eventually, when the data temperature cools down, the data chunks get evicted from FAST Cache and will be replaced by newer hot data. FAST Cache uses a simple Least Recently Used (LRU) mechanism to evict the data chunks.
FAST Cache is built on the premise that the overall applications' latencies can improve when most frequently accessed data is maintained on a relatively smaller sized, but faster storage medium, like Flash drives. FAST Cache identifies the most frequently accessed data which is temporary and copies it to the flash drives automatically and non-disruptively. The data movement is completely transparent to applications, thereby making this technology application-agnostic and management-free. For example, FAST Cache can be enabled or disabled on any storage pool simply by selecting/clearing the "FAST Cache" storage pool property in advanced settings.
FAST Cache can be selectively enabled on a few or all storage pools within a storage array, depending on application performance requirements and SLAs.
There are several distinctions to EMC FAST Cache:
•
•
•
•
Figure 10 EMC FAST Cache
Additional information on EMC Fast Cache is documented in the white paper titled EMC FAST Cache - A Detailed Review which is available at:
http://www.emc.com/collateral/software/white-papers/h8046-clariion-celerra-unified-fast-cache-wp.pdf
FAST VP
VNX FAST VP is a policy-based auto-tiering solution for enterprise applications. FAST VP operates at a granularity of 1 GB, referred to as a "slice." The goal of FAST VP is to efficiently utilize storage tiers to lower customers' TCO by tiering colder slices of data to high-capacity drives, such as NL-SAS, and to increase performance by keeping hotter slices of data on performance drives, such as Flash drives. This occurs automatically and transparently to the host environment. High locality of data is important to realize the benefits of FAST VP. When FAST VP relocates data, it will move the entire slice to the new storage tier. To successfully identify and move the correct slices, FAST VP automatically collects and analyzes statistics prior to relocating data. Customers can initiate the relocation of slices manually or automatically by using a configurable, automated scheduler that can be accessed from the Unisphere management tool. The multi-tiered storage pool allows FAST VP to fully utilize all the storage tiers: Flash, SAS, and NL-SAS. The creation of a storage pool allows for the aggregation of multiple RAID groups, using different storage tiers, into one object. The LUNs created out of the storage pool can be either thickly or thinly provisioned. These "pool LUNs" are no longer bound to a single storage tier. Instead, they can be spread across different storage tiers within the same storage pool. If you create a storage pool with one tier (Flash, SAS, or NL-SAS), then FAST VP has no impact on the performance of the system. To operate FAST VP, you need at least two tiers.
Additional information on EMC FAST VP for Unified Storage is documented in the white paper titled EMC FAST VP for Unified Storage System - A Detailed Review, see:
http://www.emc.com/collateral/software/white-papers/h8058-fast-vp-unified-storage-wp.pdf
FAST Cache and FAST VP are offered in a FAST Suite package as part of the VNX Total Efficiency Pack. This pack includes the FAST Suite which automatically optimizes for the highest system performance and lowest storage cost simultaneously. In addition, this pack includes the Security and Compliance Suite which keeps data safe from changes, deletions, and malicious activity. For additional information on this Total Efficiency Pack as well as other offerings such as the Total Protection Pack, see:
http://www.emc.com/collateral/software/data-sheet/h8509-vnx-software-suites-ds.pdf
EMC PowerPath
EMC PowerPath is host-based software that provides automated data path management and load-balancing capabilities for heterogeneous server, network, and storage deployed in physical and virtual environments. A critical IT challenge is being able to provide predictable, consistent application availability and performance across a diverse collection of platforms. PowerPath is designed to address those challenges, helping IT meet service-level agreements and scale-out mission-critical applications.
This software supports up to 32 paths from multiple HBAs (iSCSI TCI/IP Offload Engines [TOEs] or FCoE CNAs) to multiple storage ports when the multipathing license is applied. Without the multipathing license, PowerPath will use only a single port of one adapter (PowerPath SE). In this mode, the single active port can be zoned to a maximum of two storage ports. This configuration provides storage port failover only, not host-based load balancing or host-based failover. It is supported, but not recommended, if the customer wants true I/O load balancing at the host and also HBA failover.
PowerPath balances the I/O load on a host-by-host basis. It maintains statistics on all I/O for all paths. For each I/O request, PowerPath intelligently chooses the most under utilized path available. The available under utilized path is chosen based on statistics and heuristics, and the load-balancing and failover policy in effect.
In addition to the load balancing capability, PowerPath also automates path failover and recovery for high availability. If a path fails, I/O is redirected to another viable path within the path set. This redirection is transparent to the application, which is not aware of the error on the initial path. This avoids sending I/O errors to the application. Important features of PowerPath include standardized path management, optimized load balancing, and automated I/O path failover and recovery.
For more information on Powerpath, see:
http://www.emc.com/collateral/software/data-sheet/l751-powerpath-ve-multipathing-ds.pdf
Oracle VM
Oracle VM is a platform that provides a fully equipped environment with all the latest benefits of virtualization technology. Oracle VM enables you to deploy operating systems and application software within a supported virtualization environment. Oracle VM is a Xen-based hypervisor that runs at nearly bare-metal speeds.
Oracle VM Architecture
Figure 11 elaborates on the Oracle VM architecture. The Oracle VM architecture has three important components:
Figure 11 Oracle VM Architecture
•
•
•
Advantage of using Oracle VM for Oracle JD Edwards EnterpriseOne with Oracle RAC Database
Oracle's virtualization technologies are an excellent delivery vehicle for Independent Software Vendors (ISV's) looking for a simple, easy-to-install and easy-to-support application delivery solution.
Oracle now offers a highly available, grid-ready virtualization solution for your data center, combining all the benefits of a fully virtualized environment.
The combination of Oracle VM and Oracle JD Edwards EnterpriseOne enables a better server consolidation (bare-metal deployments with under utilized CPU resources can often benefit from consolidation with other workloads using server virtualization) sub-capacity licensing, and rapid provisioning.
Oracle RAC on Oracle VM also supports the creation of non-production virtual clusters on a single physical server for production demos and test/dev environments. This deployment combination permits dynamic changes to pre-configured database resources for agile responses to changing service level requirements common in consolidated environments.
Moreover, Oracle VM is the only software based virtualization solution that is fully supported and certified for both Oracle JD Edwards and Oracle real Application Clusters.
There are several reasons why customers may want to run Oracle JD Edwards EnterpriseOne in an Oracle VM environment. Some of the more common ones are:
•
•
•
–
–
–
•
Oracle VM Templates for JD Edwards EnterpriseOne enable rapid implementation of your JD Edwards EnterpriseOne system. With these pre configured, pretested templates of the JD Edwards EnterpriseOne enterprise, database, and HTML servers, you can have your system up and running in just a few days. Oracle VM Templates contain the complete technology stack necessary to run the server: operating system, database, web server, and JD Edwards EnterpriseOne tools and applications.
Paravirtualized VM (PVM)
Guest virtual machines running on Oracle VM server can be configured in para-virtualized mode. In this mode the kernel of the guest operating system is modified to distinguish that it is running on a hypervisor instead of on the bare metal hardware. As a result, I/O actions and system clock timers in particular are handled more efficiently, as compared with non-paravirtualized systems where I/O hardware and timers have to be emulated in the operating system. Oracle VM supports PV kernels for Oracle Linux and Red Hat Enterprise Linux, offering better performance and scalability.
Oracle JD Edwards EnterpriseOne
Oracle JD Edwards is the ERP solution of choice for many small and medium-sized businesses (SMB). JDE E1 offers an attractive combination of a large number of easy-to-deploy and easy-to-use ERP applications across multiple industries. These applications include Supply Chain Management (SCM), Human Capital Management (HCM), Supplier Relationship Management (SRM), Financials, and Customer Relationship Management (CRM). The various components of Oracle JD Edwards EnterpriseOne are shown in Figure 12.
Figure 12 JD Edwards EnterpriseOne Components
HTML Server
HTML Server is the interface of the JDE to the outside world. It allows the JDE ERP users to connect to their Applications using their Browsers via the Web server. It is one of the Tiers of the standard three-tier JDE Architecture. HTML Server is just not an interface, it has logic and runs Web Services which processes some of the data and only the result set is sent through the WAN to the end users.
Enterprise Server
Enterprise Server hosts the JDE Applications that execute all the basic functions of the JDE ERP systems, like running the Transaction processing service, Batch Services, data replication, security and the entire time stamp and distributed processing happens at this layer. Multiple enterprise servers can be added for scalability, especially when we need to apply Electronic Software Updates (ESU) to one server while the other is online.
Database Server
Database Server in a JDE environment is used to host the data. It simply is a data repository and is not used to process JDE logic. The JDE Database server can run many supported databases such as, Oracle, SQL Server, DB2 or Access. Since this server does not run any Applications as mentioned, the only licensing that is required for this server is the database license; hence the server should be sized correctly. If this server has excess capacity the UBE's can be run on this server to improve their performance.
Deployment Server
Deployment server essentially is a centralized software (C Code) repository for deploying software packages to all the servers and workstations that are part of the Cisco JDE solution. Although the Deployment server is not a business critical server, it is very important to note that it is a critical piece of the JDE Architecture, without which the Installation, upgrade, development or modification of packages (codes) or reports would become impossible.
Server Manager
The Server Manager is a key JDE software component that helps customers deploy the latest JDE tools software onto various JDE Servers that are registered with the server manager. The server manager is web based and enables life cycle management of JDE products like the Enterprise server and HTML server via a web based console. It has in-built abilities for configuration management and it maintains an audit history of changes made to the components and configuration of various JDE Server software.
Batch Server
Batch processes (UBE) are background processes requiring no operator intervention or interactivity. One of most important batch process in JDE is MRP Process. Batch Process can be scheduled using a Process Scheduler which runs in the Batch Server. JDE customers running a high volume of reports often split the load on their Enterprise server such that they have one or more Batch servers which handle the high volume reporting (UBE) loads, thereby freeing up their enterprise server to handle interactive user loads more efficiently. This leads to better interactive application and UBE performance due to the expanded scaling provided by the additional hardware.
Design Considerations for Oracle JD Edwards EnterpriseOne Implementation on Cisco UCS with Oracle VM
The design document provides best practices for designing Oracle JD Edwards EnterpriseOne virtualized environment with Cisco UCS, EMC VNX with Oracle VM. There are several factors which need to be considered vis-à-vis JD Edwards HTML Server, JDE E1 application server for interactive and batch (UBE) processes and most importantly the scalability, ease of deployment and maintenance of hardware installed for JD Edwards deployment.
Scalable Architecture Using Cisco UCS Servers
An obvious immediate benefit with Cisco is a single trusted vendor providing all the components needed for a JD Edwards EnterpriseOne deployment with the ability to provide scalable platform, dynamic provisioning, failover with minimal downtimes and reliability.
Some of the capabilities offered by Cisco UCS, which complement the scalable architecture include the following:
•
•
–
–
–
Unified Fabric for Oracle JD Edwards deployment helps to use same fabric path for both Ethernet and FC traffic. This provides multiple storage options using either block or file level storage. 10 Gigabit loseless Ethernet connectivity provides high throughput between Oracle Virtual Servers, thus helping to reduce long running UBE execution time such as MRP.
To accurately design JD Edwards EnterpriseOne on any hardware configuration, we need to understand the characteristics of each tier in JDE deployment vis-à-vis CPU, memory and I/O operations. For instance JDE Enterprise Server for Interactive is both CPU and memory intensive, but is low on disk utilization, whereas, the database server is more memory and disk intensive rather than CPU utilization. Some of the important characteristics to design JD Edwards EnterpriseOne on Cisco UCS Server are listed in Table 1:
Boot from SAN
Boot from SAN is a critical feature which helps to maximize the benefits of Cisco UCS stateless computing, which does not require a static binding between a physical server and the OS/ hypervisor hosted on that server. The hypervisor is installed on a SAN LUN and is booted using the service profile. When the service profile is moved to another server, the server policy and the PWWN of the HBAs also moves along with the service profile. Thereby, the new server takes the identity of the old server and looks identical to the old server.
The following are the benefits of boot from SAN:
•
–
–
–
–
•
•
•
•
•
With boot from SAN, the hypervisor image resides on the SAN and the server communicates with the SAN through a Host Bus Adapter (HBA). The HBA BIOS contain instructions that enable the server to find the boot disk. After Power On-Self Test (POST), the server hardware component fetches the designated boot device in the hardware BOIS settings. Once the hardware detects the boot device, it follows the regular boot process.
Sizing Guidelines for Oracle JD Edwards EnterpriseOne
Sizing ERP deployments on a virtualized environment is a complex, and setting it up correctly depends on the input provided by the customers on how they intend to use the ERP system, and what their priorities are in terms of end user as well as corporate expectations.
Some of the common questions related to ERP sizing are: the number of concurrent interactive users using the system, total number of ERP end users, the kind of applications that the end users will access as well as number of reports and type of reports generated during peak activity can help size the system for optimal performance. Analyzing the demands and its characteristics, the JDE system is expected to handle during different time periods in the fiscal year is necessary to do a proper sizing.
The Oracle JD Edwards EnterpriseOne configuration used in the present deployment is geared to handle a very high workload of end users running heavy SRM interactive applications as well as a high number of batch processes. Each of the HTML Server, Enterprise Server and Database nodes are load balanced with an active-active setup. Each Oracle Virtual Server is hosting an instance of HTML Server, Enterprise Server and Oracle RAC node, thus providing failover during maintenance or failure of Oracle Virtual Server nodes.
The following sections briefly describe the sizing aspects of each tier of the three tier Oracle JD Edwards EnterpriseOne deployment architecture.
Oracle JDE HTML Server
The Oracle JDE HTML server, serves end user interactive application requests from JDE users. The JDE HTML server loads the application forms and requests services from the JDE Enterprise server for application processing based on form input. Some very lightweight application logic also runs on the JDE HTML Server. Client requests do result in significant load on the JDE HTML servers since the JDE HTML servers make and manage database as well as network connections. The JDE HTML server's utilization of CPU and memory depends heavily on the number of interactive users using the server. Disk utilization is not a major factor in the sizing of the JDE HTML Server.
Typically, on the Oracle Linux Server, the number of interactive users per JVM when deployed on an Oracle VM should be capped to around 150 to 200 interactive users for optimal performance. This allows minimizing the heap size for each of the JVM thus reduces Full GC pause time. It is recommended to having multiple JVMs load balanced through Oracle HTTP Server, thus allowing reduced GC pause time and high availability of application. Due to high number of JVMs, typically an HTML server is configured with medium memory and medium computing power.
Oracle JDE Enterprise Server
The Oracle JDE Enterprise Server acts as the central point for serving requests for application logic. The Oracle JDE EnterpriseOne clients make requests for application processing. Depending upon the JDE environment used as well as user preferences, the input data is then processed and returned back to the client. The Call Object kernels running on the JDE Enterprise server are delegated the responsibility of processing end user application processing requests and the Security kernel handles the responsibility of ensuring authentication of the end users. The application processing is CPU intensive and the CPU frequency and number of cores available to the Enterprise server plays a large part affecting the performance and throughput of the system. As the number of interactive users requests grow, the memory requirements of the JDE Enterprise server also increases. This is also true for the batch (UBE) reports that the JDE Enterprise server processes.
The typical sizing recommendation for number of users per call object kernel on Oracle VM would be between 8 - 12 users/call object kernels and about 1 security kernel for every 50 interactive users. The in memory cache usage of call object kernels increase with user load so it is typical for the memory usage of individual call objects to increase with increase in user loads.
Oracle JDE Database Server
The JDE Database Server services the data requests made by both the JDE Enterprise and JDE HTML Servers. The JDE Database Server sizing depends on the type of reports being processed as well as the interactive user loads. Some JDE reports can be very Disk I/O intensive and depending on the kind of reports being processed, careful consideration needs to be given to disk layout. If the Oracle Database Server database has ample memory available to it and the memory is utilized to cache data it can benefit application performance by reducing disk I/O operations. The JDE Database server typically benefits from having faster disk and high memory allocation. The choice of Oracle RAC database for Oracle JD Edwards EnterpriseOne environment helps to provide high availability, increased transaction response time for concurrent UBE processes and enhanced scalability for JD Edwards EnterpriseOne online users. Moreover, the DATA and LOG disk must be configured as a physical disk to Guest OS configured for Oracle RAC nodes. In the present deployment, we have used virtual machines with 10 vCPU and 124 GB of physical memory for each of the Oracle RAC nodes.
Oracle JD Edwards EnterpriseOne Deployment Architecture
The deployment architecture of Oracle JD Edwards EnterpriseOne on is shown in Figure 13.
Figure 13 Deployment Architecture of JD Edwards EnterpriseOne on Oracle VM
The configuration for each of the Oracle JD Edward's Guest OS deployed on Oracle VM is shown in Figure 14.
Figure 14 Guest OS Configuration of Oracle JD Edwards EnterpriseOne on Oracle VM
The configuration presented in this document is based on the following main configuration components.
Infrastructure Setup
This section elaborates the infrastructure setup details used to deploy Oracle JD Edwards EnterpriseOne. The high-level workflow to configure the system is detailed in Figure 15.
Figure 15 Workflow to Setup Infrastructure for Oracle JD Edwards EnterpriseOne
Cisco Nexus Switch Configuration
This section elaborates on the procedure to configure Cisco Nexus 5548UP Switches. Some of the important configurations detailed in the section are:
•
•
•
•
•
First Time Setup of Cisco Nexus 5548 UP Switch
The NX-OS setup should automatically start on initial boot and connection to the serial or console port of the switch. Enter the following commands to configure the Cisco Nexus 5548UP Switch:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
Note
22.
Note
Note
Follow similar to configure the Cisco Nexus 5548 Switch B (Nexus5k-B).
Enabling Features and Global Configuration
To enable appropriate Nexus 5548 switch licensing, follow these steps on Cisco Nexus 5548 A - (Nexus5k-A), and Cisco Nexus 5548 B - (Nexus5k-B) separately:
1.
2.
3.
4.
5.
6.
Note
Verification: The figure given below, lists the enabled features on Cisco Nexus 5548 (show feature | include enabled").
Figure 16 Features Enabled in Cisco Nexus 5548
Creating VSAN and Adding FC Interfaces
To create VSAN and add FC interfaces, follow these steps on Cisco Nexus 5548 A - (Nexus5k-A), and Cisco Nexus 5548 B - (Nexus5k -B) separately:
1.
2.
3.
4.
5.
6.
Verification: The command sh vsan membership should lists ports fc1/29-32 under "vsan 20".
To configure the ports 29-32 as FC ports, follow these steps on Cisco Nexus 5548 A - (Nexus5k-A), and Cisco Nexus 5548 B - (Nexus5k-B) separately:
1.
2.
3.
4.
5.
Follow the steps in the previous section to configure the Cisco Nexus Switch B.
Verification: The command "show interface brief" should list these interfaces as FC (Admin Mode "F").
VLAN Configuration
A VLAN is a group of end stations in a switched network that is logically segmented by function, project team, or application, irrespective of the physical locations of the users. VLANs have the same attributes as physical LANs, but the end stations can be grouped even if they are not physically located on the same LAN segment.
In this design the VLANs configured are listed in Table 3.
To configure the VLANs, follow these steps:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
Verification: The command show vlan should list the vlans and interfaces assigned to it. Or, the command show run interface <interface name> should show the configuration for a given interface or port channel. The figure given below, lists the executed command:
Nexus5k-A# sh vlanVLAN Name Status Ports---- -------------------------------- --------- -------------------------------1 default active Po1, Eth1/7, Eth1/8, Eth1/9Eth1/10, Eth1/11, Eth1/14Eth1/15, Eth1/16, Eth1/17Eth1/18, Eth1/19, Eth1/20Eth1/21, Eth1/22, Eth1/23Eth1/24, Eth1/25, Eth1/26Eth1/31, Eth1/32191 Private active Eth1/17, Eth1/19613 VLAN0613 active Eth1/17, Eth1/19, Eth1/32760 Public active Eth1/17, Eth1/19, Eth1/32
Note
Virtual Port Channel (vPC) Configuration
A virtual port channel (vPC) allows links that are physically connected to two different Cisco Nexus 5000 Series devices to appear as a single port channel by a third device.
Cisco Nexus 5548UP vPC configurations with the vPC domains and corresponding vPC names and IDs for the present setup is listed in Table 4 and Table 5. To provide Layer 2 and Layer 3 switching, a pair of Cisco Nexus 5548UP Switches with upstream switching are deployed, providing high availability in the event of failure to Cisco UCS to handle management, application, and Network storage data traffic. In the Cisco Nexus 5548UP switch topology, a single vPC feature is enabled to provide high availability, faster convergence in the event of a failure, and greater throughput.
Note
In the vPC design table, a single vPC domain, Domain 100, is created across Cisco Nexus 5548UP member switches to define vPCs to carry specific network traffic. This topology defines two vPCs. These vPCs are managed within the Cisco Nexus 5548UP, which connects Cisco UCS Fabric Interconnects and the EMC storage system.
To create vPC, follow these steps for both Cisco Nexus 5548 A and Cisco Nexus 5548 B separately with configuration details as per Table 4 and Table 5.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
Note
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
Verification: When configuring the Cisco Nexus 5548UP with vPCs, make sure that the status for all vPCs is "Up" for connected Ethernet ports by running the commands (show vpc) from the CLI on the Cisco Nexus 5548UP Switch.
Figure 17 vPC Status on Cisco Nexus 5548UP A and Nexus 5548UP B
For more information on vPC configuration for Cisco Nexus 5548UP switches, see:
Setting Up Jumbo Frames on Cisco Nexus 5500
Jumbo frames with an mtu size 9000 have to be setup on Cisco Nexus 5500. Oracle Interconnect traffic under normal conditions does not go to the northbound switch like the Cisco Nexus 5500, as all the private vNICs are configured in Fabric B. However if there is a partial link or IOM failure, the private interconnect traffic has to go to the immediate northbound switch (Nexus 5500 in our case) to reach Fabric B.
Follow these commands to configure Jumbo Frames on Cisco Nexus 5500 Fabric A Switch:
Nexus5k-A# conf terminalEnter configuration commands, one per line. End with CNTL/Z.Nexus5k-A(config)# class-map type network-qos class-platinumNexus5k-A(config-cmap-nq)# exitNexus5k-A(config)# policy-map type network-qos jumboNexus5k-A(config-pmap-nq)# class type network-qos class-defaultNexus5k-A(config-pmap-nq-c)# mtu 9216Nexus5k-A(config-pmap-nq-c)# multicast-optimizeNexus5k-A(config-pmap-nq-c)# exitNexus5k-A(config-pmap-nq)# system qosNexus5k-A(config-sys-qos)# service-policy type network-qos jumboNexus5k-A(config-sys-qos)# exitNexus5k-A(config)# copy running-config startup-config[########################################] 100%Nexus5k-A(config)#Similarly, enable Jumbo Frames on Nexus 5500 Fabric B Switch.
Cisco Unified Computing System Configuration
This section details the Cisco UCS configuration that is performed as part of the infrastructure build for deployment of Oracle JD Edwards EnterpriseOne. The racking, power, and installation steps can be referred from the Installation Guide, as it is beyond the scope of this document. For more details on each step refer:
•
http://www.cisco.com/en/US/docs/unified_computing/ucs/hw/chassis/install/install.html
•
•
Validate Installed Firmware
To login to Cisco UCS Manager, follow these steps:
1.
2.
3.
4.
5.
6.
7.
For more information on Firmware Management, see: http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/firmware-mgmt/gui/2.1/b_GUI_Firmware_Management_21_chapter_0100.html
Verification: The Installed Firmware should be displayed as 2.1(1a) as shown in Figure 18.
Figure 18 Verifying the Installed Firmware Version
Chassis Discovery Policy
To edit the chassis discovery policy, follow these steps:
1.
2.
3.
4.
Verification: The chassis discovery policy configured to 8-link is displayed in Figure 19.
Figure 19 Displaying the Chassis Discovery Policy Configuration
Enabling Network Components
To enable Fibre Channel, servers, and uplink ports, follow these steps:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
Verification: Check if all configured links show their status as "up" as shown in Figure 20 for Fabric Interconnect A. This can also be verified on the Cisco Nexus switch side by running "show int status" and all the ports connected to the Cisco UCS Fabric Interconnects are shown as "up".
Note
Figure 20 Enabling the Server Ports on Cisco UCS Fabric Interconnect
Creating MAC Address Pools
To create MAC Address pools, follow these steps:
1.
2.
Note
3.
4.
5.
6.
7.
8.
Note
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
Note
19.
20.
21.
22.
Verification: Select LAN tab > Pools > root. Select MAC Pools and it expands to show the MAC pools created. On the right pane, details of the MAC pools are displayed as shown in Figure 21.
Figure 21 MAC Pool Details
Creating WWPN Pools
To create WWPN pools, follow these steps:
1.
2.
Note
3.
4.
5.
6.
7.
8.
Note
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
Verification: As displayed in Figure 22, the new name with the 24 block size.
Figure 22 WWPN Pool Details
Creating WWNN Pools
To create WWNN pools, follow these steps:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
Verification: The new name with the 24 block size displays in the right panel when WWNN pools is selected on the left panel, as shown in Figure 23. (WWNN Pool Size "24" is just for demonstration and not a hard limit for WWNN Pool).
Figure 23 WWNN Pool Details
Creating UUID Suffix Pools
To create UUID suffix pools, follow these steps:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
Verification: Ensure that the UUID suffix pools created are displayed as shown in Figure 24.
Figure 24 UUID Suffix Pool Details
Creating VLANs
To create VLANs, follow these steps:
1.
Note
2.
3.
4.
5.
6.
Verification: Select LAN tab > LAN Cloud > VLANs. Open VLANs and all of the created VLANs are displayed. The right pane gives the details of all individual VLANs as shown in Figure 25.
Figure 25 Details of Created VLANs
Creating Uplink Ports Channels
To create uplink port channels to Cisco Nexus 5548 switches, follow these steps:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
Verification: Select LAN tab > LAN Cloud. On the Right Pane, select the LAN Uplinks and expand the Port channels listed as shown in Figure 26.
Note
Figure 26 Details of Port Channels
Creating VSANs
All the servers under JD Edwards EnterpriseOne deployment on UCS with EMC VNX 5300 are booted from SAN. The following section elaborates on VSAN creation which we configured under Cisco Nexus Switch Configuration configuration
To create VSANs, follow these steps:
1.
2.
3.
4.
5.
6.
7.
Verification: Select SAN tab >SAN Cloud >VSANs on the left panel. The right panel displays the created VSANs as shown in Figure 27.
Figure 27 Details of VSANs
SAN Boot Policy Configuration
In the present deployment, the UCS Servers are booted from SAN (EMC VNX5300). With boot from SAN, the OS image resides on the SAN and the server communicates with the SAN through a host bus adapter (HBA). The HBAs BIOS contain the instructions that enable the server to find the boot disk. After power on self test (POST), the server hardware component fetches the boot device that is designated as the boot device in the hardware BOIS settings. When the hardware detects the boot device, it follows the regular boot process.
Modifying Service Profile for Boot Policy
In this setup, vhba0 and vhba1 are used both for SAN Boot and JD Edwards EnterpriseOne Installation. Storage SAN WWPN ports will be connected in the boot policy as shown:
To modify the Service Profile for boot policy, follow these steps:
1.
2.
3.
4.
5.
6.
Note
7.
8.
9.
10.
Figure 28 Boot Policy Service Profile
Note
BIOS Policy
BIOS policy is one of the features of Cisco UCS Service Profiles which enable users to incorporate similar BIOS settings all across the deployed servers. It helps to create application specific BIOS setting, For instance in a virtualized environment we would enable VT for direct IO wherein in the non-virtualized environment we would leave that as default. This helps a consistent configuration and the Administrators do not need to interrupt the boot process on each server to alter the BIOS setting. This helps faster deployment as users do not have to use the F2 option to change BIOS of hundreds of servers deployed in a Data Center. The BIOS policy configured for JDE deployment is shown in Figure 29.
Figure 29 BIOS Setting for CPU Performance
Figure 30 BIOS setting for Physical Memory Performance
Note
UCS Service Profile Configuration
An important aspect of configuring a physical server in a Cisco UCS 5108 chassis is to develop a service profile through Cisco UCS Manager. Service profile is an extension of the virtual machine abstraction applied to physical servers. The definition has been expanded to include elements of the environment that span the entire data center, encapsulating the server identity (LAN and SAN addressing, I/O configurations, firmware versions, boot order, network VLAN, physical port, and quality-of-service [QoS] policies) in logical "service profiles" that can be dynamically created and associated with any physical server in the system within minutes rather than hours or days. The association of service profiles with physical servers is performed as a simple, single operation. It enables migration of identities between servers in the environment without requiring any physical configuration changes and facilitates rapid bare metal provisioning of replacements for failed servers.
Service profiles can be created in several ways:
•
•
•
In the present scenario we created a Service profile updating template and thereafter instantiated service profile through the template. A service profile template parameterizes the UIDs that differentiate one instance of an otherwise identical server from another. Templates can be categorized into two types:
•
•
Service profiles, templates, and other management data is stored in high-speed persistent storage on the Cisco UCS Fabric Interconnects, with mirroring between fault-tolerant pairs of Fabric Interconnects.
Create vNIC Templates
To configure the vNIC template, follow these steps:
1.
2.
3.
4.
5.
6.
7.
8.
Figure 31 vNIC Template for Public Network
Figure 32 vNIC Template for Private Network
Figure 33 vNIC Template for Management Network
Create vHBA Templates
To create a HBA template, follow these steps:
1.
2.
3.
4.
5.
6.
Figure 34 vHBA Template
Creating Service Profile Templates
To create service profile templates in UCSM, follow these steps:
1.
2.
3.
4.
a.
b.
c.
d.
5.
a.
b.
c.
d.
e.
f.
g.
6.
a.
b.
c.
d.
e.
f.
g.
h.
i.
j.
7.
a.
8.
a.
b.
c.
9.
a.
10.
11.
A server pool contains a set of servers. These servers typically share the same characteristics. Those characteristics can be their location in the chassis, or an attribute such as server type, amount of memory, local storage, type of CPU, or local drive configuration. You can manually assign a server to a server pool, or use server pool policies and server pool policy qualifications to automate the assignment.
If your system implements multi-tenancy through organizations, you can designate one or more server pools to be used by a specific organization. For example, a pool that includes all servers with two CPUs could be assigned to the Marketing organization, while all servers with 64 GB memory could be assigned to the Finance organization.
For Oracle JD Edwards deployment, this feature can help in assigning servers to a Server Pool with specific CPU and memory for batch workload or for interactive workload. With the help of UCS XML API we can easily integrate Oracle Enterprise Manager (OEM) and provide automatic failover during the failure of server assigned to specific Server Pool. For instance, in the event of a batch server failure assigned to a Batch server pool, OEM can initiate a call to UCS XML scripts and a server from Spare Server Pool can move to a Batch Server Pool and can be automatically assigned to the Service Profile of Failed Batch Server.
12.
13.
Figure 35 Service Profile Template
Creating Service Profile from the Template and Associating it to Cisco UCS Blade Server
To create a service profile from the template and associating it to a blade, follow these steps:
1.
2.
3.
4.
5.
6.
Figure 36 Successful Association of the Service Profile
Creation of Zoneset and Zones on Cisco Nexus 5548UP Switch
Once the server association to a Service Profile is done, we can see the published WWPNs of vhab0 and vhba1 on Cisco Nexus 5548 A, and 5548 B respectively for that specific server. This section details the steps to configure zoneset and zones on each of the Nexus 5548 switch for server Oracle VM Similar the same steps need to be followed for second server on the Oracle JD Edwards deployment with Oracle VM on Cisco Nexus 5548 A, and 5548 B.
Follow these steps to configure zoneset and zones on Nexus 5548UP Switches:
1.
Nexus5k-A# conf tEnter configuration commands, one per line. End with CNTL/Z.Nexus5k-A(config)# sh flogi database--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------fc2/1 20 0x7801ef 50:06:01:60:3e:a0:05:47 50:06:01:60:be:a0:05:47fc2/2 20 0x7800ef 50:06:01:68:3e:a0:05:47 50:06:01:60:be:a0:05:47fc2/3 20 0x780002 20:4d:54:7f:ee:56:ca:00 20:14:54:7f:ee:56:ca:01fc2/3 20 0x780005 20:00:00:25:b5:de:0a:0e 20:00:00:25:b5:de:de:0efc2/4 20 0x780000 20:4e:54:7f:ee:56:ca:00 20:14:54:7f:ee:56:ca:01fc2/5 16 0x6d0000 50:0a:09:83:9d:93:40:7f 50:0a:09:80:8d:93:40:7ffc2/6 16 0x6d0001 50:0a:09:84:8d:93:40:7f 50:0a:09:80:8d:93:40:7ffc2/7 16 0x6d0002 20:4f:54:7f:ee:56:ca:00 20:10:54:7f:ee:56:ca:01fc2/7 16 0x6d0003 20:00:00:25:b5:de:aa:0e 20:de:00:25:b5:00:00:0efc2/7 16 0x6d0005 20:00:00:25:b5:de:aa:0a 20:de:00:25:b5:00:00:0afc2/8 16 0x6d0004 20:00:00:25:b5:de:aa:0c 20:de:00:25:b5:00:00:0cfc2/8 16 0x6d0006 20:00:00:25:b5:de:aa:09 20:de:00:25:b5:00:00:09fc2/8 16 0x6d000a 20:50:54:7f:ee:56:ca:00 20:10:54:7f:ee:56:ca:01fc2/8 16 0x6d000b 20:00:00:25:b5:de:aa:0f 20:de:00:25:b5:00:00:0fTotal number of flogi = 14.
Note
2.
Nexus5k-A(config)# zone name jde-ovm1-vhba0 vsan 20Nexus5k-A(config-zone)# member pwwn 50:06:01:60:3e:a0:05:47Nexus5k-A(config-zone)# member pwwn 50:06:01:68:3e:a0:05:47Nexus5k-A(config-zone)# member pwwn 20:00:00:25:b5:de:0a:0eNexus5k-A(config-zone)# zoneset name jde-ovm-n5kA vsan 20Nexus5k-A(config-zoneset)# member jde-ovm1-vhba0Nexus5k-A(config-zoneset)# zoneset activate name jde-ovm-n5kA vsan 20WARNING: You are trying to activate zoneset jde-ovm-n5kA, which is different from current active zoneset jdeovm-n5ka. Do you want to continue? (y/n) [n] yZoneset activation initiated. check zone statusNexus5k-A(config)# sh zoneset active vsan 20zoneset name jde-ovm-n5kA vsan 20zone name jde-ovm1-vhba0 vsan 20* fcid 0x7801ef [pwwn 50:06:01:60:3e:a0:05:47]* fcid 0x7800ef [pwwn 50:06:01:68:3e:a0:05:47]* fcid 0x780005 [pwwn 20:00:00:25:b5:de:0a:0e]Nexus5k-A(config)# copy r s[########################################] 100%Copy complete, now saving to disk (please wait)Similar configuration has to be done on Nexus 5548 B for vhba1 of JDE-OVM-1 Service Profile. Once we configure zones and zoneset on Nexus 5548 B we would see following configuration.
Nexus5k-B(config)# sh zoneset active vsan 20zoneset name jde-ovm1-n5kB vsan 20zone name jde-ovm1-vhba1 vsan 20* fcid 0xdf01ef [pwwn 50:06:01:61:3e:a0:05:47]* fcid 0xdf00ef [pwwn 50:06:01:69:3e:a0:05:47]* fcid 0xdf0005 [pwwn 20:00:00:25:b5:de:b0:0e]Configuring Connectivity for EMC VNX Storage
The present JD Edwards deployment leverages EMC VNX5300 storage, connected to the Cisco UCS system. It leverages the EMC FAST Cache as well as EMC FAST Virtual Pool capability.
As Edwards Enterprise One Server has very frequent and random access to the database; the Oracle RAC data files would require very fast read/write operations. For such access patterns a storage pool with Flash disks also known as Solid State Disks (SSD) are chosen with RAID level 5. SSD disks provide faster response time for random access IO patterns. The Data LUN would be configured as Physical disk to the Oracle RAC Guest VMs Oracle RAC log files) involves sequential write intensive operations, hence SAS drives with RAID level 10 is chosen. The log LUN would be configured as physical disk to the Oracle RAC Guest VMs Oracle RAC ASM luns is configured with RAID level 10 and attached as physical disk to Oracle RAC guest VMs. For boot LUNs, RAID level 5 is chosen for reliability. All the boot luns would be configured as Virtual Disk to each of the Guest VMs. The installation directory for HTML, JDE E1 and Database resides on RAID5, using multiple SAS drives. For Backing up LUNs, NL-SAS drives are chosen for their high capacity and lower cost characteristics that align with the backup model.
Figure 37 illustrates the disk layout configured for Oracle JD Edwards on Cisco UCS with Oracle VM.
Figure 37 Disk layout
LUNs carved from the EMC VNX storage forming the disk layout for the Oracle JD Edwards EnterpriseOne deployment are shown in Table 6.
Note
Creation of Storage Pools/ RAID Groups
To create storage pools/RAID groups, follow these steps:
1.
2.
a.
b.
c.
d.
3.
4.
5.
6.
7.
a.
b.
Note
Figure 38 Host ID on Storage Group
Since the zones are configured in the Cisco Nexus switches with the Host HBA WWPNs, they will appear in the EMC VNX5300 - "Host connectivity status".
To establish the Host connectivity at the EMC VNX5300 array, follow these steps:
1.
2.
3.
4.
5.
6.
To associate the LUN for this blade, associate LUN to the already created Storage group. For the JD Edwards HTML Server, the storage group created is "html1-install-lun". To do this, click Hosts > Storage Groups (html1-install-lun) > Connect Hosts; this will open Storage group properties pop-up box > Hosts tab. Locate the "JDEOVM1" from the available hosts section and Click OK. Associating host to the Storage Groups (JDEOVM1) is shown in Figure 39.
Figure 39 Storage Group - Host Association
Oracle Virtual Server (OVS) Configuration
It is not within the scope of this document to include the specifics of Oracle Virtual Server (OVS) Installation. You should refer to the Oracle installation documentation for specific installation instructions for your Environment.
Oracle Virtual Server Installation
Some of the important steps during Oracle Virtual Server installation are:
1.
2.
Figure 40 Single FC Path for OVS Installation
3.
Note
Figure 41 OVS ISO Attached as Virtual Media to KVM Console
4.
Figure 42 Single FC Path for OS Installation
5.
Figure 43 EMC Boot LUN
6.
Figure 44 Boot from Multipath Device
7.
Figure 45 vNIC Mgmt Configured for Management IP
8.
Figure 46 OVS Installation Status
Similar steps need to be executed to install the second Oracle Virtual Server.
Some of the important steps executed post OVS installation are:
1.
Figure 47 Reconfigure Boot Policy
2.
[root@jde-ovm1 ~]# modinfo fnicfilename: /lib/modules/2.6.39-200.37.1.el5uek/kernel/drivers/scsi/fnic/fnic.koversion: 1.5.0.1license: GPL v2author: Abhijeet Joglekar <abjoglek@cisco.com>, Joseph R. Eykholt <jeykholt@cisco.com>description: Cisco FCoE HBA Driversrcversion: 60031A0CDAEE010E0A0B25Aalias: pci:v00001137d00000045sv*sd*bc*sc*i*depends: libfcoe,libfc,scsi_transport_fcvermagic: 2.6.39-200.37.1.el5uek SMP mod_unload modversionsparm: fnic_log_level:bit mask of fnic logging levels (int)[root@jde-ovm1 ~]# mkdir /test[root@jde-ovm1 ~]# mount -o loop,ro dd-fnic-1.5.0.18-oracle-ovs-3.1.1.iso /test[root@jde-ovm1 ~]# cd /test[root@jde-ovm1 test]# rpm -ivh --nodeps /test/rpms/x86_64/kmod-fnic-1.5.0.18-el6uek_ovm_3.1.1.x86_64.rpmPreparing... ########################################### [100%]1:kmod-fnic ########################################### [100%]WARNING: The initrd image will contain multipath boot from san[root@jde-ovm1 2.6.39-200.37.1.el5uek]# uname -aLinux jde-ovm1 2.6.39-200.37.1.el5uek #1 SMP Fri Feb 8 03:53:53 PST 2013 x86_64 x86_64 x86_64 GNU/Linux[root@jde-ovm1 /]# cd lib/modules/2.6.39-200.37.1.el5uek/kernel/drivers/scsi/fnic/[root@jde-ovm1 fnic]# cp /lib/modules/2.6.39-200.1.1.el5uek/extra/fnic/fnic.ko .cp: overwrite `./fnic.ko'? y[root@jde-ovm1 fnic]# modprobe fnic[root@jde-ovm1 fnic]# modinfo fnicfilename: /lib/modules/2.6.39-200.37.1.el5uek/kernel/drivers/scsi/fnic/fnic.koversion: 1.5.0.18license: GPL v2author: Abhijeet Joglekar <abjoglek@cisco.com>, Joseph R. Eykholt <jeykholt@cisco.com>description: Cisco FCoE HBA Driversrcversion: 24F8E443F0EEDBDF4802F20alias: pci:v00001137d00000045sv*sd*bc*sc*i*depends: libfc,libfcoe,scsi_transport_fcvermagic: 2.6.39-200.1.1.el5uek SMP mod_unload modversionsparm: fnic_log_level:bit mask of fnic logging levels (int)parm: fnic_trace_max_pages:Total allocated memory pages for fnic trace buffer (uint)3.
device {vendor "EMC"product "SYMMETRIX"getuid_callout "/sbin/scsi_id -g -u -ppre-spc3-83 -s /block/%n"features "0"hardware_handler "0"path_grouping_policy multibusrr_weight uniformrr_min_io 8path_checker tur}4.
[root@jde-ovm1 ~]# multipath -l3600601604c3029009a6d5f53d7d4e211 dm-0 DGC,VRAIDsize=40G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw`-+- policy='round-robin 0' prio=1 status=active|- 0:0:0:0 sda 8:0 active ready running|- 0:0:1:0 sdo 8:224 active ready running|- 1:0:0:0 sdac 65:192 active ready running`- 1:0:1:0 sdaq 66:160 active ready running3600601604c302900cc4b23b27bd5e211 dm-1 DGC,VRAIDsize=550G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw`-+- policy='round-robin 0' prio=1 status=active|- 0:0:0:1 sdb 8:16 active ready running|- 0:0:1:1 sdp 8:240 active ready running|- 1:0:0:1 sdad 65:208 active ready running`- 1:0:1:1 sdar 66:176 active ready running5.
# ls /sys/class/fc_hosthost0 host1# echo "1" > /sys/class/fc_host/host0/issue_lip# echo "- - -" > /sys/class/scsi_host/host0/scan# echo "1" > /sys/class/fc_host/host1/issue_lip# echo "- - -" > /sys/class/scsi_host/host1/scan6.
OVM Manager Installation
Oracle VM Manager is installed as a production level; this is the preferred installation type, with options for selecting Oracle SE or EE database as the location for the Oracle VM Manager repository. Prior to installing Oracle VM Manager, ensure that Oracle SE Database is installed. Please follow the steps below to successfully install OVM Manager.
[root@ovmmanager ovmmanager]# ./runInstaller.shOracle VM Manager Release 3.1.1 InstallerOracle VM Manager Installer log file:/tmp/install-2013-06-10-164951.logPlease select an installation type:1: Demo2: Production3: Uninstall4: HelpSelect Number (1-4): 2Starting production installation ...Verifying installation prerequisites ...Oracle Database Repository==========================Use an existing Oracle databaseEnter the Oracle Database hostname [localhost]: ovmmanagerEnter the Oracle Database System ID (SID) [XE]: orclEnter the Oracle Database SYSTEM password:Enter the Oracle Database listener port [1521]: 1521Enter the Oracle VM Manager database schema [ovs]: ovs1Enter the Oracle VM Manager database schema password:Invalid password.Passwords need to be between 8 and 16 characters in length.Passwords must contain at least 1 lower case and 1 upper case letter.Passwords must contain at least 1 numeric value.Enter the Oracle VM Manager database schema password:Enter the Oracle VM Manager database schema password (confirm):Oracle Weblogic Server 11g==========================Enter the Oracle WebLogic Server 11g user [weblogic]:Enter the Oracle WebLogic Server 11g user password:Enter the Oracle WebLogic Server 11g user password (confirm):Passwords do not matchEnter the Oracle WebLogic Server 11g user password:Enter the Oracle WebLogic Server 11g user password (confirm):Oracle VM Manager application=============================Enter the username for the Oracle VM Manager administration user [admin]:Enter the admin user password:Enter the admin user password (confirm):Verifying configuration ...Start installing the configured components:1: Continue2: AbortSelect Number (1-2): 1Step 1 of 9 : Database ...Installing Database ...Database installation skipped ...Step 2 of 9 : Java ...Installing Java ...Step 3 of 9 : Database Schema ...Creating database schema 'ovs1' ...Step 4 of 9 : WebLogic ...Retrieving Oracle WebLogic Server 11g ...Installing Oracle WebLogic Server 11g ...Step 5 of 9 : ADF ...Retrieving Oracle Application Development Framework (ADF) ...Unzipping Oracle ADF ...Installing Oracle ADF ...Installing Oracle ADF Patch...Step 6 of 9 : Oracle VM ...Retrieving Oracle VM Manager Application ...Extracting Oracle VM Manager Application ...Installing Oracle VM Manager Core ...Step 7 of 9 : Domain creation ...Creating Oracle WebLogic Server domain ...Starting Oracle WebLogic Server 11g ...Configuring data source 'OVMDS' ...Creating Oracle VM Manager user 'admin' ...Step 8 of 9 : Deploy ...Deploying Oracle VM Manager Core container ...Deploying Oracle VM Manager UI Console ...Deploying Oracle VM Manager Help ...Enabling HTTPS ...Granting ovm-admin role to user 'admin' ...Step 9 of 9 : Oracle VM Manager Shell ...Retrieving Oracle VM Manager Shell & API ...Extracting Oracle VM Manager Shell & API ...Installing Oracle VM Manager Shell & API ...Retrieving Oracle VM Manager Upgrade tool ...Extracting Oracle VM Manager Upgrade tool ...Installing Oracle VM Manager Upgrade tool ...Copying Oracle VM Manager shell to '/usr/bin/ovm_shell.sh' ...Installing ovm_admin.sh in '/u01/app/oracle/ovm-manager-3/bin' ...Installing ovm_upgrade.sh in '/u01/app/oracle/ovm-manager-3/bin' ...Enabling Oracle VM Manager service ...Shutting down Oracle VM Manager instance ...Restarting Oracle VM Manager instance ...Waiting 15 seconds for the application to initialize ...Oracle VM Manager is running ...Oracle VM Manager installed.Please wait while WebLogic configures the applications... This can take up to 5 minutes.Installation Summary--------------------Database configuration:Database host name : ovmmanagerDatabase instance name (SID): orclDatabase listener port : 1521Application Express port : NoneOracle VM Manager schema : ovs1Weblogic Server configuration:Administration username : weblogicOracle VM Manager configuration:Username : adminCore management port : 54321UUID : 0004fb00000100000a5c59c7f7487ffePasswords:There are no default passwords for any users. The passwords to use for Oracle VM Manager, Oracle Database 11g XE, and Oracle WebLogic Server have been set by you during this installation. In the case of a default install, all passwords are the same.Oracle VM Manager UI:http://ovmmanager:7001/ovm/consolehttps://ovmmanager:7002/ovm/consoleLog in with the user 'admin', and the password you set during the installation.Please note that you need to install tightvnc-java on this computer to access a virtual machine's console.For more information about Oracle Virtualization, please visit:http://www.oracle.com/virtualization/Oracle VM Manager installation complete.Please remove configuration file /tmp/ovm_configzFYrq_.Post OVM Manager installation, apply Oracle VM Manager 3.1.1 Patch Update (Build 365) [ID 1530546.1]. This would help in resolution of time out issues on creation of Oracle VM 3.1.1.
Oracle VM Configuration
Some of the important steps to configure Oracle VM environment are shown in Figure 48.
Figure 48 Oracle VM Configuration Steps
7.
Figure 49 Unassigned Servers
8.
Figure 50 Network Configuration
9.
Figure 51 Cluster Pool
10.
Figure 52 Storage Repository
11.
Figure 53 Guest VMs Details
Figure 54 Guest VMs
12.
13.
Figure 55 Disk Configuration for JDE Application Server
Figure 56 Disk Configuration for Oracle RAC Nodes
Once the Guest VMs are created we would proceed to Installation of Oracle Linux 5.8 on each of the VMs.
Oracle Linux Installation
Some of the important steps during Oracle Linux installation are:
1.
Figure 57 http Setup for PVM Installation
2.
Figure 58 Text Mode Selection
3.
Figure 59 Virtual Disk Selection
4.
Figure 60 Confirm OS Installation
Some of the important steps executed post Oracle Linux installation are:
1.
2.
3.
4.
kernel: nf_conntrack: table full, dropping packet.kernel: nf_conntrack: table full, dropping packet.Download package iptables-1.3.5-9.2.0.2.el5_8.x86_64 from URL= https://linux.oracle.com => Home > Channels > Channel: Oracle Linux 5 Update 10 Patch (x86_64)>Packages.
Update the rpm with rpm -Uvh iptables-1.3.5-9.2.0.2.el5_8.x86_64.rpm.
Apply the above patch to the all the Guest VMs with Oracle Linux 5.8 (uek kernel).
Once we complete the post-installation steps we would proceed to Oracle RAC setup on Guest VMs.
Oracle RAC Setup
This section describes the deployment of Oracle Database 11g R2 GRID Infrastructure with RAC option for Oracle JD Edwards EnterpriseOne deployment on Oracle Virtualized environment. After installation of Oracle Linux 5.8 on each RAC node, verify that all the required rpms are installed as part of OS installation which is required for Oracle GRID Installation.
As a best practice recommended from Oracle, ready-to-go Oracle VM Templates for Oracle RAC can be downloaded from Oracle Software Delivery Cloud for faster deployment.
Use the following Oracle document for pre-installation tasks, such as setting up the kernel parameters, RPM packages, user creation, and so on.
(http://download.oracle.com/docs/cd/E11882_01/install.112/e10812/prelinux.htm#BABHJHCJ)
Follow the steps to complete the Oracle Database 11g R2 GRID Infrastructure with RAC option Installation. For this solution the user, group, directory structure, kernel parameters and user limits can be created as shown in the steps below. One can resize the kernel parameter, user limits and rename the directory structure, user name, groups as per the business requirement.
1.
groupadd -g 1000 oinstallgroupadd -g 1200 dbauseradd -u 2000 -g oinstall -G dba gridpasswd griduseradd -u 1100 -g oinstall -G dba oraclepasswd oracle
Note
2.
3.
Disk /dev/xvdb: 429.4 GB, 429496729600 bytes - orcl-data-aDisk /dev/xvdc: 214.7 GB, 214748364800 bytes- orcl-log-aDisk /dev/xvdd: 107.3 GB, 107374182400 bytes - orcl-asmDisk /dev/xvde: 429.4 GB, 429496729600 bytes - orcl-data-bDisk /dev/xvdf: 214.7 GB, 214748364800 bytes - orcl-log-bDisk /dev/xvdg: 107.3 GB - orcl-asm14.
[root@vm-db-node1 ~]# fdisk -lDisk /dev/xvda: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System/dev/xvda1 * 1 13 104391 83 Linux/dev/xvda2 14 39162 314464342+ 8e Linux LVMDisk /dev/dm-0: 216.7 GB, 216795185152 bytes255 heads, 63 sectors/track, 26357 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDisk /dev/dm-0 doesn't contain a valid partition tableDisk /dev/dm-1: 105.1 GB, 105193144320 bytes255 heads, 63 sectors/track, 12789 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDisk /dev/dm-1 doesn't contain a valid partition tableDisk /dev/xvdb: 429.4 GB, 429496729600 bytes255 heads, 63 sectors/track, 52216 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System/dev/xvdb1 1 52216 419424988+ 83 LinuxDisk /dev/xvdc: 214.7 GB, 214748364800 bytes255 heads, 63 sectors/track, 26108 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System/dev/xvdc1 1 26108 209712478+ 83 LinuxDisk /dev/xvdd: 107.3 GB, 107374182400 bytes255 heads, 63 sectors/track, 13054 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System/dev/xvdd1 1 1305 10482381 83 Linux/dev/xvdd2 1306 2710 11285662+ 83 Linux/dev/xvdd3 2711 4015 10482412+ 83 LinuxDisk /dev/xvde: 429.4 GB, 429496729600 bytes255 heads, 63 sectors/track, 52216 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System/dev/xvde1 1 52216 419424988+ 83 LinuxDisk /dev/xvdf: 214.7 GB, 214748364800 bytes255 heads, 63 sectors/track, 26108 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System/dev/xvdf1 1 26108 209712478+ 83 LinuxDisk /dev/xvdg: 107.3 GB, 107374182400 bytes255 heads, 63 sectors/track, 13054 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System/dev/xvdg1 1 1305 10482381 83 Linux/dev/xvdg2 1306 2710 11285662+ 83 Linux
Note
5.
[root@vm-db-node1 /]# /etc/rc.d/init.d/oracleasm configureConfiguring the Oracle ASM library driver.Default user to own the driver interface [grid]:Default group to own the driver interface [grid]: oinstallStart Oracle ASM library driver on boot (y/n) [y]:Scan for Oracle ASM disks on boot (y/n) [y]:Writing Oracle ASM library driver configuration: doneInitializing the Oracle ASMLib driver: [ OK ]Scanning the system for Oracle ASMLib disks: [ OK ]6.
[root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk ocrvote1 /dev/xvdd1Marking disk "ocrvote1" as an ASM disk: [ OK ][root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk ocrvote2 /dev/xvdd2Marking disk "ocrvote2" as an ASM disk: [ OK ][root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk ocrvote3 /dev/xvdd3Marking disk "ocrvote3" as an ASM disk: [ OK ][root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk ocrvote4 /dev/xvdg1Marking disk "ocrvote4" as an ASM disk: [ OK ][root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk ocrvote5 /dev/xvdg2Marking disk "ocrvote5" as an ASM disk: [ OK ]Create data and log disk[root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk data_A /dev/xvdb1Marking disk "data_A" as an ASM disk: [ OK ][root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk data_B /dev/xvde1Marking disk "data_B" as an ASM disk: [ OK ][root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk log_A /dev/xvdc1Marking disk "log_A" as an ASM disk: [ OK ][root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm createdisk log_B /dev/xvdf1Marking disk "log_B" as an ASM disk:7.
[root@vm-db-node1 etc]# /etc/rc.d/init.d/oracleasm listdisksDATA_ADATA_BLOG_ALOG_BOCRVOTE1OCRVOTE2OCRVOTE3OCRVOTE4OCRVOTE58.
[root@vm-db-node2 ~]# /etc/rc.d/init.d/oracleasm scandisksScanning the system for Oracle ASMLib disks: [ OK ][root@vm-db-node2 ~]# /etc/rc.d/init.d/oracleasm listdisksDATA_ADATA_BLOG_ALOG_BOCRVOTE1OCRVOTE2OCRVOTE3OCRVOTE4OCRVOTE5Once ASM disks are configured on the both the Oracle RAC node VMs, we would proceed to Oracle Grid Installation.
Oracle Grid Installation
Some of the important steps to configure Oracle Grid Infrastructure are:
1.
Figure 61 Execute Grid Installer
2.
3.
Figure 62 Select Grid Installation
4.
Figure 63 Scan and Cluster Name
5.
Figure 64 Add Cluster Node Information
6.
Figure 65 SSH Connectivity
7.
Figure 66 Specify Interface Type
8.
Figure 67 Storage Options
9.
Figure 68 ASM Disk Group
Figure 69 Select Disks
10.
Figure 70 Specify Installation Location
11.
Figure 71 Installation Summary
12.
13.
Figure 72 Execute Configuration Script
14.
Figure 73 Installed Product - Progress and Status
Once Oracle RAC Grid infrastructure is installed, proceed to Oracle Database Installation.
For more information on Oracle RAC Grid, see Grid Infrastructure Installation Guide for Linux:
http://www.oracle.com/pls/db112/to_toc?pathname=install.112/e10812/toc.htm
Oracle Database Installation
Some of the important steps to configure Oracle Grid Infrastructure are:
1.
Figure 74 Execute Database Installer
2.
Figure 75 Select Install Database Software
3.
Figure 76 Select Nodes
4.
Figure 77 Configure SSH Connectivity
5.
Figure 78 Installation Summary
6.
Figure 79 Execute Root Scripts
7.
Figure 80 Installation Successful
For complete details on Oracle RAC database installation, see Real Application Clusters Installation Guide for Linux and UNIX for detailed installation instructions (http://www.oracle.com/pls/db112/to_toc?pathname=install.112/e10813/toc.htm)
Oracle RAC Post Installation
After successfully installation Grid Infrastructure and Oracle Database binaries we would proceed to configure of Oracle RAC instance. This involves creation of ASM disk group for Oracle Data & Oracle Logs through asmca and configuration of DB instance through dbca.
Following are some of the important Oracle RAC Post installation steps:
1.
Figure 81 Select Oracle Data path for Creation of DATA Disk Group
2.
Figure 82 Create Disk Group
3.
Figure 83 Create Oracle LOG Disk Group
4.
5.
Figure 84 Oracle RAC Instance on Each VM
Oracle JD Edwards EnterpriseOne Installation
The Installation of Oracle JD Edwards (JDE) 9.0.2 suite on Oracle VM with Oracle Linux 5.8 and Oracle RAC 11.2.0.3 as the RDBMS is described in the sections below:
Pre-Requisites
•
•
•
General Install Requirements
Following are generic pre-requisites before installing the Oracle JDE Enterprise Server.
•
•
•
•
Oracle JD Edwards EnterpriseOne Specific Install Requirements
The below mentioned are some of the requirements to be considered for JD Edwards EnterpriseOne installation
Note
JDE Install Port Numbers
Table 8 shows the port numbers that are used for WLS, JDE, HTTP ports, and so on.
Table 8 Port numbers
WLS
7503-7581
JDE Enterprise Server
9700
HTTP
7777
SQLServer
14501
The steps followed to install Oracle JD Edwards on Cisco UCS are shown in Figure 85:
Figure 85 Oracle JD Edwards EnterpriseOne Installation Workflow
Oracle JDE Deployment Server Install
The Deployment server is used as a repository of JD Edwards installation and upgrade software and data artifacts. This section shows the steps for installing the JD Edwards deployment server. The installation steps shown here forth are specific to the JD Edwards 9.0.2 application suite used in conjunction with JD Edwards tools release 8.98.4.10.
To Install the JDE Deployment Server, follow these steps:
1.
Run the steps below, as referenced from Oracle solution ID: Document 1310036.1
Follow the below mentioned steps to download the installer for the database and the installer for the .NET Framework from Microsoft. In addition, you must follow these directions to download a program from Oracle that runs the two Microsoft installers, passing them parameters that EnterpriseOne needs to run properly.
Note
2.
a.
b.
c.
d.
e.
3.
a.
b.
c.
d.
4.
5.
V24818-01 JD Edwards EnterpriseOne Tools 8.98.4.2 - Microsoft SQL Server 2005 Express SP3 Local Database Installer for Deployment Server and Development Client
Place the SSE 2005 SP3 installer SQLEXPR.exe and the .NET Framework 4 installer dotNetFx40_Full_setup.exe onto your Deployment Server in this directory: <dep_svr_install_dir>\OneWorld Client Install\ThirdParty\SSE
6.
7.
8.
9.
10.
11.
12.
13.
14.
Figure 86 JD Edwards Enterprise One Deployment Server Installation Progress
15.
16.
Figure 87 Installing the Visual Studio C Runtime libraries
Figure 88 Accepting the License Agreement Prior to Install
Tools Upgrade on the Deployment Server
After the deployment server install is completed successfully, a tools upgrade to 8.98.4.10 is performed. To upgrade the tools release, follow these steps:
1.
2.
Figure 89 Installing the JD Edwards Enterprise One Workstation
3.
Figure 90 Selecting Options to Proceed Installation
Figure 91 Selecting the Tools Upgrade Setup Type
Install Planner ESU
To download and install the Install Planner ESU, follow these steps:
1.
Figure 92 Sign in to the JD Edwards EnterpriseOne Client Install Using the JDE User
Figure 93 Selecting the Setup Type
2.
3.
Figure 94 Changing the Directory to Scripts
4.
Figure 95 Running the command SpecialInstrs
5.
Figure 96 Selecting the SQL Server Express
6.
Figure 97 Copying the Control Records
7.
8.
Figure 98 Commenting the AS400 and Oracle Drivers
9.
Figure 99 Incrementing the Connection Timeout
10.
11.
12.
13.
Figure 100 Searching for the Batch Application
14.
Figure 101 Selecting the Batch Application Version
15.
Figure 102 Setting the Processing Options
16.
Figure 103 Defining the Path to the Planner Data Folder
17.
Figure 104 Submitting the Data Selection
18.
Figure 105 Selecting the Report Destination
19.
Figure 106 Completion of the Report download
Enterprise Server Install
In this setup, the JDE Enterprise server and JDE database server are on different machines. The platform pack software pertaining to these two JDE servers has to be installed separately onto these two machines. Download the relevant platform pack software (part id V12449-01 -> V12452-01) from Oracle eDelivery and extract its contents by unzipping the zip files. Please note that all zip files must be extracted in place within a folder as sub-folders disk1, disk2 etc.
To install the enterprise server, follow these steps:
1.
Figure 107 Running the install.sh command
2.
Figure 108 Proceeding the JDE EnterpriseOne UNIX Platform Pack
3.
Figure 109 Defining the Directory
4.
Figure 110 Selecting the Install option
5.
Figure 111 Selecting the Installation type
6.
Note
Figure 112 Selecting the Features to be Installed
7.
Figure 113 Defining the DB Attributes
8.
Figure 114 Defining the concurrent user number
9.
Figure 115 Naming the Database Instance
10.
Figure 116 Summary of the Enterprise Server features
11.
DB Server Install
The PlatformPack binaries used for the Enterprise server are copied onto the DB server.
Figure 117 Launching the JD Edwards EnterpriseOne UNIX Platform Pack
1.
Figure 118 Defining the Directory
2.
Figure 119 Defining the Install Option
3.
Figure 120 Defining the Installation type
4.
Figure 121 Selecting the DB Server features
5.
Figure 122 Defining the DB Options
6.
Figure 123 Defining the Database Instance References
7.
Figure 124 Verifying the Creation of new database
8.
Figure 125 Summary of the Database Server Features
The DB platform pack is installed successfully.
Installation Plan
To Install the Installation Plan, follow these steps:
1.
2.
3.
4.
Figure 126 Defining the Verbose Mode
5.
Figure 127 Defining the Process Mode
6.
Figure 128 Defining the Default Mode Values
7.
Figure 129 Defining the Detail Status
8.
Figure 130 Defining the Process after Completion
9.
Figure 131 Defining the Values for Replication
10.
Figure 132 Defining the Package Push Options
11.
Figure 133 Adding the Custom Installation Plan
12.
Figure 134 Define the Name and Description for the Plan
13.
Figure 135 Adding the Location
14.
Figure 136 Defining the Location Parameters
15.
Figure 137 Defining the JDE Servers Details
16.
Figure 138 Defining the Installation Path and Server Details
17.
Figure 139 Confirming the Data Source
18.
19.
Figure 140 Defining the Data Server on Separate Server
20.
21.
Figure 141 Verify all Data Sources
22.
Figure 142 Defining the Environment Data
23.
Figure 143 Not Defining Other Location
24.
Figure 144 Completion of Plan
25.
Figure 145 Running the Planner Validation Report
26.
Installation Workbench
To install the JDE Workbench, follow these steps:
1.
2.
3.
4.
Figure 146 Defining the Workbench Mode
5.
Figure 147 Search the Plans Available
6.
Figure 148 Selecting the Plan
7.
Figure 149 Completion of the Installation Plan
8.
Figure 150 Verifying the Plan Status
Change Assistant
To download and install the Change assistant, follow these steps:
1.
2.
Figure 151 Downloading the Change Assistant
Baseline ESU Install
To Install the Baseline ESU, follow these steps:
1.
2.
Figure 152 Selecting the Unattended Mode
3.
Figure 153 Downloading the Baseline ESU
4.
Figure 154 Summary of the ESU Downloaded
Note
UL2 Install
The Oracle-JDE DIL kit, which best represents a standard customer workload for JD Edwards applications, needs the JDE application level to be at 9.0 update level 2, so UL2 was applied to the installed base application.
To install the UL2, follow these steps:
1.
2.
Figure 155 Clicking the UL2 Update Installation
3.
Figure 156 Proceeding with the UL2 Installation
4.
5.
Figure 157 Successful Installation of the UL2
6.
Figure 158 Defining the Software Update Parameters
Figure 159 Select the Environment (DV900 was chosen)
Figure 160 Uncheck the Create OMW Project and Package Assembly
Figure 161 Completion of the Software Update
HTML Server Install
Download the Weblogic (WLS) 10.3.6 binaries from Oracle eDelivery, and install the WLS Sever on the HTML server. To install Oracle Weblogic Server 10.3.6, see, http://docs.oracle.com/cd/E21764_01/doc.1111/e14142.pdf.
After the installation of WLS, perform the steps described in this section to create a cluster on WLS and deploy JDE Html server on it. A cluster defines groups of Weblogic Server servers that work together to improve scalability and reliability.
Create a domain. Configure the admin server port to 7501 as shown in above link in Appendix B.
Configuring the Cluster
To configure a cluster, follow these steps:
1.
Figure 162 Weblogic Admin Console Page
2.
Figure 163 Naming the Cluster
3.
Figure 164 Summary of the Clusters Created
4.
Figure 165 Create a new Machine entry
Figure 166 Summary of Machines Added
5.
Figure 167 Defining Properties for the New Server
6.
Figure 168 Configuring the General Features
7.
Figure 169 Defining the Instance Properties
8.
Figure 170 Validating the Configuration Items
9.
Figure 171 Completing the Creation of the Managed Instance
Oracle HTTP Server Installation
Install the Oracle HTTP server on the JDE HTML server.
To install Oracle JRF and Oracle HTTP server, see:
•
•
JDE User Creation
To create the JDE user sign into the DEP900 environment on the deployment server, and follow these steps:
1.
Figure 172 Logging into the Deployment Server
2.
Figure 173 Loading the P980001
3.
Figure 174 Adding the Users
4.
5.
Figure 175 Searching for the User Created
Figure 176 Verifying the User
6.
Figure 177 Loading the P98OWSEC
7.
Figure 178 Verifying the User
8.
Figure 179 Setting the Password Management Options
Figure 180 Modifying the Password Change Frequency
Oneworld Client Install
Use the shared location for client installation binaries on deployment server to install the client.
To Install the Oneworld Client, follow these steps:
1.
Figure 181 Running the OneWorld Install Files
2.
Figure 182 Clicking Workstation Installation Link
3.
Figure 183 Proceeding with the Installation
4.
Figure 184 Selecting the Development Full Package
Full Package Build
To Build the Full package, follow these steps:
1.
2.
Figure 185 Loading the GH9083
3.
Figure 186 Selecting the Package Assembly
4.
Figure 187 Defining the new Package Assembly
5.
Figure 188 Launching the Package Assembly Director
6.
Figure 189 Defining the Package Information
7.
Figure 190 Summary of the Package Features
8.
Figure 191 Selecting the Package Build
9.
Figure 192 Proceeding to Package Building
10.
Figure 193 Defining the Package Build Location
11.
Figure 194 Selecting Enterprise Servers
12.
Figure 195 Server Build Specifications
13.
Figure 196 Adding the Package Sequence
14.
Figure 197 Activating the Package Build
15.
Figure 198 Submitting the Package Build Defined
16.
Figure 199 Selecting the Report Output Destination
Note
Summary
The above sections described the approach taken for installing JD Edwards 9.0.2 on a Oracle Virtualized environment with two-node Oracle RAC database. The benchmarking effort required the use of a weblogic cluster with an Oracle Http server front end to load balance the users among the various cluster nodes. It is very important to check Oracle support documents for information regarding the latest support statements in the various JDE MTRs as well as to check for recently released patches.
Oracle JD Edwards EnterpriseOne Performance and Scalability
Workload Description
The Oracle JD Edwards Day in the life (DIL) kit is an attempt to capture how a typical customer interacts with the JDE system during the course of a typical day. The DIL kit accomplishes this with a set of scripts for 17 interactive applications as well as a set of JDE reports (UBEs) which processes a specific set of data which is part of the DIL database. Due to the availability of this standard set of scripts and UBEs, various hardware vendors, including Cisco have endeavored to characterize JDE implementations on their hardware platform to deliver a value proposition for prospective customers.
The DIL kit interactive application workload skews more towards SRM applications which feature prominently in the application workloads used by the large JDE customer base in the mid-scale manufacturing industry segment. The UBE workload is also representative of the type of reports that would be run by customers in this segment, though it does incorporate reports that cater to a larger audience of customers.
The DIL workload incorporates a good mix of applications ranging from multiple line items Sales Order and Purchase Order entries, coupled with light weight applications like supplier ledger enquiry. Similarly, the UBEs range from long running MRP processing and General Ledger Post reports to the short running Company constants and Business unit reports.
The LoadRunner scripts for the JDE interactive applications that the DIL kit incorporates measure the response times for certain key, representative transactions and these are incorporated in this whitepaper. The UBE performance is measured in terms of the total time taken to generate the report, as measured by timings recorded in the JDE logs for those UBEs.
Test Methodology
The interactive and batch version of the JDE E1 DIL kit was run to capture the end-user response time variation and Batch execution rate with important system characteristics such as CPU, memory, and I/O across the servers in the test system. All four components of the JDE E1 deployment—HTML server, Enterprise Server for interactive user, Enterprise Server for Batch and Oracle database server—were monitored through nmon Linux monitoring tool. EMC nar files were used to measure the total IOPs and latency on EMC VNX5300.
Test scenarios included a broad range of JDE applications ensuring that they closely mimic how a potential customer would utilize Oracle JD Edwards EnterpriseOne. The documented response times and the best practices to deploy Oracle JDE E1 server would give customers a good indication on how they could expect such a configuration to perform when deployed in their production environments.
Cisco endeavored to truly stress the hardware configuration, as well as provide customers with scenarios which provide a mix of interactive and batch processes running concurrently. Cisco devised various scenarios to test and record the impact of running a mixed batch workload would have on the interactive performance of JDE applications, since batch processes are typically resource hungry, thereby impacting the responsiveness of JDE interactive applications.
The elaborate test scenarios executed for JDE deployment on Cisco UCS are as follows:
•
•
•
Interactive Workload Mix
The JDE E1 DIL kit is a set of 17 scripts that include Oracle SCM, SRM, HCM, CRM, and Financials applications.
Table 9 shows the transaction mix used for the JD Edwards EnterpriseOne interactive test with the JDE E1 DIL kit.
Interactive workload Scaling
The test scenario for Interactive Scaling was executed to determine the variation in end-user response time in the event of increasing interactive users from 1000 to 7,000 concurrent users. The system resource utilization such as CPU, memory and disk IO was captured across all the three tiers i.e. HTML Server, JDE E1 Server and Oracle Database server. For 2500 concurrent interactive users, a single HTML Server VM and E1 server VM with two-node Oracle RAC VMs was utilized. For users beyond 2500 concurrent users two HTML VMs and two E1 servers VMs with a common two-node Oracle RAC VM was utilized. This was deployed to enable high interactive user scaling as well as demonstrate failover setup for HTML and E1 server.
Oracle's JDE E1 DIL Kit Illustrating Weighted Average Response Time for Interactive UsersShows the weighted average response time for 1000 to 7,000 interactive users.
Figure 200 Oracle's JDE E1 DIL Kit Weighted Average Response Time for Interactive Users
As illustrated in Figure 200, JDE E1 deployment on a Cisco UCS blade server infrastructure on Oracle Virtualized environment scales exceptionally well, with an almost flat response time of around 0.14 second for 1000 to 7,000 concurrent users. It elaborates how well Oracle JD Edwards EnterpriseOne can be consolidated on just two Cisco UCS B200 M3 servers, with peak of 7000 users.
User Response Time
User response time was captured at the LoadRunner Controller for all 17 interactive JDE E1 DIL test scripts. The five important JD Edwards EnterpriseOne applications measured using the JDE E1 DIL Kit were:
•
•
•
•
•
The transaction mix for these applications is detailed in the Workload Mix section of this document.
Oracle's JDE E1 DIL Kit Weighted Average Response Time, shows the weighted average response time for all 17 JDE E1 DIL kit scripts and for the five JD Edwards applications.
Figure 201 Oracle's JDE E1 DIL Kit Weighted Average Response Time
As detailed in Figure 201, the average response time of any of the JD Edwards application for interactive user tests was around 0.6 seconds during the scalability from 1000 to 7,000 concurrent users.
CPU Utilization
Each of the tiers; HTML server,Enterprise Server and two-node Oracle RAC database was deployed on separate Oracle VMs deployed on just two Cisco UCS B200 M3 Blade Servers, equipped with two-socket Intel Xeon E5-2690 processors. Figure 202 illustrates the average CPU utilization across the 3-Tier JD Edwards Technology Stack.
Figure 202 JDE E1 CPU Utilization
Observations:
•
•
•
•
•
Memory Utilization
Memory utilization across each of the Oracle Guest VMs for the test with 1000 to 7,000 Interactive users across the 3-Tier JD Edwards technology stack is illustrated JDE E1 Memory Utilization.
Each of the JDE OVM Servers was deployed on Cisco UCS B200 M3 server equipped with 256 GB of memory.
Figure 203 JDE E1 Memory Utilization
Observations:
•
•
•
I/O Performance
The EMC VNX5300 was configured as the storage system for each of the three components of the JDE E1 deployment: HTML server, JDE E1 server, and Oracle RAC database server. The Cisco UCS servers were booted from the SAN (EMC VNX5300), which allows unleashing of the full capabilities of Cisco UCS statelessness. Cisco UCS stateless configuration allows migration of Cisco UCS service profiles from a failed physical server to a standby server.
JDE E1 Average IOPs on VNX5300 illustrates total disk I/O performance captured with the help of EMC NAR files, during the scalability test for 1000 to 7000 JDE E1 interactive users.
Figure 204 JDE E1 Average IOPs on VNX5300
Observations:
•
•
•
Individual UBEs
Batch processing is another critical activity in Oracle JD Edwards EnterpriseOne environment and it was important to test and determine the execution time for long running individual UBE processes. This helps in proper sizing of batch servers for a Oracle JD Edwards deployment. In real-world JD Edwards deployment, several long running UBEs are run after business hours, and they must complete within a fixed duration of time since they are end of day reports and should not spill over to the following day affecting interactive users. The performance characteristics of these UBEs are summarized in Table 10, with a brief description of what the UBE provides and what dataset they operate on. These long running reports ran on a fixed set of data with a standard set of processing options against the JDE DIL database, as described below.
Only Batch Execution
There are customers who run only JDE Batch processing for many business functions during a dedicated window and it was imperative on Cisco to test and provide enough information for such customers to make an informed decision regarding their deployment of JDE on Cisco UCS using Oracle Virtualization. For such customers a test scenario was configured wherein, high volume of short running UBEs as well as 3 long running UBEs were executed, and the impact of running this test scenario was measured in terms of CPU and memory consumed on the Enterprise server, Database server and each of the two Oracle Virtual Servers.
The test successfully achieved 320 UBEs/minute. The average IOPS measure on the EMC VNX5300 was around 3200. It would be a good strategy for real-world JD Edwards EnterpriseOne customers, to schedule very high volume of UBEs during those non-peak hours when minimal interactive users are logged in to the JDE system
Resource Utilization for Only Batch Execution, illustrates the CPU and memory utilization on JDE Enterprise Server and two-node Oracle RAC database during the execution of Only Batch Processes.
Figure 205 Resource Utilization for Only Batch Execution
Best Practices & Tuning Recommendations
Oracle JD Edwards EnterpriseOne deployed on Cisco UCS with EMC Storage system with Oracle VM was configured for small to medium scale ERP deployment. The benchmark of JDE DIL Kit demonstrated exceptional performance for JDE interactive users and JDE Batch processes. The subsequent sections elaborate on the tuning parameters and best practices incorporated across hardware and software stack.
System Configuration
All the tests were executed using two Cisco UCS B200 M3 servers. Each server was equipped with two E5-2690 processor and 256 GB of physical memory. Six Guest VMs were carved on the two B200 M3, with two instance each of HTML Server, JDE EnterpriseOne Server and Oracle RAC node.
Both of the Cisco UCS Blade Servers were attached to a BIOS policy. BIOS policy is one of the features of UCS Service Profiles which enable users to incorporate similar BIOS settings all across the deployed servers. This helps a consistent configuration and the Administrators do not need to interrupt the boot process on each server to alter the BIOS setting. The BIOS policy configured for Oracle JDE deployment is shown in Figure 206.
Figure 206 BIOS setting for CPU Performance
Figure 207 BIOS setting for Indirect IO
Figure 208 BIOS Setting for Physical Memory
Disable I/O MMU Virtualization Settings (VT-d). See,
http://docs.oracle.com/cd/E27300_01/E27307/html/vmrns-bugs.html
Jumbo Frames
Jumbo frames are supported in Oracle VM 3.1.1, this enables utilization of 10G ports for Oracle RAC interconnect traffic. Please ensure to use Oracle VM 3.1.1 Build 819 and above to successfully configure jumbo frames Oracle RAC private network.
Configuration of jumbo frames is demonstrated in the Infrastructure Setup Section of this document.
TCP Parameter Configuration
Default TCP parameters in most Linux VM distributions are conservative, and are tuned to handle 100Mb/s or 1Gb/s port speeds, and result in buffer sizes that are too small for 10Gb networks. Modifying these values can lead to significant performance gain in a 10Gbs VM network.
Buffer sizes should be adjusted to permit the maximum number of bytes in transit and prevent traffic throttling. The following values can be set on the virtual machines as well as dom0. Edit /etc/sysctl.conf for the below mentioned changes. These can be changed both on virtual machines as well as dom0. (See, Oracle VM 3: 10GbE Network Performance Tuning whitepaper
net.core.rmem_max = 134217728net.core.wmem_max = 134217728net.ipv4.tcp_rmem = 4096 87380 134217728net.ipv4.tcp_wmem = 4096 65536 134217728net.core.netdev_max_backlog = 300000net.ipv4.tcp_moderate_rcvbuf =1Additionally, for Oracle VM Server for x86 prior to version 3.2, netfilter should be turned off on bridge devices. This is done automatically in OVM version 3.2 and later.
net.bridge.bridge-nf-call-iptables=0net.bridge.bridge-nf-call-arptables=0net.bridge.bridge-nf-call-ip6tables=0Oracle RAC Configuration
Several Settings were changed on Oracle RAC cluster VMs to support the high load that the RDBMS was handling. Some of the important tuning parameters are:
•
•
•
•
•
For more details on Disk layout for Oracle RAC database, please refer Table 11. LUN Configuration for JD Edwards EnterpriseOne deployment
Some of the important Oracle RAC configuration parameters are detailed below:
Each of the Oracle RAC VM resided on a separate B200M3 server (Ref: Figure 14. Guest OS configuration of Oracle JD Edwards EnterpriseOne on OVM). Oracle RAC VM was configured with 10 vCPU and 120 GB of physical memory. There is a significant impact to the database memory consumption with dedicated connections that can be avoided by using shared server connections. (Ref: Doc ID 1188240.1 on https://support.oracle.com. In order to scale to 7000 users and with limited available memory for Oracle RAC VM, the present Oracle RAC setup was configured in shared connection mode.
WebLogic Server Configuration
The JRockit Java Virtual Machine (JVM) was used along with WebLogic 10.3.6. A vertical cluster of up to 20 JVMs was created and Oracle HTTP Server was used to load balance the load among the various nodes of the vertical cluster
Some of the important configuration details for the WebLogic Server are:
•
•
•
•
•
Oracle JD Edwards Enterprise Server Configuration
The JD Edwards tools release 8.98.4.10 was used with JD Edwards application release 9.0.2. The number of interactive users per callobject peaked at around 9/callobject kernel.
Some of the important configuration settings for JDE initialization files are detailed below
•
–
- Security kernels 70
- Call Object kernels 400
- Workflow kernels 30
- Metadata kernels 1
–
- maxNetProcesses=60
- maxNetConnections=8000
- maxKernelProcesses=1000
- maxNumSocketMsgQueue=1000
- maxIPCQueueMsgs=600
- maxLenInlineData=4096
- maxLenFixedData=16384
- maxFixedDataPackets=2000
- internalQueueTimeOut=90
–
- maxNumberOfResources=4000
- maxNumberOfSemaphores=2000
- startIPCKeyValue=6001
- avgResourceNameLength=40
- avgHandles=200
- hashBucketSize=53
- maxMsgqMsgBytes=5096
- maxMsgqEntries=1024
- maxMsgqBytes=65536
- msgQueueDelayTimeMillis=40
•
–
- minConnection=5
- maxConnection=800
- poolGrowth=5
- initialConnection=25
- maxSize=500
•
–
- MAXUser=500
- OWVirtualThreadPoolSize=800
–
- maxPoolSize=500
Conclusion
This CVD demonstrates how Cisco UCS servers on Oracle VM along with the latest EMC VNX storage technologies, form a highly reliable, robust solution for Oracle JDE Edwards implementation.
Enterprise Resource Planning (ERP) has been around for many decades and has provided agile IT practices to the Business. Organizations that have used the ERP Packages have immensely benefitted by streamlining their back end processes to improve management and improve ROI.
ERP being a business critical Application, takes a long time to implement and test; there is always a concern to move to newer technologies or experiment with the advanced features that are available today. Since this is a Business Critical Application, one of the most important concerns is predictability; will it work for us, how and at what cost?
Cisco has invested considerable time and effort to test, validate and characterize Oracle JD Edwards on Cisco UCS Platform using EMC VNX Storage with Oracle VM, thus providing comprehensive scalable architecture and best practices. By leveraging the best practices and expertise in our extensive Oracle JDE benchmark activity, customers can confidently deploy and consolidate JD Edwards EnterpriseOne on Cisco UCS platform with Oracle VM virtualization and EMC VNX storage family.
Cisco Oracle Competency Center has provided considerable information in this document by testing and characterizing the Cisco UCS on Oracle Virtual environment using the Oracle JD Edwards EnterpriseOne software stack. With the scalability demonstrated by the test results, Cisco is confident that these astounding results will prove that Cisco UCS, Oracle VM and EMC VNX storage is a solid fit for any customer considering virtualization of Oracle JD Edwards EnterpriseOne.
Bill of Materials
Table 12 and Table 13 give details of all the hardware and software components used in the CVD.
Appendix A - Workload Mix for Batch and Interactive Test
Appendix B - Reference Documents
•
•
•
•
•
Appendix C - Reference Links
•
http://www.cisco.com/en/US/docs/unified_computing/ucs/hw/chassis/install/ucs5108_install.html
•
•
•
http://www.emc.com/collateral/software/white-papers/h8058-fast-vp-unified-storage-wp.pdf
•
http://www.emc.com/collateral/software/white-papers/h8046-clariion-celerra-unified-fast-cache-wp.pdf
•
http://www.emc.com/collateral/software/white-papers/h8058-fast-vp-unified-storage-wp.pdf
•
http://www.emc.com/collateral/hardware/data-sheets/h8520-vnx-family-ds.pdf
Feedback