FlexPod Datacenter for SAP Solution using FibreChannel SAN with Cisco UCS Manager 4.0 and NetApp ONTAP 9.7

FlexPod Datacenter for SAP Solution using FibreChannel SAN with Cisco UCS Manager 4.0 and NetApp ONTAP 9.7

Deployment Guide for FibreChannel-based FlexPod Datacenter Solution for SAP and SAP HANA with Cisco UCS 4^th Generation and NetApp ONTAP 9.7

Published: December 2020

NOTE: Works with document’s Advanced Properties “Last Updated” property. Click File | Properties | Advanced Properties | Custom.

In partnership with:

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, go to:

http://www.cisco.com/go/designzone.

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

© 2020 Cisco Systems, Inc. All rights reserved.

 

Contents

Executive Summary. 4

Solution Overview.. 5

Deployment Hardware and Software. 6

Network Switch Configuration. 11

Storage Configuration. 18

Cisco UCS Configuration. 38

SAN Switch Configuration. 109

HANA Node Preparation. 122

System Provisioning for SAP HANA. 154

SAP HANA Installation. 172

Cisco Intersight 176

Monitor SAP HANA with AppDynamics. 183

Appendix. 189

About the Authors. 192

Feedback. 193

Executive Summary

Cisco and NetApp have partnered to deliver FlexPod, which serves as the foundation for a variety of SAP and SAP HANA workloads and enables efficient architectural designs that are based on customer requirements. FlexPod Datacenter solution is a validated approach for deploying Cisco and NetApp technologies as a shared infrastructure platform for SAP HANA implementation in Tailored DataCenter Integration (TDI) mode.

This document describes the Cisco and NetApp FlexPod Datacenter with NetApp ONTAP 9.7 on NetApp AFF A400 storage, Cisco UCS Manager unified software release 4.1(1) with 2^nd Generation Intel Xeon Scalable Processors for SAP HANA in particular.

FlexPod Datacenter with NetApp ONTAP 9.7 and Cisco UCS unified software release 4.1(1) is a predesigned, best-practice datacenter architecture built on the Cisco Unified Computing System (Cisco UCS), the Cisco Nexus 9000 family of switches, MDS 9000 multilayer fabric switches, and NetApp AFF A-Series storage arrays running ONTAP 9.7 storage OS.

Solution Overview

Introduction

The current industry trend in datacenter design is towards shared infrastructures. With pre-validated IT platforms, enterprise customers have embarked on the journey to the cloud by moving away from application silos and toward shared infrastructure that can be quickly deployed, thereby increasing agility, and reducing costs.

Cisco and NetApp have partnered to deliver FlexPod, which uses best of breed storage, server, and network components to serve as the foundation for a variety of SAP workloads in general and SAP HANA in particular, enabling efficient architectural designs that can be quickly and confidently deployed.

Audience

The audience for this document includes, but is not limited to; sales engineers, field consultants, professional services, IT managers, partner engineers, and customers who want to take advantage of an infrastructure built to deliver IT efficiency and enable IT innovation.

Purpose of this Document

This document provides a step-by-step configuration and implementation guide for the FlexPod Datacenter with Cisco UCS Fabric Interconnects, NetApp AFF storage, Cisco MDS, and Cisco Nexus 9000 solution.

What’s New in this Release?

The following design elements distinguish this version of FlexPod from previous FlexPod architectures:

●     Cisco UCS 4.1(1) unified software release, Cisco UCS B480-M5 with 2^nd Generation Intel Xeon Scalable Processors, and Cisco 1400 Series Virtual Interface Cards (VICs)

●     Cisco UCS 6454 Fabric Interconnects

●     Cisco UCS 2408 Fabric Extender

●     Cisco Intersight Software as a Service (SaaS) Management

●     NetApp AFF A400 Storage Controller

●     NetApp ONTAP® 9.7

●     NetApp SnapCenter and NetApp SnapCenter Plug-in for SAP HANA Version 4.3

●     Fibre channel and NFS  storage design

●     Unified Extensible Firmware Interface (UEFI) Secure Boot with SLES for SAP Applications 15 SP2 and RHEL for SAP HANA 8.1 Operating Systems.

●     32 Gigabit per second Fibre Channel Connectivity

Deployment Hardware and Software

Architecture

FlexPod is a defined set of hardware and software that serves as an integrated foundation for both virtualized and non-virtualized SAP HANA workloads. The FlexPod design is flexible enough that the networking, computing, and storage can fit in one data center rack or be deployed according to a customer's data center design. Port density enables the networking components to accommodate multiple configurations of this kind.

One benefit of the FlexPod architecture is the ability to customize or "flex" the environment to suit a customer's requirements. A FlexPod can easily be scaled as requirements and demand change. The unit can be scaled both up (adding resources to a FlexPod unit) and out (adding more FlexPod units). The reference architecture detailed in this document highlights the resiliency, cost benefit, and ease of deployment of a Fibre Channel and IP-based storage solution. A storage system capable of serving multiple protocols across a single interface allows for customer choice and investment protection because it truly is a wire-once architecture.

Figure 1 shows the VMware vSphere built on FlexPod components and the network connections for a configuration with the Cisco UCS 6454 Fabric Interconnects. This design has port-channeled 25 Gb Ethernet connections between the Cisco UCS 5108 Blade Chassis and the Cisco UCS Fabric Interconnects via the Cisco UCS 2408 Fabric Extenders, 25 Gb Ethernet connections between the Cisco UCS Fabric Interconnect and Cisco Nexus 9000, and between Cisco Nexus 9000 and NetApp AFF A400 storage array. This infrastructure option is expanded with Cisco MDS switches placed between the Cisco UCS Fabric Interconnect and the NetApp AFF A400 to provide FC-booted hosts with 32 Gb FC block-level access to storage serving them SAP HANA persistence as well.

Topology

Figure 1.       FlexPod with Cisco UCS 6454 Fabric Interconnects and NetApp AFF A400

The reference architecture hardware configuration includes:

●     Two Cisco Nexus 93180YC-FX switches

●     Two Cisco UCS 6454 fabric interconnects

●     Two Cisco MDS 9148T multilayer fabric switches

●     One NetApp AFF 400 (HA pair) running ONTAP 9.7 with external NVMe Disk shelf NS224 SSD disks

Software Revisions

Table 1 lists the software revisions for this solution.

Table 1.          Software Revisions

Configuration Guidelines

This document explains how to configure a fully redundant, highly available configuration for a FlexPod unit with ONTAP storage. Therefore, reference is made to which component is being configured with each step, either 01 or 02 or A and B. For example, node01 and node02 are used to identify the two NetApp storage controllers that are provisioned with this document, and Cisco Nexus A or Cisco Nexus B identifies the pair of Cisco Nexus switches that are configured. The Cisco UCS Fabric Interconnects are similarly configured. Additionally, this document details the steps for provisioning multiple Cisco UCS hosts, and these examples are identified as: HANA-scaleup-01, HANA-scaleout-01 etc to represent infrastructure hosts deployed to each of the fabric interconnects in this document. Finally, to indicate that you should include information pertinent to your environment in a given step, <text> appears as part of the command structure. See the following example for the network port vlan create command:

Usage:

network port vlan create ?

 [-node] <nodename>                  Node

 { [-vlan-name] {<netport>|<ifgrp>}  VLAN Name

 |  -port {<netport>|<ifgrp>}        Associated Network Port

 [-vlan-id] <integer> }              Network Switch VLAN Identifier

Example:

network port vlan create -node <node01> -vlan-name a0a-<vlan id>

This document is intended to enable you to fully configure the customer environment. In this process, various steps require you to insert customer-specific naming conventions, IP addresses, and VLAN schemes, as well as to record appropriate MAC addresses. Table 2 describes the VLANs necessary for deployment as outlined in this guide.

Table 2.          Necessary VLANs

Physical Infrastructure

FlexPod Cabling

The information in this section is provided as a reference for cabling the physical equipment in a FlexPod environment. To simplify cabling requirements, a cabling diagram was used.

The cabling diagram in this section contains details for the prescribed and supported configuration of the NetApp AFF 400 running NetApp ONTAP® 9.7.

This document assumes that out-of-band management ports are plugged into an existing management infrastructure at the deployment site. These interfaces will be used in various configuration steps.

The NetApp storage controller and disk shelves should be connected according to best practices for the specific storage controller and disk shelves. For disk shelf cabling, refer to NetApp Support.

Figure 2 details the cable connections used in the validation lab for the FlexPod topology based on the Cisco UCS 6454 fabric interconnect. Four 32Gb uplinks connect as port-channels to each Cisco UCS Fabric Interconnect from the MDS switches, and a total of eight 32Gb links connect the MDS switches to the NetApp AFF controllers.  Also, 25Gb links connect the Cisco UCS Fabric Interconnects to the Cisco Nexus Switches and the NetApp AFF controllers to the Cisco Nexus Switches. Additional 1Gb management connections will be needed for an out-of-band network switch that sits apart from the FlexPod infrastructure. Each Cisco UCS fabric interconnect and Cisco Nexus switch is connected to the out-of-band network switch, and each AFF controller has a connection to the out-of-band network switch.

Figure 2.       FlexPod Cabling with Cisco UCS 6454 Fabric Interconnect and NetApp AFF 400

Network Switch Configuration

This section provides a detailed procedure to configure the Cisco Nexus 9000s to use in a FlexPod environment.

Physical Connectivity

Follow the physical connectivity guidelines for FlexPod as explained in the section FlexPod Cabling.

FlexPod Cisco Nexus Base

The following procedures describe how to configure the Cisco Nexus switches for use in a base FlexPod environment. This procedure assumes you’re using Cisco Nexus 9000 7.0(3)I7(6), the Cisco suggested Nexus switch release at the time of this validation.

Table 3 lists the VLANs necessary for deployment as outlined in this guide.

Table 3.          Configuration Variables

Set Up Initial Configuration

Cisco Nexus A

To set up the initial configuration for the Cisco Nexus A switch on <nexus-A-hostname>, follow these steps:

1.    Configure the switch.

On initial boot and connection to the serial or console port of the switch, the NX-OS setup should automatically start and attempt to enter Power on Auto Provisioning.

2.    Review the configuration summary before enabling the configuration.

Use this configuration and save it? (yes/no) [y]: Enter

Cisco Nexus B

To set up the initial configuration for the Cisco Nexus B switch on <nexus-B-hostname>, follow these steps:

1.    Configure the switch.

2.    Review the configuration summary before enabling the configuration.

Use this configuration and save it? (yes/no) [y]: Enter

FlexPod Cisco Nexus Switch Configuration

Enable Licenses

Cisco Nexus A and Cisco Nexus B

To license the Cisco Nexus switches, follow these steps:

1.    Log in as admin.

2.    Run the following commands to enable the required features:

Create VLANs

Cisco Nexus A and Cisco Nexus B

To create the necessary virtual local area networks (VLANs), follow this step on both switches:

1.    From the global configuration mode, run the following commands:

Create Port Channels and assign interfaces

Cisco Nexus A

To create the necessary port channels between devices, follow this step:

1.    From the global configuration mode, run the following commands:

Cisco Nexus B

To create the necessary port channels between devices, follow this step:

1.    From the global configuration mode, run the following commands:

Configure Port Channel Parameters

Cisco Nexus A and Cisco Nexus B

To configure port channel parameters, follow this step on both switches:

1.    From the global configuration mode, run the following commands:

Configure Virtual Port Channels

Cisco Nexus A

To configure virtual port channels (vPCs) for switch A, follow this step:

1.    From the global configuration mode, run the following commands:

 

Cisco Nexus B

To configure vPCs for switch B, follow this step:

1.    From the global configuration mode, run the following commands:

 

Uplink into Existing Network Infrastructure

Depending on the available network infrastructure, several methods and features can be used to uplink the FlexPod environment. If an existing Cisco Nexus environment is present, it is recommended to use vPCs to uplink the Cisco Nexus switches included in the FlexPod environment into the infrastructure. Use this procedure to create an uplink vPC to the existing environment. Make sure to run copy run start to save the configuration on each switch after the configuration is completed.

Storage Configuration

Complete Configuration Worksheet

Before running the setup script, complete the cluster setup worksheet and review the configuration worksheets in the ONTAP 9.7 Software Setup Guide  to learn about configuring ONTAP software.

Configure ONTAP Nodes

Table 4 and Table 5 list the information needed to configure two ONTAP nodes. Customize the cluster detail values with the information applicable to your deployment.

Table 4.          ONTAP Software Installation Prerequisites

Set Up ONTAP Cluster

Table 5 lists all the parameters required to set up the ONTAP cluster.

Table 5.          ONTAP Cluster Prerequisites

To set up an ONTAP cluster, follow these steps:

1.    From a console port program attached to the storage controller A (node 01) console port, run the node setup script. This script appears when ONTAP 9.7 software boots on the node for the first time.

2.    Follow the prompts to set up node 01:

3.    Press Enter to continue the cluster setup via CLI.

4.    Create a new cluster:

5.    Press Enter for the default option “no” to setup a single node cluster:

6.    Create the cluster interface configuration. Choose Yes if you want to use the default settings.

7.    Provide the cluster administrator’s password:

8.    Create the cluster and provide a cluster name :

9.    Add the necessary license keys:

10.  Create the vserver for cluster administration:

11.  Provide the DNS domain names and DNS server IP address:

12.  Finish the first part of the setup:

13.  From a console port program attached to the storage controller B (node 02) console port, run the node setup script. This script appears when ONTAP 9.7 software boots on the node for the first time.

14.  Follow the prompts to set up node 02:

15.  Press Enter to continue the cluster setup via CLI.

16.  Join the new cluster:

17.  Create the cluster interface configuration. Choose Yes if you want to use the default settings:

18.  Enter an IP address of the private cluster network from node 1:

19.  Finish the second part of the setup:

20.  Open an SSH connection to either the cluster IP or host name.

21.  Log in with the admin user with the password you provided earlier.

Set Auto-Revert on Cluster Management

To set the auto-revert parameter on the cluster management interface, run the following command:

Set Up Management Broadcast Domain

By default, all network ports are included in the default broadcast domain. Network ports used for data services (for example, e0d, e1a, and e1e) should be removed from the default broadcast domain, leaving just the management network ports (e0M).

To make the changes, the following commands must be executed for each storage node. Storage nodes are named after the cluster name with an appended number.

Create Aggregates

An aggregate containing the root volume for each storage controller is created during the ONTAP software setup process. To create additional aggregates, determine the aggregate name, the node on which to create it, and the number of disks it should contain.

To create new aggregates, run the following commands:

 

Optional: Rename the root aggregate on node 01 to match the naming convention for this aggregate on node 02. The aggregate is automatically renamed if system-guided setup is used.

Verify Storage Failover

To confirm that storage failover is enabled, run the following commands for a failover pair:

1.    Verify the status of storage failover.

2.    Enable failover on one of the two nodes.

3.    Verify the HA status for a two-node cluster.

4.    Continue with step 6 if high availability is configured.

5.    Only enable HA mode for two-node clusters. Do not run this command for clusters with more than two nodes because it causes problems with failover.

6.    Verify that hardware assist is correctly configured and, if needed, modify the partner IP address.

Verify FCP Ports are Set to Target Ports

To change the FCP to target mode if they are configured as initiator, follow these steps:

1.    Check the ports if they configured correctly:

2.    Disable all ports which need to be changed:

3.    Change the HBAs mode to target:

4.    Reboot  each controller node:

5.    Wait until the first node is back up and running and reboot the second node:

Disable Flow Control on 25/40/100GbE Ports

NetApp recommends disabling flow control on all the 25/40/100GbE ports that are connected to external devices. To disable flow control, follow these steps:

1.    Run the following commands to configure node 1:

2.    Run the following commands to configure node 2:

Configure AutoSupport

NetApp AutoSupport^® sends support summary information to NetApp through HTTPS. To configure AutoSupport, run the following command:

Enable Cisco Discovery Protocol

To enable the Cisco Discovery Protocol (CDP) on the NetApp storage controllers, run the following command:

Create Broadcast Domain for NFS access

For this setup, the VLAN ID 228 has been used for NFS access which is needed for /hana/shared for SAP HANA multiple host setups.

The broadcast domains must be created with an MTU size of 9000 (jumbo frames):

Create Interface Groups

To create the LACP interface groups for the 25GbE data interfaces, run the following commands:

Create VLAN for NFS access

To create VLANs, follow these steps:

1.    Set the MTU size of the interface groups.

2.    Create HANA NFS VLAN ports and add them to the NFS broadcast domain.

Configure HTTPS Access

For each of the SVMs and the cluster node, create a certificate to allow secure communication with HTTPS. For each of the certificates, specify the individual values listed in Table 6.

Table 6.           ONTAP Software Parameters Needed to Enable HTTPS

To configure secure access to the storage controller, follow these steps:

1.    Increase the privilege level to access the certificate commands.

2.    Generally, a self-signed certificate is already in place. Verify the certificate and obtain parameters (for example the <serial-number>) by running the following command:

3.    For each SVM shown, the certificate common name should match the DNS FQDN of the SVM. Delete the two default certificates and replace them with either self-signed certificates or certificates from a certificate authority (CA). To delete the default certificates, run the following commands:

4.    To generate and install self-signed certificates, run the following commands as one-time commands. Generate a server certificate for the Infra-SVM, the HANA SVM, and the cluster SVM. Use tab completion to aid in the completion of these commands.

5.    To obtain the values for the parameters required in the next step  (<cert-ca> and <cert-serial>), run the security certificate show command.

6.    Enable each certificate that was just created by using the –server-enabled true and –client-enabled false parameters. Use tab completion to aid in the completion of these commands.

7.    Disable HTTP cluster management access.

8.    Change back to the normal admin privilege level and set up the system so that  SVM logs  are available through the web.

Configure SVM for the Infrastructure

Table 7 and Figure 3 describe the infrastructure SVM together with all required storage objects (volumes and LIFs).

Figure 3.       Overview of Infrastructure SVM Components

Table 7.          ONTAP Software Parameters for Infrastructure SVMs

Create SVM for the Infrastructure

To create an infrastructure SVM, follow these steps:

1.    Run the vserver create command.

2.    Select the SVM data protocols to configure, keeping iSCSI and NFS.

3.    Add the data aggregates to the SVM aggregate list.

4.    Enable and run the NFS protocol in the SVM.

Create Load-Sharing Mirrors

To create a load-sharing mirror of an SVM root volume, follow these steps:

1.    Create a volume to be the load-sharing mirror of the infrastructure SVM root volume on each node.

2.    Create the mirroring relationships.

3.    Initialize the mirroring relationship.

Add Infrastructure SVM Management LIF

To add the infrastructure SVM administration LIF in the out-of-band management network, follow these steps:

1.    Run the following commands:

2.    Create a default route to allow the SVM management interface to reach the outside world.

Create FCP LIFs

To create the eight FCP LIFs (four on each node), run the following commands:

Create Block Protocol (FCP) Service

Run the following command to create the FCP service. This command also starts the iSCSI service.

Create FlexVol Volumes

To create FlexVol volumes, run the following commands:

Configure LUNs for FCP Boot

Create Boot LUNs for Servers

To create boot LUNs, run the following commands. This example creates boot LUNs for four servers. Repeat the command with a different LUN name to create additional boot LUNs for additional servers.

Create Portset

To create a portset that includes four FCP LIFs, run the following commands:

Create igroups

To create igroups, run the following commands:

Repeat the command by using the WWPNs of additional servers to create additional igroups for additional servers.

Map Boot LUNs to igroups

To map server boot LUNs to igroups, run the following commands. Repeat this command to map additional boot LUNs to additional servers.

Configure SVM for HANA

Table 8 and Figure 4 describe the HANA SVM together with all the required storage objects (volumes, export-policies, and LIFs).

Figure 4.       Overview of SAP HANA SVM Components

Table 8.          ONTAP Software Parameter for HANA SVM

Create SVM for SAP HANA

To create an SVM for SAP HANA volumes, follow these steps:

1.    Run the vserver create command.

2.    Select the SVM data protocols to configure, keeping NFS and FCP.

3.    Add the two data aggregates to the hana-svm aggregate list.

4.    Disable any QOS policy at the vserver level

5.    Enable and run the NFS protocol in the Infra-SVM.

6.    Enable a large NFS transfer size.

7.    Set the group ID of the user root to 0.

Create Load-Sharing Mirrors

To create a load-sharing mirror of an SVM root volume, follow these steps:

1.    Create a volume to be the load-sharing mirror of the HANA SVM root volume on each node.

2.    Create the mirroring relationships.

3.    Initialize the mirroring relationship.

Create Export Policies for the Root Volumes

To configure the NFS export policies on the SVM, follow these steps:

1.    Create a new rule for the infrastructure NFS subnet in the default export policy.

2.    Assign the FlexPod export policy to the infrastructure SVM root volume.

Add HANA SVM Management Interface and Administrator

To add the HANA SVM administrator and SVM administration LIF in the out-of-band management network, follow these steps:

1.    Run the following commands:

2.    Create a default route to allow the SVM management interface to reach the outside world.

3.    Set a password for the SVM vsadmin user and unlock the user.

Create Export Policies for the HANA SVM

1.    Create a new export policy for the HANA data and log subnet.

2.    Create a rule for this policy.

Create NFS LIF for SAP HANA Shared

To create the NFS LIFs for SAP HANA data, run the following commands:

Create FCP LIFs

To create the eight FCP LIFs (four on each node), run the following commands:

Create Portset

To create a portset that includes all FCP LIFs, run the following commands:

Create igroups for SAP HANA Servers

Use the WWPN information you defined in section Create WWPN Pools to create the igroups.

To create igroups, run the following commands. Repeat this command by using the WWPNs of additional servers to create additional igroups for additional servers.

Cisco UCS Configuration

Cisco UCS Base Configuration

This FlexPod deployment explains the configuration steps for the Cisco UCS 6454 Fabric Interconnects (FI) in a design that will support Fibre Channel SAN boot.

Perform Initial Setup of Cisco UCS 6454 Fabric Interconnects for FlexPod Environments

This section provides the detailed procedures for configuring the Cisco Unified Computing System (Cisco UCS) for use in a FlexPod environment. The steps are necessary to provision the Cisco UCS B-Series and C-Series servers and should be followed precisely to avoid improper configuration.

Cisco UCS Fabric Interconnect A

To configure the Cisco UCS for use in a FlexPod environment, follow these steps:

1.    Connect to the console port on the first Cisco UCS fabric interconnect.

2.    Wait for the login prompt for the Cisco UCS Fabric Interconnect A before proceeding to the next section.

Cisco UCS Fabric Interconnect B

To configure the Cisco UCS for use in a FlexPod environment, follow these steps:

1.    Connect to the console port on the second Cisco UCS fabric interconnect.

2.    Wait for the login prompt for the Cisco UCS Fabric Interconnect B before proceeding to the next section.

Cisco UCS Setup

Log into Cisco UCS Manager

To log into the Cisco Unified Computing System (Cisco UCS) environment, follow these steps:

1.    Open a web browser and navigate to the Cisco UCS fabric interconnect cluster address.

2.    Click the Launch UCS Manager link to launch Cisco UCS Manager.

3.    If prompted to accept security certificates, accept as necessary.

4.    When prompted, enter admin for the user name and enter the administrative password.

5.    Click Login to log into Cisco UCS Manager.

Anonymous Reporting

To enable anonymous reporting, follow this step:

1.    In the Anonymous Reporting window, choose whether to send anonymous data to Cisco for improving future products. If you choose Yes, enter the IP address of your SMTP Server. Click OK.

Upgrade Cisco UCS Manager Software to Version 4.1(1d)

This document assumes you’re using Cisco UCS 4.1(1d). To upgrade the Cisco UCS Manager software and the Cisco UCS Fabric Interconnect software to version 4.1(1d), refer to Cisco UCS Manager Install and Upgrade Guides.

Synchronize Cisco UCS to NTP

To synchronize the Cisco UCS environment to the NTP servers in the Cisco Nexus switches, follow these steps:

1.    In Cisco UCS Manager, click Admin.

2.    Expand All > Time Zone Management.

3.    Choose Timezone.

4.    In the Properties pane, choose the appropriate time zone in the Timezone menu.

5.    Click Save Changes and then click OK.

6.    Click Add NTP Server.

7.    Enter <nexus-A-mgmt0-ip> and click OK. Click OK to confirm.

8.    Click Add NTP Server.

9.    Enter <nexus-B-mgmt0-ip> and click OK, then click OK again.

Add Additional DNS Server(s)

To add one or more additional DNS servers to the UCS environment, follow these steps:

1.    In Cisco UCS Manager, click Admin.

2.    Expand All > Communications Management.

3.    Choose DNS Management.

4.    In the Properties pane, choose Specify DNS Server.

5.    Enter the IP address of the additional DNS server.

6.    Click OK and then click OK again. Repeat this process for any additional DNS servers.

Configure Unified Ports (FCP)

Fibre Channel port configurations differ between the Cisco UCS 6454, 6332-16UP and the 6248UP fabric interconnects.  All fabric interconnects have a slider mechanism within the Cisco UCS Manager GUI interface, but the fibre channel port selection options for the Cisco UCS 6454 are from the first 16 ports starting from the first port and configured in increments of 4 ports from the left. For the Cisco UCS 6332-16UP the port selection options are from the first 16 ports starting from the first port, and configured in increments of the first 6, 12, or all 16 of the unified ports.  With the 6248UP, the port selection options will start from the right of the 32 fixed ports, or the right of the 16 ports of the expansion module, going down in contiguous increments of 2. The remainder of this section shows configuration of the 6454. Modify as necessary for the Cisco UCS 6332-16UP or 6248UP.

To enable the fibre channel ports, follow these steps for the Cisco UCS 6454:

1.    In Cisco UCS Manager, click Equipment.

2.    Choose Equipment > Fabric Interconnects > Fabric Interconnect B (subordinate).

3.    Choose Configure Unified Ports.

4.    Click Yes in the pop-up window warning that changes to the fixed module will require a reboot of the fabric interconnect and changes to the expansion module will require a reboot of that module.

5.    Within the Configured Fixed Ports pop-up window move the gray slider bar from the left to the right to choose either 4, 8, 12, or 16 ports to be set as FC Uplinks.

6.    Click OK, then click Yes, then click OK to continue.

7.    Choose Equipment > Fabric Interconnects > Fabric Interconnect A (primary).

8.    Choose Configure Unified Ports.

9.    Click Yes on the pop-up window warning that changes to the fixed module will require a reboot of the fabric interconnect and changes to the expansion module will require a reboot of that module.

10.  Within the Configured Fixed Ports pop-up window move the gray slider bar from the left to the right to choose either 4 or 8 ports to be set as FC Uplinks.

11.  Click OK, then click Yes, then OK to continue.

12.  Wait for both Fabric Interconnects to reboot.

13.  Log back into Cisco UCS Manager.

Edit Chassis Discovery Policy

Setting the discovery policy simplifies the addition of Cisco UCS B-Series chassis and of additional fabric extenders for further Cisco UCS C-Series connectivity. To modify the chassis discovery policy, follow these steps:

1.    In Cisco UCS Manager, click Equipment and choose the Policies tab.

2.    Under Global Policies, set the Chassis/FEX Discovery Policy to match the minimum number of ports that are cabled between the chassis or fabric extenders (FEXes) and the fabric interconnects.

3.    On the Cisco UCS 6454 Fabric Interconnects, the Link Grouping Preference is automatically set to Port Channel and is greyed out. On a Cisco UCS 6300 Series or Cisco UCS 6200 Series Fabric Interconnect, set the Link Grouping Preference to Port Channel. If Backplane Speed Preference appears, leave it set at 40G.

4.    If any changes have been made, click Save Changes, and then click OK.

Enable Port Auto-Discovery Policy

Setting the port auto-discovery policy enables automatic discovery of Cisco UCS B-Series chassis server ports. To modify the port auto-discovery policy, follow these steps:

1.    In Cisco UCS Manager, click Equipment, choose All > Equipment in the Navigation Pane, and choose the Policies tab.

2.    Under Port Auto-Discovery Policy, set Auto Configure Server Port to Enabled.

3.    Click Save Changes and then click OK.

Enable Server and Uplink Ports

To enable and verify server and uplink ports, follow these steps:

1.    In Cisco UCS Manager, click Equipment.

2.    Expand Equipment > Fabric Interconnects > Fabric Interconnect A (primary) > Fixed Module.

3.    Expand and choose Ethernet Ports.

4.    Verify that all ports connected to UCS chassis are configured as Server ports and have a status of Up.

5.    Click Yes to confirm server ports and click OK.

6.    Verify that the ports connected to the chassis are now configured as server ports.

7.    Choose the ports that are connected to the Cisco Nexus switches, right-click them, and choose Configure as Uplink Port.

8.    Click Yes to confirm uplink ports and click OK.

9.    Choose Equipment > Fabric Interconnects > Fabric Interconnect B (subordinate) > Fixed Module.

10.  Expand and choose Ethernet Ports.

11.  Verify that all ports connected to UCS chassis are configured as Server ports and have a status of Up.

12.  Click Yes to confirm server ports and click OK.

13.  Verify that the ports connected to the chassis are now configured as server ports.

14.  Choose the ports that are connected to the Cisco Nexus switches, right-click them, and choose Configure as Uplink Port.

15.  Click Yes to confirm the uplink ports and click OK.

Enable Info Policy for Neighbor Discovery

Enabling the info policy enables Fabric Interconnect neighbor information to be displayed. To modify the info policy, follow these steps:

1.    In Cisco UCS Manager, click Equipment, choose All > Equipment in the Navigation Pane, and choose the Policies tab on the right.

2.    Under Global Policies, scroll down to Info Policy and choose Enabled for Action.

3.    Click Save Changes and then click OK.

4.    Under Equipment, choose Fabric Interconnect A or B. On the right, choose the Neighbors tab. CDP information is shown under the LAN tab and LLDP information is shown under the LLDP tab.

Acknowledge Cisco UCS Chassis and FEX

To acknowledge all Cisco UCS chassis and any external FEX modules, follow these steps:

1.    In Cisco UCS Manager, click Equipment.

2.    Expand Chassis and choose each chassis that is listed.

3.    Right-click each chassis and choose Acknowledge Chassis.

4.    Click Yes and then click OK to complete acknowledging the chassis.

5.    If Nexus FEXes are part of the configuration, expand Rack Mounts and FEX.

6.    Right-click each FEX that is listed and choose Acknowledge FEX.

7.    Click Yes and then click OK to complete acknowledging the FEX.

Create an Organization

Now all items have been deployed at the root level in Cisco UCS Manager. To allow this Cisco UCS to be shared among different projects, UCS Organizations can be created.  In this validation, the organization for this FlexPod deployment is FlexPod. To create an organization for this FlexPod deployment, follow these steps:

1.    In Cisco UCS Manager, click Servers.

2.    In the Navigation Pane, expand Servers > Service Profiles.

3.    Right-click root under Service Profiles and choose Create Organization.

4.    Provide a name for the Organization to indicate this FlexPod deployment and optionally provide a Description.

5.    Click OK then click OK again to complete creating the organization.

Create a WWNN Pool for FC Boot (FCP)

In this FlexPod implementation, a WWNN pool is created at the root organization level to avoid WWNN address pool overlaps. If your deployment plan calls for different WWNN ranges in different UCS organizations, place the WWNN pool at the organizational level. To configure the necessary WWNN pool for the Cisco UCS environment, follow these steps on Cisco UCS Manager:

1.    Choose SAN.

2.    Choose Pools > root.

3.    Right-click WWNN Pools under the root organization.

4.    Choose Create WWNN Pool to create the WWNN pool.

5.    Enter WWNN-Pool for the name of the WWNN pool.

6.    Optional: Enter a description for the WWNN pool.

7.    Choose Sequential for Assignment Order.

8.    Click Next.

9.    Click Add.

10.  Modify the From field as necessary for the UCS Environment

11.  Specify a size of the WWNN block sufficient to support the available server resources. In this example, with the WWNN block modification, a maximum of 128 addresses are available.

12.  Click OK.

13.  Click Finish and click OK to complete creating the WWNN pool.

Create WWPN Pools (FCP)

In this FlexPod implementation, WWPN address pools are created at the root organization level to avoid WWPN address pool overlaps. If your deployment plan calls for different WWPN address ranges in different UCS organizations, place the WWPN pools at the organizational level. To configure the necessary WWPN pools for the Cisco UCS environment, follow these steps:

1.    In Cisco UCS Manager, click SAN.

2.    Choose Pools > root.

3.    Right-click WWPN Pools under the root organization.

4.    Choose Create WWPN Pool to create the WWPN pool.

5.    Enter WWPN-Pool-A for the name of the WWPN pool.

6.    Optional: Enter a description for the WWPN pool.

7.    Choose Sequential for Assignment Order.

8.    Click Next.

9.    Click Add.

10.  Specify a starting WWPN.

11.  Specify a size for the WWPN pool that is sufficient to support the available blade or server resources remembering that servers could have multiple vHBAs and unassociated service profiles could be created. In this example, with the WWPN block modification, a maximum of 128 addresses are available.

12.  Click OK.

13.  Click Finish.

14.  In the confirmation message, click OK.

15.  Right-click WWPN Pools under the root organization.

16.  Choose Create WWPN Pool to create the WWPN pool.

17.  Enter WWPN-Pool-B for the name of the WWPN pool.

18.  Optional: Enter a description for the WWPN pool.

19.  Choose Sequential for Assignment Order.

20.  Click Next.

21.  Click Add.

22.  Specify a starting WWPN.

23.  Specify a size for the WWPN address pool that is sufficient to support the available blade or server resources remembering that servers could have multiple vHBAs and unassociated service profiles could be created. In this example, with the WWPN block modification, a maximum of 128 addresses are available.

24.  Click OK.

25.  Click Finish.

26.  In the confirmation message, click OK.

Create VSANs (FCP)

To configure the necessary virtual storage area networks (VSANs) for the FlexPod Organization in the Cisco UCS environment, follow these steps:

1.    In Cisco UCS Manager, click SAN.

2.    Choose SAN > SAN Cloud.

3.    Right-click VSANs.

4.    Choose Create VSAN.

5.    Enter VSAN-A for the name of the VSAN to be used for Fabric A.

6.    Leave FC Zoning set at Disabled.

7.    Choose Fabric A.

8.    Enter a unique VSAN ID and a corresponding FCoE VLAN ID that matches the configuration in the MDS switch for Fabric A.  It is recommended to use the same ID for both parameters and to use something other than 1.

9.    Click OK and then click OK again.

10.  Under SAN Cloud, right-click VSANs.

11.  Choose Create VSAN.

12.  Enter VSAN-B for the name of the VSAN to be used for Fabric B.

13.  Leave FC Zoning set at Disabled.

14.  Choose Fabric B.

15.  Enter a unique VSAN ID and a corresponding FCoE VLAN ID that matches the configuration in the MDS switch for Fabric B.  It is recommended use the same ID for both parameters and to use something other than 1.

16.  Click OK and then click OK again.

Create FC Uplink Port Channels (FCP)

To create the FC Uplink Port Channels and assign the appropriate VSANs to them for the Cisco UCS environment, follow these steps:

1.    In Cisco UCS Manager, click SAN.

2.    Choose SAN > SAN Cloud.

3.    Expand Fabric A and choose FC Port Channels.

4.    Right-click FC Port Channels and choose Create FC Port Channel.

5.    Set a unique ID for the port channel and provide a unique name for the port channel.

6.    Click Next.

7.    Choose the appropriate Port Channel Admin Speed.

8.    Choose the ports connected to Cisco MDS A and use >> to add them to the port channel.

9.    Click Finish to complete creating the port channel.

10.  Click OK on the confirmation.

11.  Under FC Port-Channels, choose the newly created port channel.

12.  From the drop-down list to choose Fab-A.

13.  Click Save Changes to assign the VSAN.

14.  Click OK.

15.  Under FC Port Channels, expand FC Port-Channel 10. Under FC Port-Channel 11 choose FC Interface 1/1. Enter a User Label to indicate the connectivity on the MDS 9148T switch, such as <mds-A-hostname>:fc1/1. Click Save Changes and OK. Repeat this process for FC Interface 1/2.

16.  Expand Fabric B and choose FC Port Channels.

17.  Right-click FC Port Channels and choose Create FC Port Channel.

18.  Set a unique ID for the port channel and provide a unique name for the port channel.

19.  Click Next.

20.  Choose the ports connected to Cisco MDS B and use >> to add them to the port channel.

21.  Click Finish to complete creating the port channel.

22.  Click OK on the confirmation.

23.  Under FC Port-Channels, choose the newly created port channel.

24.  In the right pane, use the drop-down to choose Fab-B.

25.  Click Save Changes to assign the VSAN.

26.  Click OK.

27.  Under FC Port Channels, expand FC Port-Channel 20. Under FC Port-Channel 20 choose FC Interface 1/1. Enter a User Label to indicate the connectivity on the MDS 9148T switch, such as <mds-B-hostname>:fc1/1. Click Save Changes and OK. Repeat this process for FC Interface 1/2.

Create vHBA Templates (FCP)

To create the necessary virtual host bus adapter (vHBA) templates for the Cisco UCS environment within the HANA organization, follow these steps:

1.    In Cisco UCS Manager, click SAN.

2.    Expand Policies > root > Sub-Organizations > HANA.

3.    Right-click vHBA Templates under the HANA Organization.

4.    Choose Create vHBA Template.

5.    Enter vHBA-A for the vHBA template name.

6.    Keep Fabric A selected.

7.    Leave Redundancy Type set to No Redundancy.

8.    Choose VSAN Fab-A.

9.    Leave Initial Template for the Template Type.

10.   Choose WWPN-Pool-A for the WWPN Pool.

11.  Click OK to create the vHBA template.

12.  Click OK.

13.  Right-click vHBA Templates under the HANA Organization.

14.  Choose Create vHBA Template.

15.  Enter vHBA-B for the vHBA template name.

16.  Choose B for the Fabric ID.

17.  Leave Redundancy Type set to No Redundancy.

18.  Choose VSAN Fab-B.

19.  Leave Initial Template for the Template Type.

20.  Choose WWPN-Pool-B for the WWPN Pool.

21.  Click OK to create the vHBA template.

22.  Click OK.

Create SAN Connectivity Policy (FCP)

To configure the necessary Infrastructure SAN Connectivity Policy within the HANA organization, follow these steps:

1.    In Cisco UCS Manager, click SAN.

2.    Choose SAN > Policies > root > Sub-Organizations > HANA.

3.    Right-click SAN Connectivity Policies under the HANA Organization.

4.    Choose Create SAN Connectivity Policy.

5.    Enter FC-Boot for the name of the policy.

6.    Choose the previously created WWNN-Pool for the WWNN Assignment.

7.    Click Add to add a vHBA.

8.    In the Create vHBA dialog box, enter vHBA-Fab-A for the name of the vHBA.

9.    Choose the Use vHBA Template checkbox.

10.  In the vHBA Template list, choose vHBA-A.

11.  In the Adapter Policy list, choose Linux.

12.  Click OK.

13.  Click Add to add a second vHBA.

14.  In the Create vHBA dialog box, enter vHBA-Fab-B for the name of the vHBA.

15.  Choose the Use vHBA Template checkbox.

16.  In the vHBA Template list, choose vHBA-B.

17.  In the Adapter Policy list, choose Linux.

18.  Click OK.

19.  Click OK to create the SAN Connectivity Policy.

20.  Click OK to confirm creation.

Add Block of IP Addresses for KVM Access

To create a block of IP addresses for in band server Keyboard, Video, Mouse (KVM) access in the Cisco UCS environment, follow these steps:

1.    In Cisco UCS Manager, click LAN.

2.    Expand Pools > root > IP Pools.

3.    Right-click IP Pool ext-mgmt and choose Create Block of IPv4 Addresses.

4.    Enter the starting IP address of the block, number of IP addresses required, and the subnet mask and gateway information. Optionally, enter the Primary and Secondary DNS server addresses.

5.    Click OK to create the block.

6.    Click OK in the confirmation message.

Create Uplink Port Channels to Cisco Nexus Switches

To configure the necessary port channels out of the Cisco UCS environment, follow these steps:

1.    In Cisco UCS Manager, click LAN.

2.    Under LAN > LAN Cloud, expand the Fabric A tree.

3.    Right-click Port Channels under Fabric A.

4.    Choose Create Port Channel.

5.    Enter 41 for the unique ID of the port channel.

6.    Enter Po41-Nexus for the name of the port channel.

7.    Click Next.

8.    Choose the uplink ports connected to the Nexus switches to be added to the port channel.

9.    Click >> to add the ports to the port channel.

10.  Click Finish to create the port channel.

11.  Click OK.

12.  In the navigation pane, under LAN > LAN Cloud > Fabric A > Port Channels, choose Port-Channel 41. Choose Auto for the Admin Speed.

13.  Click Save Changes and OK. After a few minutes, verify that the Overall Status is Up, and the Operational Speed is correct.

14.  In the navigation pane, under LAN > LAN Cloud > Fabric A > Port Channels, expand Port-Channel 41. Under Port-Channel 41, choose Eth Interface 1/49. In the center pane under Properties, enter a User Label to indicate the port connectivity, such as <nexus-a-hostname>:Eth1/33 Click Save Changes and click OK. Repeat steps 1-14 for the remaining uplink ports.

15.  In the navigation pane, under LAN > LAN Cloud, expand the Fabric B tree.

16.  Right-click Port Channels under Fabric B.

17.  Choose Create Port Channel.

18.  Enter 42 for the unique ID of the port channel.

19.  Enter Po42-Nexus for the name of the port channel.

20.  Click Next.

21.  Choose the ports connected to the Nexus switches to be added to the port channel:

22.  Click >> to add the ports to the port channel.

23.  Click Finish to create the port channel.

24.  Click OK.

25.  In the navigation pane, under LAN > LAN Cloud > Fabric B > Port Channels, choose Port-Channel 42. Choose Auto Gbps for the Admin Speed.

26.  Click Save Changes and OK. After a few minutes, verify that the Overall Status is Up, and the Operational Speed is correct.

27.  In the navigation pane, under LAN > LAN Cloud > Fabric A > Port Channels, expand Port-Channel 42. Under Port-Channel 42, choose Eth Interface 1/49. In the center pane under Properties, enter a User Label to indicate the port connectivity, such as <nexus-b-hostname>:Eth1/33 Click Save Changes and click OK. Repeat steps 1-27 for the remaining uplink ports.

Set Jumbo Frames in Cisco UCS Fabric

Jumbo Frames are used in FlexPod for the NFS storage protocols. The normal best practice in FlexPod has been to set the MTU of the Best Effort QoS System Class in Cisco UCS Manager to 9216 for Jumbo Frames. In the Cisco UCS 6454 Fabric Interconnect with UCS Manager version 4.0 software the MTU for the Best Effort QoS System Class is fixed at normal and cannot be changed.  With this setting of normal in the 6454, Jumbo Frames can pass through the Cisco UCS fabric without being dropped. In UCS Manager version 4.1, the MTU for the Best Effort QoS System Class is again settable.  To configure jumbo frames in the UCS fabric, follow these steps:

1.    In Cisco UCS Manager, click LAN.

2.    Expand LAN > LAN Cloud > QoS System Class.

3.    In the right pane, click the General tab.

4.    On the Best Effort row, enter 9216 in the box under the MTU column.

5.    Click Save Changes.

6.    Click OK.

Create VLANs

Within Cisco UCS, all the network types for an SAP HANA system are manifested by defined VLANs. Network design guideline from SAP recommends seven SAP HANA related networks and two infrastructure related networks.

Even though nine VLANs are defined, VLANs for all the networks are not necessary if the solution will not use those networks. For example, if the Replication Network is not used in the solution, then VLAN ID 225 need not be created.

The VLAN IDs can be changed if required to match the VLAN IDs in the customer’s network – for example, ID 222 for backup should match the configured VLAN ID at the customer uplink network switches.

To configure the necessary VLANs for the Cisco UCS environment, follow these steps:

1.    In Cisco UCS Manager, click LAN.

2.    Expand LAN > LAN Cloud.

3.    Right-click VLANs and select Create VLANs.

4.    Enter HANA-Mgmt for the name of the VLAN to be used as HANA node management network.

5.    Keep the Common/Global option selected for the scope of the VLAN.

6.    Enter <var_mgmt_vlan_id> for the ID of the HANA Node-to-Node network.

7.    Keep the Sharing Type as None.

8.    Click OK and then click OK again.

9.    Create VLAN HANA-AppServer using <var_appserver_vlan_id>

10.  Create VLAN HANA-Backup using <var_backup_vlan_id>

11.  Create VLAN HANA-Client using <var_client_vlan_id>.

12.  Create VLAN HANA-DataSource using <var_datasource_vlan_id>.

13.  Create VLAN HANA-Internal Node-to-Node traffic using <var_internal_vlan_id>.

14.  Create VLAN HANA-Replication using <var_replication_vlan_id>.

15.  Create VLAN HANA-sharednfs for /hana/shared NFS network.

Create MAC Address Pools

In this FlexPod deployment, MAC address pools are created at the root organization level to avoid MAC address pool overlaps. If your deployment plan calls for different MAC address ranges in different UCS organizations, place the MAC pools at the organizational level. To configure the necessary MAC address pools for the Cisco UCS environment, follow these steps:

1.    In Cisco UCS Manager, click LAN.

2.    Expand Pools > root.

3.    Right-click MAC Pools under the root organization.

4.    Choose Create MAC Pool to create the MAC address pool.

5.    Enter MAC-Pool-A for the name of the MAC pool.

6.    Optional: Enter a description for the MAC pool.

7.    Choose Sequential for the Assignment Order.

8.    Click Next.

9.    Click Add.

10.  Specify a starting MAC address.

11.  Specify a size for the MAC address pool that is sufficient to support the available blade or server resources remembering that a server may contain multiple vNICs and that multiple unassociated Service Profiles can be created. In this example, with the MAC block modification, a maximum of 256 addresses are available.

12.  Click OK.

13.  Click Finish.

14.  In the confirmation message, click OK.

15.  Right-click MAC Pools under the root organization.

16.  Choose Create MAC Pool to create the MAC address pool.

17.  Enter MAC-Pool-B for the name of the MAC pool.

18.  Optional: Enter a description for the MAC pool.

19.  Choose Sequential for the Assignment Order.

20.  Click Next.

21.  Click Add.

22.  Specify a starting MAC address.

23.  Specify a size for the MAC address pool that is sufficient to support the available blade or server resources remembering that a server may contain multiple vNICs and that multiple unassociated Service Profiles can be created.

24.  Click OK.

25.  Click Finish.

26.  In the confirmation message, click OK.

Create Network Control Policy for Cisco Discovery Protocol (CDP) and Link Layer Discovery Protocol (LLDP)

To create a network control policy that enables CDP and LLDP on the server virtual network controller (vNIC) ports, follow these steps:

1.    In Cisco UCS Manager, click LAN.

2.    Expand Policies > root.

3.    Right-click Network Control Policies.

4.    Choose Create Network Control Policy.

5.    Enter Enable-CDP-LLDP for the policy name.

6.    For CDP, choose the Enabled option.

7.    For LLDP, scroll down and choose Enabled for both Transmit and Receive.

8.    Click OK to create the network control policy.

9.    Click OK.

Create vNIC Templates

Each VLAN is mapped to a vNIC template to specify the characteristic of a specific network To create multiple virtual network interface card (vNIC) templates within the HANA organization, follow these steps.

1.    In Cisco UCS Manager, click LAN.

2.    Expand Policies > root > Sub-Organizations > HANA

3.    Under the HANA Organization, right-click vNIC Templates.

4.    Choose Create vNIC Template.

5.    Enter HANA-Internal for the vNIC template name.

6.    Keep Fabric A selected (alter to Fabric B for the next vNIC Template).

7.    Check the Enable Failover checkbox.

8.    Leave the Peer Redundancy Template set to <not set>.

9.    Under Target, make sure that only the Adapter checkbox is selected.

10.  Choose Updating Template for the Template Type.

11.  Under VLANs, choose the checkboxes for HANA-Internal VLAN.

12.  Enable the native VLAN radio button for the VLAN HANA-Internal VLAN.

13.  Choose vNIC Name for the CDN Source.

14.  For MTU, enter 9000.

15.  In the MAC Pool list, choose MAC-Pool-A (Select FI-B for the next vNIC Template).

16.  In the Network Control Policy list, choose Enable-CDP-LLDP.

17.  Click OK.

18.  Continue to create a vNIC template for each VLAN altering the FI-A and FI-B assignments to distribute the traffic between the Fabrics.

Create vNIC/vHBA Placement Policy

Cisco UCS assigns virtual network interface connections (vCons) to the PCIe adapter cards in the server. Each vCon is a virtual representation of a physical adapter that can be assigned vNICs and vHBAs.

For blade or rack servers that contain one adapter, Cisco UCS assigns all vCons to that adapter.

For blade or rack servers that contain two or three adapters, Cisco UCS assigns the vCons based on the selected virtual slot mapping scheme. This can be one of the following:

●     Round Robin— In a server with two adapter cards, Cisco UCS assigns vCon1 and vCon3 to Adapter1, then assigns vCon2 and vCon4 to Adapter2. In a server with three adapter cards, Cisco UCS assigns vCon1 to Adapter1, vCon2 and vCon4 to Adapter2, and vCon3 to Adapter3. This is the default scheme.

●     Linear Ordered— In a server with two adapter cards, Cisco UCS assigns vCon1 and vCon2 to Adapter1, then assigns vCon3 and vCon4 to Adapter2.In a server with three adapter cards, Cisco UCS assigns vCon1 to Adapter1 and vCon2 to Adapter2, then assigns vCon3 and vCon4 to Adapter3.

To create a vNIC/vHBA placement policy for the SAP HANA hosts, follow these steps:

1.    Find the installed adapters in the system:

2.    Choose a virtual slot mapping scheme and create a placement policy.

3.    In Cisco UCS Manager, click the Servers tab in the navigation pane.

4.    Select Policies > root > Sub-Organization > HANA.

5.    Right-click vNIC/vHBA Placement Policies.

6.    Select Create Placement Policy.

7.    Enter HANA for the name of the placement policy and select Liner Ordered for Virtual Slot Mapping Scheme.

 

For the Placement policy, considering the installed adapters helps to be sure of what vCONs to use for vNIC assignment.

Create LAN Connectivity Policy

With LAN connectivity policy, you need to define the vNICs that a system needs to have in order to cater to the specific networks the use-case demands.

For both SAP HANA scale-up and scale-out system use cases, apart from the admin/management the node may need backup and application server network connections, at a minimum. It could have other networks depending on usage.

However, for the scale-out system scenario, inter-node communication is mandatory and that is what distinguishes it from a single host system’s policy.

In the following steps, you will create separate LAN connectivity policies for the two-standard use cases. This further simplifies the service profile template creation by having the pre-determined vNICs already part of the network configuration.

SAP HANA Scale-Up System Use Case

To configure the necessary LAN Connectivity Policy within the HANA Organization, follow these steps:

1.    In Cisco UCS Manager, click LAN.

2.    Expand LAN > Policies > root > Sub-Organizations > HANA.

3.    Right-click LAN Connectivity Policies.

4.    Select Create LAN Connectivity Policy.

5.    Enter scale-up for the name of the policy.

6.    Click Add to add a vNIC.

7.    In the Create vNIC dialog box, enter Mgmt for the name of the vNIC.

8.    Select the Use vNIC Template checkbox.

9.    In the vNIC Template list, select HANA-Mgmt.

10.  In the Adapter Policy list, select Linux.

11.  Click OK to add this vNIC to the policy.

12.  Click Add to add another vNIC to the policy.

13.  In the Create vNIC box, enter appserver for the name of the vNIC.

14.  Select the Use vNIC Template checkbox.

15.  In the vNIC Template list, select HANA-AppServer.

16.  In the Adapter Policy list, select Linux.

17.  Click OK to add the vNIC to the policy.

18.  Click Add to add another vNIC to the policy.

19.  In the Create vNIC dialog box, enter backup for the name of the vNIC.

20.  Select the Use vNIC Template checkbox.

21.  In the vNIC Template list, select HANA-Backup.

22.  In the Adapter Policy list, select Linux.

23.  Click OK to add this vNIC to the policy.

24.  Click Add to add another vNIC to the policy.

25.  In the Create vNIC dialog box, enter sharednfs for the name of the vNIC.

26.  Select the Use vNIC Template checkbox.

27.  In the vNIC Template list, select HANA-sharednfs.

28.  In the Adapter Policy list, select Linux.

29.  Click OK to add this vNIC to the policy.

30.  Click Add to add a vNIC.

31.  In the Create vNIC dialog box, enter sysrep for the name of the vNIC.

32.  Select the Use vNIC Template checkbox.

33.  In the vNIC Template list, select HANA-Replication.

34.  In the Adapter Policy list, select Linux.

35.  Click OK to add this vNIC to the policy.

SAP HANA Scale-Out System Use Case

To configure the necessary LAN Connectivity Policy within the T01-HANA Organization, follow these steps:

1.    In Cisco UCS Manager, click LAN.

2.    Expand LAN > Policies > root > Sub-Organizations > HANA Organization.

3.    Right-click LAN Connectivity Policies.

4.    Select Create LAN Connectivity Policy.

5.    Enter scale-out for the name of the policy.

6.    Click Add to add a vNIC.

7.    In the Create vNIC dialog box, enter Mgmt  for the name of the vNIC.

8.    Check the Use vNIC Template checkbox.

9.    In the vNIC Template list, select HANA-Mgmt.

10.  In the Adapter Policy list, select Linux.

11.  Click OK to add this vNIC to the policy.

12.  Click Add to add another vNIC to the policy.

13.  In the Create vNIC box, enter internode for the name of the vNIC.

14.  Check the Use vNIC Template checkbox.

15.  In the vNIC Template list, select HANA-Internal.

16.  In the Adapter Policy list, select HANA.

17.  Click OK to add the vNIC to the policy.

18.  Click Add to add a vNIC.

19.  In the Create vNIC dialog box, enter appserver for the name of the vNIC.

20.  Check the Use vNIC Template checkbox.

21.  In the vNIC Template list, select HANA_AppServer.

22.  In the Adapter Policy list, select Linux.

23.  Click OK to add this vNIC to the policy.

24.  Click Add to add a vNIC.

25.  In the Create vNIC dialog box, enter sharednfs for the name of the vNIC.

26.  Check the Use vNIC Template checkbox.

27.  In the vNIC Template list, select HANA-sharednfs.

28.  In the Adapter Policy list, select HANA.

29.  Click OK to add this vNIC to the policy.

30.  Click Add to add a vNIC.

31.  In the Create vNIC dialog box, enter backup for the name of the vNIC.

32.  Check the Use vNIC Template checkbox.

33.  In the vNIC Template list, select HANA-Backup.

34.  In the Adapter Policy list, select Linux.

35.  Click OK to add this vNIC to the policy.

Create UUID Suffix Pool

To configure the necessary universally unique identifier (UUID) suffix pool for the Cisco UCS environment, follow these steps:

1.    In Cisco UCS Manager, click Servers.

2.    Expand Pools > root.

3.    Right-click UUID Suffix Pools.

4.    Choose Create UUID Suffix Pool.

5.    Enter UUID-Pool for the name of the UUID suffix pool.

6.    Optional: Enter a description for the UUID suffix pool.

7.    Keep the prefix at the derived option.

8.    Choose Sequential for the Assignment Order.

9.    Click Next.

10.  Click Add to add a block of UUIDs.

11.  Keep the From field at the default setting.

12.  Specify a size for the UUID block that is sufficient to support the available blade or server resources and the number of Service Profiles that will be created.

13.  Click OK.

14.  Click Finish.

15.  Click OK.

Modify Default Host Firmware Package

Firmware management policies allow the administrator to choose the corresponding packages for a given server configuration. These policies often include packages for adapter, BIOS, board controller, FC adapters, host bus adapter (HBA) option ROM, and storage controller properties.

To modify the default firmware management policy in the Cisco UCS environment, follow these steps:

1.    In Cisco UCS Manager, click Servers.

2.    Expand Policies > root.

3.    Expand Host Firmware Packages.

4.    Choose default.

5.    In the Actions pane, choose Modify Package Versions.

6.    Choose version 4.1(1d) for both the Blade and Rack Packages.

7.    Click OK, then click OK again to modify the host firmware package.

Create Local Disk Configuration Policy (Optional)

A local disk configuration specifying no local disks for the Cisco UCS environment can be used to ensure that servers with no local disks are used for SAN Boot. local disks.

To create a local disk configuration policy, follow these steps:

1.    In Cisco UCS Manager, click Servers.

2.    Expand Policies > root.

3.    Right-click Local Disk Config Policies.

4.    Choose Create Local Disk Configuration Policy.

5.    Enter SAN-Boot for the local disk configuration policy name.

6.    Change the mode to No Local Storage.

7.    Click OK to create the local disk configuration policy.

8.    Click OK.

Create Power Control Policy

To create a power control policy for the Cisco UCS environment, follow these steps:

1.    In Cisco UCS Manager, click Servers.

2.    Expand Policies > root.

3.    Right-click Power Control Policies.

4.    Choose Create Power Control Policy.

5.    Enter No-Power-Cap for the power control policy name.

6.    Change the power capping setting to No Cap.

7.    Click OK to create the power control policy.

8.    Click OK.

 Update the Default Maintenance Policy

To update the default Maintenance Policy to either require user acknowledgement before server boot when service profiles change or to make the changes on the next server reboot, follow these steps:

1.    In Cisco UCS Manager, click Servers.

2.    Expand Policies > root.

3.    Choose Maintenance Policies > default.

4.    Change the Reboot Policy to User Ack.

5.    Choose “On Next Boot” to delegate maintenance windows to server administrators.

6.    Click Save Changes.

7.    Click OK to accept the changes.

Create Server BIOS Policy

Best performance of the SAP HANA environment requires to configure the Server BIOS accurately. Create a server BIOS policy for the Cisco UCS environment:

1.    In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.    Select Policies > root > Sub-Organization > HANA > BIOS Policies.

3.    Right-click BIOS Policies and select Create BIOS Policy.

4.    Enter HANA-BIOS as BIOS policy name.

5.    Select Reboot on BIOS Settings Change.

6.    Click OK and confirm the new BIOS Policy with OK.

7.    Select the HANA-BIOS policy in the navigation pane.

8.    On the Main sub tab, change the Quiet Boot setting from Platform Default to Disabled.

9.    Click the Advanced tab.

10.  Click the Processor tab and change CPU Performance from Platform Default to Enterprise:

a.     Keep processor C State on platform default.

b.    Keep Processor C1E on Platform Default.

c.     Change Processor C3, C6 and C7 Report to disabled.

d.    Change Power Technology from Platform Default to Performance.

e.    Change Energy Performance from Platform Default to Performance.

11.  In the RAS Memory tab change the LV DDR Mode to performance mode and enable NUMA. Keep the memory RAS configuration on platform default.

12.  Enable Serial Port A in the Serial Port tab.

13.  In the Server Management tab, configure the Console Redirection to serial-port-a with the BAUD Rate 115200 and enable the feature Legacy OS redirect. This is used for Serial Console Access over LAN to all SAP HANA servers.

14.  Click Save Changes to update the BIOS Policy.

15.  Click OK.

Create FC Boot Policy (FCP)

This procedure applies to a Cisco UCS environment in which at least two Fibre Channel logical interfaces (LIFs) are on cluster node 1 (fcp-1a and fcp-1b) and two Fibre Channel LIFs are on cluster node 2 (fcp-2a and fcp-2b). Also, it is assumed that the A LIFs are connected to Fabric A (Cisco UCS Fabric Interconnect A) and the B LIFs are connected to Fabric B (Cisco UCS Fabric Interconnect B).

To create the FC boot policy, follow these steps:

1.    Make sure the NetApp array for the LIFs is configured for infra SVM:

2.    Make a note of the FC ports and their physical WWPN IDs from the ONTAP system manager Network > FC Ports page, as shown below:

3.    Log into MDS A and B and verify the fabric logins:

4.    Based on the port mappings, make a reference connection diagram for easy visualization as shown below:

We chose the WWPNs corresponding to infra SVM for the boot policy as targets. In this example, we used LIFs fcp_1_c and fcp_2_c, such as 20:07:d0:39:ea:18:ba:5a and 20:03:d0:39:ea:18:ba:5a for SAN Primary and Secondary targets. LIFs fcp_2_a and fcp_1_a, such as 20:01:d0:39:ea:18:ba:5a and 20:05:d0:39:ea:18:ba:5a are used as SAN Secondary’s primary and secondary targets.

To create a boot policy for the within the HANA organization, follow these steps:

1.    In Cisco UCS Manager, click Servers.

2.    Expand Policies > root > Sub-Organizations > HANA.

3.    Under the HANA Organization, right-click Boot Policies.

4.    Choose Create Boot Policy.

5.    Enter Boot-FCP for the name of the boot policy.

6.    Optional: Enter a description for the boot policy.

7.    Do not select the Reboot on Boot Order Change checkbox.

8.    Choose the Uefi Boot Mode.

9.    Do not choose the Boot Security checkbox.

10.  Expand Local Devices and choose Add Remote CD/DVD.

11.  Expand vHBAs and choose Add SAN Boot.

12.  Choose Primary for the type field.

13.  Enter vHBA-Fab-A in the vHBA field.

14.  Click OK.

15.  From vHBAs, choose Add SAN Boot Target.

16.  Keep 0 for the value for Boot Target LUN.

17.  Enter the WWPN for LIF fcp_1_c.

18.  Choose Primary for the SAN boot target type.

19.  Click OK to add the SAN boot target.

20.  From vHBAs, choose Add SAN Boot Target.

21.  Enter 0 for the value for Boot Target LUN.

22.  Enter the WWPN for LIF fcp_2_c.

23.  Click OK to add the SAN boot target.

24.  Add SAN Secondary. Add its primary and secondary targets. We chose controllers fcp_1_a and fcp_2_a for this deployment.

25.  From vHBAs, choose Add SAN Boot.

26.  In the Add SAN Boot dialog box, enter FCP-Fabric-B in the vHBA box.

27.  The SAN boot type should automatically be set to Secondary.

28.  Click OK.

29.  From vHBAs, choose Add SAN Boot Target.

30.  Keep 0 for the value for Boot Target LUN.

31.  Enter the WWPN for LIF fcp_2_a.

32.  Choose Primary for the SAN boot target type.

33.  Click OK to add the SAN boot target.

34.  From vHBAs, choose Add SAN Boot Target.

35.  Keep 0 for the value for Boot Target LUN.

36.  Enter the WWPN for LIF fcp_1_a.

37.  Click OK to add the SAN boot target.

38.  Click OK, then click OK again to create the boot policy.

Create Service Profile Template (FCP)

In this procedure, one service profile template for HANA nodes is created with FC boot within the HANA organization. The steps to create service profile template to instantiate HANA nodes for SAP HANA scale-up or scale-out use case depends on the LAN connectivity policy you select and corresponding placement of vNICs per vCONs.

To create the service profile template, follow these steps:

1.    In Cisco UCS Manager, click Servers.

2.    Expand Service Profile Templates > root > Sub-Organizations > HANA.

3.    Right-click the HANA Organization.

4.    Choose Create Service Profile Template to open the Create Service Profile Template wizard.

5.    Enter HANA-node for the name of the service profile template.

6.    Choose the Updating Template option.

7.    Under UUID, choose UUID-Pool for the UUID pool.

8.    Click Next.

Configure Storage Provisioning

To configure storage provisioning, follow these steps:

1.    If you have servers with no physical disks, click the Local Disk Configuration Policy tab and choose the SAN-Boot Local Storage Policy. Otherwise, choose the default Local Storage Policy.

2.    Click Next.

Configure Networking

To configure the network options for HANA node intended to be scale-up system, follow these steps:

1.    Keep the default setting for Dynamic vNIC Connection Policy.

2.    Choose the “Use Connectivity Policy” option to configure the LAN connectivity.

3.    Choose ‘scale-up’ from the LAN Connectivity Policy drop-down list.

 

4.    Click Next.

OR

To configure the network options for the HANA node that is intended to be part of the scale-out cluster, follow these steps:

1.    Keep the default setting for Dynamic vNIC Connection Policy.

2.    Select the “Use Connectivity Policy” option to configure the LAN connectivity.

3.    Select scale-out from the LAN Connectivity Policy drop-down list.

4.    Click Next.

Configure SAN Connectivity

To configure SAN connectivity, follow these steps:

1.    Choose the Use Connectivity Policy option for the “How would you like to configure SAN connectivity?” field.

2.    Choose the FC-Boot option from the SAN Connectivity Policy drop-down list.

3.    Click Next.

Configure Zoning

To configure zoning, follow this step:

1.    Set no zoning options and click Next.

Configure vNIC/vHBA Placement

To configure the vNIC/vHBA placement in case of Scale-up system, follow these steps:

1.    In the “Select Placement” list, Select HANA.

2.    Manually assign the vNICs to vCon1 and vCon3 as below:

3.    Click Next.

To configure the vNIC/HBA placement for a scale-out system, follow these steps:

1.    In the “Select Placement” list, choose HANA.

2.    Manually assign the vNICs to vCon1 and vCon3 as below:

3.    Click Next.

Configure vMedia Policy

To configure the vMedia policy, follow these steps:

1.    Do not select a vMedia Policy.

2.    Click Next.

Configure Server Boot Order

To configure the server boot order, follow these steps:

1.    Choose FC-Boot for Boot Policy.

2.    Click Next.

Configure Maintenance Policy

To configure the maintenance policy, follow these steps:

1.    Change the Maintenance Policy to default.

 

2.    Click Next.

Configure Server Assignment

To configure the server assignment, follow this step:

1.    Select defaults with Server assignment and Firmware Management. Click Next.

Configure Operational Policies

To configure the operational policies, follow these steps:

1.    In the BIOS Policy list, choose HANA-BIOS.

2.    Expand Management IP Address. For Outband IPV4 select ext-mgmt from the drop-down list.

3.    Expand Power Control Policy Configuration and choose No-Power-Cap in the Power Control Policy list.

 

4.    Click Finish to create the service profile template.

5.    Click OK in the confirmation message.

Create Service Profiles

Depending on the LAN connectivity policy you configured while creating the Service Profile Template, to create the Service Profiles for the number of nodes you need, follow these steps:

1.    Connect to UCS Manager and click Servers.

2.    Choose Service Profile Templates > root > Sub-Organizations > HANA > Service Template HANA-node.

3.    Right-click HANA-node and choose Create Service Profiles from Template.

4.    Enter HANA-Scaleup-0 for the service profile prefix (assuming the scale-up LAN connectivity was used, and you need to generate a service profile for a single scale-up node).

5.    Enter 1 for the “Name Suffix Starting Number.”

6.    Enter 1 for the “Number of Instances.”

7.    Click OK to create the service profiles.

8.    Click OK in the confirmation message.

SAN Switch Configuration

This section explains how to configure the Cisco MDS 9000s for use in a FlexPod environment. Follow the steps precisely because failure to do so could result in an improper configuration.

If directly connecting storage to the Cisco UCS fabric interconnects, skip this section.

Follow the physical connectivity guidelines for FlexPod as explained in the section FlexPod Cabling.

It’s useful to prepare a connectivity diagram of the FC setup for quick reference. The Cisco MDS specific connectivity in the validation setup is as shown below (the diagram shows physical and it’s infra-svm LIFs of NetApp array):

The procedures in this section describe how to configure the Cisco Nexus switches for use in a base FlexPod environment. This procedure assumes you are using the Cisco MDS 9148T with NX-OS 8.4(1).

Cisco MDS 9148T A

To set up the initial configuration for the Cisco MDS A switch, <mds-A-hostname>, follow these steps:

1.    Configure the switch using the command line.

2.    Review the configuration.

Cisco MDS 9148T B

To set up the initial configuration for the Cisco MDS B switch, <mds-B-hostname>, follow these steps:

On initial boot and connection to the serial or console port of the switch, the NX-OS setup should automatically start and attempt to enter Power on Auto Provisioning. Enter y to get to the System Admin Account Setup.

1.    Configure the switch using the command line.

2.    Review the configuration.

FlexPod Cisco MDS Switch Configuration

Cisco MDS 9148T A and Cisco MDS 9148T B

To enable the correct features on the Cisco MDS switches, follow these steps:

1.    Log in as admin.

2.    Run the following commands:

Configure Individual Ports

Cisco MDS 9148T A

To configure individual ports and port-channels for switch A, follow this step:

1.    From the global configuration mode, run the following commands:

Cisco MDS 9148T B

To configure individual ports and port-channels for switch B, follow this step:

1.    From the global configuration mode, run the following commands:

Create VSANs

Cisco MDS 9148T A

To create the necessary VSANs for fabric A and add ports to them, follow this step:

1.    From the global configuration mode, run the following commands:

Cisco MDS 9148T B

To create the necessary VSANs for fabric B and add ports to them, follow this step:

1.    From the global configuration mode, run the following commands:

Create Device Aliases

Cisco MDS 9148T A

To create device aliases for Fabric A that will be used to create zones, follow these steps:

1.    LIFs information can be acquired from the array configuration.

2.    From the global configuration mode, run the following commands:

Cisco MDS 9148T B

To create device aliases for Fabric B that will be used to create zones, this step:

1.    From the global configuration mode, run the following commands:

Create Zones and Zoneset

To create the infra-svm specific zone that is used solely for booting purpose follow these steps:

Cisco MDS 9148T A

To create the required zones and zoneset on Fabric A, run the following commands:

1.    LIFs information can be acquired from the array configuration.

Cisco MDS 9148T B

To create the required zones and zoneset on Fabric B, run the following commands:

Create hana-svm Specific Zone for HANA Persistence LUNs Presentation

To create the hana-svm specific zone that is used solely for presentation of HANA persistence LUNs to designated hosts, you need the information of the logical ports it is connected to and the LIF information. The connectivity diagram helps map the LIFs corresponding to the hana-svm. The LIF information is shown in the array command line:

Cisco MDS 9148T A

To add device aliases for LIFs specific to hana-svm for Fabric A that will be used to create zone, follow this step:

1.    From the global configuration mode, run the following commands:

Cisco MDS 9148T A

To create the required zones and zoneset on Fabric A, run the following commands:

Cisco MDS 9148T B

To add device aliases for LIFs specific to hana-svm for Fabric B that will be used to create zone, follow this step:

1.    From the global configuration mode, run the following commands:

To create the required zone and zoneset on Fabric B, run the following commands:

HANA Node Preparation

This section details the preparation of HANA nodes based on SLES for SAP Applications 15 SP2 and RHEL 8.1 and provides the Linux Operating System installation procedure using SAN Boot and includes operating system customization to fulfill all SAP HANA requirements. If you plan to install Red Hat Enterprise Linux for SAP Solutions skip the first SUSE Linux Enterprise Server for SAP Applications installation section. 

SLES for SAP Applications 15 SP2

SLES for SAP 15 Installation

SUSE® Linux Enterprise Server for SAP Applications is the reference platform for the software development of SAP. It is optimized for SAP applications like SAP HANA. The installation follows the installation workflow documented in chapter 3.1 of https://documentation.suse.com/sles-sap/15-SP2/html/SLES-SAP-guide/index.html and this section lists where the lab installation deviates from the installation workflow.

The supplement SUSE information available from the SAP notes system is as follows:

●     SAP Note 2578899 - SUSE Linux Enterprise Server 15: Installation Note

●     SAP Note 2684254 - SAP HANA DB: Recommended OS settings for SLES 15 for SAP Applications 15

●     SAP Note 1275776 - Linux: Preparing SLES for SAP environments

Download the ISO image from https://download.suse.com.

To map the installation ISO image in the KVM console, follow these steps:

1.    In the Navigation pane of the Cisco UCS Manager, click Servers.

2.    Select Service Profile > root > Sub-Organization > HANA > HANA-Server01.

3.    In the Actions section click KVM Console.

4.    Choose Virtual Media > Activate Virtual Devices.

5.    For Unencrypted Virtual Media Session, select Accept this Session and then click Apply.

6.    Click Virtual Media and choose Map CD/DVD.

7.    Click Browse to navigate to the ISO media location. Select SLE-15-SP2-Full-x86_64-GM-Media1..SO and click Open.

8.    Click Map Device.

9.    In the KVM Console menu, click Boot Server.

10.  During the VIC FC boot driver verification at the server boot time the NetApp array target WWPN numbers are listed during the connection verification.

11.  The System will automatically boot from the ISO image into the installation wizard.

12.  Follow the SUSE Linux Enterprise installation workflow and choose SUSE Linux Enterprise Server for SAP Applications 15 SP2.

13.  Language, Keyboard and Product Selection – Choose the desired Language and Keyboard layout. Select SLES for SAP Applications 15 SP2 for Product to Install. Click Next.

14.  On the Agreement page, tick the agree checkbox and click Next.

15.  On the network settings screen configure the management network interface:

16.  Identify the Ethernet device to vNIC interface mapping first from the Cisco UCS Manager:

a.     In the Navigation pane of UCSM, click Servers.

b.    Select Service Profile > root > Sub-Organizations > HANA > HANA-Server01.

c.     In the network tab, scroll down to the vNIC section and list the vNICs with their MAC addresses.

d.    Note down the MAC address of the HANA-Mgmt vNIC, in this lab installation “00:25:B5:FD:AA:01”

17.  In the SUSE network settings screen, find the network interface with the same MAC address, in this lab installation eth2 and click Edit.

18.  Provide the IP address <var_server01_mgmt_ip>, the subnet mask <var_oob_vlan_net> and the fully qualified host name <var_server01_mgmt_hostname> in the General tab. Click Next.

19.  Select the Hostname/DNS tab.

20.  Provide the server hostname: <var_server01_hostname>

21.  Change the Set Hostname from the DHCP drop-down list to no.

22.  Enter the name server IP: <var_nameserver_ip>

23.  Enter the domain name in the domain search field: <var_dns_domain_name>

24.  Choose the Routing table.

25.  Enter the default IPv4 Gateway IP address: <var_os_default_IPv4_gateway> and change the device field to:

26.  Configure the IP addresses for the rest of networks configured.

27.  Click Next and continue with the SUSE Linux Enterprise installation workflow.

28.  During the System Probing stage, for the  pop-up “The system seems to have multipath hardware. Do you want to activate multipath?” click Yes.

29.  Skip the registration for now. Click OK and then click Next.

30.  Extension and Module Selection. Choose as shown below and click Next.

31.  Add-On Product Installation – verify the selection and click Next.

32.  System Role – Select SLES for SAP Applications.

33.  Choose OS Edition – uncheck both options. Click Next.

34.  Suggested Partitioning:

a.     On the Suggested Partitioning dialog select Expert Partitioner and start with the current proposal.

b.    Under Hostname > Volume Management delete all volumes.

c.     Under Hostname > Hard Disks,  look for the 100G NetApp device for the boot LUN. Select the multipath device. The Used devices tab list the NetApp LUN scsi devices.

d.    Click the middle Partitions tab. Under Partition Table select Create new partition table.

e.    Choose GPT as new Partition Table Type and click Next.

f.      Change to the Partitions tab and click Add Partition.

g.    Choose 0.5 GiB as new partition custom size and click Next.

h.     Choose the Role EFI Boot Partition and click Next.

i.      Keep the default selections -> Format the device file system in format type FAT and mount the device mount point /boot/efi. Click Next.

j.      Create another partition and click Add Partition.

k.     Choose Custom Size of 2GiB and Click Next. Select Swap role and click Next. Accept the default formatting options.

l.      Create another partition and click Add Partition.

m.   Allocate the remaining maximum size (97.49 GiB) and click Next.

n.     Choose the role Operating System and click Next.

o.    For the Formatting Options, choose appropriate filesystem for root filesystem and Mount Device as / [root filesystem]. XFS was used in the validation setup.

p.    Accept the changes and leave the expert partitioner. Suggested Partitioning page now summarizes the configuration.

q.    Click Next to accept the suggested partitioning and continue with the SUSE Linux Enterprise installation workflow.

35.  Set the Clock and Timezone as required.

36.  Provide the system administrator root password <var_os_root_pw> and click Next.

37.  Several customization steps are recommended from the Installation Settings screen.

38.  Click Software to apply the following changes:

a.     Deselect GNOME Desktop Environment.

b.    Select Fonts.

c.     Select X Window System.

d.    (Optional)Select Enhanced Base System.

e.    Select SAP HANA Server Base.

f.      Deselect primary function SAP Application Sever Base.

g.    (Optional) Select primary function high availability.

39.  Under Security > Firewall will be enabled click disable.

40.  Click Kdump to disable kdump.

41.  Set Default system target to text mode.

42.  Click Install and select Install again on the Confirmation pop-up.

43.  The server will reboot automatically to finish the installation. Click abort on the post-install configuration wizard.

SLES for SAP 15 Post Installation

Apply the post installation steps to prepare the operating system for SAP HANA installation. Connect to the SSH server terminal.

Network Interface Configuration

To configure the network interface, follow these steps:

1.    Configure any remaining network interfaces.

2.    In the SUSE network settings screen, find the network interface with the same MAC address and click edit to provide the appropriate IP address matching to the correct VLAN and provide a fully qualified hostname.

3.    Verify all interfaces come up successfully.

4.    Verify the default gateway setting.

5.    Perform a network service restart to enact the updates relating to IP settings. With admin IP configuration, you would be able to SSH to the host.

6.    Update the /etc/hosts with IP address of all networks and their alias hostnames:

Proxy Configuration

To configure the proxy, follow this step:

1.    Enter and test your proxy configuration.

SLES System Update

To update the SLES system, follow these steps:

1.    Register the system with SUSE to get the latest patches. The system must have access to the Internet to proceed with this step.

2.    Update the system with the following command.

3.    Follow the on-screen instructions to complete the update process. Reboot the server, as suggested, and log in to the system again.

4.    Reboot the system.

Implement SAP Notes Recommendations

To optimize the HANA DB with SLES for SAP 15 SP2, follow the instructions in the SAP Note 2684254:

1.    SAP Note 1275776 describes how to apply recommended operating system settings for running SAP applications on SLES. There are three ways to implement the same – sapconf, sapture or manually. It is important to note when using sapconf or saptune, verify that parameters handled by these tools  are not configured elsewhere (for example, boot parameter, sysctl.conf, and so on). This can cause inconsistent system behavior and makes debugging very difficult.

2.    This CVD uses the saptune (version 2) method which can prepare the operating system for SAP applications based on implementing specific SAP Notes.

3.    Install saptune:

4.    Configuration – activate saptune:

5.    All available solutions and notes can be listed with:

6.    Apply SAP HANA solution/notes:

7.    Verify the solution applied:

Update Cisco fnic/enic Drivers

Based on the serer type/model, processor version, OS release and version information download the firmware bundle corresponding to the UCS Server firmware installed from the Cisco UCS Hardware and Software Compatibility site.

To extract the rpm files of the fnic and enic driver from the driver bundle and copy them to the server, follow these steps:

1.    Verify the current driver:

2.    RPM install the drivers:

3.    Reboot the server:

Enable System Monitoring (Optional)

To enable the system monitoring, follow these steps:

1.    Enable system utilization monitoring:

2.    (Optional) Install rsyslog to bring the /var/log/messages file back:

Persistent Memory Configuration

Configure and manage Intel Optane DC PMM from the command line with the ipmctl and ndctl utilities. The tools are not installed by default but are required to manage the libnvdimm (non-volatile memory device) sub-system in the Linux kernel.

To install the host tools, open an SSH prompt as root and follow these steps:

1.    Install the ipmctl host utility:

2.    Install the ndctl utility:

3.    Verify the persistent memory modules have been discovered and the software can communicate with them:

4.    Add a UDEV rule:

5.    Create the goal:

6.    Confirm with Y and reboot the server to apply the new memory allocations.

7.    Verify the regions have been created:

8.    Create a name space for each region; on a 4-socket server invoke the command four times:

9.    Verify the namespace have been created successfully:

10.  Construct an XFS file system on the block devices:

11.  Create directories and mount the block devices using the DAX file system option:

12.  Change the permission of the mount points:

13.  Finally add the mount points to /etc/fstab to persist them:

14.  The device names chosen by the kernel are subject to creation order and discovery. For static configuration they usually don’t change, alternatively consider using persistent naming instead to mount the pmem namespace.

The persistent name for a pmem namespace in /etc/fstab will look like the following:

RHEL 8.1

Red Hat Enterprise Linux 8 (RHEL) is the reference platform for the software deployment of SAP. It is optimized for SAP applications like SAP HANA. Install the operating system as described in the standard RHEL installation guide. This section lists where the lab installation deviates from the installation workflow.

The following is the supplement RHEL information available from the SAP notes system:

●     SAP Note 2526952 - Red Hat Enterprise Linux for SAP Solutions

●     SAP Note 2772999 - Red Hat Enterprise Linux 8.x: Installation and Configuration

●     SAP Note 2777782 - SAP HANA DB: Recommended OS Settings for RHEL 8

RHEL 8.1 Installation

Download the standard RHEL ISO image from https://access.redhat.com/downloads. To map the installation ISO image in the KVM console, follow these steps:

1.    In the Navigation pane of the Cisco UCS Manager, click Servers.

2.    Select Service Profile > root > Sub-Organization > HANA > HANA-Server01.

3.    In the Actions section click KVM Console.

4.    Choose Virtual Media > Activate Virtual Devices.

5.    For Unencrypted Virtual Media Session, select Accept this Session and then click Apply.

6.    Click Virtual Media and choose Map CD/DVD.

7.    Click Browse to navigate to the ISO media location. Select rhel-8.1-x86_64.ISO. Click Open.

8.    Click Map Device.

9.    In the KVM Console menu, click Boot Server.

10.  During the VIC FC boot driver verification at the server boot time the FlexPod target WWPN numbers are listed during the connection verification.

11.  The system will automatically boot from the ISO image into the installation wizard.

12.  Select Install Red Hat Enterprise Linux 8.1.0 to start the interactive installation process using the server base installation option.

13.  At the welcome screen select the language and click Continue.

14.  The installation summary page appears. Complete all items before starting the installation.

15.  Click Software Selection and use the “Server” Base Environment. No Add-Ons are required during installation. Click Done to return to the main screen.

16.  Click Time & Date. Select the timezone of your choice and ensure the date and time are set correct.

17.  Click Installation Destination and Add a disk.

18.  Choose the 100 GB boot LUN created before. Click Done.

19.  Change the radio button Storage Configuration to Custom. Click Done.

20.  Change the selection to Standard Partition for “New mount points will use the following partition scheme:” and Click the link to create the file system layout automatically.

21.  Delete the home filesystem from the list pressing the minus (-) button.

22.  Select the root volume and resize to 94.4 GiB. Click Done.

23.  Uncheck Enable KDump.

24.  Select Network & Host Name:

a.     Enter a short host name (cishana01) and click Apply.

b.    Identify the Ethernet device to vNIC interface mapping first from the Cisco UCS Manager.

              i.    In the Navigation pane of UCSM, click Servers.

             ii.    Select Service Profile > root > Sub-Organizations > HANA > HANA-Server01.

            iii.    In the network tab, scroll down to the vNIC section and list the vNICs with their MAC addresses.

c.     Compare the Ethernet hardware addresses and configure the network interfaces.

d.    Switch the interfaces On and click Done. Repeat the above steps of network configuration for the remaining interfaces.

25.  Select System Purpose Role: Red Hat Enterprise Linux Server and the appropriate SLA and usage information. Click Done.

26.  Click Begin Installation and provide a root password while the installation is running in the background. Set the Root user password.

27.  Before rebooting, unmap the ISO image from the KVM console.

RHEL System Post Installation

Configure the Network

Verify the network configuration; that all networks have been assigned the IP addresses and that they are up. Ensure the network devices configured during installation will be enabled during boot. The following command will change the ONBOOT variable in line 15 of the network configuration file to yes. Verify the successful change. If needed, restart the network to effect any changes and verify the interfaces are up.

To configure the network, follow these steps:

1.    Update the hostname with hostnamectl command:

2.    Disable Firewall:

3.    Disable SELinux:

4.    Update the /etc/hosts with IP address of all networks and their alias hostnames:

5.    Add any additional full qualified hostname to the /etc/hosts file as well, like for the client, datasource or internode network based on the use-case.

Review Network Time and Date Configuration

To review the network time and date configuration, follow these steps:

1.    During the installation a local NTP server was configured. Review the configuration is working:

2.    Restart the chronyd service:

3.    Validate the service is running and connected to the local time server:

4.    Perform a system reboot to effect SELinux setting:

Update the Red Hat System

To update the Red Hat system, follow these steps:

1.    Configure a proxy for the subscription-manager:

2.    Configure a proxy for the rhn-register and up2date services:

Refer to the Red Hat KB article https://access.redhat.com/solutions/65300 for more information about how to prepare the system to access the Red Hat Subscription Manager through proxy.

To register the system to the Red Hat portal and attach the SAP HANA subscription, follow the instructions in the Red Hat KB article https://access.redhat.com/solutions/4714781.

To update the Red Hat System, follow these steps:

1.    Set the release to the minor release and clear the yum cache and subscribe to the channels:

 

 

 

2.    Check for the available repositories:

3.    Update all packages (including kernel and glibc) to the latest version available in the official RHEL 8 repositories after the initial OS installation:

4.    Install other additional required packages required for running SAP HANA on RHEL 8:

5.    Reboot the system to effect kernel switch.

6.    Python configuration:

Optimize the System for SAP HANA Database

To optimize the system for the SAP HANA database, follow these steps:

7.    Configure tuned to use profile “sap-hana:”

a.     Install and activate the tuned profile "sap-hana" and check if it is active. With sap-hana profile, the THP is disabled, CPU Governor as well as EPB are set to performance.

b.    Install tuned-profiles-sap-hana:

c.     Start and enable the tuned:

d.    Apply the profile for sap-hana:

e.    Verify the solution applied:

8.    Disable ABRT, Core Dumps and kdump:

9.    To disable the core file creation for all users, open or create the file /etc/security/limits.d/99-sap.conf and add the following lines:

10.  Turn off auto-numa balancing. In RHEL 8, auto-numa balancing is switched off by default. To verify, use the following command:

11.  To disable automatic NUMA balancing, use the following command:

12.  If package numad is installed on your server, please make sure that it is switched off and disabled:

13.  Configure C-Stares for lower latency. Append processor.max_cstate=1 intel_idle.max_cstate=1 to the Kernel command line starting with GRUB_CMDLINE_LINUX in /etc/default/grub.

14.  Update the GRUB2 configuration file:

15.  Increase kernel.pid_max -> Add the following line to /etc/sysctl.d/sap.conf (create the file if it doesn't already exist): kernel.pid_max=4194304

16.  Reboot the system to affect this parameter change.

Update Cisco fnic/enic Drivers

Based on the server type/model, processor version, OS release, and version information, download the firmware bundle corresponding to the Cisco UCS Server firmware installed from the Cisco UCS Hardware and Software Compatibility site.

To extract the rpm files of the fnic and enic driver from the driver bundle and copy them to the server, follow these steps:

1.    Verify the current driver:

2.    RPM install the drivers:

3.    Reboot the server.

4.    Verify the driver installation after the reboot:

Persistent Memory Configuration

To configure and manage the Intel Optane DC PMM from the command line with the ipmctl and ndctl utilities, follow these steps. The tools are not installed by default but are required to manage the libnvdimm (non-volatile memory device) sub-system in the Linux kernel.

1.    Open an SSH prompt as root to install the host tools.

2.    EPEL packages assume that the 'codeready-builder' repository is enabled.

3.    Enable the EPEL 8 repository or download the required rpm file from https://dl.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/.

4.    Install the ndctl utility:

5.    Verify the persistent memory modules have been discovered and the software can communicate with them:

6.    Add a UDEV rule:

7.    Create the goal:

8.    Confirm with Y and reboot the server to apply the new memory allocations.

9.    Verify regions had been created:

10.  Create a name space for each region; on a 4-socket server invoke the command four times:

11.  Verify the namespace have been created successfully:

12.  Construct an XFS file system on the block devices:

13.  Create directories and mount the block devices using the DAX file system option:

14.  Change the permission of the mount points:

15.  Add the mount points to /etc/fstab to persist them:

The persistent name for pmem namespace 0 in /etc/fstab will look like the following:

System Provisioning for SAP HANA

This section describes the sequence of steps required to provision nodes for SAP HANA installation starting with the storage volume configuration, LUN configuration, OS configuration needed to mount the storage LUNS and subsequent use-case specific preparation tasks. The underlying infrastructure configuration has already been defined in the previous sections of this document.

Table 9 lists the required variables used in this section.

Table 9.          Required Variables

Each SAP HANA host uses at least two FCP ports for the data and log LUNs of the SAP HANA system and has a network interfaces connected to the storage network, which is used for the SAP HANA shared volume in case of a Scale-out SAP HANA system. The LUNs must be distributed to the storage nodes, as shown in Figure 5, so that a maximum of eight data and eight log volumes are stored on a single storage node.

The limitation of having six SAP HANA hosts per storage node is only valid for production SAP HANA systems for which the storage-performance key performance indicators defined by SAP must be fulfilled. For nonproduction SAP HANA systems, the maximum number is higher and must be determined during the sizing process.

Configure SAP HANA Scale-up Systems

Figure 5 shows the volume configuration of four scale-up SAP HANA systems. The data and log volumes of each SAP HANA system are distributed to different storage controllers. For example, volume SID1_data_mnt00001 is configured on controller A and volume SID1_log_mnt00001 is configured on controller B. Within each volume, a single LUN is configured.

Figure 5.       Volume Layout for SAP HANA Scale-up Systems

For each SAP HANA host, a data volume, a log volume, and a volume for /hana/shared are configured. Table 10 lists an example configuration with four SAP HANA scale-up systems.

Table 10.       Volume Configuration for SAP HANA Scale-up Systems

Table 11 lists an example mount point configuration for a scale-up system.

Table 11.       Mount Points for Scale-up Systems

Configuration Example for an SAP HANA Scale-up System

The following examples show an SAP HANA database with SID=FCS and a server RAM size of 1TB. For different server RAM sizes, the required volume sizes are different.

For a detailed description of the capacity requirements for SAP HANA, see the SAP HANA Storage Requirements white paper.

Create Volumes and Adjust Volume Options

To create a data, log, and shared volume and adjust the volume options, run the following commands:

Create the LUNs

To create a data, log and shared LUNS run the following commands:

Map the LUNs to the hosts

To create a map the LUNs to the host server-01, run the following commands:

Update the Load-Sharing Mirror Relation

To update the load-sharing mirror relation, run the following command:

Host Setup

Before setting up the host, NetApp SAN host utilities must be downloaded from the NetApp Support site and installed on the HANA servers. The host utility documentation includes information about additional software that must be installed depending on the FCP HBA used.

The documentation also contains information on multipath configurations that are specific to the Linux version used. This document covers the required configuration steps for SLES 12 SP1 or higher and RHEL 7.2 or later, as described in the Linux Host Utilities 7.1 Installation and Setup Guide.

Configure Multipathing

To configure multipathing, follow these steps:

1.    Run the Linux rescan-scsi-bus.sh -a command on each server to discover new LUNs.

2.    Run the sanlun lun show command and verify that all required LUNs are visible. The following example shows the sanlun lun show command output for scale-up HANA system with one data LUN, one LOG LUN, and a LUN for /hana/shared. The output shows the LUNs and the corresponding device files, such as LUN FCS_data_mnt00001 and the device file /dev/sdag Each LUN has eight FC paths from the host to the storage controllers.

3.    Run the multipath -ll command to get the worldwide identifiers (WWIDs) for the device file names:

4.    Edit the /etc/multipath.conf file and add the WWIDs and alias names.

5.    Run the multipath -r command to reload the device map.

6.    Verify the configuration by running the multipath -ll command to list all the LUNs, alias names, and active and standby paths.

Create File Systems

To create the XFS file system on each LUN belonging to the HANA system, take the following actions:

Create Mount Points

To create the required mount-point directories, take one of the following actions:

Mount File Systems

To mount file systems during system boot using the /etc/fstab configuration file, follow these steps:

1.    Add the required file systems to the /etc/fstab configuration file.

2.    Run mount –a to mount the file systems on the host.

Configure SAP HANA Scale-out Systems

Volume and LUN Configuration for SAP HANA Scale-out Systems

Figure 6 shows the volume configuration of a 4+1 scale-out SAP HANA system. The data volumes and log volumes of each SAP HANA host are distributed to different storage controllers. For example, the volume SID_data_mnt00001 is configured on controller A and the volume SID_log_mnt00001 is configured on controller B. One LUN is configured within each volume.

The /hana/shared volume must be accessible by all HANA hosts and is therefore exported by using NFS. Even though there are no specific performance KPIs for the /hana/shared file system, NetApp recommends using a 10Gb Ethernet connection.

Figure 6.       Volume Layout for SAP HANA Scale-out Systems

For each SAP HANA host, a data volume and a log volume are created. The /hana/shared volume is used by all hosts of the SAP HANA system. Table 12 lists an example configuration for a 4+1 scale-out SAP HANA system.

Table 12.       Volume Configuration for SAP HANA Scale-out Systems

Table 13 lists the configuration and the mount points of a scale-out system with four active SAP HANA hosts.

Table 13.       Mount Points for Scale-out Systems

Configuration Example for SAP HANA Scale-out Systems

The following examples show a 4+1 SAP HANA scale-out database with SID=FLX and a server with a RAM size of 2TB. For different server RAM sizes, the required volume sizes are different.

For a detailed description of the capacity requirements for SAP HANA, see the SAP HANA Storage Requirements white paper.

Creating LUNs, Volumes, and Mapping LUNs to Initiator Groups

You can use NetApp ONTAP System Manager and CLI to create storage volumes and LUNs and map them to the servers.

NetApp offers an automated application wizard for SAP HANA within ONTAP System Manager, which simplifies the volume and LUN provisioning process significantly. It creates and configures the volumes and LUNs automatically according to NetApp best practices for SAP HANA.

Using the sanlun tool, run the following command to obtain the worldwide port names (WWPNs) of each SAP HANA host:

To configure a 4+1 scale-out HANA system with the SID FLX, follow these steps:

1.    Start the Application Provisioning wizard for SAP HANA in System Manager and provide the required information. All initiators (WWPNs) from all hosts need to be added.

2.    Confirm that storage is successfully provisioned.

SAP HANA Storage Connector API

A storage connector is required in scale-out environments that have failover capabilities. In scale-out setups, SAP HANA provides high-availability functionality so that an SAP HANA database host can fail over to a standby host. In this case, the LUNs of the failed host is accessed and used by the standby host. The storage connector is used to make sure that a storage partition can be actively accessed by only one database host at a time.

In SAP HANA scale-out configurations with NetApp storage, the standard storage connector delivered by SAP is used. The “SAP HANA Fibre Channel Storage Connector Admin Guide” can be found as an attachment to SAP note 1900823.

Host Setup

Before setting up the host, NetApp SAN host utilities must be downloaded from the NetApp Support site and installed on the HANA servers. The host utility documentation includes information about additional software that must be installed depending on the FCP HBA used.

The documentation also contains information on multipath configurations that are specific to the Linux version used. This document covers the required configuration steps for SLES 12 SP1 or higher and RHEL 7.2 or later, as described in the Linux Host Utilities 7.1 Installation and Setup Guide.

Configure Multipathing

To configure multipathing, follow these steps:

1.    Run the Linux rescan-scsi-bus.sh -a command on each server to discover new LUNs.

2.    Run the sanlun lun show command and verify that all required LUNs are visible. The following example shows the sanlun lun show command output for a 2+1 scale-out HANA system with two data LUNs and two log LUNs. The output shows the LUNs and the corresponding device files, such as LUN SS3_data_mnt00001 and the device file /dev/sdag Each LUN has eight FC paths from the host to the storage controllers.

3.    Run the multipath -r command to get the worldwide identifiers (WWIDs) for the device file names:

4.    Edit the /etc/multipath.conf file and add the WWIDs and alias names.

5.    Run the multipath -r command to reload the device map.

6.    Verify the configuration by running the multipath -ll command to list all the LUNs, alias names, and active and standby paths.

Create File Systems

To create the XFS file system on each LUN belonging to the HANA system, take one of the following actions:

Create Mount Points

Set permissions and create mount points on all worker and standby hosts.

Mount File Systems

To mount file systems during system boot using the /etc/fstab configuration file, follow these steps:

1.    Add the /hana/shared file system to the /etc/fstab configuration file of each host.

2.    To mount the file systems, run the mount –a command at each host.

3.    A sample global.ini file from the installed Scale-out SAP HANA system is shown below:

SAP HANA Installation

For information about the SAP HANA installation, please use the official SAP documentation, which describes the installation process with and without the SAP unified installer.

SAP HANA Server Installation Guide

All other SAP installation and administration documentation is available here: http://service.sap.com/instguides.

Important SAP Notes

Read the following SAP Notes before you start the installation. These SAP Notes contain the latest information about the installation, as well as corrections to the installation documentation.

The latest SAP Notes can be found here: https://service.sap.com/notes.

SAP HANA IMDB Related Notes

SAP Note 1514967   - SAP HANA: Central Note

SAP Note 1523337   - SAP HANA Database: Central Note

SAP Note 2000003   - FAQ: SAP HANA

SAP Note 1730999 - Configuration changes in SAP HANA appliance

SAP Note 1514966 - SAP HANA 1.0: Sizing SAP In-Memory Database

SAP Note 1743225 - SAP HANA: Potential failure of connections with scale out nodes

SAP Note 1681092   - Support for multiple SAP HANA databases on a single SAP HANA appliance

SAP Note 1514966   - SAP HANA: Sizing SAP HANA Database

SAP Note 1637145   - SAP BW on HANA: Sizing SAP HANA Database

SAP Note 1793345   - Sizing for Suite on HANA

Linux Related Notes

SAP Note 2235581   - SAP HANA: Supported Operating Systems

2578899 - SUSE Linux Enterprise Server 15: Installation Note

SAP Note 2009879   - SAP HANA Guidelines for Red Hat Enterprise Linux (RHEL)

SAP Note 1731000 - Non-recommended configuration changes

SAP Note 2382421 - Optimizing the Network Configuration on HANA- and OS-Level

SAP Note 1829651 - Time zone settings in SAP HANA scale out landscapes

SAP Application Related Notes

SAP Note 1658845 - SAP HANA DB hardware check

SAP Note 1637145 - SAP BW on SAP HANA: Sizing SAP In-Memory Database

SAP Note 1661202 - Support for multiple applications on SAP HANA

SAP Note 1681092 - Support for multiple SAP HANA databases one HANA aka Multi SID

SAP Note 1577128 - Supported clients for SAP HANA 1.0

SAP Note 1808450 - Homogenous system landscape for on BW-HANA

SAP Note 1976729   - Application Component Hierarchy for SAP HANA

SAP Note 1927949   - Standard Behavior for SAP Logon Tickets

SAP Note 1577128   - Supported clients for SAP HANA

SAP Note 2186744   - FAQ: SAP HANA Parameters

SAP Note 2267798   - Configuration of the SAP HANA Database during Installation Using hdbparam

SAP Note 2156526   - Parameter constraint validation on section indices does not work correctly with hdbparam

SAP Note 2399079   - Elimination of hdbparam in HANA 2

Third-Party Software

SAP Note 1730928 - Using external software in an SAP HANA appliance

SAP Note 1730929 - Using external tools in an SAP HANA appliance

SAP Note 1730930 - Using antivirus software in an SAP HANA appliance

SAP Note 1730932 - Using backup tools with Backint for SAP HANA

NetApp Technical reports

TR-4436-SAP HANA on NetApp AFF Systems with FCP

TR-4614-SAP HANA Backup and Recovery with SnapCenter

TR-4646-SAP HANA Disaster Recovery with  Storage Replication

High-Availability Configuration for Scale-out Systems

Before beginning the installation, create a global.ini file to enable use of the SAP storage connector during the installation process. The SAP storage connector mounts the required file systems at the worker hosts during the installation process. The global.ini file must be available in a file system that is accessible from all hosts, such as the /hana/shared file system.

Before installing SAP HANA software on a scale-out system, the following steps must be completed:

1.    Add the following mount options for the data LUNs and the log LUNs to the global.ini file:

a.     relatime and inode64 for the data file system

b.    relatime and inode64 for the log file system

2.    Add the WWIDs of the data and log partitions. The WWIDs must match the alias names configured in the /etc/multipath.conf file.

3.    Prepare global.ini file for the installation of a Scale-out SAP HANA system. Example is shown below:

4.    Using the SAP hdblcm installation tool, start the installation by running the following command at one of the worker hosts. Use the addhosts option to add the further worker and the standby hosts

SAP HANA Data Volume Size

A default SAP HANA instance uses only one data volume per SAP HANA service. Due to the max file size limitation of the file system, NetApp recommends limiting max data volume size.

To automatically limit the max data volume size, set the following parameter in the global.ini within section (persistence):

This creates a new data volume when the limit of 8000GB is reached.

SAP Note 240005 question 15 provides more information: https://launchpad.support.sap.com/#/notes/2400005

Cisco Intersight

Cisco Intersight is an intelligent Software-as-a-Service (SaaS) platform for IT staff to manage and get support for their Intersight-connected environment when and where they need it. It simplifies the infrastructure management and provides proactive support for the FlexPod environment.

Cisco Intersight Assist helps to add endpoint devices Cisco UCS Domain, Hypervisors (in case of virtual implementations) and more to Cisco Intersight and provides the connection mechanism to claim the device in Cisco Intersight.

Cisco Intersight Assist is available as part of the Cisco Intersight Virtual Appliance, which is distributed as a deployable virtual machine contained within an Open Virtual Appliance (OVA) file format. Install the appliance on an ESXi server on-premise. For more information see the Cisco Intersight Assist Getting Started Guide.

The Cisco Intersight Help Center provides an overview and information on how to get started with Cisco Intersight. The Help Center is available from https://intersight.com/help/home.

Requirements

The following prerequisites are necessary to setup access to Cisco Intersight and to connect the core FlexPod components to Intersight:

●     A valid Cisco Intersight account. Navigate to https://intersight.com to create an account and follow the account creation instructions. The account creation requires at least one device to be registered in Intersight including its Device ID and Claim ID information.  

●     Valid Advanced or Premier License for Cisco Intersight.

●     Cisco UCS Fabric Interconnects must be able to do a DNS lookup to access Cisco Intersight.

●     External endpoints registration messages to be routed to Cisco Intersight.

●     Calls from non-registered endpoints (or other infrastructure devices) to be routed to Cisco Intersight.

Table 14.       Connectivity Requirements (direct or through HTTP proxy)

Cisco Intersight Virtual Assist Installation

Cisco Intersight Assist helps to add endpoint devices to Cisco Intersight which do not connect directly with Cisco Intersight. The NetApp array doesn’t connect directly with Cisco Intersight and requires a connection mechanism. Cisco Intersight Assist provides that connection mechanism which helps to add the NetApp array into Cisco Intersight.

Requirements

The following are the Cisco Intersight requirements:

●     VMWare ESXi 6.0 and higher

●     VMWare vSphere WebClient 6.5 and higher

●     System Requirements are 8 to 23 vCPU and 16 to 64 GB of main memory.

●     DNS Setup

◦     myhost.example.com (A record and PTR record) with a valid IP address

◦     dc-myhost.example.com (CNAME record of myhost.example.com)

Deploy Cisco Intersight Virtual Appliance

The Getting Started Guide provides an overview of the Cisco Intersight Virtual Appliances and details the required installation steps.

To deploy the Cisco Intersight Virtual Appliance, follow these steps:

1.    Log into the VMWare vSphere VCenter via Web Client with administrator credentials.

2.    Select the creation type – Deploy OVF template.

3.    Select the name of the OVF and deployment location.

4.    Select the destination compute resource and go to the next page.

5.    Review the details and go to the next page.

6.    Select the appropriate deployment size.

7.    Select a destination and thin provision to optimize disk usage on the select storage page.

8.    Select a source network and map it to the destination network. This could be the VM management network which has access to all infrastructure elements management.

9.    On the Customize Template page, customize the deployment properties of the OVF template and provide network details, an admin password, and NTP and DNS domain/server information. Click Next.

10.  Click Finish on the Ready to complete page.

Set Up Cisco Intersight Assist

To set up Cisco Intersight Assist, follow these steps:

1.    After installing Cisco Intersight Virtual Appliance, start the VM host in vCenter. After a short period of time, connect to the host using your web browser to proceed with the configuration. Provide your proxy configuration in the settings section to connect the installer to the Internet.

2.    Select Intersight Assist and click Proceed to start the two-step approach installation:

a.     Connect Intersight Virtual Appliance.

b.    Software Download.

The first step provides the Device ID and Claim Code to claim the Virtual Appliance like a standard device in Cisco Intersight.

3.    Once connected click Continue to move to the second setup step and wait for the download to complete which can take some time.

Claim the Fabric Interconnects

To connect and access Cisco Intersight, follow these steps:

1.    In Cisco UCS Manager, click the Admin tab in the navigation pane.

2.    Choose Device Connector from the drop-down list.

3.    Click Settings and provide the DNS, NTP and proxy configuration. Click Save.

4.    Enable Intersight Management.

5.    In Cisco Intersight, click the Admin > Targets tab in the navigation pane.

6.    Click Claim a Net Target.

7.    Copy & Paste the Device ID and Claim Code from Cisco USCM.

8.    Click Claim to connect the fabric interconnects to Cisco Intersight.

9.    Click one of the Fabric Interconnects to review the chassis and server status. From the tool menu it is possible to open a TAC support case from Intersight by uploading the technical support information for further troubleshooting.

Monitor SAP HANA with AppDynamics

Introduction

AppDynamics is an Application Performance Monitoring (APM) Platform that helps you to understand and optimize the performance of your business, from its software to infrastructure to business journeys.

The AppDynamics APM Platform enables you to monitor and manage your entire application-delivery ecosystem, from the mobile app or browser client request through your network, backend databases and application servers and more. AppDynamics APM gives you a single view across your application landscape, letting you quickly navigate from the global perspective of your distributed application right down to the call graphs or exception reports generated on individual hosts.

AppDynamics has an agent-based architecture. Once our agents are installed it gives you a dynamic flow map or topography of your application. It uses the concept of traffic lights to indicate the health of your application (green is good, yellow is slow, and red indicates potential issues) with dynamic baselining. AppDynamics measures application performance based on business transactions which essentially are the key functionality of the application. When the application deviates from the baseline AppDynamics captures and provides deeper diagnostic information to help be more proactive in troubleshooting and reduce the MTTR (mean time to resolution).

SAP Landscape Monitoring

AppDynamics has a one of its kind ABAP agent for monitoring SAP ABAP systems. We have comprehensive coverage of the SAP landscape with our ABAP, Java, .net and Server visibility agents. In addition, Datavard Insights extends the AppDynamics for SAP solution with system-level monitoring for the overall SAP systems and SAP HANA databases. While AppDynamics agents provides transaction-level visibility, Datavard Insights collects performance metrics, logs, and events, including processes outside of the user business transactions, such as background jobs or IDocs processing.

The complex and proprietary nature of SAP applications makes it difficult to diagnose issues. AppDynamics allows enterprises to instrument SAP applications, monitor performance, and understand the root cause of performance bottlenecks.

 

Trial Registration

To register for a free trial, follow these steps:

1.    Connect to https://www.appdynamics.com/free-trial/.

2.    Provide the details to sign up for a free trial utilizing an AppDynamics SaaS controller.

3.    Once the AppDynamics SaaS Controller has been provisioned, you will receive an email with the information you need for you to login to the Controller.

4.    You can download the Java Agent and the Server Visibility Agent directly from the Controller

 

5.    You can use the email and password you provided to sign up for the trial to login to the agent download site at the URL listed below and download the ABAP Agent: https://download.appdynamics.com

Agent Installation

AppDynamics has several types of agents to monitor different language applications to user Experience to Infrastructure monitoring. Based on the SAP landscape and the underlying technology of the SAP systems the agents are installed.

The most frequently installed agents are:

●     Java Agent - For Java based SAP Systems

●     ABAP Agent - For ABAP based SAP systems

●     Server Visibility Agent - Provides extended hardware metrics and Service Availability Monitoring

Prerequisites

Please see the link below to verify the supported SAP environments:

https://docs.appdynamics.com/display/SAP/SAP+Supported+Environments

Java Agent Installation

The Java Agent must be installed on SAP Java application servers (for example, Enterprise Portal and PO application servers).

The high-level steps for installing the Java Agent are listed below:

1.    Ensure you are logged into the host server as the appropriate <SID>adm OS user.

2.    Create the permanent installation directory for the agent.

3.    Unzip the file in the permanent directory (such as /usr/sap/appdyn/app).

4.    Change the permissions on the contents of the agents' installation directory to give full read/write privileges at the owner and group levels.

5.    Edit the configuration file in the agent installation directory to configure the agent to connect to the controller, provide the identity of the JVM, and so on.

6.    You will need to add parameters to the SAP JVM to start the Java agent when the SAP system is started up by logging into the SAP app server as the “sidadm” user.

7.    Use the SAP NetWeaver Administrator or the AS Java Config Tool (depending on your SAP system) to edit the JVM startup parameters.  For more detailed information, see: Configuring AppDynamics Java Agent in SAP

8.    Restart the SAP JVM for the settings to take effect

9.    Validate the Java Agent is reporting to the controller by logging into the controller UI

For detailed information, see: Install the AppDynamics Java Agent

ABAP Agent Installation

The ABAP Agent needs to be installed on the SAP servers utilizing the ABAP stack.

There are 4 primary steps to perform, each with secondary steps involved. The four primary steps are as follows:

1.    Copy and unzip the ABAP Agent.

2.    Import the ABAP Agent Transports.

3.    Configure ABAP Agent and Install HTTP SDK.

4.    Activate Datavard Insight Collectors.

Copy and Unzip ABAP Agent

The high-level steps to copy and unzip the ABAP Agent are listed below:

1.    Ensure you are logged into the host server as the appropriate <SID>adm OS user.

2.    Copy the agent binary to a temporary directory on the server.

3.    Unzip the file into a temporary directory.

4.    Change the permissions on the contents of the agents' installation directory to give full read/write privileges at the owner and group levels.

Import the ABAP Agent Transports

The high-level steps to import the ABAP agent transports are listed below:

1.    The ABAP Agents data files and cofiles should be copied from the temporary directories where they were unzipped over to the appropriate transport directories for the SAP module in question. For example, for ECC we would copy the transports to "/usr/sap/trans/ECC2/cofiles" and "/usr/sap/trans/ECC2/data."

2.    Set the permissions on the cofiles and data files to allow read/write access at the owner and group levels.

3.    Log into the SAP system, execute transaction STMS:

a.     Go to the import queue of the system where you want to install the ABAP agent.

b.    Select "Extras > Other Requests > Add" from the menu bar and add the vendor transports to the import queue one at a time.

4.    Import the transports in the appropriate order

The order of the transports is critical and is specified in the “readme.txt” file of the ABAP Agent subdirectory that is specific to the version of NetWeaver in use. For more information please see: Install on SAP NetWeaver Systems

Make sure that when selecting the "Execution" tab in the "Import Transport Request" pop-up dialog box to select the option "Synchronous". When selecting the "Options" tab, put a checkmark next to "Ignore Invalid Component Version".

Configure ABAP Agent / Install HTTP SDK

The high-level steps to configure the ABAP agent and install the HTTP SDK are listed below:

1.    Log into the SAP system and execute transaction "/DVD/APPD_CUST".

2.    Switch to edit mode.

3.    Fill in the fields on the screen to configure the agent to connect to the controller, SDK settings, and Analytics API settings.

4.    Click Activate integration.

5.    Click SDK Installation. This will take you to the "AppDynamics HTTP SDK Installation Manager" screen.

6.    Go to Edit > Change Directory:

a.     Enter the path that was used for the agents' permanent base install directory (i.e. /usr/sap/appdyn) in the field displayed in the pop-up dialog box shown below, and then click OK.

b.    Click Install SDK.

c.     Click the green checkmark to exit the screen and return to the "AppDynamics settings" screen.

d.    Click Status. This will take you to the "AppDynamics status check" screen.

e.    Click Start to start the HTTP SDK proxy on each SAP app server.

Activate Datavard Insight Collectors

Datavard Insights collect detailed performance data for an SAP system. It uses collector jobs that run as periodic background processes in the SAP system. These jobs must be scheduled to run.

Refer to the following links to the related documentation:

Datavard Insights Integration

Performance Collector Settings

SAP Dashboards

Mapping Between AppDynamics Metrics and Datavard Insights KPIs

Server Visibility Agent Installation

The Server Visibility Agent must be installed on every application server and central services server that will be monitored.

The high-level steps for installing the Server Visibility agent are listed below:

1.    Ensure you are logged into the host server as the appropriate <SID>adm OS user.

2.    Create the permanent installation directory for the agent.

3.    Unzip the file in the permanent directory (such as /usr/sap/appdyn/machine).

4.    Change the permissions on the contents of the agents' installation directory to give full read/write privileges at the owner and group levels.

5.    Edit the configuration files in the agent installation directory to configure the agent to connect to the controller, provide the identity of the host, and so on.

6.    Start the server visibility agent with the script provided in the agents’ bin directory.

7.    Validate the server visibility is reporting to the controller by logging into the controller UI.

Appendix

FlexPod Backups

Cisco UCS Backup

Automated backup of the UCS domain is important for recovery of the UCS Domain from issues ranging catastrophic failure to human error.  There is a native backup solution within Cisco UCS that allows local or remote backup using FTP/TFTP/SCP/SFTP as options.

Backups created can be a binary file containing the Full State, which can be used for a restore to the original or a replacement pair of fabric interconnects.  Alternately create the XML configuration file consisting of All configurations, just System configurations, or just Logical configurations of the UCS Domain.  For scheduled backups, options will be Full State or All Configuration, backup of just the System or Logical configurations can be manually initiated.

To configure the backup, using the Cisco UCS Manager GUI, follow these steps:

1.    Choose Admin within the Navigation pane and choose All.

2.    Click the Policy Backup & Export tab within All.

3.    For a Full State Backup, All Configuration Backup, or both, specify the following:

a.     Hostname: <IP or FQDN of host that will receive the backup>

b.    Protocol: [FTP/TFTP/SCP/SFTP]

c.     User: <account on host to authenticate>

d.    Password: <password for account on host>

e.    Remote File: <full path and filename prefix for backup file>

f.      Admin State: <choose Enable to activate the schedule on save, Disable to disable schedule on Save>

g.    Schedule: (Daily/Weekly/Bi Weekly)

 

4.    Click Save Changes to create the Policy.

Cisco Nexus and MDS Backups

The configuration of the Cisco Nexus 9000 and Cisco MDS 9148T switches can be backed up manually at any time with the copy command, but automated backups can be put in place with the NX-OS feature scheduler.  An example of setting up automated configuration backups of one of the FlexPod 9336C-FX2 switches is shown below:

Show the job that has been setup:

The documentation for the feature scheduler can be found here: http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/system_management/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_System_Management_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_System_Management_Configuration_Guide_7x_chapter_01010.html

About the Authors

Pramod Ramamurthy, Technical Marketing Engineer, Cisco Systems, Inc.

Pramod is a Technical Marketing Engineer with Cisco UCS Solutions and Performance Group. Pramod has more than 15 years of experience in the IT industry focusing on SAP technologies. Pramod is currently focusing on the Converged Infrastructure Solutions design, validation and associated collaterals build for SAP HANA.

Marco Schoen, Technical Marketing Engineer, NetApp, Inc.

Marco is a Technical Marketing Engineer with NetApp and has over 20 years of experience in the IT industry focusing on SAP technologies. His specialization areas include SAP NetWeaver Basis technology and SAP HANA. He is currently focusing on the SAP HANA infrastructure design, validation and certification on NetApp Storage solutions and products including various server technologies.

Acknowledgements

For their support and contribution to the validation of this Cisco Validated Design, we would like to thank:

●     Shailendra Mruthunjaya, Cisco Systems Inc.

Feedback

For comments and suggestions about this guide and related guides, join the discussion on Cisco Community at https://cs.co/en-cvds.

NOTE: Available paragraph styles are listed in the Quick Styles Gallery in the Styles group on the Home tab. Alternatively, they can be accessed via the Styles window (press Alt + Ctrl + Shift + S).