Have an account?

  •   Personalized content
  •   Your products and support

Need an account?

Create an account

Provision Software-Defined Storage Infrastructure at Cloud Scale with Terraform Provider for Cisco Intersight Platform

White Paper

Available Languages

Download Options

  • PDF
    (1.2 MB)
    View with Adobe Reader on a variety of devices
Updated:May 6, 2020

Available Languages

Download Options

  • PDF
    (1.2 MB)
    View with Adobe Reader on a variety of devices
Updated:May 6, 2020
 

 

This document describes how to use the Terraform provider for the Cisco Intersight to achieve automated, zero-touch infrastructure deployment for cloud-scale software-defined storage on the Cisco Unified Computing System (Cisco UCS®). This guide provides an overview of the design and development of a Terraform configuration for provisioning infrastructure and installing the operating system for a generic software-defined storage solution using the Terraform provider for the Cisco Intersight platform.

Executive summary

This document describes how to use the Terraform provider for Cisco Intersight software with the Cisco Unified Computing System (Cisco UCS®) platform to automate infrastructure provisioning and operating system deployment for cloud-scale storage. This guide provides the framework for deploying any software-defined storage solution on three Cisco UCS C240 M5 Rack Server nodes with two Cisco UCS C220 M5 Rack Servers as controller and load-balancer nodes. Cisco UCS provides computing, network, and storage components as a unified platform for software-defined storage solutions. Cisco Intersight software provides a systems management platform that delivers intuitive computing through cloud-powered intelligence. The Cisco Intersight platform provides infrastructure management for Cisco UCS and Cisco HyperFlex systems. It supports the use of the Terraform provider to develop infrastructure as code, facilitating zero-touch deployment.

Introduction

This section provides an overview of the Terraform provider, Cisco Intersight and Software-defined storage solution with Cisco Intersight.

Cisco Intersight platform

The Cisco Intersight platform provides benefits for the entire infrastructure lifecycle. Cisco UCS Manager, Cisco UCS Director, and Cisco Integrated Management Controller (IMC) focus on day-0 and day-1 activities related to deployment and configuration. The Cisco Intersight platform augments this focus with benefits for day 2 and beyond as well. In addition to monitoring and alerting, Cisco Intersight provides integration with the Cisco® Technical Assistance Center (TAC), predictive analytics, and resource optimization to address ongoing operations and systems upgrades.

Figure 1 shows a high-level view of multiple management stages with the Cisco Intersight platform.

A picture containing objectDescription automatically generated

Figure 1.            

Cisco Intersight infrastructure lifecycle

HashiCorp Terraform

Terraform is software for building complex, version-controlled, and collaborative infrastructure with high productivity. The infrastructure Terraform can manage includes low-level components such as computing instances, storage, and networking, as well as high-level components such as Domain Name System (DNS) entries and software-as-a-service (SaaS) features.

The set of files used to describe infrastructure in Terraform is known as a Terraform configuration. The configuration is written using HashiCorp Configuration Language (HCL), a simple human-readable configuration language, to define a desired topology of infrastructure resources.

Why  Terraform provider for the Cisco Intersight?

The Terraform provider allows organizations to develop Cisco Intersight resources as self-service infrastructure using code rather than manual provisioning.

This approach provides several benefits:

      You can more quickly and easily scale Cisco Intersight resources. You can provision infrastructure in minutes, with little effort, using the automated workflows, performing the same tasks that used to take days.

      The operating model of Terraform is well suited for the Cisco Intersight platform, because it accommodates the shift from static to dynamic infrastructure provisioning. For example, if a resource is deleted in the Terraform configuration, it will be reflected in the Cisco Intersight platform when the new configuration is applied.

      Terraform maintains a state file, which is a record of the currently provisioned resources. State files provide a version history of Cisco Intersight resources, enabling a detailed audit trail of changes.

      The provider enables idempotency, producing the same result and state with repeated API calls.

 

Software-defined storage solution with Cisco UCS and Cisco Intersight

The Cisco Intersight platform provides a comprehensive architecture for deploying and managing software-defined storage. Figure 2 shows the hardware design for a generic software-defined storage solution.

A screenshot of a cell phoneDescription automatically generated

Figure 2.            

Solution hardware design

The design includes the following components:

      The Cisco Intersight platform is deployed as a SaaS solution.

      Controller and load-balancer nodes are deployed on Cisco UCS C220 M5 servers.

      Storage nodes are deployed on Cisco UCS C240 M5 servers.

      Cisco UCS C240 and C220 servers are connected to Cisco Nexus® 93240YC-FX2 Switches with 25-Gbps line speed.

Solution deployment

The deployment of the whole solution consists of several main steps:

      Create an account on the Cisco Intersight platform and claim all storage nodes.

      Create a Terraform configuration environment for the Cisco Intersight platform.

      Apply Terraform configurations for the following:

    Update firmware on all storage nodes.

    Create server policies and profiles.

    Deploy server profiles for all nodes.

    Install the operating system on all nodes.

      Deploy software-defined storage software.

Creating an account on the Cisco Intersight platform and claiming nodes

You create an account on the Cisco Intersight platform by claiming a Cisco IMC device. Before you claim the device, you should perform a preliminary check of the Cisco Intersight device connector in the IMC. The current device connector requirements are described in https://www.cisco.com/c/en/us/td/docs/unified_computing/Intersight/b_Cisco_Intersight_Appliance_Getting_Started_Guide/b_Cisco_Intersight_Appliance_Install_and_Upgrade_Guide_chapter_010.pdf and summarized in Table 1.

Table 1.             Device connector requirements

Component

Minimum software version

Supported device connector version

Releases that include supported device connectors

Cisco UCS Manager

Release 3.2(I)

Release 1.0.9-2290

Release 4.0(2a) or later

Cisco IMC Supervisor

For M5 servers: Release 3.1(3a)

For M4 servers: Release 3.0(4)

Release 1.0.9-335

Release 4.0(2c) or later

Cisco HyperFlex Connect and Cisco HyperFlex HX Data Platform

Release 2.6

Release 1.0.9-1335

Release 3.5(2a) or later

 

To create an account on the Cisco Intersight platform, follow these steps:

1.     Log in to https://www.intersight.com/.

2.     Sign in with your Cisco ID (Figure 3).

Related image, diagram or screenshot

Figure 3.            

Cisco Intersight main screen

3.     Read the offer description and accept it.

4.     Copy the device ID and claim code in the Cisco Intersight Account Creation wizard from the Cisco IMC.

5.     Enter a name for the account and click Create.

The device claim process can take a few minutes. If required, the device connector will automatically be upgraded as part of the process. Account creation and basic configuration are now finished.

6.     On the Cisco Intersight dashboard, choose Devices > Claim a New Device (Figure 4).

Related image, diagram or screenshot

Figure 4.            

Claiming a new device

7.     Copy the device ID and claim code for the second Cisco UCS C240 M5L server and click Claim.

8.     Repeat the same process for the third Cisco UCS C240 M5L server and for the Cisco UCS C220 M5 servers.

9.     After you claim all the devices, you should see the devices listed under the server tab at the left.

 

Create a Terraform configuration environment for the Cisco Intersight platform

Next you create the Terraform configuration environment.

Install Terraform

On the deployment host, make sure that Terraform is installed. The Terraform binary is available for major distributions.

1.     Download the Terraform zip file from https://www.terraform.io/downloads.html according to the operating system of your deployment host.

2.     Extract the zip file and move it to a directory of your choice.

3.     Add the path of the directory to the PATH variable of the system.

For more information about how to install Terraform, see https://learn.hashicorp.com/terraform/getting-started/install.html.

Clone the repository

Use the following command to clone the repository on your deployment host. This repository contains code to deploy the whole solution.

After the repository has been cloned, you should see five directories: firmware_update, create_infra, provision_infra, os_deployment, and unbind_profiles. Separate directories are created for logical separation and easy understanding of workflows.

If required, you can merge all the resources into a single file in your deployment with fewer modifications

Copy the Terraform provider binary file

The Terraform binary file for the Cisco Intersight platform is built for Microsoft Windows, Darwin (Mac OS), and Linux 64-bit architecture systems. Choose the appropriate binary file according to the operating system in your deployment host.

Follow these steps:

4.     Go to https://github.com/cisco-intersight/terraform-provider-intersight.

5.     Navigate to the Releases tab. Download the terraform-provider-intersight.zip file for the latest release.

6.     Extract the files and navigate to the terraform-provider-intersight binary file for the operating system installed on your deployment host.

7.     Copy the terraform-provider-intersight binary file to all the directories of the downloaded repository.

Generate API keys

To use the Cisco Intersight provider, you need an API key, a secret key, and the Cisco Intersight endpoint URL. To generate the keys, follow these steps:

8.     Log in to https://www.intersight.com/.

9.     On the Settings screen, click the Settings menu.

10.  On the General page, choose API > API Keys > Generate API Key.

11.  On the Generate New API Key screen, enter the purpose for the API key and click Generate. The API key ID and RSA private key are displayed.

12.  Copy the API key.

13.  Save the private key information in a .pem file. Save it in a location in the downloaded repository.

Define the Cisco Intersight provider

Now define the Cisco Intersight provider.

Navigate to the create_infra directory in the cloned repository and open main.tf file. Enter the API key you copied. Also provide the secret key file. The endpoint changes if you are using  Cisco Intersight appliance.

provider "intersight" {
  apikey    = "API Key generated from previous step”
  secretkeyfile = "Secret key file generated from previous step "
  endpoint = "https://intersight.com"
}

For simplicity in this document, multiple directories are defined for each workflow (to create infrastructure, apply server profiles, upgrade firmware, etc.). Copy the same main.tf file in all the directories. You can also combine all the Terraform configurations in one directory.

Configure variables

You need to define some of the basic inputs required for various workflows to provision the infrastructure. You will define all these as variables in a file named variables.tf.

Each configuration takes a value in the variables.tf file as shown here:

variable "variable_name " {
  default = <value of the variable>
}

You need to keep several image files, such as the operating system image file, Cisco Server Configuration Utility (SCU), and Cisco Host Update Utility (HUU), in a remote server for firmware upgrades and operating system installation. Download the files from software.cisco.com and keep them in a local Network File System (NFS), HTTP, or Common Internet File System (CIFS) share that is accessible to the Cisco IMC for the servers. Then you can edit the variables for the remote server and images.

In variables.tf, define all the VLAN IDs that you need for the management, client, cluster, and replication networks.

Now you need the managed object ID (MOID) for the organization and claimed storage nodes. To get these values, follow these steps:

14.  Log in to https://www.intersight.com/.

15.  From the Help screen, click Get More Help from Cisco Intersight.

16.  Navigate to API Documentation.

17.  Click API Reference (Figure 5).

A screenshot of a cell phoneDescription automatically generated

Figure 5.            

API Reference

18.  Search for compute/PhysicalSummary.

19.  Click GET: Read a 'compute.PhysicalSummary' resource (Figure 6).

 

A screenshot of a cell phoneDescription automatically generated

Figure 6.            

Supported methods for compute/PhysicalSummary resource

20.  In the representational state transfer (REST) client, click Send to get a response from the compute/PhysicalSummary API (Figure 7).

A screenshot of a cell phoneDescription automatically generated

Figure 7.            

REST Client screen

21.  The response is an array of physical servers claimed with all the information about the claimed servers. The first entry is the MOID for the server. Copy the MOIDs for all the claimed servers and the organization MOID.

22.  You have all the basic input required to apply the Terraform configurations. Copy the same varables.tf file in all the directories.


 

Understanding the Cisco Intersight provider and Terraform configuration

The resource objects names in the configuration are in this format:

intersight_<model_name_in_snake_case>

For example, intersight_ntp_policy is the resource object for the Network Time Protocol (NTP) policy, and intersight_storage_disk_group_policy is the resource object for the storage disk group policy. The following is an example of a complete resource definition for a server profile and the NTP policy attached to the server profile:

resource "intersight_server_profile" "storage-node1" {
  name = "storage-node1"
  organization {
    object_type = "organization.Organization"
    moid = var.org_moid
  }
  assigned_server {
  moid = var.storage-node1
  object_type = "compute.RackUnit"
  }
}
 
resource "intersight_ntp_policy" "sds-ntp-policy" {
  name    = "sds-ntp-policy"
  enabled = true
  ntp_servers = [
    "ntp.esl.cisco.com",
    "171.68.38.65",
    "173.38.201.115"
  ]
  organization {
    object_type = "organization.Organization"
    moid = var.org_moid
  }
 
  profiles {
    moid        = intersight_server_profile.storage-node1.moid
    object_type = "server.Profile"
  }
}

Each resource is assigned a name, which can later be used for tracking and referencing. This name will not be reflected anywhere in the Cisco Intersight platform. It is only for reference among the .tf files. The NTP policy is attached to the server profile created earlier. This is accomplished by referencing the storage-node1 profile in profiles.moid. A resource can point to or reference another resource using the format <resource>.<resource_name>.<property_name>.

View the logs

TF_LOG is a terraform variable that is used for viewing different categories of logs. By default, this variable is left empty. To view logs for Terraform operations, this variable must be set to DEBUG.

In Mac OS and Linux, you accomplish this with export TF_LOG=debug. In Windows PowerShell, use $env:TF_LOG=“DEBUG”.

View documentation

Documentation about provider resources and configuration options can be found at https://github.com/cisco-intersight/terraform-provider-intersight/tree/master/website/docs.

Implement the Terraform configuration

You need to run terraform init whenever you start with new Terraform code.

The terraform init command will scan the code, identify the provider you are using, and download the appropriate provider. In the case here, because you copied the provider manually, Terraform use the Cisco Intersight provider.

You next run the terraform plan command.

The plan command allows you to see what Terraform will do to the resources before any changes are actually made. This command provides a good way to check the potential results before you make any changes to the infrastructure.

In the output, the symbols show you the following:

      Resources with a plus sign (+) will be created.

      Resources with a minus sign (-) will be deleted.

      Resources with a tilde (~) will be modified in place.

Finally, run terraform apply to deploy the configuration. Observe that the apply command shows the same output as the plan command. To actually proceed with the plan, you need to type yes to confirm.

Applying Terraform configurations for workflows

Next apply the Terraform configurations to the various workflows.

Upgrade firmware

You should upgrade the firmware for the Cisco UCS C240 M5 servers to a suggested release. On the Cisco Intersight platform, you upgrade firmware using a noninteractive Cisco HUU to upgrade the BIOS, Cisco IMC, PCI adapters, RAID controllers, and other firmware to compatible versions.

You can upgrade firmware either through a network share or using utility storage. In this example, the firmware is upgraded through a network share using the NFS protocol. The HUU image file is mounted directly in the Cisco IMC. Therefore, it requires uninterrupted connectivity between the remote file server and Cisco IMC.

To apply the Terraform configuration for firmware update, follow these steps:

23.  Go to the firmware_update directory.

24.  Append all the servers that require firmware updates to firmware_update.tf.

25.  If running the configuration for the first time, run terraform init.

26.  Run terraform apply to initiate the firmware updates.

Create server policies

You need multiple server policies for any generic software-defined storage solution. Table 2 lists the policies required and the corresponding resource objects in the Terraform provider for the Cisco Intersight platform.

Table 2.             Terraform provider policies and resource objects

Policy

Terraform rResource oObject

Comments

Adapter configuration

intersight_adapter_config_policy

Specify the ID of the PCI slot ID in which the Cisco virtual interface card (VIC) adapter is placed.

Ethernet adapter

intersight_vnic_eth_adapter_policy

 

Ethernet network

intersight_vnic_eth_network_policy

Create these resources based on the number of networks required for the software-defined storage solution. In the example here, four networks are being created.

Ethernet quality of service

intersight_vnic_eth_qos_policy

 

LAN connectivity

intersight_vnic_lan_connectivity_policy

intersight_vnic_eth_if

The first resource creates LAN connectivity policy. The second resource creates multiple virtual interfaces.

NTP

intersight_ntp_policy

 

Disk group

intersight_storage_disk_group_policy

The purpose of this resource is to create RAID from 2 solid-state disks (SSDs) present in the server. This resource will be used for OS installation.

Storage

intersight_storage_storage_policy

This resource will include disk group policy and also set policies for virtual drives. This resource will also set storage disks to JBOD mode.

Boot order

intersight_boot_precision_policy

 

 

To apply the Terraform configuration to create server policies, follow these steps:

27.  Go to the create_infra directory.

28.  Update the server_profiles.tf file with the number of server profiles required.

29.  If you are running the configuration for the first time, run terraform init.

30.  Run terraform apply to initiate the firmware updates.

31.  After the updates are applied, verify that all the server profiles are created in the Cisco Intersight portal by navigating to Policies > Server Policies (Figure 8).

A screenshot of a computerDescription automatically generated

Figure 8.            

Policies created by Terraform configuration

Associate server profiles

After all the required polices have been created, you need to associate the server profiles with each physical server. To associate the profiles with the servers, follow these steps:

32.  Go to the provision_infra directory.

33.  Update the server_profiles.tf file with the resource intersight_server_profile for each server profile.

34.  For each server profile, define the server with which you want to associate it in the assigned_server attribute.

35.  If you are running the configuration for the first time, run terraform init.

36.  Run terraform apply to initiate the firmware updates.

37.  The process for applying the server profiles takes a few minutes. You can see that association is in progress from the Requests screen (Figure 9).

 

A screenshot of a cell phoneDescription automatically generated

Figure 9.            

Server profile association in progress

38.  After the server profiles have been applied, verify server profile association in the Cisco Intersight portal by navigating to Profiles > Server Profiles (Figure 10).

A screenshot of a stereoDescription automatically generated

Figure 10.         

Server profile associated with physical servers

Make changes to resources

If any of the resources needs updating, you first need to unbind the server profile, then make the changes, and then bind the server profiles back again.

Follow these steps to make any changes:

39.  Go to the unbind_profiles directory.

40.  Update unbind_profiles.tf file with the server profiles that you want to unbind.

41.   Run terraform apply.

42.  Make any necessary changes in either policies or profiles.

43.  Go to provision_infra directory and run terraform apply.

Install the operating system

The operating system used for this document is Red Hat Enterprise Linux (RHEL) 7.6. However, you can choose a different operating system based on the Cisco UCS Hardware and Software Compatibility Matrix and support from your storage software vendor.

To install the OS, follow these steps:

44.  Go to the os_deployment directory.

45.  In repo_setup.tf file, define all the OS images in the resource intersight_softwarerepository_operating_system_file. Use a separate resource for each OS type.

46.  In the same file, configure the resource for the Cisco SCU in the intersight_firmware_server_configuration_utility_distributable resource.

47.  In os_install.tf file, define the resource intersight_os_install for each node.

48.  Run terraform apply to install the OS.

Complying with the Cisco Hardware Compatibility List

The Cisco Intersight platform evaluates the compatibility of your Cisco UCS and Cisco HyperFlex systems to verify that the hardware and software have been tested and validated by Cisco or Cisco partners. The Cisco Intersight platform reports validation issues after checking the compatibility of the server hardware (server model, CPU, and server firmware version), server software (current OS vendor and OS version), and adapter compliance (adapter model, driver protocol, and driver version for the firmware).

You can manually determine the recommended hardware and firmware versions for your server configuration with the Cisco UCS Hardware and Software Compatibility tool at https://ucshcltool.cloudapps.cisco.com/public/.

The Cisco Intersight platform provides am open-source tool called the OS Discovery Tool to collect the OS and driver information need to evaluate Cisco Hardware Compatibility List (HCL) compliance in Linux operating system versions (Figure 11).

Related image, diagram or screenshot

Figure 11.         

OS Discovery Tool

Follow these steps to evaluate compliance with the HCL:

49.  Set up any Linux virtual or physical machine that has access to the on-premise network. Install Python 2.7 on this control node.

50.  On this control node, clone the Python software development kit (SDK) repository for the Cisco Intersight platform:

51.  Install the Python SDK:

cd intersight-python
python setup.py install

52.  In the intersight-python directory, go to os-discovery-tool.

53.  Edit the discovery_config_linux.json file to include intersight_api_key and path intersight_secret_file.

54.  Run the following script:

./get_linux_inv_to_intersight.py --log-inventory --configfile=discovery_config_linux.json

55.  The script collects information about the operating system and device drivers on all the nodes and routes this information to the Cisco Intersight platform to help evaluate compliance with the HCL. After this process is complete, you should be able to see the compliance status. In the Servers list, choose one of the servers and navigate to the HCL tab. You should see the HCL status (Figure 12).

Related image, diagram or screenshot

Figure 12.         

Hardware and software compatibility status

56.  For all the components, you can also see recommended versions and download the drivers by clicking Get Recommended Drivers (Figure 13).

A screenshot of a cell phoneDescription automatically generated

Figure 13.         

HCL recommendations

Deploy software-defined storage software

After the server profiles are attached and operating system is installed on all the nodes, you can deploy your software-defined storage solution. Refer to the software-defined storage vendor’s installation guides to deploy the solution.

Typically, you need to install two components: controllers and load balancer nodes and claiming storage nodes.

You can customize your Terraform resources, such as the number of networks and LAN connectivity policy, based on the requirements of your storage software vendor.

Conclusion

The Cisco Intersight SaaS platform transforms the way that customers deploy and manage Cisco UCS and Cisco HyperFlex systems. The Cisco Intersight platform supports the Terraform provider, which offers an excellent way to easily build, scale, and manage the lifecycle of any scale-out storage software solution with Cisco UCS servers connected to a switched environment. The Cisco Intersight platform offers resources for updating firmware, configuring profiles, associating profiles with physical servers, managing profiles, and installing the operating system.

For more information

For additional information, see the following resources:

      Terraform provider for the Cisco Intersight platform on GitHub:
https://github.com/cisco-intersight/terraform-provider-intersight

      GitHub repository for the solution:
https://github.com/ucs-compute-solutions/terraform-intersight-sds

      Cisco Intersight online help: https://intersight.com/help/home

      Cisco Intersight data sheet: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/intersight/datasheet-c78-739433.html#FlexibleDeploymentOptions

 

About the Author

Paniraja Koppa, Cisco Systems, Inc.

Paniraja Koppa is a Technical Marketing Engineer for UCS Solutions. He has more than 13 years of experience with a primary focus on data center technologies such as Cisco UCS, Storage, Operating systems, Automation, Virtualization and Cloud. In his current role at Cisco Systems, he works on best practices, optimization, automation and performance tuning of software defined storage on Cisco UCS platforms. Prior to this, he has led QA efforts for 4 new virtual adapter card’s firmware and software features for Cisco UCS.  He also worked as customer support engineer and advocate in the Data Center Virtualization space.

Acknowledgements

For their support and contribution, the authors would like to thank:

      Vikrant Balyan, Cisco Systems, Inc.

      Aanisha Mishra, Cisco Systems, Inc.

      Chris O'Brien, Cisco Systems, Inc.

      Oliver Walsdorf, Cisco Systems, Inc.

      Jawwad Memon, Cisco Systems, Inc.

      David Soper, Cisco Systems, Inc.

Learn more