This document describes how to use the Terraform provider for the Cisco Intersight™ to achieve automated, zero-touch infrastructure deployment for cloud-scale software-defined storage on the Cisco Unified Computing System™ (Cisco UCS®). This guide provides an overview of the design and development of a Terraform configuration for provisioning infrastructure and installing the operating system for a generic software-defined storage solution using the Terraform provider for the Cisco Intersight platform.
This document describes how to use the Terraform provider for Cisco Intersight™ software with the Cisco Unified Computing System™ (Cisco UCS®) platform to automate infrastructure provisioning and operating system deployment for cloud-scale storage. This guide provides the framework for deploying any software-defined storage solution on three Cisco UCS C240 M5 Rack Server nodes with two Cisco UCS C220 M5 Rack Servers as controller and load-balancer nodes. Cisco UCS provides computing, network, and storage components as a unified platform for software-defined storage solutions. Cisco Intersight software provides a systems management platform that delivers intuitive computing through cloud-powered intelligence. The Cisco Intersight platform provides infrastructure management for Cisco UCS and Cisco HyperFlex™ systems. It supports the use of the Terraform provider to develop infrastructure as code, facilitating zero-touch deployment.
This section provides an overview of the Terraform provider, Cisco Intersight and Software-defined storage solution with Cisco Intersight.
The Cisco Intersight platform provides benefits for the entire infrastructure lifecycle. Cisco UCS Manager, Cisco UCS Director, and Cisco Integrated Management Controller (IMC) focus on day-0 and day-1 activities related to deployment and configuration. The Cisco Intersight platform augments this focus with benefits for day 2 and beyond as well. In addition to monitoring and alerting, Cisco Intersight provides integration with the Cisco® Technical Assistance Center (TAC), predictive analytics, and resource optimization to address ongoing operations and systems upgrades.
Figure 1 shows a high-level view of multiple management stages with the Cisco Intersight platform.
Cisco Intersight infrastructure lifecycle
Terraform is software for building complex, version-controlled, and collaborative infrastructure with high productivity. The infrastructure Terraform can manage includes low-level components such as computing instances, storage, and networking, as well as high-level components such as Domain Name System (DNS) entries and software-as-a-service (SaaS) features.
The set of files used to describe infrastructure in Terraform is known as a Terraform configuration. The configuration is written using HashiCorp Configuration Language (HCL), a simple human-readable configuration language, to define a desired topology of infrastructure resources.
The Terraform provider allows organizations to develop Cisco Intersight resources as self-service infrastructure using code rather than manual provisioning.
This approach provides several benefits:
● You can more quickly and easily scale Cisco Intersight resources. You can provision infrastructure in minutes, with little effort, using the automated workflows, performing the same tasks that used to take days.
● The operating model of Terraform is well suited for the Cisco Intersight platform, because it accommodates the shift from static to dynamic infrastructure provisioning. For example, if a resource is deleted in the Terraform configuration, it will be reflected in the Cisco Intersight platform when the new configuration is applied.
● Terraform maintains a state file, which is a record of the currently provisioned resources. State files provide a version history of Cisco Intersight resources, enabling a detailed audit trail of changes.
The Cisco Intersight platform provides a comprehensive architecture for deploying and managing software-defined storage. Figure 2 shows the hardware design for a generic software-defined storage solution.
Solution hardware design
The design includes the following components:
● The Cisco Intersight platform is deployed as a SaaS solution.
● Controller and load-balancer nodes are deployed on Cisco UCS C220 M5 servers.
● Storage nodes are deployed on Cisco UCS C240 M5 servers.
● Cisco UCS C240 and C220 servers are connected to Cisco Nexus® 93240YC-FX2 Switches with 25-Gbps line speed.
The deployment of the whole solution consists of several main steps:
● Create an account on the Cisco Intersight platform and claim all storage nodes.
● Create a Terraform configuration environment for the Cisco Intersight platform.
● Apply Terraform configurations for the following:
◦ Update firmware on all storage nodes.
◦ Create server policies and profiles.
◦ Deploy server profiles for all nodes.
◦ Install the operating system on all nodes.
● Deploy software-defined storage software.
You create an account on the Cisco Intersight platform by claiming a Cisco IMC device. Before you claim the device, you should perform a preliminary check of the Cisco Intersight device connector in the IMC. The current device connector requirements are described in https://www.cisco.com/c/en/us/td/docs/unified_computing/Intersight/b_Cisco_Intersight_Appliance_Getting_Started_Guide/b_Cisco_Intersight_Appliance_Install_and_Upgrade_Guide_chapter_010.pdf and summarized in Table 1.
Table 1. Device connector requirements
Minimum software version
Supported device connector version
Releases that include supported device connectors
Cisco UCS Manager
Release 4.0(2a) or later
Cisco IMC Supervisor
For M5 servers: Release 3.1(3a)
For M4 servers: Release 3.0(4)
Release 4.0(2c) or later
Cisco HyperFlex Connect and Cisco HyperFlex HX Data Platform
Release 3.5(2a) or later
To create an account on the Cisco Intersight platform, follow these steps:
1. Log in to https://www.intersight.com/.
2. Sign in with your Cisco ID (Figure 3).
Cisco Intersight main screen
3. Read the offer description and accept it.
4. Copy the device ID and claim code in the Cisco Intersight Account Creation wizard from the Cisco IMC.
5. Enter a name for the account and click Create.
The device claim process can take a few minutes. If required, the device connector will automatically be upgraded as part of the process. Account creation and basic configuration are now finished.
6. On the Cisco Intersight dashboard, choose Devices > Claim a New Device (Figure 4).
Claiming a new device
7. Copy the device ID and claim code for the second Cisco UCS C240 M5L server and click Claim.
8. Repeat the same process for the third Cisco UCS C240 M5L server and for the Cisco UCS C220 M5 servers.
9. After you claim all the devices, you should see the devices listed under the server tab at the left.
Create a Terraform configuration environment for the Cisco Intersight platform
Next you create the Terraform configuration environment.
On the deployment host, make sure that Terraform is installed. The Terraform binary is available for major distributions.
1. Download the Terraform zip file from https://www.terraform.io/downloads.html according to the operating system of your deployment host.
2. Extract the zip file and move it to a directory of your choice.
3. Add the path of the directory to the PATH variable of the system.
For more information about how to install Terraform, see https://learn.hashicorp.com/terraform/getting-started/install.html.
Use the following command to clone the repository on your deployment host. This repository contains code to deploy the whole solution.
After the repository has been cloned, you should see five directories: firmware_update, create_infra, provision_infra, os_deployment, and unbind_profiles. Separate directories are created for logical separation and easy understanding of workflows.
Copy the Terraform provider binary file
The Terraform binary file for the Cisco Intersight platform is built for Microsoft Windows, Darwin (Mac OS), and Linux 64-bit architecture systems. Choose the appropriate binary file according to the operating system in your deployment host.
Follow these steps:
5. Navigate to the Releases tab. Download the terraform-provider-intersight.zip file for the latest release.
6. Extract the files and navigate to the terraform-provider-intersight binary file for the operating system installed on your deployment host.
7. Copy the terraform-provider-intersight binary file to all the directories of the downloaded repository.
To use the Cisco Intersight provider, you need an API key, a secret key, and the Cisco Intersight endpoint URL. To generate the keys, follow these steps:
8. Log in to https://www.intersight.com/.
9. On the Settings screen, click the Settings menu.
10. On the General page, choose API > API Keys > Generate API Key.
11. On the Generate New API Key screen, enter the purpose for the API key and click Generate. The API key ID and RSA private key are displayed.
12. Copy the API key.
13. Save the private key information in a .pem file. Save it in a location in the downloaded repository.
Now define the Cisco Intersight provider.
Navigate to the create_infra directory in the cloned repository and open main.tf file. Enter the API key you copied. Also provide the secret key file. The endpoint changes if you are using Cisco Intersight appliance.
For simplicity in this document, multiple directories are defined for each workflow (to create infrastructure, apply server profiles, upgrade firmware, etc.). Copy the same main.tf file in all the directories. You can also combine all the Terraform configurations in one directory.
You need to define some of the basic inputs required for various workflows to provision the infrastructure. You will define all these as variables in a file named variables.tf.
Each configuration takes a value in the variables.tf file as shown here:
You need to keep several image files, such as the operating system image file, Cisco Server Configuration Utility (SCU), and Cisco Host Update Utility (HUU), in a remote server for firmware upgrades and operating system installation. Download the files from software.cisco.com and keep them in a local Network File System (NFS), HTTP, or Common Internet File System (CIFS) share that is accessible to the Cisco IMC for the servers. Then you can edit the variables for the remote server and images.
In variables.tf, define all the VLAN IDs that you need for the management, client, cluster, and replication networks.
Now you need the managed object ID (MOID) for the organization and claimed storage nodes. To get these values, follow these steps:
14. Log in to https://www.intersight.com/.
15. From the Help screen, click Get More Help from Cisco Intersight.
16. Navigate to API Documentation.
17. Click API Reference (Figure 5).
18. Search for compute/PhysicalSummary.
19. Click GET: Read a 'compute.PhysicalSummary' resource (Figure 6).
Supported methods for compute/PhysicalSummary resource
20. In the representational state transfer (REST) client, click Send to get a response from the compute/PhysicalSummary API (Figure 7).
REST Client screen
21. The response is an array of physical servers claimed with all the information about the claimed servers. The first entry is the MOID for the server. Copy the MOIDs for all the claimed servers and the organization MOID.
22. You have all the basic input required to apply the Terraform configurations. Copy the same varables.tf file in all the directories.
The resource objects names in the configuration are in this format:
For example, intersight_ntp_policy is the resource object for the Network Time Protocol (NTP) policy, and intersight_storage_disk_group_policy is the resource object for the storage disk group policy. The following is an example of a complete resource definition for a server profile and the NTP policy attached to the server profile:
Each resource is assigned a name, which can later be used for tracking and referencing. This name will not be reflected anywhere in the Cisco Intersight platform. It is only for reference among the .tf files. The NTP policy is attached to the server profile created earlier. This is accomplished by referencing the storage-node1 profile in profiles.moid. A resource can point to or reference another resource using the format <resource>.<resource_name>.<property_name>.
TF_LOG is a terraform variable that is used for viewing different categories of logs. By default, this variable is left empty. To view logs for Terraform operations, this variable must be set to DEBUG.
In Mac OS and Linux, you accomplish this with export TF_LOG=debug. In Windows PowerShell, use $env:TF_LOG=“DEBUG”.
Documentation about provider resources and configuration options can be found at https://github.com/cisco-intersight/terraform-provider-intersight/tree/master/website/docs.
You need to run terraform init whenever you start with new Terraform code.
The terraform init command will scan the code, identify the provider you are using, and download the appropriate provider. In the case here, because you copied the provider manually, Terraform use the Cisco Intersight provider.
You next run the terraform plan command.
The plan command allows you to see what Terraform will do to the resources before any changes are actually made. This command provides a good way to check the potential results before you make any changes to the infrastructure.
In the output, the symbols show you the following:
● Resources with a plus sign (+) will be created.
● Resources with a minus sign (-) will be deleted.
● Resources with a tilde (~) will be modified in place.
Finally, run terraform apply to deploy the configuration. Observe that the apply command shows the same output as the plan command. To actually proceed with the plan, you need to type yes to confirm.
Next apply the Terraform configurations to the various workflows.
You should upgrade the firmware for the Cisco UCS C240 M5 servers to a suggested release. On the Cisco Intersight platform, you upgrade firmware using a noninteractive Cisco HUU to upgrade the BIOS, Cisco IMC, PCI adapters, RAID controllers, and other firmware to compatible versions.
You can upgrade firmware either through a network share or using utility storage. In this example, the firmware is upgraded through a network share using the NFS protocol. The HUU image file is mounted directly in the Cisco IMC. Therefore, it requires uninterrupted connectivity between the remote file server and Cisco IMC.
To apply the Terraform configuration for firmware update, follow these steps:
23. Go to the firmware_update directory.
24. Append all the servers that require firmware updates to firmware_update.tf.
25. If running the configuration for the first time, run terraform init.
Create server policies
You need multiple server policies for any generic software-defined storage solution. Table 2 lists the policies required and the corresponding resource objects in the Terraform provider for the Cisco Intersight platform.
Table 2. Terraform provider policies and resource objects
Terraform rResource oObject
Specify the ID of the PCI slot ID in which the Cisco virtual interface card (VIC) adapter is placed.
Create these resources based on the number of networks required for the software-defined storage solution. In the example here, four networks are being created.
Ethernet quality of service
The first resource creates LAN connectivity policy. The second resource creates multiple virtual interfaces.
The purpose of this resource is to create RAID from 2 solid-state disks (SSDs) present in the server. This resource will be used for OS installation.
This resource will include disk group policy and also set policies for virtual drives. This resource will also set storage disks to JBOD mode.
To apply the Terraform configuration to create server policies, follow these steps:
27. Go to the create_infra directory.
28. Update the server_profiles.tf file with the number of server profiles required.
29. If you are running the configuration for the first time, run terraform init.
30. Run terraform apply to initiate the firmware updates.
31. After the updates are applied, verify that all the server profiles are created in the Cisco Intersight portal by navigating to Policies > Server Policies (Figure 8).
Policies created by Terraform configuration
After all the required polices have been created, you need to associate the server profiles with each physical server. To associate the profiles with the servers, follow these steps:
32. Go to the provision_infra directory.
33. Update the server_profiles.tf file with the resource intersight_server_profile for each server profile.
34. For each server profile, define the server with which you want to associate it in the assigned_server attribute.
35. If you are running the configuration for the first time, run terraform init.
36. Run terraform apply to initiate the firmware updates.
37. The process for applying the server profiles takes a few minutes. You can see that association is in progress from the Requests screen (Figure 9).
Server profile association in progress
38. After the server profiles have been applied, verify server profile association in the Cisco Intersight portal by navigating to Profiles > Server Profiles (Figure 10).
Server profile associated with physical servers
If any of the resources needs updating, you first need to unbind the server profile, then make the changes, and then bind the server profiles back again.
Follow these steps to make any changes:
39. Go to the unbind_profiles directory.
40. Update unbind_profiles.tf file with the server profiles that you want to unbind.
41. Run terraform apply.
42. Make any necessary changes in either policies or profiles.
43. Go to provision_infra directory and run terraform apply.
The operating system used for this document is Red Hat Enterprise Linux (RHEL) 7.6. However, you can choose a different operating system based on the Cisco UCS Hardware and Software Compatibility Matrix and support from your storage software vendor.
To install the OS, follow these steps:
44. Go to the os_deployment directory.
45. In repo_setup.tf file, define all the OS images in the resource intersight_softwarerepository_operating_system_file. Use a separate resource for each OS type.
46. In the same file, configure the resource for the Cisco SCU in the intersight_firmware_server_configuration_utility_distributable resource.
47. In os_install.tf file, define the resource intersight_os_install for each node.
48. Run terraform apply to install the OS.
The Cisco Intersight platform evaluates the compatibility of your Cisco UCS and Cisco HyperFlex systems to verify that the hardware and software have been tested and validated by Cisco or Cisco partners. The Cisco Intersight platform reports validation issues after checking the compatibility of the server hardware (server model, CPU, and server firmware version), server software (current OS vendor and OS version), and adapter compliance (adapter model, driver protocol, and driver version for the firmware).
You can manually determine the recommended hardware and firmware versions for your server configuration with the Cisco UCS Hardware and Software Compatibility tool at https://ucshcltool.cloudapps.cisco.com/public/.
The Cisco Intersight platform provides am open-source tool called the OS Discovery Tool to collect the OS and driver information need to evaluate Cisco Hardware Compatibility List (HCL) compliance in Linux operating system versions (Figure 11).
OS Discovery Tool
Follow these steps to evaluate compliance with the HCL:
49. Set up any Linux virtual or physical machine that has access to the on-premise network. Install Python 2.7 on this control node.
50. On this control node, clone the Python software development kit (SDK) repository for the Cisco Intersight platform:
51. Install the Python SDK:
52. In the intersight-python directory, go to os-discovery-tool.
53. Edit the discovery_config_linux.json file to include intersight_api_key and path intersight_secret_file.
54. Run the following script:
55. The script collects information about the operating system and device drivers on all the nodes and routes this information to the Cisco Intersight platform to help evaluate compliance with the HCL. After this process is complete, you should be able to see the compliance status. In the Servers list, choose one of the servers and navigate to the HCL tab. You should see the HCL status (Figure 12).
Hardware and software compatibility status
56. For all the components, you can also see recommended versions and download the drivers by clicking Get Recommended Drivers (Figure 13).
After the server profiles are attached and operating system is installed on all the nodes, you can deploy your software-defined storage solution. Refer to the software-defined storage vendor’s installation guides to deploy the solution.
Typically, you need to install two components: controllers and load balancer nodes and claiming storage nodes.
The Cisco Intersight SaaS platform transforms the way that customers deploy and manage Cisco UCS and Cisco HyperFlex systems. The Cisco Intersight platform supports the Terraform provider, which offers an excellent way to easily build, scale, and manage the lifecycle of any scale-out storage software solution with Cisco UCS servers connected to a switched environment. The Cisco Intersight platform offers resources for updating firmware, configuring profiles, associating profiles with physical servers, managing profiles, and installing the operating system.
For additional information, see the following resources:
● GitHub repository for the solution:
● Cisco Intersight online help: https://intersight.com/help/home
Paniraja Koppa, Cisco Systems, Inc.
Paniraja Koppa is a Technical Marketing Engineer for UCS Solutions. He has more than 13 years of experience with a primary focus on data center technologies such as Cisco UCS, Storage, Operating systems, Automation, Virtualization and Cloud. In his current role at Cisco Systems, he works on best practices, optimization, automation and performance tuning of software defined storage on Cisco UCS platforms. Prior to this, he has led QA efforts for 4 new virtual adapter card’s firmware and software features for Cisco UCS. He also worked as customer support engineer and advocate in the Data Center Virtualization space.
For their support and contribution, the authors would like to thank:
● Vikrant Balyan, Cisco Systems, Inc.
● Aanisha Mishra, Cisco Systems, Inc.
● Chris O'Brien, Cisco Systems, Inc.
● Oliver Walsdorf, Cisco Systems, Inc.
● Jawwad Memon, Cisco Systems, Inc.
● David Soper, Cisco Systems, Inc.