Splunk POD: A Turn-Key Solution for Cisco UCS Infrastructure and Splunk Deployment Guide

 

Published: August 2026

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, go to: https://www.cisco.com/go/designzone

Executive Summary

Splunk POD is an integrated hardware and software solution that combines Cisco UCS servers, Cisco Nexus switches, Cisco Intersight, and Splunk Enterprise platform to deliver an "appliance-like" on-premises deployment experience. This document provides technical guidance for physically connecting the hardware, further configuring servers in Cisco Intersight, and installing the operating system to prepare them to operate as a cluster for Splunk POD.

Key Features

●     Predictable Performance: Cisco-validated S/M/L bundles ensure consistent performance

●     Simplified Deployment: Kubernetes-based automation reduces deployment time from weeks to hours

●     Unified Support: Single vendor support for entire hardware and software stack

●     Enhanced Security: Optional Enterprise Security (ES) integration for advanced threat detection

Solution Overview

This chapter contains the following:

●     Audience

●     Purpose of this document

●     Sizing Options

●     Solution Summary

Splunk POD is a turnkey solution that includes:

●     Splunk Kubernetes Installer

●     Splunk Enterprise platform running in Kubernetes (SOK) created with the Kubernetes Installer for Splunk POD

●     Cisco UCS servers (Cisco UCS C225 and Cisco UCS C240 models)

●     Cisco Nexus 9000 switches

●     Cisco Intersight management

●     Optional: Splunk Enterprise Security (ES)

Audience

The intended audience for this document includes, but is not limited to, sales engineers, field consultants, professional services, IT managers, IT engineers and IT architects, partners, and customers who are interested in deploying Splunk POD.

Purpose of this document

This document provides a step-by-step guide to configure and prepare the hardware for Splunk POD. This guide explains the host, network, and rack configuration requirements.

Sizing Options

Splunk POD offers several pre-defined sizing options to meet the needs of different size organizations (Table 1). These pre-defined options feature different amounts of hot storage (indexer-local storage), and cold storage (SmartStore).

Additionally, cold storage is configured to keep three replicas of data for the Small and Medium options to provide data redundancy. For the Large and Extra-Large options, cold storage uses 4+2 erasure coding. Erasure coding is a technique that splits data into data chunks and parity chunks. In this case, 4+2 means there are 4 data chunks and 2 parity chunks for data protection.

Table 1.             Size Options

Table 2.             Data Retention and Redundancy

Solution Summary

Some of the key benefits of Splunk POD include:

●     Simplified Management and Deployment: Splunk POD combines the ease of server management provided by Cisco Intersight with the UCS platform for automated hardware configuration and deployment with the power of Kubernetes and the Splunk Operator for Kubernetes to provide rapid and simplified deployment. Using POD’s Kubernetes Installer, Splunk can be automatically configured and deployed in about 20 minutes from host/OS configuration using pre-configured specifications that match the POD hardware.

●     High Availability and Reliability: Splunk POD focuses on providing resiliency to customer deployments with hardware and software redundancies. The power of Kubernetes allows self-healing and protection against outages.

●     Real Time Insights and Proactive Monitoring: Using a combination of technologies, Intersight, Splunk, and other open-source tools, Splunk POD provides visibility into the health of the cluster.

●     Accelerated Deployment and Reduced Risk: Validated reference architectures and Cisco Validated Designs (CVDs) provide prescriptive, step-by-step guidance for deploying Splunk Enterprise on Cisco UCS, accelerating time-to-value, and minimizing deployment risks

This architecture for running Splunk Enterprise on Cisco UCS uses the following infrastructure components for compute, network, and storage:

●     Cisco UCS Nexus 9000 Switches

●     Cisco UCS C-Series M8 Series C225 and Cisco UCS C240 Rack Servers

 

Splunk POD servers can be divided into two categories, with additional server types in each category, depending on server role.

Control Plane services consist of three Cisco UCS C225 servers that host the Kubernetes Control Plane services and the local OCI image registries used by the cluster. Two of those Cisco UCS C225s are both control nodes and workers. The third is a dedicated control node.

Workers host the Kubernetes workloads. Workloads are carried out on either only Cisco UCS C225s or Cisco UCS 240s and Cisco UCS C225s depending on which configuration was purchased.

Technology Overview

This chapter contains the following:

●     Cisco Unified Computing System

●     Splunk Enterprise

●     Rook-Ceph Overview

Cisco Unified Computing System

Cisco Unified Computing System (Cisco UCS) is a next-generation data center platform that integrates computing, networking, storage access, and virtualization resources into a cohesive system designed to reduce total cost of ownership and increase business agility.

Cisco UCS C225 M8 Rack Server

The Cisco UCS C225 M8 Rack Server is a versatile general-purpose infrastructure and application server. This high-density, 1RU, single-socket rack server delivers industry-leading performance and efficiency for a wide range of workloads, including virtualization, collaboration, and bare-metal applications. The Cisco UCS C225 M8 Rack Server extends the capabilities of the Cisco UCS Rack Server portfolio. It powers 5th Gen and 4th Gen AMD EPYC Processors with 150 percent more cores per socket designed using AMD’s chiplet architecture. With advanced features such as AMD Infinity Guard, compute-intensive applications will see significant performance improvements and reap other benefits such as power and cost efficiencies.

You can deploy the Cisco UCS C-Series Rack Servers as standalone servers or as part of the Cisco Unified Computing System managed by Cisco Intersight or Cisco UCS Manager to take advantage of Cisco standards-based unified computing innovations that can help reduce your Total Cost of Ownership (TCO) and increase your business agility.

The Cisco UCS C225 M8 Rack Server brings many new innovations to the Cisco UCS AMD Rack Server portfolio. With the introduction of PCIe Gen 5.0 for high-speed I/O, a DDR5 memory bus, and expanded storage capabilities, the server delivers significant performance and efficiency gains that will improve your application performance. For more details, go to: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/ucs-c225-m8-rack-server-ds.html

Cisco UCS C240 M8 Rack Server

The 2RU, 2-socket Cisco UCS C240 M8 RTX PRO Server offers I/O flexibility and larger storage capacity. It combines the fastest Intel processors and is a versatile general-purpose application and infrastructure server delivering leading performance and efficiency for a wide range of workloads, including AI, big-data analytics, databases, collaboration, virtualization, and high-performance computing.

The Cisco UCS C240 M8 Rack Server extends the capabilities of the Cisco UCS rack server portfolio by incorporating Intel Xeon 6 CPUs and NVIDIA RTX PRO Blackwell Server Edition GPUs. It improves security, performance, and efficiency while helping you achieve your sustainability goals with built-in accelerators such as Intel Trust Domain Extensions (TDX), Intel Data Streaming Accelerator (DSA), Intel QuickAssist Technology (QAT), Intel Advanced Matrix Extensions (AMX), and Intel In-Memory Analytics Accelerator (IAA).

For more information, go please see: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/ucs-c240-m8-rack-server-ds.html

Splunk Enterprise

Splunk Enterprise is a software product that enables you to search, analyze, and visualize the data gathered from the components of your IT infrastructure or business. Splunk Enterprise collects data from any source, including metrics, logs, clickstreams, sensors, stream network traffic, web servers, custom applications, hypervisors, containers, social media, and cloud services. It enables you to search, monitor and analyze that data to discover powerful insights across multiple use cases like security, IT operations, application delivery and many more. With Splunk Enterprise, everyone from data and security analyst to business users can gain insights to drive operational performance and business results. Splunk makes it easy to input data from virtually any source — without the limitations of database structures.

Splunk POD runs Splunk Enterprise in Kubernetes via the Splunk Operator for Kubernetes (SOK). SOK provides a scalable, Kubernetes-native solution for deploying and managing Splunk Enterprise. It leverages custom resource objects to streamline operations and ensure high availability.

Rook-Ceph Overview

Ceph is software-defined, distributed storage system that supports block storage and object storage for Splunk. Splunk POD utilizes Ceph for storage for Splunk components such as the Search Heads, License Manager and Cluster Manager, for miscellaneous pod storage, Splunk Operator for Kubernetes app framework and for Splunk SmartStore. Splunk indexer storage remains node-local.

Rook is the Kubernetes operator for Ceph. It automates the management, scaling, and healing of Ceph storage, allowing the easy deployment and management of a Ceph cluster.

Ceph is composed of Monitors, Managers, OSDs, and RGWs:

●     Monitor daemons maintain maps of the cluster state, including the monitor map, manager map, the OSD map, the MDS map, and the CRUSH map. These maps are critical cluster state required for Ceph daemons to coordinate with each other

●     Manager daemons keep track of runtime metrics and the current state of the Ceph cluster

●     Ceph OSDs: stores data, handles data replication, recovery, rebalancing, and provides some monitoring information to Ceph Monitors and Managers

●     RGWs: daemon provides a RESTful gateway between applications and Ceph storage clusters that is S3-compatible

The Kubernetes Installer configuration file allows users to select either replication or the erasure coding level. Documentation and the CVD for XL and Large will recommend this to allow the 1-year retention for large and 180 days for XL. The ordering SKU is sized appropriately to support a general estimate of that much data. 

Figure 2 illustrates the difference between Replica 3 and 4:2 erasure coding.

Solution Design

This chapter contains the following:

●     Hardware Specifications

●     Hardware Inventory and Bill of Materials

●     Physical Components

●     Physical Topology

●     Network Configuration

●     Software Topology

Hardware Specifications

The following tables list the hardware specifications for this solution. Splunk POD utilizes NVMe drives for all indexer storage and miscellaneous Kubernetes pod storage. Cold storage via Splunk SmartStore utilizes NVMe or HDD for reduced cost. Splunk POD uses 2x of the Cisco UCS C225 as dual-purpose controller and worker nodes such that they  are utilizes for both the Kubernetes control plane and as general workers. There is also a single Reserved Controller which is a Cisco UCS C225 Reserved Controller with reduced memory.

Table 3.             Small POD (NVMe)

Table 4.             Small POD (HDD)

Table 5.             Medium POD (NVMe)

Table 6.             Medium POD (HDD)

Table 7.             Large POD (NVMe)

Table 8.             Large POD (HDD)

Table 9.             Extra-Large POD (NVMe)

Table 10.          Extra-Large POD (HDD)

Hardware Inventory and Bill of Materials

This is the current hardware specifications for Splunk POD. The servers in each Splunk POD tier are configured the same. The difference is the quantity of drives in each machine changes within each tier.

Table 11.          Cisco UCS C225 – Controller Plane (Reserved)

Table 12.          Cisco UCS C225

Table 13.          Cisco UCS C240

As previously mentioned, Splunk POD utilizes NVMe drives for all indexer storage and miscellaneous Kubernetes pod storage. Cold storage via Splunk SmartStore utilizes NVMe or HDD for reduced cost. Table 14 lists the type of storage devices utilized by the solution.

Table 14.          Storage Devices

Table 15 and 16 list the type and quantity of each drive as well as the total number of servers depending on which solution was purchased. As previously mentioned, all SKU’s utilize the same machine types as outlined above, and difference is the number of drives in each SKU. They also list the quantity of NVMe drives to be used for Indexer storage.

Table 15.          NVMe SKUs

Table 16.          HDD SKUs

Physical Components

Figure 3 illustrates the Splunk POD rack configuration in a 42U server rack for the Medium NVMe sizing option. Other deployment types are not shown. Each server connects to Cisco 9000 Series Nexus switches at the top of the rack, as detailed in the Physical Topology section.

Figure 4 illustrates the Medium HDD sizing option. Other deployment types are not shown. The HDD option comes with Cisco UCS C240 which contain the HDD.

 

Figure 5 illustrates the Splunk POD rack configuration for the Extra-Large (NVMe) tier.

 

Figure 6 illustrates the Splunk POD Extra-Large (HDD) rack configuration which requires two 42U racks. It is recommended to split the nodes as evenly as possible between the two racks. For example, place half the C240’s on one rack and the other half on the other.

Physical Topology

This reference design shows a typical network configuration for the Medium NVMe sizing option. All servers connect to Nexus switches using 100G cables.

Network Configuration

Host Network

Connect all 100Gbps network cards on each host to the Nexus 9000 switches as shown in Figure 7. Configure the Cisco VIC adapter in Physical NIC mode with an OS HA bond to provide network redundancy for the cluster. The following sections include step-by-step instructions for configuring the VICs.

Console Network

All servers in Splunk POD have a 1Gbps console interface port. This allows you to access the virtual console and must be connected to an out-of-band switch that supports 1Gbps connections.

Software Topology

POD Deployment

Figure 7 illustrates where the software is deployed in relation to the hardware. Splunk POD does not utilize strict placement rules. Scheduling is generally decided by Kubernetes with one notable exception; indexers are deployed to Cisco UCS C225 machines and utilize the /mnt/splunk mount.

Ceph is also configured to use any available disk and does not require specific formatting. Ceph will detect these automatically and configure them appropriately.

Cisco UCS Install and Configure

This chapter contains the following: 

●     Cisco UCS Configuration

●     Configure Policies for Cisco Server Profile Templates for Each Server Role

This chapter details the Cisco Intersight deployed Cisco UCS C225 and Cisco UCS C240 M8 Rack Servers.

Cisco UCS Configuration

Before deploying Splunk, you must configure the UCS servers appropriately. This section contains the required procedures:

●     Procedure 1. Configure Cisco UCS Rack Servers

●     Procedure 2. Claim Rack Servers in Cisco Intersight Platform

●     Procedure 3. Configure Cisco Intersight Account and System Settings

The compute nodes in Cisco UCS are configured using server profiles defined in Cisco Intersight. These server profiles derive all the server characteristics from various policies and templates.

Procedure 1.    Configure Cisco UCS Rack Servers

Step 1.          Configure CIMC To Standalone Mode; monitor the server boot process until you reach the Cisco menu and press F8 to enter Cisco IMC Configuration Utility again.

Step 2.          Apply the following configuration:

-        NIC mode selected to Dedicated

-        IP to IPV4

-        CIMC IP with an IP in the same subnet as your computer

-        NIC redundancy to none

-        No VLAN

-        IP Address for the CIMC

-        Subnet for the CIMC

-        Gateway for the CIMC

-        Password for the Admin user. This password will be used to log into the CIMC Console later.

Step 3.          Press F10 to save changes and reboot the server.

Step 4.          Connect your computer to the physical Management Port on the server and open a web browser.

Step 5.          Use the IP you configured https://x.x.x.x

Step 6.          Log into the CIMC of your server by accessing https://<CIMC IP Address> and using the username and password. The user is "admin" and the password is the password set in Step 1 during CIMC configuration.

Step 7.          Configure the DNS settings by entering your Domain Name and DNS Server.

Step 8.          Configure the NTP server.

Step 9.          Configure any proxy you desire to use by entering its host or IP and the appropriate port.

Step 10.       Pause on the screen containing the device and claim codes. You will use these in the following section.

Procedure 2.    Claim Rack Servers in Cisco Intersight Platform

Step 1.          Go to https://intersight.com/.

Step 2.          Sign in with your Cisco ID or if you don’t have one, click Sign Up and set up your account.

Step 3.          After logging into your Cisco Intersight account, go to System > Targets > Claim a New Target.

Step 4.          For the Select Target Type, select Cisco UCS Server (Standalone) and click Start.

Step 5.          Enter the Device ID and Claim Code which was previously captured on the CIMC page. Click Claim to claim this device in Cisco Intersight.

Step 6.          Repeat this procedure for the rest of your servers.

Procedure 3.    Configure Cisco Intersight Account and System Settings

Step 1.          Go to System > Account Details. For more information, see: https://intersight.com/help/saas/system/settings https://intersight.com/help/saas/system/settings

Step 2.          In the System tab > Select Resource Group. Create New resource group.

Step 3.          Select the Targets to be part of this resource group and click Create.

Step 4.          Use the Spk-Resource group for this solution. Go to System menu, select Organizations then click Create Organization.

Step 5.          Enter the name for the new Organization creation.

Step 6.          (Optional) Check the box to share resources with other organizations. Click Next.

Step 7.          In the configuration option, select the “Spk-Resource” configured earlier and click Next.

Step 8.          Verify the summary page and then click Create to create organization with resource group for this deployment as shown below:

Step 9.          To configure Allow Tech Support Bundle Collection, go to Settings > Security & Privacy > and enable the option and then click Save.

Configure Policies for Cisco Server Profile Templates for Each Server Role

A server profile enables resource management by simplifying policy alignment and server configuration. The server profile wizard groups the server policies into the following categories to provide a quick summary view of the policies that are attached to a profile:

●     Compute Configuration: BIOS, Boot Order, and Virtual Media policies.

●     Management Configuration: Certificate Management, IMC Access, IPMI (Intelligent Platform Management Interface) Over LAN, Local User, Serial Over LAN, SNMP (Simple Network Management Protocol), Syslog and Virtual KVM (Keyboard, Video, and Mouse).

●     Storage Configuration: SD Card, Storage.

●     Network Configuration: LAN connectivity and SAN connectivity policies.

Some of the characteristics of the server profile template for this solution are as follows:

●     BIOS policy is created to specify various server parameters in accordance with CPU manufacturer’s best practices.

●     Boot order policy defines virtual media (KVM mapper DVD) and local boot through M.2 SSD.

●     IMC access policy defines the management IP address pool for KVM access.

●     LAN connectivity policy is used to create a bonded network for increased network performance.

This section contains the following procedures to configure the various Cisco UCS Server Templates. It details:

●     Procedure 1. Create UUID Pool

●     Procedure 2. Configure BIOS Policy

●     Procedure 3. Create Boot Order Policy

●     Procedure 4. Create an Adapter Policy

●     Procedure 5. Create Storage Policy

●     Procedure 6. Configure Ethernet Network Policy

●     Procedure 7. Configure MAC Pool

●     Procedure 8. Configure Ethernet Adapter Policy

●     Procedure 9. Configure Ethernet QoS Policy  

●     Procedure 10. Configure LAN Connectivity Policy

●     Procedure 11. Create Server Profile Template

●     Procedure 12. Assign and Deploy Server Profiles

Procedure 1.    Configure UUID Pool

Step 1.          To create UUID Pool for a Cisco UCS, go to Configure > Pools > and click Create Pool. Then Select UUID.

Step 2.          In the UUID Pool Create section, for the Organization, select “Spk-Org” and for the Policy name “Spk-UUID.” Click Next.

Step 3.          Select Prefix, UUID block and size according to your environment and click Create as shown below:

Procedure 2.    Configure BIOS Policy

Table 17.          BIOS recommendations for Splunk Analytical Enterprise Workloads

 

Step 1.          Go to Configure > Policies > and select Platform type as UCS Server and select on BIOS and click Start.

Step 2.          In the BIOS create general menu, for the Organization, select Spk-Org and for the Policy name enter Default-BIOS. Click Next.

Step 3.          Apply the parameters from the above table to configure the BIOS.

Step 4.          Click Create to create the BIOS policy.

Procedure 3.    Create Boot Order Policy

Step 1.          To configure Boot Order Policy for a Cisco UCS Server Profile template profile, go to Configure > Policies > and click Create Policy. Select UCS Server and then Boot Order. Click Start.

Step 2.          Select Unified Extensible Firmware Interface (UEFI) and select UCS Server (Standalone).

Step 3.          Create the first Boot Device by clicking Add Boot Device. For Device Name, enter “KVM-Mapped-DVD” and for Sub-Type, select KVM MAPPED DVD.

Step 4.          Create the second Boot Device by clicking Add Boot Device again. For Device Name, enter “M2-SSD” and for Slot, enter “MSTOR-RAID.”

Step 5.          Ensure both slots are set to Enabled then click Create.

Procedure 4.    Create An Adapter Policy

In this procedure, you will configure the network adapter policy for the servers

Step 1.          Go to Configure > Policies > and click Create Policy. For the Platform Type, select UCS Server, for the Policy select Adapter Configuration. Click Start.

Step 2.          Select Spk-Org for Organization and name the policy “Spk-Physical-adapter-policy.” Click Next.

Step 3.          Click Add VIC Adapter Configuration.

Step 4.          Update the Configuration with the following parameters:

-        Set the PCI slot to MLOM

-        Disable LLDP

-        Disable FIP

-        Disable Port Channel

-        Enable Physical NIC Mode

-        FEC Mode for all interfaces is cl91

Step 5.          Click Add.

Step 6.          Click Create.

Procedure 5.    Create Storage Policy

In this procedure, you will configure the storage policies for the Cisco UCS C225 and C240’s. Both servers can utilize the same storage policies.

Step 1.          Go to Configure > Policies > and click Create Policy. For the platform type select UCS Server and for the Policy select Storage.

Step 2.          For the Organization, select Spk-Org, and for the name, enter “JBOD-Storage-Policy” Click Next.

Step 3.          In the Policy Details section select UCS Server (Standalone) as the policy.

Step 4.          Enter in the following options:

-        Use JBOD drives for Virtual Drive creation: Disabled

-        Unused Disks State: No Change

-        Default Drive State: JBOD

-        M.2 RAID Configuration: MStorBootVD

-        Slot of the M.2 RAID Controller: MSTOR-RAID-1

Step 5.          Click Create.

Procedure 6.    Configure Ethernet Network Policy

Step 1.          To configure the Ethernet Network Group Policy for the UCS server profile, go to > Configure > Policies > and click Create Policy.

Step 2.          For the platform type select UCS Server (Standalone) and for the policy select Ethernet Network.

Step 3.          For the Organization select Spk-Org and for the name input “Eth-Network.”

Step 4.          Select Access. For the VLAN Mode, Disable the toggle “Enable QinQ Tunneling” and leave the Default VLAN empty.

Step 5.          Click Create.

Procedure 7.    Configure MAC Pool

Step 1.          To configure a MAC Pool for a Cisco UCS Domain profile, go to > Configure > Pools > and click Create Pool. Select option MAC to create MAC Pool.

Step 2.          In the MAC Pool Create section, for the Organization, select Spk-Org and for the Policy name Spk-MAC-Pool. Click Next.

Step 3.          Enter the MAC Blocks From and Size of the pool according to your environment and click Create.

Procedure 8.    Configure Ethernet Adapter Policy

Step 1.          To configure the Ethernet Adapter Policy for the UCS Server profile, go to > Configure > Policies > and click Create Policy.

Step 2.          For the platform type select UCS Server and for the policy select Ethernet Adapter.

Step 3.          In the Ethernet Adapter Configuration section, for the Organization select Spk-Org and for the policy name enter Eth-Adapter.

Step 4.          Select Cisco Provided Ethernet Adapter Configuration and click the option “Select Cisco Provided Configuration” and then click Linux as shown below:

Step 5.          In the Policy Details section, for the recommended performance on the ethernet adapter, keep the “Interrupt Settings” parameter.

Step 6.          Click Create to create this policy.

Procedure 9.    Configure Ethernet QoS Policy

Step 1.          In the Create Ethernet QoS Configuration section, for the Organization select Spk-Org and for the policy name enter “Eth-QoS-9000.” Click Next.

Step 2.          Enter QoS Settings as shown below to configure 9000 MTU for replication vNIC traffic.

Step 3.          Click Create.

Procedure 10.    Configure LAN Connectivity Policy

Step 1.          Go to > Configure > Policies and click Create Policy. For the platform type, select “UCS Server” and for the policy select “LAN Connectivity.”

Step 2.          In the LAN Connectivity Policy Create section, for the Organization select “Spk-Org” for the policy name enter “Spk-LAN-Connectivity” and for the Target Platform select UCS Server (Standalone). Click Next.

Step 3.          In the Policy Details click Add vNIC.

Step 4.           In the Add vNIC section, for the first vNIC enter eno5. In the Edit vNIC section, for the vNIC name enter "eno5" and for the MAC Pool select Spk-MAC-Pool. In the Placement option, select Slot ID as MLOM, PCI Link as 0, and PCI Order as 0. Also disable the Failover option.

Step 5.          For the Ethernet Network Policy, select Eth-Network. For the Ethernet Network Control Policy select Eth-Network-Control. For Ethernet QoS, select Eth-QoS-9000, and for the Ethernet Adapter, select Eth-Adapter. Click Add to add eno5 to this policy.

Step 6.          Add a second vNIC. For the name enter "eno6" and for the MAC Pool select Spk-MAC-Pool. In the Placement option, select Slot ID as MLOM. Again, disable the Failover option.

Step 7.          For the Ethernet Network Policy, select Eth-Network. For the Ethernet Network Control Policy select Eth-Network-Control. For Ethernet QoS, select Eth-QoS-9000, and for the Ethernet Adapter, select Eth-Adapter. Click Add to add eno6 to this policy.

Step 8.          Click Create to create the policy.

Procedure 11.    Create Server Profile Templates

Create server profile templates for the servers. All servers will use the same template

Step 1.          Go to Configure > Templates > UCS Server Profile Templates and click Create Server Profile Template.

Step 2.          For the Organization, select Spk-Org, enter in “Server-Template” for the name, and select UCS Server (Standalone) for the Target Platform. Click Next.

Step 3.          In the Compute Configuration step, select the Default-BIOS policy, and select M.2-Boot for the Boot Order which was created in prior steps. Click Next.

Step 4.          Click Next in the Management Configuration screen.

Step 5.          From Storage Configuration, select the JBOD-Storage-Policy storage policy.

Step 6.          On the Network Configuration screen,  select the Spk-LAN-Connectivity policy

Step 7.          Click Derive Profiles in the Summary screen.

Step 8.          In the following screen, enter the number of profiles to derive. This corresponds to total number of servers in your profile.

Step 9.          Click Next.

Step 10.       In the Details screen, expand the Derive option at the bottom, enter in “Server-Profile-” as the Profile Name Prefix, enter “1” for Digits Count, and “1” for Start Index for Suffix.

Step 11.       Click Next.

Step 12.       Click Derive in the Summary screen.

Procedure 12.    Assign and Deploy Server Profiles

Step 1.          Go to > Configure > UCS Server Profiles > then select the server profile “Server-Profile-1” and choose Assign Server from the drop-down list.

Step 2.          From the Assign Server to UCS server Profile, select the specific Server where you want to apply this server profile and click Assign.

Step 3.          After you assign the server profile to the appropriate server, go to Configure > Profiles > UCS Server Profile > select the same server and click Deploy to configure server as shown below:

Step 4.          Check the boxes and click Deploy to activate the server profile.

Step 5.          Repeat steps 1 – 4 for each server profile.

Install and Configure Red Hat Enterprise Linux 9.6

This chapter contains the following:

●     Install Red Hat Enterprise Linux (RHEL) 9.6

●     Post OS Install

This chapter provides detailed procedures for installing Red Hat Enterprise Linux Server on Cisco UCS C225 and Cisco UCS C240 M8 servers. There are multiple ways to install the RHEL operating system. The installation procedure described in this deployment guide uses ISM automated workflow to install the operating system on all the servers through Intersight. For more information, see: https://intersight.com/help/saas/resources/installing_an_operating_system#performing_os_installation_in _cisco_mode

Install Red Hat Enterprise Linux (RHEL) 9.6

This section contains the following procedures:

●     Procedure 1. Add OS Image Link

●     Procedure 2. Add Server Configuration Utility Image

●     Procedure 3. Install the Operating System

●     Procedure 4. (Optional) Manual Operating System Install

Procedure 1.    Add OS Image Link

Step 1.          Log into Intersight account.

Step 2.          Go to Systems > Software Repository > OS Image Links tab and click the Add OS Image Link icon as shown below:

Step 3.          Add the image source of the operating system along with details of the file share location and the protocol (CIFS/NFS/HTTPS) to the software repository.

Step 4.          Provide the details for Operating System image, modify as required, and save the Operating System image as shown below. Click Add.

Procedure 2.    Add Server Configuration Utility Image

Step 1.          Log into Intersight Account.

Step 2.          Go to Systems > Software Repository > SCU Links tab and click Add SCU Link as shown below:

Step 3.          Review the Server Configuration Utility image details, modify as required, and save the Server Configuration Utility image.

Procedure 3.    Install the Operating System

Step 1.          Log into Intersight Account.

Step 2.          Go to Operate > Servers > and then select a server.

Step 3.          Click the ellipses and select Install Operating System as shown below:

Step 4.          From the Install Operating System menu, make sure all the relevant servers are selected and click Next.

Step 5.          Select the OS Image Link previously configured.

Step 6.          From the Configuration menu, select the configuration sources.

 

Step 7.          From the Configuration menu, provide the details for the RHEL host with the appropriate IP Address, Netmask, Gateway, Preferred Name Server, Hostname, and password then click Next.

Step 8.          From the Server Configuration Utility menu, select the SCU Link previously configured as shown below:

Step 9.          From the Installation Target menu, select M.2 for the Installation target. When you select the M.2 option, all the servers will automatically detect the Boot Drive previously configured into the UCS Boot drive setup.

Step 10.       Click Next and from the Summary menu, verify the details of your selections, make changes where required, and click Install to install the Operating System.

Step 11.       To check the status of the task, click Request then click the individual task to see the execution flow as shown below:

Step 12.       After the OS installation finishes, reboot the server, and complete the appropriate registration steps.

Procedure 4.    (Optional) Manual Operating System Install Example

Step 1.          Download the Red Hat Enterprise Linux 9.6 OS image and save the ISO file to local disk.

Step 2.          Launch the vKVM console on your server by going to Cisco Intersight > Operate > Servers > click one of the server nodes and then from the Actions drop-down list select Launch vKVM.

Step 3.          Click Accept security and open KVM. Click Virtual Media > vKVM-Mapped vDVD. Click Browse and map the RHEL ISO image, click Open and then click Map Drive. After mapping the ISO file, click Power > Power Cycle System to reboot the server.

Step 4.          During the server boot order, it detects the virtual media connected as RHEL ISO DVD media and it will launch the RHEL OS installer.

Step 5.          Select language and for the Installation destination assign the local virtual drive. Apply the hostname and click Configure Network to configure any or all the network interfaces. Alternatively, you can configure only the “Public Network” in this step. You can configure additional interfaces as part of post OS install steps.

Step 6.          After the OS installation finishes, reboot the server, and complete the appropriate registration steps.

Step 7.          Repeat steps 1-6 on all server nodes and install RHEL 9.6 on all the server nodes.

Post OS Install

This section contains the following procedures:

●     Procedure 1. Set up Remote Login from the Bastion

●     Procedure 2. Disable the Linux Firewall

●     Procedure 3. Upgrade Cisco UCS VIC Driver for Cisco UCS VIC

●     Procedure 4. Configure Chrony

●     Procedure 5. Disable Transparent Huge Pages

●     Procedure 6. Configure Indexers

●     Procedure 7. Optional Disable C-States in Linux for C240 Servers

●     Procedure 8. Create Bonded Network Interface

●     Procedure 9. Run the Kubernetes Installer for Splunk POD

Procedure 1.    Set up Remote Login from the Bastion

To manage all the nodes in a cluster from the Bastion node, SSH keys for remote access needs to be set up. This is required for the Kubernetes Installer for Splunk POD to function correctly.

This example deploys an open SSH key. It is strongly recommended to use a passphrase protected SSH key and the toolchain.

Step 1.          Log into the Bastion:

Step 2.          Run the ssh-keygen command to create both public and private keys on the admin node:

Step 3.          Create an Ansible inventory file containing all hosts in the cluster. Example below: 

Step 4.          Ensure that the service account user has passwordless sudo. This is required to install and manage Kubernetes.

The result should look like this for all servers:

Step 5.          When that is created, run the following command from the admin node to copy the public key id_rsa.pub to all the nodes of the cluster. ssh-copy-id appends the keys to the remote-hosts .ssh/authorized_keys

Step 6.          Enter the password of the remote host(s). Ensure the key has been deployed to all servers.

Step 7.          Enable RHEL subscription for all machines:

Procedure 2.    Disable the Linux Firewall

Step 1.          Run the following commands to disable the Linux firewall:

Procedure 3.    Upgrade Cisco UCS VIC Driver for Cisco UCS VIC

The latest Cisco Network driver is required for performance and updates. The latest drivers can be downloaded from here: https://software.cisco.com/download/home/283862063/type/283853158/release/4.3(5e)

Step 1.          In the ISO image, the required driver can be located here:

Step 2.          From a node connected to the Internet, download, extract, and transfer “kmod-enic-*.rpm to Bastion server.

Step 3.          Copy the rpm on all nodes of the cluster using the following Ansible commands. For this example, the rpm is assumed to be in present working directory of the Bastion host:

Step 4.          Use the yum module to install the “enic” driver rpm file on all the nodes through Ansible:

Step 5.          Make sure that the above installed version of “kmod-enic” driver is being used on all nodes by running the command "modinfo enic" on all nodes:

Procedure 4.    Configure Chrony

Step 1.          Edit “/etc/chrony.conf” file:

Step 2.          Copy “chrony.conf” file from the admin node to the “/etc/” of all nodes by running command below:

Step 3.          Start Chrony service. Adjust timezone accordingly for the POD:

Procedure 5.    Disable Transparent Huge Pages

Step 1.          Run the following command:

Step 2.          Update the kernels to disable transparent huge pages permanently. This command should be run on all servers:

Procedure 6.    Configure Indexers

Cisco UCS C225 hosts ship with identical drive configurations, regardless of role. Each host has a RAID1 m2 boot volume, as configured above, and a varying amount of unconfigured drives depending on the SKU. We will configure these drives into a volume for consumption by Splunk indexers using mdadm.

Step 1.          Use lsblk to determine the drive names which are not the root volume:

 

Step 2.          Use mdadm to create a RAID5 device across the drives:

Step 3.          Format the resulting RAID5 device as ext4. This command is non-destructive by default. If a drive is already formatted, it will error without wiping the partition.

Example Shell command

Step 4.          Update /etc/fstab and mount volumes:

Procedure 7.    Optional Disable C-States in Linux for C240 Servers

We configured a BIOS policy that disabled C-States at the BIOS level. In this section, we will configure Linux to disable C-States to guarantee expected behavior on UCS C240 machines in the event of BIOS drift.

Step 1.          Disable C-States via Grubby for Linux C240 servers:

Procedure 8.    Create Bonded Network Interface

This procedure details how to configure a bonded interface called bond0.

Step 1.          Create bond from the two virtual NICs:

Step 2.          Step MTU (Maximum Transmission Unit) on eno5 and eno6 to 9000 (jumbo frames):

Step 3.          Start the interfaces:

Step 4.          Repeat for all nodes in the cluster.

Procedure 9.    Run the Kubernetes Installer for Splunk POD

Step 1.          Run the Splunk Kubernetes Installer once you have successfully prepared the hardware. The installer contains all necessary dependencies to install and run Splunk on Kubernetes on your recently configured hardware.

Step 2.          Access the Splunk POD documentation here: https://help.splunk.com/?resourceId=Splunk_POD_overview and proceed with the steps detailed in the Deploy Splunk POD section to continue configuring the software required for Splunk POD.

Conclusion

Cisco UCS provides a tightly integrated platform combining compute, storage, and networking, which is purpose-built to support high-performance, scalable workloads like Splunk. This integration ensures predictable performance and high availability for Splunk Enterprise deployments. Deploying Splunk Enterprise on Cisco UCS servers managed by Cisco Intersight offers a unified, high-performance, and scalable infrastructure for operational analytics.

About the authors

Eugene Minchenko, Global Solutions Engineer Cisco Systems, Inc.

Eugene Minchenko is a Global Solution Architect at Cisco Systems with over 15 years of experience in architecting and implementing enterprise data center, AI and security solutions for global customers. He holds a master’s in computer science and specializes in sales engineering, developing reference architectures, and authoring technical documentation for data centers and cloud workloads.

Michael Guenther, Principal Software Engineer, Splunk LLC.

Michael Guenther is a Principal Engineer at Splunk responsible for the architecture of the Splunk POD solution. He has worked with numerous distributed systems. He joined Splunk in 2017 and has worked extensively on a wide range of Splunk products.

Michael Lusher, Software Engineer, Splunk LLC.

Michael Lusher is a Software Engineer at Splunk working on the Splunk POD solution. He joined Splunk in 2021 and has worked on a variety of Splunk products.

Acknowledgements

For their support and contribution to the design, validation, and creation of this Cisco Validated Design, the authors would like to thank

●     Hardikkumar Vyas, Technical Marketing Engineer, Cisco Systems, Inc.

CVD Program

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS X-Series, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study,  LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trade-marks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. (LDW_U2_P4)

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)