Splunk POD: A Turn-Key Solution for Cisco UCS Infrastructure and Splunk Deployment Guide

 

Published: March 2026

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, go to: https://www.cisco.com/go/designzone

Executive Summary

Splunk POD is an integrated hardware and software solution that combines Cisco UCS servers, Cisco Nexus switches, Cisco Intersight, and Splunk Enterprise platform to deliver an "appliance-like" on-premises deployment experience. This document provides technical guidance for physically connecting the hardware, further configuring servers in Cisco Intersight, and installing the operating system to prepare them to operate as a cluster for Splunk POD.

Key Features

●     Predictable Performance: Cisco-validated S/M/L bundles ensure consistent performance

●     Simplified Deployment: Kubernetes-based automation reduces deployment time from weeks to hours

●     Unified Support: Single vendor support for entire hardware and software stack

●     Enhanced Security: Optional Enterprise Security (ES) integration for advanced threat detection

Solution Overview

This chapter contains the following:

Audience

Purpose of this document

Sizing Options

Solution Summary

Splunk POD is a turnkey solution that includes:

●     Splunk Kubernetes Installer

●     Splunk Enterprise platform running in Kubernetes (SOK) created with the Kubernetes Installer for Splunk POD

●     Cisco UCS servers (Cisco UCS C225 and Cisco UCS C245 models)

●     Cisco Nexus 9000 switches

●     Cisco Intersight management

●     Optional: Splunk Enterprise Security (ES)

Audience

The intended audience for this document includes, but is not limited to, sales engineers, field consultants, professional services, IT managers, IT engineers and IT architects, partners, and customers who are interested in deploying Splunk POD.

Purpose of this document

This document provides a step-by-step guide to configure and prepare the hardware for Splunk POD. This guide explains the host, network, and rack configuration requirements.

Sizing Options

Splunk POD offers 3 pre-defined sizing options to meet the needs of different size organizations (Table 1).    

Table 1.           Size Options

Data Retention Requirements:

●     90 days in hot/warm cache (local storage)

●     1-year total retention (SmartStore)

●     3 copies (replicas) of data stored in SmartStore to prevent data loss

Solution Summary

Some of the key benefits of Splunk POD include:

●     Simplified Management and Deployment: Splunk POD combines the ease of server management provided by Cisco Intersight with the UCS platform for automated hardware configuration and deployment with the power of Kubernetes and the Splunk Operator for Kubernetes to provide rapid and simplified deployment. Using POD’s Kubernetes Installer, Splunk can be automatically configured and deployed in about 20 minutes from host/OS configuration using pre-configured specifications that match the POD hardware.

●     High Availability and Reliability: Splunk POD focuses on providing resiliency to customer deployments with hardware and software redundancies. The power of Kubernetes allows self-healing and protection against outages.

●     Real Time Insights and Proactive Monitoring: Using a combination of technologies, Intersight, Splunk, and other open-source tools, Splunk POD provides visibility into the health of the cluster.

●     Accelerated Deployment and Reduced Risk: Validated reference architectures and Cisco Validated Designs (CVDs) provide prescriptive, step-by-step guidance for deploying Splunk Enterprise on Cisco UCS, accelerating time-to-value, and minimizing deployment risks

This architecture for running Splunk Enterprise on Cisco UCS uses the following infrastructure components for compute, network, and storage:

●     Cisco UCS Nexus 9000 Switches

●     Cisco UCS C-Series M8 Series C225 and Cisco UCS C245 Rack Servers

Splunk POD servers can be divided into two categories, with additional server types in each category, depending on server role.

Control Plane: Control Plane services consist of Cisco UCS C225 servers:

●     Bastion: A Cisco UCS C225 server that hosts the Kubernetes Installer for Splunk POD as well as the Splunk Deployment server

●     Controllers: Three Cisco UCS C225 servers that host the Kubernetes Control Plane services and the local OCI image registries used by the cluster.

Workers: Workers host the Kubernetes workloads. Apart from the Splunk Agent Management service, all Splunk services are deployed on these machines:

●     Search Heads: Cisco UCS C225 servers sized to support Splunk Search workloads.

●     Indexers: Cisco UCS C245 servers with local NVMe storage to support the deployment of Splunk Indexers with sufficient cache to store ~90 days of data in local cache

●     Volume Servers: Cisco UCS C245 servers with ultra-dense local storage designed to host, at least, one year of data in S3-compliant storage.

Technology Overview

This chapter contains the following:

Cisco Unified Computing System

Splunk Enterprise

SeaweedFS Overview

Cisco Unified Computing System

Cisco Unified Computing System (Cisco UCS) is a next-generation data center platform that integrates computing, networking, storage access, and virtualization resources into a cohesive system designed to reduce total cost of ownership and increase business agility.

Cisco UCS C225 M8 Rack Server

The Cisco UCS C225 M8 Rack Server is a versatile general-purpose infrastructure and application server. This high-density, 1RU, single-socket rack server delivers industry-leading performance and efficiency for a wide range of workloads, including virtualization, collaboration, and bare-metal applications. The Cisco UCS C225 M8 Rack Server extends the capabilities of the Cisco UCS Rack Server portfolio. It powers 5th Gen and 4th Gen AMD EPYC Processors with 150 percent more cores per socket designed using AMD’s chiplet architecture. With advanced features such as AMD Infinity Guard, compute-intensive applications will see significant performance improvements and reap other benefits such as power and cost efficiencies.

You can deploy the Cisco UCS C-Series Rack Servers as standalone servers or as part of the Cisco Unified Computing System managed by Cisco Intersight or Cisco UCS Manager to take advantage of Cisco standards-based unified computing innovations that can help reduce your Total Cost of Ownership (TCO) and increase your business agility.

The Cisco UCS C225 M8 Rack Server brings many new innovations to the Cisco UCS AMD Rack Server portfolio. With the introduction of PCIe Gen 5.0 for high-speed I/O, a DDR5 memory bus, and expanded storage capabilities, the server delivers significant performance and efficiency gains that will improve your application performance. For more details, go to: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/ucs-c225-m8-rack-server-ds.html

Cisco UCS C245 M8 Rack Server

The Cisco UCS C245 M8 Rack Server is perfectly suited for a wide range of storage and I/O-intensive applications such as big data analytics, databases, collaboration, virtualization, consolidation, AI/ML, and high-performance computing supporting up to two AMD CPUs in a 2RU form factor.

The Cisco UCS C245 M8 Rack Server extends the capabilities of the Cisco UCS Rack Server portfolio. It powers 5th Gen and 4th Gen AMD EPYC Processors with up to 160 cores per socket designed using AMD’s chiplet architecture. With advanced features like AMD Infinity Guard, compute-intensive applications will see significant performance improvements and will reap other benefits such as power and cost efficiencies.

Cisco UCS C245 M8 Rack Servers can be deployed as part of a Cisco UCS–managed environment, through Cisco Intersight, or standalone. When used in combination with Cisco Intersight, the Cisco UCS C245 M8 brings the power and automation of unified computing to enterprise applications, including Cisco Single Connect technology, drastically reducing switching and cabling requirements.

For more information, go to: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/ucs-c245-m8-rack-server-ds.html

Splunk Enterprise

Splunk Enterprise is a software product that enables you to search, analyze, and visualize the data gathered from the components of your IT infrastructure or business. Splunk Enterprise collects data from any source, including metrics, logs, clickstreams, sensors, stream network traffic, web servers, custom applications, hypervisors, containers, social media, and cloud services. It enables you to search, monitor and analyze that data to discover powerful insights across multiple use cases like security, IT operations, application delivery and many more. With Splunk Enterprise, everyone from data and security analyst to business users can gain insights to drive operational performance and business results. Splunk makes it easy to input data from virtually any source — without the limitations of database structures.

Splunk POD runs Splunk Enterprise in Kubernetes via the Splunk Operator for Kubernetes (SOK). SOK provides a scalable, Kubernetes-native solution for deploying and managing Splunk Enterprise. It leverages custom resource objects to streamline operations and ensure high availability.

SeaweedFS Overview

SeaweedFS is an S3-compatible distributed storage system for blobs, objects, files and more. Splunk POD utilizes SeaweedFS for the Splunk Operator for Kubernetes app framework and for Splunk SmartStore. It is a separate storage component from Splunk indexer storage. SeaweedFS follows a simple architecture. It is composed of a master server which coordinates volume management and metadata, volume pods which manage actual data, and filers which handle file operations and the S3 API. Splunk POD deploys 3 master pods, at least 3 filer pods, and 1 volume pod per Volume Server. SeaweedFS creates 3 replicas of every object across nodes and is safe from the simultaneous loss of 2 Volume Servers.

Solution Design

This chapter contains the following:

Hardware Specifications

Hardware Inventory and Bill of Materials

Physical Components

Physical Topology

Network Configuration

Software Topology

Hardware Specifications

The following tables list the hardware specifications for this solution.

Table 2.           Small POD

Table 3.           Medium POD

Table 4.           Large POD

Hardware Inventory and Bill of Materials

This is the current hardware specifications for Splunk POD, divided by server class.

Table 5.           Control Plane - Bastion

Table 6.           Control Plane – Kubernetes Controllers

Table 7.           Worker – Search Head

Table 8.           Worker – Indexers

Table 9.           Worker – Volume Servers

Physical Components

Figure 2 illustrates the Splunk POD rack configuration in a 42U server rack for the Medium sizing option. Other deployment types are not shown. Each server connects to Cisco 9000 series Nexus switches at the top of the rack, as detailed in the Physical Topology section.

Physical Topology

This reference design shows a typical network configuration for the Medium sizing option. All servers connect to Nexus switches using 100G cables.

Network Configuration

Host Network

Connect all 100Gbps network cards on each host to the Nexus 9000 switches as shown in Figure 3. Configure the virtual interface cards in physical NIC mode with an OS HA bond to provide network redundancy for the cluster. The following sections include step-by-step instructions for configuring the VICs.

Console Network

All servers in Splunk POD have a 1Gbps console interface port. This allows you to access the virtual console and must be connected to an out-of-band switch that supports 1Gbps connections.

Software Topology

POD Deployment by Host Type

Figure 4 illustrates where the software is deployed in relation to the hardware. Splunk POD deploys each component to different hosts:

●     Search Heads are deployed to Cisco UCS C225 machines and utilize the /data/shared mount

●     Indexers are deployed to Cisco UCS C245 machines and utilize the /data/shared mount

●     SeaweedFS volume pods are deployed to Cisco UCS C245 machines and utilize the /data/storage mount

Splunk POD uses a set of rules to assign pods to specific hosts:

●     Strict Separation (Protected Components)

◦       Indexer pods never co-locate with other indexer pods

◦       Search head pods never co-locate with other search head pods

◦       SeaweedFS volume pods never co-locate with other SeaweedFS volume pods

◦       These components are given dedicated hosts when possible

●     Flexible Placement (other components)

◦       Cluster Manager, License Manager, Deployer, Monitoring Console

◦       Can co-locate with protected components if CPU/memory resources allow

◦       May vary in placement between cluster deployments

●     Fallback Behavior

◦       If insufficient hosts with expected device names/mounts exist, pods will still schedule

◦       Allows for emergency node replacement scenarios

◦       Placement rules are soft constraints, not hard requirements

Storage Diagram Deployment

Figure 5 illustrates the architecture and deployment topology of the storage system in Splunk POD.

Cisco UCS Install and Configure

This chapter contains the following: 

Cisco UCS Configuration

Configure Policies for Cisco Server Profile Templates for Each Server Role

This chapter details the Cisco Intersight deployed Cisco UCS C225 and C245 M8 Rack Servers.

Cisco UCS Configuration

Before deploying Splunk, you must configure the UCS servers appropriately. This section contains the required procedures:

●     Procedure 1. Configure Cisco UCS Rack Servers

●     Procedure 2. Claim Rack Servers in Cisco Intersight Platform

●     Procedure 3. Configure Cisco Intersight Account and System Settings

The compute nodes in Cisco UCS are configured using server profiles defined in Cisco Intersight. These server profiles derive all the server characteristics from various policies and templates.

Procedure 1.    Configure Cisco UCS Rack Servers

Step 1.             Configure CIMC To Standalone Mode; monitor the server boot process until you reach the Cisco menu and press F8 to enter to Cisco IMC Configuration Utility again.

Step 2.             Apply the following configuration:

-       NIC mode selected to Dedicated

-       IP to IPV4

-       CIMC IP with an IP in the same subnet as your computer

-       NIC redundancy to none

-       No VLAN

-       IP Address for the CIMC

-       Subnet for the CIMC

-       Gateway for the CIMC

-       Password for the Admin user. This password will be used to log into the CIMC Console later.

Step 3.             Press F10 to save changes and reboot the server.

Step 4.             Connect your computer to the physical Management Port on the server and open a web browser.

Step 5.             Use the IP you configured https://x.x.x.x

Step 6.             Log into the CIMC of your server by accessing https://<CIMC IP Address> and using the username and password. The user is "admin" and the password is the password set in Step 1 during CIMC configuration.

Step 7.             Configure the DNS settings by entering your Domain Name and DNS Server.

Step 8.             Configure the NTP server.

Step 9.             Configure any proxy you desire to use by entering its host or IP and the appropriate port.

Step 10.         Pause on the screen containing the device and claim codes. You will use these in the following section.

Procedure 2.    Claim Rack Servers in Cisco Intersight Platform

Step 1.             Go to https://intersight.com/.

Step 2.             Sign in with your Cisco ID or if you don’t have one, click Sign Up and setup your account.

Step 3.             After logging into your Cisco Intersight account, go to System > Targets > Claim a New Target.

Step 4.             For the Select Target Type, select Cisco UCS Server (Standalone) and click Start.

Step 5.             Enter the Device ID and Claim Code which was previously captured on the CIMC page. Click Claim to claim this device in Cisco Intersight.

Step 6.             Repeat the procedure for the rest of your servers.

Procedure 3.    Configure Cisco Intersight Account and System Settings

Step 1.             Go to System > Account Details. For more information, see: https://intersight.com/help/saas/system/settings https://intersight.com/help/saas/system/settings

Step 2.             In the System tab > Select Resource Group. Create New resource group.

Step 3.             Select the Targets to be part of this resource group and click Create.

Step 4.             Use the Spk-Resource group for this solution. Go to System menu, select Organizations then click Create Organization.

Step 5.             Enter the name for the new Organization creation.

Step 6.             (Optional) Check the box to share resources with other organizations. Click Next.

Step 7.             In the configuration option, select the “Spk-Resource” configured earlier and click Next.

Step 8.             Verify the summary page and then click Create to create organization with resource group for this deployment as shown below:

Step 9.             To configure Allow Tech Support Bundle Collection, go to Settings > Security & Privacy > and enable the option and then click Save.

Configure Policies for Cisco Server Profile Templates for Each Server Role

A server profile enables resource management by simplifying policy alignment and server configuration. The server profile wizard groups the server policies into the following categories to provide a quick summary view of the policies that are attached to a profile:

●     Compute Configuration: BIOS, Boot Order, and Virtual Media policies.

●     Management Configuration: Certificate Management, IMC Access, IPMI (Intelligent Platform Management Interface) Over LAN, Local User, Serial Over LAN, SNMP (Simple Network Management Protocol), Syslog and Virtual KVM (Keyboard, Video, and Mouse).

●     Storage Configuration: SD Card, Storage.

●     Network Configuration: LAN connectivity and SAN connectivity policies.

Some of the characteristics of the server profile template for this solution are as follows:

●     BIOS policy is created to specify various server parameters in accordance with AMD CPU’s best practices.

●     Boot order policy defines virtual media (KVM mapper DVD) and local boot through M.2 SSD.

●     IMC access policy defines the management IP address pool for KVM access.

●     LAN connectivity policy is used to create two virtual network interface cards (vNICs) – One vNIC for Server Node Management and Splunk Data Ingestion Network Traffic, second vNIC for Splunk Indexing Server-to-Server Network Traffic Interface.

This section contains the following procedures to configure the various UCS Server Templates. It details:

●     Procedure 1. Create UUID Pool

●     Procedure 2. Configure BIOS Policy

●     Procedure 3. Create Boot Order Policy

●     Procedure 4. Create an Adapter Policy

●     Procedure 5. Create C245 Indexer Storage Policy

●     Procedure 6. Create C245 Volume Server Storage Policy

●     Procedure 7. Create C225 Server Storage Policy

●     Procedure 8. Create Indexer Server Profile Template

●     Procedure 9. Create Volume Server Profile Template

●     Procedure 10. Create Controllers Server Profile Template

●     Procedure 11. Create Search Head Server Profile Template

●     Procedure 12. Create Bastion Server Profile Template

●     Procedure 13. Assign and Deploy Server Profiles

Procedure 1.    Configure UUID Pool

Step 1.             To create UUID Pool for a Cisco UCS, go to Configure > Pools > and click Create Pool. Select UUID.

Step 2.             In the UUID Pool Create section, for the Organization, select “Spk-Org” and for the Policy name “Spk-UUID.” Click Next.

Step 3.             Select Prefix, UUID block and size according to your environment and click Create as shown below:

Procedure 2.    Configure BIOS Policy

 

Table 10.       BIOS recommendations for Splunk Analytical Enterprise Workloads

Step 1.             Go to Configure > Policies > and select Platform type as UCS Server and select on BIOS and click Start.

Step 2.             In the BIOS create general menu, for the Organization, select Spk-Org and for the Policy name SpkBIOS-Policy. Click Next.

Step 3.             Apply the mentioned parameters from the above table to configure the BIOS for C225 and C245 M8 Server running as Big-Data Analytical workloads.

Step 4.             Click Create to create the BIOS policy.

Procedure 3.    Create Boot Order Policy

Step 1.             To configure Boot Order Policy for a Cisco UCS Server Profile template profile, go to Configure > Polices > and click Create Policy. Select UCS Server and then Boot Order. Click Start.

Step 2.             Select Unified Extensible Firmware Interface (UEFI) and select UCS Server (Standalone).

Step 3.             Create the first Boot Device by clicking Add Boot Device. For Device Name, enter “KVM-Mapped-DVD” and for Sub-Type, select KVM MAPPED DVD.

Step 4.             Create the second Boot Device by clicking Add Boot Device again. For Device Name, enter “M2-SSD” and for Slot, enter “MSTOR-RAID.”

Step 5.             Ensure both slots are set to Enabled then click Create.

Procedure 4.    Create An Adapter Policy

In this procedure, you will configure the network adapter policy for the servers

Step 1.             Go to Configure > Polices > and click Create Policy. For the Platform Type, select UCS Server, for the Policy select Adapter Configuration. Click Start.

Step 2.             Select Spk-Org for Organization and name the policy “Spk-Physical-adapter-policy.” Click Next.

Step 3.             Click Add VIC Adapter Configuration.

Step 4.             Update the Configuration with the following parameters:

-       Set the PCI slot to MLOM

-       Disable LLDP

-       Disable FIP

-       Disable Port Channel

-       Enable Physical NIC Mode

-       FEC Mode for all interfaces is cl91

Step 5.             Click Add.

Step 6.             Click Create.

Procedure 5.    Create Indexer Server Storage Policies

In this procedure, you will configure the storage policies for the Cisco UCS C245 Indexers servers, Cisco UCS C245 Volume servers, and Cisco UCS C225 servers.

Step 1.             Go to Configure > Polices > and click Create Policy. For the platform type select UCS Server and for the Policy select Storage.

Step 2.             For the Organization, select Spk-Org, and for the name, enter “C245-Indexer-RAID1-RAID5.” Click Next.

Step 3.             In the Policy Details section select UCS Server (Standalone) as the policy.

Step 4.             On the same page, toggle the option for “MRAID/RAID Controller Configuration” and then click Add Drive Group.

Step 5.             In the Add Drive Group modal, enter “RAID5” in the Drive Group Name and select RAID5 as the RAID Level. For the Drive Array Span 0, enter in “1-11” then enter in “12” for the Dedicated Hot Spares option.

Step 6.             Click Add Virtual Drive. In the “Add Virtual Drive” modal, enter in the following options:

-       Drive Groups: RAID5

-       Virtual Drive Name: RAID5

-       Expand to Available: Toggled on

-       Strip Size: 64KiB

-       Access Policy: Default

-       Read Policy: No Read Ahead

-       Write Policy: Write Through

-       Disk Cache: Disabled

Step 7.             Click Add.

Step 8.             Click Create when the Drive Group and Virtual Drive have been configured.

Procedure 6.    Create Volume Server Storage Policies

Configure the Volume Server storage policies for the Volume servers with this procedure.

Step 1.             Go to Configure > Polices > and click Create Policy. For the platform type, select UCS Server and for the Policy select Storage.

Step 2.             In the Create section, for the Organization, select Spk-Org, for the policy name enter “C245-Volume-RAID5.” Click Next.

Step 3.             In the Policy Details section select UCS Server (Standalone) as the policy.

Step 4.             On the same page, toggle the option for “MRAID/RAID Controller Configuration” and then click Add Drive Group.

Step 5.             In the Add Drive Group modal, enter “RAID5” in the Drive Group Name and select “RAID5” as the RAID Level. For the Drive Array Span 0, enter in “1-23” then enter in “24” for the Dedicated Hot Spares option.

Step 6.             Click Add Virtual Drive. In the “Add Virtual Drive” modal, enter in the following options:

-       Drive Groups: RAID5

-       Virtual Drive Name: RAID5

-       Expand to Available: Toggled on

-       Strip Size: 64KiB

-       Access Policy: Default

-       Read Policy: No Read Ahead

-       Write Policy: Write Through

-       Disk Cache: Disabled

Step 7.             Click Add.

Step 8.             Click Create when both the Drive Group and Virtual Drive have been configured.

Procedure 7.    Create C225 Storage Policies

Create another storage policy for the Cisco UCS C225 servers. All Cisco UCS C225 machines share the same storage policy.

Step 1.             Go to Configure > Polices > and click Create Policy. For the platform type select UCS Server and for the Policy select Storage.

Step 2.             In the Create section, for the Organization, select Spk-Org for the policy name enter “C225-Storage-RAID1.” Click Next.

Step 3.             In the Policy Details section select UCS Server (Standalone) as the policy.

Step 4.             On the same page, toggle the option for “MRAID/RAID Controller Configuration” and then click Add Drive Group.

Step 5.             In the Add Drive Group modal, enter “RAID1” in the Drive Group Name and select RAID1 as the RAID Level. For the Drive Array Span 0, enter in “1-2.”

Step 6.             Click Add Virtual Drive. In the “Add Virtual Drive” modal, enter in the following options:

-       Drive Groups: RAID1

-       Virtual Drive Name: RAID1

-       Expand to Available: Toggled on

-       Strip Size: 64KiB

-       Access Policy: Default

-       Read Policy: No Read Ahead

-       Write Policy: Write Through

-       Disk Cache: Disabled

Step 7.             Click Add.

Step 8.             Click Create when both the Drive Group and Virtual Drive have been configured. You should have the following policies:

Procedure 8.    Create Server Profile Templates

Create server profile templates for the Indexer, Volume, Bastion, Controller, and Search Head servers. You do this by creating the indexer template and then cloning the rest.

Step 1.             Go to Configure > Templates > UCS Server Profile Templates and click Create Server Profile Template.

Step 2.             For the Organization, select Spk-Org, enter in “Indexer” for the name, and select UCS Server (Standalone) for the Target Platform. Click Next.

Step 3.             In the Compute Configuration step, select the C245-Default-BIOS policy, and select M.2-Boot for the Boot Order which was created in prior steps. Click Next.

Step 4.             Click Next in the Management Configuration screen.

Step 5.             From Storage Configuration, select the C245-Indexer-RAID5 storage policy.

Step 6.             Click Next in the Network Configuration screen.

Step 7.             Click Derive Profiles in the Summary screen.

Step 8.             In the following screen, enter the number of profiles to derive. This corresponds to the number of Indexers in your profile.

Step 9.             Click Next.

Step 10.         In the Details screen, expand the Derive option at the bottom, enter in “Indexer” as the Profile Name Prefix, enter “1” for Digits Count, and “1” for Start Index for Suffix.

Step 11.         Click Next.

Step 12.         Click Derive in the Summary screen.

Procedure 9.    Create a Volume Server Profile Template

Create the Volume Server Template by cloning the Indexer template.

Step 1.             Go to Configure > Templates > UCS Server Profile Templates. Locate the Indexer template we just created and click the 3 dots on the right-hand side. Click Clone.

Step 2.             In the General screen click Next.

Step 3.             In the Details page, change the Clone Name to “Volume.”

Step 4.             Click Clone.

Step 5.             When cloned, click Actions on the top-right and select Edit to configure the cloned the Template.

Step 6.             Click Next to proceed with the cloned settings until you get to Storage Configuration. Click the X icon to Detach the Storage policy.

Step 7.             Now that the previous policy is detached, attach the C245-Volume-RAID5 policy. When selected, click Next in Storage Configuration to proceed.

Step 8.             Click Next on the following screens since you do not need to make any additional changes. When you reach the Summary section, click Derive Profiles.

Step 9.             Change the number of profiles to derive according to the profile you are using. We will change the Number of Profiles to Derive to 4 for this guide since we are configuring the Medium Profile.

Step 10.         In the Details screen, expand the Derive option at the bottom, enter in “Volume” as the Profile Name Prefix, enter “1” for Digits Count, and “1” for Start Index for Suffix.

Step 11.         Click Next.

Step 12.         Click Derive in the Summary screen.

Procedure 10.    Create a Controllers Server Profile Template

Now you will create the Controller Template by cloning the Controller template.

Step 1.             Go to > Configure > Templates > UCS Server Profile Templates. Locate the Volume template, click the ellipses, and click Clone.

Step 2.             In the General screen click Next.

Step 3.             In the Details page, change the Clone Name to “Controller.” Click Clone.

Step 4.             When cloned, click Actions on the top-right and select Edit to configure the cloned the Template.

Step 5.             Click Next to proceed past the General screen to the Compute Configuration page. Click the X icon to detach the cloned BIOS policy.

Step 6.             When detached, you will create a new Boot Policy for the C225 machines. Click Select the BIOS and then click Create Policy.

Step 7.             Enter “C225-Default-BIOS” for the Name and click Next.

Step 8.             Click Next.

Step 9.             When you configured the BIOS Policy, click Next in the Compute Configuration.

Step 10.         Proceed through the Management Configuration screen by clicking Next. When you get to the Storage Configuration screen, modify the storage policy to use the “C225-Storage-Policy.”

Step 11.         Click Next.

Step 12.         Proceed through the Network Configuration screen by clicking Next. When you get to the Summary screen, click Derive Profiles. Enter 3 for the Number of Profiles to derive, choose Assign Later, and click Next.

Step 13.         In Details, enter “Controller” as the Profile Name Prefix, set Digits Count to 1 and Start Index to 1, then click Next and Derive on the Summary screen.

Procedure 11.    Create the Search Head Server Profile Template

Now you will create the Search Head Server Template by cloning the Controller template.

Step 1.             Go to Configure > Templates > UCS Server Profile Templates. Locate the Controller template, click the 3 dots on the right, and click Clone.

Step 2.             In the General screen click Next.

Step 3.             In the Details page, change the Clone Name to “Search Head.” Then click Clone.

Step 4.             When cloned, click Actions on the top-right and select Edit to configure the cloned the Template.

Step 5.             Click Next to proceed until the Summary Screen. Click Derive.

Step 6.             In the Details screen, expand the Derive option at the bottom, enter in “SearchHead” as the Profile Name Prefix, enter “1” for Digits Count, and “1” for Start Index for Suffix.

Step 7.             Click Next.

Procedure 12.    Assign and Deploy Server Profiles

Step 1.             Go to > Configure > UCS Server Profiles > then select the server profile “Indexer1” and choose Assign Server from the drop-down list.

Step 2.             From the Assign Server to UCS server Profile, select the specific Server where you want to apply this server profile and click Assign.

Step 3.             After you assign the server profile to the appropriate server, go to Configure > Profiles > UCS Server Profile > select the same server and click Deploy to configure server as shown below:

Step 4.             Check the boxes and click Deploy to activate the server profile.

Step 5.             Repeat steps 1 – 4 for each server profile.

Install and Configure Red Hat Enterprise Linux 9.6

This chapter contains the following:

Install Red Hat Enterprise Linux (RHEL) 9.6

Post OS Install

This chapter provides detailed procedures for installing Red Hat Enterprise Linux Server on Cisco UCS C225 and Cisco UCS C245 M8 servers. There are multiple ways to install the RHEL operating system. The installation procedure described in this deployment guide uses ISM automated workflow to install the operating system on all the servers through Intersight. For more information, see: https://intersight.com/help/saas/resources/installing_an_operating_system#performing_os_installation_in _cisco_mode

Install Red Hat Enterprise Linux (RHEL) 9.6

This section contains the following procedures:

●     Procedure 1. Add OS Image Link

●     Procedure 2. Add Server Configuration Utility Image

●     Procedure 3. Install the Operating System

●     Procedure 4. (Optional) Manual Operating System Install

Procedure 1.    Add OS Image Link

Step 1.             Log into Intersight account.

Step 2.             Go to Systems > Software Repository > OS Image Links tab and click the Add OS Image Link icon as shown below:

Step 3.             Add the image source of the operating system along with details of the file share location and the protocol (CIFS/NFS/HTTPS) to the software repository.

Step 4.             Provide the details for Operating System image, modify as required, and save the Operating System image as shown below. Click Add.

Procedure 2.    Add Server Configuration Utility Image

Step 1.             Log into Intersight Account.

Step 2.             Go to Systems > Software Repository > SCU Links tab and click Add SCU Link as shown below:

Step 3.             Review the Server Configuration Utility image details, modify as required, and save the Server Configuration Utility image.

Procedure 3.    Install the Operating System

Step 1.             Log into Intersight Account.

Step 2.             Go to Operate > Servers > and then select a server.

Step 3.             Click the ellipses and select Install Operating System as shown below:

Step 4.             From the Install Operating System menu, make sure all the relevant servers are selected and click Next.

Step 5.             Select the OS Image Link previously configured.

Step 6.             From the Configuration menu, select the configuration sources.

 

Step 7.             From the Configuration menu, provide the details for the RHEL host with the appropriate IP Address, Netmask, Gateway, Preferred Name Server, Hostname, and password then click Next.

Step 8.             From the Server Configuration Utility menu, select the SCU Link previously configured as shown below:

Step 9.             From the Installation Target menu, select M.2 for the Installation target. When you select the M.2 option, all the servers will automatically detect the Boot Drive previously configured into the UCS Boot drive setup.

Step 10.         Click Next and from the Summary menu, verify the details of your selections, make changes where required, and click Install to install the Operating System.

Step 11.         To check the status of the task, click Request then click the individual task to see the execution flow as shown below:

Step 12.         After the OS installation finishes, reboot the server, and complete the appropriate registration steps.

Procedure 4.    (Optional) Manual Operating System Install Example

Step 1.             Download the Red Hat Enterprise Linux 9.4 OS image and save the IOS file to local disk.

Step 2.             Launch the vKVM console on your server by going to Cisco Intersight > Operate > Servers > click one of the server node and then from the Actions drop-down list select Launch vKVM.

Step 3.             Click Accept security and open KVM. Click Virtual Media > vKVM-Mapped vDVD. Click Browse and map the RHEL ISO image, click Open and then click Map Drive. After mapping the iso file, click Power > Power Cycle System to reboot the server.

Step 4.             During the server boot order, it detects the virtual media connected as RHEL ISO DVD media and it will launch the RHEL OS installer.

Step 5.             Select language and for the Installation destination assign the local virtual drive. Apply the hostname and click Configure Network to configure any or all the network interfaces. Alternatively, you can configure only the “Public Network” in this step. You can configure additional interfaces as part of post OS install steps.

Step 6.             After the OS installation finishes, reboot the server, and complete the appropriate registration steps.

Step 7.             Repeat steps 1-6 on all server nodes and install RHEL 9.6 on all the server nodes.

Post OS Install

This section contains the following procedures:

●     Procedure 1. Set up Remote Login from the Bastion

●     Procedure 2. Disable the Linux Firewall

●     Procedure 3. Disable SELinux

●     Procedure 4. Upgrade Cisco UCS VIC Driver for Cisco UCS VIC

●     Procedure 5. Configure Chrony

●     Procedure 6. Disable Transparent Huge Pages

●     Procedure 7. Configure File System Bastion, Controllers, Search Heads, and Indexers

●     Procedure 8. Configure File System for Volume Servers

●     Procedure 9. Run the Kubernetes Installer for Splunk POD

Procedure 1.    Set up Remote Login from the Bastion

To manage all the nodes in a cluster from the Bastion node, SSH keys for remote access needs to be setup. This is required for the Kubernetes Installer for Splunk POD to function correctly.

This example deploys an open SSH key. It is strongly recommended to use a passphrase protected SSH key and the toolchain.

Step 1.             Log into the Bastion:

Step 2.             Run the ssh-keygen command to create both public and private keys on the admin node:

Step 3.             Create an Ansible inventory file containing all hosts in the cluster. Example below: 

Step 4.             Ensure that the service account user has passwordless sudo. This is required to install and manage Kubernetes.

The result should look like this for all servers:

Step 5.             When that is created, run the following command from the admin node to copy the public key id_rsa.pub to all the nodes of the cluster. ssh-copy-id appends the keys to the remote-hosts .ssh/authorized_keys

Step 6.             Enter the password of the remote host(s). Ensure the key has been deployed to all servers.

Step 7.             Enable RHEL subscription for all machines:

Procedure 2.    Disable the Linux Firewall

Step 1.             Run the following commands to disable the Linux firewall:

Procedure 3.    Disable SELinux

Step 1.             SELinux can be disabled by editing /etc/selinux/config and changing the SELINUX line to SELINUX=disabled. To disable SELinux, run the following commands. It can also be disabled via the Ansible module:

Step 2.             Reboot the machine, if needed for SELinux to be disabled in case it does not take effect. It can be checked using the following command:

Procedure 4.    Upgrade Cisco UCS VIC Driver for Cisco UCS VIC

The latest Cisco Network driver is required for performance and updates. The latest drivers can be downloaded from here: https://software.cisco.com/download/home/283862063/type/283853158/release/4.3(5e)

In the ISO image, the required driver can be located here: “\Network\Cisco\VIC\RHEL\RHEL9.4\kmodenic-4.8.0.0 1128.4.rhel9u4_5.14.0_427.13.1.x86_64.rpm”

Step 1.             From a node connected to the Internet, download, extract, and transfer “kmod-enic-*.rpm to Bastion server.

Step 2.             Copy the rpm on all nodes of the cluster using the following Ansible commands. For this example, the rpm is assumed to be in present working directory of the Bastion host:

Step 3.             Use the yum module to install the “enic” driver rpm file on all the nodes through Ansible:

Step 4.             Make sure that the above installed version of “kmod-enic” driver is being used on all nodes by running the command "modinfo enic" on all nodes:

Procedure 5.    Configure Chrony

Step 1.             Edit “/etc/chrony.conf” file:

Step 2.             Copy “chrony.conf” file from the admin node to the “/etc/” of all nodes by running command below:

Step 3.             Start Chrony service. Adjust timezone accordingly for the POD:

Procedure 6.    Disable Transparent Huge Pages

Step 1.             Run the following command:

Step 2.             Update the kernels to disable transparent huge pages permanently. This command should be run on all servers:

Procedure 7.    Configure File System for Bastion, Controllers, Search Heads, and Indexers

Cisco UCS C225 hosts ship with identical drive configurations, regardless of role. Each host has a RAID1 m2 boot volume, as configured above, and another RAID1 pairing of data drives. The Cisco UCS C245s that will host the indexers have 12 drives configured as RAID5 with a hot spare. All servers should have those volumes formatted as ext4 and mounted in the same location.

Step 1.             Use fdisk to determine the data drive name. It is the larger volume and is probably /dev/sdb:

Step 2.             Format data drive as ext4. This command is non-destructive by default. If a drive is already formatted, it will error without wiping the partition.

Example Shell command

Step 3.             Update /etc/fstab and mount volumes:

Procedure 8.    Configure File System for Volume Servers

The Cisco UCS C245s that will host the volume servers have 24 drives configured as RAID5 with a hot spare. The volume servers should be using xfs instead of ext4.

Step 1.             Use fdisk to determine the data drive name:

Step 2.             Format data drive as xfs. This command is non-destructive by default. If a drive is already formatted, it will error without wiping the partition.

Step 3.             Update /etc/fstab and mount volumes.

Procedure 9.    Run the Kubernetes Installer for Splunk POD

Step 1.             Run the Splunk Kubernetes Installer once you have successfully prepared the hardware. The installer contains all necessary dependencies to install and run Splunk on Kubernetes on your recently configured hardware.

Step 2.             Access the Splunk POD documentation here: https://help.splunk.com/?resourceId=Splunk_POD_overview

Step 3.             Proceed with the steps detailed in the “Deploy Splunk POD” section to continue configuring the software required for Splunk POD.

Conclusion

Cisco UCS provides a tightly integrated platform combining compute, storage, and networking, which is purpose-built to support high-performance, scalable workloads like Splunk. This integration ensures predictable performance and high availability for Splunk Enterprise deployments. Deploying Splunk Enterprise on Cisco UCS servers managed by Cisco Intersight offers a unified, high-performance, and scalable infrastructure for operational analytics.

About the authors

Eugene Minchenko, Global Solutions Engineer Cisco Systems, Inc.

Eugene Minchenko is a Global Solution Architect at Cisco Systems with over 15 years of experience in architecting and implementing enterprise data center, AI and security solutions for global customers. He holds a master’s in computer science and specializes in sales engineering, developing reference architectures, and authoring technical documentation for data centers and cloud workloads.

Michael Guenther, Principal Software Engineer, Splunk LLC.

Michael Guenther is a Principal Engineer at Splunk responsible for the architecture of the Splunk POD solution. He has worked with numerous distributed systems. He joined Splunk in 2017 and has worked extensively on a wide range of Splunk products.

Michael Lusher, Software Engineer, Splunk LLC.

Michael Lusher is a Software Engineer at Splunk working on the Splunk POD solution. He joined Splunk in 2021 and has worked on a variety of Splunk products.

Acknowledgements

For their support and contribution to the design, validation, and creation of this Cisco Validated Design, the authors would like to thank

●     Hardikkumar Vyas, Technical Marketing Engineer, Cisco Systems, Inc.

CVD Program

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS X-Series, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study,  LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trade-marks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. (LDW_P7)

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)