How to monitor if Splunk Cloud Elastic Load Balancer (ELB) public IP addresses have changed

Applicability

Description

Users connect to Splunk Cloud search heads from external sources to query REST API endpoints. The firewall needs to permit traffic coming from the public IP of the Splunk Cloud search head cluster's ELB 

Since these public IPs can change, which is beyond Splunk's control, on-premises instances can be disconnected from the Splunk Cloud search heads for an extended period unless they are notified of such changes and update their firewall rules.

Investigation

When Splunk Cloud stacks were originally provisioned, they used Elastic Load Balancers. There is no functionality for assigning static IP's to an ELB. Network Load Balancers are in place on Victoria stacks and they DO use static IP's.

However, since AWS is a 3rd party cloud provider for Splunk there are rare scenarios outside the control of Splunk where the IP addresses might change. Users typically interface with Splunk search heads through a load balancer and are given an FQDN like: <stack>.splunkcloud.com

This is a load balancer with search heads behind it and the public IP address that the load balancer FQDN resolves to can change over time.

Resolution

The following solution can be used to monitor for changes to the Splunk Cloud load balancer public IP addresses

1. Create a script

On a on premise Splunk utility server such as the deployment server or monitoring console instance, create a script file called elb.sh and add the following contents

#!/bin/bash# List of FQDNs to perform nslookup
fqdns=(
    "es.<stack>.splunkcloud.com"
    "<stack>.splunkcloud.com"
)# File to store the results
output_file="nslookup_results.txt"# Get the current date
current_date=$(date +"%Y-%m-%d %H:%M:%S")# Iterate through each FQDN and perform nslookup
for fqdn in "${fqdns[@]}"; do
    nslookup_result=$(nslookup "$fqdn")    # Extract canonical name
    canonical_name=$(echo "$nslookup_result" | awk '/^Non-authoritative answer:$/,0 { if ($1 == "Name:") {print $2; exit} }')    # Extract addresses below "Non-authoritative answer:"
    addresses=$(echo "$nslookup_result" | awk '/^Non-authoritative answer:$/,0 { if ($1 == "Address:") {print "elb_address="$2} }')    # Write results to the output file
    while read -r address; do
        echo "$current_date FQDN=$fqdn Name=$canonical_name $address" >> "$output_file"
    done <<< "$addresses"
doneecho "NSLOOKUP results have been saved to $output_file"

The script will need executable permison

chmod +x elb.sh

The script will create an output like this written to a log file which in this case is in /opt/splunk/bin

root@sh1:/opt/splunk/bin# cat nslookup_results.txt
2024-05-08 19:43:41 FQDN=es.<stack>.splunkcloud.com Name=<stack>-shc2-1806503139.us-east-1.elb.amazonaws.com elb_address=x.x.x.x
2024-05-08 19:43:41 FQDN=es.<stack>.splunkcloud.com Name=<stack>-shc2-1806503139.us-east-1.elb.amazonaws.com elb_address=x.x.x.x
2024-05-08 19:43:41 FQDN=es.<stack>.splunkcloud.com Name=<stack>-shc2-1806503139.us-east-1.elb.amazonaws.com elb_address=x.x.x.x
2024-05-08 19:43:41 FQDN=<stack>.splunkcloud.com Name=<stack>.splunkcloud.com elb_address=x.x.x.x
2024-05-08 19:43:41 FQDN=<stack>.splunkcloud.com Name=<stack>.splunkcloud.com elb_address=x.x.x.x
2024-05-08 19:43:41 FQDN=<stack>.splunkcloud.com Name=<stack>.splunkcloud.com elb_address=x.x.x.x

2. Create Input Monitor

Create an input monitor stanza on the utility server where the script is running to ingest the events into your on premise Splunk deployment. Events should be forwarded to your on premise indexers.

$SPLUNK_HOME/etc/system/local/inputs.conf
[monitor:///opt/splunk/bin/nslookup_results.txt]
disabled = 0
index = main
sourcetype = elb_nslookup

3. Create a cron job

Run the script every hour

0 * * * * /opt/splunk/bin/elb.sh

4. Set up a Splunk alert

Run hourly and monitor for elb ip address change where match=no
Note: For the elb_address fields, enter the IP addresses that the FQDN resolves to at the time the script is deployed

index=main sourcetype=elb_nslookup source=*/nslookup_results.txt
| stats latest(_time) as _time latest(FQDN) as FQDN latest(Name) as Name by elb_address
| table _time elb_address FQDN Name
| sort -Name
| eval match=if(elb_address="x.x.x.x" OR
elb_address="x.x.x.x" OR
elb_address="x.x.x.x" OR
elb_address="x.x.x.x" OR
elb_address="x.x.x.x" OR
elb_address="x.x.x.x", "yes", "no")
| eval note=if(match="yes", "all good", "add elb_address to firewall whitelist")
| eval env = case(
FQDN="<stack>.splunkcloud.com","Core SHC",
FQDN="es.<stack>.splunkcloud.com","ES SHC",
1=1, "unknown")
| search match=no

If there is a non match you will need to update the firewall rules for the new IPs. If for some reason the FQDN names change in the future you can update the script and modify the FQDN.