Simple method for setting Splunk Environment Variables in Linux

Applicability

Description

If your organization runs Splunk on Linux servers and allows you to use profile scripts, your life can be immensely simplified by setting permanent environment variables. When environment variables are set, Splunk commands

• no longer require being run from $SPLUNK_HOME/bin
• are no longer prefaced with ./

If environment variables are set only for the Splunk user, attempting to run a Splunk command from any directory except $SPLUNK_HOME/bin will fail. Also, entering a command without ./ will fail even if you are in the bin folder (without environment variables, Splunk is started with ./splunk start)

Investigation

Splunk environment variables can be enabled during the current login session, using this command:

source /opt/splunk/bin/setSplunkEnv

However, the variables will not persist beyond the current login session.

Resolution

To make these persist, place a script file in the splunk user:

1. SSH into the Splunk instance using the Splunk user account which is intended to run Splunk.
2. From the Splunk user's home directory, run a long listing of the home folder to identify what startup script is used by your distro:
   ls -la
3. Based on the output above, place the following into the Splunk user's Bash shell startup script " ~/.bashrc " (or "~/.bash_profile ", or " ~/.profile "). 
   For example (most debian distro's)
   

   vi ~/.profile

    

   If the file does not already exist, please research which file to use for your distro. Once the file is added or created as shown with the vi command, paste this into the file and save changes:

   source /opt/splunk/bin/setSplunkEnv

4. Restart the machine (not Splunk specifically)*
5. When the server comes back up again, SSH into it as the Splunk user. You will see the following just above the command prompt:
   

   Tab-completion of "splunk <verb> <object>" is available.

NOTE:

If these steps are used to update variables on a clustered indexer, please place the cluster into maintenance mode before restarting any one indexer. Once all have been updated, check the Cluster Manager's Settings > Indexer Clustering page to confirm all are in an "up" status and stable. Then, take it out of maintenance mode. Commands for Maintenance Mode must be run from the cluster manager:

splunk enable maintenance-mode
splunk disable maintenance-mode

Maintenance mode allows indexers to continue searching and ingesting data. If your environment uses SmartStore, it also continues placing warm buckets into your SmartStore storage. However, it stops bucket replication across peers. Maintenance Mode is a very important and useful tool for Splunk Admins when working with indexer clusters.

It is used primarily to keep Splunk from generating unnecessary fix-up tasks during maintenance activities, and allows an indexer to be shut down for a very short period using  splunk stop instead of splunk offline (which re-assigns primary buckets to all the other indexers and takes a very long time to complete).

If a cluster is unstable (one or more members "flapping" -- status switching from "Up" to "Pending", "Batch Adding", "Down", etc.), it is suggested placing the cluster into Maintenance Mode to limit the number of fix up tasks until the issue can be resolved or remediated.

Related Links

• For indexer clusters: Splunk Documentation regarding maintenance mode
• InterWebs threads on setting environment variables for a specific user:
  • ServerFault
  • StackExchange
  • AskUbuntu