How to Update Credentials for Universal Forwarders

Applicability

Description

How to Update Credentials for Universal Forwarders

Investigation

The following search will show all the Splunk fowarders that are sending their metrics logs into Splunk Cloud, which in turns means that UF credentials are still valid and have not expired.

index=_internal source=*metrics.log group=tcpin_connections hostname=*
| dedup hostname os | table hostname sourceIp os arch version fwdType

If a fowarder is not sending data into Splunk Cloud, the above search query can be used with the specific hostname.

Replace

hostname=*

with 

hostname=forwarderHostNameWithIssues

If no metrics events are returned from the host then most likely the UF credentials have expired and will need to be updated.

To determine if the UF credentials have expired, the following process can be used on the instance itself. 

In the $SPLUNK_HOME/etc/auth directory find a file named <your_stack_name>_server.pem

cd $SPLUNK_HOME/etc/auth
$SPLUNK_HOME/bin/splunk cmd openssl x509 -in <stack>_server.pem -noout -enddate

something similar to the output below will be returned:

notAfter=Mon DD HH:MM:SS YYYY TMZ

If the date is in the past, the certificate is expired and the credential package needs to be updated.

Resolution

To update the UF credentials please refer to Install and configure the Splunk Cloud Platform universal forwarder credentials package

If an existing credential package on the forwarder is being replaced, the following steps can be used:

1. download the credentials app from the Splunk Cloud Search Head, and place it into the /tmp directory on the affected Splunk forwarder. This is the credentials package.

2. From the /tmp directory, extract the package using "tar xvf splunkclouduf.spl" for nix or use unzip for Windows.

3. Once extracted, it creates a new directory, named 100_stackname_splunkcloud.

6. Go to $SPLUNK_HOME/etc/apps, and copy the the current, with expired certs, 100_stackname_splunkcloud directory, to /tmp and then delete it from the $SPLUNK_HOME/etc/apps directory.

7. Then copy the new 100_stackname_splunkcloud directory into the $SPLUNK_HOME/etc/apps.

8. Restart Splunk service by using ./splunk restart in the $SPLUNK_HOME/bin

Note: The command "$SPLUNK_HOME/bin/splunk install app /tmp/splunkclouduf.spl -update 1 -auth <username>:<password>" is not working, so the current configuration directory needs to be removed and replace from the extracted directory.

Once the UF credential is updated, the following OpenSSL command can be used to verify:

cd $SPLUNK_HOME/etc/auth 
$SPLUNK_HOME/bin/splunk cmd openssl x509 -in <stack>_server.pem -noout -enddate