S3ClientProps IamServiceAccountAwsCredentials error failed to open identity token file AWS_WEB_IDENTITY_TOKEN_FILE error=No such file or directory

Applicability

Description

The user noticed that this ERROR message appeared whenever Splunk started (or restarted) after upgrading to the 9.2.3 version.

The same ERROR message was found in the splunkd.log and splunkd-utility.log

$SPLUNK_HOME/var/log/splunk/splunkd.log

10-04-2024 09:47:24.144 +0900 INFO  ServerConfig [0 MainThread] - My GUID is 1583CEE5-4EAF-41AC-9962-E492A22214CA10-04-2024 09:47:38.747 +0900 ERROR AwsCredentials [3208 SplunkdSpecificInitThread] - S3ClientProps IamServiceAccountAwsCredentials error failed to open identity token file AWS_WEB_IDENTITY_TOKEN_FILE error=No such file or directory

$SPLUNK_HOME/var/log/splunk/splunkd-utility.log

10-04-2024 09:47:22.724 +0900 ERROR AwsCredentials - S3ClientProps IamServiceAccountAwsCredentials error failed to open identity token file AWS_WEB_IDENTITY_TOKEN_FILE error=No such file or directory

Investigation

Environment:

• The indexer was running on the Amazon EC2 instance.
• Indexer cluster (Three members)
• The user used the IAM role for the AWS S3 SmartStore authentication.
• Therefore, the user didn't need the remote.s3.access_key and remote.secret_key in the indexes.conf.

This ERROR appeared across all indexer cluster members.

$SPLUNK_HOME/etc/system/local/indexes.conf

[default]
remotePath = volume:remote_store/$_index_name
repFactor = auto[volume:remote_store]
storageType = remote
path = s3://s3-splunk-smartstore# The following S3 settings are required only if you use the access and secret keys. They are not needed if you are using AWS IAM roles.
# remote.s3.access_key = <S3 access key>
# remote.s3.secret_key = <S3 secret key>
# remote.s3.endpoint = https:|http://<S3 host>

Tested the connectivity between the indexer and the AWS S3 and the function using the rfs CLI command, and it returned the result successfully without any error.

# splunk cmd splunkd rfs -- ls --starts-with volume:remote_store

Checked the system environment variable definition with the splunk account (splunk) to see if there's any variable related to the AWS_WEB_IDENTITY_TOKEN_FILE variable after referencing the SmartStore on S3 security strategies document.

However, there was nothing related to the AWS.

$ su - splunk
$ printenv
LANG=en_US.UTF-8
HISTCONTROL=ignoredups
HOSTNAME=indexer_name
which_declare=declare -f
USER=splunk
PWD=/app/splunk
HOME=/app/splunk
MAIL=/var/spool/mail/splunk
SHELL=/bin/bash
TERM=xterm
SHLVL=1
LOGNAME=splunk
PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
HISTSIZE=1000
LESSOPEN=||/usr/bin/lesspipe.sh %s
BASH_FUNC_which%%=() { ( alias;
eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
}
_=/usr/bin/printenv

Resolution

This is a bug. The log level will be changed from ERROR to DEBUG in the future release.

Therefore, the user can ignore this message at the moment.

In the get AWS credentials call, Splunk attempts multiple options to obtain AWS credentials in the following order:1. Try retrieving credentials from memory.
2. Try retrieving from server.conf.
3. Try retrieving from IAM Role.
4. Try retrieving from IRSA(IAM Roles for Service Accounts) configuration.
5. ...The issue observed is that when attempting to use IRSA configuration, a log message is added indicating failure to obtain AWS credentials due to a missing token file. 
This message is currently logged as an error, but it should be logged as a debug message instead, given the nature of the retry process.

Related Links

SmartStore on S3 security strategies document