SQS-Based S3 inputs fail with "ExpiredToken" errors after upgrading splunk Add-on for AWS to v8.1.1

Applicability

Description

After upgrading the Splunk Add-on for AWS to version 8.1.1, SQS-Based S3 inputs configured with AssumeRole-based authentication intermittently fail with "ExpiredToken" errors. Temporary STS credentials expire and are not refreshed correctly during the ingestion workflow, causing data ingestion to stop approximately one hour after the inputs are started or restarted.

Restarting the affected inputs temporarily restores ingestion, but the issue recurs after the STS token expires again.

Investigation

1. Navigate to the Splunk Add-on for AWS and confirm the installed version is 8.1.1.
2. Identify SQS-Based S3 inputs configured with AssumeRole authentication.
3. Check internal logs for "ExpiredToken" errors related to the affected inputs:
   

   "index=_internal sourcetype=splunkd component=ExecProcessor "ExpiredToken""

4. Confirm ingestion stops approximately one hour after inputs are started and resumes only after a manual restart.

Resolution

To resolve the issue there are two ways:

Option 1: Upgrade to version 8.1.2 (permanent fix)

1. Download Splunk Add-on for AWS version 8.1.2 from SplunkBase.
2. Install the updated add-on following standard upgrade procedures.
3. Restart Splunk services.
4. Monitor the affected SQS-Based S3 inputs and confirm "ExpiredToken" errors no longer occur.

Option 2: Temporary workaround (if upgrade is not immediately possible)

1. Navigate to Splunk Add-on for AWS > Configuration > Account.
2. Identify the account used by the affected SQS-Based S3 inputs.
3. Switch the authentication method from AssumeRole to IAM user-based authentication (if permitted by AWS security policies).
4. Save the configuration and restart the affected inputs.
5. Monitor ingestion stability and confirm data flows without interruption.

Related Links

Splunk Add-on for AWS v8.1.2 on SplunkBase

Splunk Add-on for AWS Release Notes - Fixed Recurring ExpiredToken