Splunk Add-on for AWS Fails to Download Files >=10MB in size from S3 Bucket

Applicability

Description

The Splunk AWS TA is configured with the SQS-based S3 input to pull new files from the S3 bucket. This integration works fine for files smaller than 10 MB, but it has been noticed that files equal to or larger than 10 MB cannot be downloaded by the TA.

Investigation

SPL command used to find the error logs as below:

index=_internal datainput=b'<inputname>' ERROR

Upon investigation, the following error was found in the logs.

2024-02-19 08:43:12,562 level=ERROR pid=1196312 tid=Thread-5 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_process:507 | datainput="Apex" start_time=1708332011, message_id="a5e1ad69-7d08-4e16-9bce-c0eeaddab4aa" created=1708332192.371223 expires=1708332492.3712432 job_id=32e412f5-5f60-40d6-809d-5d15f946020b | message="An error occurred while processing the message."
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 469, in _process
headers = self._download(record, cache, session)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 710, in _download
return self._s3_agent.download(record, cache, session)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 891, in download
return bucket.transfer(s3_bkt, key, fileobj, **condition)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/common/s3.py", line 97, in transfer
download_fileobj(client, bucket, key, fileobj)
File "/opt/splunk/etc/apps/Splunk_TA_aws/lib/boto3/s3/inject.py", line 795, in download_fileobj
return future.result()
File "/opt/splunk/etc/apps/Splunk_TA_aws/lib/s3transfer/futures.py", line 103, in result
return self._coordinator.result()
File "/opt/splunk/etc/apps/Splunk_TA_aws/lib/s3transfer/futures.py", line 266, in result
raise self._exception
File "/opt/splunk/etc/apps/Splunk_TA_aws/lib/s3transfer/tasks.py", line 139, in __call__
return self._execute_main(kwargs)
File "/opt/splunk/etc/apps/Splunk_TA_aws/lib/s3transfer/tasks.py", line 162, in _execute_main
return_value = self._main(**kwargs)
File "/opt/splunk/etc/apps/Splunk_TA_aws/lib/s3transfer/download.py", line 570, in _main
Bucket=bucket, Key=key, **extra_args
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/common/s3.py", line 135, in get_object
return self._client.get_object(**kwargs)
File "/opt/splunk/etc/apps/Splunk_TA_aws/lib/botocore/client.py", line 535, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/splunk/etc/apps/Splunk_TA_aws/lib/botocore/client.py", line 983, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied

During investigation, it was observed that the "AccessDenied" error was misleading because all permissions are properly set up. However, the user was unable to download the file despite having required permissions for an IAM user. The issue was only observed for the files that are greater than 10 MB in size.

Resolution

It was discovered that it is required to assign "s3:ListBucketMultipartUpload" permission to the IAM user if Multipart upload process is enabled on S3 bucket to upload large size object.

Multipart upload allows users to upload a single object as a set of parts. Each part is a contiguous portion of the object's data. Users can upload these object parts independently and in any order. 

Related Links

Reference of S3 bucket Multipart Upload is documented below:
Uploading and copying objects using multipart upload