The Splunk Add-on for AWS v7.7.0 is unable to decode parquet files when using SQS-based S3 inputs for AWS Security Lake data

Applicability

Description

The Splunk Add-on for AWS version 7.7.0 is unable to decode parquet files ingested using the Security Lake data input with the aws:asl sourcetype. Internal logs indicate a decoding error with exit code 1.

Investigation

During the investigation of internal logs using below SPL in search app:

index=_internal datainput=<aws-security-data-lake-input-name>

Errors as below:

2024-09-19 09:59:35,038 level=ERROR pid=3719278 tid=Thread-6 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_process:507 | datainput="AWS_SQS_S3" start_time=1726662446, message_id="8c897bb1-4f23-4520-8e91-1a477265bfad" created=1726739974.9252675 expires=1726740274.9252875 job_id=3fcf7cc4-df89-4eff-b375-dbb7339859a0 | message="An error occurred while processing the message." 
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 494, in _process
    self._parse_csv_with_delimiter,
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 591, in _ingest_file
    for records, metadata in self._decode(fileobj, source):
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/common/decoder.py", line 175, in __call__
    for line in aws_parquet.stream_parquet(filename):
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_parquet/__init__.py", line 51, in stream_parquet
    raise parser_error
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_parquet/__init__.py", line 47, in stream_parquet
    raise subprocess.CalledProcessError(return_code, parser_cmdline)
subprocess.CalledProcessError: Command '['nice', '/opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_parquet/parquet_decoder_linux_amd64', '/opt/splunk/var/lib/splunk/modinputs/aws_sqs_based_s3/AWS_SQS_S3/tmp292ej88q']' returned non-zero exit status 1.
2024-09-19 09:59:35,038 level=ERROR pid=3719278 tid=Thread-6 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_ingest_file:635 | datainput="AWS_SQS_S3" start_time=1726662446, message_id="8c897bb1-4f23-4520-8e91-1a477265bfad" created=1726739974.9252675 expires=1726740274.9252875 job_id=3fcf7cc4-df89-4eff-b375-dbb7339859a0 | message="Failed to ingest file." uri="s3://aws-security-data-lake-eu-west-1-mapanubxyh6h29ozef7ukwie9zwi2u/aws/ROUTE53/2.0/region=eu-west-1/accountId=246136079538/eventDay=20240919/88c904b7daf48405901b527b36215fc2.gz.parquet"
2024-09-19 09:59:35,038 level=ERROR pid=3719278 tid=Thread-6 logger=aws_parquet pos=__init__.py:stream_parquet:50 | datainput="AWS_SQS_S3" start_time=1726662446, message_id="8c897bb1-4f23-4520-8e91-1a477265bfad" created=1726739974.9252675 expires=1726740274.9252875 job_id=3fcf7cc4-df89-4eff-b375-dbb7339859a0 | message="there was an error within the parquet parser: Command '['nice', '/opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_parquet/parquet_decoder_linux_amd64', '/opt/splunk/var/lib/splunk/modinputs/aws_sqs_based_s3/AWS_SQS_S3/tmp292ej88q']' returned non-zero exit status 1." 

Investigation also revealed that this issue was addressed in v.7.6.0 as per the Splunk Doc.

Resolution

Root cause: This is because binary files are used to decode parquet files, and they are not compatible with Ubuntu 20.04. This input works fine in AWS v7.6.0 on Ubuntu 20.04 OS.

Fix: The decoding issue will be fixed in a new patch release of the add-on.