Generic S3 input stopped collecting new logs configured in Splunk Add-on for Amazon Web Services(AWS)

Applicability

Description

The Generic S3 input that utilizes AWS S3 Bucket to fetch the objects at a given internal and is configured in the Splunk Add-on for Amazon Web Services (AWS) is encountering a failure, accompanied by the error message: "TypeError: cannot unpack non-iterable int object".

Another observation made was that the other input types ie. SQS Based S3, CloudWatch Logs, etc were working as expected

Investigation

Upon investigating and checking the internal logs, with the below search query to check if there are any error messages in the _internal logs:

SPL:

index="_internal" sourcetype=*aws* datainput="input_name" level=ERROR

Found the below error messages which are generated repeatedly whenever a Generic S3 input is executed. The error message reads as below:

Error Logs:

2024-05-27 12:48:07,290 level=ERROR pid=2205 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:index_data:114 | datainput="<input_name>" bucket_name="<bucket_name>" | message="Failed to collect data through generic S3." start_time=1716814086 job_uid="9a6566b5-c700-42f8-97ab-1ff7b556422c"
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 108, in index_data
    self._do_index_data()
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 131, in _do_index_data
    self.collect_data()
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 166, in collect_data
    index_store = s3ckpt.S3IndexCheckpointer(self._config)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 165, in __init__
    config[asc.data_input], config[tac.checkpoint_dir]
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 103, in get
    S3CkptPool.CKPT_POOL[ckpt_name] = S3CkptPool.S3CkptItem(ckpt_name, ckpt_dir)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 90, in __init__
    self.idx_ckpt = LocalKVStore.open_always(ckpt_file_idx)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 177, in open_always
    indexes = cls.build_indexes(fp)
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 184, in build_indexes
    for flag, key, pos in cls._replay(fp):
  File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 111, in _replay
    flag, key, _ = umsgpack.unpack(fp)
TypeError: cannot unpack non-iterable int object

Upon further validation and debugging of the internal logs, it was discovered that the root cause of the issue lies within the checkpoints, which became corrupted.

Consequently, whenever an input invocation occurred, it failed with the error: "TypeError: cannot unpack non-iterable int object."

This error arose because the input fetched the last checkpoint value to pull new objects from the S3 bucket.

Resolution