Logs are not getting ingested from S3 bucket via Splunk Add-on for AWS and getting Error as "UnicodeEncodeError: 'latin-1' codec can't encode character '\u0107'"

Applicability

Description

When a user attempts to ingest logs containing special characters, such as apostrophes ('), or accented letters like 'ć', from an S3 bucket into Splunk using the Splunk Add-on for AWS, these logs fail to ingest into Splunk. However, the logs containing the special characters are still present in the AWS S3 bucket.

Investigation

1. SPL query used to check the Error logs as below:

index=_internal sourcetype=*aws* level=ERROR

Below is the UnicodeEncodeError that has been reported in the internal logs.

2024-08-08 20:58:01,342 level=ERROR pid=398 tid=Thread-7 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_process:507 | datainput="secops-medallia-windows-other-logs" start_time=1723150352, message_id="8dcdcaf8-2e73-49e4-9002-0c923bf472b7" created=1723150680.809199 expires=1723150980.8092139 job_id=79788787-bf57-45bc-b064-c9fb70fbfb2b | message="An error occurred while processing the message."
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 494, in _process
self._parse_csv_with_delimiter,
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 632, in _ingest_file
volume = self._event_writer.write_fileobj(records, **metadata)
File "/opt/splunk/etc/apps/Splunk_TA_aws/lib/splunksdc/event_writer.py", line 174, in write_fileobj
self._write(data)
File "/opt/splunk/etc/apps/Splunk_TA_aws/lib/splunksdc/event_writer.py", line 132, in _write
self._dev.write(data)
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0107' in position 3235566: ordinal not in range(256)

2. Execute the below command on the Instance where Splunk Add-on for AWS Installed.

/opt/splunk/bin/splunk cmd pythonimport sys, locale, os;print(sys.stdout.encoding);print(locale.getpreferredencoding());print(sys.getfilesystemencoding());print(locale.getdefaultlocale());

The response is as below:

UTF-8UTF-8UTF-8

3. Executed the below command on the Instance

cat /etc/default/locale

The Output is below

LANG="en_US"
LANGUAGE="en_US:"

Resolution

To address this issue, it is recommended to adjust the "locale" settings at the OS level by modifying the file `/etc/default/locale` and adding the following stanza and then reboot the server.

LANG=en_US.UTF-8
LANGUAGE=en.UTF-8

Related Links

https://stackoverflow.com/questions/55077450/linode-lish-bash-warning-setlocale-lc-all-cannot-change-locale-en-us-utf-8