UTF-8 characters are being hex-encoded in Splunk logs

Applicability

Splunk Enterprise 9.x 8.2.12 8.2.11.2 8.2.11.3 8.1.14.1

Description

The UTF-8 characters (international characters: Korean) are not displayed correctly in the many of log files in $SPLUNK_HOME/var/log/splunk directory.
Such as audit.log, configuration_change.log, scheduler.log, etc.

For instance,
The below savedsearch name is "demo_KOR_한글버전".
 

$ cat savedsearches.conf 
[demo_KOR_한글버전]
action.email = 1
action.email.to = test@localhost.com
action.email.useNSSubject = 1
action.webhook.enable_allowlist = 0
alert.track = 0
cron_schedule = * * * * *
dispatch.earliest_time = -24h@h
dispatch.latest_time = now
display.visualizations.show = 0
enableSched = 1
request.ui_dispatch_app = search
request.ui_dispatch_view = search
search = index=_internal ERROR
$

However, most of the log files didn't display it correctly as below.

savedsearch_id="admin;search;demo_KOR_\xed\x95\x9c\xea\xb8\x80\xeb\xb2\x84\xec\xa0\x84"

savedsearch_name="demo_KOR_\xed\x95\x9c\xea\xb8\x80\xeb\xb2\x84\xec\xa0\x84"

scheduler.log

09-28-2023 03:35:01.229 +0000 INFO  SavedSplunker - savedsearch_id="admin;search;demo_KOR_\xed\x95\x9c\xea\xb8\x80\xeb\xb2\x84\xec\xa0\x84", search_type="scheduled", search_streaming=0, user="admin", app="search", savedsearch_name="demo_KOR_\xed\x95\x9c\xea\xb8\x80\xeb\xb2\x84\xec\xa0\x84", priority=default, status=success, digest_mode=1, durable_cursor=0, scheduled_time=1695872100, window_time=0, dispatch_time=1695872100, run_time=0.288, result_count=30, alert_actions="", sid="scheduler__admin__search__RMD5f15e6c41068bdbfd_at_1695872100_7", suppressed=0, action_time_ms=5, thread_id="AlertNotifierWorker-0", workload_pool=""

audit.log

09-28-2023 03:32:07.548 +0000 INFO  AuditLogger - Audit:[timestamp=09-28-2023 03:32:07.548, user=admin, action=create_saved_search, info=succeeded, savedsearch_name="demo_KOR_\xed\x95\x9c\xea\xb8\x80\xeb\xb2\x84\xec\xa0\x84", app="search", owner="admin", disabled="false", type="scheduled", dispatch_earliest="-24h@h", dispatch_latest="now"]

configuration_change.log

{"datetime":"09-28-2023 03:32:07.556 +0000","log_level":"INFO ","component":"ConfigChange","data":{"path":"/opt/splunk/etc/users/admin/search/local/savedsearches.conf","action":"add","modtime":"Thu Sep 28 03:32:07 2023","epoch_time":"1695871927","new_checksum":"0x36a4405f972d4535","old_checksum":"0x0","changes":[{"stanza":"demo_KOR_\xed\x95\x9c\xea\xb8\x80\xeb\xb2\x84\xec\xa0\x84","properties":[{"name":"action.email.useNSSubject","new_value":"1","old_value":""},{"name":"action.webhook.enable_allowlist","new_value":"0","old_value":""},{"name":"dispatch.earliest_time","new_value":"-24h@h","old_value":""},{"name":"dispatch.latest_time","new_value":"now","old_value":""},{"name":"display.visualizations.show","new_value":"0","old_value":""},{"name":"request.ui_dispatch_app","new_value":"search","old_value":""},{"name":"request.ui_dispatch_view","new_value":"search","old_value":""},{"name":"search","new_value":"index=_internal ERROR","old_value":""}]}]}}

Investigation

This is a consequence of new escaping code added in recent versions of Splunk to prevent log injection vulnerabilities.
SPL-241330 [Hackerone] Log injection vulnerability in /en-GB/splunkd/__raw/services/messages
 

Resolution

This has been resolved in Splunk Version 9.2.0+

For previous versions the following Workaround can be applied:
We can apply the following workaround to restore the non-ASCII content in their log files.
This should be done in the $SPLUNK_HOME/etc/log-local.cfg file  (create the file if it does not exist) and the restart of Splunk is required.
For each appender defined in the log.cfg file, add a line like the following log-local.cfg

appender.<name>.escape=false

For instance,
Add the following line for the scheduler.log

appender.scheduler.escape=false

Add the following line for the audit.log

appender.audittrail.escape=false

Add the following line for the configuration_change.log

appender.confchangelogger.escape=false

The full list to be added to $SPLUNK_HOME\etc\log-local.cfg can be found below:

[splunkd]
appender.A1.escape=false
appender.license_usage.escape=false
appender.license_usage_summary.escape=false
appender.A4.escape=false
appender.metrics.escape=false
appender.audittrail.escape=false
appender.accesslog.escape=false
appender.uiaccess.escape=false
appender.scheduler.escape=false
appender.remotesearches.escape=false
appender.idata_ResourceUsage.escape=false
appender.conf.escape=false
appender.idata_KVStore.escape=false
appender.kvstore_appender.escape=false
appender.watchdog_appender.escape=false
appender.searchmsgs_appender.escape=false
appender.workloadmanager.escape=false
appender.confchangelogger.escape=false

In $SPLUNK_HOME\etc\log-searchprocess.cfg add the following: 

appender.searchprocessAppender.escape=false
appender.hcMetrics.escape=false
appender.watchdog_appender.escape=false
appender.searchTelemetryAppender.escape=false

In $SPLUNK_HOME\etc\splunk-launch.conf add the following line:

SPLUNK_PYTHON_DONT_ESCAPE_PRINTABLE = 1

Restart Splunk to apply the change.

Related Links

SPL-245166

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release