btool ignores configuration file when filename contains trailing invisible/zero-width space characters

Applicability

Description

Administrators deploy or edit configuration files (for example, indexes.conf or server.conf) under an app directory such as:

$SPLUNK_HOME/etc/manager-apps/<app_name>/local/
$SPLUNK_HOME/etc/apps/<app_name>/local/

Although the file appears to exist and standard file commands output seemingly normal results, the stanzas and attributes defined in the file are completely ignored by btool and Splunk processes.

Symptoms:

• splunk btool <conf_name> list --debug returns only default/system values and omits the configuration in the app layer.
• Bundle push or deployment succeeds, but target peer_apps do not apply the configurations.
• File content and stanzas appear completely valid upon inspection with text editors.

Investigation

Invisible characters (such as Unicode Zero-Width Space U+200B) do not render on standard terminal displays, making the file look deceptively normal:

1. Deceptive Output from Standard Commands:

   • Running ls -la may display a trailing question mark:

     -rw-r--r--  1 splunk  splunk   78 Oct  8 18:29 indexes.conf?

   • Running file * hides the character completely:

     app.conf:     ASCII text
     indexes.conf​: ASCII text
     limits.conf:  ASCII text
     server.conf:  ASCII text

2. Quick Detection via Shell Tab Auto-Completion:

   In a Bash or Zsh terminal, type the start of the filename and press [Tab]:

   file index[Tab]

   The shell automatically escapes the non-printable UTF-8 bytes into octal notation:

   file indexes.conf$'\342\200\213'

   (The appearance of $'\342\200\213' immediately confirms trailing invisible characters).

3. Inspect Filenames Using Octal Escapes (ls -lb):

   ls -lb $SPLUNK_HOME/etc/manager-apps/<app_name>/local/

   Observed Output:

   -rw-r--r-- 1 splunk splunk 78 Nov  4 13:20 indexes.conf\342\200\213

4. Confirm btool Behavior:

   $SPLUNK_HOME/bin/splunk btool indexes list --debug | grep -i <stanza_or_setting>

   If settings are missing from the app layer despite correct file permissions and syntax, filename corruption is confirmed.

Resolution

Root Cause

Splunk's configuration parser and btool strictly evaluate files ending with the exact extension .conf. When a trailing non-printable or invisible character (such as Zero-Width Space U+200B / UTF-8 octal \342\200\213) is appended to the filename, Splunk treats the file as having an unrecognized extension and completely skips it during startup, reload, and btool evaluations.

Fix / Workaround

1. Rename the file to remove the trailing invisible character:

   Use shell expansion or press [Tab] to complete the corrupted filename:

   mv indexes.conf$'\342\200\213' indexes.conf

2. Verify the filename is clean:

   ls -lb indexes.conf

   Confirm that no trailing escape sequence (\342\200\213) appears.

3. Verify with btool:

   $SPLUNK_HOME/bin/splunk btool indexes list --debug

   Confirm that btool now recognizes the stanza from the app layer.

4. Apply Cluster Bundle if applicable:

   On the Cluster Manager, validate and apply the bundle to distribute the updated configuration to the indexers' peer_apps:

   splunk validate cluster-bundle
   splunk apply cluster-bundle

Related Links

• Splunk Documentation: Use btool to troubleshoot configurations
• Splunk Documentation: About configuration files