xmlkv command not working on Search Head Cluster

Applicability

Description

xmlkv command not working in Search Head cluster giving the below error in the webUI,

"Streamed search execute failed because: Error in 'xmlkv' command: Cannot find program 'xmlkv' or script 'xmlkv'."

Investigation

1. The observed error is related to xmlkv.py present under /opt/splunk/etc/search/bin/xmlkv.py and states that the Search Head which was running the xmlkv command could not find the xmlkv.py file on the Indexer and hence could not run the search.

2. Usually the error occurs when the ./etc/search/bin directory is backlisted to be replicated to indexer on the Search Head. If it is blacklisted for bundle replication, then the commands which require ./bin directory scripts will not run properly as indexers will not be able to retrieve the required script to run them.

3. Looked into any blacklist for the ./search/bin directory on the Search Head Cluster, could see a blacklist for ./apps/search/bin in 

distsearch.conf.
./apps/CA-DistributedSearch/default/distsearch.conf:     
[replicationBlacklist]
bl_bin_23 = apps/search/bin/...

4. Since the blacklist was added due to a specific use case, the blacklist could not be removed. The only was to fix the issue was to enable Search Head to run the command locally.

Resolution

As there was a backlist on the SHC and cannot be removed, we enabled Search Head to use xmlkv.py from its own ./search/bin/..

./commands.conf
[xmlkv]
filename = xmlkv.py
local = true
streaming = trueRestart Splunk service