Log Observer Connect Does Not Display Indexer-Hosted Indexes in Distributed Splunk Enterprise Deployment

Applicability

Description

In distributed Splunk Enterprise deployments, Log Observer Connect retrieves the list of available indexes using REST APIs exposed by the Search Head. These REST endpoints return index metadata based on the index definitions known to the Search Head instance.

If indexes are defined only in indexes.conf on the Indexers and do not appear in any indexes.conf on the Search Head, the Search Head does not have local knowledge of those index names. As a result, these indexes are not returned by the Search Head REST API and do not appear in Log Observer Connect, even though:

• Data exists in the indexes on the Indexers

• Distributed searches against those indexes work correctly

• User permissions allow access to those indexes

This behaviour is consistent with REST-based index discovery in distributed Splunk Enterprise environments where index metadata visibility depends on Search Head configuration.

Investigation

Observed Behavior

• Log Observer Connect only displayed indexes created locally on the Search Head

• Required indexes existed on the Indexer but were not visible in LOC

• REST call to the Search Head returned only SH-local index:

  /services/data/indexes

• Indexer-hosted indexes were not returned.

Validation Performed

• Verified connectivity between Log Observer Connect and Search Head

• Verified authentication and service user configuration

• Verified RBAC permissions and role configuration

• Verified distributed search configuration and search peer setup

• Verified Indexer contained required index definitions

• Verified REST response from Search Head returned only locally defined indexes

• Created a test index on Search Head, which immediately appeared in LOC

Conclusion from Testing:

The Search Head REST endpoint returned only index definitions present in the Search Head configuration. Indexes defined only on the Indexer were not exposed through the Search Head REST API and therefore were not visible to Log Observer Connect.

Resolution

Workaround (Recommended Operational Fix)

Define placeholder index stanzas for the required indexes in indexes.conf on the Search Head.

The placeholder configuration only needs:

• Index name

• Basic database paths

Example:

[<index name>]
homePath = $SPLUNK_DB/<index name>/db
coldPath = $SPLUNK_DB/<index name>/colddb
thawedPath = $SPLUNK_DB/<index name>/thaweddb

These placeholder stanzas:

• Make the Search Head aware of index names

• Allow Log Observer Connect to display the indexes

• Do not impact actual index configuration or data storage on the Indexer

• Do not override Indexer settings

• Enable SPL auto-complete for index names

After updating indexes.conf, restart Splunk on the Search Head.

Expected Result After Fix

• Index appears in Search Head REST response

• Index appears in Log Observer Connect

• Distributed search behavior remains unchanged

Related Links

https://help.splunk.com/en/splunk-observability-cloud/manage-data/view-splunk-platform-logs/set-up-log-observer-connect-for-splunk-enterprise

https://help.splunk.com/en/splunk-enterprise/leverage-rest-apis/rest-api-user-manual/10.2/rest-api-user-manual/basic-concepts-about-the-splunk-platform-rest-api