Splunk searches fail because of connectivity problems

Applicability

Splunk Enterprise

Description

When running searches the following errors appear:

12 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors.
[subsearch]: Unknown error for indexer: Indexer36. Search Results might be incomplete! If this occurs frequently, check on the peer.
[subsearch]: Unknown error for indexer: Indexer39. Search Results might be incomplete! If this occurs frequently, check on the peer.
[subsearch]: Unknown error for indexer: Indexer41. Search Results might be incomplete! If this occurs frequently, check on the peer.
[subsearch]: Unknown error for indexer: Indexer43. Search Results might be incomplete! If this occurs frequently, check on the peer.
[subsearch]: Unknown error for indexer: Indexer46. Search Results might be incomplete! If this occurs frequently, check on the peer.
[subsearch]: Unknown error for indexer: Indexer47. Search Results might be incomplete! If this occurs frequently, check on the peer.
[subsearch]: Unknown error for indexer: Indexer48. Search Results might be incomplete! If this occurs frequently, check on the peer.
[subsearch]: Unknown error for indexer: Indexer49. Search Results might be incomplete! If this occurs frequently, check on the peer.
[subsearch]: Unknown error for indexer: Indexer50. Search Results might be incomplete! If this occurs frequently, check on the peer.
[subsearch]: Unknown error for indexer: Indexer51. Search Results might be incomplete! If this occurs frequently, check on the peer.
[subsearch]: Unknown error for indexer: Indexer53. Search Results might be incomplete! If this occurs frequently, check on the peer.
[subsearch]: Unknown error for indexer: Indexer60. Search Results might be incomplete! If this occurs frequently, check on the peer.

This environment has 15 Search Heads, 60 Indexers, and an Indexer Cluster Master.

Investigation

The bad searches were:

1. Throttled SPO NRT tasks rate
2. Stuck Grabber
3. Lograbber - CannotSaveTokenException raised
 

Errors and WARN

 

01-01-2020 08:40:03.188 +0000 ERROR SearchOperator:savedsplunk - Error in 'savedsearch' command: Unable to find saved search named 'Lograbber - Found tenant services 
with no Test API results'.
host = SearchHead1source = /opt/splunk/SearchHead1-sh_ls_-20200101-102734-vBHRe7/log/splunkd.logsourcetype = splunkd

01-01-2020 08:05:09.333 +0000 WARN SavedSplunker - Reached maximum amount of time allowed to spend in per-result alerts for savedsearch_id="nobody;search;Proxy - 
Nginx - Level Error", sid="scheduler__oliezhov__search__RMD5a49f7473db4b124b_at_1577865600_333_C347B6F1-D5EF-46C3-9B20-5B6237A19BF9", max_alerts_time=300 (seconds), 
fired_alerts=466
host = SearchHead1source = /opt/splunk/SearchHead1-sh_ls_-20200101-102734-vBHRe7/log/scheduler.logsourcetype = scheduler

The types of errors that were logged for the user 'fred' were as follows:

2020-01-01 08:49:46,750 INFO [5e0c5d2ab57f4ff478e690] view:456 - loading SID "scheduler__fred__search__RMD5f442d11fa5911fd8_at_1577865600_392_D1C92F89-E031-4CC0-A5CD-
97C996915C21"
host = SearchHead1source = /opt/splunk/SearchHead1-sh_ls_-20200101-102734-vBHRe7/log/web_service.logsourcetype = splunk_web_service

 

01-01-2020 10:00:04.284 +0000 INFO SavedSplunker - savedsearch_id="nobody;search;Lograbber - Redis rate limiter Errors", search_type="", user="fred", app="search", 
savedsearch_name="Lograbber - Redis rate limiter Errors", priority=default, status=success, digest_mode=1, scheduled_time=1577872800, window_time=0, dispatch_time=
1577872803, run_time=0.887, result_count=0, alert_actions="", sid="scheduler__fred__search__RMD52ce6131efe702eb8_at_1577872800_584_C347B6F1-D5EF-46C3-9B20-5B6237A19BF9", 
suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""
host = SearchHead1source = /opt/splunk/SearchHead1-sh_ls_-20200101-102734-vBHRe7/log/scheduler.logsourcetype = scheduler

 

01-01-2020 09:00:41.187 +0000 INFO SavedSplunker - savedsearch_id="nobody;search;Lograbber - CannotSaveTokenException raised", search_type="", user="fred", app="search", 
savedsearch_name="Lograbber - CannotSaveTokenException raised", priority=default, status=success, digest_mode=1, scheduled_time=1577869200, window_time=900, 
dispatch_time=1577869204, run_time=36.119, result_count=0, alert_actions="", sid="scheduler__fred__search__RMD5c434f1137997b9f1_at_1577869200_477_C347B6F1-D5EF-46C3-9B20
-5B6237A19BF9", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""
host = SearchHead1source = /opt/splunk/SearchHead1-sh_ls_-20200101-102734-vBHRe7/log/scheduler.logsourcetype = scheduler

 

01-01-2020 08:04:55.765 +0000 INFO SavedSplunker - savedsearch_id="fred;search;Stuck Grabber", search_type="", user="fred", app="search", savedsearch_name="Stuck Grabber",
priority=default, status=success, digest_mode=1, scheduled_time=1577865600, window_time=0, dispatch_time=1577865604, run_time=291.153, result_count=0, alert_actions="", 
sid="scheduler__fred__search__RMD51d89c4ce0be19629_at_1577865600_352_C347B6F1-D5EF-46C3-9B20-5B6237A19BF9", suppressed=0, thread_id="AlertNotifierWorker-22", 
workload_pool=""
host = SearchHead1source = /opt/splunk/SearchHead1-sh_ls_-20200101-102734-vBHRe7/log/scheduler.logsourcetype = scheduler

Resolution

Following the review of the error:

[subsearch]: Unknown error for indexer Search Results might be incomplete! If this occurs frequently, check on the peer

1- Disable SSL compression, If enabled on the Search Head
 

[sslConfig]
useClientSSLCompression=false

(defaults)

https://docs.splunk.com/Documentation/Splunk/7.3.3/Admin/Serverconf
 

[sslConfig]useClientSSLCompression = <boolean>
* Turns on HTTP client compression.
* Server-side compression is turned on by default. Setting this on the
client-side enables compression between server and client.
* Enabling this potentially gives you much faster distributed searches
across multiple Splunk instances.
* Default: true

The current value on SearchHead1 was:
 

[sslConfig]
useClientSSLCompression = true

2- Set the parameters below to these values in server.conf on the Indexers.
 

streamInWriteTimeout=120

And

busyKeepAliveIdleTimeout =120

https://docs.splunk.com/Documentation/Splunk/7.3.3/Admin/Serverconf
 

busyKeepAliveIdleTimeout = <integer>
* How long, in seconds, that the Splunkd HTTP server allows a keep-alive
connection to remain idle while in a busy state before forcibly
disconnecting it.
* Use caution when configuring this setting as a value that is too large
can result in file descriptor exhaustion due to idling connections.
* If this number is less than 12, it is set to 12.
* Default: 12streamInWriteTimeout = <positive number>
* The timeout, in seconds, for uploading data to the http server.
* When uploading data to http server, if the http server is unable
to write data to the receiver for the specified value, the operation
aborts.
* Default: 5

The current values on IndexerMaster1 were:

[httpServer]
busyKeepAliveIdleTimeout = 12
streamInWriteTimeout = 5

ULIMIT values

The ulimit values for the number of open files (nofiles) need to be increased from 1024 to a minimum of 64000 as explained below:


SearchHead1 was:

ULIMIT

nofiles 1024 (under)
process 515631


IndexerMaster1 was:

nofiles 1024
process 257592

The other indexers should also be verified to ensure the nofile setting is 64000 or above,


Documentation

https://docs.splunk.com/Documentation/Splunk/7.3.3/Installation/Systemrequirements

System-wide Resource ulimit invocation Minimum recommended values:

• Open files ulimit -n 64000
• User processes ulimit -u 16000

Other solutions:

1. A rolling restart of the indexers 
2. Tuning on connectionTimeout mitigates the issue

 

Revision History

Revision	Publish Date	Comments
1.0	28-Sep-2026	Initial Release