Splunk DB Connect: Experiencing "Cannot communicate with task server" error on Splunk cloud instance

Applicability

Description

Experiencing "Cannot communicate with task server" error on Splunk cloud instance

Investigation

1. Validate Java version using below command . 

java -version

Note: Compatible java version are 11,17 and 18

2. Check the KV store status by executing below command which should be in Ready status

./SPLUNK_HOME show kvstore-status

3. Verify whether 9998 and 9999 port is not in use by any other services.
4. Search for the error logs and found below error:
SPL:

index="_internal" sourcetype=splunkd *dbx* *task*

Error:

07-02-2024 08:04:32.727 +0000 ERROR ExecProcessor [373762 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/bin/server.sh" Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: Expected 0 or 1 Atom entries; found 2
07-02-2024 08:04:32.727 +0000 ERROR ExecProcessor [373762 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/bin/server.sh" 	at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollectorFactory.createFromStackHecConfiguration(HttpEventCollectorFactory.java:80)
07-02-2024 08:04:32.727 +0000 ERROR ExecProcessor [373762 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/bin/server.sh" 	at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollectorFactory.create(HttpEventCollectorFactory.java:50)

This error occurs when HEC configuration is not recognised by Splunk DB Connect. So in order to resolve this issue follow steps provided in Resolution.

Resolution

1. Go to local folder of Splunk DB Connect folder through CLI. 
Path of dbx_settings.conf: (/opt/splunk/etc/apps/splunk_app_db_connect/local/dbx_settings.conf)
2. Edit dbx_settings.conf file and add below stanza and provide required information as per the suggestion and save the file. 

[hec]
hecUris = <hec endpoint> ( You will find it in $SPLUNK_HOME/etc/apps/100-whisper-searchhead app > local > cloud.conf )
hecToken = <> ( You will find it Settings > Data Input > dbconnect-httpinput and if not present, create new token from Settings > Data inputs )

3. Restart Splunk Services after making the changes.