Can't communicate with the task server error when setting Up Splunk DB Connect

Applicability

Splunk Enterprise Splunk 8.1.0. Splunk DB Connect is version 3.11.1

Description

Issues Setting Up Splunk DB Connect (3.11.1) while trying to install DBConnect on Heavy Forwarder, so it can tap into a SQL DB and run certain queries. As a result, got KVStore message warnings for DB Connect app.

Investigation

1.How to test the error:

• Confirm Splunk DB Connect is at the latest version, along with Splunk.
• Run java -version command and confirm Java version installed is open jdk version.
• Confirm Splunk DB Connect installation method. The installation of JAVA was using the yum command.
• Use netstat command and confirm default port is open and no Firewall rules blocking it.
• Assure the JAVA_HOME path is the right one by running find / -name java command to find right location of java path and use it on app settings. ** Try to point to the JRE path instead of the JDK.

On $SPLUNK_HOME/var/log/splunk/splunkd.log 

From splunkd.log:
 

01-16-2023 14:23:57.778 -0800 INFO ExecProcessor - message from "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.sh" action=start_dbxquery_server, configFile=/opt/splunk/etc/apps/splunk_app_db_connect/config/dbxquery_server.yml
01-16-2023 14:23:57.878 -0800 INFO ExecProcessor - message from "/opt/splunk/etc/apps/splunk_app_db_connect/bin/server.sh" action=start_task_server, configFile=/opt/splunk/etc/apps/splunk_app_db_connect/config/dbx_task_server.yml
01-16-2023 14:23:58.279 -0800 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.sh" 14:23:58.274 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - initializing secret kv store collection
01-16-2023 14:23:58.476 -0800 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/splunk_app_db_connect/bin/server.sh" 14:23:58.467 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - initializing secret kv store collection
01-16-2023 14:23:58.753 -0800 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.sh" 14:23:58.752 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - secret KV Store not found, creating
01-16-2023 14:23:58.760 -0800 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.sh" action=dbxquery_server_start_failed error=com.splunk.HttpException: HTTP 503 -- KV Store initialization failed. Please contact your system administrator. stack=com.splunk.HttpException.create(HttpException.java:84)\\com.splunk.DBXService.sendImpl(DBXService.java:132)\\com.splunk.DBXService.send(DBXService.java:44)\\com.splunk.HttpService.post(HttpService.java:387)\\com.splunk.EntityCollection.create(EntityCollection.java:95)\\com.splunk.EntityCollection.create(EntityCollection.java:83)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.initialize(SecurityFileGenerationUtil.java:249)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.initEncryption(SecurityFileGenerationUtil.java:49)\\com.splunk.dbx.command.DbxQueryServerStart.startDbxQueryServer(DbxQueryServerStart.java:82)\\com.splunk.dbx.command.DbxQueryServerStart.streamEvents(DbxQueryServerStart.java:50)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.command.DbxQueryServerStart.main(DbxQueryServerStart.java:95)\\

On $SPLUNK_HOME/var/log/splunk/mongod.log
 

***aborting after fassert() failure
2023-01-16T18:48:40.613Z W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.
2023-01-16T18:48:40.629Z F NETWORK [main] The provided SSL certificate is expired or not yet valid.
2023-01-16T18:48:40.629Z F - [main] Fatal Assertion 28652 at src/mongo/util/net/ssl_manager.cpp 1120
2023-01-16T18:48:40.629Z F - [main]

2. Execute the following:

• Run $SPLUNK_HOME/bin/splunk show kvstore-status command and confirm the KVStore status. It should be in "ready" status.
• Use the openssl x509 -enddate -noout -in ./server.pem command to confirm Splunk certificate status.

3. Results after testing:

• The kvstore status was "failed".
• The server.pem certificate was expired.
• After running searches index=_internal sourcetype=*dbx* error and index=_internal splunk_app_db_connect didnt receive any result.

 

Moreover, if you find any warning message like below on Splunk UI, you can check and validate the Splunk user permissions on DB connect app directory ( /opt/splunk/etc/apps/splunk_app_db_connect )

Introspecting scheme=server: Unable to run “/opt/splunk/etc/apps/splunk_app_db_connect/bin/server.sh --scheme”: child failed to start: Permission denied

 

Resolution

1. Based on SSL certificate messages, renew Splunk certificate if is expired.  If using the Splunk default certificates, move the $SPLUNK_HOME/etc/auth/server.pem file and restart Splunk to regenerate the certificate.

2. Check KVstore and confirm that KVstore is on ready status:

3.Confirm that "Can't communicate with task server " messages are  not longer present..

4.If there is any permission error on Splunk UI, provide 755 permission to Splunk user on DB connect app folder.

Related Links

Splunk_DB_Connect_Release_Notes
DeployDBX/Installdatabasedrivers
Admin/TroubleshootKVstore

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release