The Splunk DB Connect inputs sometime experience intermittent failures while collecting data resulting a miss during data collection.

Applicability

Description

The Splunk DB Connect input was running properly; however, it was observed that inputs sometime fails to intermittently collect the data that resulting a miss during data collection with the following error logged in Splunk:

2024-08-27 16:02:55.480 -0800 [QuartzScheduler_Worker-4] INFO- Job 'RPAUTOMATION_W093558_PROD-BPA-Terminated_log' finished with status: FAILED

Investigation

During investigation, it was observed that it was observed that few inputs sometime fails to intermittently collect the data which results in missing data in Splunk, with the following error logged in Splunk:

2024-08-27 16:02:55.480 -0800 [QuartzScheduler_Worker-4] INFO org.easybatch.core.job.BatchJob - Job 'RPAUTOMATION_W093558_PROD-BPA-Terminated_log' finished with status: FAILED

2024-08-27 16:02:55.518 -0800 [QuartzScheduler_Worker-31] ERROR org.easybatch.core.job.BatchJob - Unable to write records
org.apache.http.conn.ConnectTimeoutException: Connect to 127.0.0.1:8088 [/127.0.0.1] failed: Read timed out
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEventBatch(HttpEventCollector.java:141)
at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEvents(HttpEventCollector.java:113)
at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollectorLoadBalancer.uploadEvents(HttpEventCollectorLoadBalancer.java:52)
at com.splunk.dbx.server.dbinput.recordwriter.HecEventWriter.writeRecords(HecEventWriter.java:36)
at org.easybatch.core.job.BatchJob.writeBatch(BatchJob.java:234)
at org.easybatch.core.job.BatchJob.call(BatchJob.java:102)
at org.easybatch.extensions.quartz.Job.execute(Job.java:59)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:475)
at sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:469)
at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:159)
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1320)
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1233)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:417)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:389)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
... 19 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out

Based on the error, it appears that HEC ingestion is failing due to a "Socket Read Timeout." This indicates that the HEC is not processing requests within the expected timeframe.

Resolution

To mitigate this issue, please follow the steps outlined below:

1. Set dedicatedIoThreads = 8 in $SPLUNK_HOME/etc/system/local/inputs.conf
   This allows HEC to process multiple requests simultaneously but should not exceed the number of CPU cores.
2. Set parallelIngestionPipelines = 2 in $SPLUNK_HOME/etc/system/local/server.conf
   Increasing the number of parallel ingestion pipelines enhances forwarder throughput and allows for the simultaneous processing of multiple inputs.
3. Set batch_upload_size = 500 in $SPLUNK_HOME/etc/apps/splunk_app_db_connect/local/db_inputs.conf
   This specifies the number of rows to upload to HEC in one batch.
4. Set maxHecContentLength = 5242880 in $SPLUNK_HOME/etc/apps/splunk_app_db_connect/local/dbx_settings.conf
   This defines the maximum size of each batch when uploading to HEC.

After adding these parameters, user needs to restart the Splunk service.