DB connect input stopped sending events to indexer but other DB connect inputs are working fine

Applicability

Splunk Enterprise Splunk DB Connect

Description

DB connect input stopped sending events to indexer but other DB connect inputs are working fine

 

Investigation

How to identify :

Check the db_connect logs (index=_internal sourcetype=dbx*) for messages similar to the example below :
 

2018-10-01 12:07:33.885 +0100 INFO c.s.dbx.server.task.listeners.JobMetricsListener - action=collect_job_metrics connection=XXXXX jdbc_url=null 
db_read_time=91 hec_record_process_time=826 format_hec_success_count=1000 status=FAILED input_name=XXXXXXXXX batch_size=1000 error_threshold=N/A 
is_jmx_monitoring=false start_time=2018-10-01_12:07:27 end_time=2018-10-12_01:07:30 duration=3193 read_count=2095000 write_count=0 filtered_count=0 error_count=0
....
2018-10-01 12:07:33.885 +0100 [QuartzScheduler_Worker-27] INFO org.easybatch.core.job.BatchJob - Job 'XXXXXX' finished with status: FAILED
.....
at com.splunk.dbx.server.dbinput.recordwriter.HecEventWriter.writeRecords(HecEventWriter.java:36)
at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEvents(HttpEventCollector.java:89)
at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEventBatch(HttpEventCollector.java:112)
java.io (https://java.io) .IOException: HTTP Error 400: Bad Request
2018-10-01 12:07:33.885 +0100 [QuartzScheduler_Worker-27] ERROR org.easybatch.core.job.BatchJob - Unable to write records

 

Resolution

Caused by the following bug :

DBX-4614 - DBX 3.1.3 inputs throwing "HTTP Error 400: Bad Request" error when large field length is encountered

Which is a duplicate of :

SPL-156315 [Regression] After upgrade to 7.x, HEC events greater than 512KB are dropped with parsing errors, resulting in degrade of indexing throughput

If you try to send an event to HEC which is > 512KB then the the event is dropped. If DBX is sending the event via HEC and hits this limit it will keep trying to send the event and fail meaning no further events will be indexed unless the event is manually skipped or a substring of the column is sent instead.

This issue is fixed in Splunk 7.0.5 and above, 7.1.3 and above, and 7.2.0 and above by increasing the default maximum HEC event size from 512K to 5MB. A new configuration option maxEventSize was also added to inputs.conf to allow increasing this higher if required :


https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf#http:_.28HTTP_Event_Collector.29
 

maxEventSize = <positive integer>[KB|MB|GB]
* The maximum size of a single HEC (HTTP Event Collector) event.
* HEC disregards and triggers a parsing error for events whose size is
greater than 'maxEventSize'.
* Default: 5MB.

 

Related Links

Splunk Documentation

http: (HTTP Event Collector)
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf#http:_.28HTTP_Event_Collector.29

 

Revision History

Revision	Publish Date	Comments
1.0	28-Sep-2026	Initial Release