HEC ingestion showing error codes 503/504

Applicability

Description

During failure periods the next is observed:

• HEC clients received 503 and 504 responses.
• Ingestion loss occurred when sender retries were exhausted.
• Indexer CPU was high or saturated.
• Parse-pipeline queues showed sustained utilization or blocking:
• httpinputq
• parsingqueue
• typingqueue
• aggqueue
• indexqueue could remain relatively low/empty, indicating the bottleneck was upstream of indexing/disk write activity.
• Processor-level inspection identified RegexProcessor as a primary CPU consumer.
• The HEC queue became blocked during high-load periods.

Investigation

The affected deployment used ingest-action rulesets for eks:audit and databricks:json to reduce indexed volume. These filters performed regex evaluation against every incoming event before routing matching events to nullQueue.

Although the rules reduced license consumption by dropping a large percentage of incoming data, they imposed regex-processing overhead on the indexers for the full pre-drop data volume.

Observed behavior:

1. Parse-side queues filled while indexqueue remained comparatively low.
2. Regex processing was the dominant CPU consumer.
3. eks:audit and databricks:json were the primary sourcetypes associated with regex-processing load.
4. HEC 503/504 errors increased during parse-tier saturation.
5. Reducing unrelated search workload improved general CPU headroom but did not independently eliminate HEC failures.
6. A Databricks truncation configuration issue was corrected, but it was determined to predate the HEC incident and was not established as the original trigger.

Index-time regex-based ingest filters contributed significant CPU consumption in the parsing pipeline. The filters evaluated regex expressions against all received events—including events ultimately discarded—causing RegexProcessor load and parse-tier queue saturation.

When parse-tier capacity was exhausted, HEC ingestion experienced backpressure and returned 503/504 responses.

Resolution

1. Removed the ingest-action filter rulesets for:

• eks:audit
• databricks:json

        2.Confirmed that parse-pipeline queues drained after removal:

• typingqueue dropped to near-idle levels.
• parsingqueue and aggqueue dropped to near-idle levels.
• RegexProcessor-related pressure was removed.

After removal of the EKS ingest-action filter, HEC 503 errors dropped to zero and remained at zero for more than 48 hours, confirming situation does persist.

Recommended best practices:

Avoid expensive index-time regex filtering on indexers

Do not rely on index-time regex drop rules as the primary volume-control mechanism for high-volume HEC data when the indexer tier is parse-CPU constrained. Even dropped events require receipt, parsing, and regex evaluation before they can be discarded.