Splunk Limits Index-Time Fields to 1000 Characters via HTTP Event Collector (HEC)

Applicability

Description

This issue occurs when data is ingested via Splunk's HTTP Event Collector (HEC) and the fields parameter defines index-time fields. If any field value exceeds 1000 characters, it is truncated, irrespective of deployment type or sourcetype configurations like the TRUNCATE property.

This limitation directly affects how Splunk handles field values in the field parameter, causing data loss for excessively long strings.

Investigation

The issue was reproduced using the following steps:

1. Initial Payload Submission:

A curl request was sent to the Splunk HEC endpoint with a fields parameter containing a field value exceeding 1000 characters:

curl https://http-inputs-cloudstack.splunkcloud.com:443/services/collector/event \
-H 'Authorization: Splunk xxxxx-xxxx-xxxx-xxxx-xxxxxxx' \
-d '{"time":1730810951.533,"index":"temp","host":"OCP-JC-TE-V4-PRO","sourcetype":"sourcetype_name","event":"{\"msgLength\":1013}","fields":{"kubernetes":"<long_string_of_characters>"}}'

Outcome: When reviewed in Splunk, the kubernetes field was truncated to 1000 characters.

2. Attempt to Modify props.conf:

The TRUNCATE property in props.conf was adjusted to TRUNCATE=0 under the relevant sourcetype configuration:
[sourcetype_name] 
TRUNCATE=0
Outcome: The truncation issue persisted, confirming that sourcetype-level configurations do not impact the behavior of the fields parameter.

3. Verification Across Deployment Types:

Tests were conducted on both Splunk Cloud and on-premises Splunk Enterprise environments.
Outcome: The truncation occurred consistently in both environments, confirming that this behavior is not specific to a deployment type.

4. Comparison with event Parameter:

The same payload was modified to include the long field value under the event section instead of the fields parameter:

curl https://http-inputs-cloudstack.splunkcloud.com:443/services/collector/event \
-H 'Authorization: Splunk xxxxx-xxxx-xxxx-xxxx-xxxxxxx' \
-d '{"time":1730810951.533,"index":"temp","host":"OCP-JC-TE-V4-PRO","sourcetype":"sourcetype_name","event":"{\"kubernetes\":\"<long_string_of_characters>\"}"}'

Outcome: The field value was successfully ingested without truncation.

Resolution

To work around the limitation of the fields parameter, consider one of the following approaches:

Option 1: Use the event Field Instead of fields

If you need to include long field values, store them in the event field rather than the fields parameter. This way, the values won’t get cut off.

Modified Payload Example:

curl https://http-inputs-temis.splunkcloud.com:443/services/collector/event \ 
-H 'Authorization: Splunk xxxxx-xxxx-xxxx-xxxx-xxxxxxx' \ 
-d '{"time":1730810951.533,"index":"temp","host":"OCP-JC-TE-V4-PRO","sourcetype":"sourcetype_name","event":"{\"kubernetes\":\"<long_string_of_characters>\"}"}'

Option 2: Use tstats to Retrieve Field Values

If you continue to use the fields parameter and some values are truncated, you can still query those values efficiently using tstats.

SPL Query Example:

| tstats count where index="your_index_name" by kubernetes 
| eval length = len(kubernetes)

This lets you check the content and length of the truncated fields while improving query performance. 

Related Links

https://docs.splunk.com/Documentation/Splunk/9.4.0/Admin/Propsconf
https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/UsetheHTTPEventCollector