Data is not getting indexed using HEC token due to data channel missing

Applicability

Description

A user encountered an issue where they were unable to view or receive any security events from ManageEngine ADSelfServicePlus.
They have a ManageEngine ADSelfServicePlus solution deployed and configured to integrate with their on-premise Splunk Enterprise solution.
They configured ADSelfServicePlus to integrate with Splunk using the HTTP event collector.

Both ADSelfServicePlus and Splunk are configured with the settings as described.
However, over at Splunk, when the user tried to search for the logs from ADSelfServicePlus, they were unable to search for any security event logs. 

Resolution

Root Cause:
Observed consistent errors from the logs, such as below for the token that user has configured:

11-25-2021 15:44:46.499 +0800 ERROR HttpInputDataHandler - Failed processing http input, token name=ADSelfService Plus, channel=n/a, source_IP=203.127.255.12, reply=10, events_processed=0, http_input_body_size=258, parsing_err=""
11-25-2021 15:44:46.519 +0800 ERROR HttpInputDataHandler - Failed processing http input, token name=ADSelfService Plus, channel=n/a, source_IP=203.127.255.12, reply=10, events_processed=0, http_input_body_size=233, parsing_err=""
11-25-2021 15:44:46.841 +0800 ERROR HttpInputDataHandler - Failed processing http input, token name=ADSelfService Plus, channel=n/a, source_IP=203.127.255.12, reply=10, events_processed=0, http_input_body_size=528, parsing_err=""

As can bee seen, the reply as 10 in the logs and according to the HEC reply codes, 10 means missing data channel.

HEC reply codes:
Reply HttpInputReply status event_message: 

0 Success OK Success
1 TokenDisabled FORBIDDEN Token disabled
2 NoAuthorization UNAUTHORIZED Token is required
3 InvalidAuthorization UNAUTHORIZED Invalid authorization
4 TokenNotFound FORBIDDEN Invalid token
5 NoData BAD_REQUEST No data
6 InvalidData BAD_REQUEST Invalid data format
7 IncorrectIndex BAD_REQUEST Incorrect index
8 ServerError has been removed as it is not used anywhere
9 ServerBusy SERVICE_UNAVAILABLE Server is busy
10 NoChannel BAD_REQUEST Data channel is missing
11 InvalidChannel BAD_REQUEST Invalid data channel
12 NoEvent BAD_REQUEST Event field is required
13 BlankEvent BAD_REQUEST Event field cannot be blank
14 AckDisabled BAD_REQUEST ACK is disabled
15 UnsupportedType BAD_REQUEST Error in handling indexed fields
16 QueryStringAuthNotEnabled BAD_REQUEST Query string authorization is not enabled
17 HECHealthy OK HEC is healthy
18 QueuesFull SERVICE_UNAVAILABLE HEC is unhealthy, queues are full
19 AckUnavailable SERVICE_UNAVAILABLE HEC is unhealthy, ack service unavailable
20 QueuesFullAckUnavailable SERVICE_UNAVAILABLE Hec is unhealthy, queues are full, ack service unavailable

As you can see that the data channel in the log is missing (channel=n/a), this happens because the "indexer acknowledgement" is enabled when creating a HEC token and hence Splunk is expecting channel ID for acknowledgement.

Resolution/Recommendation:

• Edit the global settings to have a default index as "main".
• Create a new token with "Enable Indexer Acknowledgement" disabled.
• Choose index as "main"
• Make sure that the communication from the application side contains a data channel for Splunk to authenticate and index the data.

Related Links