User Behavior Analytics Not Sending Threats and Anomalies to Splunk ES as Notable Events

Applicability

Description

Splunk UBA is not sending threats and anomalies to Splunk Enterprise Security (ES) as Notable Events. Despite following the documentation steps to configure the Output Connector and verifying the connection settings, threats generated in UBA do not appear as Notable Events in Splunk ES. This prevents security analysts from seeing UBA-detected threats in their centralized ES console, reducing visibility and impacting incident response capabilities.

 

Investigation

During investigation, it was found that despite following all steps in the Splunk UBA documentation for configuring the Output Connector to send threats to Splunk ES, the anomalies and threats were still not appearing as Notable Events in Splunk ES. Further investigation revealed that the issue was related to the "HTTP Event Collector (HEC) token"configuration. Specifically, the HEC token was either not properly configured in the UBA properties file, invalid, expired, or not synchronized correctly with the Splunk ES HEC configuration. The Output Connector logs showed authentication failures with error code 4 ("Invalid token"), confirming that UBA could not authenticate with the Splunk HEC endpoint to send threat data.

The root cause was determined to be missing or incorrect HEC configuration parameters in the uba-site.properties file, which are required for UBA to properly authenticate and send events to the Splunk HEC endpoint. While the Output Connector UI may show as configured, the underlying HEC parameters in the properties file must also be correctly set for the integration to function.

Resolution

To resolve this issue and enable UBA to send threats and anomalies to Splunk ES as Notable Events, follow these steps:

Step 1: Obtain the HEC Token from Splunk Enterprise

In Splunk Enterprise:

1. Navigate to Settings → Data Inputs → HTTP Event Collector
2. Locate the existing HEC token named "SplunkES-UBA-Integration.v1" 
3. Click on the token name to view details
4. Copy the Token Value

Step 2: Add HEC Configuration to uba-site.properties

On the UBA management node, edit the properties file:

sudo vi /etc/caspida/local/conf/uba-site.properties

Add the following stanza at the end of the file:

splunkes.hec.host=<Splunk_IP>
splunkes.hec.token.value=<HEC_Token_Value>
splunkes.hec.port=8088

Example for Splunk Enterprise:

splunkes.hec.host=a.aa.bb.cc
splunkes.hec.token.value=12345678-1234-1234-1234-123456789abc
splunkes.hec.port=8088

Example for Splunk Cloud:

splunkes.hec.host=stack.splunkcloud.com
splunkes.hec.token.value=12345678-1234-1234-1234-123456789abc
splunkes.hec.port=443

Important Notes:

1. Replace <Splunk_IP_or_Hostname> with your actual Splunk server IP address or hostname
2. Replace <HEC_Token_Value> with the token copied from Step 1
3. For Splunk Cloud with custom HEC configurations, use port 443 instead of 8088
4. For standard Splunk Enterprise, use port 8088

Step 3: Sync Configuration Across Cluster (Multi-node Only)

For multi-node UBA cluster, sync the configuration to all nodes:

/opt/caspida/bin/Caspida sync-cluster /etc/caspida/local/conf/

Step 4: Restart UBA Services

Restart all UBA services to apply the configuration changes:

/opt/caspida/bin/Caspida stop-all
/opt/caspida/bin/Caspida start-all

Step 5: Test HEC Connectivity

From the UBA server, test the HEC endpoint:

curl -k -H "Authorization: Splunk <HEC_Token_Value>" https://<Splunk_Host>:8088/services/collector/event -d '{"event":"test from UBA"}'

Expected Output:

Related Links

Document Link: Reference