Splunk On-Call and HEC

Applicability

Description

Splunk Enterprise and Splunk Cloud can be connected to Splunk On-Call via HEC by setting up a webhook in Splunk On-Call. Before setting up the connection from On-Call, the HTTP Event Collector must be configured and enabled for a Splunk instance.

Tip: When setting up the HEC in Splunk, create a new Source Type for the type of data that you’re sending in. This allows for you to send in and keep track of multiple different types of OnCall data like chats, incident action logs, different teams incidents, etc.

Outgoing webhooks can be created to trigger on any available On-Call event type (Incident, Paging, On-Call, Chat). For all created webhooks, the Method, Content-Type, Custom Headers, and URL can remain the same, with the only necessary change being to the payload.

Method: POSTContent-Type: application/json

While the URL will be the same for each webhook, keep in mind that the URL will vary with different deployments of Splunk:

Note: Some Splunk instances may use port 443 instead of 8088 for event ingestion.

The header will be the same for all webhooks and Splunk deployments. Be sure to replace <token> with the appropriate value for the HEC.

From here, the payload will depend on the webhook's event type and use case, and can be configured accordingly.

Investigation

If the both the webhook and the HEC are set up correctly, but Splunk On-Call data is not reaching Splunk, it is likely because the call is being blocked by Splunk. Allowlisting the Splunk On-Call Public IP, 34.208.78.175, should allow for the connection to flow.

Resolution

If you've confirmed both the HEC and the On-Call webhook are set up correctly, as well as allowed the Splunk On-Call IP address in Splunk, please reach out to Splunk On-Call Support for further assistance!

Related Links

Splunk HEC Setup: https://docs.splunk.com/Documentation/Splunk/9.0.2/Data/UsetheHTTPEventCollector

Splunk On-Call HEC Integration: https://docs.splunk.com/observability/en/sp-oncall/spoc-integrations/splunking-victorops-data.html

Splunk On-Call Outgoing Webhooks: https://docs.splunk.com/observability/en/sp-oncall/admin/get-started/custom-outbound-webhooks.html

Splunk On-Call IP Ingress/Egress Filtering: https://docs.splunk.com/observability/en/sp-oncall/admin/get-started/egress-filtering-cloudflare.html