HTTP 403 Forbidden -- b'{"text":"Invalid token","code":4}' - splunklib.binding.HTTPError

Applicability

Description

ITSI "setseverityfields" command failed with the following HEC token error.

2024-09-18 23:08:36,758+0000 process:820349 thread:MainThread ERROR [command_set_severity_fields_v2] [__init__:1433] [exception] Notable event creation failed.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/notable_event_kpi_alert.py", line 202, in create_notable_events
event_ids = notable_event.create_bulk(alerts)
File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/base_event_management.py", line 34, in wrapper
r = fx(*args, **kwargs)
File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/notable_event.py", line 333, in create_bulk
sourcetype=events[0].get('sourcetype'))
File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/push_event_manager.py", line 176, in push_events
return self.push_event_object.write_events(events_to_push)
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/modular_input/event_writer.py", line 460, in write_events
raise last_ex
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/solnlib/modular_input/event_writer.py", line 446, in write_events
headers=self.headers,
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/splunklib/binding.py", line 321, in wrapper
return request_fun(self, *args, **kwargs)
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/splunklib/binding.py", line 76, in new_f
val = f(*args, **kwargs)
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/splunklib/binding.py", line 809, in post
response = self.http.post(path, all_headers, **query)
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/splunklib/binding.py", line 1308, in post
return self.request(url, message)
File "/opt/splunk/etc/apps/SA-ITOA/lib/SA_ITOA_app_common/splunklib/binding.py", line 1337, in request
raise HTTPError(response)
splunklib.binding.HTTPError: HTTP 403 Forbidden -- b'{"text":"Invalid token","code":4}'

The following error remains during the traceback of the exemption.

Traceback (most recent call last):
File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/notable_event_kpi_alert.py", line 92, in transform_to_kpi_events
self.create_notable_events(events_data)
File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/notable_event_kpi_alert.py", line 206, in create_notable_events
raise NotableEventKPIAlertException(_('Unable to create notable event, %s') % e)
ITOA.event_management.notable_event_kpi_alert.NotableEventKPIAlertException: Unable to create notable event, HTTP 403 Forbidden -- b'{"text":"Invalid token","code":4}'

Investigation

The HEC tokens were responsible for POSTing the generated notables and episodes into the Splunk index in ITSI.

Resolution

There are two ways to resolve this issue with the HEC token.

Method 1: Disabling and Enabling the HEC token in the Stack.

1. Navigate to Settings and click "Data Inputs".
2. In the list of inputs, click "HTTP Event Collector" input.
3. Type ITSI in the filter to filter the list.
4. Disable the "Auto Generated ITSI Event Management Token" using the "Disable" link in the Actions.

5. Click Disable.
6. Reload the same page.
7. Enable the "Auto Generated ITSI Event Management Token" using the "Enable" link in the Actions.

Method 2: Deleing the existing HEC token for the input.

1. Navigate to Settings and click "Data Inputs".
2. In the list of inputs, click "HTTP Event Collector" input.
3. Type ITSI in the filter to filter the list.
4. Disable the "Auto Generated ITSI Event Management Token" using the "Delete" link in the Actions.

5. Click Delete.
6. Reload the same page.
7. Navigate to Settings, and choose "Server Controls".
8. Click Restart.
9. After the restart, the new HEC input will be created for the same deleted input.

Related Links

Splunk Documentation - Link