How to audit self-service changes in Splunk Cloud

Applicability

Splunk Cloud

Description

You would like to identify what changes were performed and who performed the changes as well. There is no standalone log which indicates this. 

Note: It is important to indicate the investigation was done to identify the user who changed the retention policy of an index. Those are the examples used. However, it should be possible to track down other changes as well.

 

Investigation

• Find out what is the endpoint or uri used to perform the change
  • original_uri=/servicesNS/nobody/cloud_administration/cluster_blaster_indexes/sh_indexes_manager/rq_test_delete
  • Where rq_test_delete is the name of the index modified
  • python.log should help you find the uri
• Run this search using the uri: 
  • Where "rq_test_delete" is the index and frozenTimePeriodInSecs the setting that was modified

index="_internal" rq_test_delete sourcetype="splunk_python" frozenTimePeriodInSecs

• The search should be ran using a timeframe for when the change was done
• The results should be analyzed to confirm there was indeed a change in the configuration
• Once you identified the event showing the change, gather the "x-request-id" at the end of the event
• Run the following search:
  • Where c5af7333-e217-40 is the request id

index="_internal" c5af7333-e217-40 sourcetype="splunkd_access"

• The results will tell you who performed the change 
• The co-relation of these 2 logs is provided by the "x-request-id"

Resolution

Here you can see an example of the steps described in the investigation:
 

index="_internal" c5af7333-e217-40 sourcetype="splunkd_access"

 

2024-01-22 16:42:45,952 +0000, level=INFO, pid=2820742, tid=140569762883392, comp=remote_configs_proxy, loc=remote_configs_proxy:339, performing GET after redirected method=POST, original_uri=/servicesNS/nobody/cloud_administration/cluster_blaster_indexes/sh_indexes_manager/rq_test_delete, shaded_uri=/servicesNS/nobody/cloud_administration/admin/sh_indexes_manager/rq_test_delete, query={'output_mode': 'json'}, form={'datatype': 'event', 'frozenTimePeriodInSecs': '5616000', 'maxGlobalDataSizeMB': '0', 'maxGlobalRawDataSizeMB': '1024', 'maxTotalDataSizeMB': '0', 'metric.timestampResolution': 's', 'archiver.enableDataArchive': '0', 'archiver.maxDataArchiveRetentionPeriod': '0'}, x-request-id=c5af7333-e217-40

The original value for frozenTimePeriodInSecs was 6480000
 

index="_internal" c5af7333-e217-40 sourcetype="splunkd_access"

 

127.0.0.1 - user@domain.com [22/Jan/2024:16:42:45.959 +0000] "GET /servicesNS/nobody/cloud_administration/admin/sh_indexes_manager/rq_test_delete?output_mode=json HTTP/1.0" 200 2078 "-" "Python-httplib2/0.20.4 (gzip)" "-" "c5af7333-e217-40" - 8125ms

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release