Unable to Delete Saved Searches from the UI.

Applicability

Description

This issue extends beyond saved searches and affects all knowledge objects migrated from the old environment, including dashboards, lookups, and other related objects. It appears that none of the migrated objects can be deleted in the new environment. In some instances, errors are encountered when attempting deletion, while in other cases, the "delete" button is not even displayed. This behavior seems to be consistent across all migrated objects.

Whenever the admin or the user having an admin role try to delete the saved searches, they were encountering the below error message.

“This saved search failed to handle removal request due to Object id=“savedsearch_name” cannot be deleted in config=savedsearches.”

Investigation

1. Deteremined that this savedsearch has not been orphaned by navigating into "Settings -> All configurations -> Reassign Knowledge Objects". 

2. Tried to delete the saved search search from the CLI using the CURL command:
curl -k -u admin:pass --request DELETE https://localhost:8089/servicesNS/admin/search/saved/searches/MySavedSearch
Unfortunately, it didn't work.

3. Checked the configuration location for the saved search, whether it is present on the default directory or local directory and determined that the savedsearches.conf file was present on the local directory.

4. Validated the permission for the report and found that it is not set to "Private".

5. To identify the location of the savedsearch in the conf files, executed the following command on one of the Search Head.

Command:

grep -R "alert_name" /opt/splunk/

The output looked like below:

/opt/splunk/var/run/splunk/confsnapshot/baseline_local/apps/app_name/local/savedsearches.conf:
[savedsearch_name] 

6. Found that the owner for the savedsearch has set to nobody in the local.meta file. Therefore, changed the owner from nobody to admin and performed the restart. Unfortunately, the issue still persists.

Before:

[savedsearches/savedsearch_name]
access = read : [ * ], write : [ admin ]
export = none
owner = nobody
version = 9.2.1
modtime = 1716807026.377801000

After:

[savedsearches/savedsearch_name]
access = read : [ * ], write : [ admin ]
export = none
owner = admin
version = 9.2.1
modtime = 1716807026.377801000

By following the steps outlined in the resolution column, able to delete the savedsearches successfully from the UI without any issues.

Resolution

1. On the Deployer, both default.meta and local.meta files from the old environment were present. This caused issues when pushing the configuration to the Search Head (SH) nodes.

2. During the deployment process, these two files (default.meta and local.meta) were merged into "one default.meta file" on the SH nodes, causing objects to be referenced incorrectly and preventing deletion.

3. Removed the local.meta file from the Deployer, as it was not supposed to be there in the first place.

4. Once the unnecessary local.meta file was deleted, able to successfully delete the knowledge objects that were previously referenced in the default.meta file.