After upgrade to 9.2.1 REST endpoint /services/deployment/server/clients not working as expected.

Applicability

Description

After upgrade to 9.2.1 REST endpoint /services/deployment/server/clients to list the deployment clients was not working as expected for custom user/role even though "list_deployment_server" and "list_deployment_client" capabilities were preset along with rest command capabilities.

Investigation

1. After upgrade to 9.2.1, the rest endpoint to list deployment clients was not working for custom roles but admin was still able to list the clients using rest api.

2. Created a separate rest user for test purpose with the below capabilities all necessary to list the clients and run rest command.

rest_apps_view
rest_properties_get
rest_properties_set
rtsearch
schedule_search
search
list_deployment_server
list_deployment_client

3. Ran the below rest search to list the clients but the rest user was not able to list the clients even though the required capabilities were given to the user/role. But the admin user could list the clients with the same search.

|rest /services/deployment/server/clients

4. Tried adding the below configuration on forwarder outputs.conf so as to whitelist all the relevant logs to be sent to Indexer layer so that the monitoring console can read them. But this did not help to fix the issue.

./outputs.conf
[tcpout] 
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker|_dsclient|_dsphonehome|_dsappevent)

5. The issue was not with the rest endpoint but was with the indexes permission of the rest user/role.

Resolution

In Splunk 9.2.x, 2 new indexes are introduced namely "_dsphonehome, _dsclient, and _dsappevent which are related to the logs from Deployment server and deployment client.

By default, only admin user will have permission to search the 3 newly added indexes. By giving the rest user/role to search these 3 new indexes, the issue was resolved and the endpoint started to list the clients.

Settings-->Role-->rest role-->Indexes-->select the 3 indexes--->save

Related Links

Deployment Client Rest Endpoint

Required Capabilities to list Clients

Editing roles