After upgrade to 9.2.x, Deployment Server Forwarder Management UI experiences latency (slowness)

Applicability

Description

After upgrading the Deployment Server to version 9.2, performance degradation has been observed. 

• On Deployment Server Web UI,  select Settings -> Forwarder Management,  it took at least 10-20 seconds for UI to respond. 
• Searching for any Deployment Client by hostname is taking longer than expected.

Investigation

1) Deployment Server is forwarding its internal logs to indexer Clusters on (9.2). 

2) According to this Documentation Upgradepre-9.2deploymentservers , following configurations are in place: 

a) Configured outputs.conf on Deployment Server:

[indexAndForward]
index = true
selectiveIndexing = true

b) Ensured Indexer tier have the following internal indexes created: 

[_dsphonehome]
[_dsclient]
[_dsappevent]

c) From Deployment Server - Search and Reporting:

Following SPL query returns results from all three internal indexes mentioned above. 

index= _ds*

3) From  Deployment Server Web UI,  select Settings -> Forwarder Management, 

 All clients are listed under the UI, but it takes a long time for the UI to respond, this only happens after upgrade to 9.2 verison. 

The WebUI is calling a REST API to retrieve client data.

| rest splunk_server=local /services/deployment/server/clients

From version 9.2.x, the REST API implementation was changed to use SPL searches. 

These searches used can be found from $SPLUNK_HOME/etc/apps/SplunkDeploymentServerConfig/default/savedsearches.conf

Resolution

Reason for UI slowness

9.2 introduced a Deployment Server Scalability feature. This feature enables Deployment Server(DS) to handle more agents, create a deployment server cluster to handle incoming traffic from clients better and did some code improvements to handle parallel calls to deployment server more efficiently. It also changed the way DS works internally which in some cases might impact performance of certain endpoints.

With Splunk 9.2+, DS now uses internal indexes to gather certain data. Specifically, deployment/server/clients obtains phonehome data up to the DS reload time. In case of a large amount of phonehomes(a lot of clients/short phonehome interval) this can cause a slowdown, as observed in this issue. 

Splunk's Development Team has fixed the issue and the fix is available in Splunk Enterprise versions 10.0.x, 9.4.2, 9.3.4. 9.2.6.

Workaround

The following solutions can be applied all together, or you might decide to only use some of the solutions after evaluating the impact and your use case. 

Option (1):

Modify $SPLUNK_HOME/etc/system/local/deploymentclient.conf file on the deployment clients, modifying the phoneHomeIntervalInSecs entry to a bigger value. This will decrease the amount of data incoming to the _dsphonehome index, resulting in faster searches.

Implications of the Impact:
Changing the phonehome interval will decrease the amount of data coming over time. This will make the performance degradation happen more slowly, but it will still happen. 

As an example:

(a) In an environment, with phoneHomeIntervalInSec=60 seconds, performance degradation happens over the 5 day period after which the performance is not satisfying. By changing the phonehome interval to 5 minutes( 300 seconds ), the performance will degrade 5 times slower - over the course of 25 days.
(b) Changing phonehome interval means that the application deployment will be affected - since it is tied to the phonehome, the applications will take 5 minutes to be updated instead of 60 seconds.
(c) In this particular environment, there is a search that runs every 10 minutes, collecting information about the online clients. Changing phonehome to 5 minutes will affect the results of this query. 
(d) Changing phonehome interval means that after reloading DS, clients will show up after sending a phonehome - for example, changing the phonehome interval to 5 minutes means that it might take up to 5 minutes for all the clients to show up in the forwarder management UI.

Option (2):
Reduce the _dsphonehome index retention from the default value of 604800 (7 days) to something smaller. Two to three days retention is usually sufficient for a typical use case, however, adjust according to your business need.

For example: To modify the retention period to 259200 (3 days), edit the indexes.conf and modify the frozenTimePeriodInSecs parameter under the [_dsphonehome] stanza to the value stated:

[_dsphonehome]
frozenTimePeriodInSecs = 259200

The change must be applied to the host that stores the index. If the index is stored in the indexer layer, push the change to all relevant indexers. If the indexers are clustered, the change must be pushed from the Cluster Manager. 

Implications of the Impact:
Reducing the _dsphonehome index retention naturally reduces the number of phone home events that you can retrieve. This is usually not a problem since the events are only used by the Deployment Server. This option should ideally be combined with option 1 for the best result.

Option (3):
Reload the deployment server when performance is degraded by running splunk reload deploy-server. This will cause DS to reload, making the queries consider a smaller amount of data, improving performance for some time. They can also restart the whole splunkd instance, the effect will be the same.

Implications of the Impact:
Reloading DS will cause the clients to be considered offline until they send a phone home. This means that after the reload, some clients are not visible until they phone home. After the initial phone home interval, the clients should have phoned home and will show up on the UI. The DS will continue working normally.

Option (4):
Modify the SPL used for DS deployment/server/clients endpoint to take less data into account. See explanation below for implications and the updated query.

Implications of the Impact:
The SPL change that suggested will modify the behavior of the deployment/server/clients to only consider data from shorter time period after a reload. Let us assume 24-hour time period. 

1. This means that offline clients(the clients that stopped phone-homing) will not show up after 24-hours of being offline.
2. This means that clients must phone-home at least once in 24-hours to show up on the dashboard. Clients that haven’t phoned-home in the last 24 hours will not be considered connected to DS and will not show up.
3. This solution would not degrade over time as the amount of data searched will be constant.
4. The exact time period will depend on use-case. 24-hours is only an example; You might want a longer time period to be considered. The longer the time period, the slower potential performance of the endpoint.

For Option (4) to work, on DS, create a folder $SPLUNK_HOME/etc/apps/SplunkDeploymentServerConfig/local. Then, create a file $SPLUNK_HOME/etc/apps/SplunkDeploymentServerConfig/local/savedsearches.conf.
Add the following entry to that file. If an entry with the same stanza exists, delete the entry and paste the new one that can be found below:

[DeploymentServerClients]
search = | tstats prestats=t latest(data.lastPhoneHomeTime) values(data.lastPhoneHomeTime) latest(data.updaterRunning) where index=_dsphonehome [| makeresults \
  | eval comparisonTime=if($reloadTime$ > relative_time(now(), "-24h"), $reloadTime$, relative_time(now(), "-24h")) \
  | return earliest=comparisonTime ] \
  latest=now `filter_by_client_id("$clientId$")` by data.clientId ``` Obtain phonehome values to calculate average PH time.``` \
```Obtain data from dsclient index to enrich phonehome data with client information.``` \
| tstats prestats=t append=t latest(data.build) latest(data.connectionId) latest(data.dns) \
           latest(data.guid) latest(data.hostname) latest(data.instanceId) \
           latest(data.instanceName) latest(data.ip) latest(data.mgmt) latest(data.name) latest(data.package) \
           latest(data.packageType) latest(data.splunkVersion) latest(data.utsname) where index=_dsclient earliest=1 latest=now `filter_by_client_id("$clientId$")` by \
           data.clientId \
```Instead of doing join operation on dsclient index, previous tstats queries obtain data from relevant indexes and merge them with stats command.``` \
| stats latest(data.lastPhoneHomeTime) as lastPhoneHomeTime values(data.lastPhoneHomeTime) as allphs latest(data.updaterRunning) as updaterRunning latest(data.build) as data.build latest(data.connectionId) as data.connectionId latest(data.dns) as data.dns \
           latest(data.guid) as data.guid latest(data.hostname) as data.hostname latest(data.instanceId) as data.instanceId \
           latest(data.instanceName) as data.instanceName latest(data.ip) as data.ip latest(data.mgmt) as data.mgmt latest(data.name) as data.name latest(data.package) as data.package \
           latest(data.packageType) as data.packageType latest(data.splunkVersion) as data.splunkVersion latest(data.utsname) as data.utsname by data.clientId \
```Discard entries from dsclient that do not have a matching entry from dsphonehome index. This emulates join operation.``` \
```A reverse is not necessary - every client that stored phonehome data will have a corresponding entry in dsclient index.``` \
| where lastPhoneHomeTime!="" \
| search `filter_by_phonehome($minPhoneHomeTime$)` \
```Obtain time after the initial processing of the phonehomes is finished.``` \
```Extract last 4 phonehomes and calculate averagePhoneHomeInterval``` \
| eval last4phs=mvindex(allphs,-4,-1) \
``` First phoneHome is the oldest one``` \
| eval oldestph=mvindex(last4phs,0) \
| eval newestph=mvindex(last4phs,-1) \
| eval deltacount=max(1,mvcount(last4phs)-1) \
| eval averagePhoneHomeInterval=floor((newestph-oldestph)/deltacount) \
```Use time() instead of now(), since now() runs at the beginning of search which makes it possible for lastPhoneHomeTime > now``` \
```Use floor() to discard millisecond resolution``` \
| eval actual_ratio=(floor(time())-lastPhoneHomeTime)/averagePhoneHomeInterval \
| search `filter_by_ratio("$greater_than$", $ratio$)` \
| fields - allphs oldestph newestph deltacount last4phs \
```Enrich client data with its latest applications data``` \
| join data.clientId type=left max=0 [ \
    ```Obtain latest application status.``` \
    | tstats latest(data.appName) as appName latest(data.serverClassName) as serverClassName \
      latest(data.action) as action latest(data.result) as result latest(data.failedReason) as failedReason \
      latest(data.checksum) as checksum latest(data.timestamp) as timestamp latest(data.clientId) \
      as data.clientId where index=_dsappevent earliest=$reloadTime$ latest=now \
      `filter_by_client_id("$clientId$")` `filter_by_app("$appName$")` `filter_by_action("$action$")` `filter_by_checksum($checksum$)` \
      `filter_by_error("$has_error$")` by data.key \
    | stats list(appName) as appName list(serverClassName) as serverClassName \
      list(action) as action list(result) as result list(failedReason) as failedReason \
      list(checksum) as checksum list(timestamp) as timestamp count(appName) as countApps by data.clientId \
    | eval appName=mvjoin(appName, "#") \
    | eval serverClassName=mvjoin(serverClassName, "#") \
    | eval action=mvjoin(action, "#") \
    | eval result=mvjoin(result, "#") \
    | eval failedReason=mvjoin(failedReason, "#") \
    | eval checksum=mvjoin(checksum, "#") \
    | eval timestamp=mvjoin(timestamp, "#") \
    ] \
| eval appName=split(appName, "#") \
| eval serverClassName=split(serverClassName, "#") \
| eval action=split(action, "#") \
| eval result=split(result, "#") \
| eval failedReason=split(failedReason, "#") \
| eval checksum=split(checksum, "#") \
| eval timestamp=split(timestamp, "#") \
| search `filter_by_app_count("$has_app_filter$")` \
| rename data.* as *[DeploymentServerClientsBasic]
search = | tstats prestats=t latest(data.lastPhoneHomeTime) where index=_dsphonehome [| makeresults \
  | eval comparisonTime=if($reloadTime$ > relative_time(now(), "-24h"), $reloadTime$, relative_time(now(), "-24h")) \
  | return earliest=comparisonTime ] latest=now by data.clientId \
| tstats prestats=t append=t latest(data.build) latest(data.connectionId) latest(data.dns) \
           latest(data.guid) latest(data.hostname) latest(data.instanceId) \
           latest(data.instanceName) latest(data.ip) latest(data.mgmt) latest(data.name) latest(data.package) \
           latest(data.packageType) latest(data.splunkVersion) latest(data.utsname) where index=_dsclient earliest=1 latest=now \
           by data.clientId \
| stats latest(data.lastPhoneHomeTime) as lastPhoneHomeTime latest(data.build) as data.build latest(data.connectionId) as data.connectionId latest(data.dns) as data.dns \
           latest(data.guid) as data.guid latest(data.hostname) as data.hostname latest(data.instanceId) as data.instanceId \
           latest(data.instanceName) as data.instanceName latest(data.ip) as data.ip latest(data.mgmt) as data.mgmt latest(data.name) as data.name latest(data.package) as data.package \
           latest(data.packageType) as data.packageType latest(data.splunkVersion) as data.splunkVersion latest(data.utsname) as data.utsname by data.clientId \
| where lastPhoneHomeTime != "" \
| where "data.dns" != "" \
| rename data.* as * \
| eventstats count as total \
| head $offset_end$ \
| streamstats count as paginator \
| where paginator>$offset_start$

Afterwards, reload deployment server: $SPLUNK_HOME/bin/splunk reload deploy-server

This change will make the DeploymentServerClients SPL search (used by /services/deployment/server/clients endpoint) limit the search of the _dsphonehome index to reloadTime or the last 24 hours, whichever is shorter (notice the -24h entry). The time can be reduced further (for example to -12h), but keep in mind that this means that any deployment client that has not sent a phone-home in that time period (e.g. 12 hours) will not be reported.

If you decide to further modify that time value, note that there are two instances of -24h - you would need to modify both. The same applies to the DeploymentServerClientsBasic SPL search.

Related Links

Deploymentclientconf
Upgradepre-9.2deploymentservers
Troubleshootingperformance