After upgrading to version 9.2.x, the list of Deployment Clients (Forwarders) displayed under the Deployment Server is incomplete Or showing error "No clients phoned home"

Applicability

Description

• After upgrade to 9.2.x, the deployment clients can be shown under Deployment Server's Forwarder Management UI page, but the list is incomplete.
• This issue appears to occur only when Deployment Server (DS) is co-located with Distributed Monitoring Console (DMC) 
• The issue can sometimes be seen as error "No clients phoned home". 

Investigation

As an example here, the Deployment Server in question is showing only 21 out of 60 under its Forwarder Management page, and the number varies each day.

Step 1:

SSH to Deployment Server and run following command to check its deployment clients:

splunk list deploy-clients

▶  The result shows only 21 deployment clients IP. 

Step 2:

On Deployment Server's Web UI, run this SPL query to check how many clients recently phone home from REST API:

|rest splunk_server=local /services/deployment/server/clients
|table hostname ip instanceName utsname package splunkVersion lastPhoneHomeTime

▶  The result also shows 21 deployment clients. 

Step 3: 

To check how many clients actually phone home:

On Deployment Server, from $SPLUNK_HOME/var/log/client_events, check latest phone home logs:

cat phonehomes_xxxxx.log | jq -r '.data.connectionId' | sort | uniq -c | wc -l

▶  The result shows 60 different connection IP phone home. 

Step 4: 

Ensure following settings required for 9.2.x version are already in outputs.conf on Deployment Server:

[indexAndForward]
index = true
selectiveIndexing = true   

index=_internal source=*splunkd* component IN ("DC:DeploymentClient", "HttpPubSubConnection") log_level IN (WARN, ERROR)

index=_dsclient OR index=_dsphonehome OR index=_dsappevent

Resolution

Resolution: 

Step 1: 

Locally index Deployment Server related data, without forwarding to indexer tier:

To do that, update the  $SPLUNK_HOME/etc/system/local/outputs.conf to the following to stop forwarding Deployment Server related data (removed _dsclient, _dsphonehome, _dsappevent)

[tcpout]
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker)

Step 2: 

Distributed Monitoring Console:  Ensure the Distributed Monitoring Console can search the deployment server indexes, by adding the Indexer role.  This can be configured using the web interface or command line.

Web Interface Configuration:

• Navigate to the Distributed Monitoring Console Web interface > Settings > Monitoring Console > Setting (from Monitoring Console tab) > General Settings.
• Select Edit > Edit Server roles next to the Monitoring Console instance.
• Place a check mark next to the Indexer role and click Save.

• Click on Apply Changes in the top right to save the changes.

Command Line Configuration:

• On $SPLUNK_HOME/etc/system/local/distsearch.conf,under the stanza that has default=true add localhost:localhost to the list of servers to ensure that search will look into itself to search Deployment Server data locally:

[distributedSearch:dmc_group_indexer]
default = true
servers =localhost:localhost, Indexer1_IP:8089, Indexer2_IP:8089

Related Links

Upgrade pre-9.2 deployment servers