Deployment Client bypasses Deployment Server and shows on License Manager's/Indexer's GUI in Forwarder Management

Applicability

Description

There were changes introduced in the way Deployment Server communicates to its own clients and indexers: 

• Deployment Server (DS) pos-9.2.0 stores all incoming traffic from Deployment Clients (DC) inside the directory: $SPLUNK_HOME/var/log/client_events/ in 3 types of files:

clients_<DS_instance_guid>.log
phonehomes_<DS_instance_guid>.log
appevents_<DS_instance_guid>.log

• When logs about incoming connections are saved in this directory, they are indexed by the indexer and available in indexes: _dsclient, _dsphonehome and _dsappevent.
  • After that Forwarder Management UI makes internal rest calls to the indexers to query those indexes so he can know about its DCs

As a representation of a simplistic Splunk Environment in which this scenario might happen:

• have 3 Splunk instances
  • DC + DS + LicenseManager/Indexer (LM)

  • LM works simultaneously as an Indexer.

  • DS is configured to forward its logs to LM/indexer and doesn’t index them on its own.

Investigation

Deployment Server has been configured to forward all its logs to the License Manager/Indexer instance, without indexing them on its own.

Upon migration from pre 9.2.0 to pos-9.2.0 these indexes weren't located in the DS itself:

clients_<DS_instance_guid>.log
phonehomes_<DS_instance_guid>.log
appevents_<DS_instance_guid>.log

In that situation DC phones home to DS okay and apps and the whole communication in the backend works okay and as it should.

The issue it's only visible in the GUI in both, Forwarder Management on the DS and LicenseManager/Indexer (LM)

• Forwarder Management on the DS doesn't show DC - wrong behaviour
• Forwarder Management on the LicenseManager/Indexer (LM) shows DC - wrong behaviour

Because all data about communication with the DC has been forwarded to the indexer that is the reason why the Deployment Clients appeared on a not-DS instance.

Resolution

To fix:

• setup DS to locally index and not forward _dsclient, _dsphonehome and _dsappevent indexes.
• add/modify outputs.conf $SPLUNK_HOME/etc/apps/SplunkDeploymentServerConfig/local/outputs.conf

[indexAndForward]
index = true
selectiveIndexing = true[tcpout]
forwardedindex.filter.disable = false
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker)

• DCs will be visible in DS's GUI Forwarder Management

Little temporarily behaviour as a side note:

• DCs will momentarily/ temporarily be visible on LicenseManager/Indexer (LM)'s GUI Forwarder Management
  • Delete Record button is deprecated and unusable.
  • The only way to delete information about DC from on LicenseManager/Indexer (LM)'s GUI Forwarder Management is to reload whole DS/DS cluster, which will delete information about all phoned DCs.
    
    • Forwarder Management UI is automatically activated when DC phones home.

Related Links

Upgrade DS from pre-9.2