"DMC Alert - Missing Forwarders"

Applicability

Splunk Enterprise

Description

Splunk monitoring console is reporting DMC Alert - missing forwarders.
This alert can be triggered either due to the missing forwarders or the updates related to certain lookups.
DMC Alert - Missing Forwarders fetch missing forwarders from dmc_forwarder_assets lookup and that lookup file updates based on the frequency you set in Monitoring Console > Settings > Forwarder Monitoring Setup > Data Collection Interval and lookup file updates as per provided.

Investigation

Once this Alert is received, Check for certain attributions;

• Check whether all forwarders are available.
• If no forwarders are missing, then check the frequency at which this alert is set in Monitoring Console.

Check in Monitoring Console > Settings > Forwarder Monitoring Setup > Data Collection Interval and lookup file updates with below query :

(fwdType=* group=tcpin_connections guid=* index=_internal sourcetype=splunkd (connectionType=cooked OR connectionType=cookedSSL)) 
| stats values(fwdType) as forwarder_type, latest(version) as version, values(arch) as arch, values(os) as os, max(_time) as last_connected, sum(kb) as new_sum_kb, sparkline(avg(tcp_KBps), 1m) as new_avg_tcp_kbps_sparkline, avg(tcp_KBps) as new_avg_tcp_kbps, avg(tcp_eps) as new_avg_tcp_eps by guid, hostname 
| inputlookup dmc_forwarder_assets append=1 
| stats values(forwarder_type) as forwarder_type, max(version) as version, values(arch) as arch, values(os) as os, max(last_connected) as last_connected, values(new_sum_kb) as sum_kb, values(new_avg_tcp_kbps_sparkline) as avg_tcp_kbps_sparkline, values(new_avg_tcp_kbps) as avg_tcp_kbps, values(new_avg_tcp_eps) as avg_tcp_eps by guid, hostname 
| addinfo 
| eval status=if(((isnull(sum_kb) OR (sum_kb <= 0)) OR (last_connected < (info_max_time - 900))),"missing","active"), sum_kb=round(sum_kb,2), avg_tcp_kbps=round(avg_tcp_kbps,2), avg_tcp_eps=round(avg_tcp_eps,2) 
| fields guid, hostname, forwarder_type, version, arch, os, status, last_connected, sum_kb, avg_tcp_kbps_sparkline, avg_tcp_kbps, avg_tcp_eps

So here you need to look at | stats .... which is doing statistics on forwarder_type, version, os, arch, last_connected, ..... guid, hostname, if guid changed for UF then this query treats that UF as new Universal Forwarder and it keeps throwing alert for same UF with old guid as a missing forwarder.

If all forwarders are forwarding as expected, you should see this table update regularly, according to the Data Collection Interval.

 

Resolution

After verifying forwarders are sending data to Indexers and the lookup is not updating, forcing a rebuild of the forwarder asset table cleared the alert:
Monitoring Console > Settings > Forwarder Monitoring Setup > Click on Rebuild forwarder assets

 

Related Links

https://community.splunk.com/t5/Getting-Data-In/monitoring-console-triggered-alerts-missing-forwarders-but-they/m-p/458517
 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release