Troubleshooting Splunk Universal Forwarder Data Ingestion Issues

Available Languages

Updated:October 1, 2026

Document ID:SPLK053987

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Cloud Splunk Universal Forwarder

Description

Data from Universal Forwarders is missing, delayed, or partially indexed. The forwarder may be running, but logs are not visible in Splunk.

Resolution

Confirm Forwarder Service Status

Confirms whether Splunk is running locally. If it is not running, the issue is on the forwarder itself.

Linux
$SPLUNK_HOME/bin/splunk statusWindows
"%SPLUNK_HOME%\bin\splunk.exe" status

Also, on Linux, you can check if there are any Splunk services running with the following command, you should see several services running such as splunkd

ps -ef | grep splunk

USER	PID	PPID	C	STIME	TTY	TIME	CMD
splunk	3793	3162	0	2024	?	00:34:56	compsup daemon
splunk	3945	3793	0	2024	?	00:34:59	/opt/splunk/var/run/supervisor/pkg-run/pkg-identity3181374198/identity
splunk	4430	3162	1	2024	?	23:13:35	/opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore
splunk	4803	3162	0	2024	?	02:26:49	/opt/splunk/bin/python3.9 -O /opt/splunk/lib/python3.9/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
splunk	241578	3162	3	18:48	?	00:02:51	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	241584	241578	0	18:48	?	00:00:00	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	245903	3162	3	18:53	?	00:03:02	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	245907	245903	0	18:53	?	00:00:00	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	265162	3162	3	19:15	?	00:01:58	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	265163	265162	0	19:15	?	00:00:00	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	297970	3162	3	19:53	?	00:00:43	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	297971	297970	0	19:53	?	00:00:00	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	298724	3162	1	19:54	?	00:00:24	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	298725	298724	0	19:54	?	00:00:00	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	306872	3162	3	20:03	?	00:00:31	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	306873	306872	0	20:03	?	00:00:00	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	308645	3162	3	20:05	?	00:00:24	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	308646	308645	0	20:05	?	00:00:00	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	312760	3162	5	20:10	?	00:00:20	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	312761	312760	0	20:10	?	00:00:00	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	313017	3162	3	20:10	?	00:00:11	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	313018	313017	0	20:10	?	00:00:00	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	317128	3162	2	20:14	?	00:00:02	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	317130	317128	0	20:14	?	00:00:00	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	317543	3162	8	20:15	?	00:00:06	[splunkd pid=3161] [search-launcher] [process-runner]
splunk	317545	317543	0	20:15	?	00:00:00	[splunkd pid=3161] [search-launcher] [process-runner]

Validate Network Connectivity

Confirms the forwarder can reach the destination (indexer or Splunk Cloud). If this fails, the issue is network-related.

Linux
telnet <host> 9997
nc -vz <host> 9997Windows
Test-NetConnection -ComputerName <host> -Port 9997

BTOOL

Btool shows the final merged configuration across all layers (default, app, local).

Use | grep local

Filtering with local helps you confirm what is explicitly configured on that forwarder, instead of inherited defaults.

Outputs Configuration

This hows active forwarding configuration defined locally. In this case it is necessary to check paths ending in /local/outputs.conf, correct server = host:9997, and correct tcpout group

Linux:
splunk btool outputs list --debug | grep -i localWindows:
splunk btool outputs list --debug | findstr /i local

Outputs.conf

Configure forwarding with outputs.conf

Inputs Configuration

This confirms what data is being monitored locally, it is required to look for entries from /local/inputs.conf, correct file paths, and Inputs enabled.

Linux:
splunk btool inputs list --debug | grep -i localWindows:
splunk btool inputs list --debug | findstr /i local

inputs.conf

Props Configuration

This shows local parsing rules.

Linux:
splunk btool props list --debug | grep -i localWindows:
splunk btool props list --debug | findstr /i local

Transforms Configuration

It shows routing rules. In this case check the rules sending data to nullQueue

Linux:
splunk btool transforms list --debug | grep -i localWindows:
splunk btool transforms list --debug | findstr /i local

Universal Forwarder with Structured Data and INDEXED_EXTRACTIONS - Props/Transforms Forwarding not working

Full Configuration Check

It identifies configuration conflicts.

splunk btool check

See the links below for additional details:

Command line tools for use with Support

Use btool to troubleshoot configurations

Log Analysis

splunkd.log

It shows runtime errors such as connectivity, SSL, or forwarding failures.

Location:

$SPLUNK_HOME/var/log/splunk/splunkd.log

Linux:
grep -i error splunkd.log
grep -i tcp splunkd.logWindows:
findstr /i error splunkd.log

metrics.log

It shows the pipeline health.

Linux:
grep -i blocked metrics.log
grep -i queue metrics.log

Internal Logs via Search

index=_internal sourcetype=splunkd host=<forwarder_host>

Reference:
What Splunk software logs about itself

Validate File Monitoring and Permissions

Confirms Splunk can read the file.

Linux:
ls -l <file_path>

Monitoring Console (MC) – Splunk Enterprise

Configure the Monitoring Console to monitor your forwarders:

Reference:
Configure forwarder monitoring for the Monitoring Console

---

Forwarder Connections

Whether forwarders are connected.

Indexing Performance

Ingestion rate and latency.

Indexing QueuesPipeline health and bottlenecks.

Resource Usage

System health (CPU, memory, disk).

Useful Searches

Forwarder connections:

index=_internal sourcetype=splunkd component=TcpInputProc

Throughput:

index=_internal source=*metrics.log group=tcpin_connections

Blocked queues:

index=_internal source=*metrics.log blocked=true

Generate Diagnostic File

Generate a diag and upload it to your support case, this captures logs and configs for deeper troubleshooting.

splunk diag

Generate a diagnostic file

Troubleshooting Splunk Universal Forwarder Data Ingestion Issues

Available Languages

Bias-Free Language

Applicability

Description

Resolution

Related Links

Revision History

Was this Document Helpful?

Contact Cisco

This Document Applies to These Products