Splunk Universal Forwarder stops Forwarding events when installed on AIX - TcpOutEloop Segmentation Fault

Available Languages

Updated:September 29, 2026
Document ID:SPLK048148

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Universal Forwarder AIX 7.1 & 7.2

Description

After installing Universal Forwarder version 9.1.0 or 9.1.1 on AIX 7.1 or 7.2 it is found that forwarding intermittently stops causing logs  from this Forwarder to be missing when searching the indexes.

When the issue occurs a crash log is generated in $SPLUNK_HOME/var/log/splunk/crash-<date/time>.log with crashing thread TcpOutEloop

Example Crash log header:

[build 64e843ea36b1] 1970-01-01 08:00:04
Received fatal signal 11 (Segmentation fault) on PID 17891640.
 Crashing thread: TcpOutEloop
...

Investigation

When this issue occurs it also causes Splunk to use the incorrect date/time and the splunkd.log will have a timestamp starting at epoch.

The issue also typically occurs after forwarding to the next Splunk instance fails or timeouts.

For Example, when checking $SPLUNK_HOME/var/log/splunk/splunkd.log it is found that the Cooked connection to ip=X.X.X.X:9997 timed out and the proceeding logs have the incorrect timestamp of 01-01-1970 08:00:06.304 +0700

01-06-2025 15:51:17.069 +0700 INFO  TailReader [45351307 tailreader0] - Batch input finished reading file='/opt/splunkforwarder/var/spool/splunk/tracker.log'
01-06-2025 15:51:34.563 +0700 WARN  AutoLoadBalancedConnectionStrategy [45810075 TcpOutEloop] - Cooked connection to ip=X.X.X.X:9997 timed out
01-01-1970 08:00:06.304 +0700 INFO  HttpPubSubConnection [77201681 HttpClientPollingThread_X] - Running phone uri=/services/broker/phonehome/connection_X.X.X.X_8089_X.X.X.X_UNIX-UF
01-01-1970 08:00:06.304 +0700 INFO  HttpPubSubConnection [77201681 HttpClientPollingThread_X] - Running phone uri=/services/broker/phonehome/connection_X.X.X.X_8089_X.X.X.X_UNIX-UF

Restarting Splunk temporarily resolves the issue with forwarding resuming but will likely re-occur when or if the connection again fails.


Resolution

This behavior has been identified as a known issue.

A permanent fix has been included in the following Splunk forwarder versions 9.2.4, 9.3.2, 9.4.0.

It is recommended to upgrade the Splunk Universal Forwarder to one of the fixed versions listed above to permanently resolve this issue.

 


Splunk Docs: SPL-245807 

Revision History

Revision Publish Date Comments
1.0
29-Sep-2026
Initial Release

Was this Document Helpful?

FeedbackFeedback

Contact Cisco

This Document Applies to These Products

Related Articles