Universal Forwarder in AIX crashing after upgrading to 9.1.1.

Applicability

Description

1. Universal forwarder crashing continiously, after the upgrade from the 8.2.x to 9.1.X, as it is not able to connect to the recevier point (Ex: Indexer, Heavy forwarder).

2. During the process of Universal Forwarder installation in the AIX system, splunkd is up for a few seconds and then goes       down.

3.  With the crashing thread of TcpOutEloop.

[build 64e843ea36b1] 1970-01-01 01:00:04
Received fatal signal 11 (Segmentation fault) on PID 23855400.
 Crashing thread: TcpOutEloop

Investigation

1. Confirmed the Aix version, as AIX supported version are 7.1 or 7.2

2. Checked splunkd.log in the mentioned path (/opt/splunk/var/log/splunk)

  In the Splunkd.log could see the events which were showing the timestamp as "1970".

 

01-01-1970 01:00:06.167 +0100 ERROR TcpOutputFd [66191707 TcpOutEloop] - Connection to host=10.X.x.x:9997 failed
01-01-1970 01:00:06.167 +0100 WARN  TcpOutputFd [66191707 TcpOutEloop] - Connect to 10.X.x.x:9997 failed. Connection refused
01-01-1970 01:00:06.167 +0100 WARN  AutoLoadBalancedConnectionStrategy [66191707 TcpOutEloop] - Applying quarantine to ip=10.X.x.x port=9997 connid=0 _numberOfFailures=2
01-01-1970 01:00:06.167 +0100 INFO  AutoLoadBalancedConnectionStrategy [66191707 TcpOutEloop] - Connection to 10.X.x.x:9997 closed. Connection closed by server.
01-01-1970 01:00:06.167 +0100 INFO  HttpPubSubConnection [66650435 HttpClientPollingThread_SP-IT-UF-AIX-abc] - Running phone uri=/services/broker/phonehome/connection_10.X.x.x_8089_ahc 
01-01-1970 01:00:06.167 +0100 INFO  HttpPubSubConnection [66650435 HttpClientPollingThread_SP-IT-UF-AIX-abc] - Running phone uri=/services/broker/phonehome/connection_10.X.x.x_8089_xyz
01-01-1970 01:00:06.167 +0100 INFO  HttpPubSubConnection [66650435 HttpClientPollingThread_SP-IT-UF-AIX-abc] - Running phone uri=/services/broker/phonehome/connection_10.X.x.x_8089_abc
01-01-1970 01:00:06.167 +0100 INFO  HttpPubSubConnection [66650435 HttpClientPollingThread_SP-IT-UF-AIX-abc] - Running phone uri=/services/broker/phonehome/connection_10.X.x.x_8089_abc
01-01-1970 01:00:06.167 +0100 INFO  HttpPubSubConnection [66650435 HttpClientPollingThread_SP-IT-UF-AIX-abc] - Running phone uri=/services/broker/phonehome/connection_10.X.x.x_8089_abc
01-01-1970 01:00:06.167 +0100 INFO  HttpPubSubConnection [66650435 HttpClientPollingThread_SP-IT-UF-AIX-abc] - Running phone uri=/services/broker/phonehome/connection_10.X.x.x_8089_abc
[1:00 PM] 01-01-1970 01:00:06.167 +0100 INFO  AutoLoadBalancedConnectionStrategy [66191707 TcpOutEloop] - Connected to idx=10.x.x.x:9997:0, pset=0, reuse=0. autoBatch=1
01-01-1970 01:00:06.148 +0100 ERROR TcpOutputFd [96207137 TcpOutEloop] - Connection to host=10.x.x.x:9997 failed
01-01-1970 01:00:06.148 +0100 WARN  TcpOutputFd [96207137 TcpOutEloop] - Connect to 10.x.x.x:9997 failed. Connection refused
01-01-1970 01:00:06.148 +0100 INFO  AutoLoadBalancedConnectionStrategy [96207137 TcpOutEloop] - Connection to 10.x.x.x:9997 closed. Connection closed by server.
01-01-1970 01:00:06.148 +0100 INFO  HttpPubSubConnection [64225543 HttpClientPollingThread_SP-IT-UF-ABC] - Running phone uri=/services/broker/phonehome/connection_10.x.0.0_8089_temshebrc.inprod.abc
[1:01 PM] 01-01-1970 01:00:04.560 +0100 INFO  AutoLoadBalancedConnectionStrategy [100532693 TcpOutEloop] - Connected to idx=10.223.0.54:9995:0, pset=0, reuse=0. autoBatch=1
01-01-1970 01:00:04.560 +0100 INFO  AutoLoadBalancedConnectionStrategy [100532693 TcpOutEloop] - Connected to idx=10.0.0.0:9997:1, pset=0, reuse=0. autoBatch=1

3.  Checked the crash.log recorded in the UF's $SPLUNK_HOME/var/log/splunk/ you can find the below details:

[build 64e843ea36b1] 1970-01-01 01:00:04
Received fatal signal 11 (Segmentation fault) on PID 23855400.
 Crashing thread: TcpOutEloop

4. Collect the crash core dump by AIX.

a. Below document can be shared with the customer to generate  the crash core dump by AIX.

b. Note :  

Splunk is not responsible for the dump generation, because it’s all on their AIX system admin. If their Splunk admin has no idea whether or how to generate it, they should contact their sys admin to evaluate the solution, in order to comply with their company’s (security, operation, privacy, etc.) policies. 

Resolution

1.  The Build as given by the dev, which needs to be installed, this can also help to stop the crash.

2. If not the Fix is included in 9.1.2 & 9.2.0 versions.

3. And also you can downgrade the Universal Forwarder to 9.0.x to workaround it.