Segmentation Fault errors while upgrading Splunk Enterprise.

Applicability

Description

Unable to upgrade Splunk HF from 9.1.2 to 9.4.X version on Amazon Linux AMI OS ending with the below error while upgrading.

ERROR: 

sh: line 1: [PID] Segmentation fault splunk migrate renew-certs 2>&1
ERROR while running renew-certs migration.

Investigation

1) This error message indicates either a permission issue or a certificate issue. 

2) While investigation, it was observed the error regarding server.pem

ERROR:

ERROR ServerConfig [0 MainThread] - No '/opt/splunk/etc/auth/server.pem' certificate found.  Splunkd communication will not work without this!

3) But, in this scenario even after validating the permissions of the certificate and creating a new server.pem certificate did not resolve the issue.

4) Upon further investigation, noticed the below error in the splunkd.log - 

ERROR CgroupsLocation [0 MainThread] - Cannot set location using location=auto, error=Error code="Not supported", function=detectSubPath, error="Memory, cpu, and V2 controllers are not present in cgroup file"

5) This error log indicates that the code is "Not Supported".

6) Furthermore, validated the OS which is being used with the command "uname -a"
 

7) The OS being used is Amazon Linux AMI x86_64. 

Resolution

The OS being used was not supported for Splunk 9.4.x version and because of this reason the upgradation was interrupted.

As per the Splunk document, Amazon Linux 2 x86 (64-bit) is marked as D which indicates Splunk supports this platform and architecture but might remove support in a future release. 

The error while upgrading was indicating a certificate issue, which is not the actual issue. As a first step the kernel version should be checked and verify support for the upgraded/new version of Splunk. 

Related Links

System Requirements