Splunkd won't start with message "failed with code 1"

Applicability

Description

After the OS upgrade from 7.5 to 8.10 (Redhat) the Splunk was started, while Starting the below errors were triggered.

"coldPath='/var/lib/splunk/<INDEX_NAME>/colddb' of index=<INDEX_NAME> on unusable filesystem.Validating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue"Checking prerequisites...
Checking http port [X]: open
Checking mgmt port [X]: open
Checking appserver port [X.X.X.X:X]: open
Checking kvstore port [X]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
coldPath='/var/lib/splunk/<INDEX_NAME>/colddb' of index=<INDEX_NAME> on unusable filesystem.
Validating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue

Investigation

1.  Check the permission of the Splunk, where it is being installed if it owns the appropriate permission.

2. According to the " FS Locking " document - 
It states the exact error which was seen and it is advised to login as a root and set :

OPTIMISTIC_ABOUT_FILE_LOCKING=1 in Splunk-launch.conf and restart Splunk.

3.  Check if the above parameter is set to 1,  here even after having this configuration set as 1, error was still being triggered.

4. Try changing permissions once again for the Splunk directory using "chown" command and check if the error still persist, if so continue with further steps.

5. On further investigation, noticed that the cold and frozen directories are totally full as per the below logs:

ERROR BucketMover  - Failed to clean new-style bucket='/var/lib/splunk/<INDEX_NAME>/colddb/db_XXX' before freezingERROR ScopedFileMutex - failed to create file /var/lib/splunk/<INDEX_NAME>/colddb/db_XXX/bucket_info.csv: No such file or directoryWARN IndexerService  - Removing bucket artifact dir="/var/lib/splunk/<INDEX_NAME>/colddb/db_<index>.rbsentinel"WARN DiskMon - MinFreeSpace=20000. The diskspace remaining=0 is less than 1 x minFreeSpace on /var/lib/splunk/<INDEX_NAME>/colddbWARN DatabaseDirectoryManager  - Writing the manifest in bucket path="/var/lib/splunk/<INDEX_NAME>/colddb/db_XXX" failed, but unless you see this frequently, this should not be of much concern: the bucket can move while you attempt to persist the frozen bit, and this is expected.

Resolution

If the above errors exist related to diskspace try to free up space from colddir and frozendir by removing old bucket files using 

rm -rf <bucket_name>

Post that try starting the Splunk, this should help to resolve the issue.

Since it was noticed that the frozen and cold directories were completely filled, removing old buckets from them and making some free space inside there directories resolved the issue.

Setting OPTIMISTIC_ABOUT_FILE_LOCKING=1 is not a best practice. Once the issue is resolved, this value should be switched back to OPTIMISTIC_ABOUT_FILE_LOCKING=0 in splunk-launch.conf, followed by a Splunk restart.

Related Links

File_Locking_Isuue