Introduction
This document describes how to repair the MongoDB on the Secure Network Analytics (formerly Stealthwatch) Manager appliance after an unclean shutdown.
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command."
Review Log Data
Use the less command to review the mongodb.log file.
732smc:~# less /lancope/var/mongodb/log/mongodb.log
2021-06-21T14:54:43.029+0000 I CONTROL ***** SERVER RESTARTED *****
2021-06-21T14:54:43.033+0000 I CONTROL [initandlisten] MongoDB starting : pid=87057 port=27017 dbpath=/lancope/var/database/dbs/mdb/ 64-bit host=ussecrapstwsmc1
2021-06-21T14:54:43.033+0000 I CONTROL [initandlisten] db version v3.0.15
2021-06-21T14:54:43.033+0000 I CONTROL [initandlisten] git version: b8ff507269c382bc100fc52f75f48d54cd42ec3b
2021-06-21T14:54:43.033+0000 I CONTROL [initandlisten] build info: Linux 3555b2234f08 4.9.0-2-amd64 #1 SMP Debian 4.9.13-1 (2017-02-27) x86_64 BOOST_LIB_VERSION=1_49
2021-06-21T14:54:43.033+0000 I CONTROL [initandlisten] allocator: tcmalloc
2021-06-21T14:54:43.033+0000 I CONTROL [initandlisten] options: { config: "/etc/mongodb/mongodb.conf", net: { port: 27017 }, processManagement: { fork: true }, storage: { dbPath: "/lancope/var/database/dbs/mdb/" }, systemLog: { destination: "file", logAppend: true, path: "/lancope/var/mongodb/log/mongodb.log" } }
2021-06-21T14:54:43.050+0000 W - [initandlisten] Detected unclean shutdown - /lancope/var/database/dbs/mdb/mongod.lock is not empty.
2021-06-21T14:54:43.063+0000 I STORAGE [initandlisten] **************
old lock file: /lancope/var/database/dbs/mdb/mongod.lock. probably means unclean shutdown,
but there are no journal files to recover.
this is likely human error or filesystem corruption.
please make sure that your journal directory is mounted.
found 2 dbs.
see: http://dochub.mongodb.org/core/repair for more information
*************
2021-06-21T14:54:43.063+0000 I STORAGE [initandlisten] exception in initAndListen: 12596 old lock file, terminating
2021-06-21T14:54:43.063+0000 I CONTROL [initandlisten] dbexit: rc: 100
Repair the Database if it does not Start
Step 1. Check Mongo Status
To check the status of lc-mongodb.service, issue the systemctl is-active lc-mongodb command.
If Mongo is in an active state, your results would resemble:
732smc:/# systemctl is-active lc-mongodb
active
732smc:/#
If Mongo is not in an active state, your results would resemble:
732smc:/# systemctl is-active lc-mongodb
inactive
732smc:/#
Step 2. Stop the Mongo Service
If the lc-mongodb service was found to be in an active state, stop the service with the /bin/systemctl stop lc-mongodb.service command.
732smc:/# /bin/systemctl stop lc-mongodb.service
732smc:/# /bin/systemctl status lc-mongodb.service | grep Active
Active: inactive (dead) since Thu 2022-04-07 12:33:49 UTC; 1s ago7
Wait a few moments and ensure that mongo remains in a stopped state. Use the /bin/systemctl status lc-mongodb.service command as needed to ensure that the service is in a inactive state.
Step 3. Gather Process ID (PID)
Check to see if the lock file still contains a PID. Issue the cat /lancope/var/database/dbs/mdb/mongod.lock command.
This output shows that the lock file contains the PID of the mongo service. This file must only contain data if the service is in an active state.
Note: Take note of the PID if one is returned, as it is used in Step 4
732smc:/# cat /lancope/var/database/dbs/mdb/mongod.lock
14259
732smc:/#
This output shows that the lock file does not contain a PID. This file must be empty if the process in not in an active state. If there is no PID continue to Step 7.
732smc:/# cat /lancope/var/database/dbs/mdb/mongod.lock
732smc:/#
Step 4. Check PID Status
If mongod.lock file checked in Step 3 contained a PID, run the ps fuax | grep [1]4259 command (change the [1]4259 with your PID from step 3) to check the existence of the PID and then subsequently kill that PID if it is found.
Note: The bracket expression is not required but results in the exclusion of the "grep" command in the output.
732smc:/# ps faux | grep [1]4259
mongodb 14259 0.3 0.4 516180 71520 ? Sl 12:38 0:03 /lancope/mongodb/bin/mongod --fork --config /etc/mongodb/mongodb.conf
732smc:/# kill -9 14259
732smc:/#
Step 5. Clear lock file contents
Clear the content of the lock file with the > /lancope/var/database/dbs/mdb/mongod.lock command. Verify the file is now empty with the cat /lancope/var/database/dbs/mdb/mongo.lock command.
732smc:/# > /lancope/var/database/dbs/mdb/mongod.lock
732smc:/# cat /lancope/var/database/dbs/mdb/mongod.lock
732smc:/#
Step 6. Attempt to start the MongoDB
Attempt to start the lc-mongodb service with the /bin/systemctl start lc-mongodb.service command. Once your prompt is returned, check the status of the process with the /bin/systemctl status lc-mongodb.service | grep Active command.
732smc:/# /bin/systemctl start lc-mongodb.service
732smc:/# /bin/systemctl status lc-mongodb.service | grep Active
Active: active (running) since Thu 2022-04-07 12:38:37 UTC; 27s ago
732smc:/#
If the process is in an active state, check again in a few minutes to ensure that it remains in an active state. You do not need to repair the database if it remains in a functioning state. If the process fails to remain active, proceed to step 7 and initiate a repair process.
Step 7. Initiate repair
Issue the sudo -u mongodb /lancope/mongodb/bin/mongod --dbpath /lancope/var/database/dbs/mdb --repair command
732smc:/# sudo -u mongodb /lancope/mongodb/bin/mongod --dbpath /lancope/var/database/dbs/mdb/ --repair
732smc:/#
Step 8. Attempt to start the repaired MongoDB
Run the the /bin/systemctl start lc-mongodb.service command to start the service. The process must remain in an active state and can be checked with the /bin/systemctl status lc-mongodb.service command.
732smc:/# /bin/systemctl start lc-mongodb.service
732smc:/# /bin/systemctl status lc-mongodb.service | grep Active
Active: active (running) since Thu 2022-04-07 12:38:37 UTC; 27s ago
Feedback