Contents

Introduction

Prerequisites

Requirements

Components Used

Review Log Data

Repair the Database if it does not Start

Introduction

This document describes how to repair the MongoDB on the Secure Network Analytics (formerly Stealthwatch) Manager appliance after an unclean shutdown.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command."

Review Log Data

Use the less command to review the mongodb.log file.

732smc:~# less /lancope/var/mongodb/log/mongodb.log

2021-06-21T14:54:43.029+0000 I CONTROL ***** SERVER RESTARTED *****

2021-06-21T14:54:43.033+0000 I CONTROL [initandlisten] MongoDB starting : pid=87057 port=27017 dbpath=/lancope/var/database/dbs/mdb/ 64-bit host=ussecrapstwsmc1

2021-06-21T14:54:43.033+0000 I CONTROL [initandlisten] db version v3.0.15

2021-06-21T14:54:43.033+0000 I CONTROL [initandlisten] git version: b8ff507269c382bc100fc52f75f48d54cd42ec3b

2021-06-21T14:54:43.033+0000 I CONTROL [initandlisten] build info: Linux 3555b2234f08 4.9.0-2-amd64 #1 SMP Debian 4.9.13-1 (2017-02-27) x86_64 BOOST_LIB_VERSION=1_49

2021-06-21T14:54:43.033+0000 I CONTROL [initandlisten] allocator: tcmalloc

2021-06-21T14:54:43.033+0000 I CONTROL [initandlisten] options: { config: "/etc/mongodb/mongodb.conf", net: { port: 27017 }, processManagement: { fork: true }, storage: { dbPath: "/lancope/var/database/dbs/mdb/" }, systemLog: { destination: "file", logAppend: true, path: "/lancope/var/mongodb/log/mongodb.log" } }

2021-06-21T14:54:43.050+0000 W - [initandlisten] Detected unclean shutdown - /lancope/var/database/dbs/mdb/mongod.lock is not empty.

2021-06-21T14:54:43.063+0000 I STORAGE [initandlisten] **************

old lock file: /lancope/var/database/dbs/mdb/mongod.lock. probably means unclean shutdown,

but there are no journal files to recover.

this is likely human error or filesystem corruption.

please make sure that your journal directory is mounted.

found 2 dbs.

see: http://dochub.mongodb.org/core/repair for more information

*************

2021-06-21T14:54:43.063+0000 I STORAGE [initandlisten] exception in initAndListen: 12596 old lock file, terminating

2021-06-21T14:54:43.063+0000 I CONTROL [initandlisten] dbexit: rc: 100

Repair the Database if it does not Start

Step 1. Check Mongo Status

To check the status of lc-mongodb.service, issue the systemctl is-active lc-mongodb command.

If Mongo is in an active state, your results would resemble:

732smc:/# systemctl is-active lc-mongodb

active

732smc:/#

If Mongo is not in an active state, your results would resemble:

732smc:/# systemctl is-active lc-mongodb

inactive

732smc:/#

Step 2. Stop the Mongo Service

If the lc-mongodb service was found to be in an active state, stop the service with the /bin/systemctl stop lc-mongodb.service command.

732smc:/# /bin/systemctl stop lc-mongodb.service

732smc:/# /bin/systemctl status lc-mongodb.service | grep Active

Active: inactive (dead) since Thu 2022-04-07 12:33:49 UTC; 1s ago7

Wait a few moments and ensure that mongo remains in a stopped state. Use the /bin/systemctl status lc-mongodb.service command as needed to ensure that the service is in a inactive state.

Step 3. Gather Process ID (PID)

Check to see if the lock file still contains a PID. Issue the cat /lancope/var/database/dbs/mdb/mongod.lock command.

This output shows that the lock file contains the PID of the mongo service. This file must only contain data if the service is in an active state.

Note: Take note of the PID if one is returned, as it is used in Step 4

732smc:/# cat /lancope/var/database/dbs/mdb/mongod.lock

14259

732smc:/#

This output shows that the lock file does not contain a PID. This file must be empty if the process in not in an active state. If there is no PID continue to Step 7.

732smc:/# cat /lancope/var/database/dbs/mdb/mongod.lock

732smc:/#

Step 4. Check PID Status

If mongod.lock file checked in Step 3 contained a PID, run the ps fuax | grep [1]4259 command (change the [1]4259 with your PID from step 3) to check the existence of the PID and then subsequently kill that PID if it is found.

Note: The bracket expression is not required but results in the exclusion of the "grep" command in the output.

732smc:/# ps faux | grep [1]4259

mongodb 14259 0.3 0.4 516180 71520 ? Sl 12:38 0:03 /lancope/mongodb/bin/mongod --fork --config /etc/mongodb/mongodb.conf

732smc:/# kill -9 14259

732smc:/#

Step 5. Clear lock file contents

Clear the content of the lock file with the > /lancope/var/database/dbs/mdb/mongod.lock command. Verify the file is now empty with the cat /lancope/var/database/dbs/mdb/mongo.lock command.

732smc:/# > /lancope/var/database/dbs/mdb/mongod.lock

732smc:/# cat /lancope/var/database/dbs/mdb/mongod.lock

732smc:/#

Step 6. Attempt to start the MongoDB

Attempt to start the lc-mongodb service with the /bin/systemctl start lc-mongodb.service command. Once your prompt is returned, check the status of the process with the /bin/systemctl status lc-mongodb.service | grep Active command.

732smc:/# /bin/systemctl start lc-mongodb.service

732smc:/# /bin/systemctl status lc-mongodb.service | grep Active

Active: active (running) since Thu 2022-04-07 12:38:37 UTC; 27s ago

732smc:/#

If the process is in an active state, check again in a few minutes to ensure that it remains in an active state. You do not need to repair the database if it remains in a functioning state.  If the process fails to remain active, proceed to step 7 and initiate a repair process.

Step 7. Initiate repair

Issue the sudo -u mongodb /lancope/mongodb/bin/mongod --dbpath /lancope/var/database/dbs/mdb --repair command

732smc:/# sudo -u mongodb /lancope/mongodb/bin/mongod --dbpath /lancope/var/database/dbs/mdb/ --repair

732smc:/#

Step 8. Attempt to start the repaired MongoDB

Run the the /bin/systemctl start lc-mongodb.service command to start the service.  The process must remain in an active state and can be checked with the /bin/systemctl status lc-mongodb.service command.

732smc:/# /bin/systemctl start lc-mongodb.service

732smc:/# /bin/systemctl status lc-mongodb.service | grep Active

Active: active (running) since Thu 2022-04-07 12:38:37 UTC; 27s ago