Restart mongod service after digital ocean hard reboot due to permission issue / wired tiger files corruption

March 29, 2018 953 views
MongoDB DigitalOcean Ubuntu Ubuntu 16.04

So after Digital Ocean gave absolutely no warning and hard rebooted our servers, everything was fine except, big surprise, the MongoDB servers! I have finally fixed the configuration files, only to realize they were fine all along, and have concluded that there is some kind of a permission issue. I am attempting to restart the mongod service and pull from the log files. Also sorry, but I updated the question twice with more information and Digital Ocean then rejected it as spam so I am attempting to post again.

When I run the command
chown -R mongodb:mongodb /mongo-metadata/ ,

Afterwards we do a
"sudo service mongod restart" and the status we get is:

root@mongoprimary:/mongo-metadata# sudo service mongod status
● mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2018-03-29 16:50:28 UTC; 5s ago
     Docs: https://docs.mongodb.org/manual
  Process: 14317 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=0/SUCCESS)
 Main PID: 14317 (code=exited, status=0/SUCCESS)

Mar 29 16:50:28 mongoprimary.******.com systemd[1]: Started High-performance, schema-free document-oriented database.
Mar 29 16:50:28 mongoprimary.*****.com mongod[14317]: about to fork child process, waiting until server is ready for connectio
Mar 29 16:50:28 mongoprimary.*****.com mongod[14317]: forked process: 14320

the output of my log file to better understand the error:

root@mongoprimary:/mongo-metadata# tail /var/log/mongodb/mongod.log 
2018-03-29T16:50:28.633+0000 I REPL     [replExecDBWorker-2] transition to STARTUP2
2018-03-29T16:50:28.633+0000 I REPL     [replExecDBWorker-2] Starting replication storage threads
2018-03-29T16:50:28.633+0000 I REPL     [signalProcessingThread] Stopping replication storage threads
2018-03-29T16:50:28.633+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongosecondary1.******.com:27017
2018-03-29T16:50:28.633+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongosecondary2.******.com:27017
2018-03-29T16:50:28.634+0000 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2018-03-29T16:50:28.634+0000 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2018-03-29T16:50:28.688+0000 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2018-03-29T16:50:28.688+0000 I CONTROL  [signalProcessingThread] now exiting
2018-03-29T16:50:28.688+0000 I CONTROL  [signalProcessingThread] shutting down with code:0

After I run a db repair command (both with and without sudo), command I run:

"mongod --repair --dbpath /mongo-metadata --storageEngine wiredTiger"

Last line of the output repair function:

2018-03-29T16:47:41.513+0000 I INDEX    [initandlisten] build index on: meteor.users properties: { v: 2, key: { vendorId: 1 }, name: "vendorId_1", ns: "meteor.users" }
2018-03-29T16:47:41.513+0000 I INDEX    [initandlisten]      building index using bulk method; build may temporarily use up to 38 megabytes of RAM
2018-03-29T16:47:41.607+0000 I STORAGE  [initandlisten] finished checking dbs
2018-03-29T16:47:41.607+0000 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2018-03-29T16:47:41.607+0000 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-03-29T16:47:41.607+0000 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2018-03-29T16:47:41.607+0000 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2018-03-29T16:47:41.614+0000 I STORAGE  [initandlisten] shutdown: removing fs lock...
2018-03-29T16:47:41.614+0000 I CONTROL  [initandlisten] now exiting
2018-03-29T16:47:41.614+0000 I CONTROL  [initandlisten] shutting down with code:0

Then when I attempt to restart:

root@mongoprimary:/mongo-metadata# sudo service mongod restart
root@mongoprimary:/mongo-metadata# tail /var/log/mongodb/mongod.log 
2018-03-29T16:48:03.770+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-03-29T16:48:03.770+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=488M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-03-29T16:48:03.777+0000 E STORAGE  [initandlisten] WiredTiger error (13) [1522342083:777174][14299:0x7f3e0cf7ad00], file:WiredTiger.wt, connection: /mongo-metadata/WiredTiger.turtle: handle-open: open: Permission denied
2018-03-29T16:48:03.777+0000 I -        [initandlisten] Assertion: 28595:13: Permission denied src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 269
2018-03-29T16:48:03.777+0000 I STORAGE  [initandlisten] exception in initAndListen: 28595 13: Permission denied, terminating
2018-03-29T16:48:03.778+0000 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2018-03-29T16:48:03.778+0000 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-03-29T16:48:03.778+0000 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2018-03-29T16:48:03.778+0000 I CONTROL  [initandlisten] now exiting
2018-03-29T16:48:03.778+0000 I CONTROL  [initandlisten] shutting down with code:100

Please help me get the mongoDB servers back up and running. I have contacted Digital Ocean 3 times and still have heard no response, it is so frustrating that I was literally in the hospital when they notified me of this hard reboot and do absolutely nothing to help fix the problem they created :( I won't dispute my configuration is less than ideal, if not downright shitty, but some communication on their part would go along way, even if it consisted of you're a dumbass. Instead of lets hard reboot your server and ignore you since we don't give a shit that we crashed your apps.

Will definitely get anyone who helps me fix this problem a beer / whiskey / joint of their choosing :) Thank you!

Be the first one to answer this question.