Question

Restart mongod service after digital ocean hard reboot due to permission issue / wired tiger files corruption

So after Digital Ocean gave absolutely no warning and hard rebooted our servers, everything was fine except, big surprise, the MongoDB servers! I have finally fixed the configuration files, only to realize they were fine all along, and have concluded that there is some kind of a permission issue. I am attempting to restart the mongod service and pull from the log files. Also sorry, but I updated the question twice with more information and Digital Ocean then rejected it as spam so I am attempting to post again.

When I run the command chown -R mongodb:mongodb /mongo-metadata/ ,

Afterwards we do a “sudo service mongod restart” and the status we get is:

root@mongoprimary:/mongo-metadata# sudo service mongod status
● mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2018-03-29 16:50:28 UTC; 5s ago
     Docs: https://docs.mongodb.org/manual
  Process: 14317 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=0/SUCCESS)
 Main PID: 14317 (code=exited, status=0/SUCCESS)

Mar 29 16:50:28 mongoprimary.******.com systemd[1]: Started High-performance, schema-free document-oriented database.
Mar 29 16:50:28 mongoprimary.*****.com mongod[14317]: about to fork child process, waiting until server is ready for connectio
Mar 29 16:50:28 mongoprimary.*****.com mongod[14317]: forked process: 14320

the output of my log file to better understand the error:

root@mongoprimary:/mongo-metadata# tail /var/log/mongodb/mongod.log 
2018-03-29T16:50:28.633+0000 I REPL     [replExecDBWorker-2] transition to STARTUP2
2018-03-29T16:50:28.633+0000 I REPL     [replExecDBWorker-2] Starting replication storage threads
2018-03-29T16:50:28.633+0000 I REPL     [signalProcessingThread] Stopping replication storage threads
2018-03-29T16:50:28.633+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongosecondary1.******.com:27017
2018-03-29T16:50:28.633+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongosecondary2.******.com:27017
2018-03-29T16:50:28.634+0000 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2018-03-29T16:50:28.634+0000 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2018-03-29T16:50:28.688+0000 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2018-03-29T16:50:28.688+0000 I CONTROL  [signalProcessingThread] now exiting
2018-03-29T16:50:28.688+0000 I CONTROL  [signalProcessingThread] shutting down with code:0

After I run a db repair command (both with and without sudo), command I run:
“mongod --repair --dbpath /mongo-metadata --storageEngine wiredTiger”

Last line of the output repair function:

2018-03-29T16:47:41.513+0000 I INDEX    [initandlisten] build index on: meteor.users properties: { v: 2, key: { vendorId: 1 }, name: "vendorId_1", ns: "meteor.users" }
2018-03-29T16:47:41.513+0000 I INDEX    [initandlisten] 	 building index using bulk method; build may temporarily use up to 38 megabytes of RAM
2018-03-29T16:47:41.607+0000 I STORAGE  [initandlisten] finished checking dbs
2018-03-29T16:47:41.607+0000 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2018-03-29T16:47:41.607+0000 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-03-29T16:47:41.607+0000 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2018-03-29T16:47:41.607+0000 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2018-03-29T16:47:41.614+0000 I STORAGE  [initandlisten] shutdown: removing fs lock...
2018-03-29T16:47:41.614+0000 I CONTROL  [initandlisten] now exiting
2018-03-29T16:47:41.614+0000 I CONTROL  [initandlisten] shutting down with code:0

Then when I attempt to restart:

root@mongoprimary:/mongo-metadata# sudo service mongod restart
root@mongoprimary:/mongo-metadata# tail /var/log/mongodb/mongod.log 
2018-03-29T16:48:03.770+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-03-29T16:48:03.770+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=488M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-03-29T16:48:03.777+0000 E STORAGE  [initandlisten] WiredTiger error (13) [1522342083:777174][14299:0x7f3e0cf7ad00], file:WiredTiger.wt, connection: /mongo-metadata/WiredTiger.turtle: handle-open: open: Permission denied
2018-03-29T16:48:03.777+0000 I -        [initandlisten] Assertion: 28595:13: Permission denied src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 269
2018-03-29T16:48:03.777+0000 I STORAGE  [initandlisten] exception in initAndListen: 28595 13: Permission denied, terminating
2018-03-29T16:48:03.778+0000 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2018-03-29T16:48:03.778+0000 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-03-29T16:48:03.778+0000 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2018-03-29T16:48:03.778+0000 I CONTROL  [initandlisten] now exiting
2018-03-29T16:48:03.778+0000 I CONTROL  [initandlisten] shutting down with code:100

Please help me get the mongoDB servers back up and running. I have contacted Digital Ocean 3 times and still have heard no response, it is so frustrating that I was literally in the hospital when they notified me of this hard reboot and do absolutely nothing to help fix the problem they created :( I won’t dispute my configuration is less than ideal, if not downright shitty, but some communication on their part would go along way, even if it consisted of you’re a dumbass. Instead of lets hard reboot your server and ignore you since we don’t give a shit that we crashed your apps.

Will definitely get anyone who helps me fix this problem a beer / whiskey / joint of their choosing :) Thank you!

Subscribe
Share

Submit an answer
You can type!ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!