By wmnserver
So after Digital Ocean gave absolutely no warning and hard rebooted our servers, everything was fine except, big surprise, the MongoDB servers! I have finally fixed the configuration files, only to realize they were fine all along, and have concluded that there is some kind of a permission issue. I am attempting to restart the mongod service and pull from the log files. Also sorry, but I updated the question twice with more information and Digital Ocean then rejected it as spam so I am attempting to post again.
When I run the command chown -R mongodb:mongodb /mongo-metadata/ ,
Afterwards we do a “sudo service mongod restart” and the status we get is:
root@mongoprimary:/mongo-metadata# sudo service mongod status
● mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Thu 2018-03-29 16:50:28 UTC; 5s ago
Docs: https://docs.mongodb.org/manual
Process: 14317 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=0/SUCCESS)
Main PID: 14317 (code=exited, status=0/SUCCESS)
Mar 29 16:50:28 mongoprimary.******.com systemd[1]: Started High-performance, schema-free document-oriented database.
Mar 29 16:50:28 mongoprimary.*****.com mongod[14317]: about to fork child process, waiting until server is ready for connectio
Mar 29 16:50:28 mongoprimary.*****.com mongod[14317]: forked process: 14320
the output of my log file to better understand the error:
root@mongoprimary:/mongo-metadata# tail /var/log/mongodb/mongod.log
2018-03-29T16:50:28.633+0000 I REPL [replExecDBWorker-2] transition to STARTUP2
2018-03-29T16:50:28.633+0000 I REPL [replExecDBWorker-2] Starting replication storage threads
2018-03-29T16:50:28.633+0000 I REPL [signalProcessingThread] Stopping replication storage threads
2018-03-29T16:50:28.633+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to mongosecondary1.******.com:27017
2018-03-29T16:50:28.633+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to mongosecondary2.******.com:27017
2018-03-29T16:50:28.634+0000 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture
2018-03-29T16:50:28.634+0000 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
2018-03-29T16:50:28.688+0000 I STORAGE [signalProcessingThread] shutdown: removing fs lock...
2018-03-29T16:50:28.688+0000 I CONTROL [signalProcessingThread] now exiting
2018-03-29T16:50:28.688+0000 I CONTROL [signalProcessingThread] shutting down with code:0
After I run a db repair command (both with and without sudo), command I run:
“mongod --repair --dbpath /mongo-metadata --storageEngine wiredTiger”
Last line of the output repair function:
2018-03-29T16:47:41.513+0000 I INDEX [initandlisten] build index on: meteor.users properties: { v: 2, key: { vendorId: 1 }, name: "vendorId_1", ns: "meteor.users" }
2018-03-29T16:47:41.513+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 38 megabytes of RAM
2018-03-29T16:47:41.607+0000 I STORAGE [initandlisten] finished checking dbs
2018-03-29T16:47:41.607+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2018-03-29T16:47:41.607+0000 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-03-29T16:47:41.607+0000 I NETWORK [initandlisten] shutdown: going to flush diaglog...
2018-03-29T16:47:41.607+0000 I STORAGE [initandlisten] WiredTigerKVEngine shutting down
2018-03-29T16:47:41.614+0000 I STORAGE [initandlisten] shutdown: removing fs lock...
2018-03-29T16:47:41.614+0000 I CONTROL [initandlisten] now exiting
2018-03-29T16:47:41.614+0000 I CONTROL [initandlisten] shutting down with code:0
Then when I attempt to restart:
root@mongoprimary:/mongo-metadata# sudo service mongod restart
root@mongoprimary:/mongo-metadata# tail /var/log/mongodb/mongod.log
2018-03-29T16:48:03.770+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-03-29T16:48:03.770+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=488M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-03-29T16:48:03.777+0000 E STORAGE [initandlisten] WiredTiger error (13) [1522342083:777174][14299:0x7f3e0cf7ad00], file:WiredTiger.wt, connection: /mongo-metadata/WiredTiger.turtle: handle-open: open: Permission denied
2018-03-29T16:48:03.777+0000 I - [initandlisten] Assertion: 28595:13: Permission denied src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 269
2018-03-29T16:48:03.777+0000 I STORAGE [initandlisten] exception in initAndListen: 28595 13: Permission denied, terminating
2018-03-29T16:48:03.778+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2018-03-29T16:48:03.778+0000 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-03-29T16:48:03.778+0000 I NETWORK [initandlisten] shutdown: going to flush diaglog...
2018-03-29T16:48:03.778+0000 I CONTROL [initandlisten] now exiting
2018-03-29T16:48:03.778+0000 I CONTROL [initandlisten] shutting down with code:100
Please help me get the mongoDB servers back up and running. I have contacted Digital Ocean 3 times and still have heard no response, it is so frustrating that I was literally in the hospital when they notified me of this hard reboot and do absolutely nothing to help fix the problem they created :( I won’t dispute my configuration is less than ideal, if not downright shitty, but some communication on their part would go along way, even if it consisted of you’re a dumbass. Instead of lets hard reboot your server and ignore you since we don’t give a shit that we crashed your apps.
Will definitely get anyone who helps me fix this problem a beer / whiskey / joint of their choosing :) Thank you!
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Hello,
Just came across this answer and decided to write some general guidelines for anyone who comes across this in the future despite the old question.
Firstly, I’m sorry to hear about the troubles you’re having, let’s try to get your MongoDB back up and running.
From the log, it appears that MongoDB is indeed running into a permission issue when it tries to access the file /mongo-metadata/WiredTiger.turtle.
Assuming MongoDB is running under the user ‘mongodb’, try running the following commands:
sudo chown -R mongodb:mongodb /mongo-metadata/
sudo chmod -R 755 /mongo-metadata/
These commands ensure that the MongoDB user has the appropriate permissions on the /mongo-metadata/ directory.
The chown command changes the owner of the directory (and all subdirectories and files, because of the -R flag for recursive) to the ‘mongodb’ user and group.
The chmod command changes the permissions of the directory (and all subdirectories and files) to 755. In Linux permission model, 755 means the owner (which now is ‘mongodb’) can read, write and execute, while the group and others can read and execute.
After running these commands, try restarting the MongoDB service again and see if it starts as normal.
In the event that you are still experiencing issues after changing the permissions, another possibility is that the abrupt shutdown caused some corruption to the MongoDB data files. MongoDB uses the WiredTiger storage engine, which generally recovers well from unclean shutdowns, but it’s not perfect.
If changing permissions doesn’t work, you may have to restore from a backup. If you don’t have a backup, you might try to repair the MongoDB, but be aware that this operation can result in data loss.
Also after fixing the permissions, try to attempt another repair as you did earlier by using the --repair option when starting MongoDB. You should make a backup of your data directory first, just in case.
I hope this information helps anyone who comes across this in the future.
A good option to consider is using a managed MongoDB service so that you don’t have to worry about managing such incidents:
Best,
Bobby
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
Full documentation for every DigitalOcean product.
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.
New accounts only. By submitting your email you agree to our Privacy Policy
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.