Here is a follow-up - it may be a little long - maybe it will help others make informed decisions regarding using DO or VMs in general.
So, ultimately, at the end of the day, my server became hosed and I was forced to provision another one to bring my app back up. It was down for 16*+ hours. We (the owner who pays me) got 80 bucks to compensate and some friendly guesses regarding what might of happened - 'probably because I had shut down improperly' - unlikely because I simply had not shutdown. More likely, after doing some research, the need for fsck was caused by faulty (or faltering) hardware - or faulty disk mounting (of SSDs). Would I bet my life on it? No. They did mention no others had experienced the same problem - which would blow that theory out of the water. Which is interesting since they have a kernel mode specifically for booting and running fsck.
In any case this was the second time in 3 months that my app was left down. I've been running this same app for *years* without issues both on hardware(a mac mini which had a hybrid persistance hd/ssd, which I do miss sometimes) and on other VMs.
I've moved my app server to Linode - leaving my asset host in place (which thank goodness also had backups from the server that was hosed). I should have done this before to mitigate risk anyway.
So my lesson, learned:
1) Make damn sure you are making backups. Do them yourself the old fashioned cron/rsync way. Use 2+ DCs. (yea, I had done this)
2) Have a fallback. Like I set I got set up in Linode - and kept my DO asset host/backup andI *might* come back someday and setup an app server again - and fail over to it if Linode craps out
4) Experiment with 'managed' plans. It *might* be worth it. It *might* have helped in this case
5) Reboot and run fsck periodically - investigate what it finds, examining specifically for possible faulty sectors/drives/mounts
My theory: SSDs are awesome. But may not be ideal for servers (that write often).
I won't be severing ties completely with DO, but I won't be recommending them either.
Hope that helps someone.
Paul
Hi All,
I faced the same issue in 3 months it get down 2nd time, 1 time i loss my data.
i have open ticket but they are just replacing the boot option and nothing more that.
my server down more than 36 hours i have migrate sites but mysql data still there.
anyone can help me how to backup mysql data,
we don’t have access in terminal, only VNC is access but showing error: read only file system