Resizing my Droplet made it unbootable

February 3, 2018 1.3k views
Storage CoreOS
slang
By:
slang

I resized my Droplet from the old 512MB RAM plan, to the new 5USD/mo 1GB RAM plan, and now it's not booting. When I view the screen via the QEMU console, it says "Booting from Hard Disk... Boot failed: not a bootable disk".

I opened a ticket 9 hours ago and I haven't gotten any response.

I've also tried taking a snapshot of the broken Droplet and creating a new droplet from that snapshot, but it just hangs at the "Booting from Hard Disk..." step and never errors out or boots. I've even tried this in another region, just in case NYC1 is having problems, but the exact same issue happens.

I can't download my snapshot, so my data is trapped in Digital Ocean's data center and I have no idea how to get it out. I also can't mount the recovery kernel because the CoreOS image doesn't have this option enabled.

What can I do?

Log In to Comment
2 Answers
mzelem February 6, 2018

I had a similar issue when I resized my droplet. It was a ubuntu 14.04 8GB droplet and when the resize was finished I saw the following error:

since I can't attach in image I'll type the error by hand:
error: no such device: xxxxx-xxxxxxx-xxxxx-xxxxx-xxxxx.

Press any key to continue...

Gave up waiting for root device. Common problems:

  • Boot args (cat /proc/cmdline)
  • Check rootdelay= (did the system wait long enough?)
  • Check root= (did the system wait for the right device?)
  • Missing modules (cat /proc/modules; ls /dev) ALERT! /dev/disk/by-uuid/xxxxx-xxxxxxx-xxxxx-xxxxx-xxxxx does not exist. Dropping to a shell!

I also got the same problem if I tried to restore from a backup.

However, I discovered that going to the 'kernel' tab and changing the kernel from the recovery kernel that it was set to to using an ubuntu 14.04 kernel seemed to fix the problem. Hopefully this helps.

Reply
  • cases.webapp April 15, 2018

    Just faced exactly the same problem after users of my system reported that they couldn't reach website. SSH access to the server just timed out. Went to console access in DigitalOcean and saw exactly the same error as you mentioned.

    Tried fsck from recovery kernel as mentioned here with no luck.

    Followed your advice (switched to 'Ubuntu 14.04 x64 vmlinuz-4.2.0-36-generic' kernel) and it logged in perfectly.

    Looks like DigitalOcean introduced new kernels (named 'DigitalOcean GrubLoader v0.1 (20160527) Kernel' and 'DigitalOcean GrubLoader v0.2 (20160714)') OR automatically switched my droplet to these and they are not working with my droplet.

    Thanks a lot mzelem for this advice!

    How To Recover from File System Corruption Using Fsck and a Recovery ISO
    by Justin Ellingwood
    Linux includes a filesystem checking utility called fsck that can be used to recover your system in cases of corruption. In this guide, we will discuss how to attempt to recover your filesystem if there are filesystem issues by using the fsck kernel and a recovery ISO.
slang February 6, 2018

CoreOS isn't able to change kernels, so I just waited a full day for the Digital Ocean support people to enable the Recovery ISO. From there I used dd and gzip to transfer the entire disk over ssh and recover files from it on my desktop machine.

The issue ended up being that the resize tool had destroyed the partition table of the disk and I needed to manually look for ext4 superblocks with xxd and grep. Hint: xxd disk.img | grep -E 'ff ?ff ?53 ?ef' will give you a reasonably sized list to look through. From there I managed to mount the disk with an offset I calculated based on the position of the superblock, recover the files, and move them over to a new VPS.

Digital Ocean support provided basically no help and they haven't even explained why resizing a droplet destroyed my filesystem.

Reply
Have another answer? Share your knowledge.

Related Questions