Lost access to sites/servers and known good backups won't boot...

February 7, 2018 1.5k views
Backups Ubuntu

I am unable to access some of my servers/sites.
When I restore from a known good backup to the existing droplet or new droplets, I see the same error/warning:


DIGITALOCEAN Droplet Recovery System [17.10.1 (Artful Albacore)]

This system was placed into recovery mode by DigitalOcean support
or because the droplet cannot boot due a missing kernel.

  • Network access and SSH have been enabled automatically *
  • SSH Host keys will NOT match while using this rescue system *

Root Password has been randomly set to: 7459-bc92-d2c4-1056
You are strongly encouraged to use SSH keys.

Select One of the options below:

  1. Check Filesystem
  2. Reset Droplet Root Password
  3. Configure Keyboard
  4. Attempt to 'chroot' into installed system
  5. Interactive Shell [/bin/bash]
  6. Install a DigitalOcean provided Ubuntu 16.04 LTS kernel
  7. Install/update kernel on disk

Choose (1-7) and press Enter to continue.

I attempted to restore several times using snapshots that have previously worked.

The IP refuses an SSH connection so I used the console and observed a red "fail" message briefly displayed on the screen followed by the message posted above.

I tried the following options with no success at restoring normal use and access.

  1. Check Filesystem - passes
  2. I am able to chroot into the system and see no issues.
  3. Install a DigitalOcean provided Ubuntu 16.04 LTS kernel does not solve it
  4. Updating the Kernel does not work.

Why am I unable to use my legacy servers as they were configured?

1 comment
  • Same issue here for me today. I love how there is no response to a support ticket for over 8 hours and no way to use your backups. Really unhappy with DO.

13 Answers

I'm getting the same problem and I found the solution!

Short Answer:

1) Go to your Dashboard and Power Off your droplet.
2) Go to Kernel tab and change your kernel to use: DigitalOcean GrubLoader v0.2
3) Go to Access tab and click Launch Console
4) You should see the console screen with some options to choose (hit Enter key if you don't see any)
5) Choose the interactive bash command option
6) Type "reboot" and wait, don't close the window
7) You'll be prompted to select from 3 options. Choose option #2 (boot kernel from GrubLoader)
8) Done

Your droplet should start correctly again (at least in my case)

Long Answer:

My droplet went down and unable to boot (always in recovery mode) after the NYC1 Hypervisor Reboots maintenance completed by DO:

https://status.digitalocean.com/incidents/qwh4c27w73fr

Looking further into their post:
https://blog.digitalocean.com/a-message-about-intel-security-findings/

The most important information:
*
If your distribution is not included in the list of patched versions below, we highly recommend you move your data to a new Droplet running a version that is receiving security updates. To simplify the act of patching, we have recently updated Droplets to utilize a GrubLoader, which allows Droplets that use our in-control panel legacy system to boot into internally installed kernels. On certain legacy Droplets, this may cause issues if the kernel is not upgraded.*

My short answer above is based on the above information from DO.

  • hello my friend, in my case, my droplet dont have a kernel, only option is

    MOUNT RECOVERY KERNEL

    when i click the message is :
    THE IMAGE FOR THIS DROPLET DOES NOT ALLOW KERNELS TO BE CHANGED.

    • In this case skip step 1-2 and do step 3-8. Which is basically rebooting your droplet using the console (not the dashboard), so that you'll have the option to choose the kernel to boot during startup.

      • in my case when select "7) You'll be prompted to select from 3 options. Choose option #2 (boot kernel from GrubLoader)" this show a error fast and return to menu blue with 3 options. not have a option GRUB KERNEL.

        THE FAST ERROR IS: error: file /boot/grub/i386-pc/part_mbr.mod not found

        If i select the second option 3x the system boot and go to login root of ssh, but after reboot the system return to grub and selected the first option DigitalOcean Rescue Enviroment.

  • Thank you so much! I had the same exact problem and it worked for me too.

  • Thank you, same problem and it worked fine.

  • Thanks for the tips. I had a similar misfortune when my droplet was rebooted after the Spectre/Meltdown mitigation and would only boot into recovery mode due to a "missing kernel".

    The Kernel tab in my Account said I was on "DigitalOcean GrubLoader v0.2". I followed most of the steps above and installed a DigitalOcean provided Ubuntu 16.04 LTS kernel. When typing "reboot", nothing happened and my Chrome window eventually froze. Followed these steps 2-3 times in sequence, but still no luck.

    I eventually applied some system updates by ssh'ing directly into the box. Then I powered down the machine to perform a snapshot.

    When I powered things back up, my droplet booted correctly. Classic IT support trick to the rescue!

Contact DO support and ask that they change your droplet's active boot device back to its hard drive.

Thank you.
No response from support on the original ticket that matches my question here (10 hrs ago).
I'll add another ticket. Thank you for the recommendation.

  • Why would you open more tickets instead of just waiting for the original one to be answered?

Desperation.

  • Hey Alex, did you find a resolution to this issue? Happening to me as well.

    • Still no response from DO.
      :(

      My solution was to move a local backup to Linode to recover from the outage for my clients. I'm very disappointed with the lack of response to say the least.

The same problem! You will get this Rescue-mode also in case of power cycle for existing droplet.
SUPPORT! PLEASE HELP (ticket is created)

Any help on this? Same thing happened to me and I am beginning to loose it.

  • I'm going on 24h with no response to my ticket. I ended up moving a local backup to Linode to recover from the outage for my clients. I'm very disappointed with the lack of response to say the least.

Damn... how did you get your SQL databases? How are you able to pull the backup from ocean and move it? Am I missing something? This is horrible.

  • Offsite backup.
    You may be able to run commands through the recovery utility and then SCP or RSYNC out.

    I've been paying for what amount to useless backups for years - snapshots are not working either (on new Droplets).

    Left to DO support, we'd still be offline.
    Really, really lame.

    I tried everything I could think of and have come to the conclusion that old kernels are not going to work on DO post Intel revelation and patch frenzy...

@alex458755 Have you find solution? I have the same issues today. And Tried to contact to DO, but there is no support.

Same issue here. Anyone find an answer!?! :/

Same here... no response via ticket system; left to my own misunderstandings...

My droplet is down for 25 hours. Digital ocean didn't contact me or any of that sort. I've since issued two tickets that are still unanswered. This is horrible and the digitalocean support takes days!!

We are in the same boat. The servers that are down are legacy Ubuntu 12.04.
Maybe it's time to start posting about this fiasco on social media?

Has Digital ocean in making the changes they have that have prevented access of a number of there client committed criminal offences under 156 of the New York Penal code
S 156.10 Computer trespass.
A person is guilty of computer trespass when he or she knowingly uses,
causes to be used, or accesses a computer, computer service, or computer
network without authorization and:

  1. he or she does so with an intent to commit or attempt to commit or further the commission of any felony; or
  2. he or she thereby knowingly gains access to computer material. Computer trespass is a class E felony.

S 156.20 Computer tampering in the fourth degree.
A person is guilty of computer tampering in the fourth degree when he
or she uses, causes to be used, or accesses a computer, computer
service, or computer network without authorization and he or she
intentionally alters in any manner or destroys computer data or a
computer program of another person.
S 156.27 Computer tampering in the first degree.
A person is guilty of computer tampering in the first degree when he
commits the crime of computer tampering in the fourth degree and he
intentionally alters in any manner or destroys computer data or a
computer program so as to cause damages in an aggregate amount exceeding
fifty thousand dollars.
Computer tampering in the first degree is a class C felony.

Have another answer? Share your knowledge.