Having application issues after spectre/meltdown reboots?

March 7, 2018 3.4k views
Applications

(Written by Nate, Mike, and Jonathan from DO Support Team)

Due to reboots as a result of our efforts to mitigate the Spectre and Meltdown vulnerabilities, we've seen a number of customers reporting issues including failure to boot, services failing to start, etc. While in most cases, these issues are not caused by the reboot itself, or DigitalOcean's platform, our goal is to help our customers troubleshoot as quickly and as efficiently as possible.

Below we've provided a few links to guides that will help troubleshoot common issues that will help you, as a customer, and us on the support side better understand where the issue may lie. If you're unable to determine the root cause after troubleshooting, please don't hesitate to reach out and our support team will be happy to further assist!

Troubleshooting Network Access and Connectivity

Kernel

Networking can fail if you are running an older droplet which has a kernel selection box in the control panel. This selection box chooses the kernel that we try to boot, and just because the system boots does not mean that the correct one is selected. In this case, you will want to follow this guide to make sure you are booting under the right kernel:

https://www.digitalocean.com/community/tutorials/how-to-update-a-digitalocean-server-s-kernel

A valid test may simply be to set the kernel to our Grub Loader kernel and see if this restores your networking. If you are already on the Grub Loader kernel, it can be beneficial to try setting the original kernel used on the droplet. You may not be aware of what that kernel was, and our support team may be able to assist you with this via support ticket.

ping

Attempting to ping your Droplet will allow you to confirm that your Droplet is reachable over network from either your local environment, or another Droplet or server. If you've disabled ICMP requests, ping will fail. This is common if you're using the default ruleset provided by our Cloud Firewall service, or if you've modified ufw, iptables, firewalld, or your firewall of choice to do the same.

Example Usage

ping -c 5 11.22.33.44

The above will send 5 packages to the IP 11.22.33.44 and return packet loss (if any) and round-trip time. The response times for ping will vary based on your location and where your Droplet is located, thus for example, pinging a Droplet in NYC3 from India will likely be higher than someone pinging a Droplet in NYC3 from within the United States.

nmap

Using nmap will allow you to run a port scan against your Droplets IP to confirm whether a port is open, filtered, or closed. Most commonly, we see that ports 80 (HTTP) or 443 (HTTPS/SSL) have been firewalled off, thus preventing web access to your website or application.

Example Usage

nmap -p 22,80,443

The above command will check ports 22 (SSH), 80 (HTTP), and 443 (HTTPS/SSL) for a response and return their current status.

mtr

Using mtr allows you to troubleshooting connectivity issues between Droplets, or from your local environment to a Droplet. It provides details such as latency, hops between routes, etc.

Using the DigitalOcean Web Console

If you're unable to access your Droplet via SSH, whether due to the firewall blocking access, or due to a larger issue, DigitalOcean provides access to a web based console. The web Console functions much like a physical attached console in that even if SSH isn't working, you should be able to access your Droplet.

The web Console does require a root password to be used as it does not currently provide support for SSH keys. If you've locked the root user, or you've forgot the password for the root user, you may need to do a password reset before you'll be able to access console. To reset your root password, please:

  • Visit https://cloud.digitalocean.com
  • Click on the name of the Droplet you wish to reset the root password on.
  • Click on Access from the left side menu.
  • Under Reset Root Password, please click on the "Reset Root Password" button.

Note: This will reboot your Droplet.

Troubleshooting Your Web Server (Apache/NGINX)

One of the most common web server issues is a 40x or 50x error in the browser. This indicates that the Droplet itself is working properly, however the web server may not be able to serve the request as expected. For information on web server status codes, please see:

The most popular or often seen error codes are:

  • 200 - OK
  • 404 - File Not Found
  • 500 - Internal Server Error
  • 503 - Service Unavailable
  • 504 - Gateway Timeout

When it comes to errors, the error log for the web server is going to be the best file to refer to. This file will often detail additional information associated with the error received and in some cases, application specific information that can also be helpful in determining the root cause. We can use the tail command to check the last X lines of the log (where X is any number you choose).

For Apache

tail -20 /var/log/apache2/error.log

For NGINX

tail -20 /var/log/nginx/error.log

Troubleshooting Virtual Hosts (Apache) and Server Blocks (NGINX)

When it comes to Virtual Hosts and Server Blocks, a missing } or and extra line or space can often be the difference between the web server starting and failing. While the guides below do target Ubuntu, the configuration for Virtual Hosts and Server Blocks is not OS-specific and applies globally to all distributions.

General Troubleshooting

Common Linux Troubleshooting

Troubleshooting Logs (ex: system)

Reaching Out to DigitalOcean Support

If the above guides do not aid in resolving the issue that you're currently experiencing, please feel free to reach out to our support team by submitting a support ticket within our support portal:

Please keep in mind, our support team does not have access to the internals of your Droplet, thus we're unable to execute commands or pull information directly. Any details that you can provide from the troubleshooting steps above will assist us in troubleshooting the issues you're experiencing and working with you towards a resolution.

4 Answers
eric654 April 12, 2018
Accepted Answer

This page needs an update to tell people what to do when their droplet no longer boots. Users need to go to the dashboard, click on the droplet that won't boot. Then select kernel from the menu on the left. The kernel probably shows Grubload v0.2 which works sometimes, but sometimes does not. Confirm by selecting Access from the left menu, and launch console. Then look for an initramfs prompt where the booting stops.

If it's stuck at initramfs, click on the kernels menu and select a kernel from the list by clicking on Select a kernel. Your choice must match your architecture, probably 64 bit, and your OS version, probably 14.04 if it's a couple of years old.

After the recent NYC2 maintenance, my droplet failed to boot. The web console showed:

Giving up waiting for root device.
ALERT! /dev/disk/by-uuid/XXXX does not exist.
Dropping to shell!

This is caused because the recent maintenance changed the UUID of the root disk.

Here's how to fix it:

  1. find the new disk uuid:
    (initramfs) ls /dev/disk/by-uuid
    YYYY

  2. reboot into grub
    (initramfs) reboot

when the grub menu appears, hit 'e' to edit the command line
The command line looks like: /boot/vmlinuz-3.13.0-48-generic root=UUID=XXXX

Change the old UUID (XXXX) to YYYY and type ^X to reboot.

  1. once you droplet has rebooted ok, regenerate grub config with the new UUID $ sudo update-grub
  • That's awesome, thank you for posting it. We honestly hadn't been seeing that work with the recent round of tickets on the support side, and saw the kernel instructions working quite consistently. If this worked for you, it's definitely worth documenting here.

  • Work's nicely, i would like found this before lost 200 customers on my database today, very sad.

    I open several tickets and no one responds, nice support DO.

Emergency thing is that my droplet ip can't visit!!! My website and my data are all stock in it. Ok?

  • While I'm not able to promise that I can quickly get to requests made in the comments here, I can tell you that I'm on top of this right now.

    Kind Regards,
    Jarland
    Platform Support Lead

I'm also having this problem since the 12th. Can we get any sort of update?

Have another answer? Share your knowledge.