Having application issues after spectre/meltdown reboots?

Question

(Written by Nate, Mike, and Jonathan from DO Support Team)

Due to reboots as a result of our efforts to mitigate the Spectre and Meltdown vulnerabilities, we’ve seen a number of customers reporting issues including failure to boot, services failing to start, etc. While in most cases, these issues are not caused by the reboot itself, or DigitalOcean’s platform, our goal is to help our customers troubleshoot as quickly and as efficiently as possible.

Below we’ve provided a few links to guides that will help troubleshoot common issues that will help you, as a customer, and us on the support side better understand where the issue may lie. If you’re unable to determine the root cause after troubleshooting, please don’t hesitate to reach out and our support team will be happy to further assist!

Troubleshooting Network Access and Connectivity

Kernel

Networking can fail if you are running an older droplet which has a kernel selection box in the control panel. This selection box chooses the kernel that we try to boot, and just because the system boots does not mean that the correct one is selected. In this case, you will want to follow this guide to make sure you are booting under the right kernel:

https://www.digitalocean.com/community/tutorials/how-to-update-a-digitalocean-server-s-kernel

A valid test may simply be to set the kernel to our Grub Loader kernel and see if this restores your networking. If you are already on the Grub Loader kernel, it can be beneficial to try setting the original kernel used on the droplet. You may not be aware of what that kernel was, and our support team may be able to assist you with this via support ticket.

ping

Using ping and ping6

Attempting to ping your Droplet will allow you to confirm that your Droplet is reachable over network from either your local environment, or another Droplet or server. If you’ve disabled ICMP requests, ping will fail. This is common if you’re using the default ruleset provided by our Cloud Firewall service, or if you’ve modified ufw, iptables, firewalld, or your firewall of choice to do the same.

Example Usage

ping -c 5 11.22.33.44

The above will send 5 packages to the IP 11.22.33.44 and return packet loss (if any) and round-trip time. The response times for ping will vary based on your location and where your Droplet is located, thus for example, pinging a Droplet in NYC3 from India will likely be higher than someone pinging a Droplet in NYC3 from within the United States.

nmap

Using nmap will allow you to run a port scan against your Droplets IP to confirm whether a port is open, filtered, or closed. Most commonly, we see that ports 80 (HTTP) or 443 (HTTPS/SSL) have been firewalled off, thus preventing web access to your website or application.

Example Usage

nmap -p 22,80,443

The above command will check ports 22 (SSH), 80 (HTTP), and 443 (HTTPS/SSL) for a response and return their current status.

mtr

How To Use Traceroute and MTR to Diagnose Network Issues

Using mtr allows you to troubleshooting connectivity issues between Droplets, or from your local environment to a Droplet. It provides details such as latency, hops between routes, etc.

Using the DigitalOcean Web Console

How To Use the DigitalOcean Console to Access your Droplet

If you’re unable to access your Droplet via SSH, whether due to the firewall blocking access, or due to a larger issue, DigitalOcean provides access to a web based console. The web Console functions much like a physical attached console in that even if SSH isn’t working, you should be able to access your Droplet.

The web Console does require a root password to be used as it does not currently provide support for SSH keys. If you’ve locked the root user, or you’ve forgot the password for the root user, you may need to do a password reset before you’ll be able to access console. To reset your root password, please:

Visit https://cloud.digitalocean.com
Click on the name of the Droplet you wish to reset the root password on.
Click on Access from the left side menu.
Under Reset Root Password, please click on the “Reset Root Password” button.

Note: This will reboot your Droplet.

Troubleshooting Your Web Server (Apache/NGINX)

One of the most common web server issues is a 40x or 50x error in the browser. This indicates that the Droplet itself is working properly, however the web server may not be able to serve the request as expected. For information on web server status codes, please see:

HTTP Status Codes

The most popular or often seen error codes are:

200 - OK
404 - File Not Found
500 - Internal Server Error
503 - Service Unavailable
504 - Gateway Timeout

When it comes to errors, the error log for the web server is going to be the best file to refer to. This file will often detail additional information associated with the error received and in some cases, application specific information that can also be helpful in determining the root cause. We can use the tail command to check the last X lines of the log (where X is any number you choose).

For Apache

tail -20 /var/log/apache2/error.log

For NGINX

tail -20 /var/log/nginx/error.log

Troubleshooting Virtual Hosts (Apache) and Server Blocks (NGINX)

When it comes to Virtual Hosts and Server Blocks, a missing } or and extra line or space can often be the difference between the web server starting and failing. While the guides below do target Ubuntu, the configuration for Virtual Hosts and Server Blocks is not OS-specific and applies globally to all distributions.

General Troubleshooting

Common Linux Troubleshooting

How To Troubleshoot Common Site Issues on a Linux Server

Troubleshooting Logs (ex: system)

Reaching Out to DigitalOcean Support

If the above guides do not aid in resolving the issue that you’re currently experiencing, please feel free to reach out to our support team by submitting a support ticket within our support portal:

https://cloudsupport.digitalocean.com

Please keep in mind, our support team does not have access to the internals of your Droplet, thus we’re unable to execute commands or pull information directly. Any details that you can provide from the troubleshooting steps above will assist us in troubleshooting the issues you’re experiencing and working with you towards a resolution.

eric654 · Accepted Answer

This page needs an update to tell people what to do when their droplet no longer boots. Users need to go to the dashboard, click on the droplet that won’t boot. Then select kernel from the menu on the left. The kernel probably shows Grubload v0.2 which works sometimes, but sometimes does not. Confirm by selecting Access from the left menu, and launch console. Then look for an initramfs prompt where the booting stops.

If it’s stuck at initramfs, click on the kernels menu and select a kernel from the list by clicking on Select a kernel. Your choice must match your architecture, probably 64 bit, and your OS version, probably 14.04 if it’s a couple of years old.