How To Recover from a Compromised Droplet Sending an Outgoing Flood or DDoS
This tutorial explains the next steps to take after receiving a message from DigitalOcean support that your Droplet is sending an outgoing flood or DDoS.
This article's content is based on the popular post by Will from DigitalOcean's support team. This tutorial focuses on the post-mortem aspect of recovering from a compromised Droplet, while also providing a good overview of all the steps needed for a full recovery.
Why did I get this message?
You received this message because your Droplet is sending out an unusually large amount of traffic. In most cases, this indicates that your Droplet's security has been compromised and that someone is using it to send spam email or malicious traffic.
To protect others from being harmed, your Droplet's normal Internet access has been disabled, but you can still access it through the control panel console.
What should I do next?
We recommend that you create a plan to recover your content, get it back online, and resolve current and future security issues.
Each server setup is unique, but in many cases, these are the general steps you should follow:
- Recover content from the compromised Droplet
- Deploy a new Droplet
- Secure your new Droplet
- Deploy content to new Droplet
- Conduct a post-mortem on your old Droplet
- Make regular backups
Step 1 — Recovering Content
First, you need to access your server through the console in the DigitalOcean control panel. The link looks like this:
XXXXX is your Droplet's ID.
After you get access to your server, you can recover content from it.
You'll need a password for root, so if you don't have one please contact support for further advice.
You can read a detailed article on how to do this:
Step 2 — Deploying a New Droplet
In most cases, it's faster to move to a fresh, uncompromised Droplet.
For detailed instructions on how to spin up a new Droplet, please read this tutorial:
Step 3 — Securing Your Droplet
Next, secure your new Droplet. You'll want to make sure your new Droplet doesn't get hacked in the same way as your old one.
Some good security practices include:
- Disable the root user
- Use an SSH key
- Keep your software up to date
- Use strong passwords
- Keep your firewall settings as strict as possible
You can read an entire article on security best practices here:
Step 4 — Deploying Content to New Droplet
Now that you have a new, secured Droplet set up, it's time to redeploy your content. Your individual setup will be unique in this case.
As you install your software and upload your content, it's a good idea to do a quick sanity check for security at each step. A few best practices to keep in mind:
- Run the latest versions of software
- Run applications as unprivileged users
- Use strong passwords
- Use certificates and keys where possible
- Never use 777 permissions
- Remove installation files and unused forms
Step 5 — Conducting a Post-Mortem
Now that your content is back online, you have some breathing space to figure out what went wrong the first time. These steps can help you find evidence of viruses and trojans on your old server.
Log into your old server using the console in our control panel.
Note: If you find anything suspicious in this step that could lead to a similar compromise on your new Droplet, take steps to prevent the issue from recurring.
Once you are logged in from the console, use one of these commands to try to find an unfamiliar process running:
This command, if installed, shows programs holding open a network socket:
This command will show all running processes:
Adding a pipe to an output paging program may help for long output. Examples:
lsof -i | less ps -ef | less
At this point, you may have a process or two that you want to investigate. Make a note of the process ID for each one, which is a string of numbers.
Next we want to find the malicious files on your system.
You can use this command to locate the executable file that is the origin of a particular process. Replace XXXX with the process ID (PID) you found earlier:
ls -al /proc/XXXX/exe
You can repeat this command for any suspicious processes you noted earlier.
You can also search for suspicious files yourself. Common places trojans hide are:
You can use this command to list all content in a particular folder, including dot files. This example is for the
ls -al /boot
If you find something foreign, check the ownership of the files for hints on what user was used to install the malicious code.
Review your log files to try to find out how the code was installed so that you can work on preventing it from happening again.
If you need any advice, send DigitalOcean support whatever data you are looking at that you need help with and they will try to point you in the right direction. The best way is to screenshot the console showing the data you are uncertain of, upload to a file sharing service (imgur.com, dropbox.com, etc.) and send the URL in the ticket.
Some programs that may also help are:
If you can't find anything, let support know via a support ticket for advice.
Now you can kill any malicious processes and remove the files.
Double-check your new Droplet to make sure nothing suspicious was copied over. If you identified the user or program that was compromised the first time, take extra steps to secure that user or program on the new server.
If you have success finding malicious processes and files, post your results in the comments to help other people. Additional recovery tips are also welcome!
Step 6 — Making Backups
Security is important, and so is being prepared. Should your Droplet ever become compromised again in the future, backups will make your recovery process much easier. We recommend acquainting yourself with backup strategies and choosing the one that's right for you:
Having a compromised Droplet is no fun, but a good recovery plan can get you back on your feet in no time. Taking the time to do a post-mortem on your compromised Droplet can help you avoid running into the same problems twice.
Additional copy by Sharon Campbell