Update: As of April 12, 2018 we have removed the block on port 8083. We will continue to observe the result of this change and may reinstate the block if we see an uptick in abusive traffic. We will continue to post updates here, as necessary.
Details about the exploit
Due to a vulnerability within the VestaCP software, an exploit is being used to gain root access to Droplets running this software. Exploited Droplets are then being used to perform a DoS attack to remote servers by sending large amounts of traffic.
VestaCP has also acknowledged the issue and is investigating the behavior further themselves: https://forum.vestacp.com/viewtopic.php?p=68594#p68594
What is being done about this?
DigitalOcean is working to mitigate traffic from Droplets that have already been compromised. We have also blocked all traffic to TCP port 8083, the default port used by VestaCP for API and login requests. Additionally, we have disabled networking on a subset of Droplets suspected of running this software in an attempt to avoid a potential compromise or to prevent abusive traffic from being sent from these Droplets. Vesta has officially released an update that is meant to mitigate the vulnerability.
These packages are available via package manager updates or fresh installs. Though this update is meant to mitigate against new attacks, it will not help your system if it has already been compromised. It is advised that you create a new instance of Vesta, rather than updating old systems.
Disclaimer: These packages have not been tested by DigitalOcean to ensure the vulnerability is officially patched. Furthermore, we have not seen enough details of either the exploit or the patch to have a high degree of confidence in its ability to mitigate the vulnerability. This means that, even post patch, your Droplet could still be vulnerable. Please read on for additional information about ways to potentially reduce your risk of exploitation.
If you create a new instance of Vesta, we recommend that you change the default port away from 8083 (which is currently blocked by DigitalOcean). We have prepared a tutorial to assist users with installing VestaCP and migrating their data.
https://www.digitalocean.com/community/tutorials/how-to-install-vestacp-and-migrate-user-data
Our Support Team is working as quickly and efficiently as possible to respond to each ticket in a timely manner. As additional updates are available, we’ll modify this post. We appreciate your patience as we work to resolve outstanding issues.
How can I check if my Droplet was compromised?
/etc/cron.hourly/gcc.sh
file. as we have seen this cron particularly affected./lib/libudev.so: Unix.Trojan.DDoS_XOR-1 FOUND
What steps can I take to prevent my Droplet from being compromised?
service vesta stop
systemctl stop vesta
Block public inbound connections to port 8083. If needed, you can still establish connections through the use of an SSH tunnel by following our tutorial How To Set Up SSH Tunneling on a VPS .
Use DigitalOcean’s Cloud Firewalls to block SSH traffic and VestaCP control panel traffic from all IPs other than the one IP you use to manage your Droplet.
Follow our Community article, which includes mitigation steps, to spin up a new Droplet.
Additional Measures
Though Vesta has updated their package, there is still a chance the system is vulnerable (there is always a chance that any system is vulnerable). In light of this vulnerability, we recommend taking the following general security precautions to avoid takeovers of your Droplet(s). Compromised Droplets are typically used for cryptocurrency mining or Denial of Service (DoS) attacks, both of which will impact your ability to use your Droplet.
/bin/false
in /etc/passwd
-Consider running proactive security software like ossec or fail2banThese answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
After updating server via SSH how to change VestaCP port and add firewall rules
I run a web host that uses VestaCP as a backbone API for managing resources only, but everything has been patched and wanted to share a script to update your VestaCP port from 8083 to 5600. But only do this AFTER update and patch. It will create the firewall rule on VestaCP also.
curl http://www.nodehost.ca/scripts/sh/vestacp_changeport.sh > vestacp_changeport.sh && bash vestacp_changeport.sh
Feel free to download and see what the file actually does, and if you want to upload a file and run local here is the script in the file.
echo "NodeHost Custom VESTACP Script"
echo "JOB: Changing VESTACP port"
string="listen 8083;"
stringnew="listen 5600;"
grep "$stringnew" /usr/local/vesta/nginx/conf/nginx.conf || sed -i "s/$string/$stringnew/g" /usr/local/vesta/nginx/conf/nginx.conf
echo "JOB: Complete"
echo "JOB: Changing VESTACP firewall rule for new port"
v-add-firewall-rule ACCEPT 0.0.0.0/0 5600 TCP
echo "JOB: Complete"
echo "JOB: Restarting VESTACP"
service vesta restart
echo "JOB: Complete"
echo "JOB: Port has been changed to 5600 from 8083"
I hope this helps.
I find some of the comments here appalling. I had problems with the network on my servers almost a day and then I found out the result was that a bunch of other people on the network had chosen to install this vestacp thing that was vulnerable and caused their servers to attack the network. Digital ocean shut them down as they should have, only for them to come here and talk trash and act like digital ocean is responsible for their own stupid choices.
thanks digital ocean. don’t listen to the haters.
Ok, Below is my case
My Server ports were closed, but i was able to login and use SSH. So i updated vesta cp and changed ports. Also activated firewall from digital ocean account. Checked for Cron jobs didn’t find anything. Found Nothing Checked /etc/passwd file, make all changes of shell access(Only checked because everything was already in place). Also monitored ls /etc/cron.hourly/ ls /lib/ ls /etc/rc.* ls /etc/systemd/* Didn’t find anything.
Server is working fine for now but one question is left. Are these steps are enough?
Hello good day to all, really it is a very serious problem that is happening right now, many fallen sites, and many people losing traffic.
I had several fallen sites, and solved it in the following way:
1.- Make a backup of the database of each of the websites:
2.- Install Vesta on a Centos server (create a new droplet with Centos7)
3.- Change the port of vesta, it could be to 2083.
4.- Update the rules of the firewall in vesta via console:
5.- copy the files from the previous server to the current server: rsync -avzhie “ssh -p 22” root@IP.DEL.SERVER.OFTERIOR: /mnt/home/admin/backup.tar.gz/home/admin/backups-new-server/
6.- Restore all files.
If you have any questions with the steps above, let me know, I will be very happy to help you… :) a big hug and successes!
Hola buen día a todos, realmente es un problema muy grave el que está pasando ahora mismo, muchos sitos caidos, y muchas personas perdiendo tráfico.
Yo tuve varios sitios caidos, y lo solucioné de la siguiente manera:
1.- Hacer un backup de la base de datos de cada uno de los sitios web:
2.- Instalar Vesta en un servidor Centos (para ello cree una nueva gotita con Centos7)
3.- Cambiar el puerto de vesta, podría ser al 2083.
4.- Actualizar las reglas del firewall en vesta via consola:
5.- copiar los archivos desde el servidor anterior al servidor actual: rsync -avzhie “ssh -p 22” root@IP.DEL.SERVER.ANTERIOR:/mnt/home/admin/backup.tar.gz /home/admin/backups-new-server/
6.- Restaurar todos los archivos.
Si tienes alguna duda con los pasos anteriores, avísame, estaré muy contento de ayudarte… :) un fuerte abrazo y éxitos!
Hi DO
Please, please… can we have a reply to our ticket, it’s nearly been 24 hours and still not one response!?
Ticket #1443819
Thank you.
How we can send you full details about what exactly happend to VestaCP? We think there are STILL vulnerable servers on digitalocean, because hole was not at vestacp service. We found where is it.
Access the VPS via the emergency console.
scan your VPS
clamscan -r -i /
I found the trojan here
/lib/libudev.so: Unix.Trojan.DDoS_XOR-1 FOUND
I successfully removed the trojan using the instructions on https://admin-ahead.com/forum/server-security-hardening/unix-trojan-ddos_xor-1-chinese-chicken-multiplatform-dos-botnets-trojan/
reboot your vps
scan your VPS again
clamscan -r -i /
and make sure the cron job has been removed
ls -la /etc/cron.hourly/
Now that the trojan has been removed. You can request to remove the ban on your VPS network.
Hello, please check my ticket :)
==> Ticket #1474222
Hello. I checking and deleting the vulnerable files and disabled vesta servise. Please, turn networking back on my droplets. My tickets: 1461992 1455447 1462500 1462122 1461845 1461882 1462015 1445568 1462125 1456021 1462120 1462123 1456046
Hi, my droplet was affected by the VestaCP vulnerability. I host several sites on this droplet. I requested to boot it in the recovery tool to download data from it. I did it immediately and completely rebuilt the droplet with a new Debian image. I requested to boot the droplet back to it’s disk, but I get no answer… So now it is almost 5 days since my droplet is offline and DigitalOcean do not answer on tickets. I need the same IP address, because many domains are redirected to this IP, which I can not change. So rebuilding the droplet is the only option for me.
I am using DigitalOcean for more than 4 years now, but this failure of support is shocking. Please respond something on my ticket: #1448429
Click below to sign up and get $100 of credit to try our products over 60 days!
Hi guys! Did you resolved your issues? My tickets still waiting for answer
Hi, I am facing the same issue. I can’t access my rest API using 8083 port couple of mobile apps are running using this API http://[IP Removed]:8083/ and my ticket is Ticket #1447852. Don’t know how to solve this and also would like to know is this issue with this particular port and how much time it will take to solve this issue.
I’ve checked and my droplets were not compromised. I’ve patched Vesta, and changed it’s port, please check my Ticket #1447289.
thanks in advance
Hi ,
I have check and gone through all the necessary steps and can rest assured it is not compromise There is no such file gcc.sh was found in /etc/cron.hourly/gcc.sh and no such file libudev.so in /lib/ path.
I have taken measure to change the vestacp login port from 8083 to 56000.
For the moment i have stop the vesta service until it is patch and secure but i could not do so as there is no internet to from the droplet.
Kindly do reply as soon as possible to my ticket #1446648
Thank You.