How do I determine the impact of VestaCP vulnerability from April 8th, 2018?

Question

Update: As of April 12, 2018 we have removed the block on port 8083. We will continue to observe the result of this change and may reinstate the block if we see an uptick in abusive traffic. We will continue to post updates here, as necessary.

Details about the exploit

Due to a vulnerability within the VestaCP software, an exploit is being used to gain root access to Droplets running this software. Exploited Droplets are then being used to perform a DoS attack to remote servers by sending large amounts of traffic.

VestaCP has also acknowledged the issue and is investigating the behavior further themselves: https://forum.vestacp.com/viewtopic.php?p=68594#p68594

What is being done about this?

DigitalOcean is working to mitigate traffic from Droplets that have already been compromised. We have also blocked all traffic to TCP port 8083, the default port used by VestaCP for API and login requests. Additionally, we have disabled networking on a subset of Droplets suspected of running this software in an attempt to avoid a potential compromise or to prevent abusive traffic from being sent from these Droplets. Vesta has officially released an update that is meant to mitigate the vulnerability.

These packages are available via package manager updates or fresh installs. Though this update is meant to mitigate against new attacks, it will not help your system if it has already been compromised. It is advised that you create a new instance of Vesta, rather than updating old systems.

Disclaimer: These packages have not been tested by DigitalOcean to ensure the vulnerability is officially patched. Furthermore, we have not seen enough details of either the exploit or the patch to have a high degree of confidence in its ability to mitigate the vulnerability. This means that, even post patch, your Droplet could still be vulnerable. Please read on for additional information about ways to potentially reduce your risk of exploitation.

If you create a new instance of Vesta, we recommend that you change the default port away from 8083 (which is currently blocked by DigitalOcean). We have prepared a tutorial to assist users with installing VestaCP and migrating their data.

https://www.digitalocean.com/community/tutorials/how-to-install-vestacp-and-migrate-user-data

Our Support Team is working as quickly and efficiently as possible to respond to each ticket in a timely manner. As additional updates are available, we’ll modify this post. We appreciate your patience as we work to resolve outstanding issues.

How can I check if my Droplet was compromised?

Check all cron jobs on your machine for malicious activity. Specifically, check the contents of the /etc/cron.hourly/gcc.sh file. as we have seen this cron particularly affected.
Install an antivirus software, run a scan and then check for any results similar to /lib/libudev.so: Unix.Trojan.DDoS_XOR-1 FOUND

What steps can I take to prevent my Droplet from being compromised?

Consider stopping the Vesta service if you do not have an immediate need for it:

service vesta stop
systemctl stop vesta

Block public inbound connections to port 8083. If needed, you can still establish connections through the use of an SSH tunnel by following our tutorial How To Set Up SSH Tunneling on a VPS .
Use DigitalOcean’s Cloud Firewalls to block SSH traffic and VestaCP control panel traffic from all IPs other than the one IP you use to manage your Droplet.
Follow our Community article, which includes mitigation steps, to spin up a new Droplet.

Additional Measures

Though Vesta has updated their package, there is still a chance the system is vulnerable (there is always a chance that any system is vulnerable). In light of this vulnerability, we recommend taking the following general security precautions to avoid takeovers of your Droplet(s). Compromised Droplets are typically used for cryptocurrency mining or Denial of Service (DoS) attacks, both of which will impact your ability to use your Droplet.

Change any default passwords on your installed software (eg, admin/admin), as these are commonly used in takeover scenarios
Enable your local firewall (eg, iptables) to prevent all inbound connections by default, and only allow connections on ports you’re specifically serving
Add monitoring software that notifies you when your resources, especially CPU or bandwidth, are extremely high
Ensure your password is sufficiently complex, preferably longer than ten characters
If possible, keep your software up-to-date by enabling auto-update features
Periodically scan through your user lists in /etc/passwd and /etc/shadow for unknown users
Ensure that users who don’t need interactive console access have their shell set to /usr/sbin/nologin or /bin/false in /etc/passwd -Consider running proactive security software like ossec or fail2ban

Anthony Rossbach · Answer

After updating server via SSH how to change VestaCP port and add firewall rules I run a web host that uses VestaCP as a backbone API for managing resources only, but everything has been patched and wanted to share a script to update your VestaCP port from 8083 to 5600. But only do this AFTER update and patch. It will create the firewall rule on VestaCP also. curl http://www.nodehost.ca/scripts/sh/vestacp_changeport.sh > vestacp_changeport.sh && bash vestacp_changeport.sh Feel free to download and see what the file actually does, and if you want to upload a file and run local here is the script in the file. echo "NodeHost Custom VESTACP Script" echo "JOB: Changing VESTACP port" string="listen 8083;" stringnew="listen 5600;" grep "$stringnew" /usr/local/vesta/nginx/conf/nginx.conf || sed -i "s/$string/$stringnew/g" /usr/local/vesta/nginx/conf/nginx.conf echo "JOB: Complete" echo "JOB: Changing VESTACP firewall rule for new port" v-add-firewall-rule ACCEPT 0.0.0.0/0 5600 TCP echo "JOB: Complete" echo "JOB: Restarting VESTACP" service vesta restart echo "JOB: Complete" echo "JOB: Port has been changed to 5600 from 8083" I hope this helps.

MaryRWebb · Answer

I find some of the comments here appalling. I had problems with the network on my servers almost a day and then I found out the result was that a bunch of other people on the network had chosen to install this vestacp thing that was vulnerable and caused their servers to attack the network. Digital ocean shut them down as they should have, only for them to come here and talk trash and act like digital ocean is responsible for their own stupid choices.

thanks digital ocean. don’t listen to the haters.

narinderbansalmonk · Answer

Ok, Below is my case

My Server ports were closed, but i was able to login and use SSH. So i updated vesta cp and changed ports. Also activated firewall from digital ocean account. Checked for Cron jobs didn’t find anything. Found Nothing Checked /etc/passwd file, make all changes of shell access(Only checked because everything was already in place). Also monitored ls /etc/cron.hourly/ ls /lib/ ls /etc/rc.* ls /etc/systemd/* Didn’t find anything.

Server is working fine for now but one question is left. Are these steps are enough?

alexcastaneda · Answer

Hello good day to all, really it is a very serious problem that is happening right now, many fallen sites, and many people losing traffic.

I had several fallen sites, and solved it in the following way:

1.- Make a backup of the database of each of the websites:

mysqldump -u userdb -p dbname - p dbname > name_file_backup.sql

2.- Install Vesta on a Centos server (create a new droplet with Centos7)

3.- Change the port of vesta, it could be to 2083.

cd /usr/local/vesta/nginx/conf/ && nano nginx.conf
change “listen 8083;” to “listen 2083;”
service vesta restart

4.- Update the rules of the firewall in vesta via console:

v-change-firewall-rule Accept 0.0.0.0/0 2083 TCP

5.- copy the files from the previous server to the current server: rsync -avzhie “ssh -p 22” root@IP.DEL.SERVER.OFTERIOR: /mnt/home/admin/backup.tar.gz/home/admin/backups-new-server/

6.- Restore all files.

If you have any questions with the steps above, let me know, I will be very happy to help you… :) a big hug and successes!

Hola buen día a todos, realmente es un problema muy grave el que está pasando ahora mismo, muchos sitos caidos, y muchas personas perdiendo tráfico.

Yo tuve varios sitios caidos, y lo solucioné de la siguiente manera:

1.- Hacer un backup de la base de datos de cada uno de los sitios web:

mysqldump -u usuariodb -p nombrebd - p nombre_bd > archivo.sql

2.- Instalar Vesta en un servidor Centos (para ello cree una nueva gotita con Centos7)

3.- Cambiar el puerto de vesta, podría ser al 2083.

cd /usr/local/vesta/nginx/conf/ && nano nginx.conf
cambiar " listen 8083; " por " listen 2083; "
service vesta restart

4.- Actualizar las reglas del firewall en vesta via consola:

v-change-firewall-rule Accept 0.0.0.0/0 2083 TCP

5.- copiar los archivos desde el servidor anterior al servidor actual: rsync -avzhie “ssh -p 22” root@IP.DEL.SERVER.ANTERIOR:/mnt/home/admin/backup.tar.gz /home/admin/backups-new-server/

6.- Restaurar todos los archivos.

Si tienes alguna duda con los pasos anteriores, avísame, estaré muy contento de ayudarte… :) un fuerte abrazo y éxitos!

tomapryor · Answer

Hi DO Please, please… can we have a reply to our ticket, it’s nearly been 24 hours and still not one response!? Ticket #1443819 Thank you.

vestalovers · Answer

How we can send you full details about what exactly happend to VestaCP?We think there are STILL vulnerable servers on digitalocean, because hole was not at vestacp service.We found where is it.

karimhb · Answer

Access the VPS via the emergency console.

scan your VPS

clamscan -r -i /

I found the trojan here

/lib/libudev.so: Unix.Trojan.DDoS_XOR-1 FOUND

I successfully removed the trojan using the instructions on https://admin-ahead.com/forum/server-security-hardening/unix-trojan-ddos_xor-1-chinese-chicken-multiplatform-dos-botnets-trojan/

reboot your vps

scan your VPS again

clamscan -r -i /

and make sure the cron job has been removed

 ls -la /etc/cron.hourly/

Now that the trojan has been removed. You can request to remove the ban on your VPS network.

spidoyman · Answer

Hello,please check my ticket :) ==> Ticket #1474222

ooo34rus · Answer

Hello.I checking and deleting the vulnerable files and disabled vesta servise.Please, turn networking back on my droplets.My tickets:1461992145544714625001462122146184514618821462015144556814621251456021146212014621231456046

Daniel Toth · Answer

Hi, my droplet was affected by the VestaCP vulnerability. I host several sites on this droplet. I requested to boot it in the recovery tool to download data from it. I did it immediately and completely rebuilt the droplet with a new Debian image. I requested to boot the droplet back to it’s disk, but I get no answer… So now it is almost 5 days since my droplet is offline and DigitalOcean do not answer on tickets. I need the same IP address, because many domains are redirected to this IP, which I can not change. So rebuilding the droplet is the only option for me.

I am using DigitalOcean for more than 4 years now, but this failure of support is shocking. Please respond something on my ticket: #1448429

Related

Question

How do I determine the impact of VestaCP vulnerability from April 8th, 2018?

Related

Try DigitalOcean for free