Related

how to configure exim in centos 7 Cpanel/Whm ? Question Question
How did my database got hacked ? Question Question

Question

How do I determine the impact of VestaCP vulnerability from April 8th, 2018?

Posted April 8, 2018 76.1k views
SecurityControl PanelsApplications

Update: As of April 12, 2018 we have removed the block on port 8083. We will continue to observe the result of this change and may reinstate the block if we see an uptick in abusive traffic. We will continue to post updates here, as necessary.

Details about the exploit

Due to a vulnerability within the VestaCP software, an exploit is being used to gain root access to Droplets running this software. Exploited Droplets are then being used to perform a DoS attack to remote servers by sending large amounts of traffic.

VestaCP has also acknowledged the issue and is investigating the behavior further themselves: https://forum.vestacp.com/viewtopic.php?p=68594#p68594

What is being done about this?

DigitalOcean is working to mitigate traffic from Droplets that have already been compromised. We have also blocked all traffic to TCP port 8083, the default port used by VestaCP for API and login requests. Additionally, we have disabled networking on a subset of Droplets suspected of running this software in an attempt to avoid a potential compromise or to prevent abusive traffic from being sent from these Droplets.
Vesta has officially released an update that is meant to mitigate the vulnerability.

These packages are available via package manager updates or fresh installs. Though this update is meant to mitigate against new attacks, it will not help your system if it has already been compromised. It is advised that you create a new instance of Vesta, rather than updating old systems.

Disclaimer: These packages have not been tested by DigitalOcean to ensure the vulnerability is officially patched. Furthermore, we have not seen enough details of either the exploit or the patch to have a high degree of confidence in its ability to mitigate the vulnerability. This means that, even post patch, your Droplet could still be vulnerable. Please read on for additional information about ways to potentially reduce your risk of exploitation.

If you create a new instance of Vesta, we recommend that you change the default port away from 8083 (which is currently blocked by DigitalOcean). We have prepared a tutorial to assist users with installing VestaCP and migrating their data.

https://www.digitalocean.com/community/tutorials/how-to-install-vestacp-and-migrate-user-data

Our Support Team is working as quickly and efficiently as possible to respond to each ticket in a timely manner. As additional updates are available, we’ll modify this post. We appreciate your patience as we work to resolve outstanding issues.

How can I check if my Droplet was compromised?

  • Check all cron jobs on your machine for malicious activity. Specifically, check the contents of the /etc/cron.hourly/gcc.sh file. as we have seen this cron particularly affected.
  • Install an antivirus software, run a scan and then check for any results similar to /lib/libudev.so: Unix.Trojan.DDoS_XOR-1 FOUND

What steps can I take to prevent my Droplet from being compromised?

  • Consider stopping the Vesta service if you do not have an immediate need for it:
service vesta stop
systemctl stop vesta

  • Block public inbound connections to port 8083. If needed, you can still establish connections through the use of an SSH tunnel by following our tutorial How To Set Up SSH Tunneling on a VPS .

  • Use DigitalOcean’s Cloud Firewalls to block SSH traffic and VestaCP control panel traffic from all IPs other than the one IP you use to manage your Droplet.

  • Follow our Community article, which includes mitigation steps, to spin up a new Droplet.

Additional Measures

Though Vesta has updated their package, there is still a chance the system is vulnerable (there is always a chance that any system is vulnerable). In light of this vulnerability, we recommend taking the following general security precautions to avoid takeovers of your Droplet(s). Compromised Droplets are typically used for cryptocurrency mining or Denial of Service (DoS) attacks, both of which will impact your ability to use your Droplet.

  • Change any default passwords on your installed software (eg, admin/admin), as these are commonly used in takeover scenarios
  • Enable your local firewall (eg, iptables) to prevent all inbound connections by default, and only allow connections on ports you’re specifically serving
  • Add monitoring software that notifies you when your resources, especially CPU or bandwidth, are extremely high
  • Ensure your password is sufficiently complex, preferably longer than ten characters
  • If possible, keep your software up-to-date by enabling auto-update features
  • Periodically scan through your user lists in /etc/passwd and /etc/shadow for unknown users
  • Ensure that users who don’t need interactive console access have their shell set to /usr/sbin/nologin or /bin/false in /etc/passwd -Consider running proactive security software like ossec or fail2ban
Add a comment
sbeckstrand
By:
sbeckstrand
edited by jarland

Related

how to configure exim in centos 7 Cpanel/Whm ? Question Question
How did my database got hacked ? Question Question
Add a comment
4 comments
  • tcaseng86 April 9, 2018
    Reply Report

    Hi ,

    I have check and gone through all the necessary steps and can rest assured it is not compromise There is no such file gcc.sh was found in /etc/cron.hourly/gcc.sh and no such file libudev.so in /lib/ path.

    I have taken measure to change the vestacp login port from 8083 to 56000.

    For the moment i have stop the vesta service until it is patch and secure but i could not do so as there is no internet to from the droplet.

    Kindly do reply as soon as possible to my ticket #1446648

    Thank You.

  • ricardoe56cf7fd April 9, 2018
    Reply Report

    I’ve checked and my droplets were not compromised. I’ve patched Vesta, and changed it’s port, please check my Ticket #1447289.

    thanks in advance

  • lakshmithaneru April 9, 2018
    Reply Report

    Hi, I am facing the same issue. I can’t access my rest API using 8083 port couple of mobile apps are running using this API http://[IP Removed]:8083/ and my ticket is Ticket #1447852. Don’t know how to solve this and also would like to know is this issue with this particular port and how much time it will take to solve this issue.

    edited by wpa
  • sanmiguel April 9, 2018
    Reply Report

    Hi guys!
    Did you resolved your issues? My tickets still waiting for answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
73 answers
anthonyrossbach April 9, 2018

After updating server via SSH how to change VestaCP port and add firewall rules

I run a web host that uses VestaCP as a backbone API for managing resources only, but everything has been patched and wanted to share a script to update your VestaCP port from 8083 to 5600. But only do this AFTER update and patch. It will create the firewall rule on VestaCP also.

curl http://www.nodehost.ca/scripts/sh/vestacp_changeport.sh > vestacp_changeport.sh && bash vestacp_changeport.sh

Feel free to download and see what the file actually does, and if you want to upload a file and run local here is the script in the file.

echo "NodeHost Custom VESTACP Script"

echo "JOB: Changing VESTACP port"
string="listen          8083;"
stringnew="listen          5600;"
grep "$stringnew" /usr/local/vesta/nginx/conf/nginx.conf || sed -i "s/$string/$stringnew/g" /usr/local/vesta/nginx/conf/nginx.conf
echo "JOB: Complete"

echo "JOB: Changing VESTACP firewall rule for new port"
v-add-firewall-rule ACCEPT 0.0.0.0/0 5600 TCP
echo "JOB: Complete"

echo "JOB: Restarting VESTACP"
service vesta restart
echo "JOB: Complete"

echo "JOB: Port has been changed to 5600 from 8083"

I hope this helps.

Reply Report
MaryRWebb April 9, 2018

I find some of the comments here appalling. I had problems with the network on my servers almost a day and then I found out the result was that a bunch of other people on the network had chosen to install this vestacp thing that was vulnerable and caused their servers to attack the network. Digital ocean shut them down as they should have, only for them to come here and talk trash and act like digital ocean is responsible for their own stupid choices.

thanks digital ocean. don’t listen to the haters.

Reply Report
  • anthonyrossbach April 9, 2018

    Yes, I use VestaCP as a API only and did find that what DO did was required, none of my systems where affected or had problems only because no direct sharing of addresses.

    Acting fast is what DigitalOcean is good at, and that is a great thing.

    Reply Report
  • Shadowhaxor April 9, 2018

    Sorry if I’m being rude, but I doubt you understand the situtation here. The fact that you said “This Vesta CP thing” is troublesome. As someone who works at a very large MSP, a few things here bother me.

    1. Droplets that have VestaCP had their networking disabled. Regardless if their Vesta CP instance was compromised or not. There are a number of people here who had already hardened their instance by changing the Vesta CP port, added firewall rules or both.

    2. Zero proof has been proved that a droplet was compomised. I can understand the panic and doing the shutdowns, but if we can show that the was no compromise, our droplets should have connectivity re-established. So far, that hasn’t happened.

    It has not thing to with hating and everything to do with protocol and correct handling. Even now, there is a patch to correct the issue, but without network connectivity, it can’t be applied. All that needed to be done was to force a firewall rule to deny port 8083 and any outbound traffic until proof was provided that the patch was applied / and that the droplet wasn’t compromised.

    Reply Report
    • jarland April 9, 2018

      Hey Shadowhaxor,

      To clarify any misunderstanding, the droplets that were disabled were found to have VestaCP listening on port 8083 and there was not a software patch available at this time. If an instance had been hardened in the way you’ve mentioned, it would not have been included in our process. Any droplet running VestaCP that was included in our process and was not compromised would have been compromised within a few hours at most, and this was proven to us throughout the day as we observed the patterns prior to taking this action.

      Kind Regards,
      Jarland
      Platform Support Lead

      Reply Report
      • bangbible April 9, 2018

        One datacenter was blocked. We had vestacp in every data center. You disabled one server in a NYC datacenter that was no compromised nor would it have been.

        You caused serious harm to real businesses by doing so.

        There have been many reports of servers running vesta on other ports then 8083 being compromised. Again you are taking action and harming businesses on a hunch instead of removing compromised servers.

        Reply Report
      • jasonengage April 11, 2018

        I really appreciate your response in disabling the ports! Super critical action-taking. A++

        Reply Report
    • jasonengage April 11, 2018

      I have 3 droplets running vesta. One was compromised. Has to rebuild it. Luckily DO disabled the 8083 port to prevent my other droplets from being infected. Thank you DO.

      I updated my vesta and changed the port on my other 2 servers. Works fine now. Can access no problem.

      Reply Report
tomapryor April 9, 2018

Hi DO

Please, please… can we have a reply to our ticket, it’s nearly been 24 hours and still not one response!?

Ticket #1443819

Thank you.

Reply Report
alexcastaneda April 9, 2018

Hello good day to all, really it is a very serious problem that is happening right now, many fallen sites, and many people losing traffic.

I had several fallen sites, and solved it in the following way:

1.- Make a backup of the database of each of the websites:

  • mysqldump -u userdb -p dbname - p dbname > namefilebackup.sql

2.- Install Vesta on a Centos server (create a new droplet with Centos7)

3.- Change the port of vesta, it could be to 2083.

  • cd /usr/local/vesta/nginx/conf/ && nano nginx.conf
  • change “listen 8083;” to “listen 2083;”
  • service vesta restart

4.- Update the rules of the firewall in vesta via console:

  • v-change-firewall-rule Accept 0.0.0.0/0 2083 TCP

5.- copy the files from the previous server to the current server:
rsync -avzhie “ssh -p 22” root@IP.DEL.SERVER.OFTERIOR: /mnt/home/admin/backup.tar.gz/home/admin/backups-new-server/

6.- Restore all files.

If you have any questions with the steps above, let me know, I will be very happy to help you… :) a big hug and successes!

Hola buen día a todos, realmente es un problema muy grave el que está pasando ahora mismo, muchos sitos caidos, y muchas personas perdiendo tráfico.

Yo tuve varios sitios caidos, y lo solucioné de la siguiente manera:

1.- Hacer un backup de la base de datos de cada uno de los sitios web:

  • mysqldump -u usuariodb -p nombrebd - p nombre_bd > archivo.sql

2.- Instalar Vesta en un servidor Centos (para ello cree una nueva gotita con Centos7)

3.- Cambiar el puerto de vesta, podría ser al 2083.

  • cd /usr/local/vesta/nginx/conf/ && nano nginx.conf
  • cambiar “ listen 8083; ” por “ listen 2083; ”
  • service vesta restart

4.- Actualizar las reglas del firewall en vesta via consola:

  • v-change-firewall-rule Accept 0.0.0.0/0 2083 TCP

5.- copiar los archivos desde el servidor anterior al servidor actual:
rsync -avzhie “ssh -p 22” root@IP.DEL.SERVER.ANTERIOR:/mnt/home/admin/backup.tar.gz /home/admin/backups-new-server/

6.- Restaurar todos los archivos.

Si tienes alguna duda con los pasos anteriores, avísame, estaré muy contento de ayudarte… :) un fuerte abrazo y éxitos!

Reply Report
  • jaimegdelrio April 10, 2018

    Hola Alex
    Tu sabes que puedo hacer si en uno de mis droplets, pensando que era un error, lo apagué y prendí, pero luego cuando quise entrar en la consola, me aparecía la pantalla de “rescue enviroment”. Es posible todavía recuperar mis archivos, base de datos… ?

    Agradezco tu ayuda.

    Reply Report
  • sanmiguel April 10, 2018

    Alex, thank you!

    I’m getting “no route to host message” this must be because I’m offline and this server only can be accessed from whitin DO virtual terminal, what do you suggest?

    King Regards

    Alex, gracias!

    Tengo el mensaje “no route to host” debe ser porque el servidor esta fuera de linea y solo puede accederse con la terminal virtual de DO, ¿que sugerencia me puedes dar?

    Muchas gracias.

    Reply Report
    • alexcastaneda April 10, 2018

      Hello and good day! … I think the best thing is to create another droplet in DO, request that they activate the network in your current droplet, and then recover and copy all the files to your new droplet. if you want I can help you via teamviewer. send me a message to: aquino.alex25 (@) gmail . com

      Reply Report
narinderbansalmonk April 11, 2018

Ok, Below is my case

My Server ports were closed, but i was able to login and use SSH.
So i updated vesta cp and changed ports.
Also activated firewall from digital ocean account.
Checked for Cron jobs didn’t find anything. Found Nothing
Checked /etc/passwd file, make all changes of shell access(Only checked because everything was already in place).
Also monitored
ls /etc/cron.hourly/
ls /lib/
ls /etc/rc.*
ls /etc/systemd/*
Didn’t find anything.

Server is working fine for now but one question is left.
Are these steps are enough?

Reply Report
  • cach April 11, 2018

    Same here. Updated, changed port, added http authentification on top, changed passwords, checked for files, crons, processes,etc, added firewall to droplet, I already had fail2ban set to ban ips for 1 year on 3 bad login attempts.
    The server is running smoothly. Hope no one at DO decides to take it down for no good reason.

    Reply Report
JosephShenton April 8, 2018

This is simply unacceptable. I understand you have to make sure everyone’s Droplets are safe however I had changed the default port and made sure users require a login via .htaccess BEFORE ever reaching the login page of VestaCP. I was safe, however you shutdown my droplet TWO DAYS IN A ROW. This has made me lose $125 while the droplets were offline so far. You guys need to understand that I use VestaCP because I do not need cPanel, and I can’t put my droplet back online until you guys allow VestaCP again meaning this could be upwards of 1-4 Weeks! I’ll be losing money and having to pay refunds to over 10,000 people due to this. I am putting my droplet back online right now and if you take it down you will be getting an email to your legal department. Thank you.

Reply Report
  • eris April 8, 2018

    If your Droplet was ifdowned, this was because our security team determined the Droplet is running vestacp and is vulnerable to compromise (or has been already compromised and sending out DDOSes) – two things we explicitly checked for is if users are running this on a different port and if they’ve implemented http authentication.

    I re-enabled networking temporarily to check over your Droplet and honestly it does not look fully secured yet, so I was required to take it back down 😞. I recommend following up in the ticket we opened on your account so that we can help resolve this issue and get you back up and running.

    I’m sorry for the trouble this has caused. 😞

    Best,
    Eris

    Reply Report
    • JosephShenton April 8, 2018

      Ok, I can just use console to modify the droplet correct? If so I’ll double check that the htaccess is working.

      Also, it’s not your fault :)

      Reply Report
      • eris April 8, 2018

        Yep, you can use the console to access the Droplet right now. That said, this is a very complex vulnerability and the issue isn’t fully fixed by the vestacp team. Your best bet is going to be to stop vestacp temporarily until a patch is available, and manually remove any trojans that were installed.

        Let us know in the ticket once that’s done and we can re-enable networking for you.

        - Eris

        Reply Report
        • JosephShenton April 8, 2018

          Ok will do! Thanks Eris!

          Reply Report
          • JosephShenton April 8, 2018

            Hi, I am wondering what alternative to VESTACP I can use? I need to keep my subdomains.

            Reply Report
          • jarland April 8, 2018

            It’s hard to say on alternatives. Personally, I like cPanel and for my money I think it’s great to have a team of developers available 24/7 (or on call) to fix a major issue with their software because it is in their financial interest to do so.

            That, however, is me personally. I also use VestaCP though, and I love it. I recommend it all the time. Perhaps one answer is simply to see how they respond and patch it, and work with them rather than move to a new system.

            Reply Report
          • JosephShenton April 9, 2018

            I’ve changed the port for VestaCP and I’ve fixed the HTTP Authentication. Please can you check again? I’ve also updated VestaCP which has the patch released 3 hours ago. Well, i tried to. Just realised networking being disabled means I cannot update it. I also removed the trojan as I had been infected.

            Reply Report
          • eris April 9, 2018

            Should be all good, @JosephShenton. Next time please use our support system – easier to track that way.

            Reply Report
    • bangbible April 9, 2018

      There is no proof this is tied to all vesta installs or even vesta at this point. You are harming real businesses here just to save you from doing work and blocking the affected servers

      Reply Report
digitaloceand6fe3884bd3694 April 8, 2018

“Additionally, we have disabled networking on all Droplets suspected of running this software in an attempt to avoid a potential compromise or to prevent abusive traffic from being sent from these Droplets”

“How can I check if my Droplet was compromised? Check all cron jobs on your machine for malicious activity. ”

How can I do that with networking blocked? I have a root login disabled with ssh key access only!

Reply Report
  • eris April 8, 2018

    Root login will still work via password on the Console – it’s not SSH.

    If that does not work, I recommend reaching out to our support team for help. We’re working on a process to help get this resolved for everyone.

    - Eris

    Reply Report
    • digitaloceand6fe3884bd3694 April 8, 2018

      Why is SSH blocked when this is a problem with TCP/8083?

      Reply Report
      • eris April 8, 2018

        We don’t have access to the internals of your Droplet. This means, until your Droplet actually sends out a DDOS, we can’t tell if it’s compromised or not — only that it’s vulnerable.

        Once someone gains access thru that vulnerability, it doesn’t go away because we blocked the port they used to gain access. If you’ve been running vestacp, your Droplet could have been compromised days ago and you won’t know until you check manually. Even with the port blocked, a DDOS can still be triggered using a Trojan installed on the Droplet. Normally this would be fine and we can react as it happens. However, this was happening so much in some of our datacenter that we needed to take preventative measures.

        For this reason, we need to disable networking completely, and not just block the entry port.

        - Eris

        Reply Report
  • jarland April 8, 2018

    Blocking root login on SSH won’t block it on the web console, as this is like attaching a mouse, keyboard, and monitor directly to the system.

    Reply Report
CodeP April 8, 2018

Please check the support ticket. Need to get reply asap.

Reply Report
  • jarland April 8, 2018

    There will be many people ticketing in about this as this control panel appears to have reasonable popularity. I will definitely do my best to see to a speedy response, but want to prepare you for the reality of what our support team will be dealing with on this issue. What is your ticket number?

    To speed up the process for self resolution, we have now mounted the recovery ISO to each droplet that was found to be running Vesta and had networking disabled as a result. Powering the droplet off and back on will boot into that. Note that backing up MySQL databases will be much easier prior to booting the ISO.

    Reply Report
thetigcoder April 8, 2018

Only one of my 4 droplets is down for this problem, i want restore online, how have to do? I don’t have time, my customer in impatient.

Reply Report
  • jarland April 8, 2018

    Understood and sorry to hear about the impact this is having on your and your customers. If you do not have time to focus on the issue yourself, you may be able to work with the support for VestaCP:

    http://vestacp.com/support

    Otherwise if you have any Linux admin friends that you trust to assist you on this, that may help you recover more quickly as well.

    Reply Report
nlcaldwell April 8, 2018

Taking down droplets that have not been compromised but which are running VestaCP is an absurd over-reaction on DigitalOcean’s part. Is DigitalOcean allowed under the Terms of Service to disable networking when a Droplet is not compromised?

Reply Report
  • digitaloceand6fe3884bd3694 April 8, 2018

    I too would like a definite answer on this.

    Reply Report
  • thetigcoder April 8, 2018

    I quote you, i want know if that is under the terms of service, because also for me one droplet were down but i checked already now and non is infected (/etc/cron.hourly/gcc.sh not found).

    Reply Report
  • jarland April 8, 2018

    If you were fortunate to not be compromised prior to this reaction, and have taken steps to prevent it from happening, we can definitely get that back online for you. Please note that if we disabled your droplet it was vulnerable and would have been compromised within a number of hours at most, as VestaCP was actively responding to requests on port 8083 and, at the time, had not been patched.

    If you have a ticket number to share, I will escalate that.

    Reply Report
    • nlcaldwell April 8, 2018

      No where did you answer my question: does DigitalOcean’s actions in disabling networking for uncompromised servers violate the Terms of Service? I have already resolved the issue on my end, and since this vulnerability has been known for a couple days now, the notion that it was only hours away from being compromised is playing on ignorance.

      I can only assume that since you did not answer my question, what DigitalOcean did violates the Terms of Service. These actions are egregious and absurd.

      Reply Report
      • bangbible April 9, 2018

        They wont answer this because it will leave them open to lawsuits.

        I am agree its insane. There is no proof that any server was in danger.

        Reply Report
Previous 1 2 3 4 5 6 7 8 Next
Load More Answers

Related

Looking for something else?

Ask a new question Search for more help