Question

How do I determine the impact of VestaCP vulnerability from April 8th, 2018?

Posted April 8, 2018 60.2k views
SecurityControl PanelsApplications

Update: As of April 12, 2018 we have removed the block on port 8083. We will continue to observe the result of this change and may reinstate the block if we see an uptick in abusive traffic. We will continue to post updates here, as necessary.

Details about the exploit

Due to a vulnerability within the VestaCP software, an exploit is being used to gain root access to Droplets running this software. Exploited Droplets are then being used to perform a DoS attack to remote servers by sending large amounts of traffic.

VestaCP has also acknowledged the issue and is investigating the behavior further themselves: https://forum.vestacp.com/viewtopic.php?p=68594#p68594

What is being done about this?

DigitalOcean is working to mitigate traffic from Droplets that have already been compromised. We have also blocked all traffic to TCP port 8083, the default port used by VestaCP for API and login requests. Additionally, we have disabled networking on a subset of Droplets suspected of running this software in an attempt to avoid a potential compromise or to prevent abusive traffic from being sent from these Droplets.
Vesta has officially released an update that is meant to mitigate the vulnerability.

These packages are available via package manager updates or fresh installs. Though this update is meant to mitigate against new attacks, it will not help your system if it has already been compromised. It is advised that you create a new instance of Vesta, rather than updating old systems.

Disclaimer: These packages have not been tested by DigitalOcean to ensure the vulnerability is officially patched. Furthermore, we have not seen enough details of either the exploit or the patch to have a high degree of confidence in its ability to mitigate the vulnerability. This means that, even post patch, your Droplet could still be vulnerable. Please read on for additional information about ways to potentially reduce your risk of exploitation.

If you create a new instance of Vesta, we recommend that you change the default port away from 8083 (which is currently blocked by DigitalOcean). We have prepared a tutorial to assist users with installing VestaCP and migrating their data.

https://www.digitalocean.com/community/tutorials/how-to-install-vestacp-and-migrate-user-data

Our Support Team is working as quickly and efficiently as possible to respond to each ticket in a timely manner. As additional updates are available, we’ll modify this post. We appreciate your patience as we work to resolve outstanding issues.

How can I check if my Droplet was compromised?

  • Check all cron jobs on your machine for malicious activity. Specifically, check the contents of the /etc/cron.hourly/gcc.sh file. as we have seen this cron particularly affected.
  • Install an antivirus software, run a scan and then check for any results similar to /lib/libudev.so: Unix.Trojan.DDoS_XOR-1 FOUND

What steps can I take to prevent my Droplet from being compromised?

  • Consider stopping the Vesta service if you do not have an immediate need for it:
service vesta stop
systemctl stop vesta

  • Block public inbound connections to port 8083. If needed, you can still establish connections through the use of an SSH tunnel by following our tutorial How To Set Up SSH Tunneling on a VPS .

  • Use DigitalOcean’s Cloud Firewalls to block SSH traffic and VestaCP control panel traffic from all IPs other than the one IP you use to manage your Droplet.

  • Follow our Community article, which includes mitigation steps, to spin up a new Droplet.

Additional Measures

Though Vesta has updated their package, there is still a chance the system is vulnerable (there is always a chance that any system is vulnerable). In light of this vulnerability, we recommend taking the following general security precautions to avoid takeovers of your Droplet(s). Compromised Droplets are typically used for cryptocurrency mining or Denial of Service (DoS) attacks, both of which will impact your ability to use your Droplet.

  • Change any default passwords on your installed software (eg, admin/admin), as these are commonly used in takeover scenarios
  • Enable your local firewall (eg, iptables) to prevent all inbound connections by default, and only allow connections on ports you’re specifically serving
  • Add monitoring software that notifies you when your resources, especially CPU or bandwidth, are extremely high
  • Ensure your password is sufficiently complex, preferably longer than ten characters
  • If possible, keep your software up-to-date by enabling auto-update features
  • Periodically scan through your user lists in /etc/passwd and /etc/shadow for unknown users
  • Ensure that users who don’t need interactive console access have their shell set to /usr/sbin/nologin or /bin/false in /etc/passwd -Consider running proactive security software like ossec or fail2ban
Add a comment
sbeckstrand
By:
sbeckstrand
edited by jarland
Add a comment
4 comments
  • tcaseng86 April 9, 2018
    Reply Report

    Hi ,

    I have check and gone through all the necessary steps and can rest assured it is not compromise There is no such file gcc.sh was found in /etc/cron.hourly/gcc.sh and no such file libudev.so in /lib/ path.

    I have taken measure to change the vestacp login port from 8083 to 56000.

    For the moment i have stop the vesta service until it is patch and secure but i could not do so as there is no internet to from the droplet.

    Kindly do reply as soon as possible to my ticket #1446648

    Thank You.

  • ricardoe56cf7fd April 9, 2018
    Reply Report

    I’ve checked and my droplets were not compromised. I’ve patched Vesta, and changed it’s port, please check my Ticket #1447289.

    thanks in advance

  • lakshmithaneru April 9, 2018
    Reply Report

    Hi, I am facing the same issue. I can’t access my rest API using 8083 port couple of mobile apps are running using this API http://[IP Removed]:8083/ and my ticket is Ticket #1447852. Don’t know how to solve this and also would like to know is this issue with this particular port and how much time it will take to solve this issue.

    edited by wpa
  • sanmiguel April 9, 2018
    Reply Report

    Hi guys!
    Did you resolved your issues? My tickets still waiting for answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
73 answers
anthonyrossbach April 9, 2018

After updating server via SSH how to change VestaCP port and add firewall rules

I run a web host that uses VestaCP as a backbone API for managing resources only, but everything has been patched and wanted to share a script to update your VestaCP port from 8083 to 5600. But only do this AFTER update and patch. It will create the firewall rule on VestaCP also.

curl http://www.nodehost.ca/scripts/sh/vestacp_changeport.sh > vestacp_changeport.sh && bash vestacp_changeport.sh

Feel free to download and see what the file actually does, and if you want to upload a file and run local here is the script in the file.

echo "NodeHost Custom VESTACP Script"

echo "JOB: Changing VESTACP port"
string="listen          8083;"
stringnew="listen          5600;"
grep "$stringnew" /usr/local/vesta/nginx/conf/nginx.conf || sed -i "s/$string/$stringnew/g" /usr/local/vesta/nginx/conf/nginx.conf
echo "JOB: Complete"

echo "JOB: Changing VESTACP firewall rule for new port"
v-add-firewall-rule ACCEPT 0.0.0.0/0 5600 TCP
echo "JOB: Complete"

echo "JOB: Restarting VESTACP"
service vesta restart
echo "JOB: Complete"

echo "JOB: Port has been changed to 5600 from 8083"

I hope this helps.

Reply Report
MaryRWebb April 9, 2018

I find some of the comments here appalling. I had problems with the network on my servers almost a day and then I found out the result was that a bunch of other people on the network had chosen to install this vestacp thing that was vulnerable and caused their servers to attack the network. Digital ocean shut them down as they should have, only for them to come here and talk trash and act like digital ocean is responsible for their own stupid choices.

thanks digital ocean. don’t listen to the haters.

Reply Report
  • anthonyrossbach April 9, 2018

    Yes, I use VestaCP as a API only and did find that what DO did was required, none of my systems where affected or had problems only because no direct sharing of addresses.

    Acting fast is what DigitalOcean is good at, and that is a great thing.

    Reply Report
  • Shadowhaxor April 9, 2018

    Sorry if I’m being rude, but I doubt you understand the situtation here. The fact that you said “This Vesta CP thing” is troublesome. As someone who works at a very large MSP, a few things here bother me.

    1. Droplets that have VestaCP had their networking disabled. Regardless if their Vesta CP instance was compromised or not. There are a number of people here who had already hardened their instance by changing the Vesta CP port, added firewall rules or both.

    2. Zero proof has been proved that a droplet was compomised. I can understand the panic and doing the shutdowns, but if we can show that the was no compromise, our droplets should have connectivity re-established. So far, that hasn’t happened.

    It has not thing to with hating and everything to do with protocol and correct handling. Even now, there is a patch to correct the issue, but without network connectivity, it can’t be applied. All that needed to be done was to force a firewall rule to deny port 8083 and any outbound traffic until proof was provided that the patch was applied / and that the droplet wasn’t compromised.

    Reply Report
    • jarland April 9, 2018

      Hey Shadowhaxor,

      To clarify any misunderstanding, the droplets that were disabled were found to have VestaCP listening on port 8083 and there was not a software patch available at this time. If an instance had been hardened in the way you’ve mentioned, it would not have been included in our process. Any droplet running VestaCP that was included in our process and was not compromised would have been compromised within a few hours at most, and this was proven to us throughout the day as we observed the patterns prior to taking this action.

      Kind Regards,
      Jarland
      Platform Support Lead

      Reply Report
      • bangbible April 9, 2018

        One datacenter was blocked. We had vestacp in every data center. You disabled one server in a NYC datacenter that was no compromised nor would it have been.

        You caused serious harm to real businesses by doing so.

        There have been many reports of servers running vesta on other ports then 8083 being compromised. Again you are taking action and harming businesses on a hunch instead of removing compromised servers.

        Reply Report
      • jasonengage April 11, 2018

        I really appreciate your response in disabling the ports! Super critical action-taking. A++

        Reply Report
    • jasonengage April 11, 2018

      I have 3 droplets running vesta. One was compromised. Has to rebuild it. Luckily DO disabled the 8083 port to prevent my other droplets from being infected. Thank you DO.

      I updated my vesta and changed the port on my other 2 servers. Works fine now. Can access no problem.

      Reply Report
tomapryor April 9, 2018

Hi DO

Please, please… can we have a reply to our ticket, it’s nearly been 24 hours and still not one response!?

Ticket #1443819

Thank you.

Reply Report
alexcastaneda April 9, 2018

Hello good day to all, really it is a very serious problem that is happening right now, many fallen sites, and many people losing traffic.

I had several fallen sites, and solved it in the following way:

1.- Make a backup of the database of each of the websites:

  • mysqldump -u userdb -p dbname - p dbname > namefilebackup.sql

2.- Install Vesta on a Centos server (create a new droplet with Centos7)

3.- Change the port of vesta, it could be to 2083.

  • cd /usr/local/vesta/nginx/conf/ && nano nginx.conf
  • change “listen 8083;” to “listen 2083;”
  • service vesta restart

4.- Update the rules of the firewall in vesta via console:

  • v-change-firewall-rule Accept 0.0.0.0/0 2083 TCP

5.- copy the files from the previous server to the current server:
rsync -avzhie “ssh -p 22” root@IP.DEL.SERVER.OFTERIOR: /mnt/home/admin/backup.tar.gz/home/admin/backups-new-server/

6.- Restore all files.

If you have any questions with the steps above, let me know, I will be very happy to help you… :) a big hug and successes!

Hola buen día a todos, realmente es un problema muy grave el que está pasando ahora mismo, muchos sitos caidos, y muchas personas perdiendo tráfico.

Yo tuve varios sitios caidos, y lo solucioné de la siguiente manera:

1.- Hacer un backup de la base de datos de cada uno de los sitios web:

  • mysqldump -u usuariodb -p nombrebd - p nombre_bd > archivo.sql

2.- Instalar Vesta en un servidor Centos (para ello cree una nueva gotita con Centos7)

3.- Cambiar el puerto de vesta, podría ser al 2083.

  • cd /usr/local/vesta/nginx/conf/ && nano nginx.conf
  • cambiar “ listen 8083; ” por “ listen 2083; ”
  • service vesta restart

4.- Actualizar las reglas del firewall en vesta via consola:

  • v-change-firewall-rule Accept 0.0.0.0/0 2083 TCP

5.- copiar los archivos desde el servidor anterior al servidor actual:
rsync -avzhie “ssh -p 22” root@IP.DEL.SERVER.ANTERIOR:/mnt/home/admin/backup.tar.gz /home/admin/backups-new-server/

6.- Restaurar todos los archivos.

Si tienes alguna duda con los pasos anteriores, avísame, estaré muy contento de ayudarte… :) un fuerte abrazo y éxitos!

Reply Report
  • jaimegdelrio April 10, 2018

    Hola Alex
    Tu sabes que puedo hacer si en uno de mis droplets, pensando que era un error, lo apagué y prendí, pero luego cuando quise entrar en la consola, me aparecía la pantalla de “rescue enviroment”. Es posible todavía recuperar mis archivos, base de datos… ?

    Agradezco tu ayuda.

    Reply Report
  • sanmiguel April 10, 2018

    Alex, thank you!

    I’m getting “no route to host message” this must be because I’m offline and this server only can be accessed from whitin DO virtual terminal, what do you suggest?

    King Regards

    Alex, gracias!

    Tengo el mensaje “no route to host” debe ser porque el servidor esta fuera de linea y solo puede accederse con la terminal virtual de DO, ¿que sugerencia me puedes dar?

    Muchas gracias.

    Reply Report
    • alexcastaneda April 10, 2018

      Hello and good day! … I think the best thing is to create another droplet in DO, request that they activate the network in your current droplet, and then recover and copy all the files to your new droplet. if you want I can help you via teamviewer. send me a message to: aquino.alex25 (@) gmail . com

      Reply Report
narinderbansalmonk April 11, 2018

Ok, Below is my case

My Server ports were closed, but i was able to login and use SSH.
So i updated vesta cp and changed ports.
Also activated firewall from digital ocean account.
Checked for Cron jobs didn’t find anything. Found Nothing
Checked /etc/passwd file, make all changes of shell access(Only checked because everything was already in place).
Also monitored
ls /etc/cron.hourly/
ls /lib/
ls /etc/rc.*
ls /etc/systemd/*
Didn’t find anything.

Server is working fine for now but one question is left.
Are these steps are enough?

Reply Report
  • cach April 11, 2018

    Same here. Updated, changed port, added http authentification on top, changed passwords, checked for files, crons, processes,etc, added firewall to droplet, I already had fail2ban set to ban ips for 1 year on 3 bad login attempts.
    The server is running smoothly. Hope no one at DO decides to take it down for no good reason.

    Reply Report
JosephShenton April 8, 2018

This is simply unacceptable. I understand you have to make sure everyone’s Droplets are safe however I had changed the default port and made sure users require a login via .htaccess BEFORE ever reaching the login page of VestaCP. I was safe, however you shutdown my droplet TWO DAYS IN A ROW. This has made me lose $125 while the droplets were offline so far. You guys need to understand that I use VestaCP because I do not need cPanel, and I can’t put my droplet back online until you guys allow VestaCP again meaning this could be upwards of 1-4 Weeks! I’ll be losing money and having to pay refunds to over 10,000 people due to this. I am putting my droplet back online right now and if you take it down you will be getting an email to your legal department. Thank you.

Reply Report
  • eris April 8, 2018

    If your Droplet was ifdowned, this was because our security team determined the Droplet is running vestacp and is vulnerable to compromise (or has been already compromised and sending out DDOSes) – two things we explicitly checked for is if users are running this on a different port and if they’ve implemented http authentication.

    I re-enabled networking temporarily to check over your Droplet and honestly it does not look fully secured yet, so I was required to take it back down 😞. I recommend following up in the ticket we opened on your account so that we can help resolve this issue and get you back up and running.

    I’m sorry for the trouble this has caused. 😞

    Best,
    Eris

    Reply Report
    • JosephShenton April 8, 2018

      Ok, I can just use console to modify the droplet correct? If so I’ll double check that the htaccess is working.

      Also, it’s not your fault :)

      Reply Report
      • eris April 8, 2018

        Yep, you can use the console to access the Droplet right now. That said, this is a very complex vulnerability and the issue isn’t fully fixed by the vestacp team. Your best bet is going to be to stop vestacp temporarily until a patch is available, and manually remove any trojans that were installed.

        Let us know in the ticket once that’s done and we can re-enable networking for you.

        - Eris

        Reply Report
        • JosephShenton April 8, 2018

          Ok will do! Thanks Eris!

          Reply Report
          • JosephShenton April 8, 2018

            Hi, I am wondering what alternative to VESTACP I can use? I need to keep my subdomains.

            Reply Report
          • jarland April 8, 2018

            It’s hard to say on alternatives. Personally, I like cPanel and for my money I think it’s great to have a team of developers available 24/7 (or on call) to fix a major issue with their software because it is in their financial interest to do so.

            That, however, is me personally. I also use VestaCP though, and I love it. I recommend it all the time. Perhaps one answer is simply to see how they respond and patch it, and work with them rather than move to a new system.

            Reply Report
          • JosephShenton April 9, 2018

            I’ve changed the port for VestaCP and I’ve fixed the HTTP Authentication. Please can you check again? I’ve also updated VestaCP which has the patch released 3 hours ago. Well, i tried to. Just realised networking being disabled means I cannot update it. I also removed the trojan as I had been infected.

            Reply Report
          • eris April 9, 2018

            Should be all good, @JosephShenton. Next time please use our support system – easier to track that way.

            Reply Report
    • bangbible April 9, 2018

      There is no proof this is tied to all vesta installs or even vesta at this point. You are harming real businesses here just to save you from doing work and blocking the affected servers

      Reply Report
digitaloceand6fe3884bd3694 April 8, 2018

“Additionally, we have disabled networking on all Droplets suspected of running this software in an attempt to avoid a potential compromise or to prevent abusive traffic from being sent from these Droplets”

“How can I check if my Droplet was compromised? Check all cron jobs on your machine for malicious activity. ”

How can I do that with networking blocked? I have a root login disabled with ssh key access only!

Reply Report
  • eris April 8, 2018

    Root login will still work via password on the Console – it’s not SSH.

    If that does not work, I recommend reaching out to our support team for help. We’re working on a process to help get this resolved for everyone.

    - Eris

    Reply Report
    • digitaloceand6fe3884bd3694 April 8, 2018

      Why is SSH blocked when this is a problem with TCP/8083?

      Reply Report
      • eris April 8, 2018

        We don’t have access to the internals of your Droplet. This means, until your Droplet actually sends out a DDOS, we can’t tell if it’s compromised or not — only that it’s vulnerable.

        Once someone gains access thru that vulnerability, it doesn’t go away because we blocked the port they used to gain access. If you’ve been running vestacp, your Droplet could have been compromised days ago and you won’t know until you check manually. Even with the port blocked, a DDOS can still be triggered using a Trojan installed on the Droplet. Normally this would be fine and we can react as it happens. However, this was happening so much in some of our datacenter that we needed to take preventative measures.

        For this reason, we need to disable networking completely, and not just block the entry port.

        - Eris

        Reply Report
  • jarland April 8, 2018

    Blocking root login on SSH won’t block it on the web console, as this is like attaching a mouse, keyboard, and monitor directly to the system.

    Reply Report
CodeP April 8, 2018

Please check the support ticket. Need to get reply asap.

Reply Report
  • jarland April 8, 2018

    There will be many people ticketing in about this as this control panel appears to have reasonable popularity. I will definitely do my best to see to a speedy response, but want to prepare you for the reality of what our support team will be dealing with on this issue. What is your ticket number?

    To speed up the process for self resolution, we have now mounted the recovery ISO to each droplet that was found to be running Vesta and had networking disabled as a result. Powering the droplet off and back on will boot into that. Note that backing up MySQL databases will be much easier prior to booting the ISO.

    Reply Report
thetigcoder April 8, 2018

Only one of my 4 droplets is down for this problem, i want restore online, how have to do? I don’t have time, my customer in impatient.

Reply Report
  • jarland April 8, 2018

    Understood and sorry to hear about the impact this is having on your and your customers. If you do not have time to focus on the issue yourself, you may be able to work with the support for VestaCP:

    http://vestacp.com/support

    Otherwise if you have any Linux admin friends that you trust to assist you on this, that may help you recover more quickly as well.

    Reply Report
nlcaldwell April 8, 2018

Taking down droplets that have not been compromised but which are running VestaCP is an absurd over-reaction on DigitalOcean’s part. Is DigitalOcean allowed under the Terms of Service to disable networking when a Droplet is not compromised?

Reply Report
  • digitaloceand6fe3884bd3694 April 8, 2018

    I too would like a definite answer on this.

    Reply Report
  • thetigcoder April 8, 2018

    I quote you, i want know if that is under the terms of service, because also for me one droplet were down but i checked already now and non is infected (/etc/cron.hourly/gcc.sh not found).

    Reply Report
  • jarland April 8, 2018

    If you were fortunate to not be compromised prior to this reaction, and have taken steps to prevent it from happening, we can definitely get that back online for you. Please note that if we disabled your droplet it was vulnerable and would have been compromised within a number of hours at most, as VestaCP was actively responding to requests on port 8083 and, at the time, had not been patched.

    If you have a ticket number to share, I will escalate that.

    Reply Report
    • nlcaldwell April 8, 2018

      No where did you answer my question: does DigitalOcean’s actions in disabling networking for uncompromised servers violate the Terms of Service? I have already resolved the issue on my end, and since this vulnerability has been known for a couple days now, the notion that it was only hours away from being compromised is playing on ignorance.

      I can only assume that since you did not answer my question, what DigitalOcean did violates the Terms of Service. These actions are egregious and absurd.

      Reply Report
      • bangbible April 9, 2018

        They wont answer this because it will leave them open to lawsuits.

        I am agree its insane. There is no proof that any server was in danger.

        Reply Report
user45456 April 8, 2018

We just need to make backup of database! How to do it ASAP!!

Reply Report
user45456 April 8, 2018

Reply to my ticket (Ticket #1446190 )

Reply Report
azartcitycom April 8, 2018

Hi there, how can i enable networking for my droplet ? even if i want to change vestacp to another software i need to access my droplet.
I have no any ability to show my users any message :/
I scan my droplets(as you write ) and there was no any vulnerability.
I also open ticket but still no answer .

Reply Report
Tyrion April 8, 2018

Hello, can you please turn networking back on for my droplet. VestaCP has launched an update I want to install that should patch this vulnerability for unaffected servers like mine. I also put in a droplet firewall to block all non acceptable ports. I have also checked all the cron folders and crontab nothing matched the vulnerabilities and that other lib file doesn’t exist. I’ve had a Ticket for several hours yet no response as of yet(#1446345). I spun up a new droplet as per instructions but had no way of accessing the backups since networking was taken down. I then shutdown my droplet to copy my backups and then I’m now trying to get my droplet back to running(It is stuck in recovery mode). I also am not happy to have to pay for not only these outages but the time wasted with the extra droplet I spun up without any way to setup a new instance due to no backups(I’m turning on Digital Ocean Droplet backups going forward). I also would like to get my site back online promptly. Thank You. I need to know whether you can help me quickly or whether I need to rebuild my server and wait for DNS to propagate.

Reply Report
  • Tyrion April 9, 2018

    Got an email I just need to reboot my droplet WRONG, I have rebooted my droplet 8 times now it is stuck in the recovery image and as the image states I need to contact support(which I have done) to get it back to its disk. still no response from my last message. my company website has been offline for 5 hours now please fix this soon and get my server out of recovery mode.

    Reply Report
  • jarland April 9, 2018

    Hey Tyrion,

    I hope you were able to get what you needed to move forward on this. I’m happy to offer further assistance. I see that the last reply to your ticket was from us, so we’ll wait to hear back from you :)

    Kind Regards,
    Jarland
    Platform Support Lead

    Reply Report
agenciaglobales April 8, 2018

I’m going to create a new drop, can I do this with vestacp?
Correction pacode is already available.

Reply Report
SS88 April 8, 2018

Hello.

I’m part of a team and one Droplet is currently offline despite NOT being compromised and already PATCHED.

I opened a ticket (1446281) HOURS ago and it has no replies.

Reply Report
  • jarland April 9, 2018

    Hey SS88,

    I hope you were able to get what you needed to move forward here. It looks like we have worked this out with you, but please let me know if this appears any different from your side at this time.

    Kind Regards,
    Jarland
    Platform Support Lead

    Reply Report
dougfda9158d1d815899733026 April 8, 2018

Can you please respond to my Ticket #1446492:? I have created a new droplet with new IP and updated the networking for all domains (2+ hours ago).

Reply Report
  • dougfda9158d1d815899733026 April 9, 2018

    I have applied the Update patch from VestaCP on this new Droplet.
    0.9.8-20 (security)

    Reply Report
  • jarland April 9, 2018

    Hey there,

    I sent a reply over. Please reply to the ticket if you have any further questions. While there are several other customers working with us on this issue, we will do our best to provide a speedy response.

    Kind Regards,
    Jarland
    Platform Support Lead

    Reply Report
elninja April 9, 2018

Related to this, I have two droplets, I have disabled VestaCP and I’m waiting on you to restart the affected droplets with networking and the previous kernel.

Ticket #1446891

If you’re wondering how to disable VestaCP, install the rcconf and scroll down until you find vesta and disable it.

apt install rcconf -y
rcconf # opens a cli app

Check for any suspicious cron files

including the ones stored in /etc/crontab
and by listing the root jobs crontab -l root

Reply Report
  • elninja April 9, 2018

    It’s been more than a day since I disabled the vesta service!

    Please reboot back normally so I can backup the MySQL database!

    I have many sites down with the email, dns and web servers down!

    Reply Report
tcaseng86 April 9, 2018

Hi there,

I have check and gone through all the necessary steps and can rest assured it is not compromise. There is no such file gcc.sh was found in /etc/cron.hourly/gcc.sh and no such file libudev.so in /lib/ path.

I have taken measure to change the vestacp login port from 8083 to 56000.

All the above mention can check in my support ticket as there are screenshot proof for it.

For the moment i have stop the vesta service until it is patch and secure but i could not do so as there is no internet to from the droplet.

There is already a patch release which can be found here:

[https://lowendtalk.com/discussion/141728/vestacp-possibly-hit-with-zeroday-exploit-patch-released](http://)

but i could not do so as mention no internet connection from the droplet.

Please reply to my ticket [Ticket #1446648] as soon as possible.

Thank You

Reply Report
  • jarland April 9, 2018

    Hey there,

    Looks like we replied so you should be all set. I hope that you have everything you need to move forward, please let us know if not!

    Kind Regards,
    Jarland
    Platform Support Lead

    Reply Report
secretsaiyan April 9, 2018

You guys mentioned that TCP/8083 got blocked for the SFO2 region.

Today I was unable to login to Vesta CP through port 8083 and my droplet is in SF01. Did you guys also block 8083 for SF01 region? Today I did apt-get update and I noticed that there was an update for Vesta CP, so I want to know if the Control Panel stopped being accessible to me due to the update or because you guys blocked port 8083 for SF01 droplets. Thank you!

Reply Report
  • eris April 9, 2018

    Yes, port 8083 is blocked in all regions. We recommend running VestaCP on a different port for the time being.

    Please open a support ticket if you have any further questions.

    - Eris

    Reply Report
    • secretsaiyan April 9, 2018

      Thank you so much for the fast response, Eris. I hope you guys get this issue sorted out for all regions :) I’ll change the port later, but for now I just did a service stop vesta to keep the control panel shut down for now. I don’t need to access Vesta CP at the moment.

      Reply Report
jassdesigngroup April 9, 2018

Hi there

Can you help me to turn networking back on for my droplet to take action and fix this problem?

Ticket #1447067

Reply Report
fandavy April 9, 2018

Hi Can you please take a look at 1446838

Reply Report
Acasuso April 9, 2018

Still waiting for response 1446685 it’s been 6 hours since my server had vestacp stoped, antivirus verified, ports and firewall changed and email responded.

Reply Report
finanzashvg April 9, 2018

My ticket #1447361
The temporal solution is turn off the vesta service (sudo service vesta stop)
My droplet was not compromised.
Please, turn networking back on my droplet.

Reply Report
Shadowhaxor April 9, 2018

Ticket #1447097
I had already set up my vestacp so only I could access it. I verified that my droplet was not compromised and fired off a ticket. My vesta cp port was also changed. No response as of yet. Please, turn networking back on my droplet.

Reply Report
neha April 9, 2018

Till what time 8083 port will be blocked?? This is weird way of resolving issues by blocking a specific port across all infrastructure.

There can be other applications which are running on 8083 port and you guys made them stopped.

1445131

Reply Report
  • anthonyrossbach April 9, 2018

    You can easily change the port now if you need, and can change back later if you need, I posted the script below. https://www.digitalocean.com/community/questions/how-do-i-determine-the-impact-of-vestacp-vulnerability-from-april-8th-2018?answer=42884

    Reply Report
    • neha April 9, 2018

      We can not change the port. It’s a IOT server and more then 5000 remote devices connect to this server on 8083 port.

      I am still waiting for the reasoning to block a port globally and when it will be opened again?

      Reply Report
      • anthonyrossbach April 9, 2018

        Reason is un patched systems missed can be infected if the ports open. Its to prevent more servers from needing to be taken offline that are not infected but also not updated.

        I suspect it may not be removed for some time.

        Reply Report
      • anthonyrossbach April 9, 2018

        If posible I would for any devices using the API or port to be easily changed, for me I updated all my systems and scripts by changing a single DB value for the connection port and address for each server running VestaCP. It’s a good idea to have a way to easily switch addresses and ports.

        Reply Report
        • neha April 9, 2018

          We are not using VestaCP at all. Our application is an IOT server which connect with remote devices on different ports over public networks.

          There is no way to switch the ports on these device remotely.

          We regret to use digital ocean which does not have capability to block port only on effective servers rather then doing it globally and effects other..

          Reply Report
          • tim3b April 10, 2018

            We are not using VestaCP at all, either. And we also regret how Digital Ocean has blocked legitimate services depending on port 8083 for other purposes.

            In our case we have a native application configured to access web services on that port – and switching represents no simple matter. We have been down in the water for two days now, without receiving any substantive response from Digital Ocean (eg: 1444863).

            Frustrating that status.digitalocean.com shows this incident as “Resolved” when clearly customers are still experiencing significant downtime as a result.

            Reply Report
sophyxed April 9, 2018

I have like 7 droplets and like 30-40-something websites running under vestacp on all droplets! only 1 droplet so far has been shut-off that was hosting 2 websites.

Are you guys telling me that you are going to shut-off those 30-40 websites sooner or later???? This is a total painnnnnnn in the ash if my option is migrating for immediate traffic recovery! Do you have a fix for this??????

Reply Report
jonathancelio April 9, 2018

Hi guys,

In this night, suddenly, my websites goes down, but my droplet don’t. So I entered in the server via SSH in my console and then show me that:

“DigitalOcean Recovery Environment 17.04.1”

So I enter here and see this huge problem with VestaCP, and my server is running in VestaCP, so I think that is the problem, but I don’t know what to do.

I just use VestaCP to manage all sites and databases in one place, i dont use than to log in the control panel. ALL my web pages are offline and I need to going back all immediatly.

Please take a look at Ticket #1446953

Reply Report
MohdSohail April 9, 2018

Hey Guys!

I followed instructions to checking and deleting the vulnerable files. I have also disabled vesta service. Vesta team has also released the security patch. I can’t install the patch without the internet connection.

You guys please activate networking so that I can install the patch. And I am saying it again that I have disabled the vesta service. I also have changed the default vesta port from 8083 to something else. Please do it otherwise it’s a loss every day like everyone else.

https://forum.vestacp.com/viewtopic.php?f=25&t=16575

The ticket I have raised - #1447354

Reply Report
bbnchile April 9, 2018

Hey there!
Please!!!

1447790

vestacp release a patch, but i can’t apply with no connection…

HELP PLEASE!

Reply Report
pasaro April 9, 2018

Hello DO team,

I use vestacp and my droplet seems like got compromised.
I have checked there are gcc.sh and livudev.so.
Anybody here can pointed me out how to clean this step by step?
There is a huge long thread on vestacp forum that make difficulty to summarized a simple step by step to handle this situation.
I try to clone the droplet and immediately got another take down.
My ticket number:
first droplet : #1444597
second droplet: #1446615

Thanks in advance.

Reply Report
neotec April 9, 2018

My droplet was taken down for this issue. Now I wasn’t able to access my files to do necessary backups in order to move to another droplet. I can’t SSH and I can’t use the web console either (no access to existing files). How do I move my existing files.

How do I back up and move my existing files to a new droplet?

Please help with tieckt #1447622 ASAP!

Thanks!

Reply Report
neotec April 9, 2018

I have received an email from DigitalOcean saying “Because your Droplet may have been compromised, you’ll need to back up your data and transfer it to a new Droplet. We have a recovery tool to assist you…”

I have two questions regarding this:

1) Where to find and how to use the recovery tool mentioned in the email?

2) The email suggests I move my files to a new droplet. But do I need to install VestaCP first on the new droplet and I should start with a blank droplet?

Thanks!

Reply Report
speti91 April 9, 2018

VestaCP: The fix has been released just now!

You can check here: https://forum.vestacp.com/viewtopic.php?f=10&t=16556&start=260#p68893

Reply Report
roncooletz April 9, 2018

I have already changed the default port. Now my droplet needs connectivity on the network. Please see my ticket #1444662. Thank you.

Reply Report
romkond April 9, 2018

Hi,
how can I download files (eg. data base dump) from my server to my PC (Windows 7) using DO console (can’t use FTP clients and PuTTy because you blocked them)?

Reply Report
blissrob April 9, 2018

Please review Ticket #1447360

Like many others, this VestaCP complete network block is impacting many many live users.

Many of my servers have only recovery console available via direct console. This is an issue.

Also, I only use secure SSH (alternate port with key) to access my servers and have no known root password. So for others that do not have recovery console, I cannot access either.

This is 100% downtime on all my droplets.
I can run script to resolve issue across all droplets quickly if networking is reenabled.

Please review Ticket #1447360

Reply Report
guru3dc3b30da436aa9ab0c5c7 April 9, 2018

Hey,

About 10 hours i am waiting reply about my ticket: #1443782

Please help

Reply Report
dld April 9, 2018

8 hours wait here. Droplet checked. No infection. Admin port changed. Htauth set up for admin site. But no way to apply patch without network. Please re-connect me.

1447615

Also unsure why this server was singled out. We’re running 4 vesta servers. None infected. Only this one disabled. Other three patched within minutes. This one … still unable to patch.

Reply Report
keloxers April 9, 2018

Please Helpme Ticket #1449452

Reply Report
marianoconsoni April 9, 2018

Hi, due to time difference (I’m in GMT-3) I see this email monday morning when my customers started yelling me because they can’t use their emails and websites. I don’t need vesta control panel by now, just email and webservers.
I don’t receive answer on my ticket.
Ticket #1449553

Thanks.

Reply Report
  • marianoconsoni April 9, 2018

    I don’t know what to do. My customers can’t access their emails (so they can’t work) and want to send me a legal demand. And Digital Ocean does not respond my tickets.
    Is there any phone number or a way to contact DO apart of tickets?

    Reply Report
blissrob April 9, 2018

Please review Ticket #1447360
I need access to my droplets in order to access data and update.
@ryanpq- Can you help?
@etel- Maybe you too? We need an inside advocate on this.
@jarland

Reply Report
cloudd1fb0b307e April 9, 2018

I need help with Ticket #1449474. Please, answer me.

Reply Report
blissrob April 9, 2018

@ryanpq - Are you able to help with this. You’ve always been very helpful to me and others in the past. We have thousands of users impacted and need an advocate.

Reply Report
waldsonpatricio April 9, 2018

Please, check mine: #1446775

Reply Report
aronjayvo April 9, 2018

Please check ticket number 1448697. The response time is awful.

Reply Report
vsib April 9, 2018

Please check ticket 1443289. Requesting the Recovery Environment

Reply Report
jaimegdelrio April 9, 2018

Four of my droplets are reboot and all files deleted, please, attend my ticket #1450226

Help

Reply Report
prnnk April 9, 2018

Please check my ticket [Ticket #1446734]

Reply Report
bangbible April 9, 2018

You systematically removed servers from the network without knowing the root of the issue. VestaCP its self was not the entire issue. Certain programs selected and installed along with Vesta were. You didn’t check if servers were running these programs you just removed paying customers from your network without notice.

We were not compromised and you caused damage. You are still restricting access over an issue that does not affect us.

You will not reply to tickets asking if you plan on removing servers in other locations for the same reasoning.

You have not provided proof that such power was granted in the tos before this event.

Reply Report
ooo34rus April 9, 2018

Hello.
I checking and deleting the vulnerable files and disabled vesta servise.
Please, turn networking back on my droplets.
My tickets:
1443988
1444356
1443758
1445561
1444295
1445503
1443986
1445408
1444179
1445568
1444180
1444228
1444550
1443958
1444881
1438893
1444417

Reply Report
carlosbelcha April 9, 2018

Hi everyone with this problem TEMP SOLUTION

Change the port by following: Change Port

Then you can access on the new port with no issues (also make sure you update to 0.9.8-20 before changing the port), you can do that with:

Check current version:
v-list-sys-vesta-updates
If not, 0.9.8-20 then run the following from CLI:
v-update-sys-vesta-all
That has the patch for the attack that was launched on Saturday 7th April that caused ports 8083 to be blocked in the first place.

Reply Report
  • sanmiguel April 9, 2018

    Hi Carlos,
    My droplet don’t allow me to run this command, I made another froplet and was fine, btw when this droplet goes to find the update and download it but droplet is offline, how you made this? Regards

    Reply Report
prnnk April 9, 2018

Please answer my ticket #1446734

Reply Report
zcheung April 9, 2018

I need help reactivating my server

Ticket #1447558

Reply Report
alexcastaneda April 10, 2018
[deleted]
Reply Report
purvibenvasavada April 10, 2018

guys, one of my expert guy fixed this..

you can contact him to do so..

https://www.fiverr.com/sudolinux

Reply Report
ms980896 April 10, 2018

Please answer my ticket #1444482

Reply Report
archergod April 10, 2018

Since VestaCP release a patch and some of us already applied it. any idea when you will open ports? Or changing port in Vesta is only option we have for now?

Reply Report
  • sanmiguel April 10, 2018

    I can see DO is doing a search and close procedure for those with Vesta no matter if you changed the port to another, this is insulting because they aren’t answering tickets, Vesta is a very good option and full of support, this case is really really dab because decision made by DO

    Reply Report
    • archergod April 10, 2018

      I agree this is not good decision to block port, as port is not only used by vesta but other services. But when the Vesta Team solve the issue, they can open port for those who upgrade it .. it is 2 days and I am not sure if I wait or change port or both. They are bit on “our system, we do what we want” mode.

      Reply Report
      • wpa April 10, 2018

        If you have changed the VestaCP port from the default, please open a ticket up with us.

        To help expedite the process please provide us with a screenshot of the output of the following commands in the ticket as well:

        ls /etc/cron.hourly/
ls /lib/
ls /etc/rc.*
ls /etc/systemd/*
grep "listen" /usr/local/vesta/nginx/conf/nginx.conf

        If your droplet is currently booted using our recovery ISO, please ensure the droplet disk is mounted and provide the following instead:

        ls /mnt/etc/cron.hourly/
ls /mnt/lib/
ls /mnt/etc/rc.*
ls /mnt/etc/systemd/*
grep "listen" /mnt/usr/local/vesta/nginx/conf/nginx.conf
        Reply Report
prnnk April 10, 2018

Please check ticket #1448499, false disconnect, server was not compromised.

Reply Report
wellington78dd896a2f4c0dc7 April 10, 2018

Please answer my ticket #1456045

Reply Report
archosaur2 April 10, 2018

You just block my panel without any reson by your suspitions?
Guys in DO are you crazy ????
I’ll better switch hosting…

Reply Report
  • anthonyrossbach April 11, 2018

    You do know it was a known flaw and thousands of panels where hijacked and used for DDOS attacks… They acted as they should in this case, and the fix on your end is simple, update, and change port if you want to access now via the web.

    Reply Report
    • archosaur2 April 11, 2018

      They should block outer traffic from droplet, not access to droplet for owner.
      Now I loose my production site (broken mysql, had 500 error), must reinstall all, restore database on new droplet.
      Just because stupid action - block access to admin panel.
      At least they can send me warnin BEFORE blocking, not after.

      Reply Report
goncharoveg April 10, 2018

Hi, I disabled VestaCP on my servers. Can I be sure that the servers will continue to work and will not be disabled by you?

Reply Report
  • wpa April 10, 2018

    Please open a ticket up with us so we can take a look and make sure the droplet has not been compromised.

    So that we can help you get this resolved quickly please also provide us with the output of the following in this ticket:

    ls /etc/cron.hourly/
ls /lib/
ls /etc/rc.*
ls /etc/systemd/*
grep "listen" /usr/local/vesta/nginx/conf/nginx.conf

    If your droplet is currently booted using our recovery ISO, please ensure the droplet disk is mounted and provide the following instead:

    ls /mnt/etc/cron.hourly/
ls /mnt/lib/
ls /mnt/etc/rc.*
ls /mnt/etc/systemd/*
grep "listen" /mnt/usr/local/vesta/nginx/conf/nginx.conf
    Reply Report
hakanbaysal April 11, 2018

Hi,

Please follow the 4 steps below

Firstly list firewall -> v-list-firewall
Change VestaAdmin port -> v-change-firewall-rule 2 ACCEPT 0.0.0.0/0 8888 TCP VestaAdmin
Edit conf file 8083 to 8888 -> sudo nano/usr/local/vesta/nginx/conf/nginx.conf
Restart vesta -> /etc/init.d/vesta restart

Reply Report
  • romkond April 11, 2018

    It’s a valuable info! Though, there’s a typo and you don’t need “sudo” if you log in under root. Thus:
    Firstly list firewall-> v-list-firewall
    Change VestaAdmin port -> v-change-firewall-rule 2 ACCEPT 0.0.0.0/0 8888 TCP VestaAdmin
    Edit conf file 8083 to 8888 -> nano usr/local/vesta/nginx/conf/nginx.conf
    Restart vesta -> /etc/init.d/vesta restart

    Reply Report
penchanda April 11, 2018

hi, please check my ticket. #1447504

Reply Report
reg64848581f6004cc3770ae75 April 11, 2018
[deleted]
Reply Report
tim3b April 11, 2018

Ticket #1444863 continues to go without resolution for three days now.

We have never used VestaCP though we depend specifically on port 8083 being available and open – as believed would be the case when choosing Digital Ocean over a year ago.

It is unbelievable that Digital Ocean has marked the overall incident as “resolved” while paying customers are still experiencing significant downtime as a result…

Reply Report
neha April 11, 2018

We are on the same boat. Even today they closed the ticket #1445131 without opening 8083 port. Weird way to resolving things by blocking a port globally.

Reply Report
archosaur2 April 12, 2018

Moved to Linode, panel and site works excellent, no need to begging weird support to unblock something.
PS. I used DO since 2015 but, no more.

Reply Report
tothdaniel87 April 13, 2018

Hi, my droplet was affected by the VestaCP vulnerability. I host several sites on this droplet. I requested to boot it in the recovery tool to download data from it. I did it immediately and completely rebuilt the droplet with a new Debian image. I requested to boot the droplet back to it’s disk, but I get no answer…
So now it is almost 5 days since my droplet is offline and DigitalOcean do not answer on tickets.
I need the same IP address, because many domains are redirected to this IP, which I can not change. So rebuilding the droplet is the only option for me.

I am using DigitalOcean for more than 4 years now, but this failure of support is shocking. Please respond something on my ticket: #1448429

Reply Report
ooo34rus April 13, 2018

Hello.
I checking and deleting the vulnerable files and disabled vesta servise.
Please, turn networking back on my droplets.
My tickets:
1461992
1455447
1462500
1462122
1461845
1461882
1462015
1445568
1462125
1456021
1462120
1462123
1456046

Reply Report
spidoyman April 13, 2018

Hello,
please check my ticket :)

==> Ticket #1474222

Reply Report
karimhb April 14, 2018

Access the VPS via the emergency console.

scan your VPS 

clamscan -r -i /

I found the trojan here 

/lib/libudev.so: Unix.Trojan.DDoS_XOR-1 FOUND

I successfully removed the trojan using the instructions on https://admin-ahead.com/forum/server-security-hardening/unix-trojan-ddos_xor-1-chinese-chicken-multiplatform-dos-botnets-trojan/

reboot your vps

scan your VPS again

clamscan -r -i /

and make sure the cron job has been removed

 ls -la /etc/cron.hourly/

Now that the trojan has been removed. You can request to remove the ban on your VPS network.

Reply Report
vestalovers May 22, 2018

How we can send you full details about what exactly happend to VestaCP?
We think there are STILL vulnerable servers on digitalocean, because hole was not at vestacp service.
We found where is it.

Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help