Why has my droplet networking been disabled?

August 2, 2017 3k views
DigitalOcean Miscellaneous Security
auroa
By:
auroa

First I get an email from DigitalOcean thanking me for reporting my own droplet to them?

Thank you for your submission. A member of the Trust & Safety team will review the details as soon as possible. If appropriate, the information will be forwarded to the associated customer in its entirety.

If there is additional information you'd like to provide, please respond to this email. If you did not provide logs, we would - at a minimum - require an IP address and timestamp (with associated time zone) to identify the correct server where the activity originated. We will contact you with any follow-up questions, should they arise.

Please note that as an unmanaged cloud hosting provider, we do not create, administer, or have direct access to our customers' virtual servers. We, therefore, cannot make direct changes to any programs or websites hosted there.

Additionally, our privacy policy does not allow us, under any circumstances, to share information about the customer with third parties.

Regards,

DigitalOcean

Message makes no sense, I start to think someone has access to my digitalocean account and is reporting my own droplets to shut me down... I panic. Message digital ocean, get a reply after 12+ hours the following:

The traffic we noticed was a SYN flood (http://en.wikipedia.org/wiki/Syn_flood) being launched from your Droplet against a remote server at 60.191.143.212 - not any form of legitimate traffic. This could not have been from a remote system, as there was no inbound traffic (from your client) during this incident.

Now they claim someone hacked into my droplet and performed a DDOS attack. I have been with digitalocean for more than 4 years, and this is the first time something like this happens.

I am upset of how bad their first email was, forcing me to spend an entire day in paranoia over what is going on, trying to migrate code and redeploy. Secondly, the second email reveals nothing useful.

There is nothing hosted on the IP that they say someone DDOSed. Likely someone hacked into my droplet, but something doesn't sound right. Just doesn't make sense.

Either way, my question and concern is, do I get the blame for any of this DDOS attack?
Should I keep doing business here?
Does my standing as a DO customer change because of this incident?

There are other people reporting that there is abuse of the "droplet report system".

Is DO going to do anything about this? This day because of all this mess, I lost $500+ I am not expecting to be compensated by DO, but I can't afford having my droplets shut down for no reason.

Their support system is very slow, especially for critical issues like this.

Anyone can chip in?

Log In to Comment
2 Answers
hansen August 3, 2017

Hi @auroa

There's a lot we, the community, cannot answer or see, since we're just users too.

First off, if DigitalOcean says that your droplet has been used in conjunction with something, then it's probably true. They are looking at their network logs and can see activity from your droplet.
You should then thoroughly examine your droplet for any signs of intrusion - make a copy of every log (/var/log/) and investigate. It's easy to manipulate logs if the intruder got access to your droplet with root access, so if you don't find any evidence then it doesn't mean nothing occurred.

To your question about blame. Yes, it's your responsibility to secure your server, so someone else doesn't use it for something bad.
If you should keep doing business with DigitalOcean, well, that's your decision.
I don't know if DigitalOcean keeps track, that's for DigitalOcean to answer.

What do you mean about "other people reporting" - can you link to those?

What do you think DigitalOcean should be doing? I understand you want a bit more clearer information, I agree, but besides that, I don't see what they can do differently.

And if you're losing $500/day, then you should consider a high availability system using multiple servers, different regions, different providers. Then it doesn't matter if one of them closes down. I don't earn that amount on most of servers, but I have two clients, which runs with this setup, since downtime for them would be very costly.

DigitalOcean has been slow with support lately, but they're working on a new Cloud Support system, where tickets hopefully will get much faster response. I thought it would launch Tuesday, but it doesn't look like it's finished yet.

Out of curiosity, what are you running on your server - and is everything fully updated and secured?

@jtittle Can you comment on behalf of DigitalOcean?

Reply
jtittle MOD August 3, 2017

@auroa @hansen

When it comes to management, DigitalOcean is a self-managed provider (which is often referred to as un-managed). We do not have access to a customers' Droplet(s), which prevents us from logging in or executing commands. We do have monitoring in place to detect issues, such as the one your Droplet was flagged for, though that's network level.

On our end, we manage:

  • The Hardware (hard drives, RAM, networking, etc)
  • The Network (connectivity, bandwidth, etc)
  • The Hypervisor (the physical node where your Droplets are hosted)
  • Our Control Panel and API

...

The customer would be responsible for day to day management of their Droplets, which includes:

  • Configuration and Setup
  • OS Updates and Upgrades (ensuring the latest packages are up to date)
  • Security (OS, software, applications, etc)
  • Performance (tuning your stack, managing resources, etc)
  • General sysadmin duties (i.e server management in general)

...

We do our best to provide guidance (though support ticket), though we're not a replacement for a sysadmin.

It's much like if you set up a home server on a spare desktop or laptop. You'd be responsible for the server and all that comes with managing a server. If that server was hacked and used to send a flood or used to DDoS another IP or host, they too would cut the connection as they wouldn't have access to your server to do anything.

While we do our best to provide the best support possible, when it comes to issues like these, we have to consider the network and the effect on it. If you're Droplet was hacked and is sending out a level of traffic that affects the network, we have to take action otherwise we jeopardize the network as a whole.

When networking is cut, you still have access to console and can troubleshoot any issues from there. In general, if the server has been hacked, we normally advise setting up a new Droplet and working to migrate data, though you'd need to make sure cleanup is performed to ensure you're not moving any data that caused the hack over to the new Droplet, otherwise it's very likely the same thing could happen again.

..

Please keep in mind, this isn't specific to DigitalOcean. I've worked with close to 1,000 providers over the last 15 years. The common term for this would be null route, meaning the provider simply nullifies any and all traffic. It's actually not uncommon and could happen with any provider if your VPS, Shared hosting account, or even Dedicated Server is being used to flood the network.

I can't comment on your account specifically as this is a public forum / community and any account specific details would need to stay in the ticket regarding the issue. I would definitely welcome you to follow up, ask for details, provide any logs that you can gather from Console, and ask for help.

When it comes to things like this, we deal with them daily. We're not, however, refusing to work with you or help as best we can, though we are limited in what we have access to and resolving a hack or issue such as this would be something you'd need to be willing to handle as it's one of the tasks that comes with managing a server.

Reply
  • auroa August 3, 2017

    First of all, I never expected you to log in or manage my droplet. I understand that you are not responsible for whatever is happening "inside" the droplet. But instead of sending me a generic email that blocks access to my droplet through SSH. Which means I don't have access to my data don't know what is going on. This is a scary situation to put someone in, that their livelihood depends on your servers.

    My problem is that you disabled my droplet without informing me what is going on, you send a wrongly worded email (e.g. "thanks for reporting this droplet"), I had to wait 12 hours to learn why I don't have access to my droplet, by that time I had already deleted and created new droplets to meet my clients requirements. THAT IS MY PROBLEM. Your bad communication skills when it comes down to such serious problems. If I would have known the droplet was compromised I would have been careful of which backups I would use, of what I would do next.

    Don't try to make the managed/unmanaged provider argument. You were the ones that disabled my droplet and should have given me a valid reason to help me resolve this issue in my new droplets and mitigate the problem FAST.

    • jtittle MOD August 3, 2017

      @auroa

      I'm not trying to argue either way. I'm simply providing a basic overview as, while you may be aware of the differences between managed and self-managed, others reading this may not and it's important to clarify. That is to say, we see the question quite often :-).

      When it comes to the e-mail you received, I'll agree that it's not clear. That is something that I can bring up on our end and I'd be more than happy to just that.

      ...

      In regards to resolving the issue, without access to the Droplet, we would not be able to tell you why it's happening or what software / application is causing it.

      We can see the traffic over the network, but not what's happening within the Droplet, thus it's be hard to identify why or how it's happening, or provide exact details on how to resolve the flood or prevent it on another Droplet that you migrate to.

      The only real insight we have as to what's running on your Droplet is basic information and what you tell us. We can use nmap to check ports -- that may let us know you're running a web server if ports 80/443 are open. We can check header responses to see what web server you may be using, if you're not behind a proxy (i.e CloudFlare), but honestly, that's really all we have to go when it comes to making an educated guess as to what you're running on the Droplet.

      ...

      Moving forward, my recommendations would be:

      • Ensure the firewall is properly setup. Block incoming/outgoing traffic on ports you don't specifically use, or that are not used by the OS. You can disable the firewall on the Droplet and setup a Cloud Firewall (they're free of charge) and have a nice GUI from which to work with.
      • Install an IPS, such as Fail2Ban and use it to protect SSH and other services that allow any type of log in.
      • Use a VPN and lock SSH access to that VPN IP. This way you're locking down SSH to a single IP. If you lock yourself out or change VPN's, you can access the web Console to make any changes (i.e allowing a new IP, removing the old IP). I normally use Algo.
      • Install Lynis to audit your servers. Use it often -- it's an excellent tool. Some items may not apply since you're on a VPS, but most are very much valid and will allow you effectively secure your VPS.
      • Install Maldet and ClamAV for malware and virus scanning.

      ...

      I would also look in to security for your applications. For example, if you were running WordPress, there's a ton of plugins to help prevent attacks on the front end. iThemes Security is just one of them:

      https://wordpress.org/plugins/better-wp-security/

      For other applications, it varies. Web server security also comes in to play -- I don't use Apache often myself as I prefer NGINX, so my recommendations there would be limited to NGINX -- though there are ways to prevent certain request types from your basic POST, GET, etc all the way to requests that would normally be used for SQL injection.

      ...

      Beyond those recommendations, I'd at least speak with a sysadmin. Most will at least speak to potential clients and provide a detail of what they can do for you. I know I always welcomed questions when I was freelancing as a sysadmin and even answer questions from clients I still work with outside of DigitalOcean.

      When it comes to security, it's a very broad subject and impossible to cover every last bit, but the above will at least get you headed in the right direction and hopefully prevent such from happening again.

      • auroa August 3, 2017

        Ok thank you, I understand 100% of your recommendations and I will try to be much more careful of my droplet security.

        And I appreciate that you will try to bring up the email issue in regards to the information.

        The only way my droplets could have got hacked (was two of them at the same time, same region) both were using the same numeric password of length 10 (random digits, nonsequential), that is still 10000000000 possible combinations. maybe they got lucky. One of the servers was clean fresh install from DO, nothing installed, the other had some stuff installed, but most of it regulated by the firewall. Both servers hacked at the same time. They have only been up for couple of days. I have number of servers but these were the only ones. And they belonged to the same region.

        Also the IPs that the logs showed I was SYN Flooding is not an IP address that has anything. There is no website, no server hosted there. The IP is part of some chinesse hosting company's subnet. This part makes 0 sense.

        So overall it is weird. I know you try to help as sysadmin but I also have a feeling that DO is not being transparent.

        • jtittle MOD August 3, 2017

          @auroa

          There are other locations you can check:

          /var/log/auth.log

          and

          /etc/passwd
/etc/shadow

          Generally you want to look for unknown users. If you were hacked and they were decent hackers, you'd never know they were there and that's what makes it hard.

          Script Kiddies generally leave traces -- it's often sloppy work. Anyone who knows what files do X, Y, Z -- they'll clean up their tracks. In such cases, you may only be able to find the software or application causing the issue.

          Checking top or using htop will show you running processes that you can sort. If you see anything odd, find out what it is. Generally you can use which to find the real path to the executable.

          For example, for NGINX:

          which nginx

          You can also check .bash_history for users. By default, root is:

          /root/.bash_history

          That'll tell you what commands were executed by the user. It's saved when the session is terminated.

          ...

          When it comes to passwords, having dealt with my fair share of compromises, the only safe size in my eyes is 32-64 characters and it should be mixed -- letters (upper and lower case) as well as numbers.

          If special characters are supported, use them :-).

          For example:

          =S1Ch68c~Vs^+,oXv4u`Q9"kg~ezi"<hO-{"Io3GIWDLe/Wg+LeN;Gu(u,c@

          or

          GTbu1RkQiSOD2HYyaeMlnOkQRu3eHnEDfd27XmQ2sZ4keaV9bQFqpzxei7Ndhzd

          Of course, you need a way to remember those too, so a secure password manager comes in handy. I've used both LastPass and Dashlane with success.

          You can also use phrases -- which are nearly impossible to guess if done right.

          this-is-an-example-phase-which-is-easier-to-remember-but-difficult-to-guess

          ...

          As far as the target, even though the IP may lead to a blank page, it could be a case of when accessing the IP, the web server is configured to show a blank page to non specific requests (easily setup in Apache and NGINX). The traffic would still do damage in this case, due to sheer size.

Have another answer? Share your knowledge.

Related Questions