Bruteforce attack through our server

May 3, 2017 6k views
Server Optimization CentOS
developerc9740c6fa8b2fbffc
By:
developerc9740c6fa8b2fbffc

Hello.

Recently I've received an email from Digital Ocean(DO) that someone claimed a hack attack from our IP. The day before, we received a Brute force attack abuse from DO that our IP/Server has attacked another server.

I have scanned the server for any suspicious malware/virus, also I did a security upgrade at our wordpress sites, but I have not found anything suspicious yet. I haven't either been able to find any suspicious outgoing traffic from our server, so it's quiet strange.

I am curious what your steps are to find the cause of an bruteforce attack/hack attack?
There is nothing alarming in our logs as far as I can see :/

Regards, Simon

Log In to Comment
2 Answers
jtittle1 May 3, 2017

@developerc9740c6fa8b2fbffc

At the server level, I would first look over the contents of /var/log/auth.log.

tail -50 /var/log/auth.log

That'll give you an idea of who has logged in, or attempted to, and whether it was successful or not. If you don't recognize one of the users, see if they exist.

cat /etc/passwd

One a clean (freshly deployed) Ubuntu 16.04 Droplet, where no additional users have been added, the above file looks like:

root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
systemd-timesync:x:100:102:systemd Time Synchronization,,,:/run/systemd:/bin/false
systemd-network:x:101:103:systemd Network Management,,,:/run/systemd/netif:/bin/false
systemd-resolve:x:102:104:systemd Resolver,,,:/run/systemd/resolve:/bin/false
systemd-bus-proxy:x:103:105:systemd Bus Proxy,,,:/run/systemd:/bin/false
syslog:x:104:108::/home/syslog:/bin/false
_apt:x:105:65534::/nonexistent:/bin/false
lxd:x:106:65534::/var/lib/lxd/:/bin/false
messagebus:x:107:111::/var/run/dbus:/bin/false
uuidd:x:108:112::/run/uuidd:/bin/false
dnsmasq:x:109:65534:dnsmasq,,,:/var/lib/misc:/bin/false
sshd:x:110:65534::/var/run/sshd:/usr/sbin/nologin
pollinate:x:111:1::/var/cache/pollinate:/bin/false

If you see someone you don't recognize and you're sure it's not a user that's associated with your site or services you're running, you can remove them using userdel. Be careful there as if you delete a user that is associated with a service, it will fail.

userdel username

Beyond the basics of checking logs and the current system users, run a root kit check using either rkhunter or chrootkit. Using DenyHosts is also a pretty good option to block certain attempts at logging in, though as with any tool that can deny access, you need to be careful not to block and lock yourself out.

http://denyhosts.sourceforge.net/

http://www.chkrootkit.org/

http://rkhunter.sourceforge.net/

...

If you're not already, use SSH Keys over plain-text password authentication when accessing SSH. An RSA key with a bit size of 4096 bits or even 8192 is ideal, though you can also use ed25519 as well. You'd generate the key(s), download the private key (if you can't generate them locally) and make sure the public key is in your users authorized_keys file.

SSH Keys are the preferred way to login, though I would also advise making sure your key has a valid passphrase protecting it, otherwise key-based login can be as insecure as password-based login since there's no valid protection preventing someone from using your key(s) if they get ahold of them.

Once you have SSH Keys setup, completely disable password-based login.

...

Also server side, check permissions on files and directories. If any directories or files are using a chmod of 777, fix it now. There's no reason to give such higher level access. Files should be at best using a chmod of 644 and directories a chmod of 755 (some may get by with 750).

...

On the WordPress side, check out WordFence:

https://wordpress.org/plugins/wordfence/

Also check your theme files, plugins, etc -- make sure they are 100% up to date as well.

Reply
  • developerc9740c6fa8b2fbffc May 3, 2017
    [deleted]
  • developerc9740c6fa8b2fbffc May 3, 2017
    [deleted]
  • developerc9740c6fa8b2fbffc May 3, 2017

    @jtittle Thank you Indeed for your answer, it helped me a lot, by investigating the attack. I've found a strange root login attempt, which I am not sure how it should be interpreted.
    As far as I can see, it's a successfull login, But I'm not sure:

    May 3 14:15:18 s2 sshd[19170]: Accepted publickey for xxxx from xx.xx.xx.xx port 52242 ssh2: RSA xx:xx...
    May 3 14:15:18 s2 sshd[19170]: pamunix(sshd:session): session opened for user root by (uid=0)
    May 3 14:15:22 s2 sshd[19170]: pam    unix(sshd:session): session closed for user root

jtittle1 May 4, 2017

@developerc9740c6fa8b2fbffc

If you're using port 55242 for SSH instead of the default port 22, those messages are simply telling you that a session was initiated for root and then rather quickly closed out.

If you're not using 55242 for SSH, then that very well may be a cause for concern and you'd need to investigate further.

When it comes to being compromised, it's often best to begin looking at migration options and then, setting up a more secure environment. During the migration, you'd evaluate your code, database, etc to ensure that there's not something that would allow a would-be attacker to get back in.

...

What I would recommend, on a new server is:

1). Deploy with SSH Keys -- this disables password authentication with DigitalOcean.

2). Setup a sudo user with their own SSH Key, confirm it works (i.e. you can run commands), then lock the root user. This ensures no-one can login with or attempt to login with the root user. All root-level commands are then executed by the sudo user.

3). Setup firewall rules that deny incoming and allow outgoing by default policy, then only open the ports that you absolutely need open such as SSH (22), HTTP (80), and HTTPS (443). Unless you're running an e-mail or DNS server, or have some very specific requirement, that's generally all you need to open as when it comes to NodeJS, Python, etc -- you'd be proxying locally or proxying over the private network. Make sure the firewall is active.

4). When it comes to SSH and Port 22, it's best to restrict by IP. If you don't have a static IP, this can be a problem as your IP can change at any time. In such a case, I'd recommend setting up a VPN, then restricting access to the VPN IP since it will be static. You'll have to be connected to the VPN to get in but it's far more secure.

5). Don't use sudo users for access when it comes to your sites. Use normal, un-privileged users that can't login to a shell. Allowing SFTP access to these users is fine, but they should be prevented from logging in to any form of SSH unless it's absolutely required.

6). Use individual users for each site. You can use www-data for your first site, but create a new user and a new home directory for each site after, even if you use www-data2, www-data3 etc. This will prevent one user from accessing files of another as long as permissions are setup correctly.

7). Make sure you're not exposing information that shouldn't be exposed. For example, with PHP, using phpinfo() will provide you with a lot of useful information about your setup, but you don't want that all exposed to the public.

Beyond the above, the previous information still applies.

Reply
  • developerc9740c6fa8b2fbffc May 6, 2017
    [deleted]
  • developerc9740c6fa8b2fbffc May 6, 2017

    @jtittle Thanks again for your excellent answer. Regarding the strange login with ssh key, It was one of Deploybot's IP's which is valid. They are using 5 different IP's so I did not notice that It was one of theirs.

    So clarifying that no one has been inside our server, I have now received information that the bruteforce attack is happening through Wordpress xmlrpc.php. Is there any idea in updating the theme files right now, rather than revert/reinstall the server?

    We are using Wordfence at our wordpress installations.

    • jtittle1 May 6, 2017

      @developerc9740c6fa8b2fbffc

      You can block or restrict access to a file using .htaccess, though you need to be careful as the file is used by some services, such as JetPack (if you have that plugin installed).

      For example, if you're using Apache 2.2, you might add something such as:

      <Files "xmlrpc.php">
    Order Deny,Allow
    Deny from All
    Allow from IP_ADDRESS_01
    Allow from IP_ADDRESS_02
</Files>

      Where IP_ADDRESS_01 and IP_ADDRESS_02 are two IP's that you want to allow access to. You can add any number of IP addresses, and you can use an IP range as well.

      If you're using Apache 2.4, you'll need to use a slightly different setup:

      <Files "xmlrpc.php">
    Require all denied
    Require host IP_ADDRESS_01
    Require host IP_ADDRESS_02
</Files>
Have another answer? Share your knowledge.

Related Questions