I have a Droplet running Ubuntu 16.04.

Virtualmin is installed and running virtual servers with Wordpress sites.

I can no longer access any site or the Virtualmin platform.

I think I’m being hacked but not sure how to stop this happening.

It happened earlier in the week and I restored a working backup and made sure Fail2Ban was on via Virtualmin.

Today it’s happened again…

I’ve tried running

sudo service mysql restart

With result

Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.

Running systemctl status mysql.service gives

mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: activating (start-post) (Result: exit-code) since Fri 2020-04-17 10:45:34 UTC; 17s ago
  Process: 16667 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
  Process: 16659 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 16667 (code=exited, status=1/FAILURE);         : 16668 (mysql-systemd-s)
    Tasks: 2
   Memory: 220.0K
      CPU: 343ms
   CGroup: /system.slice/mysql.service
           └─control
             ├─16668 /bin/bash /usr/share/mysql/mysql-systemd-start post
             └─16709 sleep 1

and sudo journalctl -xe

Apr 17 10:45:57 *******.**********.co.uk sshd[16724]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser=
Apr 17 10:45:58 *******.**********.co.uk sshd[16720]: Invalid user x from 165.227.225.195
Apr 17 10:45:58 *******.**********.co.uk sshd[16720]: input_userauth_request: invalid user x [preauth]
Apr 17 10:45:58 *******.**********.co.uk sshd[16720]: pam_unix(sshd:auth): check pass; user unknown
Apr 17 10:45:58 *******.**********.co.uk sshd[16720]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser=
Apr 17 10:46:00 *******.**********.co.uk sshd[16724]: Failed password for invalid user space from 45.6.18.28 port 23179 ssh2
Apr 17 10:46:00 *******.**********.co.uk sshd[16720]: Failed password for invalid user x from 165.227.225.195 port 45756 ssh2
Apr 17 10:46:00 *******.**********.co.uk sshd[16720]: Received disconnect from 165.227.225.195 port 45756:11: Bye Bye [preauth]
Apr 17 10:46:00 *******.**********.co.uk sshd[16720]: Disconnected from 165.227.225.195 port 45756 [preauth]
Apr 17 10:46:04 *******.**********.co.uk systemd[1]: Failed to start MySQL Community Server.
-- Subject: Unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has failed.
--
-- The result is failed.
Apr 17 10:46:04 *******.**********.co.uk systemd[1]: mysql.service: Unit entered failed state.
Apr 17 10:46:04 *******.**********.co.uk systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 17 10:46:04 *******.**********.co.uk systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Apr 17 10:46:04 *******.**********.co.uk systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
Apr 17 10:46:04 *******.**********.co.uk systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
Apr 17 10:46:04 *******.**********.co.uk systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Apr 17 10:46:09 *******.**********.co.uk postfix/smtpd[6120]: connect from unknown[45.142.195.2]
Apr 17 10:46:14 *******.**********.co.uk saslauthd[1834]: pam_unix(smtp:auth): check pass; user unknown
Apr 17 10:46:14 *******.**********.co.uk saslauthd[1834]: pam_unix(smtp:auth): authentication failure; logname= uid=0 euid=0 tty= ruser=
Apr 17 10:46:14 *******.**********.co.uk sshd[16787]: Invalid user hhy from 201.48.192.60
Apr 17 10:46:14 *******.**********.co.uk sshd[16787]: input_userauth_request: invalid user hhy [preauth]
Apr 17 10:46:14 *******.**********.co.uk sshd[16787]: pam_unix(sshd:auth): check pass; user unknown
Apr 17 10:46:14 *******.**********.co.uk sshd[16787]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser=
Apr 17 10:46:15 *******.**********.co.uk postfix/anvil[5663]: statistics: max connection rate 2/60s for (smtp:45.142.195.2) at Apr 17 10
Apr 17 10:46:15 *******.**********.co.uk postfix/anvil[5663]: statistics: max connection count 1 for (smtp:45.142.195.2) at Apr 17 10:36
Apr 17 10:46:15 *******.**********.co.uk postfix/anvil[5663]: statistics: max cache size 3 at Apr 17 10:37:12
Apr 17 10:46:15 *******.**********.co.uk saslauthd[1834]: DEBUG: auth_pam: pam_authenticate failed: Authentication failure
Apr 17 10:46:15 *******.**********.co.uk saslauthd[1834]: do_auth         : auth failure: [user=apple@co.uk] [service=smtp] [realm=co.uk
Apr 17 10:46:15 *******.**********.co.uk postfix/smtpd[6120]: warning: unknown[45.142.195.2]: SASL LOGIN authentication failed: authenti
Apr 17 10:46:15 *******.**********.co.uk sshd[16787]: Failed password for invalid user hhy from 201.48.192.60 port 46535 ssh2
Apr 17 10:46:16 *******.**********.co.uk sshd[16787]: Received disconnect from 201.48.192.60 port 46535:11: Bye Bye [preauth]
Apr 17 10:46:16 *******.**********.co.uk sshd[16787]: Disconnected from 201.48.192.60 port 46535 [preauth]
Apr 17 10:46:16 *******.**********.co.uk postfix/smtpd[6120]: disconnect from unknown[45.142.195.2] ehlo=1 auth=0/1 rset=1 quit=1 comman
Apr 17 10:46:20 *******.**********.co.uk postfix/smtpd[16803]: warning: hostname ip-38-83.ZervDNS does not resolve to address 92.118.38.
Apr 17 10:46:20 *******.**********.co.uk postfix/smtpd[16803]: connect from unknown[92.118.38.83]
Apr 17 10:46:27 *******.**********.co.uk sshd[16813]: Invalid user bl from 129.211.26.12
Apr 17 10:46:27 *******.**********.co.uk sshd[16813]: input_userauth_request: invalid user bl [preauth]
Apr 17 10:46:27 *******.**********.co.uk sshd[16813]: pam_unix(sshd:auth): check pass; user unknown
Apr 17 10:46:27 *******.**********.co.uk sshd[16813]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser=
Apr 17 10:46:28 *******.**********.co.uk sshd[16813]: Failed password for invalid user bl from 129.211.26.12 port 51562 ssh2
Apr 17 10:46:29 *******.**********.co.uk sshd[16813]: Received disconnect from 129.211.26.12 port 51562:11: Bye Bye [preauth]
Apr 17 10:46:29 *******.**********.co.uk sshd[16813]: Disconnected from 129.211.26.12 port 51562 [preauth]
Apr 17 10:46:30 *******.**********.co.uk sudo[16835]: userwithsudo : TTY=pts/0 ; PWD=/home/userwithsudo ; USER=root ; COMMAND=/bin/journalctl -x
Apr 17 10:46:30 *******.**********.co.uk sudo[16835]: pam_unix(sudo:session): session opened for user root by userwithsudo(uid=0)
1 comment
  • I had remedied the situation (or so I thought) by adding tighter SSH rules to the firewall.

    This morning, however, MySQL isn’t working again.

    On running service mysql restart I get:

    Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
    

    Then systemctl status mysql.service

    mysql.service - MySQL Community Server
       Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
       Active: activating (start-post) (Result: exit-code) since Tue 2020-04-21 07:21:01 UTC; 19s ago
      Process: 3479 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
      Process: 3471 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
     Main PID: 3479 (code=exited, status=1/FAILURE);         : 3480 (mysql-systemd-s)
        Tasks: 2
       Memory: 232.0K
          CPU: 342ms
       CGroup: /system.slice/mysql.service
               └─control
                 ├─3480 /bin/bash /usr/share/mysql/mysql-systemd-start post
                 └─3526 sleep 1
    

    Then

    sudo journalctl -xe gives

    Apr 21 07:23:33 staging.••••••••••••.co.uk systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
    Apr 21 07:23:33 staging.••••••••••••.co.uk systemd[1]: Stopped MySQL Community Server.
    -- Subject: Unit mysql.service has finished shutting down
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit mysql.service has finished shutting down.
    Apr 21 07:23:33 staging.••••••••••••.co.uk systemd[1]: Starting MySQL Community Server...
    -- Subject: Unit mysql.service has begun start-up
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit mysql.service has begun starting up.
    Apr 21 07:23:34 staging.••••••••••••.co.uk systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
    Apr 21 07:24:04 staging.••••••••••••.co.uk systemd[1]: Failed to start MySQL Community Server.
    -- Subject: Unit mysql.service has failed
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit mysql.service has failed.
    --
    -- The result is failed.
    Apr 21 07:24:04 staging.••••••••••••.co.uk systemd[1]: mysql.service: Unit entered failed state.
    Apr 21 07:24:04 staging.••••••••••••.co.uk systemd[1]: mysql.service: Failed with result 'exit-code'.
    Apr 21 07:24:04 staging.••••••••••••.co.uk systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
    Apr 21 07:24:04 staging.••••••••••••.co.uk systemd[1]: Stopped MySQL Community Server.
    -- Subject: Unit mysql.service has finished shutting down
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit mysql.service has finished shutting down.
    Apr 21 07:24:04 staging.••••••••••••.co.uk systemd[1]: Starting MySQL Community Server...
    -- Subject: Unit mysql.service has begun start-up
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit mysql.service has begun starting up.
    Apr 21 07:24:04 staging.••••••••••••.co.uk systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
    Apr 21 07:24:34 staging.••••••••••••.co.uk systemd[1]: Failed to start MySQL Community Server.
    -- Subject: Unit mysql.service has failed
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit mysql.service has failed.
    --
    -- The result is failed.
    Apr 21 07:24:34 staging.••••••••••••.co.uk systemd[1]: mysql.service: Unit entered failed state.
    Apr 21 07:24:34 staging.••••••••••••.co.uk systemd[1]: mysql.service: Failed with result 'exit-code'.
    Apr 21 07:24:34 staging.••••••••••••.co.uk systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
    Apr 21 07:24:34 staging.••••••••••••.co.uk systemd[1]: Stopped MySQL Community Server.
    -- Subject: Unit mysql.service has finished shutting down
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit mysql.service has finished shutting down.
    Apr 21 07:24:34 staging.••••••••••••.co.uk systemd[1]: Starting MySQL Community Server...
    -- Subject: Unit mysql.service has begun start-up
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit mysql.service has begun starting up.
    Apr 21 07:24:35 staging.••••••••••••.co.uk systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
    Apr 21 07:24:49 staging.••••••••••••.co.uk sudo[4069]: userwithsudo : TTY=pts/0 ; PWD=/home/userwithsudo ; USER=root ; COMMAND=/bin/journalctl -
    

    Here’s what’s at the end of the MySQL Error log

    2020-04-21T07:27:37.888005Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
    2020-04-21T07:27:38.066341Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_times$
    2020-04-21T07:27:38.068693Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.29-0ubuntu0.16.04.1) starting as process 4532 ...
    2020-04-21T07:27:38.078617Z 0 [Note] InnoDB: PUNCH HOLE support available
    2020-04-21T07:27:38.078650Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2020-04-21T07:27:38.078661Z 0 [Note] InnoDB: Uses event mutexes
    2020-04-21T07:27:38.078670Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
    2020-04-21T07:27:38.078679Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
    2020-04-21T07:27:38.078687Z 0 [Note] InnoDB: Using Linux native AIO
    2020-04-21T07:27:38.079055Z 0 [Note] InnoDB: Number of pools: 1
    2020-04-21T07:27:38.079195Z 0 [Note] InnoDB: Not using CPU crc32 instructions
    2020-04-21T07:27:38.081240Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
    2020-04-21T07:27:38.081295Z 0 [ERROR] InnoDB: mmap(136151040 bytes) failed; errno 12
    2020-04-21T07:27:38.081311Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
    2020-04-21T07:27:38.081322Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
    2020-04-21T07:27:38.081330Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
    2020-04-21T07:27:38.081337Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
    2020-04-21T07:27:38.081344Z 0 [ERROR] Failed to initialize builtin plugins.
    2020-04-21T07:27:38.081350Z 0 [ERROR] Aborting
    
    2020-04-21T07:27:38.081358Z 0 [Note] Binlog end
    2020-04-21T07:27:38.081422Z 0 [Note] Shutting down plugin 'MyISAM'
    2020-04-21T07:27:38.082208Z 0 [Note] /usr/sbin/mysqld: Shutdown complete```
    

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer

@paulmist

please check the tail of the mysql error log and post here if it doesn’t make any sense. The location should be in the my.cnf file or by default in /var/log.

BR

Andrew

Submit an Answer