MariaDB stopped working until restart

June 19, 2017 2.2k views
MariaDB MySQL Ubuntu 16.04
DigitalCarrot
By:
DigitalCarrot

Today our website displayed a "Database connection error"

Via the command line I tried to connect to MariaDB and it failed:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")

I did:

service mysql restart

The service restarted and I was able to connect again.

The mysql/mariadb error logs are empty. Where can I look to find out what happened?

EDIT

OK I think I answered my own question, after looking in syslog:

Jun 17 12:12:42 svr kernel: [751960.001590] Out of memory: Kill process 1644 (mysqld) score 160 or sacrifice child
Jun 17 12:12:42 svr kernel: [751960.002941] Killed process 1644 (mysqld) total-vm:602692kB, anon-rss:79864kB, file-rss:0kB
Jun 17 12:12:42 svr mysqld_safe: Number of processes running now: 0
Jun 17 12:12:42 svr mysqld_safe: mysqld restarted
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [Note] /usr/sbin/mysqld (mysqld 10.0.29-MariaDB-0ubuntu0.16.04.1) starting as process 3572 ...
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [ERROR] mysqld: Out of memory (Needed 128909312 bytes)
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [Note] InnoDB: The InnoDB memory heap is disabled
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [Note] InnoDB: Compressed tables use zlib 1.2.8
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [Note] InnoDB: Using Linux native AIO
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [Note] InnoDB: Using CPU crc32 instructions
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [Note] InnoDB: Initializing buffer pool, size = 128.0M
Jun 17 12:12:42 svr mysqld: InnoDB: mmap(139722752 bytes) failed; errno 12
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [ERROR] Plugin 'InnoDB' init function returned error.
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [Note] Plugin 'FEEDBACK' is disabled.
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [ERROR] Unknown/unsupported storage engine: InnoDB
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [ERROR] Aborting
Jun 17 12:12:42 svr mysqld:
Jun 17 12:12:42 svr mysqld: 170617 13:12:42 [Note] /usr/sbin/mysqld: Shutdown complete
Jun 17 12:12:42 svr mysqld:
Jun 17 12:12:42 svr mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended

Looks like we need a bigger droplet?

Log In to Comment
2 Answers
hansen June 19, 2017
Accepted Answer

Hi @DigitalCarrot
Yes, total system ran out of RAM, so resize the droplet up. And it also looks like your database storage is actually 140MB, but the buffer pool size is the default 128MB, which means it's swapping constantly to disk, making it much slower. Your pool should be greater than the amount of data.

Reply
  • DigitalCarrot June 19, 2017

    Thanks @hansen I will scale the droplet up, and come back to you on the buffer pool size if I need help with that. Thanks for pointing that out.

etlekana June 26, 2017

Hi @DigitalCarrot

I had same issues on a low memory droplet, and i read up on it.

This should be a solution for the problem, i am still testing if it is working on 3 different droplets atm.

https://mariadb.com/resources/blog/starting-mysql-low-memory-virtual-machines

Hope this helps :)

Kind Regards
Etlekana

Reply
  • DigitalCarrot July 13, 2017

    Thanks @etlekana

    However, I added the following to my apache2.conf:

    <files xmlrpc.php>
        order allow,deny
        deny from all
</files>

    My issues with MariaDB running out of RAM seem to have disappeared after doing this, so I suspect it could have been an XML RPC attack against WordPress.

Have another answer? Share your knowledge.

Related Questions