How to solve MariaDB memory issue

March 8, 2019 2.7k views
MariaDB Debian 9

Hello,

Suddenly, MariaDB stopped working and was unable to start. The logs reffer to some memory issue related to InnoDB. After several hours trying to solve it reading some forums and sources of information, the mysql sever is still down, unable to start.

Maybe this terminal output can help.

mariadb.service - MariaDB 10.1.37 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
           └─oom.conf
Active: failed (Result: exit-code) since Fri 2019-03-08 19:35:41 UTC; 18s ago
Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
Process: 14298 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION 
Process: 14186 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bi
Process: 14182 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exi
Process: 14178 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exit
Main PID: 14298 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"

Mar 08 19:35:40 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Starting MariaDB 10.1.37 database ser
Mar 08 19:35:41 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[14298]: 2019-03-08 19:35:41 13967806592249
Mar 08 19:35:41 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Main process exited,
Mar 08 19:35:41 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Failed to start MariaDB 10.1.37 datab
Mar 08 19:35:41 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Unit entered failed 
Mar 08 19:35:41 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Failed with result 

Thank you very much in advance.

1 comment
  • Hola! It looks like the systemctl output is cut off. Try this to get the full output:

    journalctl -u mariadb --no-pager
    
3 Answers

Oh, thank you! Here goes the new output:

-- Logs begin at Fri 2019-03-08 17:45:38 UTC, end at Fri 2019-03-08 20:14:01 UTC. --
Mar 08 17:52:35 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Starting MariaDB 10.1.37 database server…
Mar 08 17:52:36 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 17:52:36 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Failed to start MariaDB 10.1.37 database server.
Mar 08 17:52:36 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Unit entered failed state.
Mar 08 17:52:36 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Mar 08 17:54:12 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Starting MariaDB 10.1.37 database server…
Mar 08 17:54:13 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 17:54:13 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Failed to start MariaDB 10.1.37 database server.
Mar 08 17:54:13 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Unit entered failed state.
Mar 08 17:54:13 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Mar 08 18:25:01 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Starting MariaDB 10.1.37 database server…
Mar 08 18:25:01 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[11526]: 2019-03-08 18:25:01 140492701168064 [Note] Using unique option prefix 'host' is error-prone and can break in the future. Please use the full name 'host_cache_size' instead.
Mar 08 18:25:01 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[11526]: Unknown suffix '.' used for variable 'host_cache_size' (value '127.0.0.1')
Mar 08 18:25:01 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[11526]: 2019-03-08 18:25:01 140492701168064 [ERROR] /usr/sbin/mysqld: Error while setting value '127.0.0.1' to 'host_cache_size'
Mar 08 18:25:01 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[11526]: 2019-03-08 18:25:01 140492701168064 [ERROR] Aborting
Mar 08 18:25:01 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 18:25:01 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Failed to start MariaDB 10.1.37 database server.
Mar 08 18:25:01 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Unit entered failed state.
Mar 08 18:25:01 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Mar 08 18:29:32 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Starting MariaDB 10.1.37 database server…
Mar 08 18:29:33 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[11747]: 2019-03-08 18:29:33 140157977944512 [Note] Using unique option prefix 'host' is error-prone and can break in the future. Please use the full name 'host_cache_size' instead.
Mar 08 18:29:33 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[11747]: Unknown suffix '.' used for variable 'host_cache_size' (value '127.0.0.1')
Mar 08 18:29:33 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[11747]: 2019-03-08 18:29:33 140157977944512 [ERROR] /usr/sbin/mysqld: Error while setting value '127.0.0.1' to 'host_cache_size'
Mar 08 18:29:33 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[11747]: 2019-03-08 18:29:33 140157977944512 [ERROR] Aborting
Mar 08 18:29:33 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 18:29:33 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Failed to start MariaDB 10.1.37 database server.
Mar 08 18:29:33 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Unit entered failed state.
Mar 08 18:29:33 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Mar 08 18:35:28 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Starting MariaDB 10.1.37 database server…
Mar 08 18:35:29 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[11988]: 2019-03-08 18:35:29 140150647464384 [Note] /usr/sbin/mysqld (mysqld 10.1.37-MariaDB-0+deb9u1) starting as process 11988 …
Mar 08 18:35:29 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 18:35:29 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Failed to start MariaDB 10.1.37 database server.
Mar 08 18:35:29 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Unit entered failed state.
Mar 08 18:35:29 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Mar 08 18:46:03 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Starting MariaDB 10.1.37 database server…
Mar 08 18:46:04 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[12410]: 2019-03-08 18:46:04 140313808466368 [Note] /usr/sbin/mysqld (mysqld 10.1.37-MariaDB-0+deb9u1) starting as process 12410 …
Mar 08 18:46:04 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 18:46:04 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Failed to start MariaDB 10.1.37 database server.
Mar 08 18:46:04 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Unit entered failed state.
Mar 08 18:46:04 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Mar 08 18:55:54 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Starting MariaDB 10.1.37 database server…
Mar 08 18:55:54 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[12758]: 2019-03-08 18:55:54 139911274983872 [Note] /usr/sbin/mysqld (mysqld 10.1.37-MariaDB-0+deb9u1) starting as process 12758 …
Mar 08 18:55:55 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 18:55:55 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Failed to start MariaDB 10.1.37 database server.
Mar 08 18:55:55 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Unit entered failed state.
Mar 08 18:55:55 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Mar 08 19:19:17 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Starting MariaDB 10.1.37 database server…
Mar 08 19:19:18 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[13546]: 2019-03-08 19:19:18 140145203065280 [Note] /usr/sbin/mysqld (mysqld 10.1.37-MariaDB-0+deb9u1) starting as process 13546 …
Mar 08 19:19:18 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 19:19:18 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Failed to start MariaDB 10.1.37 database server.
Mar 08 19:19:18 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Unit entered failed state.
Mar 08 19:19:18 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Mar 08 19:29:07 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Starting MariaDB 10.1.37 database server…
Mar 08 19:29:08 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[13983]: 2019-03-08 19:29:08 139703135859136 [Note] /usr/sbin/mysqld (mysqld 10.1.37-MariaDB-0+deb9u1) starting as process 13983 …
Mar 08 19:29:08 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 19:29:08 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Failed to start MariaDB 10.1.37 database server.
Mar 08 19:29:08 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Unit entered failed state.
Mar 08 19:29:08 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Mar 08 19:35:40 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Starting MariaDB 10.1.37 database server…
Mar 08 19:35:41 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[14298]: 2019-03-08 19:35:41 139678065922496 [Note] /usr/sbin/mysqld (mysqld 10.1.37-MariaDB-0+deb9u1) starting as process 14298 …
Mar 08 19:35:41 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 19:35:41 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: Failed to start MariaDB 10.1.37 database server.
Mar 08 19:35:41 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Unit entered failed state.
Mar 08 19:35:41 blaudat-debian-s-3vcpu-1gb-fra1-01 systemd[1]: mariadb.service: Failed with result 'exit-code'.
  • Thank you for that. I’m assuming you’ve already taken care of this since I didn’t see it pop up again later:

    Mar 08 18:29:33 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[11747]: Unknown suffix '.' used for variable 'host_cache_size' (value '127.0.0.1')
    Mar 08 18:29:33 blaudat-debian-s-3vcpu-1gb-fra1-01 mysqld[11747]: 2019-03-08 18:29:33 140157977944512 [ERROR] /usr/sbin/mysqld: Error while setting value '127.0.0.1' to 'host_cache_size'
    

    After that though, the systemd log seems to be pretty empty on usable information. In trying to think what to do next, that led me to what is a surprisingly decent article that I think you might find equally interesting:

    https://mariadb.com/kb/en/library/what-to-do-if-mariadb-doesnt-start/

    Perhaps the actual log, which may perhaps be in /var/log/mysql, will offer more information about why it’s failing to start. The article goes on to talk about some common reasons. Feel free to share your findings here. If myself or someone else can help from what you find, we’ll certainly be glad to. If you find the answer in your search, I’d love to hear about it as well :)

Very nice of you, jarland.

I’m going to read this article and insisting to find some solutions.

Hope to confirm in a short time.

Regards!

Hiii, I found the solution… :)

https://unix.stackexchange.com/questions/413782/mariadb-wont-start-even-after-reinstalling/505221#505221

Delete the file /var/lib/mysql/tc.log reboot the system and start maria db again. Also ensure there is enough space in /var and /tmp if you’re using separate partition for them.

It seems that the /var partition got full and so mariadb failed to create tc.log file and ended up creating a 0 bit file which is causing the problem.

Hope it helps!

Have another answer? Share your knowledge.