Related

Product Now Available: Managed MySQL Databases Product
NGINX Config Generator Tool
Connect to DB installed into dedicated VPS (private networking) Question Question
my site i down please see http://www.cheri3a.com/ Question Question

Question

centos 8 mysql keeps stoped status

Posted October 19, 2020 274 views
NginxMySQLCentOS 8

mysql craahed tarting today, and it keeps down.
what I get if I want to restar mysql service

systemctl restart mysqld
Job for mysqld.service failed because the control process exited with error code.
See "systemctl status mysqld.service" and "journalctl -xe" for details.

my error log

2020-10-19T09:10:14.068125Z 0 [System] [MY-010116] [Server] /usr/libexec/mysqld (mysqld 8.0.21) starting as process 320061
2020-10-19T09:10:14.088157Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2020-10-19T09:10:14.145172Z 0 [ERROR] [MY-012681] [InnoDB] mmap(137363456 bytes) failed; errno 12
2020-10-19T09:10:14.145376Z 1 [ERROR] [MY-012956] [InnoDB] Cannot allocate memory for the buffer pool
2020-10-19T09:10:14.145460Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2020-10-19T09:10:14.145573Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2020-10-19T09:10:14.145831Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2020-10-19T09:10:14.145973Z 0 [ERROR] [MY-010119] [Server] Aborting
2020-10-19T09:10:14.146611Z 0 [System] [MY-010910] [Server] /usr/libexec/mysqld: Shutdown complete (mysqld 8.0.21)  Source distribution.


journalctl -xe

Oct 19 09:11:03 centos-s-1vcpu-1gb-sgp1-01 sshd[320118]: Invalid user 111111 from 185.153.196.230 port 32742
Oct 19 09:11:03 centos-s-1vcpu-1gb-sgp1-01 sshd[320118]: pam_unix(sshd:auth): check pass; user unknown
Oct 19 09:11:03 centos-s-1vcpu-1gb-sgp1-01 sshd[320118]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=185.153.196.230
Oct 19 09:11:05 centos-s-1vcpu-1gb-sgp1-01 sshd[320118]: Failed password for invalid user 111111 from 185.153.196.230 port 32742 ssh2
Oct 19 09:11:06 centos-s-1vcpu-1gb-sgp1-01 sshd[320118]: pam_unix(sshd:auth): check pass; user unknown
Oct 19 09:11:08 centos-s-1vcpu-1gb-sgp1-01 sshd[320118]: Failed password for invalid user 111111 from 185.153.196.230 port 32742 ssh2
Oct 19 09:11:09 centos-s-1vcpu-1gb-sgp1-01 sshd[320118]: Disconnecting invalid user 111111 185.153.196.230 port 32742: Change of username or service not allowed: (111111,ssh-connection) -> (admin,ssh-connectio>
Oct 19 09:11:09 centos-s-1vcpu-1gb-sgp1-01 sshd[320118]: PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=185.153.196.230

from the error log looks like a memory issue so i run free -mh command

free -mh
              total        used        free      shared  buff/cache   available
Mem:          818Mi       296Mi       291Mi       101Mi       230Mi       297Mi
Swap:            0B          0B          0B

I used of 296m but i cache 230m i should supposed to have decent amount of memory I can used am I right ?
https://www.digitalocean.com/community/questions/out-of-memory-is-it-possible

from the answer of this post , this is the what he said 

To expand on the “used” RAM above, simply because I’ve “used” 1.9GB out of the 2GB I’ve been allocated does not mean that I’ve used that much. The 1.6GB’s from the cache column is included in that 1.9GB’s, so my actual usage is only ~300MB. The other 1.3GB is cached and used by current processes until they are either stopped, killed or restarted

but once i take a close look at journalctl -xe log , there is a user 111111 trying to login with wrong password ??

Failed password for invalid user 111111 from 185.153.196.230 port 32742 ssh2

I dont know if this is actually a memory problem or another issues ??
hopefully someone have infra knowledge can advise me on this .thanks
now all of my site is down , I am using the cheapest droplet instance. which have 1G memory.
btw , I am using centos8 nginx, php , mysql stack.thanks again.

Add a comment
stupidboy
By:
stupidboy

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
3 answers
KFSys October 19, 2020

Hi @stupidboy,

It’s possible you have some misconfiguration in your configuration file - /etc/my.cnf.

Can you please post try with a default configuration file to start MySQL to see if the same errors will be presented?

Another solution would be to try and configure a SWAP file. I know you have enough memory but more often than not it will help out in such situations.

Reply Report
  • stupidboy October 19, 2020

    Hi thanks for prompt reply very appreciated (bow)
    this is what currently inside the /etc/my.cnf

    #
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

    cuurently those directory inside my.conf.d
    -rw-r–r–. 1 root root 295 May 14 2019 client.cnf
    -rw-r–r–. 1 root root 565 Sep 15 14:56 mysql-default-authentication-plugin.cnf
    -rw-r–r–. 1 root root 612 Sep 15 15:03 mysql-server.cnf

    inside mysql-server.cnf

#
# This group are read by MySQL server.
# Use it for options that only the server (but not clients) should see
#
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/en/server-configuration-defaults.html

# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mysqld according to the
# instructions in http://fedoraproject.org/wiki/Systemd

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysql/mysqld.log
pid-file=/run/mysqld/mysqld.pid

    thanks

    Reply Report
    • KFSys October 20, 2020

      Everything looks okay. This is strange!

      It seems like you’ll need to try and start MySQL in an insecure mod to see if it works or not. 

      mysql --initialize-insecure
      Reply Report
      • stupidboy October 20, 2020

        hi @KFSys I just tried but still same error . any possible root cause for this ? thanks 

        [an@centos-s-1vcpu-1gb-sgp1-01 ~]# mysql --initialize-insecure
mysql: [ERROR] unknown option '--initialize-insecure'.
[an@centos-s-1vcpu-1gb-sgp1-01 ~]# mysqld --initialize-insecure
[an@centos-s-1vcpu-1gb-sgp1-01 ~]# systemctl restart mysqld
Job for mysqld.service failed because the control process exited with error code.
See "systemctl status mysqld.service" and "journalctl -xe" for details.

        very annoying maybe need to destroy the droplet and try deploy again since I am not very a infra guy.
        (headache)

        Reply Report
pistle2020 October 19, 2020

The answer depends on the question System OS (CentOS) and MySQL Mariadb service

Solution: Just add some seconds before make service start - in my case I’ve added 45 seconds

Add the following lines under [service] section in {/etc/systemd/system/multi-user.target.wants/mariadb.service} - Of course the path depends on your system OS and MySQL service name (not all have mariadb.service)

Restart=always
RestartSec=45s
Do NOT forget to run the following commands

sudo systemctl daemon-reload

sudo systemctl restart mariadb.service

Reply Report
alexdo October 26, 2020

Hello,

The key error here is:

InnoDB: Fatal error: cannot allocate memory for the buffer pool

This crash is due to your system running out of memory. I’d suggest that you add a swap file to give yourself a bit more of a buffer. Check out this tutorial:

https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04

What you can also do is to use the MySQLTuner script.

The MySQLTuner is a script written in Perl and allows you to quickly test your MySQL configuration and it gives you suggestions for adjustments to increase performance and stability.

According to the official GitHub page, it supports 300 indicators for MySQL/MariaDB/Percona Server in this last version.

To run the script you could do the following:

  • SSH to your Droplet
  • Download the script:
wget http://mysqltuner.pl/ -O mysqltuner.pl
  • Then execute it:
perl mysqltuner.pl

The script would run multiple checks against your MySQL instance, all checks done by MySQLTuner are documented here.

Also as stated in the official documentation, it is still extremely important for you to fully understand each change you make to a MySQL database server. If you don’t understand portions of the script’s output, or if you don’t understand the recommendations, you should consult a knowledgeable DBA or system administrator that you trust.

As a good practice make sure to always test your changes on staging environments before implementing them on your production database.

On the same note, if you want to have a worry-free MySQL hosting and focus on your application, I would recommend trying out the DigitalOcean Managed Databases:

https://www.digitalocean.com/products/managed-databases-mysql/

This was mini tutorial was posted from bobbyiliev in this question in our community: https://www.digitalocean.com/community/questions/how-to-tweak-mysql-mariadb-configuration-for-increased-performance-and-stability

You can also create a simple bash script to check if MySQL is running and if not to restart it.

#!/bin/bash

# Check if MySQL is running
sudo service mysql status > /dev/null 2>&1

# Restart the MySQL service if it's not running.
if [ $? != 0 ]; then
    sudo service mysql restart
fi

Run this script every 5 minutes using a cron job like this one:

 */5 * * * * /home/user/scripts/monitor.sh > /dev/null 2>&1

Hope that this helps!
Regards,
Alex

How To Add Swap Space on Ubuntu 20.04
by Brian Boucheron
One way to guard against out-of-memory errors in applications is to add some swap space to your server. In this guide, we will cover how to add a swap file to an Ubuntu 20.04 server.
Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help