I have two Wordpress websites running off a single droplet. They are on LEMP stack.
I hadn’t checked them in a while but when I did, I found that one of them displayed the message:
Error Establishing Database Connection after disk filled up
I opened up the Digital Ocean control panel and realized that I was using 100% disk space. Upon some investigation, I found that it was because a Wordpress plugin called ‘Updraft’ was saving larger and larger db backups in the content folder.
After backing up those backups to my computer, I deleted them all from the server.
Then I went to this tutorial and started following the steps.
sudo systemctl start mysql Didn’t work.
I then created
/etc/my.cnf and added the lines
[mysql] innodb_force_recovery = 6
I still got the same error.
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
This is the result of
systemctl status mysql.service:
mysql.service - MySQL Community Server Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled) Active: activating (start-post) (Result: exit-code) since Mon 2021-01-25 02:01:19 UTC; 12s ago Process: 12744 ExecStart=/usr/sbin/mysqld (code=exited, status=2) Process: 12736 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS) Main PID: 12744 (code=exited, status=2); : 12745 (mysql-systemd-s) Tasks: 2 Memory: 224.0K CPU: 438ms CGroup: /system.slice/mysql.service └─control ├─12745 /bin/bash /usr/share/mysql/mysql-systemd-start post └─12789 sleep 1
This is the result of
sudo journalctl -xe:
Jan 25 02:02:20 BlueAdapterS2 audit: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/etc/ Jan 25 02:02:20 BlueAdapterS2 kernel: audit: type=1400 audit(1611540140.858:131): apparmor="DENIED" operation="open" prof Jan 25 02:02:21 BlueAdapterS2 systemd: mysql.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jan 25 02:02:25 BlueAdapterS2 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=b2:a6:ac:23:31:4a:80:7f:f8:66:e8:30:08:00 SRC=34.77.93 Jan 25 02:02:32 BlueAdapterS2 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=b2:a6:ac:23:31:4a:18:2a:d3:e0:df:f0:08:00 SRC=194.26.2 Jan 25 02:02:49 BlueAdapterS2 sudo: kev : TTY=pts/0 ; PWD=/home/kev ; USER=root ; COMMAND=/bin/journalctl -xe Jan 25 02:02:49 BlueAdapterS2 sudo: pam_unix(sudo:session): session opened for user root by kev(uid=0)
Finally, here is part of the mysql log:
2021-01-25T01:23:43.119785Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000) 2021-01-25T01:23:43.119858Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000) 2021-01-25T01:23:43.284921Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2021-01-25T01:23:43.287155Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.31-0ubuntu0.16.04.1) starting as process 5405 ... 2021-01-25T01:23:43.293301Z 0 [Note] InnoDB: PUNCH HOLE support available 2021-01-25T01:23:43.293342Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-01-25T01:23:43.293351Z 0 [Note] InnoDB: Uses event mutexes 2021-01-25T01:23:43.293358Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2021-01-25T01:23:43.293365Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8 2021-01-25T01:23:43.293373Z 0 [Note] InnoDB: Using Linux native AIO 2021-01-25T01:23:43.293664Z 0 [Note] InnoDB: Number of pools: 1 2021-01-25T01:23:43.293780Z 0 [Note] InnoDB: Not using CPU crc32 instructions 2021-01-25T01:23:43.295277Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2021-01-25T01:23:43.308704Z 0 [Note] InnoDB: Completed initialization of buffer pool 2021-01-25T01:23:43.310582Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-01-25T01:23:43.322585Z 0 [Note] InnoDB: Highest supported file format is Barracuda. 2021-01-25T01:23:43.323343Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 32014615639 2021-01-25T01:23:43.445307Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 32019858432 2021-01-25T01:23:43.476823Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 32022746925 2021-01-25T01:23:43.476920Z 0 [Note] InnoDB: Database was not shutdown normally! 2021-01-25T01:23:43.476926Z 0 [Note] InnoDB: Starting crash recovery. 2021-01-25T01:23:43.505719Z 0 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 2021-01-25 01:23:43 0xa42a8b40 InnoDB: Assertion failure in thread 2754251584 in file log0recv.cc line 2078 InnoDB: Failing assertion: !page || (ibool)!!page_is_comp(page) == dict_table_is_comp(index->table) InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 01:23:43 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail. key_buffer_size=16777216 read_buffer_size=131072 max_used_connections=0 max_threads=151 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 75719 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x30000 /usr/sbin/mysqld(my_print_stacktrace+0x3c)[0x8a906ec] /usr/sbin/mysqld(handle_fatal_signal+0x32a)[0x836d58a] [0xb7714bb4] [0xb7714bd1] /lib/i386-linux-gnu/libc.so.6(gsignal+0x39)[0xb7094eb9] /lib/i386-linux-gnu/libc.so.6(abort+0x157)[0xb7096417] /usr/sbin/mysqld[0x8343b13] /usr/sbin/mysqld[0x8b7cfb0] 57 58 59 /usr/sbin/mysqld(_Z22recv_recover_page_funcmP11buf_block_t+0x863)[0x8b7dbc3] /usr/sbin/mysqld(_Z20buf_page_io_completeP10buf_page_tb+0x3d9)[0x8d04b39] /usr/sbin/mysqld(_Z12fil_aio_waitm+0x125)[0x8d83695] /usr/sbin/mysqld(io_handler_thread+0xc1)[0x8c5f261] /lib/i386-linux-gnu/libpthread.so.0(+0x6295)[0xb740f295] /lib/i386-linux-gnu/libc.so.6(clone+0x6e)[0xb71501ae] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.
I’m at a complete loss on how to get the database up and running again. Any help would be very much appreciated.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.
Click below to sign up and get $100 of credit to try our products over 60 days!