Report this

What is the reason for this report?

Unsolveable epic MySQL crashes with Wordpress

Posted on August 24, 2014

Hi everybody, sorry about the dramatic title, but I’m truly desperate about an issue in my droplet with LAMP and Wordpress installed that seems to be really unsolveable.

It started in the beginning of 2014, when I tried to access my blog I got an “Error Establishing a Database Connection”. I searched on Google and here in Digital Ocean community and I found a lot of tutorials and fix strategies.

Basically, something happens on Apache that makes it overload MySQL (as I imagine). Once MySQL can’t allocate enough RAM to do whatever Apache wants, it crashes and my blog stays unavailable.

Well, I used to have a 500MB droplet (the cheapest one), and the first try was on the [swap files](http:// https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-12-04). It apparently solved the problem and the blog stayed online for one week without crashing. And then, suddenly, it started again. It started crashing and even when I restarted it all, it crashed again. In this week I had no traffic changes and no new posts/new plugins. Nothing changed on my blog.

A couple of weeks later, I tried a lot of possible solutions, like this and this. I tried also a tunning in MySQL. Nothing helped.

I tried playing with max_connections, thread_cache_size and other MySQL config params… nothing helped. After two months of daily tries, I quit and opened a ticket at Digital Ocean. The only answer? You have to upgrade your droplet for a 1GB droplet. I wasn’t believing it was that because how can a droplet without no traffic and only a few posts could demand a 1GB droplet? But I had nothing else to try and I upgraded.

The Wordpress stayed alive for two weeks! Yay! That was the solution, right!? No! It started again with the exactly same error (I’ll show the logs on the end of this post).

For the last tries, I installed Wordpress W3 Cache and some plugins to relieve MySQL calls. It wasn’t the solution neither.

Now I’m lost and have anything else to do. I won’t upgrade to a 2GB RAM because the traffic and the number os posts on my blog didn’t increase. What I did yesterday? I created a script in a crontab to restart MySQL and Apache each 15 minutes. I know it’s the ugliest thing to do, but I think this problem won’t solve.

Wanna know the good news? IT DIDN’T SOLVE ALSO! LOL! MySQL keeps crashing and doesn’t come back even with a script doing exactly the steps I do to restart MySQL and Apache2.

So, I believe you understand why I’m desperate, right? Sorry about this long testimonial, but now that I have nothing to work on my blog, since it doesn’t work, I have a lot of free time to write.

Here are the logs on Apache2 and MySQL:

Apache2

[Sun Aug 24 08:42:05 2014] [notice] child pid 17841 exit signal Segmentation fault (11)
[Sun Aug 24 08:42:11 2014] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Sun Aug 24 08:46:48 2014] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Sun Aug 24 08:46:59 2014] [notice] child pid 17900 exit signal Segmentation fault (11)
[Sun Aug 24 08:48:36 2014] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Sun Aug 24 08:48:57 2014] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Sun Aug 24 08:49:07 2014] [notice] child pid 17927 exit signal Segmentation fault (11)
[Sun Aug 24 08:49:07 2014] [notice] child pid 17928 exit signal Segmentation fault (11)
[Sun Aug 24 08:49:07 2014] [notice] child pid 17929 exit signal Segmentation fault (11)
[Sun Aug 24 08:49:07 2014] [notice] child pid 17930 exit signal Segmentation fault (11)
[Sun Aug 24 08:55:15 2014] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Sun Aug 24 08:55:42 2014] [notice] child pid 18022 exit signal Segmentation fault (11)
[Sun Aug 24 08:56:01 2014] [error] (12)Cannot allocate memory: fork: Unable to fork new process

MySQL

140824  9:06:10 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140824  9:06:10 [Note] Plugin 'FEDERATED' is disabled.
140824  9:06:10 InnoDB: The InnoDB memory heap is disabled
140824  9:06:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140824  9:06:10 InnoDB: Compressed tables use zlib 1.2.3.4
140824  9:06:10 InnoDB: Initializing buffer pool, size = 64.0M
InnoDB: mmap(68681728 bytes) failed; errno 12
140824  9:06:10 InnoDB: Completed initialization of buffer pool
140824  9:06:10 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140824  9:06:10 [ERROR] Plugin 'InnoDB' init function returned error.
140824  9:06:10 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140824  9:06:10 [ERROR] Unknown/unsupported storage engine: InnoDB
140824  9:06:10 [ERROR] Aborting

140824  9:06:10 [Note] /usr/sbin/mysqld: Shutdown complete

140824  9:06:10 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140824  9:06:10 [Note] Plugin 'FEDERATED' is disabled.
140824  9:06:10 InnoDB: The InnoDB memory heap is disabled
140824  9:06:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140824  9:06:10 InnoDB: Compressed tables use zlib 1.2.3.4
140824  9:06:10 InnoDB: Initializing buffer pool, size = 64.0M
InnoDB: mmap(68681728 bytes) failed; errno 12
140824  9:06:10 InnoDB: Completed initialization of buffer pool
140824  9:06:10 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140824  9:06:10 [ERROR] Plugin 'InnoDB' init function returned error.
140824  9:06:10 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140824  9:06:10 [ERROR] Unknown/unsupported storage engine: InnoDB
140824  9:06:10 [ERROR] Aborting

140824  9:06:10 [Note] /usr/sbin/mysqld: Shutdown complete

Thank you.



This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Probably you have a XML-RPC attack toward your wordpress. The most common visual cue of this would be an “Error connecting to database” message. The web console may also display “Out of Memory” errors.

Solution: https://www.digitalocean.com/community/tutorials/how-to-protect-wordpress-from-xml-rpc-attacks-on-ubuntu-14-04

Or if the above method is not working for you, you can install “Wordfence” plugin for protection. It’s pretty great. https://wordpress.org/plugins/wordfence/

I had a similar problem and was able to finally fix it.

Background

I tried a bunch of tutorials that recommended tuning my MySQL installation and Apache configuration with no luck.

Identifying the Problem

It turns out that my site was being subjected to a brute force attack that targeted the WordPress xmlrpc.php file. H/T @sierracircle.

In order to check for this, you should view your apache2 access logs and look for suspicious POST requests. In my case, I found a huge number of POST requests to the /xmlrpc.php file coming from the same IP address (about 2 or 3 requests per second).

cd /var/log/apache2/
cat access.log

Solution

In order to fix this, I banned the offending IP address via Iptables:

Ban IP the malicious IP address (replace example with whatever IP address you want to ban) :

iptables -A INPUT -s 46.166.139.20 -j DROP

To view blocked IP address:

iptables -L INPUT -v -n

Reference: http://www.cyberciti.biz/faq/linux-howto-check-ip-blocked-against-iptables/

The developer cloud

Scale up as you grow — whether you're running one virtual machine or ten thousand.

Get started for free

Sign up and get $200 in credit for your first 60 days with DigitalOcean.*

*This promotional offer applies to new accounts only.