Question

Unsolveable epic MySQL crashes with Wordpress

  • Posted on August 24, 2014
  • tostringAsked by tostring

Hi everybody, sorry about the dramatic title, but I’m truly desperate about an issue in my droplet with LAMP and Wordpress installed that seems to be really unsolveable.

It started in the beginning of 2014, when I tried to access my blog I got an “Error Establishing a Database Connection”. I searched on Google and here in Digital Ocean community and I found a lot of tutorials and fix strategies.

Basically, something happens on Apache that makes it overload MySQL (as I imagine). Once MySQL can’t allocate enough RAM to do whatever Apache wants, it crashes and my blog stays unavailable.

Well, I used to have a 500MB droplet (the cheapest one), and the first try was on the [swap files](http:// https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-12-04). It apparently solved the problem and the blog stayed online for one week without crashing. And then, suddenly, it started again. It started crashing and even when I restarted it all, it crashed again. In this week I had no traffic changes and no new posts/new plugins. Nothing changed on my blog.

A couple of weeks later, I tried a lot of possible solutions, like this and this. I tried also a tunning in MySQL. Nothing helped.

I tried playing with max_connections, thread_cache_size and other MySQL config params… nothing helped. After two months of daily tries, I quit and opened a ticket at Digital Ocean. The only answer? You have to upgrade your droplet for a 1GB droplet. I wasn’t believing it was that because how can a droplet without no traffic and only a few posts could demand a 1GB droplet? But I had nothing else to try and I upgraded.

The Wordpress stayed alive for two weeks! Yay! That was the solution, right!? No! It started again with the exactly same error (I’ll show the logs on the end of this post).

For the last tries, I installed Wordpress W3 Cache and some plugins to relieve MySQL calls. It wasn’t the solution neither.

Now I’m lost and have anything else to do. I won’t upgrade to a 2GB RAM because the traffic and the number os posts on my blog didn’t increase. What I did yesterday? I created a script in a crontab to restart MySQL and Apache each 15 minutes. I know it’s the ugliest thing to do, but I think this problem won’t solve.

Wanna know the good news? IT DIDN’T SOLVE ALSO! LOL! MySQL keeps crashing and doesn’t come back even with a script doing exactly the steps I do to restart MySQL and Apache2.

So, I believe you understand why I’m desperate, right? Sorry about this long testimonial, but now that I have nothing to work on my blog, since it doesn’t work, I have a lot of free time to write.

Here are the logs on Apache2 and MySQL:

Apache2

[Sun Aug 24 08:42:05 2014] [notice] child pid 17841 exit signal Segmentation fault (11)
[Sun Aug 24 08:42:11 2014] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Sun Aug 24 08:46:48 2014] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Sun Aug 24 08:46:59 2014] [notice] child pid 17900 exit signal Segmentation fault (11)
[Sun Aug 24 08:48:36 2014] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Sun Aug 24 08:48:57 2014] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Sun Aug 24 08:49:07 2014] [notice] child pid 17927 exit signal Segmentation fault (11)
[Sun Aug 24 08:49:07 2014] [notice] child pid 17928 exit signal Segmentation fault (11)
[Sun Aug 24 08:49:07 2014] [notice] child pid 17929 exit signal Segmentation fault (11)
[Sun Aug 24 08:49:07 2014] [notice] child pid 17930 exit signal Segmentation fault (11)
[Sun Aug 24 08:55:15 2014] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Sun Aug 24 08:55:42 2014] [notice] child pid 18022 exit signal Segmentation fault (11)
[Sun Aug 24 08:56:01 2014] [error] (12)Cannot allocate memory: fork: Unable to fork new process

MySQL

140824  9:06:10 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140824  9:06:10 [Note] Plugin 'FEDERATED' is disabled.
140824  9:06:10 InnoDB: The InnoDB memory heap is disabled
140824  9:06:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140824  9:06:10 InnoDB: Compressed tables use zlib 1.2.3.4
140824  9:06:10 InnoDB: Initializing buffer pool, size = 64.0M
InnoDB: mmap(68681728 bytes) failed; errno 12
140824  9:06:10 InnoDB: Completed initialization of buffer pool
140824  9:06:10 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140824  9:06:10 [ERROR] Plugin 'InnoDB' init function returned error.
140824  9:06:10 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140824  9:06:10 [ERROR] Unknown/unsupported storage engine: InnoDB
140824  9:06:10 [ERROR] Aborting

140824  9:06:10 [Note] /usr/sbin/mysqld: Shutdown complete

140824  9:06:10 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140824  9:06:10 [Note] Plugin 'FEDERATED' is disabled.
140824  9:06:10 InnoDB: The InnoDB memory heap is disabled
140824  9:06:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140824  9:06:10 InnoDB: Compressed tables use zlib 1.2.3.4
140824  9:06:10 InnoDB: Initializing buffer pool, size = 64.0M
InnoDB: mmap(68681728 bytes) failed; errno 12
140824  9:06:10 InnoDB: Completed initialization of buffer pool
140824  9:06:10 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140824  9:06:10 [ERROR] Plugin 'InnoDB' init function returned error.
140824  9:06:10 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140824  9:06:10 [ERROR] Unknown/unsupported storage engine: InnoDB
140824  9:06:10 [ERROR] Aborting

140824  9:06:10 [Note] /usr/sbin/mysqld: Shutdown complete

Thank you.

Show comments

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.

I had a similar problem and was able to finally fix it.

Background

I tried a bunch of tutorials that recommended tuning my MySQL installation and Apache configuration with no luck.

Identifying the Problem

It turns out that my site was being subjected to a brute force attack that targeted the WordPress xmlrpc.php file. H/T @sierracircle.

In order to check for this, you should view your apache2 access logs and look for suspicious POST requests. In my case, I found a huge number of POST requests to the /xmlrpc.php file coming from the same IP address (about 2 or 3 requests per second).

cd /var/log/apache2/
cat access.log

Solution

In order to fix this, I banned the offending IP address via Iptables:

Ban IP the malicious IP address (replace example with whatever IP address you want to ban) :

iptables -A INPUT -s 46.166.139.20 -j DROP

To view blocked IP address:

iptables -L INPUT -v -n

Reference: http://www.cyberciti.biz/faq/linux-howto-check-ip-blocked-against-iptables/

I had a similar problem from some time. I want to say that in my opinion 500MB is and should be enough memory for low traffic. In the past I had almost 200 users at the same time for at least 2 hours and nothing happened. And moreover there was almost nothing cashing and with original settings.

How this it looks like. IT starts with a huge number of processes from apache that consuming a lot of memory and because of that there is 100% CPU and 100% Hard drive usage I even can’t do anything in console like restart the system. Because of that mysql crashes and here you are error establishing database connection.

When it started ?, during a lot of months nothing happened so probably it was worpdress update OR wordpress update with some plugins update. I can’t tell you now if this is because of plugins but I found jetpack and sitemap creator that using cron. I assume that broken link checker, W3 cache and few other use cron either but currently under investigation. And THIS DATABASE ERROR happens periodically.

I haven’t noticed any brute attack on xmlrpc.php but could be something missed ?. Frankly nothing in apache logs.

IT could be that there is no straight answer for that and it is complex problem that include plugins and some kernel problems like cleaning cache from memory. I done some updates described here but currently not sure if that will help me and us ;(. I will write later.

I think there should some solution wrote by administrators, answer is simple it is a common problem and happens almost everyone. So few check points with solutions would be great.

I had this error and it plagued me for weeks. Sorry you got frustrated and switched to managed hosting. I’m posting this answer for others having the same problem.

For me it was a combination of heavy bot traffic and sub-optimal Apache configuration. What happens is amid the high bot traffic, Apache crashes, restarts automatically like it’s supposed to, but there isn’t enough memory to re-launch MySQL as well, hence the Database Connection Error.

Apache2Buddy is a quick diagnostic tool you can use to evaluate your current Apache settings given the amount of RAM you have and the size of your application and make recommendations:

$ curl -L http://apache2buddy.pl/ | perl

For me the MaxRequestWorkers (MaxClients for older Apache versions) was too high. Simply following the recommendations from Apache2Buddy and restarting Apache worked. (The file where the configuration lived was a little hard to find too. It’s at /etc/apache2/mods-available/mpm_prefork.conf).

At the beginning, I set up a stress test with JMeter, with which I was able to reproduce the error. It performed much better under the new Apache configs.

As for the bot traffic, there are a number of WordPress plugins for dealing with that. I like IP Geo Block.

I wrote an article on this as it was such a hassle. For more info on the stress test, plugins for dealing with bots, check it out here