WordPress server that worked flawlessly for months now runs out of memory and crashes for no apparent reason. How to troubleshoot and fix?

January 5, 2016 13.2k views
Apache WordPress Ubuntu

In the last month my server just started blowing up randomly. I had a perfect installation that worked flawlessly for 9 months but in the last month the load and memory usage just randomly start spiraling out of control.

It appears that something external causes this to happen but I have no idea what it could be. Rebooting the server will make it run normally for anywhere from a few minutes to 18-24 hours but that's about the max.

The memory usage just keeps going up and up and up until the Apache process core dumps. The load spirals up to 20+.

[Tue Jan 05 11:31:22.629436 2016] [core:notice] [pid 1246] AH00052: child pid 8127 exit signal Segmentation fault (11)

For 9 months prior to this this server operated flawlessly with loads [in top] ranging from .01 - .20.

The server is running digital ocean's one click WordPress installation image, it has 1 GB of memory and 1 GB swap file.

My list of active plugins is as follows: Blubrry PowerPress, CloudFlare, Disqus Comment System, Jetpack, Login LockDown, Monarch Plugin (Share On Theme123.Net), Nofollow Links, TinyMCE Advanced, Yoast SEO

None of the plugins have been changed in many months.

My server is running only one WordPress installation and one site. WordPress and plugins are always updated to the latest version. There are no major modifications on the site.

I have had problems in the past 100% on every WordPress installation with the sites being crashed via brute force hacking attempts to /xmlrpc.php I have had to completely deny access to that even though it screws up jetpack because I have not been able to get Order Allow,Deny to work. It either causes 520's to all URLs across the whole server or it reports "order not allowed here" in the error log and it doesn't work. This is a separate issue but I would be very grateful if anyone can explain that one either. Past experience indicates leaving xmlrpc.php open to the public will result in crashed sites 100% of the time.

Can anyone help? I'm getting really desperate here this is destroying my site. Haven't been able to keep it online for more than 24 hours since early December. Nobody has any answers.

6 Answers

Hey guys/gals, I was having this issue and found out it was a brute force attack on my site via the XML-RPC feature in Wordpress. See if this fixes your problem: https://www.digitalocean.com/community/tutorials/how-to-protect-wordpress-from-xml-rpc-attacks-on-ubuntu-14-04

WordPress is a popular and powerful CMS (content management system) platform. Its popularity can bring unwanted attention in the form of malicious traffic specially targeted at a WordPress site. There are many instances where a server that has not been protected or optimized could experience issues or errors after receiving a small amount of malicious traffic. This guide will show you how to protect WordPress from XML-RPC attacks on an Ubuntu 14.04 system.
  • If your site was running normaly and suddenly it is having memory issues, this is worth checking out.

Have you solved the issue? I have multiple Wordpress installations, all of them are running on separate droplets. All of them have the same issue you provided above.

Please let me know if you have solved it.

I have exactly the same issue, added swap but still blowing itself up!
Cant see why? Its a new install and barely used.

  • Never solved it. Had to move to managed hosting, which isn't great either (it costs 3 times as much but it isn't faster although the server never crashes). Often it was attacks on xml.rpc but this last time around I could not definitively track it down to that.

    Clone your droplet to a fresh IP, put CloudFlare protection on it with high security settings and set a page rule for xml.rpc at security level "Help I'm Under Attack!" and see if that fixes it.

    Please report your results. I'd prefer to have a VPS instead of manged hosting due to cost and lack of flexibility but I couldn't keep the server up, it was awful. I've had others start melting down recently also but these seem to be strictly due to xml.rpc attacks.

    I can't understand why WordPress uses xml.rpc for plugins and requires it be open to the public when it's such an Achilles heel. It sucks!

Check your access log and see if there are request floods, I had a similar issue recently with php in general being flooded by POST requests coming from Aurastresser

  • What's Aurastresser?

  • Wow, I just noticed I'm getting killed on access requests. What's going on?

    root@ubuntu-512mb***:~# grep "19/May/2016" /var/log/apache2/access.log | wc -l
    162671

    This is for a server I would expect to be handling < 10 visits a day.

As I am sure you can agree, it is difficult to know the exact cause without more information. To help you help us help you, I got some areas for you to check. I cannot guarantee the areas will show the problem itself, but it is a good start.

We have been seeing what appears to be a few cases of xmlrpc attacks, so that could be the cause of the issues you are experiencing. I am seeing you are reporting out of memory (OOM) errors and processes dying off or needing to be killed, which certainly are symptoms of just such an attack.

Basically, this attack is being made possible because many calls in the WordPress XMLRPC implementation required a username and password. In these attacks, we are seeing wp.getUsersBlogs being used (and ocassionally wp.getComments), but it could be other calls as well. Sadly, it is a common issue with WordPress.

If you could provide some lines from your site's access logs, we'd be more than happy to take a look! With that information, we might be able to do some blocking or mitigating at the network level. The logs would normally be at:

Apache for Ubuntu/Debian
/var/log/apache2/access.log
Apache for CentOS
/etc/httpd/logs/access_log
Nginx for Ubuntu/Debian
/var/log/nginx/access.log

In other cases this is a problem with a bad plugin or theme, but I do have a good list of steps to go through to troubleshoot the issue at hand:

1) Renamed the .htaccess to .htaccess.bak
2) Made a new .htaccess containing only the default coding
3) Cleared any cache files
4) Turned Debug mode on
5) Cleared any cache files
6) Disabled all plugins
7) Cleared any cache files
8) Changed the theme
9) Cleared any cache files

You'll notice I say to clear cache files a lot, and that is because each page load is cached, so that white page would be cached. Try each step, clear the cache files, then refresh the page to see if it fixed it. Being your admin panel might not be working, you can use the following MySQL queries to do the following:

  • Disable all plugins

UPDATE wpoptions SET optionvalue='a:0:{}' WHERE optionname='activeplugins';

  • Change theme to twentyfifeteen

UPDATE wpoptions SET optionvalue='twentyfifeteen' WHERE optionname='TEMPLATE' OR optionname='stylesheet';

Before changing any value, I would recommend saving the old values so you can put those back if you determine them to not be the issue.

For more in-depth help on troubleshooting WordPress, I would recommend contacting their community/support teams, or referring to their article on troubleshooting WordPress.

It is also possible the system configuration is not tuned for your WordPress needs. Our one-click is tuned for the "general" WordPress, which is pretty much an out-of-the-box WordPress. Such a WordPress has very little going on, and tends to get very little traffic. As such, the setup is tuned for such a thing. So if the issue is found to be resources stemming from the PHP settings or Web Server settings, you might want to look into tuning it for the traffic level and resource needs you are using.

It could also be your WordPress's MySQL needs are greater than that of a 1 GB RAM droplet. WordPress is extremely database driven. Nearly any change you make in your wp-admin panel will cause a change in the database. Every comment, user-registration, and post edit inserts or edits the database. The error you are seeing basically could be WordPress's way of letting you know it is hitting a MySQL error, meaning the service crashed, or the database is unreadable. Now the most common reason why MySQL is not running is that it stopped or failed to start as a result of not enough memory. But it could very well be crashed tables, or the need to optimize said tables, or just a plugin/theme doing terrible SQL queries and dragging the site down. It is truly hard to say. Thus we say to check the MySQL logs for memory related errors (any errors really). The MySQL log file is located at /var/log/mysql/error.log normally, and is a good place to check for any kind of errors that should not be occurring.

Hope it helps!
Jason Colyer
DigitalOcean Platform Support Lead

Disable Jetpack. After upping the RAM on my instance, moving the DB to another server at great expense, turns out it was just Jetpack. As always with Wordpress, start by disabling the plug ins first.

Have another answer? Share your knowledge.