What type of configuration should I set up for very high traffic wordpress site

February 26, 2017 1.6k views
WordPress

Hello!
I run a WordPress website with about 2-3M uniques/month on some private servers that I maintain, and I'm looking to move over to Digital Ocean. Currently I run a web server (NGINX/HHVM) and a database server, and I'd like some redundancy if the web server fails (which it's been doing recently).

I'm wondering if I should just set up a website - wordpress instance or would that put me in the same situation I'm in now?

Or should I set up something more complicated - and if so, what would you recommend?

Thanks for your time.

2 comments
  • Okay to answer your questions:

    I think most users are anon reading articles - we're mostly a news org. We do have several thousand users who log in to comment. We also have BuddyPress and MemberPress, so some folks also message each other, or read some gated content, although that is way less than the regular news/content use of the site.

    I meant unique visitors. Last month we had about 4M page views. That's higher than average though which is maybe more like 3M page views.

    What is failing is HHVM. It has begun crashing often in the last few weeks or so. I'm also generally sick of maintaining hardware and not having much redundancy.


    1). Have you tweaked/tuned your current NGINX setup at all? If so, how?
    Not really - it's pretty standard

    2). Have you tweaked/tuned your current database server at all? If so, how?
    No, I run Maria DB but I also think that's fairly standard. The DB server stands alone.

    3). Are you currently using any sort of caching -- object, Memcached, Redis, etc?
    I do not think so.

    4). What are your current server stats -- CPU(s), RAM, etc?
    web server:
    2x Intel Xeon E5-2630 2.3 GHz Six -Core 12 Threads (w H/T)
    32 GB (X9DR3) ram
    480 GB SSD

    database server:
    2x Intel Xeon E5620 2.40 GHz Quad-Core 8 Threads (w H/T)
    24 GB (X8DTi) ram
    250 GB 7200 SATA

    Both servers are greatly under-utilized.
    Currently the web server's disk is half full while the db is only at 13%, both servers use less than 5% of the CPU power, the webserver is under 5% mem, db server is about 25% mem. They can handle a lot, but HHVM or PHP-FPM tend to both fail often somehow. It's probably my fault tbh, as I am mostly a web developer!

    5). Are you using HHVM specifically because of what it offers, or would you be open to PHP 7?
    Not opposed to PHP, I used to run PHP5-FPM but found HHVM much much faster. I haven't checked v 7.

    6). How many plugins do you have active? Any resource-intensive ones?
    Tons. And yes, BuddyPress and MemberPress are both very resource intensive I believe.

    Thanks!

  • @jtittle (just in case you missed the reply)

    @vitamincee Great details!
    Even without Varnish or another high performance cache engine, you could probably come a long way simply by tuning Nginx, MariaDB and PHP-FPM7.1, and adding Redis/memcached.

    How many page view do you have at peak times? Because that would be our measurement for the initial setup.

    Where's your primary users visiting from? Country or region. The reason for asking is because you might want to use Block Storage instead of having to spin up a $320 droplet just to get 320GB.

    I completely agree with you. Maintaining hardware for something that could be hosted elsewhere can be a drag. And since scaling is done within minutes on DigitalOcean, then it's pretty straight forward.

    If you have any other info that you think is relevant, then please just add it - the more, the better.
    Will give a recommendation when you answer the two questions above. Pretty sure jtittle will voice his recommendations too.

4 Answers

Are the users mainly anonymous or do they login and have specific content shown to them?
The difference comes from being able to cache content and serve that, if the content is mainly the same for all visitors. If it's custom views for each visitor, then it's a different answer than below.

When you say 2-3M uniques, do you mean unique visitors or page views. When talking about server capacity, you should only count page views, since that would give you the real numbers of load on the server.
If you meant page views, then it's about 1 page view per second.

I would probably use Varnish (advanced cache), which connects to Nginx, then PHP-FPM 7.1 (which is more or less same speed as HHVM, but more actively developed), and database in MariaDB 10.1 (has some speed and feature advantages over MySQL and is not controlled by Oracle).
So it'll look like this:

Visitor <-> Varnish <-> Nginx+PHP (WordPress) <-> MariaDB

Where Varnish, Nginx+PHP and MariaDB would be on their own separate server.

If you want extra fail-over options, you could take advantage of DigitalOcean's new load balancer feature. It could be placed in front of any server, but I would probably just place it in front of multiple Varnish cache servers.

Visitor <-> Load Balancer <-> Varnish[2+] <-> Nginx+PHP (WordPress) <-> MariaDB

You need to measure the load of each of the services to figure out the exact capacity needed for each server. And if you need a load balancer in front of each server.

You're saying, the web server is currently sometimes failing. What is failing? The entire server, Nginx, HHVM, MySQL or ?

  • To answer your questions from above:

    We get about 5,000 page views an hour during peak times, unless a post goes viral, when it can go as high as 40,000 page views an hour (but that's really not common).

    Most of our users are from the US and Europe, with N America being maybe 75%.

    Thanks!

@vitamincee

Setup really depends on what you're currently running and how you've configured it to work. Before I could really recommend what I would transition in to, can you provide a little more information about your current setup?

1). Have you tweaked/tuned your current NGINX setup at all? If so, how?

2). Have you tweaked/tuned your current database server at all? If so, how?

3). Are you currently using any sort of caching -- object, Memcached, Redis, etc?

4). What are your current server stats -- CPU(s), RAM, etc?

5). Are you using HHVM specifically because of what it offers, or would you be open to PHP 7?

6). How many plugins do you have active? Any resource-intensive ones?

...

Examples of resource intensive plugins would be image optimizer plugins, such as Smusher and any similar plugins.

...

Beyond that, if there's anything unique about your setup, please let me know.

@vitamincee

Thank you for the answers and thank you, @hansen, for the tag!

Before we can get to the part where we suggest an ideal setup for you to transition over to, the first thing I would recommend doing is finding out why PHP is crashing/failing. If you're running a server that large and resources aren't an issue, then sockets and TCP connections most likely are.

Stock OS deployments aren't tuned for high-traffic scenarios. PHP-FPM defaults to socket mode on initial installation, though there's an option to swap over to TCP connections. In either case, you may be approaching the limits of what a stock OS is capable of if connections begin to drop.

To identify any potential areas that may speak to why such failures occur, we need to check the log files and see what's showing up.

The command we'd used to scan the logs and get the last x or xx lines would be tail followed by a -N where N is a number.

I'd recommend checking the syslog as well as the logs for both PHP and NGINX just to make sure we aren't missing anything.

tail -50 /var/log/syslog
tail -50 /var/log/nginx/error.log

The PHP error log differs, so you'd need to see where it's set to log and run the same on it.

...

I recommend checking your logs first so we can identify the why as moving to DigitalOcean won't magically fix an issue where you're hitting the walls of a stock OS that isn't tuned for high-traffic.

If it does turn out to be that you're hitting the wall, that can be tweaked accordingly, so that's not an issue, but it's a good idea to find that out now than it would be to move everything over and continue to run in to the same issues.

  • Hello!
    Thanks for your reply. I am seeing a ton of UFW BLOCK errors in the syslog, and nothing odd in the nginx log. I keep seeing SlowTimer/slow query errors in the hhvm error log.

@vitamincee

If everything appears to be normal, there's a few ways we can setup a cluster that would provide you with better reliability. Both are a little complex, and will take a little time to setup properly, and ideally, once setup, you need to work on optimizing NGINX, PHP-FPM, and MariaDB, otherwise moving to a setup like this isn't going to provide much benefit at all.

NOTE: Since I don't know your domain, I'll be using domain.com in all examples.

...

Recommendation #01

My first recommendation is the more simple way of setting up a cluster that'll allow you as the need arises, whether it's vertically or horizontally.

What I would do is separate each concern. You've already got a jumpstart on this by offloading your database to its own server, so the concept is very similar.

Step 01:

Setup a load balancer. You can use DigitalOceans' LB service, or setup NGINX as an LB that you have full control over. You'd point your DNS at this LB and it would handle traffic routing for you.

Step 02:

Setup NGINX + PHP-FPM (using PHP 7.1.x) on it's own server, much like what you have now. We'd be moving away from HHVM to PHP 7.1.x in this case. HHVM is marginally slower in some areas and marginally faster in others, though ultimately, we're looking for simplicity and consistency.

Step 03:

Setup Caching Servers, either Memcached or Redis. I'd pick which you'd like to use and setup 2x of them to start. The first one would be used to offload PHP Sessions so that we're not storing them to files (which is the default) and the second would be for caching that's handled by WordPress. When you start clustering, this is important as if you ever move to a multi-server setup to distribute content, you need session persistence and that's not possible using file-based sessions.

Step 04:

Setup a MariaDB Server, much like what you've done already. I'd aim for the 10.x release. Right now, we'd setup a single database server. Since you're not seeing extremely high loads right now, setting up a cluster or master-slave might be overkill.

Bring it Together

How this would work is something like this:

LB =>
   => NGINX/PHP-FPM => Caching Servers
   => MariaDB Server

The LB handles directing traffic to your NGINX/PHP-FPM instance. Once a visitor arrives, the caching servers come in to play and allow sessions to begin and cached data to resolve. Anything that isn't in the cache, or can't be pulled from cache would then hit the database server. On the database server, we'd need to implement some sort of caching as well (i.e. Query Caching) to speed things up there as well.

What this does is simply expand on what you already have and will allow you to increase complexity as needed.

Ideally, for the above setup, I'd recommend the following sizes (these being based off the resource information you've provided):

LB              => 2GB Droplet
NGINX/PHP-FPM   => 8GB Droplet
Caching Servers => 2GB Droplet x2
MariaDB Server  => 16GB Droplet

Now, the issue here is that 8GB Droplet for NGINX/PHP-FPM. You've stated that you're using 50% of your current 480GB SDD HD, so the resolution to his would be to use DigitalOcean's Block Storage option to attach a block device with XXX GB of additional storage. You would then configure WP to use that device, once mounted, for your storage.

WordPress does allow you to change the ./wp-content location/URL, and that'd be set in the ./wp-config.php file, so that's how we'd manage changing the content locations.

Since you're only using ~5% of the Web Servers RAM (32GB), which is about 1.6GB, an 8GB Droplet should be fine for it. Since you're using ~25% of the Database Servers RAM (24GB), which is about 6GB, I'd definitely shoot for the 16GB Droplet as that gives you over double usage and allows room for increased RAM usage from optimization and properly using RAM.

That brings the total monthly cost for the above to around $280 per month. Of course, this is a very basic setup, but we can scale it from there as needed should you find that it's not coping.

Other Notes

The above setup should provide a better working environment, but optimization is going to be key here, just as much as proper setup is (i.e. firewall, security, etc). This is what I would classify as a starter setup that gets you working more with cluster setups.

You need to make sure NGINX, PHP-FPM, MariaDB and your caching servers are optimized for the work load, otherwise you'll end up in the same situation.

...

I've setup clusters like these for multiple private clients over the years and they work well, while providing room to grow. They can be a bit complex at first glance, but they do provide better flow and handle traffic far better than a simple two-server setup. Yes, it's more to manage, though when it comes to scale, these options make such far easier and much less of a headache.

  • @vitamincee
    This is an awesome break-by-break setup by @jtittle

    It's more or less what I would have done. Though I would recommend using the DigitalOcean load balancer feature, since that would require less maintenance from you and "just works".

    @jtittle wouldn't you setup two web servers instead of two object caches? Otherwise, how would you benefit from the load balancer?

    • @hansen

      This is meant to be a starter recommendation to get him off the current setup and on to one that mimics the same setup he currently has, plus caching and easier session management.

      In my example, the two caching servers are not meant to be networked together. One will be used as a way of managing sessions, the other would be for caching. So I suppose we could rename one of the Caching Servers to Session Server :-).

      As far as the LB, it's meant to get the process started. The more complex setup would be to setup two web servers, though you then need to manage replication to ensure a persistent state exists between both servers, otherwise when the LB sends traffic to one, it may return data that doesn't exist since it was originally uploaded to the other server (i.e. images, plugins, themes, etc).

      I didn't want to start with that as I'm not sure how complex a setup he wants to go with right now and the above is already a starting point that will allow him to go in either direction.

      DigitalOcean allows you to do some really complex setups with the addition of block storage, so we could really break things down even further, though it all depends on what his budget is and how complex a setup he wants to go with.

Have another answer? Share your knowledge.