@vitamincee
If everything appears to be normal, there’s a few ways we can setup a cluster that would provide you with better reliability. Both are a little complex, and will take a little time to setup properly, and ideally, once setup, you need to work on optimizing NGINX, PHP-FPM, and MariaDB, otherwise moving to a setup like this isn’t going to provide much benefit at all.
NOTE: Since I don’t know your domain, I’ll be using domain.com
in all examples.
…
Recommendation #01
My first recommendation is the more simple way of setting up a cluster that’ll allow you as the need arises, whether it’s vertically or horizontally.
What I would do is separate each concern. You’ve already got a jumpstart on this by offloading your database to its own server, so the concept is very similar.
Step 01:
Setup a load balancer. You can use DigitalOceans’ LB service, or setup NGINX as an LB that you have full control over. You’d point your DNS at this LB and it would handle traffic routing for you.
Step 02:
Setup NGINX + PHP-FPM (using PHP 7.1.x) on it’s own server, much like what you have now. We’d be moving away from HHVM to PHP 7.1.x in this case. HHVM is marginally slower in some areas and marginally faster in others, though ultimately, we’re looking for simplicity and consistency.
Step 03:
Setup Caching Servers, either Memcached or Redis. I’d pick which you’d like to use and setup 2x of them to start. The first one would be used to offload PHP Sessions so that we’re not storing them to files (which is the default) and the second would be for caching that’s handled by WordPress. When you start clustering, this is important as if you ever move to a multi-server setup to distribute content, you need session persistence and that’s not possible using file-based sessions.
Step 04:
Setup a MariaDB Server, much like what you’ve done already. I’d aim for the 10.x release. Right now, we’d setup a single database server. Since you’re not seeing extremely high loads right now, setting up a cluster or master-slave might be overkill.
Bring it Together
How this would work is something like this:
LB =>
=> NGINX/PHP-FPM => Caching Servers
=> MariaDB Server
The LB handles directing traffic to your NGINX/PHP-FPM instance. Once a visitor arrives, the caching servers come in to play and allow sessions to begin and cached data to resolve. Anything that isn’t in the cache, or can’t be pulled from cache would then hit the database server. On the database server, we’d need to implement some sort of caching as well (i.e. Query Caching) to speed things up there as well.
What this does is simply expand on what you already have and will allow you to increase complexity as needed.
Ideally, for the above setup, I’d recommend the following sizes (these being based off the resource information you’ve provided):
LB => 2GB Droplet
NGINX/PHP-FPM => 8GB Droplet
Caching Servers => 2GB Droplet x2
MariaDB Server => 16GB Droplet
Now, the issue here is that 8GB Droplet for NGINX/PHP-FPM. You’ve stated that you’re using 50% of your current 480GB SDD HD, so the resolution to his would be to use DigitalOcean’s Block Storage option to attach a block device with XXX GB of additional storage. You would then configure WP to use that device, once mounted, for your storage.
WordPress does allow you to change the ./wp-content
location/URL, and that’d be set in the ./wp-config.php
file, so that’s how we’d manage changing the content locations.
Since you’re only using ~5% of the Web Servers RAM (32GB), which is about 1.6GB, an 8GB Droplet should be fine for it. Since you’re using ~25% of the Database Servers RAM (24GB), which is about 6GB, I’d definitely shoot for the 16GB Droplet as that gives you over double usage and allows room for increased RAM usage from optimization and properly using RAM.
That brings the total monthly cost for the above to around $280 per month. Of course, this is a very basic setup, but we can scale it from there as needed should you find that it’s not coping.
Other Notes
The above setup should provide a better working environment, but optimization is going to be key here, just as much as proper setup is (i.e. firewall, security, etc). This is what I would classify as a starter setup that gets you working more with cluster setups.
You need to make sure NGINX, PHP-FPM, MariaDB and your caching servers are optimized for the work load, otherwise you’ll end up in the same situation.
…
I’ve setup clusters like these for multiple private clients over the years and they work well, while providing room to grow. They can be a bit complex at first glance, but they do provide better flow and handle traffic far better than a simple two-server setup. Yes, it’s more to manage, though when it comes to scale, these options make such far easier and much less of a headache.
Okay to answer your questions:
I think most users are anon reading articles - we’re mostly a news org. We do have several thousand users who log in to comment. We also have BuddyPress and MemberPress, so some folks also message each other, or read some gated content, although that is way less than the regular news/content use of the site.
I meant unique visitors. Last month we had about 4M page views. That’s higher than average though which is maybe more like 3M page views.
What is failing is HHVM. It has begun crashing often in the last few weeks or so. I’m also generally sick of maintaining hardware and not having much redundancy.
1). Have you tweaked/tuned your current NGINX setup at all? If so, how?
Not really - it’s pretty standard
2). Have you tweaked/tuned your current database server at all? If so, how?
No, I run Maria DB but I also think that’s fairly standard. The DB server stands alone.
3). Are you currently using any sort of caching – object, Memcached, Redis, etc?
I do not think so.
4). What are your current server stats – CPU(s), RAM, etc?
web server:
2x Intel Xeon E5-2630 2.3 GHz Six -Core 12 Threads (w H/T)
32 GB (X9DR3) ram
480 GB SSD
database server:
2x Intel Xeon E5620 2.40 GHz Quad-Core 8 Threads (w H/T)
24 GB (X8DTi) ram
250 GB 7200 SATA
Both servers are greatly under-utilized.
Currently the web server’s disk is half full while the db is only at 13%, both servers use less than 5% of the CPU power, the webserver is under 5% mem, db server is about 25% mem. They can handle a lot, but HHVM or PHP-FPM tend to both fail often somehow. It’s probably my fault tbh, as I am mostly a web developer!
5). Are you using HHVM specifically because of what it offers, or would you be open to PHP 7?
Not opposed to PHP, I used to run PHP5-FPM but found HHVM much much faster. I haven’t checked v 7.
6). How many plugins do you have active? Any resource-intensive ones?
Tons. And yes, BuddyPress and MemberPress are both very resource intensive I believe.
Thanks!
@jtittle (just in case you missed the reply)
@vitamincee Great details!
Even without Varnish or another high performance cache engine, you could probably come a long way simply by tuning Nginx, MariaDB and PHP-FPM7.1, and adding Redis/memcached.
How many page view do you have at peak times? Because that would be our measurement for the initial setup.
Where’s your primary users visiting from? Country or region. The reason for asking is because you might want to use Block Storage instead of having to spin up a $320 droplet just to get 320GB.
I completely agree with you. Maintaining hardware for something that could be hosted elsewhere can be a drag. And since scaling is done within minutes on DigitalOcean, then it’s pretty straight forward.
If you have any other info that you think is relevant, then please just add it - the more, the better.
Will give a recommendation when you answer the two questions above. Pretty sure jtittle will voice his recommendations too.