@nusbaum
Interesting subject Dave. I would say, for true fail-over you should always expect the unexpected. Meaning, you shouldn't even trust a single location or a single provider.
Have servers with different providers in different locations. Have multiple DNS providers, so you don't end up in a Dyn-situation. And even run different operating systems and different service versions, if possible.

So I run a real-time system, which is hosted in Frankfurt, but the database is synchronized to London and New York. All 3 servers keep in contact, but in case of Frankfurt going offline, then London is promoted, then New York.
Each server can change the DNS, which has a TTL of 120 seconds, so there is a maximum of 150 seconds outage.
This was a setup I made several years ago. I would probably do it differently now, if I needed to remake it.

Can you describe a little more your application. I mean, I guess database sync would be important, but is file sync also part of it? And besides Nginx as proxy, what other systems are you currently using and how, and have you looked at anything else?

Thanks for your input here! I'll try to explain the stack as concisely as possible:

• We have our application deployed across two DO locations. We have NGINX as a caching reverse proxy, Apache web server and MariaDB database server installed in each location.
• We have IP addresses for both proxies advertised in our DNS.
• The application is multi-tenant and we have an account id as part of the URL. We use a hash of the account id to give all requests for that account an affinity for the same apache server. This way account users end up on the same backend web server regardless of which proxy they hit. If the web server isn't responding then NGINX will direct requests to the other web server.
• Web server files don't change often, but changes are replicated with lsyncd.
• PHP sessions and object caching is in memcached and redundant across both servers.
• MariaDB is set up with master-master replication. Since an account as an affinity for one server over another, we should not have replication conflicts other than the possibility during a failover scenario. If MariaDB isn't responding, we basically fail over the entire apache-mariadb pair rather just the database server.

I have a diagram but cannot see how to post it here.

This has been working, but with no single point of failure there is also no single point of control and flaking things could happen.

• Only one of the two NGINX servers could think a node has failed and traffic for an account could be split between two servers.
• Browsers usually handle a proxy failure (in test scenarios) ok, but not always. Sometimes they will keep trying to hit the failed server.
• Network traffic is not isolated to a private network. Everything is encrypted, but there is still more latency than if everything was in the same data center.
• Debugging production issues an be tricky at times.

This all gets back to the balance that @moisey mentioned. If DO has had one 10 minute outage in three years, this is a lot of additional complexity that may not be necessary. All the talk of floating IPs and HA got me thinking that maybe I was going too far with my multi-data center approach.

Just wanted to clarify that the only datacenter level incident that we had in 5 years was a 10 minute outage. But remember that datacenter level issues are very rare in general. We also go a long way to study the geography of each datacenter to plan around potential natural disasters, flood zones, and so forth.

We also provide the same level of consideration to other physical elements like cross connects entering a building.

However, remember that there will be network congestion, there can be a large DDoS attack is aimed at services or at the network in general that can have an impact. There are maintenances that we have to carry out and while the majority of them go smoothly there can be an issue.

By straddling two datacenters you are minimizing the potential impact as both would need be affected assuming your HA setup fails over smoothly. Otherwise it would have to be a service that is global that is impacted like DNS which can be affected and cause an issue and not route requests to your HA setup.

The reality is there is no service that has 100% uptime. And as you add more complexity as you mentioned debugging also becomes more difficult. So while you may have less service impacting issues you are more likely to cause them yourself due to configuration management or other complexities of running an HA setup.

So you really have to do a cost benefit analysis to understand how much is your cost and complexity going up, when are the times that your service must be available, and what is the impact of unavailability as well as the likelihood that it can happen.

There are certain services, like banking which can't have any downtime as it can be very damaging to their reputation, and even so online banking services like Simple had outages and issues as they were building their company.

Other services can suffer a 2 minute outage and recover with minimal impact to their business. If you look at a platform like twitter which was plagued with outages that's an extreme case of performance issues but even so they were able to pull out of it, with some impact to user growth, but not anything materially substantial.

What that ultimately means is that the question of HA is more about context around the underlying business and the tolerances it has.

@nusbaum It seems like you have a very good protection. Adding full availability, as you say, might make everything even more complicated.
All in all, it's difficult to do anything 100% perfect and adding more fail-over just makes your setup more complex.

I would say you're in pretty good standing. I would recommend that you do backups to a completely different location, and write a plan on how to get up and running with the latest backup.

About your two first issues; without looking at the configuration and exactly knowing how you application works, it is a little difficult to help with alternative ideas.
Third; True, going over the internet does add extra latency, but you're more protected of any issues in a specific data center. So it's a win-lose.
Fourth; Yes, the more complexity you add, the more difficult it'll be to do diagnostic. But this is a win-lose situation again. You have fail-over, but everything is more difficult to figure out.

Thanks for the comments @hansen. The solution is working, but I'm always looking to simplify where possible. Less complexity means less opportunity for me to make a silly mistake and mess things up.