Multi-Region High Availability

July 1, 2019 3.6k views
High Availability
vhariable
By:
vhariable

Right now I am setup in the NYC1 region. I am looking at how to mitigate some of the issues we have been experiencing in this region over the past month.

  1. What options are there to host a load balanced web server(s) in two regions (NYC and SFO for example)? Do we have to use an external LB for this?

  2. What option do we have to have a master-master type clustered database in two different regions?

What are others doing on DO to have high availability? I do not care as much about load balancing our services between two data centers, what we do want is ideally automatic failover from primary region to a backup/secondary region. So if we are experiencing issues with the NYC1 region we could manually or automatically failover to serving out of the SFO region.

1 comment
  • nusbaum October 8, 2019
    Report

    We have our solution set up in multiple regions for high availability reasons.

    1. The load balancing piece of this is the most unwieldily right now. What we are using is NGINX based caching reverse proxy servers in each data center. Our solution is multi-tenant and the proxies are both configured to route users for specific accounts to app servers in specific regions, based on the url structure. We publish IP addresses for both proxies, so a browser can hit either proxy and then have an affinity for their specific backend server. Most browsers seem to do ok at trying the other IP if one fails. If the preferred backend for a client account is down, the proxy will failover to the other data center. This isn’t as clean as we would like, but DO’s load balancer doesn’t work for us because it leaves us on one data center.
    2. We run MariaDB with a master-master setup. Since each account has an affinity to one region, and we have one MariaDB per region, records creates and updates for an account stay on one server or they all failover to the other server at the same time. Once server generates odd key and the other even, so they don’t overlap.

    I too am curious how other set up High Availability in DO? I see some big names in the list of DO customers and I cannot imagine they don’t have solid HA solutions.

    I might be old school, but I believe HA requires distribution across data centers to minimize risk of a location specific issue bringing down a solution, but maybe DO has solid isolation even within the same data center?

    For me, the biggest challenges are the lack of private networks across data centers and the lack of a load balancer that works across data centers.

Sign In to Comment
2 Answers
MattIPv4 MOD July 9, 2019

Hi there @vhariable,

Sorry for the delay in getting a response to you!
I have checked with our relevant teams about this to ensure the answer I give you is as correct as it can be.

Currently, both our load balancers and master databases are single-region. If you wanted to load balance across multiple Droplets in different regions you would, unfortunately, need to achieve this with your own LB.

With databases, we do support the ability to add read-only database nodes in different regions from the source database.

https://www.digitalocean.com/docs/databases/how-to/clusters/add-read-only-nodes/

Hope that helps!
- Matt.

Report
nusbaum December 2, 2019

I haven’t given up on DO for high availability across multiple sites, but it is definitely not as easy as we might hope. The lack of floating IPs across data centers and the fact that DO load balancers only work within a single data center are the big challenges that I have encountered. A standard IP takeover solution using something like keepalived isn’t going to work. What I am thinking about is an active-active set up with two published IP addresses that point to NGINX based load balancers in 2 different DO data centers. In most cases a browser will try to use the second IP in the first is not available. Since “most cases” isn’t really good enough, I was thinking about using something keepalived, or similar, to pull the failed IP address out of the DNS until the server is available again. This isn’t quick as even with a low TTL this could take minutes rather than seconds, but this removing the IP is a secondary measure to cover the situations where browsers don’t just roll to the second address on their own.

I realize this solution offer no balancing between load balancers and traffic will just randomly float between the two, but the goal is high availability across multiple data centers so I am no concerned if load is balanced between the balancers themselves.

I would be interested in any feedback on the viability of this approach.

If this seems feasible and people are interested, I’ll get it working and share any results with the DO community

Report
Have another answer? Share your knowledge.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Related