Related

Floating IP and BW on HA droplets Question Question
How can I verify floating IP is on a different physical server for high availability? Question Question

Question

Cross Region High Availability

Posted April 26, 2020 955 views
High Availability

Hello

On 04/22/20 there was a multi region outage with Digital ocean where NYC,SF and LO all had outages.

Since then I’ve been looking into how to setup multi region high availability and am not sure it can be done currently with Digital Ocean.

It seems like Load Balancers are locked to droplets in a single data center and Managed Databases servers are also locked to the same data center, even the backup standby nodes.

So my question is how do you setup your infrastructure in DigitalOcean so that you have redundancy setup across multiple regions?

Thanks!

Add a comment
mindglitch
By:
mindglitch

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer
Zehubert April 26, 2020

Adding a reply to comment and follow this topic. I noticed that problem and from what I remember, pretty much all DO datacenters in the USA and some in Europe suffered from that outage at various degrees.

I took away from it that DO seems to be very reliant on a backbone bandwidth provider, and an outage at that provider took all these places down or at least created a whole lot of troubles – for what seemed to be a good part of the day, and became “all green” only the day after (that I saw).

@mindglitch , you might want to look at using DNS failover (or load-balancing). Many DNS providers (Route53, DNSME etc) do provide that functionality across regions or even providers.

Cloudflare or Fastly can do this at the HTTP-Proxy level, so you could add more intelligence than just a “healthcheck”.

As far as DO is concerned, they need to convince users that such an event “cannot” happen if you pick any two or even three DO datacenters. Unfortunately, that 4-22 event showed that it did happen.

As a customer, the only solution is to be ready to go with another provider. Myself, I use Linode as a no-DO failover (ongoing replication), but it would be more convenient to stay within DO, I agree.

Reply Report
  • mindglitch April 26, 2020

    Thanks for the comment!

    We was arriving at basically he same end point as well. Using DNS to do failover to either a separate provider or a duplicated infrastructure withing Digitalocean.

    So at the moment it would look something like this right?

    • Replicate your Droplets,Managed Databases and Spaces to a different datacenter (lets say Singapore)
    • Come up with a reliable method to keep the replication infrastructure synchronized
    • Setup a failover/loadbalancing solution on your DNS (in my case Cloudflare) to switch over to the replicated infrastructure in case of outages.

    How do you handle your ongoing replication between Linode and DO?

    Reply Report
    • Zehubert April 26, 2020

      Sounds like a plan! (CF fails-over at the proxy level, not the DNS level FYI)

      Typically, it can all be scripted as export –> import. It depends on how much data it is, and how much latency you can tolerate between the instances.

      Also, if you can make the sync real-time, you can actually serve clients with both regions at the same time, and get a little more out of the extra hardware.

      Note that DO’s databases ARE cross-region, so this might be a piece that provides you with real-time data replication across regions.

      Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help