Are multiple data centers required for high availability with Digital Ocean

First off, we provide a service to non-profits that is used to manage events, including check-in services. It’s not the end of the world if the app is down, but downtime really is a problem for our users and can cause the some temporary hardships.

We came digital ocean from another hosting service and in the course of a year we actually had three data center outages that left our service inaccessible. When we got to digital ocean we built a solution that spans two data centers NYC2 and NYC3. While I’m rather proud of this solution and it’s working well, there are additional challenges and a level of unpredictability from having the solution it two independent sites. For example, we have two nginx reverse proxies and browsers could hit either. We use the hash function in nginx to try to always connect a client to the same backend (account id in URL), so regardless of which proxy the browser hits, the work should end in the same place unless there is need for a failover. A floating IP in one data center would much easier to manage.

So my question is, how is your availability within a datacenter. If I build a HA configuration and make sure my redundant servers are not on the same physical hardware can I pretty much count on the entire center not going down. The things that got us in the past…

1. failover to generator backup was never tested and failed when needed.
2. multiple network carriers on the same fiber, which ended up being cut.
3. a DNS disaster I don’t even want to get into.

Have you ever had a situation take one of the NYC data centers completely offline?

I need to start scaling our solution up and bit and need to decide how we proceed.

Thanks, Dave