How to setup auto-healing droplets?

April 5, 2018 1.4k views
API Configuration Management Deployment High Availability Load Balancing Scaling System Tools Ubuntu 16.04

We are in the process of moving from AWS where we have a highly available system setup using EC2's auto scaling feature. However, we aren't using this to change the size of the pool based on resource usage, we are strictly using it to spin up new instances when one of them fails or becomes unresponsive.

Since DO doesn't have an auto-scale type feature, what are some options to achieve this with DigitalOcean? My first thought was to create a droplet that monitors the others, but then that droplet becomes a single point of failure. Are there any services out there for this that aren't overkill for what seems like a fairly simple task or established patterns that we could script with the API that don't create a single point of failure?

3 Answers
goldfire May 14, 2018
Accepted Answer

We ended up writing our own solution to somewhat mimic the behavior in EC2. We called it healthcare.js and open-sourced it at https://github.com/goldfire/healthcare.js. Essentially, it uses the DigitalOcean API and tags for discovery, and then uses democracy.js (https://github.com/goldfire/democracy.js) to monitor which servers are running. This allows for a fully distributed self-healing system that will kill/re-build servers based on the passed server configs.

Your idea of using one node as a monitor / controller for the cluster could work, but agreed it would be a single point of failure. You could elaborate on this and use two or more droplets for this task to create a failover.

However it sounds like you would like setup an immutable infrastructure, perhaps something like a Kubernetes cluster?

-- https://go.digitalocean.com/containers-and-microservices-webinars-series

Have another answer? Share your knowledge.