Question
Pin to Kubernetes node that has block storage
- Posted November 9, 2019
- Block StorageKubernetesDockerDigitalOcean Managed Kubernetes
I’ve set up a Kubernetes cluster with 2 nodes in DigitalOcean. There’s a block storage volume that I use for persistent database storage, and as such I need my pods that rub database software to be able to access it.
At first I ran into the issue of, on deployment, the containers for the database going to a random node. If they went to the node that had the block storage, all was good. If they went to the other one, it’d go into a loop and be unable to start. It seemingly never tried to move it to the other node.
To get around this, I pinned the database to the node that has the block storage attached to it. I had to do this by hostname, as DigitalOcean does not appear to apply any labels or anything j can filter by to the node to which it attaches the block storage.
This morning I woke up to a dead service and a pile of outage emails. My node have apparently been replaced. This in itself isn’t a problem, but now of course the hostnames are different and as such Kubernetes can’t find the node it’s meant to spin up this database on. And even if it could there’s no guarantee that that node has the block storage.
I’m fairly new to Kubernetes so I might be missing something obvious here, but how are you meant to keep something running smoothly on this if you can’t guarantee that your database will be able to access the block storage it needs?
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
The way to solve this problem is to attach block storage not to the node, but to the persistent volume. There is an official guide how to do this: https://www.digitalocean.com/docs/kubernetes/how-to/add-volumes/
There is a problem though - the guide explains how to create brand new block storage, and says nothing about the situation described in your use case when we have an existing block storage. The guide for that case is here: https://github.com/digitalocean/csi-digitalocean/blob/master/examples/kubernetes/pod-single-existing-volume/README.md
Hey Rob, here’s a very good blog post on how you should set up your PostgreSQL DB with Kubernetes:
https://severalnines.com/database-blog/using-kubernetes-deploy-postgresql
You’ll definitely need to adjust the apiVersion within the YAML files. For example, if you’re using Kubernetes 1.16, then you’ll need to change the following for Deployment
change
apiVersion: extensions/v1beta1
to
apiVersion: apps/v1
Please do leverage the documentation which has good information as well as examples to get you going.
https://kubernetes.io/docs/home