When you need to write and access persistent data in a Kubernetes cluster, you can create and access DigitalOcean Block Storage Volumes by creating a
PersistentVolumeClaim (PVC) as part of your deployment.
The claim can allow cluster workers to read and write database records, user-generated website content, log files, and other data that should persist after a process has completed.
When managing persistent volume claims, note that:
Deleting a deployment will not automatically delete any PVCs that have been created. You’ll have to remove those manually with
kubectl delete pvc.
If the DigitalOcean Block Storage Volume is deleted before the PVC API object is removed, it may be in an inconsistent state and attempts to remove the PVC will stall or fail. See the troubleshooting instructions for a fix to try in this case.
If a PVC by the same name already exists, you will get an error message similar to the following:
Error from server (AlreadyExists): error when creating "pvc.yml": persistentvolumeclaims "csi-pvc" already exists
Since the volume exists, it cannot be created. The existing volume will be mounted instead.
Block storage volumes created in the control panel or via the API cannot be used by your Kubernetes clusters. You must create volumes within Kubernetes in order for your PVCs to use them..
The configuration examples below define two types of objects:
csi-pvc, which is responsible for locating the block storage volume by name if it already exists and creating the volume if it does not.
my-csi-app, which will create containers, then add a mountpoint to the first object and mount the volume there.
The first resource definition to define the PVC can look like this:
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: csi-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: do-block-storage
This example creates a 5GB block storage volume that will be available to the cluster by the name
csi-pvc. If a volume by that name does not exist, one will be created. If one already exists, then the existing volume will be mounted on the first object.
The three highlighted values,
storage can be customized as follows:
name must be lowercase alphanumeric values and dashes only and unique within the cluster. Within these constraints, you can name it whatever you want.
accessModes must be set to
ReadWriteOnce. The other parameters,
ReadWriteMany, are not supported by DigitalOcean volumes. See the Kubernetes documentation for more about accessModes.
storage value specifies the size of the volume and can be customized to meet your needs. DigitalOcean storage values can range from 1GB to 10,000GB.
Resizing block storage volumes has not yet been implemented, and changing the
storage value in the resource definition after the volume has been created will have no effect.
Billing for the block storage volume begins when the object is successfully created. To end billing, you must explicitly delete the volume. Make sure to remove the PVC from your cluster before deleting the block storage volume.
The second resource definition, which defines the pod, might look like:
--- kind: Pod apiVersion: v1 metadata: name: my-csi-app spec: containers: - name: my-frontend image: busybox volumeMounts: - mountPath: "/data" name: my-do-volume volumes: - name: my-do-volume persistentVolumeClaim: claimName: csi-pvc
This adds a pod called
my-csi-app based on the Linux BusyBox image that names the
my-do-volume and mounts it within the container at
/data on the filesystem.
The pod created in this example will immediately exit, so the API will continue to recreate the pod until you explicitly delete it.
Within the cluster, volumes will be identified by their names as defined in the
claimName parameter. In the example above, the name is
Regardless of what you set this name to be, the name of the volume on DigitalOcean will begin with
pvc- and end with a unique identifying number, something like
pvc-0213ed0abexample. You can list the storage volumes associated with a cluster with the
get pv command:
kubectl -kubeconfig=[full path to cluster config file] get pv
The output looks something like:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-0213ed0abexample 5Gi RWO Delete Bound default/csi-pvc do-block-storage 11s
By default, the filesystem owner of a DigitalOcean Block Storage Volume is
root:root. If a Pod is running as a non-root user and needs to create files or directories on the volume, this will fail due to insufficient or incorrect permissions. However, the following
mountOptions settings are not supported by DigitalOcean Kubernetes:
mountOptions: - dir_mode=0777 - file_mode=0777
The solution is to create a temporary container to change the permissions/ownership of the volume’s filesystem using
We’re going to reuse the same PVC config as described in the example above and change the pod definition.
In the example below, we’ll be using the block storage volume we just created to persist data written to a Postgres database. The pod’s resource definition might look like:
--- apiVersion: v1 kind: Pod metadata: name: my-csi-app spec: containers: - name: my-db image: postgres:latest volumeMounts: - mountPath: "/var/lib/postgresql" name: my-do-volume initContainers: - name: pgsql-data-permission-fix image: busybox command: ["/bin/chmod","-R","777", "/data"] volumeMounts: - name: my-do-volume mountPath: /data volumes: - name: my-do-volume persistentVolumeClaim: claimName: csi-pvc
This adds a pod called
my-csi-app based on the latest
postgres image that names the
my-do-volume and mounts it within the container at
/data on the filesystem. This also creates an
initContainer that temporarily mounts the volume and changes the file permissions for the specified path to 777. The initContainer then deletes itself. This all happens before the volume is mounted to the container. As a note, if you are using
securityContext in the yaml file for your Pod, you can use
chown $userid instead of
securityContext: runAsUser: 1000 fsGroup: 2000
Once the cluster has been created, you can confirm the permissions were correct by checking the log with
$ kubectl logs my-csi-app
The output should look like the following:
The files belonging to this database system will be owned by user "postgres". This user must also own the server process. The database cluster will be initialized with locale "en_US.utf8". The default database encoding has accordingly been set to "UTF8". The default text search configuration will be set to "english". Data page checksums are disabled. fixing permissions on existing directory /var/lib/postgresql/data ... ok creating subdirectories ... ok selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting dynamic shared memory implementation ... posix creating configuration files ... ok running bootstrap script ... ok performing post-bootstrap initialization ... ok syncing data to disk ... ok
As mentioned above, if the DigitalOcean Block Storage Volume is removed manually before the PVC API object is removed with
kubectl, this can cause issues. For instance, it can cause the PVC deletion to hang and never complete. If this happens, you can try the following:
The output will look something like this:
kubectl get volumeattachments
NAME CREATED AT $VOLUME_NAME 2019-03-08T21:58:24Z
Use your volume’s name, displayed by the previous command, in the commands below to gather information you’ll need to try to fix the issue.
kubectl describe volumeattachments $VOLUME_NAME kubectl edit volumeattachment $VOLUME_NAME
editcommand above will allow us to edit the PVC using a text editor. Remove the following from the volume attachment
metadatasection, and save your changes:
Now, you can try removing the PVC:
kubectl delete pvc csi-pvc
If those steps don’t work, you can open a ticket with Support.
For more about managing persistent volumes see: