How to Add Block Storage Volumes to Kubernetes Clusters

DigitalOcean Kubernetes (DOKS) is a managed Kubernetes service that lets you deploy Kubernetes clusters without the complexities of handling the control plane and containerized infrastructure. Clusters are compatible with standard Kubernetes toolchains and integrate natively with DigitalOcean Load Balancers and block storage volumes.

When you need to write and access persistent data in a Kubernetes cluster, you can create and access DigitalOcean Block Storage Volumes by creating a PersistentVolumeClaim (PVC) as part of your deployment.

The claim can allow cluster workers to read and write database records, user-generated website content, log files, and other data that should persist after a process has completed.

When managing persistent volume claims, note that:

  • Deleting a deployment will not automatically delete any PVCs that have been created. You'll have to remove those manually with kubectl delete pvc.

  • If the DigitalOcean Block Storage Volume is deleted before the PVC API object is removed, it may be in an inconsistent state and attempts to remove the PVC will stall or fail. See the troubleshooting instructions for a fix to try in this case.

  • If a PVC by the same name already exists, you will get an error message similar to the following:

    Error from server (AlreadyExists): error when creating "pvc.yml":
    persistentvolumeclaims "csi-pvc" already exists

    Since the volume exists, it cannot be created. The existing volume will be mounted instead.

  • Block storage volumes created in the control panel or via the API cannot be used by your Kubernetes clusters. You must create volumes within Kubernetes in order for your PVCs to use them.

Create a Configuration File

The configuration examples below define two types of objects:

  1. The PersistentVolumeClaim called csi-pvc, which is responsible for locating the block storage volume by name if it already exists and creating the volume if it does not.

  2. The Pod named my-csi-app, which will create containers, then add a mountpoint to the first object and mount the volume there.

Define the Persistent Volume Claim

The first resource definition to define the PVC can look like this:

apiVersion: v1
kind: PersistentVolumeClaim
  name: csi-pvc
  - ReadWriteOnce
      storage: 5Gi
  storageClassName: do-block-storage


This example creates a 5 GB block storage volume that will be available to the cluster by the name csi-pvc. If a volume by that name does not exist, one will be created. If one already exists, then the existing volume will be mounted on the first object.

The three highlighted values, name, accessModes, and storage can be customized as follows:

  1. The name must be lowercase alphanumeric values and dashes only and unique within the cluster. Within these constraints, you can name it whatever you want.

  2. accessModes must be set to ReadWriteOnce. The other parameters, ReadOnlyMany and ReadWriteMany, are not supported by DigitalOcean volumes. See the Kubernetes documentation for more about accessModes.

  3. The storage value specifies the size of the volume and can be customized to meet your needs. DigitalOcean storage values can range from 1 GB to 10,000 GB.

Block storage volumes can be resized through Kubernetes if the DOKS version is recent enough. To resize a block storage volume, update the storage value to a new target size.

Volumes can only be increased in size, but never decreased.

Billing for the block storage volume begins when the object is successfully created. To end billing, you must explicitly delete the volume. Make sure to remove the PVC from your cluster before deleting the block storage volume.

Define the Pod

The second resource definition, which defines the pod, might look like:

kind: Pod
apiVersion: v1
  name: my-csi-app
    - name: my-frontend
      image: busybox
      - mountPath: "/data"
        name: my-do-volume
    - name: my-do-volume
        claimName: csi-pvc


This adds a pod called my-csi-app based on the Linux BusyBox image that names the csi-pvc volume my-do-volume and mounts it within the container at /data on the filesystem.

The pod created in this example will immediately exit, so the API will continue to recreate the pod until you explicitly delete it.

Show Volumes

Within the cluster, volumes will be identified by their names as defined in the claimName parameter. In the example above, the name is csi-pvc.

Regardless of what you set this name to be, the name of the volume on DigitalOcean will begin with pvc- and end with a unique identifying number, something like pvc-0213ed0abexample. You can list the storage volumes associated with a cluster with the get pv command:

kubectl -kubeconfig=[full path to cluster config file] get pv

The output looks something like:

pvc-0213ed0abexample   5Gi        RWO            Delete           Bound     default/csi-pvc   do-block-storage             11s

Setting Permissions on Volumes

By default, the filesystem owner of a DigitalOcean Block Storage Volume is root:root. If a Pod is running as a non-root user and needs to create files or directories on the volume, this will fail due to insufficient or incorrect permissions. However, the following mountOptions settings are not supported by DigitalOcean Kubernetes:

- dir_mode=0777
- file_mode=0777


The solution is to create a temporary container to change the permissions/ownership of the volume's filesystem using initContainers.

We're going to reuse the same PVC config as described in the example above and change the pod definition.

Define the Pod

In the example below, we'll be using the block storage volume we just created to persist data written to a Postgres database. The pod's resource definition might look like:

apiVersion: v1
kind: Pod
  name: my-csi-app
    - name: my-db
      image: postgres:latest
      - mountPath: "/var/lib/postgresql"
        name: my-do-volume
  - name: pgsql-data-permission-fix
    image: busybox
    command: ["/bin/chmod","-R","777", "/data"]
    - name: my-do-volume
      mountPath: /data
    - name: my-do-volume
        claimName: csi-pvc


This adds a pod called my-csi-app based on the latest postgres image that names the csi-pvc volume my-do-volume and mounts it within the container at /data on the filesystem. This also creates an initContainer that temporarily mounts the volume and changes the file permissions for the specified path to 777. The initContainer then deletes itself. This all happens before the volume is mounted to the container. As a note, if you are using securityContext in the yaml file for your Pod, you can use chown $userid instead of chmod 777.


  runAsUser: 1000
  fsGroup: 2000


Check the Volume Permissions

Once the cluster has been created, you can confirm the permissions were correct by checking the log with kubectl:

$ kubectl logs my-csi-app

The output should look like the following:

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok


In the DigitalOcean Control Panel, cluster resources (worker nodes, load balancers, and block storage volumes) are listed outside the Kubernetes page. If you rename or otherwise modify these resources in the control panel, you may render them unusable to the cluster or cause the reconciler to provision replacement resources. To avoid this, manage your cluster resources exclusively with kubectl or from the control panel's Kubernetes page.

As mentioned above, if the DigitalOcean Block Storage Volume is removed manually before the PVC API object is removed with kubectl, this can cause issues. For instance, it can cause the PVC deletion to hang and never complete. If this happens, you can try the following:

kubectl get volumeattachments

The output will look something like this:

$VOLUME_NAME   2019-03-08T21:58:24Z

Use your volume's name, displayed by the previous command, in the commands below to gather information you'll need to try to fix the issue.

kubectl describe volumeattachments $VOLUME_NAME
kubectl edit volumeattachment $VOLUME_NAME

The edit command above will allow us to edit the PVC using a text editor. Remove the following from the volume attachment metadata section, and save your changes:


Now, you can try removing the PVC:

kubectl delete pvc csi-pvc


If those steps don't work, you can open a ticket with Support.


For more about managing persistent volumes see: