How to Add Block Storage Volumes to Kubernetes Clusters

When you need to write and access persistent data in a Kubernetes cluster, you can create and access DigitalOcean Block Storage Volumes by creating a PersistentVolumeClaim (PVC) as part of your deployment.

The claim can allow cluster workers to read and write database records, user-generated website content, log files, and other data that should persist after a process has completed.

When managing persistent volume claims, note that:

  • Deleting a deployment will not automatically delete any PVCs that have been created. You’ll have to remove those manually with kubectl delete pvc.

  • If the DigitalOcean Block Storage Volume is deleted before the PVC API object is removed, it may be in an inconsistent state and attempts to remove the PVC will stall or fail. See the troubleshooting instructions for a fix to try in this case.

  • If a PVC by the same name already exists, you will get an error message similar to the following:

    Error from server (AlreadyExists): error when creating "pvc.yml": 
    persistentvolumeclaims "csi-pvc" already exists
    

    Since the volume exists, it cannot be created. The existing volume will be mounted instead.

  • Block storage volumes created in the control panel or via the API cannot be used by your Kubernetes clusters. You must create volumes within Kubernetes in order for your PVCs to use them..

Create a Configuration File

The configuration examples below define two types of objects:

  1. The PersistentVolumeClaim called csi-pvc, which is responsible for locating the block storage volume by name if it already exists and creating the volume if it does not.

  2. The Pod named my-csi-app, which will create containers, then add a mountpoint to the first object and mount the volume there.

Define the Persistent Volume Claim

The first resource definition to define the PVC can look like this:

  
    
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: do-block-storage

  

This example creates a 5GB block storage volume that will be available to the cluster by the name csi-pvc. If a volume by that name does not exist, one will be created. If one already exists, then the existing volume will be mounted on the first object.

The three highlighted values, name, accessModes, and storage can be customized as follows:

  1. The name must be lowercase alphanumeric values and dashes only and unique within the cluster. Within these constraints, you can name it whatever you want.

  2. accessModes must be set to ReadWriteOnce. The other parameters, ReadOnlyMany and ReadWriteMany, are not supported by DigitalOcean volumes. See the Kubernetes documentation for more about accessModes.

  3. The storage value specifies the size of the volume and can be customized to meet your needs. DigitalOcean storage values can range from 1GB to 10,000GB.

Resizing block storage volumes has not yet been implemented, and changing the storage value in the resource definition after the volume has been created will have no effect.

Billing for the block storage volume begins when the object is successfully created. To end billing, you must explicitly delete the volume. Make sure to remove the PVC from your cluster before deleting the block storage volume.

Define the Pod

The second resource definition, which defines the pod, might look like:

  
    
---
kind: Pod
apiVersion: v1
metadata:
  name: my-csi-app
spec:
  containers:
    - name: my-frontend
      image: busybox
      volumeMounts:
      - mountPath: "/data"
        name: my-do-volume
  volumes:
    - name: my-do-volume
      persistentVolumeClaim:
        claimName: csi-pvc

  

This adds a pod called my-csi-app based on the Linux BusyBox image that names the csi-pvc volume my-do-volume and mounts it within the container at /data on the filesystem.

The pod created in this example will immediately exit, so the API will continue to recreate the pod until you explicitly delete it.

Show Volumes

Within the cluster, volumes will be identified by their names as defined in the claimName parameter. In the example above, the name is csi-pvc.

Regardless of what you set this name to be, the name of the volume on DigitalOcean will begin with pvc- and end with a unique identifying number, something like pvc-0213ed0abexample. You can list the storage volumes associated with a cluster with the get pv command:

kubectl -kubeconfig=[full path to cluster config file] get pv

The output looks something like:

NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM             STORAGECLASS       REASON    AGE
pvc-0213ed0abexample   5Gi        RWO            Delete           Bound     default/csi-pvc   do-block-storage             11s

Setting Permissions on Volumes

By default, the filesystem owner of a DigitalOcean Block Storage Volume is root:root. If a Pod is running as a non-root user and needs to create files or directories on the volume, this will fail due to insufficient or incorrect permissions. However, the following mountOptions settings are not supported by DigitalOcean Kubernetes:

  
    
mountOptions:
- dir_mode=0777
- file_mode=0777

  

The solution is to create a temporary container to change the permissions/ownership of the volume’s filesystem using initContainers.

We’re going to reuse the same PVC config as described in the example above and change the pod definition.

Define the Pod

In the example below, we’ll be using the block storage volume we just created to persist data written to a Postgres database. The pod’s resource definition might look like:

  
    
---
apiVersion: v1
kind: Pod
metadata:
  name: my-csi-app
spec:
  containers:
    - name: my-db
      image: postgres:latest
      volumeMounts:
      - mountPath: "/var/lib/postgresql"
        name: my-do-volume
  initContainers:
  - name: pgsql-data-permission-fix
    image: busybox
    command: ["/bin/chmod","-R","777", "/data"]
    volumeMounts:
    - name: my-do-volume
      mountPath: /data
  volumes:
    - name: my-do-volume
      persistentVolumeClaim:
        claimName: csi-pvc

  

This adds a pod called my-csi-app based on the latest postgres image that names the csi-pvc volume my-do-volume and mounts it within the container at /data on the filesystem. This also creates an initContainer that temporarily mounts the volume and changes the file permissions for the specified path to 777. The initContainer then deletes itself. This all happens before the volume is mounted to the container. As a note, if you are using securityContext in the yaml file for your Pod, you can use chown $userid instead of chmod 777.

Example:

  
    
securityContext:
  runAsUser: 1000
  fsGroup: 2000

  

Check the Volume Permissions

Once the cluster has been created, you can confirm the permissions were correct by checking the log with kubectl:

$ kubectl logs my-csi-app

The output should look like the following:

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Troubleshooting

As mentioned above, if the DigitalOcean Block Storage Volume is removed manually before the PVC API object is removed with kubectl, this can cause issues. For instance, it can cause the PVC deletion to hang and never complete. If this happens, you can try the following:

  
    
kubectl get volumeattachments 

  
The output will look something like this:

NAME           CREATED AT
$VOLUME_NAME   2019-03-08T21:58:24Z

Use your volume’s name, displayed by the previous command, in the commands below to gather information you’ll need to try to fix the issue.

  
    
kubectl describe volumeattachments $VOLUME_NAME 
kubectl edit volumeattachment $VOLUME_NAME 

  
The edit command above will allow us to edit the PVC using a text editor. Remove the following from the volume attachment metadata section, and save your changes:

finalizers:
  external-attacher/dobs-csi-digitalocean-com 

Now, you can try removing the PVC:

  
    
kubectl delete pvc csi-pvc

  

If those steps don’t work, you can open a ticket with Support.

References

For more about managing persistent volumes see: