Kubernetes volumes not working

March 17, 2019 1.3k views
Kubernetes

The kubernetes CSI volume driver stopped working yesterday. Now we can’t provisioning any volumes what makes the managed kubernetes almost unusable.

For example, if we try to create a simple deplyoment wit a volume as described in your docs (https://www.digitalocean.com/docs/kubernetes/how-to/add-volumes/) the volume doesn’t get provisioned.

Create first the pvc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: do-test
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: do-block-storage

Then create the deployment

apiVersion: apps/v1 
kind: Deployment
metadata:
  name: my-csi-app
  namespace: ioneaccess
spec:
  selector:
    matchLabels:
      app: my-csi-app
  replicas: 1
  template:
    metadata:
      labels:
        app: my-csi-app
    spec:
      containers:
        - name: my-frontend
          image: busybox
          volumeMounts:
          - mountPath: "/data"
            name: my-do-volume
          command: [ "sleep", "1000000" ]
      volumes:
        - name: my-do-volume
          persistentVolumeClaim:
            claimName: csi-test

After creating the PVC resource, the pvc stucks in pending state. And there are no log ouptuts from the csi-do-node pod which indicates that the driver tries or fail to create the volume.

1m          7m           26        do-test.158cc309ab1f4732                       PersistentVolumeClaim               Normal    ExternalProvisioning   persistentvolume-controller                                                           waiting for a volume to be created, either by external provisioner "dobs.csi.digitalocean.com" or manually created by system administrator
3m          7m           2         do-test.158cc309ac5b708a                       PersistentVolumeClaim               Normal    Provisioning           dobs.csi.digitalocean.com_master-ionedev-beook_e76a0359-480f-11e9-a030-229fa05727f2   External provisioner is provisioning volume for claim "ioneaccess/do-test"
21s         4m           2         do-test.158cc3392dbd9349                       PersistentVolumeClaim               Warning   ProvisioningFailed     dobs.csi.digitalocean.com_master-ionedev-beook_e76a0359-480f-11e9-a030-229fa05727f2   failed to provision volume with StorageClass "do-block-storage": timed out waiting for the condition

The volumes worked well for about 3 weeks. But after we had to recycle a kubernetes nodes due an unhealthy state, no more volumes working (except the ones which were provisioned and attached/mounted before this issue)

Anyone an idea?

6 Answers

Greetings!

This looks like an issue worthy of opening a ticket with our support team. This in particular sounds like something outside of your scope of influence:

failed to provision volume with StorageClass "do-block-storage": timed out waiting for the condition

You can open a ticket with our team here:
https://cloudsupport.digitalocean.com/s/

Jarland

hi, we are having the same issue, any news regarding this?

We couldn’t solve the problem by ourself.
But after the support restarted our master node, the volumes worked again. (The restart was required due another problem)

The issue is the same here, looks like something is wrong on DO’s side.

same for us here:

waiting for a volume to be created, either by external provisioner “dobs.csi.digitalocean.com” or manually created by system administrator

It seems that the DO API/Event issue also affects the volume mounting
https://status.digitalocean.com/incidents/1ptrlnshssy0
Since this morning we are unable to mount any volumes.

Have another answer? Share your knowledge.