Kubernetes volumes not working

The kubernetes CSI volume driver stopped working yesterday. Now we can’t provisioning any volumes what makes the managed kubernetes almost unusable.

For example, if we try to create a simple deplyoment wit a volume as described in your docs (https://www.digitalocean.com/docs/kubernetes/how-to/add-volumes/) the volume doesn’t get provisioned.

Create first the pvc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: do-test
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: do-block-storage

Then create the deployment

apiVersion: apps/v1 
kind: Deployment
metadata:
  name: my-csi-app
  namespace: ioneaccess
spec:
  selector:
    matchLabels:
      app: my-csi-app
  replicas: 1
  template:
    metadata:
      labels:
        app: my-csi-app
    spec:
      containers:
        - name: my-frontend
          image: busybox
          volumeMounts:
          - mountPath: "/data"
            name: my-do-volume
          command: [ "sleep", "1000000" ]
      volumes:
        - name: my-do-volume
          persistentVolumeClaim:
            claimName: csi-test

After creating the PVC resource, the pvc stucks in pending state. And there are no log ouptuts from the csi-do-node pod which indicates that the driver tries or fail to create the volume.

1m          7m           26        do-test.158cc309ab1f4732                       PersistentVolumeClaim               Normal    ExternalProvisioning   persistentvolume-controller                                                           waiting for a volume to be created, either by external provisioner "dobs.csi.digitalocean.com" or manually created by system administrator
3m          7m           2         do-test.158cc309ac5b708a                       PersistentVolumeClaim               Normal    Provisioning           dobs.csi.digitalocean.com_master-ionedev-beook_e76a0359-480f-11e9-a030-229fa05727f2   External provisioner is provisioning volume for claim "ioneaccess/do-test"
21s         4m           2         do-test.158cc3392dbd9349                       PersistentVolumeClaim               Warning   ProvisioningFailed     dobs.csi.digitalocean.com_master-ionedev-beook_e76a0359-480f-11e9-a030-229fa05727f2   failed to provision volume with StorageClass "do-block-storage": timed out waiting for the condition

The volumes worked well for about 3 weeks. But after we had to recycle a kubernetes nodes due an unhealthy state, no more volumes working (except the ones which were provisioned and attached/mounted before this issue)

Anyone an idea?