The kubernetes CSI volume driver stopped working yesterday. Now we can’t provisioning any volumes what makes the managed kubernetes almost unusable.
For example, if we try to create a simple deplyoment wit a volume as described in your docs (https://www.digitalocean.com/docs/kubernetes/how-to/add-volumes/) the volume doesn’t get provisioned.
Create first the pvc
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: do-test spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: do-block-storage
Then create the deployment
apiVersion: apps/v1 kind: Deployment metadata: name: my-csi-app namespace: ioneaccess spec: selector: matchLabels: app: my-csi-app replicas: 1 template: metadata: labels: app: my-csi-app spec: containers: - name: my-frontend image: busybox volumeMounts: - mountPath: "/data" name: my-do-volume command: [ "sleep", "1000000" ] volumes: - name: my-do-volume persistentVolumeClaim: claimName: csi-test
After creating the PVC resource, the pvc stucks in pending state. And there are no log ouptuts from the csi-do-node pod which indicates that the driver tries or fail to create the volume.
1m 7m 26 do-test.158cc309ab1f4732 PersistentVolumeClaim Normal ExternalProvisioning persistentvolume-controller waiting for a volume to be created, either by external provisioner "dobs.csi.digitalocean.com" or manually created by system administrator 3m 7m 2 do-test.158cc309ac5b708a PersistentVolumeClaim Normal Provisioning dobs.csi.digitalocean.com_master-ionedev-beook_e76a0359-480f-11e9-a030-229fa05727f2 External provisioner is provisioning volume for claim "ioneaccess/do-test" 21s 4m 2 do-test.158cc3392dbd9349 PersistentVolumeClaim Warning ProvisioningFailed dobs.csi.digitalocean.com_master-ionedev-beook_e76a0359-480f-11e9-a030-229fa05727f2 failed to provision volume with StorageClass "do-block-storage": timed out waiting for the condition
The volumes worked well for about 3 weeks. But after we had to recycle a kubernetes nodes due an unhealthy state, no more volumes working (except the ones which were provisioned and attached/mounted before this issue)
Anyone an idea?
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.