Report this

What is the reason for this report?

Unable to mount persistent volumes despite persistent volume claim and storage class

Posted on June 7, 2021

Hi,

My Kubernetes operators are unable to spin up any Pods because they are not able to mount persistent volumes (Block storage from Digital Ocean) and Pods remain in PENDING condition forever.

I am simply following this tutorial from Redis blog.

Here are my steps:

hostname:~$ kubectl create namespace redis
namespace/redis created

hostname:~$ kubectl config set-context --current --namespace=redis
Context "do-nyc3-k8s-1-20-2-do-0-nyc3-1623035038930" modified.

hostname:~$ kubectl get nodes
NAME                   STATUS   ROLES    AGE     VERSION
pool-yz2m3trll-8e25f   Ready    <none>   2m2s    v1.20.2
pool-yz2m3trll-8e25x   Ready    <none>   2m9s    v1.20.2
pool-yz2m3trll-8e25y   Ready    <none>   2m44s   v1.20.2

hostname:~$ kubectl get storageClass
NAME                         PROVISIONER                 RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
do-block-storage (default)   dobs.csi.digitalocean.com   Delete          Immediate           true                   5m27s
hostname:~$ kubectl get pvc
No resources found in default namespace.
hostname:~$ kubectl get pv
No resources found

hostname:~$ VERSION=`curl --silent https://api.github.com/repos/RedisLabs/redis-enterprise-k8s-docs/releases/latest | grep tag_name | awk -F'"' '{print $4}'`
hostname:~$ curl --silent -O https://raw.githubusercontent.com/RedisLabs/redis-enterprise-k8s-docs/$VERSION/bundle.yaml
hostname:~$ kubectl apply -f bundle.yaml 
role.rbac.authorization.k8s.io/redis-enterprise-operator created
rolebinding.rbac.authorization.k8s.io/redis-enterprise-operator created
serviceaccount/redis-enterprise-operator created
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/redisenterpriseclusters.app.redislabs.com created
deployment.apps/redis-enterprise-operator created
customresourcedefinition.apiextensions.k8s.io/redisenterprisedatabases.app.redislabs.com created

hostname:~$ kubectl get all --namespace=redis
NAME                                             READY   STATUS    RESTARTS   AGE
pod/redis-enterprise-operator-85d9f666fb-57gpd   1/1     Running   0          38s

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/redis-enterprise-operator   1/1     1            1           42s

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/redis-enterprise-operator-85d9f666fb   1         1         1       42s

hostname:~$ cat <<EOF | kubectl apply -f -
apiVersion: "app.redislabs.com/v1"
kind: "RedisEnterpriseCluster"
metadata:
  name: "test-cluster"
spec:
  nodes: 3
  persistentSpec:
    enabled: true
    storageClassName: "do-block-storage"
    volumeSize: "4Gi" #optional
  redisEnterpriseNodeResources:
    limits:
      cpu: 2000m
      memory: 1.4Gi
    requests:
      cpu: 2000m
      memory: 1.4Gi
EOF
redisenterprisecluster.app.redislabs.com/test-cluster created

hostname:~$ kubectl get all --namespace=redis
NAME                                                READY   STATUS    RESTARTS   AGE
pod/redis-enterprise-operator-85d9f666fb-57gpd      1/1     Running   0          6m53s
pod/test-cluster-services-rigger-869687b654-x4tmf   1/1     Running   0          3m54s

NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/test-cluster      ClusterIP   None            <none>        9443/TCP,8001/TCP,8070/TCP   3m56s
service/test-cluster-ui   ClusterIP   10.245.79.231   <none>        8443/TCP                     3m56s

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/redis-enterprise-operator      1/1     1            1           6m57s
deployment.apps/test-cluster-services-rigger   1/1     1            1           3m58s

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/redis-enterprise-operator-85d9f666fb      1         1         1       6m58s
replicaset.apps/test-cluster-services-rigger-869687b654   1         1         1       3m59s

NAME                            READY   AGE
statefulset.apps/test-cluster   0/3     4m

hostname:~$ kubectl get rec
NAME           NODES   VERSION     STATE                   SPEC STATUS   LICENSE STATE   SHARDS LIMIT   LICENSE EXPIRATION DATE   AGE
test-cluster   3       6.0.20-69   BootstrappingFirstPod   Valid                                                                  67m

I deployed the test cluster with instructions provided here: https://docs.redislabs.com/latest/platforms/kubernetes/concepts/persistent-volumes/

You can see the cluster remains in BootstrappingFirstPod state forever. What am I missing?



This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Hi there,

When Kubernetes operators are unable to mount persistent volumes and Pods remain in a PENDING state, there are several potential issues to investigate:

  1. Check Access Modes:

    • Verify that the Persistent Volume Claims (PVCs) are requesting the ReadWriteOnce access mode, as DigitalOcean Block Storage doesn’t support ReadOnlyMany or ReadWriteMany.
    • Use kubectl describe pvc -n redis to inspect the PVCs and check the AccessModes field.
  2. Inspect Storage Class:

    • Confirm that the do-block-storage Storage Class is set as the default and is properly configured to dynamically provision storage with the correct access mode.
    • Use kubectl describe sc do-block-storage to view the parameters and ensure that the provisioner is set to dobs.csi.digitalocean.com.
  3. Check PVC Binding:

    • After applying your configurations, use kubectl get pvc -n redis to ensure that PVCs are in a Bound state and not stuck in Pending.
    • If PVCs are not binding, look at the events using kubectl describe pvc PVC_NAME -n redis to identify any errors.

Best,

Bobby

The developer cloud

Scale up as you grow — whether you're running one virtual machine or ten thousand.

Get started for free

Sign up and get $200 in credit for your first 60 days with DigitalOcean.*

*This promotional offer applies to new accounts only.