Question

Unable to mount persistent volumes despite persistent volume claim and storage class

Posted June 7, 2021 50 views
RedisKubernetes

Hi,

My Kubernetes operators are unable to spin up any Pods because they are not able to mount persistent volumes (Block storage from Digital Ocean) and Pods remain in PENDING condition forever.

I am simply following this tutorial from Redis blog.

Here are my steps:

hostname:~$ kubectl create namespace redis
namespace/redis created

hostname:~$ kubectl config set-context --current --namespace=redis
Context "do-nyc3-k8s-1-20-2-do-0-nyc3-1623035038930" modified.

hostname:~$ kubectl get nodes
NAME                   STATUS   ROLES    AGE     VERSION
pool-yz2m3trll-8e25f   Ready    <none>   2m2s    v1.20.2
pool-yz2m3trll-8e25x   Ready    <none>   2m9s    v1.20.2
pool-yz2m3trll-8e25y   Ready    <none>   2m44s   v1.20.2

hostname:~$ kubectl get storageClass
NAME                         PROVISIONER                 RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
do-block-storage (default)   dobs.csi.digitalocean.com   Delete          Immediate           true                   5m27s
hostname:~$ kubectl get pvc
No resources found in default namespace.
hostname:~$ kubectl get pv
No resources found

hostname:~$ VERSION=`curl --silent https://api.github.com/repos/RedisLabs/redis-enterprise-k8s-docs/releases/latest | grep tag_name | awk -F'"' '{print $4}'`
hostname:~$ curl --silent -O https://raw.githubusercontent.com/RedisLabs/redis-enterprise-k8s-docs/$VERSION/bundle.yaml
hostname:~$ kubectl apply -f bundle.yaml 
role.rbac.authorization.k8s.io/redis-enterprise-operator created
rolebinding.rbac.authorization.k8s.io/redis-enterprise-operator created
serviceaccount/redis-enterprise-operator created
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/redisenterpriseclusters.app.redislabs.com created
deployment.apps/redis-enterprise-operator created
customresourcedefinition.apiextensions.k8s.io/redisenterprisedatabases.app.redislabs.com created

hostname:~$ kubectl get all --namespace=redis
NAME                                             READY   STATUS    RESTARTS   AGE
pod/redis-enterprise-operator-85d9f666fb-57gpd   1/1     Running   0          38s

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/redis-enterprise-operator   1/1     1            1           42s

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/redis-enterprise-operator-85d9f666fb   1         1         1       42s

hostname:~$ cat <<EOF | kubectl apply -f -
apiVersion: "app.redislabs.com/v1"
kind: "RedisEnterpriseCluster"
metadata:
  name: "test-cluster"
spec:
  nodes: 3
  persistentSpec:
    enabled: true
    storageClassName: "do-block-storage"
    volumeSize: "4Gi" #optional
  redisEnterpriseNodeResources:
    limits:
      cpu: 2000m
      memory: 1.4Gi
    requests:
      cpu: 2000m
      memory: 1.4Gi
EOF
redisenterprisecluster.app.redislabs.com/test-cluster created

hostname:~$ kubectl get all --namespace=redis
NAME                                                READY   STATUS    RESTARTS   AGE
pod/redis-enterprise-operator-85d9f666fb-57gpd      1/1     Running   0          6m53s
pod/test-cluster-services-rigger-869687b654-x4tmf   1/1     Running   0          3m54s

NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/test-cluster      ClusterIP   None            <none>        9443/TCP,8001/TCP,8070/TCP   3m56s
service/test-cluster-ui   ClusterIP   10.245.79.231   <none>        8443/TCP                     3m56s

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/redis-enterprise-operator      1/1     1            1           6m57s
deployment.apps/test-cluster-services-rigger   1/1     1            1           3m58s

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/redis-enterprise-operator-85d9f666fb      1         1         1       6m58s
replicaset.apps/test-cluster-services-rigger-869687b654   1         1         1       3m59s

NAME                            READY   AGE
statefulset.apps/test-cluster   0/3     4m

hostname:~$ kubectl get rec
NAME           NODES   VERSION     STATE                   SPEC STATUS   LICENSE STATE   SHARDS LIMIT   LICENSE EXPIRATION DATE   AGE
test-cluster   3       6.0.20-69   BootstrappingFirstPod   Valid                                                                  67m

I deployed the test cluster with instructions provided here: https://docs.redislabs.com/latest/platforms/kubernetes/concepts/persistent-volumes/

You can see the cluster remains in BootstrappingFirstPod state forever. What am I missing?

1 comment
  • Can you share the output of:
    kubectl describe pod <redis_pod> on any pods trying to spin up?
    kubectl get events -A will also be helpful to see what may be failing.

    Does the “firstPod” have any logs you can read?

Submit an answer

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!