Question

pod stuck in Terminating for 23 hours

Posted January 16, 2020 272 views
DigitalOcean Managed Kubernetes

I was trying to install ElasticSearch as described in docs https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes but it failed to start.

I then tried to delete the namespace but it is in Terminating state for 23 hours.

~ kubectl -n kube-logging get pods -o wide
NAME           READY   STATUS        RESTARTS   AGE   IP             NODE                  NOMINATED NODE   READINESS GATES
es-cluster-1   0/1     Terminating   259        46h   10.244.0.172   standard-nodes-sbvz   <none>           <none>

I tried to recycle the node but it is stuck in Upgrading. Delete is also not working: “Sorry, we were unable to delete this node”

Stuck pod:

Name:                      es-cluster-1
Namespace:                 kube-logging
Priority:                  0
Node:                      standard-nodes-sbvz/10.133.50.163
Start Time:                Tue, 14 Jan 2020 23:51:01 +0100
Labels:                    app=elasticsearch
                           controller-revision-hash=es-cluster-68c88c5d4b
                           statefulset.kubernetes.io/pod-name=es-cluster-1
Annotations:               <none>
Status:                    Terminating (lasts 23h)
Termination Grace Period:  1s
IP:                        10.244.0.172
IPs:
  IP:           10.244.0.172
Controlled By:  StatefulSet/es-cluster
Init Containers:
  fix-permissions:
    Container ID:  docker://ebffb52e7229d1f3aebfa2a7ffd662712d5170662df58738e66e72109a5d2ec7
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      chown -R 1000:1000 /usr/share/elasticsearch/data
    State:          Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /usr/share/elasticsearch/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
  increase-vm-max-map:
    Container ID:  docker://65ea1c14a8944db672e45ff8e92ad14f6460e1c0a330d94700e0dda3421d0e79
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
    Port:          <none>
    Host Port:     <none>
    Command:
      sysctl
      -w
      vm.max_map_count=262144
    State:          Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
  increase-fd-ulimit:
    Container ID:  docker://757e1ad76e8eab7e975f2d5ffdb0c9051539d4a6e9baafaab32224636a5b5814
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      ulimit -n 65536
    State:          Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
Containers:
  elasticsearch:
    Container ID:   docker://2b42ad591f67996e258b5e21a807f4262fb7e6881bc8e75aca022a62d762766c
    Image:          docker.elastic.co/elasticsearch/elasticsearch:7.5.1
    Image ID:       docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:b0960105e830085acbb1f9c8001f58626506ce118f33816ea5d38c772bfc7e6c
    Ports:          9200/TCP, 9300/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 15 Jan 2020 22:59:05 +0100
      Finished:     Wed, 15 Jan 2020 22:59:12 +0100
    Ready:          False
    Restart Count:  259
    Limits:
      cpu:  1
    Requests:
      cpu:  100m
    Environment:
      cluster.name:                  k8s-logs
      node.name:                     es-cluster-1 (v1:metadata.name)
      discovery.seed_hosts:          es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch
      cluster.initial_master_nodes:  es-cluster-0,es-cluster-1,es-cluster-2
      ES_JAVA_OPTS:                  -Xms512m -Xmx512m
    Mounts:
      /usr/share/elasticsearch/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-es-cluster-1
    ReadOnly:   false
  default-token-852sk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-852sk
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

How do I force delete pods?

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer

Hello,

Have you tried deleting the pod with the --now flag?

kubectl delete pod pod_name --now

Let me know how it goes.
Regards,
Bobby

Submit an Answer