I was trying to install ElasticSearch as described in docs https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes but it failed to start.

I then tried to delete the namespace but it is in Terminating state for 23 hours.

~ kubectl -n kube-logging get pods -o wide
NAME           READY   STATUS        RESTARTS   AGE   IP             NODE                  NOMINATED NODE   READINESS GATES
es-cluster-1   0/1     Terminating   259        46h   10.244.0.172   standard-nodes-sbvz   <none>           <none>

I tried to recycle the node but it is stuck in Upgrading. Delete is also not working: “Sorry, we were unable to delete this node”

Stuck pod:

Name:                      es-cluster-1
Namespace:                 kube-logging
Priority:                  0
Node:                      standard-nodes-sbvz/10.133.50.163
Start Time:                Tue, 14 Jan 2020 23:51:01 +0100
Labels:                    app=elasticsearch
                           controller-revision-hash=es-cluster-68c88c5d4b
                           statefulset.kubernetes.io/pod-name=es-cluster-1
Annotations:               <none>
Status:                    Terminating (lasts 23h)
Termination Grace Period:  1s
IP:                        10.244.0.172
IPs:
  IP:           10.244.0.172
Controlled By:  StatefulSet/es-cluster
Init Containers:
  fix-permissions:
    Container ID:  docker://ebffb52e7229d1f3aebfa2a7ffd662712d5170662df58738e66e72109a5d2ec7
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      chown -R 1000:1000 /usr/share/elasticsearch/data
    State:          Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /usr/share/elasticsearch/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
  increase-vm-max-map:
    Container ID:  docker://65ea1c14a8944db672e45ff8e92ad14f6460e1c0a330d94700e0dda3421d0e79
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
    Port:          <none>
    Host Port:     <none>
    Command:
      sysctl
      -w
      vm.max_map_count=262144
    State:          Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
  increase-fd-ulimit:
    Container ID:  docker://757e1ad76e8eab7e975f2d5ffdb0c9051539d4a6e9baafaab32224636a5b5814
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      ulimit -n 65536
    State:          Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
Containers:
  elasticsearch:
    Container ID:   docker://2b42ad591f67996e258b5e21a807f4262fb7e6881bc8e75aca022a62d762766c
    Image:          docker.elastic.co/elasticsearch/elasticsearch:7.5.1
    Image ID:       docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:b0960105e830085acbb1f9c8001f58626506ce118f33816ea5d38c772bfc7e6c
    Ports:          9200/TCP, 9300/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 15 Jan 2020 22:59:05 +0100
      Finished:     Wed, 15 Jan 2020 22:59:12 +0100
    Ready:          False
    Restart Count:  259
    Limits:
      cpu:  1
    Requests:
      cpu:  100m
    Environment:
      cluster.name:                  k8s-logs
      node.name:                     es-cluster-1 (v1:metadata.name)
      discovery.seed_hosts:          es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch
      cluster.initial_master_nodes:  es-cluster-0,es-cluster-1,es-cluster-2
      ES_JAVA_OPTS:                  -Xms512m -Xmx512m
    Mounts:
      /usr/share/elasticsearch/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-es-cluster-1
    ReadOnly:   false
  default-token-852sk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-852sk
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

How do I force delete pods?

1 comment
  • The same thing just happened for me. I could not find a way to delete “Terminating” resources correctly, so I just added new nodes to the cluster pool and removed nodes with these hanging resources.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer

Hello,

Have you tried deleting the pod with the --now flag?

kubectl delete pod pod_name --now

Let me know how it goes.
Regards,
Bobby

Submit an Answer