Question
pod stuck in Terminating for 23 hours
I was trying to install ElasticSearch as described in docs https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes but it failed to start.
I then tried to delete the namespace but it is in Terminating state for 23 hours.
~ kubectl -n kube-logging get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
es-cluster-1 0/1 Terminating 259 46h 10.244.0.172 standard-nodes-sbvz <none> <none>
I tried to recycle the node but it is stuck in Upgrading. Delete is also not working: “Sorry, we were unable to delete this node”
Stuck pod:
Name: es-cluster-1
Namespace: kube-logging
Priority: 0
Node: standard-nodes-sbvz/10.133.50.163
Start Time: Tue, 14 Jan 2020 23:51:01 +0100
Labels: app=elasticsearch
controller-revision-hash=es-cluster-68c88c5d4b
statefulset.kubernetes.io/pod-name=es-cluster-1
Annotations: <none>
Status: Terminating (lasts 23h)
Termination Grace Period: 1s
IP: 10.244.0.172
IPs:
IP: 10.244.0.172
Controlled By: StatefulSet/es-cluster
Init Containers:
fix-permissions:
Container ID: docker://ebffb52e7229d1f3aebfa2a7ffd662712d5170662df58738e66e72109a5d2ec7
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sh
-c
chown -R 1000:1000 /usr/share/elasticsearch/data
State: Terminated
Exit Code: 0
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/elasticsearch/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
increase-vm-max-map:
Container ID: docker://65ea1c14a8944db672e45ff8e92ad14f6460e1c0a330d94700e0dda3421d0e79
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sysctl
-w
vm.max_map_count=262144
State: Terminated
Exit Code: 0
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
increase-fd-ulimit:
Container ID: docker://757e1ad76e8eab7e975f2d5ffdb0c9051539d4a6e9baafaab32224636a5b5814
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sh
-c
ulimit -n 65536
State: Terminated
Exit Code: 0
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
Containers:
elasticsearch:
Container ID: docker://2b42ad591f67996e258b5e21a807f4262fb7e6881bc8e75aca022a62d762766c
Image: docker.elastic.co/elasticsearch/elasticsearch:7.5.1
Image ID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:b0960105e830085acbb1f9c8001f58626506ce118f33816ea5d38c772bfc7e6c
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Terminated
Exit Code: 0
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 15 Jan 2020 22:59:05 +0100
Finished: Wed, 15 Jan 2020 22:59:12 +0100
Ready: False
Restart Count: 259
Limits:
cpu: 1
Requests:
cpu: 100m
Environment:
cluster.name: k8s-logs
node.name: es-cluster-1 (v1:metadata.name)
discovery.seed_hosts: es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch
cluster.initial_master_nodes: es-cluster-0,es-cluster-1,es-cluster-2
ES_JAVA_OPTS: -Xms512m -Xmx512m
Mounts:
/usr/share/elasticsearch/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-es-cluster-1
ReadOnly: false
default-token-852sk:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-852sk
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
How do I force delete pods?
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
×
The same thing just happened for me. I could not find a way to delete “Terminating” resources correctly, so I just added new nodes to the cluster pool and removed nodes with these hanging resources.