I was trying to install ElasticSearch as described in docs https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes but it failed to start.
I then tried to delete the namespace but it is in Terminating state for 23 hours.
~ kubectl -n kube-logging get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
es-cluster-1 0/1 Terminating 259 46h 10.244.0.172 standard-nodes-sbvz <none> <none>
I tried to recycle the node but it is stuck in Upgrading. Delete is also not working: “Sorry, we were unable to delete this node”
Stuck pod:
Name: es-cluster-1
Namespace: kube-logging
Priority: 0
Node: standard-nodes-sbvz/10.133.50.163
Start Time: Tue, 14 Jan 2020 23:51:01 +0100
Labels: app=elasticsearch
controller-revision-hash=es-cluster-68c88c5d4b
statefulset.kubernetes.io/pod-name=es-cluster-1
Annotations: <none>
Status: Terminating (lasts 23h)
Termination Grace Period: 1s
IP: 10.244.0.172
IPs:
IP: 10.244.0.172
Controlled By: StatefulSet/es-cluster
Init Containers:
fix-permissions:
Container ID: docker://ebffb52e7229d1f3aebfa2a7ffd662712d5170662df58738e66e72109a5d2ec7
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sh
-c
chown -R 1000:1000 /usr/share/elasticsearch/data
State: Terminated
Exit Code: 0
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/elasticsearch/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
increase-vm-max-map:
Container ID: docker://65ea1c14a8944db672e45ff8e92ad14f6460e1c0a330d94700e0dda3421d0e79
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sysctl
-w
vm.max_map_count=262144
State: Terminated
Exit Code: 0
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
increase-fd-ulimit:
Container ID: docker://757e1ad76e8eab7e975f2d5ffdb0c9051539d4a6e9baafaab32224636a5b5814
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sh
-c
ulimit -n 65536
State: Terminated
Exit Code: 0
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
Containers:
elasticsearch:
Container ID: docker://2b42ad591f67996e258b5e21a807f4262fb7e6881bc8e75aca022a62d762766c
Image: docker.elastic.co/elasticsearch/elasticsearch:7.5.1
Image ID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:b0960105e830085acbb1f9c8001f58626506ce118f33816ea5d38c772bfc7e6c
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Terminated
Exit Code: 0
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 15 Jan 2020 22:59:05 +0100
Finished: Wed, 15 Jan 2020 22:59:12 +0100
Ready: False
Restart Count: 259
Limits:
cpu: 1
Requests:
cpu: 100m
Environment:
cluster.name: k8s-logs
node.name: es-cluster-1 (v1:metadata.name)
discovery.seed_hosts: es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch
cluster.initial_master_nodes: es-cluster-0,es-cluster-1,es-cluster-2
ES_JAVA_OPTS: -Xms512m -Xmx512m
Mounts:
/usr/share/elasticsearch/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-es-cluster-1
ReadOnly: false
default-token-852sk:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-852sk
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
How do I force delete pods?
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Same problem here, and it’s something that happens quite often.
The solution to remove the pods in Terminating state is
kubectl delete --force podname
But this is not a good solution, since as the warning message says: “warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.”
Hello,
Have you tried deleting the pod with the --now
flag?
kubectl delete pod pod_name --now
Let me know how it goes. Regards, Bobby
Click below to sign up and get $100 of credit to try our products over 60 days!
The same thing just happened for me. I could not find a way to delete “Terminating” resources correctly, so I just added new nodes to the cluster pool and removed nodes with these hanging resources.