By relgames
I was trying to install ElasticSearch as described in docs https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes but it failed to start.
I then tried to delete the namespace but it is in Terminating state for 23 hours.
~ kubectl -n kube-logging get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
es-cluster-1 0/1 Terminating 259 46h 10.244.0.172 standard-nodes-sbvz <none> <none>
I tried to recycle the node but it is stuck in Upgrading. Delete is also not working: “Sorry, we were unable to delete this node”
Stuck pod:
Name: es-cluster-1
Namespace: kube-logging
Priority: 0
Node: standard-nodes-sbvz/10.133.50.163
Start Time: Tue, 14 Jan 2020 23:51:01 +0100
Labels: app=elasticsearch
controller-revision-hash=es-cluster-68c88c5d4b
statefulset.kubernetes.io/pod-name=es-cluster-1
Annotations: <none>
Status: Terminating (lasts 23h)
Termination Grace Period: 1s
IP: 10.244.0.172
IPs:
IP: 10.244.0.172
Controlled By: StatefulSet/es-cluster
Init Containers:
fix-permissions:
Container ID: docker://ebffb52e7229d1f3aebfa2a7ffd662712d5170662df58738e66e72109a5d2ec7
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sh
-c
chown -R 1000:1000 /usr/share/elasticsearch/data
State: Terminated
Exit Code: 0
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/elasticsearch/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
increase-vm-max-map:
Container ID: docker://65ea1c14a8944db672e45ff8e92ad14f6460e1c0a330d94700e0dda3421d0e79
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sysctl
-w
vm.max_map_count=262144
State: Terminated
Exit Code: 0
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
increase-fd-ulimit:
Container ID: docker://757e1ad76e8eab7e975f2d5ffdb0c9051539d4a6e9baafaab32224636a5b5814
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sh
-c
ulimit -n 65536
State: Terminated
Exit Code: 0
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
Containers:
elasticsearch:
Container ID: docker://2b42ad591f67996e258b5e21a807f4262fb7e6881bc8e75aca022a62d762766c
Image: docker.elastic.co/elasticsearch/elasticsearch:7.5.1
Image ID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:b0960105e830085acbb1f9c8001f58626506ce118f33816ea5d38c772bfc7e6c
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Terminated
Exit Code: 0
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 15 Jan 2020 22:59:05 +0100
Finished: Wed, 15 Jan 2020 22:59:12 +0100
Ready: False
Restart Count: 259
Limits:
cpu: 1
Requests:
cpu: 100m
Environment:
cluster.name: k8s-logs
node.name: es-cluster-1 (v1:metadata.name)
discovery.seed_hosts: es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch
cluster.initial_master_nodes: es-cluster-0,es-cluster-1,es-cluster-2
ES_JAVA_OPTS: -Xms512m -Xmx512m
Mounts:
/usr/share/elasticsearch/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-852sk (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-es-cluster-1
ReadOnly: false
default-token-852sk:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-852sk
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
How do I force delete pods?
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Hello,
Have you tried deleting the pod with the --now flag?
kubectl delete pod pod_name --now
Let me know how it goes. Regards, Bobby
Same problem here, and it’s something that happens quite often.
The solution to remove the pods in Terminating state is
kubectl delete --force podname
But this is not a good solution, since as the warning message says: “warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.”
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
Full documentation for every DigitalOcean product.
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.
New accounts only. By submitting your email you agree to our Privacy Policy
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.