Garbage Collection on Kubernetes Cluster Not Working

We have been running into an issue with the managed kubernetes cluster not deleting old docker images and filling up the disk space on the nodes causing new deployments to fail when pulling down new images casing an “No space left on device” error.

Reading into the kubernetes documentation the garbage collection should take care of deleting the unused images but this doesnt appear to be the case.

Is there a way to confirm the GC is running on each node and if it is not enable it?

Our current solution to resolve the problem is to setup a new pod that connect the the nodes docker instance and run the delete images command manually.

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

We have since updated our nodes to include the Kubernetes defaults for garbage collection. As of version: 1.16.2-do.1 we also have enabled garbage collection of inodes. I would recommend upgrading to the latest cluster versions to ensure you have these changes applied.

We have the same problem, and we tried recycling the node, but after few days it gets full again. We even tried to upgrade the nodes but nothing works.

failed to garbage collect required amount of images. Wanted to free 16903334297 bytes, but freed 0 bytes

should we open a ticket?

Hi there,

Could you open a support ticket providing the UUID of your cluster so we can look into this?

Alternatively, recycling the node should be a simple although “bandage” workaround.