We have been running into an issue with the managed kubernetes cluster not deleting old docker images and filling up the disk space on the nodes causing new deployments to fail when pulling down new images casing an “No space left on device” error.
Reading into the kubernetes documentation the garbage collection should take care of deleting the unused images but this doesnt appear to be the case.
Is there a way to confirm the GC is running on each node and if it is not enable it?
Our current solution to resolve the problem is to setup a new pod that connect the the nodes docker instance and run the delete images command manually.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Hi there,
Could you open a support ticket providing the UUID of your cluster so we can look into this?
Alternatively, recycling the node should be a simple although “bandage” workaround.
We have the same problem, and we tried recycling the node, but after few days it gets full again. We even tried to upgrade the nodes but nothing works.
failed to garbage collect required amount of images. Wanted to free 16903334297 bytes, but freed 0 bytes
should we open a ticket?
We have since updated our nodes to include the Kubernetes defaults for garbage collection. As of version: 1.16.2-do.1 we also have enabled garbage collection of inodes. I would recommend upgrading to the latest cluster versions to ensure you have these changes applied.
Become a contributor for community
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
DigitalOcean Documentation
Full documentation for every DigitalOcean product.
Resources for startups and SMBs
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
The developer cloud
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Get started for free
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.