Droplet disk is 100% full but I can’t find what is using it

Question

Hi, my Droplet shows 100% on / and my app started failing with “No space left on device”. I deleted a few big log files, but df -h still shows the disk as full. When I run du -sh /* the numbers are way smaller than what df says. Where is the space going and how do I get it back without rebooting?

Bobby · Answer

Hi there, This usually happens when you delete a file that a running process still has open. The file name is gone, so du does not see it, but the space is not freed until the process closes the file or restarts. Big log files are the classic case. Check for it with: sudo lsof +L1 This lists files that are deleted but still open. Look for big ones and note the process name and PID. The clean fix is to restart that service, for example sudo systemctl restart nginx or docker compose restart app. The space comes back right away. If you can’t restart it, you can empty the file through the process: sudo truncate -s 0 /proc//fd/ The PID and FD number are in the lsof output. If lsof shows nothing, check these two other common causes: sudo du -xh / --max-depth=2 | sort -h | tail -20 df -i The first one finds the biggest folders (-x stays on one filesystem). The second checks inodes, because millions of tiny files can fill the disk even when the size looks fine. If you use Docker, docker system df often shows old images and build cache using many GB. docker system prune cleans them, but read what it will remove first. To stop this from coming back, don’t delete logs with rm while the app runs. Use logrotate with copytruncate, or set a max size for Docker logs in /etc/docker/daemon.json: { "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } } Then restart Docker. It only applies to new containers, so recreate them after. Also set a disk usage alert in the DigitalOcean monitoring page, so you know before it hits 100%. If you want to get faster with commands like du, df and lsof without touching a real server, this browser based Linux terminal game is a good place to practice.

alexdo · Answer

Hi there, Bobby covered the most common cause. If lsof +L1 comes back empty and du still doesn’t add up, there is one more case that matches “df says full, du says much less”: files hidden underneath a mount point. If anything writes into a directory while nothing is mounted there, those files land on the root disk. When a filesystem is later mounted on top of that directory, the files are still there using space, but du and ls only see the mounted filesystem, so they become invisible. On a Droplet this most often happens with Block Storage volumes. DigitalOcean’s Volumes docs note that both automatic and manual setup mount the volume under /mnt with the nofail option. That option is what lets the Droplet boot normally even if the volume isn’t attached yet. If your app or a backup job ever ran during such a window, it may have written gigabytes into /mnt/your_volume on the root disk. The same can happen with any other mount, for example a /var/lib/docker or /home you moved to a separate disk. You can look underneath the mounts without unmounting anything or rebooting, by bind-mounting the root filesystem somewhere else: sudo mkdir -p /tmp/rootfs sudo mount --bind / /tmp/rootfs sudo du -xh --max-depth=2 /tmp/rootfs | sort -h | tail -20 A bind mount of / doesn’t carry the other mounts with it, so any mount point directory that shows a large size under /tmp/rootfs is holding hidden files. Check what is in there before deleting anything, since it may be data your app thinks it saved to the volume. Move it onto the volume if you need it, then clean up and remove the bind mount: sudo umount /tmp/rootfs Regards

Become a contributor for community

DigitalOcean Documentation

Resources for startups and AI-native businesses

The developer cloud

Start building today

Report this

Droplet disk is 100% full but I can’t find what is using it

Related questions

Cron job on my Droplet does not run, but works when I run it by hand

Why does my Node.js app use way more memory in production than in local testing?

How do I handle duplicate and retried webhooks safely on my Droplet?

My Droplet got flooded with requests overnight. What should I do now, and how do I prepare for next time?

What Ubuntu packages and services should I check before deploying to a fresh Droplet?

Mysql Data disappears intermittently

What is the best way to send automated email reports from a Python script on a Droplet?

How does SMTP actually work, and why do my emails keep ending up in spam?

Self-hosting observability for your AI agents on a $6 Droplet

Best way to securely access a homelab remotely without relying only on Tailscale?

VLESS VPN connects but no traffic passes through (DigitalOcean VPS + x-ui + Xray)

Nodemailer stopped working after migrating from shared hosting to a DigitalOcean Droplet

Report this

Droplet disk is 100% full but I can’t find what is using it

Related questions

Cron job on my Droplet does not run, but works when I run it by hand

Why does my Node.js app use way more memory in production than in local testing?

How do I handle duplicate and retried webhooks safely on my Droplet?

My Droplet got flooded with requests overnight. What should I do now, and how do I prepare for next time?

What Ubuntu packages and services should I check before deploying to a fresh Droplet?

Mysql Data disappears intermittently

What is the best way to send automated email reports from a Python script on a Droplet?

How does SMTP actually work, and why do my emails keep ending up in spam?

Self-hosting observability for your AI agents on a $6 Droplet

Best way to securely access a homelab remotely without relying only on Tailscale?

VLESS VPN connects but no traffic passes through (DigitalOcean VPS + x-ui + Xray)

Nodemailer stopped working after migrating from shared hosting to a DigitalOcean Droplet