Here is what I have:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 40G 38G 0 100% /
However, I cannot find what is using the space up with du -sh *
Any ideas how to find what is using up all that space?
Another option that might help you find out what is using up space is ncdu Install it with:
sudo apt-get install ncdu
If you still can't find anything, try running:
lsof / | grep deleted
Sometimes if a file is deleted while it is opened by a process, it will not actually free up the disk space until the process is ended. This command will show if there are any files in that state.
OK for me I had similar issue: df said "29 GB used out of 30 " whereas du said "19 GB total used". Based on http://askubuntu.com/questions/280342/why-do-df-and-du-commands-show-different-disk- I attempted a reboot and sure enough 12GB magically appeared out of no where, so I must have had a process holding onto a file.
For me this revealed that some large log files I had deleted were still using up disk space, as the logging process had not closed them. systemctl restart rsyslog let the files get their proper burial
lsof / | grep deleted worked for me..
Kill all the jobs listed.
lsof / | grep deleted | awk '{ print $2 }' | xargs kill -9
and after the root volume reduced to normal.
Googled and found this question which had the same problem.
@TonyTsang That command helped me find what it was
Apparently it was a hidden log folder set by pm2.
Yes it was some hidden log set by pm2.
I almost clean up 8.8 GB deleted those logs.
rm -rf ~/.pm2/logs/*
outside of other answers here what I did was:
navigate through subfolders with ncdu and finding large folders. Among them one can delete:
/var/logs
https://bbs.archlinux.org/viewtopic.php?id=158510 // /var/logs/journal
http://askubuntu.com/questions/100004/how-can-i-free-space-from-a-massive-39-5gb-var-log-folder // other already archived (GZ-ipped) files;
and compress .git folders of projects by shrinking them with this command: git repack -a -d --depth=250 --window=250
http://stackoverflow.com/questions/5613345/how-to-shrink-the-git-folder
Thank you for responding.
ncdu shows only the disk usage under /home, apparent size 922.3MiB
lsof / | grep deleted shows 8 files of about 4K each.
df -h is still showing
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 40G 38G 0 100% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 991M 8.0K 991M 1% /dev
tmpfs 201M 380K 200M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1001M 0 1001M 0% /run/shm
none 100M 0 100M 0% /run/user
Something is still in /dev/vda1 for 38G that i cannot find
Used ncdu / to check what was using the pace, apparently like 15Gb of mail logs.
I deleted them to clear space, and a day later I got 700Mb log of mail. Considering I'm only hosting 2 wordpress blogs, seems like I got some spam issue going on :((
I'll open another thread for this, because I have no idea how to solve it.
Do you have any cron jobs set up? If so, you can disable email notifications by running crontab -e and:
MAILTO=""
>/dev/null 2>&1 to each cronjob like this, which will redirect all output to /dev/null:5 10 * * * /path/to/my/script >/dev/null 2>&1
There are no cron jobs set up. I checked the mail queue and there were like 4200 spam messages trying to be sent.
I managed to find all the infected files in the wordpress installation, delete them, clear the mail queue, restart the mail server and it seems to be working fine. Some days later, the queue is still empty, and the server usage looks normal. I guess that was an out to date wordpress plugin :S
thanks!
Hi @BreizhoO
I thought it wasn't related to my code, I thought it might be a server problem that create new partition. Apparently I'm wrong, that /dv/vda1 / is the main partition, only this time filled with some huge files, the log files.
First, stop any queue or running cron job as it might be the cause your log files is full.
I solve this by removing some log files that keeps building up when there's an error, and that error was caused by a failed job queue which is keep repeating (keep trying to run the job) and store an error log line everytime it runs. I wasn't aware until my disk was full and got a notification from newrelic.
You can follow my instructions there. Imagine you're going into a room and check the content of each room. :) Basically I'm just trying to find which dir is it. That's why I'm starting from the very top.
Go to root dir by cd /
List dirs and files: ls so you can go to each one of it.
Example results:
/usr
/var
/lib
/home
/dev
...
And do du -h [dir] for each of them:
du -h /usr
du -h /var
...
du -h /home <-- suspicious, contains couple of GBs of size
From each one of it see which one is suspicious (with largest dir size), that's why I said mine was on /home
Then I'm doing cd /home, do the same from here: ls, do du -h /home/vagrant
Then I'm doing cd /home/vagrant, do the same from here: ls, do du -h /home/vagrant/mylaravelproject
Then I'm doing cd /home/vagrant/mylaravelproject, do the same from here: ls, do du -h /home/vagrant/mylaravelproject/storage
Then I'm doing cd /home/vagrant/mylaravelproject/storage, do the same from here: ls, do du -h /home/vagrant/mylaravelproject/storage/logs
Until you finally sure that directory was the culprit.
For me it was on /home/vagrant/mylaravelproject/storage/logs
On that /logs dir there was a huge 8 to 15gb of log files X_x
I remove it.. and voila!
Does it makes sense? Some people here might have better way of finding it though.
I'm having the same issue. Did you find a resolution?
Same issue here!
I am also seeing this
But
After sending a support ticket about this issue, they point me to a direction of solving it myself.
For me there's actually a log file that stacked up until the disk is full.
Here's how I solve it:
I go to root dir:
cd /list dirs and files:
lsAnd do
du -h [dir]for each of themWhen you see something suspicious you go inside that dir, or see from the du result.
For me it's a
du -h homeWhen I trace, it points me to my laravel app storage/logs.
On a sidenote, on my issue here is caused by job queue php artisan. And on those job queue there's an error and it keeps logging and repeating, the php process also hogging the cpu. This solved me two issues altogether.
There you go. I hope it helps.
@josualeonard
Hello, I ve sent a request as well to them, and similar to fix it your self with those links .....
I don't think I have as much knowledge as you do, can you explain to me in a bit more detail how to solve this ?
Problem is the same :
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 30G 28G 0 100% /
I am not quite sure what you mean by cd / ect.. ect... It was all working fine until yesterday.
Thanks for your help !!
In response to
"And do du -h [dir] for each of them"
du -h --max-depth=1 will achieve this much quicker without having go into each folder. look for the once that looks suspicious then perform the same max depth in that folder.