Docker container working as a mail server runs out of space

October 9, 2018 1.3k views
Docker Server Optimization Email Ubuntu 16.04

I will try to describe the issue with as many details as possible.

I have a Digital Ocean server 1GB RAM, 25GB SSD with Ubuntu 16.04. I have a docker container (docker version is: Docker version 18.03.0-ce, build 0520e24).

The docker container contains postfix, dovecot, mysql, roundcube and a few other services.

Everything works fine but when start to receive or send emails the free space diminishes and I end up with no free space.

Before that happens if I run “df -h” I get:

root@mail:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            488M     0  488M   0% /dev
tmpfs           100M  9.3M   90M  10% /run
/dev/vda1        25G   11G   14G  46% /
tmpfs           497M     0  497M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           497M     0  497M   0% /sys/fs/cgroup
/dev/vda15      105M  3.4M  102M   4% /boot/efi
overlay          25G   11G   14G  46% /var/lib/docker/overlay2/17795ba052726099d7ca962eb9756c8e5354483e388c3e326781249954172746/merged
shm              64M     0   64M   0% /var/lib/docker/containers/559088336f8c6948d6940714b7e97aea765bc98ee6f237afe39406b2f3638e67/mounts/shm

When I run out of space “/dev/vda1” says “Used space 25GB” and also “overlay” says “Used 25GB”.

When I access the docker container:

docker attach SOME_ID

and I try to find directories and files of significant size or directories and files generated recently…supposing that in this way I will find the source of the issue…I can’t find anything.

This is quite confusing because according to “df -h” there is no free space and the issue is in the docker container but when I try to find suspicious files I can’t find any.

And also, if I reboot the whole machine (Turn off -> Turn on) the used space is emptied and the mail server gets back to normal. In a few days the issue happens again. The server runs out of space and I have to reboot it again.

May I please ask you for advice how to trace the source of the issue?

1 comment
  • Try du -sh * in / to figure out what’s taking all the space. When you see the map in question, cd to it and run the above again till you find the culprit. I’ve googled a bit but can’t find anything useful related to mail and docker.

2 Answers

The fact that a reboot would “resolve” the issue could imply that whatever uses up the disk space is actually created in one of the temporary folders somewhere on the server. /tmp is the main folder where temp files are created. So next time this happens run this command to find out what’s using it up:

ls -lS /tmp/
du -sh /tmp/*
find /tmp -type -size +64M -exec du -sh {} \;
  • Last command should be:

    find /tmp -type f -size +64M -exec du -sh {} \;
    
    • No, it didn’t work. When I use:

      ls -lS /var/tmp
      

      it says:

      total 0
      

      Rest of the commands say that there are no files inside. If I type:

      df -h
      

      I get this:

      root@mail:/# df -h
      Filesystem      Size  Used Avail Use% Mounted on
      overlay          25G   25G     0 100% /
      tmpfs            64M   64M     0 100% /dev
      tmpfs           497M     0  497M   0% /sys/fs/cgroup
      /dev/vda1        25G   25G     0 100% /etc/opendkim
      shm              64M     0   64M   0% /dev/shm
      tmpfs           497M     0  497M   0% /proc/scsi
      tmpfs           497M     0  497M   0% /sys/firmware
      

      But when I search for files inside /etc/opendkim, there are no big files.

What about this: find /tmp -type -size +64M -exec du -sh {} \;

  • No, nothing. It says this:

    root@mail:/# find /tmp -size +64M -exec du -sh {} \;
    root@mail:/#
    

    and by the way:

    find /tmp -type -size +64M -exec du -sh {} \;
    

    is incorrect. -type should be followed by a letter.

Have another answer? Share your knowledge.