Inode limit, how to increase inode limit on block storage volume?

December 23, 2018 1.9k views
Block Storage DigitalOcean Linux Commands Storage Ubuntu 18.04

I’ve successfully added a 100gb block storage volume to my droplet, which was originally added to store a few million .json files and a few million image files. The issue I am noticing at the moment is that after the first 3 million .json files the inode used % is already at 85%, even though the storage usage is only at 12%.

seen here: https://imgur.com/a/NEM0e3i

What I am looking to do is reformat the volume and hopefully adjust the inode table to better fit my 100gb disk space and the files I am looking to store. In total I am looking to store around 8-10 million small .json files and around 4-5 million image files.

If possible I am looking for some guidance/how to on how to adjust the inode table to allow storage of these millions of files, ideally from the digitalocean console command line.

Currently I am trying to avoid my default solution of just increasing the volume size to something like 500gb just to increase the inode limit and end up in total using 20% of the 500gb storage and paying 5x the cost.

I am not a server admin in general so most of this is learn as you go/trial and error,

Any help or insight would be appreciated,

Thanks

-Matt

1 Answer
unixynet December 26, 2018
Accepted Answer

How small/large are these files? You can set the number of inodes at format-time of the block storage. It can be as high as 4 billion inodes. So partition the new the block storage as ext4 with parted then format it with this additional parameter:

mkfs.ext4 -N 4000000000 /dev/path/to/volume

Then mount it as you mount any volume and rsync the existing json files. I wouldn’t create it with a capacity of 4 billions inodes as that’s a bit extreme. It also depends on the average size of the json files. Maybe set it to 1 or 2 billion max.

Cheers

  • Thank you for answering, this is essentially what I was looking for, the server side stuff is a bit foreign to me. I ended up creating a new volume with xfs structure instead of ext4 as from my reading will auto calibrate inodes consistently as the data is stored in the volume, though most of my reading did conclude that ext4 is still a better alternative to xfs for multiple reasons.

    Again appreciate the help.

Have another answer? Share your knowledge.