October 08, 2014
You try creating a file on a server and see this error message:
No space left on device
...but you've got plenty of space:
df Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvda1 10321208 3159012 6637908 33% /
Who is the invisible monster chewing up all of your space?
Why, the inode monster of course!
An index node (or inode) contains metadata information (file size, file type, etc.) for a file system object (like a file or a directory). There is one inode per file system object.
An inode doesn't store the file contents or the name: it simply points to a specific file or directory.
The total number of inodes and the space reserved for these inodes is set when the filesystem is first created. The inode limit can't be changed dynamically and every file system object must have an inode.
While it's unusual to run out of inodes before actual disk space, you are more likely to have inode shortages if:
-i flag to view inode usage:
df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvda1 1310720 1310720 0 100% /
You can view more detailed inode information with the following command:
tune2fs -l /dev/xvda1 | grep -i inode
If you are using up all of your inode capacity, the next step is figuring out where all of those little files are. This is a bit of a manual process.
The command below will output the number of files and directory from the top of your file system:
for i in /*; do echo $i; find $i |wc -l; done
/bin 119 /sys 9293 /tmp 1 /usr 10583
The counts above include all nested directories. If you have a single directory with many files, the command above may stall - that's a good hint that's the problem directory. Once you've narrowed it down to a specific directory, you can execute the previous command on that directory:
for i in /usr/*; do echo $i; find $i |wc -l; done
Delete all of the small files in that directory.
The bad news: on the ext* file systems, you can't simply increase the inode limit on an existing volume. You have two options:
If the disk is an LVM, increase the size of the volume.
Backup and create a new a new file system, specifying a higher inode limit:
Scout's Disk Inode Usage Plugin monitors your inode usage and can send an alert when you are approaching full capacity.