EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!

Recently, one of my servers has started to show this message in syslog:

Feb 21 21:44:18 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:18 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:18 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:18 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:18 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:29 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:29 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:29 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:29 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:29 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!

The message Directory index full! sound like inode’s problem. But after checking it, there isn’t any problem:

df -i

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 25600000 393314 25206686 2% /
tmpfs 2046261 4 2046257 1% /lib/init/rw
udev 2046261 2255 2044006 1% /dev
tmpfs 2046261 1 2046260 1% /dev/shm
/dev/sda3 95477760 23 95477737 1% /mnt/backup
/dev/sdb1 122101760 24863770 97237990 21% /var/www/willy

20% is high, but it shouldn’t be a problem. After googling about it, a found it could be a problem if some dir has lot of files. Maybe not too much to fill up the inodes in the partition, but too much for a dir.

Inside /var/www/willy are the documentroots for the server’s vhosts. So I checked how many files had each:

cd /var/www/willy
for dir in `ls -1`; do echo $dir; find ./$dir -type f|wc -l; done

domain1.com
2
domain2.com
256
domain3.com
65591
domain4.com
3
domain5.com
2
domain 6.com
154
domain7.com

The domain domain7.com spent lot of time in show his file’s number. I had to stop the script because it didn’t look to finish soon. I think find command is a slow way to count files, so I stopped the script, research inside domain7.com directory and l found this:

cd cache_files1/
ls -1|wc -l

20308227

20 millions inside only one directory. For sure it has to be a problem. After spoke with the webmaster who manage this server, he told me the domain could be deleted, because it had only 5 visits/day. He will rebuild the cache using some directories inside.

After remove the directory, the messages in syslog disappeared.