Beware of small files!

Last Saturday, several websites started to fail with mysql connection problem, or database missing.

This is a website in Yii:

The table "{{messages}}" for active record class "Message" cannot be found in the database.

In mysql everything look find, but searching in apache error logs, I found some entries like this one:

Error Can't create/write to file '/tmp/#sql_75a4_0.MYI' (Errcode: 28)

I checked my /tmp partition and it look find, only 20% space occupied. But I saw there was thousands of files from php sessions. Luckily I already saw this problem before. It was a site with a custom cache. It created thousands of small files too. The file size it was near to 0 bytes. So the problem was the partition was full, not because size, but inodes. Here there was the same problem, partition size near 27% but inodes was 100% full.

You can check your inodes with:

df -i

Usually all your partitions will be around 1%.

By the way, where there are lot of files, you will have problem if you want to delete them with:

rm sess_*

You can use instead something like:

find /tmp/ -name 'sess_*' |xargs rm