Category Archives: Troubleshooting

EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!

Recently, one of my servers has started to show this message in syslog:

Feb 21 21:44:18 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:18 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:18 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:18 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:18 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:29 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:29 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:29 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:29 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!
Feb 21 21:44:29 servername01 kernel: EXT4-fs warning (device sdb1): ext4_dx_add_entry:1535: Directory index full!

The message Directory index full! sound like inode’s problem. But after checking it, there isn’t any problem:

df -i

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 25600000 393314 25206686 2% /
tmpfs 2046261 4 2046257 1% /lib/init/rw
udev 2046261 2255 2044006 1% /dev
tmpfs 2046261 1 2046260 1% /dev/shm
/dev/sda3 95477760 23 95477737 1% /mnt/backup
/dev/sdb1 122101760 24863770 97237990 21% /var/www/willy

20% is high, but it shouldn’t be a problem. After googling about it, a found it could be a problem if some dir has lot of files. Maybe not too much to fill up the inodes in the partition, but too much for a dir.

Inside /var/www/willy are the documentroots for the server’s vhosts. So I checked how many files had each:

cd /var/www/willy
for dir in `ls -1`; do echo $dir; find ./$dir -type f|wc -l; done

domain1.com
2
domain2.com
256
domain3.com
65591
domain4.com
3
domain5.com
2
domain 6.com
154
domain7.com

The domain domain7.com spent lot of time in show his file’s number. I had to stop the script because it didn’t look to finish soon. I think find command is a slow way to count files, so I stopped the script, research inside domain7.com directory and l found this:

cd cache_files1/
ls -1|wc -l

20308227

20 millions inside only one directory. For sure it has to be a problem. After spoke with the webmaster who manage this server, he told me the domain could be deleted, because it had only 5 visits/day. He will rebuild the cache using some directories inside.

After remove the directory, the messages in syslog disappeared.

 

SSH connection timeout and freezing

Surely you have suffered this problem if you run a command through ssh and it take long to finish.

The ssh client is only waiting the command to finish, so ssh server close the connection because it doesn’t hear nothing from the client for a while.

To avoid this, you can setup in your client, to send a token each X seconds:

vim ~/.ssh/config

And add in the file:

Host *
ServerAliveInterval 180

You can change * for specific host and the configuration will only affect that site.

PHP Sessions garbage collection problem

Recently my partition size reach 100% of inodes in a couple of days. The partition had thousands of php sessions files,

sess_v3kog1m1943ue0ldu4efqo3qa0
sess_v4ila4d6i5pd0if984n4qqp5s5
sess_v7e48q0e9gc101v9hs9e1c4sb0
sess_v9ef5h87f0h7t56l9hiv6gck77
sess_vadd1reqptalvuphqdn5k80091
sess_vbj05moen6mvc8db5ovkdps3g6

Php was not cleaning the old sessions, so the partition reach the 100% size of inodes often.

I don’t know why but in Debian PHP 5.3.3, the garbage collection settings doesn’t delete the old sessions.

If you go to your /etc/php5/apache2/php.ini you will find:

session.gc_probability = 0
session.gc_divisor = 1000

It’s quite weird, because that setting shouldn’t influence in the deletion of the sessions files. You will find this settings in

session.gc_maxlifetime = 1440

But I’m pretty sure, the session files wasn’t deleted after 24min (1440 secdons / 60).

So I changed the garbage collection settings to:

session.gc_probability = 1
session.gc_divisor = 100

And now the sessions are deleted successfully.

Beware of small files!

Last Saturday, several websites started to fail with mysql connection problem, or database missing.

This is a website in Yii:

The table "{{messages}}" for active record class "Message" cannot be found in the database.

In mysql everything look find, but searching in apache error logs, I found some entries like this one:

Error Can't create/write to file '/tmp/#sql_75a4_0.MYI' (Errcode: 28)

I checked my /tmp partition and it look find, only 20% space occupied. But I saw there was thousands of files from php sessions. Luckily I already saw this problem before. It was a site with a custom cache. It created thousands of small files too. The file size it was near to 0 bytes. So the problem was the partition was full, not because size, but inodes. Here there was the same problem, partition size near 27% but inodes was 100% full.

You can check your inodes with:

df -i

Usually all your partitions will be around 1%.

By the way, where there are lot of files, you will have problem if you want to delete them with:

rm sess_*

You can use instead something like:

find /tmp/ -name 'sess_*' |xargs rm