web analytics

df showing wrong result

My mysql server suddenly started showing my disk space usage as almost full.

[root@abc /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3       550G  508G   14G  98% /
tmpfs            32G     0   32G   0% /dev/shm
/dev/sda1       2.0G  173M  1.8G   9% /tmp

But when I issued a disk usage check on / , it returned the usage to be very little.

[root@abc /]# du -sch
47G     .
47G     total

I was sure there is something wrong. Hence, I checked for deleted files. You can use either of the following commands.

[root@abc /]# lsof +L1

mysqld    23193     mysql    5u   REG    8,1          184     0       13 /tmp/ibrUf6yr (deleted)
mysqld    23193     mysql    6u   REG    8,1            0     0       14 /tmp/ibOPlmJ8 (deleted)
mysqld    23193     mysql    7u   REG    8,1        56803     0       15 /tmp/ibpGP3mE (deleted)
mysqld    23193     mysql   11u   REG    8,1            0     0       16 /tmp/ibaOw2Bu (deleted)
mysqld    23193     mysql   30r   REG    8,3 494386229248     0 32957780 /var/lib/mysql/mysql/general_log.CSV1 (deleted)

OR

[root@abc /]# ls -l /proc/*/fd/ | grep deleted

lrwx—— 1 root root 64 Oct 22 00:29 11 -> /tmp/ibaOw2Bu (deleted)
lr-x—— 1 root root 64 Oct 22 00:29 30 -> /var/lib/mysql/mysql/general_log.CSV1 (deleted)
lrwx—— 1 root root 64 Oct 22 00:29 4 -> /tmp/ibIjMQoK (deleted)
lrwx—— 1 root root 64 Oct 22 00:29 5 -> /tmp/ibrUf6yr (deleted)

It listed a few files there, out of which the general_log.CSV1 caught my attention. It was related to the mysql general log that was disabled the previous. It had a huge size. I went ahead and stopped mysql and then started it, and it was all good.

[root@abc /]# /etc/rc.d/init.d/mysqld stop

Stopping mysqld: [ OK ]

[root@abc /]# /etc/rc.d/init.d/mysqld start
Starting mysqld: [ OK ]

[root@abc /]# ls -l /proc/*/fd/ | grep deleted
lrwx—— 1 root root 64 Oct 22 01:40 11 -> /tmp/ibIpfEHv (deleted)
lrwx—— 1 root root 64 Oct 22 01:40 4 -> /tmp/ibUBMfTa (deleted)
lrwx—— 1 root root 64 Oct 22 01:40 5 -> /tmp/ibhd3mcB (deleted)
lrwx—— 1 root root 64 Oct 22 01:40 6 -> /tmp/ibeZNuv1 (deleted)
lrwx—— 1 root root 64 Oct 22 01:40 7 -> /tmp/iblMSViY (deleted)
lrwx—— 1 root root 64 Oct 22 00:29 2 -> /tmp/tmprMOXmc (deleted)

[root@abc /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 550G 48G 475G 10% /
tmpfs 32G 0 32G 0% /dev/shm
/dev/sda1 2.0G 173M 1.8G 9% /tmp

The file I mentioned earlier had gone, and the df result also is back to normal. If nothing worked, you may try rebooting the server and it should be good normally.

Leave a reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>