Where is the space in my /var going?

June 22, 2006 4:19 PM

Here's what concerns me: Filesystem Size Used Avail Use% Mounted on /dev/md3 508M 475M 33M 94% /var not much open on /var. So I go to the /var directory and use du to find where the space is going.....

Here's the end of du -k | sort -d -n:

11 ./catman/X11R6
11 ./catman/local
12 ./lost+found
33 ./catman
43 ./run
66 ./lib/mysql/mysql
74 ./lib/mysql
78 ./lock/samba
83 ./lock
180 ./log/samba
281 ./lib/ldap
498 ./local
723 ./lib/slocate
12488 ./lib/rpm
13671 ./lib
74391 ./log
88254 .
everything above ./catman/X11R6 is less 8k or less.

If I understand this right, du is telling me there's 88megs used.

So why does df say 475 used? Where are the other 387 megs? Is there some key concept I'm missing?

posted by Steve3 to Computers & Internet (12 answers total)

Response by poster: Ugh. Sorry, that formatting did not work out like I expected it to....
posted by Steve3 at 4:20 PM on June 22, 2006

An open but deleted file? (lsof might give more information).

If a file is deleted while it's open, it will stop showing up in directory listings, but the space backing it remains allocated. df just reports free space, so will see it. du walks the directory tree, and so won't.
posted by blender at 4:44 PM on June 22, 2006

What I usually like to do is go to the root of a partition, and do:

du -s -h

-h means 'human-readable'. That will show you the total file usage on the partition. You need to run that as root to get an accurate count.

Then I check it against df... if the numbers aren't pretty close, I start digging. That's a strong sign that you've been hacked... the system can be subverted so that crackers can store files on your system, but you can't see them.

You may need to boot up on a clean OS install disk, and poke through the partitions... see if you get a different du result when you do that. If so, you've very likely been rootkitted. If you find files that look suspicious, it's best to back up (and carefully inspect) critical files, and then totally nuke the system and reinstall. You can't EVER be sure a system is completely clean after it has been compromised.
posted by Malor at 4:47 PM on June 22, 2006

How many files are in /var? If you have a ton of small files, they will waste disk space. Usually a single inode is something like 4096 bytes, so a 1-byte file actually makes 4096 bytes of disk space unavailable.
posted by knave at 4:47 PM on June 22, 2006

'du' doesn't include files it can't read. Try running it as root if you aren't already.
posted by mbrubeck at 4:57 PM on June 22, 2006

Did you delete logs without restarting a service? Restart your services one at a time and see if df suddenly shows a lot more free space. I learned this the hard way, many moons ago, the first time I wrote my own Apache log rotation script.
posted by autojack at 5:00 PM on June 22, 2006

Try this:

#!/usr/bin/perl

$dir = $ARGV[0];

($ARGV[0] && (-d $ARGV[0])) || die "Please specify a directory first\n";
open (IN,"du -hx $dir  |") || die "Unable to open DU for $dir : $!\n";

while (){
        $_=~s/^\s+//;
        my $file = $_;
        next unless ($_=~s/^([\d\.]+[MKG])//);
        my $size = $1;
        if ($size=~s/G//){
                $size *= 1024;
                $size *= 1024;
        }
        elsif ($size=~s/M//){
                $size *= 1024;
        }
        else{   
                $size=~s/K//;
        }
        $file=~s/^\s+//;
        $file=~s/\s+$//;

        my $nsize =  sprintf("%030.2f",$size) . "\n";
        $entries{"$nsize$file"} = $file;
}

foreach (sort { $b <> $a } keys %entries){
        print $entries{$_} . "\n";
}

exit(0);

posted by o0o0o at 5:09 PM on June 22, 2006

Well, despite the double spaced lines, it looks like it made it through intact.
posted by o0o0o at 5:10 PM on June 22, 2006

I'll second "blender"'s suggestion about the cause of the problem. One common way to "lose" space is by thinking that

    rm /var/log/messages

will actually free up any storage. If any program (e.g., syslogd) had /var/log/messages open, the space will not be freed until it is closed.
posted by jepler at 5:32 PM on June 22, 2006

Response by poster: I think it must have been from deleting an open file, because after a restart, du says 89 megs used, while DF says 138 used. Still not an exact match, but at least I'm not worried about the partition being immediately ful.

(Yeah, I could have just restarted it before asking mefi, but now I have a better idea what went on.)

Thanks for the Perl, o0o0o, but I did have trouble running it:
% Missing $ on loop variable at ./metafilter.sh line 59.

where line 59 is

foreach (sort { $b <> $a } keys %entries){print $entries{$_} . "\n";
posted by Steve3 at 7:07 PM on June 22, 2006

There shouldn't be any need to restart the system as a whole; just restarting the offending process (or perhaps even kill -HUP it) would suffice.

If you want to find the culprit, try (as root) something like "lsof |grep deleted". On my system gconfd-2, beagled, smbd, artsd and vmware are all guilty (though not in any major way).
posted by blender at 7:59 PM on June 22, 2006

Response by poster: Unfortunately, this machine (a Cobalt Qube 3) doesn't have lsof. Is there a place to grab it from?
posted by Steve3 at 4:56 AM on June 23, 2006