Comments on: Something is spewing files on our box

Question: Something is spewing files on our box

Dag Maggot — Wed, 17 Oct 2007 18:14:03 -0800

On our Linux web server, for some reason /var/lib/php5 has filled up with files to the point that we can't even get a directory listing. We are running out of inode space for the filesystem. How can we get a directory listing to find out which files need to be removed, and remove them?

By: eschatfische

eschatfische — Wed, 17 Oct 2007 18:21:22 -0800

Does

find /var/lib/php5

not give you a listing? If so, what's the error you get?

By: Dag Maggot

Dag Maggot — Wed, 17 Oct 2007 18:32:39 -0800

Ah yes, that did it - thanks heaps! Turns out they are php session files.

By: zengargoyle

zengargoyle — Wed, 17 Oct 2007 18:36:11 -0800

Stop the webserver.

Try: 'find /var/lib/php5 -ls > /tmp/bad_dog_bad'
Look through the output in /tmp/bad_dog_bad.

Then 'find -inode XXX -delete'.

You may have to read the man pages on 'find' for some help. Normal user stuff like 'ls' tends to try and read everything and sort it or some such, 'find' just reads the directory structure in "native" order without such stuff.



$ ls . | head

001_Michael_Mills_02_Adam_Ant_and_Bow_Wow_Wow.mp3

006.jpg

016.jpg

01_godspeed_you_black_emperor_storm.mp3

02 Track 2.wma

032.jpg

034.jpg

046.jpg

04_dead_eyes.mp3

059.jpg



$ find . | head

.

./.bashrc

./.bash_profile

./.fluxbox

./.fluxbox/keys

./.fluxbox/menu

./.fluxbox/init

./.fluxbox/apps

./.fluxbox/styles

./.fluxbox/styles/Cthulhain

Good luck!

(on preview, how do you make <pre> tags not double space?)

By: fnord

fnord — Wed, 17 Oct 2007 19:05:08 -0800

I've always been partial to

cd
for i in *;do ls $i;done

That way it doesn't try to buffer it all into memory at once, it runs it a command at a time. Pretty lightweight.
I use it to comb through directories of 250,000+ files.

Find can be kind of CPU intensive.

By: philomathoholic

philomathoholic — Wed, 17 Oct 2007 19:16:55 -0800

I wonder if ls -U would work as well as find? (I don't have a large directory to test this on.) The -U is "unsorted, directory order".

With preview, it looks like the <code> tags don't double space things.

By: zengargoyle

zengargoyle — Wed, 17 Oct 2007 19:19:04 -0800

It is buffering it all into memory, it's just the shell that is doing the work, and it's not trying to sort the results. 'for i in *' will read the entire directory before it starts passing individual filenames to the 'ls' command.

'for i in *; do ls $i; done' will break most shells. The shell will expand the '*' into a full list of files... (and without the sorting that plain 'ls' would do, 'echo *' would accomplish the same thing). If a plain 'ls' wouldn't work, or take forever with tons of swapping, a 'for i in *' wouldn't work either. Same problem, has to read all of the files before it starts processing. It's probably the sorting and swapping of memory that makes things grind to a halt.

The beauty of 'find' is that it works at a lower level. It reads the directory inode, then it starts with the first file inode and processes, then on to the next file inode (or recurses into another directory inode). 'find' might take a long time, but it will always be faster than any other method, and use almost no memory. 'find' is your friend.

By: zengargoyle

zengargoyle — Wed, 17 Oct 2007 19:58:56 -0800

WOW, seems 'ls -U' is supreme. (boring stuff follows).



$ mkdir blah

$ cd blah

$ i=0; while true; do touch $i;i=$(( i + 1 )); done

( wait a while and Ctrl-c)



$ ls | sort -n | tail -1

53296

(number of files in this directory)



$ time ls -Ui > /tmp/junk

real    0m0.364s



$ time find . > /tmp/junk

real    0m0.399s



$ time ls > /tmp/junk

real    0m0.834s



$ time find . -ls > /tmp/junk

real    0m2.143s



$ time for i in *; do ls $i; done > /tmp/junk

real    4m24.637s



$ time echo * > /tmp/junk

real    0m1.375s

'ls -Ui' is now added to my cool fast things... Never thought that 'ls -U' would be faster than 'find .'.

(sorry, I spend too much time optimizing scripts.)