Something is spewing files on our box
October 17, 2007 6:14 PM Subscribe
On our Linux web server, for some reason /var/lib/php5 has filled up with files to the point that we can't even get a directory listing. We are running out of inode space for the filesystem. How can we get a directory listing to find out which files need to be removed, and remove them?
Response by poster: Ah yes, that did it - thanks heaps! Turns out they are php session files.
posted by Dag Maggot at 6:32 PM on October 17, 2007
posted by Dag Maggot at 6:32 PM on October 17, 2007
Stop the webserver.
Try: 'find /var/lib/php5 -ls > /tmp/bad_dog_bad'
Look through the output in /tmp/bad_dog_bad.
Then 'find -inode XXX -delete'.
You may have to read the man pages on 'find' for some help. Normal user stuff like 'ls' tends to try and read everything and sort it or some such, 'find' just reads the directory structure in "native" order without such stuff.
(on preview, how do you make <pre> tags not double space?)
posted by zengargoyle at 6:36 PM on October 17, 2007
Try: 'find /var/lib/php5 -ls > /tmp/bad_dog_bad'
Look through the output in /tmp/bad_dog_bad.
Then 'find -inode XXX -delete'.
You may have to read the man pages on 'find' for some help. Normal user stuff like 'ls' tends to try and read everything and sort it or some such, 'find' just reads the directory structure in "native" order without such stuff.
$ ls . | head 001_Michael_Mills_02_Adam_Ant_and_Bow_Wow_Wow.mp3 006.jpg 016.jpg 01_godspeed_you_black_emperor_storm.mp3 02 Track 2.wma 032.jpg 034.jpg 046.jpg 04_dead_eyes.mp3 059.jpg $ find . | head . ./.bashrc ./.bash_profile ./.fluxbox ./.fluxbox/keys ./.fluxbox/menu ./.fluxbox/init ./.fluxbox/apps ./.fluxbox/styles ./.fluxbox/styles/CthulhainGood luck!
(on preview, how do you make <pre> tags not double space?)
posted by zengargoyle at 6:36 PM on October 17, 2007
I've always been partial to
cd
for i in *;do ls $i;done
That way it doesn't try to buffer it all into memory at once, it runs it a command at a time. Pretty lightweight.
I use it to comb through directories of 250,000+ files.
Find can be kind of CPU intensive.
posted by fnord at 7:05 PM on October 17, 2007 [1 favorite]
cd
for i in *;do ls $i;done
That way it doesn't try to buffer it all into memory at once, it runs it a command at a time. Pretty lightweight.
I use it to comb through directories of 250,000+ files.
Find can be kind of CPU intensive.
posted by fnord at 7:05 PM on October 17, 2007 [1 favorite]
I wonder if
With preview, it looks like the
posted by philomathoholic at 7:16 PM on October 17, 2007
ls -U would work as well as find? (I don't have a large directory to test this on.) The -U is "unsorted, directory order". With preview, it looks like the
<code> tags don't double space things.posted by philomathoholic at 7:16 PM on October 17, 2007
It is buffering it all into memory, it's just the shell that is doing the work, and it's not trying to sort the results. 'for i in *' will read the entire directory before it starts passing individual filenames to the 'ls' command.
'for i in *; do ls $i; done' will break most shells. The shell will expand the '*' into a full list of files... (and without the sorting that plain 'ls' would do, 'echo *' would accomplish the same thing). If a plain 'ls' wouldn't work, or take forever with tons of swapping, a 'for i in *' wouldn't work either. Same problem, has to read all of the files before it starts processing. It's probably the sorting and swapping of memory that makes things grind to a halt.
The beauty of 'find' is that it works at a lower level. It reads the directory inode, then it starts with the first file inode and processes, then on to the next file inode (or recurses into another directory inode). 'find' might take a long time, but it will always be faster than any other method, and use almost no memory. 'find' is your friend.
posted by zengargoyle at 7:19 PM on October 17, 2007
'for i in *; do ls $i; done' will break most shells. The shell will expand the '*' into a full list of files... (and without the sorting that plain 'ls' would do, 'echo *' would accomplish the same thing). If a plain 'ls' wouldn't work, or take forever with tons of swapping, a 'for i in *' wouldn't work either. Same problem, has to read all of the files before it starts processing. It's probably the sorting and swapping of memory that makes things grind to a halt.
The beauty of 'find' is that it works at a lower level. It reads the directory inode, then it starts with the first file inode and processes, then on to the next file inode (or recurses into another directory inode). 'find' might take a long time, but it will always be faster than any other method, and use almost no memory. 'find' is your friend.
posted by zengargoyle at 7:19 PM on October 17, 2007
WOW, seems 'ls -U' is supreme. (boring stuff follows).
'ls -Ui' is now added to my cool fast things... Never thought that 'ls -U' would be faster than 'find .'.
(sorry, I spend too much time optimizing scripts.)
posted by zengargoyle at 7:58 PM on October 17, 2007 [2 favorites]
$ mkdir blah
$ cd blah
$ i=0; while true; do touch $i;i=$(( i + 1 )); done
( wait a while and Ctrl-c)
$ ls | sort -n | tail -1
53296
(number of files in this directory)
$ time ls -Ui > /tmp/junk
real 0m0.364s
$ time find . > /tmp/junk
real 0m0.399s
$ time ls > /tmp/junk
real 0m0.834s
$ time find . -ls > /tmp/junk
real 0m2.143s
$ time for i in *; do ls $i; done > /tmp/junk
real 4m24.637s
$ time echo * > /tmp/junk
real 0m1.375s
'ls -Ui' is now added to my cool fast things... Never thought that 'ls -U' would be faster than 'find .'.
(sorry, I spend too much time optimizing scripts.)
posted by zengargoyle at 7:58 PM on October 17, 2007 [2 favorites]
This thread is closed to new comments.
find /var/lib/php5
not give you a listing? If so, what's the error you get?
posted by eschatfische at 6:21 PM on October 17, 2007