Comments on: Deleting very large directory

Question: Deleting very large directory

justkevin — Thu, 10 Apr 2008 09:42:14 -0800

How to delete a huge directory on Linux without overloading the system?

I've looked at this question which is basically my exact situation, but I can't get any of the solutions described too work without killing the system.

I've tried:
rm -rf mydir
nice rm -rf mydir
nice -n 19 find mydir -type f -exec rm -v {} \;
nice -n 19 rm -rf mydir&

And many other combinations of rm, find rm, nice find rm, etc, but all cause the server load to rise to dangerous levels quickly (I kill the process when `top` hits 20, I'm assuming the machine would hang if allowed to continue).

So is it possible to remove a directory with a lot of files without killing the server?

By: nicwolff

nicwolff — Thu, 10 Apr 2008 09:56:43 -0800

find mydir -type f -print0 | perl -0 -MTime::HiRes=sleep -ne 'unlink; sleep .1;'

By: nicwolff

nicwolff — Thu, 10 Apr 2008 09:58:32 -0800

(Or sleep .05 or .01 or whatever doesn't crush your CPU.)

By: flabdablet

flabdablet — Thu, 10 Apr 2008 10:02:07 -0800

find mydir -type f | while read; do rm -v "$REPLY"; sleep 0.2; done

By: flabdablet

flabdablet — Thu, 10 Apr 2008 10:05:52 -0800

Sorry, should be

find mydir -type f | while read -r; do rm -v "$REPLY"; sleep 0.2; done

just in case any of your filenames have backslashes in them. This won't remove files with newlines in their names, but those are pretty rare.

By: flabdablet

flabdablet — Thu, 10 Apr 2008 10:09:33 -0800

Follow it up with

find mydir -depth -type d | while read -r; do rmdir -v "$REPLY"; sleep 0.2; done

to remove the directory tree, if you have tens of thousands of subdirectories and rm -rf is still too harsh.

By: TravellingDen

TravellingDen — Thu, 10 Apr 2008 10:10:13 -0800

It would be interseting to know both how many files are there (ls |wc -l) and what OS/filesystem is in use.

By: flabdablet

flabdablet — Thu, 10 Apr 2008 10:12:16 -0800

If there are enough files that rm -rf hangs you up, you don't want to be doing anything that sorts their names - TravellingDen's curiosity would be best served with

find mydir -type f | wc -l

By: dws

dws — Thu, 10 Apr 2008 10:14:52 -0800

I did some experiments on this a few years back. Between the 2.4 and 2.6 kernel series, directories became better and adding and opening files, but much slower at deleting them.

Once you've solved your immediate problem, the next step is to avoid building huge directories. One approach is to partition files into subdirectories based on something easily computable, such as the first few characters of the filename, or the md5/sha1 hash of the filename if that gets you a better distribution. This is how git (a distributed version control system) manages huge numbers of files (e.g., the source files for Linux, so if this is how Linux does it for Linux, it's an approach worth considering).

By: justkevin

justkevin — Thu, 10 Apr 2008 10:35:21 -0800

Using flabdablet's method of counting files (and nicwolf's method of deleting them), there's currently a little over 5 million, dropping at a rate of a few hundred a second. Top seems stable, hovering around 3.0, plus or minus.

It's Red Hat ES, kernel 2.6.18.

By: pharm

pharm — Thu, 10 Apr 2008 10:40:37 -0800

ionice will probably help:

$ ionice -c 3 rm -rf <dir>

puts rm in the "idle" io scheduler class, which should mean that it only gets to do IO when nobody else wants to.

By: unixrat

unixrat — Thu, 10 Apr 2008 12:08:26 -0800

The ionice/nice command loads should NOT concern you when they rise.

Trust the scheduler - trust the system. The scheduler *knows* your command has no priority and it will move it aside for other applications when they request it. The load rises because the system starts doing what you requested.

The load of a system is just a measure. The scheduler will still do the correct thing when requested. You are not "killing" anything, I promise.

By: pharm

pharm — Thu, 10 Apr 2008 12:23:09 -0800

The scheduler only knows that the command has the same priority as every other command unless you tell it otherwise.

Although I agree that a high load value isn't in and of itself a bad thing.

By: cschneid

cschneid — Thu, 10 Apr 2008 14:49:04 -0800

Agreed with using ionice, and letting the thing chug away. It will let you use your computer for anything else, since everything else will get higher priority to the disk. And the delete will go when nothing else wants to.

No reason to keep load down, use the computer to it's fullest.

By: flabdablet

flabdablet — Thu, 10 Apr 2008 17:21:05 -0800

Also: if you're regularly creating directories with millions of files in them, you might want to consider putting those on a ReiserFS file system. Reiser is good at that. But read this first.