Comments on: windows file manager for folders with hundreds of millions of files

Question: windows file manager for folders with hundreds of millions of files

yeoz — Thu, 11 Jan 2007 06:41:49 -0800

Can anyone recommend a file manager for windows that can handle directories with hundreds of millions of folders/files?

By: aubilenon

aubilenon — Thu, 11 Jan 2007 09:06:08 -0800

I would expect that EVERYTHING you could do with that would be slow. To get an idea of the maximum speed it could possibly be to load the directory contents, from a cmd prompt (Windows -> Run -> cmd.exe) type "dir /b full_path_to_directory > NUL"

I think that scripts and commandline tools are going to be the way to go here.

By: chrchr

chrchr — Thu, 11 Jan 2007 09:07:45 -0800

Another thing to consider is that the filesystem itself probably performs poorly when you have millions of files in a directory.

By: chairface

chairface — Thu, 11 Jan 2007 10:54:43 -0800

Scripts are the way to go. Most interactive tools (including dir) will attempt to sort the directory before listing it, and that kinda screws you right there. The "/b" in aubilenon's post prevents the sorting so it's a good starting benchmark. GUI interactive tools might be even worse as lots of them like to animate actions which can make any action too slow to be practical.

By: chundo

chundo — Thu, 11 Jan 2007 12:34:15 -0800

I doubt you'll find anything that can deal with it efficiently. Like aubilenon said, if you want any utility out of it (i.e. sorting), the sorting is generally done on the application side, which means sorting hundreds of millions of records AND keeping them in memory so it knows what to display next when you scroll. So any solution will be (a) an enormous memory hog and (b) slow to start up.

Please don't tell my why you want to do this, my head hurts already.

By: moochoo

moochoo — Thu, 11 Jan 2007 13:13:22 -0800

as a side note i had a folder with 1200 images in it everytime an app wrote to that folder my computer hung for a good couple fo seconds.

so i would hate to have to work in a folder that big!!

maybe booting the drive in a linux machine could be a solution?

By: scottnic

scottnic — Thu, 11 Jan 2007 15:40:57 -0800

I had some folders with tens of thousands of image files (nowhere near hundreds of millions, but they did slow Windows Explorer noticeably).

I was frustrated enough with the slowness that I wrote a little command-line program to split the files into subdirectories of files each (where is a command-line option).

It doesn't pay any attention to folders, so it may not do what you need it to do, but I'd be happy to send it to you (with the usual disclaimers -- it was written in a half-hour and I can't guarantee it won't cause problems).

If you're interested, email scott(AT){DOT}com and I can send it to you.

It does read a list of all files in the target directory first, so there may be memory issues if you truly have hundreds of millions of files...