Archiving WAY too many files

December 16, 2005 10:56 AM

Automated file archiving for Windows?

What I need is software that will sit on the server, and at the beginning of every month will automatically zip up any files that were created 2 months ago into one zip file (as in, if it's December, it automatically zips up all files created in September). Most of these files are empty, but I still need to keep them. And lots of them. Several tens of thousands each month (In the past year my software has generated approximately 500,000 files). I need software that will automatically zip up the files into one zip file, but inside the file will preserve all file attributes including creation date and time. I could probably do this using a script and the "at" command, but my boss would like something a bit more "full featured" than that.

And yes, I do actually need to keep all the files, as much as I don't want to.

posted by KirTakat to Computers & Internet (11 answers total)

Won't the Remote Storage service in 2003 Enterprise do that? Otherwise, I remember looking at a feature pack from microsoft that was a sort of HSM system for fileservers... Not finding it now, though.
posted by Triode at 11:09 AM on December 16, 2005

Might HTFileStore do the trick? It got a good review from Techworld.com.
posted by youarenothere at 11:20 AM on December 16, 2005

Response by poster: Umh, from my reading of it, it looks like Remote Storage actually leaves the physical file on the local hard drive, it just moves all the data to a different location for storage. Which is actually a pretty neat way of dealing with large files that I never knew about. Unfortunately my issue isn't storage space (like I said, most of the files are completely empty), but rather, the sheer NUMBER of files. Our backup software apparently starts to run into issues at around 600,000 files, which we are rapidly approaching. Otherwise all the files together are only a couple dozen megabytes.
posted by KirTakat at 11:22 AM on December 16, 2005

If this was a unix system, I'd use a combination of the find and tar commands.

If you have experience at the command line, you could achieve something similar with unxutils, or possibly xxcopy (a better xcopy).

Making the relevant batch script would give you complete control about how it operates.
posted by OldMansHands at 12:03 PM on December 16, 2005

Response by poster: I've considered batch scripts, suggested them, and if all else fails that's the way I'll go. I suppose my boss's point is that he would just prefer an off the shelf option so that way there aren't any issues with supporting it (not that there's much to support in a batch file).

Realistically though, it looks like most solutions that might encompass this are something along the lines of the aforementioned HTFileStore, which looks to me awfully like throwing a 900 lb monkey at a 100 lb monkey problem.
posted by KirTakat at 12:06 PM on December 16, 2005

I have to ask -- most of your files are empty, but you still need to keep them? Surely if they're genuinely empty, you really don't? Or you just need to know the names of the files for re-creation at a later date?
posted by AmbroseChapel at 1:25 PM on December 16, 2005

Use a batch file. Invoke XXCopy and move files with date stampts into a temp dir and zip.

This describes some of the XXCopy date selectors. There are lots more switches.

For 60 days, I think something like one os these will work:

XXCOPY src dst /DA#60 // After 60 days ago
XXCOPY src dst /DB#30 // Before 30 days ago
XXCOPY src dst /DA#60 /DB#30 // files with age of 30-60 days
XXCOPY src dst /DA#60 /DB#30 // files with age of 30-60 days
XXCOPY src dst /DB#60 /DA#30 // age >= 60 or age < 30/small>
posted by meehawl at 1:46 PM on December 16, 2005

Response by poster: Well, the legal department has yet to get back to us, but because these are financial files, there's a few (HA) more rules regarding what we can and can't do with them. We're more or less doing a CYA, and we have to store them for 7 years.
posted by KirTakat at 1:48 PM on December 16, 2005

Response by poster: Meehawl, as I mentioned to OldMansHands, I'm not against (or unable) to make a script for it, my boss specifically asked for a third party product to avoid having to "support" any custom solutions. Although at this point it looks like I am going to have to find a script, as other than the potential HTFileStore (which looks like way too much product for my needs), I haven't really been able to find anything.
posted by KirTakat at 1:51 PM on December 16, 2005

I'd recommend RAR. The command line version is very powerful, and it handles huge archives much better.
posted by Sharcho at 6:33 PM on December 16, 2005

third party product to avoid having to "support" any custom solutions

XXCopy is a third-party product. Perhaps we have different definitions of what "third party" means? I take it to mean "a piece of software or functionality not published or licenced by Microsoft and not included out of the box with Windows".
posted by meehawl at 6:40 AM on December 17, 2005