Non-hard drive backup needed for paperless office.

September 18, 2007 12:10 PM

Massive non-hard drive backup needed for accounting firm that wants to go almost completely paperless.

Some Background:

I work for a small accounting firm and right now we go through massive amounts of paper for a business our size. Staples is here a couple of times per week to drop off boxes and boxes of paper during our busy times. Its not just wasteful but inefficient.

So what we want to do is instead of making tons of copies for workpapers we just want to have the admin staff scan them and get everyone a dual monitor setup so that they can view client documents on one screen and work on another. Pretty standard nowadays.

The Problem:

With the increased amount of scanning going on our server data is going to sky rocket. Currently client data is sitting at about 30gb. My guess is that it will probably grow 5 gb per year, maybe 10 at most. Currently we are using two hard drives to backup the server. One is once a week and that goes home with the boss, the other is every night and it stays here.

That being said my boss doesn't really like or trust hard drive backups enough. What he wants to do is start doing a weekly backup that doesnt get overwritten at all. What he really wants is a DVD but 30 gb is way more then one DVD and creating 6-7 DVDs a week sounds like a pain in the arse. He also wants something that will stand the test of time.

So Hivemind what are some other solutions I can use other then creating 6-7 DVDs a week?

posted by Tinen to computers & internet (23 comments total)

Your "stand the test of time" requirement is going to lead not to one solution, but a system of solutions that involves an organized schedule of data transition from old to new media.
posted by odinsdream at 12:19 PM on September 18, 2007 [1 favorite]

Any problem with using DAT backups? We use them for backup of a small web server and have had no problems.

Here's a sample from HP.

Looks like they would have plenty of space for what you need.
posted by dforemsky at 12:29 PM on September 18, 2007

You could get a Blue-Ray data disc burner. They hold 50GB each. But what you really probably want is a RAID enclosure (example) since it has redundancy between the discs. Fill it with 1TB discs, and at 50 GB per backup you can have a couple of years of space available. If a drive dies, replace it, and since there is redundancy between discs, you should have no data loss. You are not protected against physical loss, in that case, though, so if there is a fire you are in trouble. I might also get a dreamhost account and keep the last few backups there. Transfer the data over the weekend.
posted by procrastination at 12:37 PM on September 18, 2007

yeah, tape is pretty much what everyone uses still. I'd do HDDs for daily/weekly incrementals and two monthlies to tape (one on-site, one off). you can get drives with magazines that can be set to automatically rotate tapes and such. also, add good backup software (Atempo looked nice last time I saw it, and there's NetBackup as well) for management and scheduling. I'd look into what you could do with network backups too; if your data set doesn't change a whole lot and/or you've got a really decent internet connection (or can get one), you could always kinda outsource the backups to a company, and let them handle dealing with tapes and all that kinda stuff. (I'd still keep a set of backups running on-site - the way to think about this is to determine how easy it'd be to restore, and if your backup provider goes away, it's very hard to do a restore.) I wouldn't do DVDs; even with an autoloader they're a pain in the ass, and they don't seem to last very long.
posted by mrg at 12:56 PM on September 18, 2007

If you're serious about your data, this boss-takes-the hard drive home stuff is bad news. What happens if he forgets, if his house is broken into, etc? Optical media is also a bad idea because it degrades fairly quickly (generally, you can only guarantee them for a few years)

You want to talk to a company that specializes in online backups They provide the server space - you set your computers to back up what has changed every night automatically. This is fairly inexpensive these days and makes recovering from serious data loss easy.
posted by chrisamiller at 1:05 PM on September 18, 2007

How secure though is the online backups? My biggest worry is that the internet isn't the most secure place in the world for highly confidential information that if ever stolen from our office could result in huge law suits since we do have some very high profile clients. That would be my biggest worry about backing up online.
posted by Tinen at 1:59 PM on September 18, 2007

I'd say go with a nice big RAID in the server to deal with 'current' data, and then tape based backup for older stuff.
posted by jjb at 2:24 PM on September 18, 2007

DVD is a poor medium for backing up because the density makes it very susceptible to damage, much more so than CDs. HD-DVD/Blu-Ray would be even worse.

And, though RAID definitely has it's uses it's NOT a backup solution and shouldn't be used as one. Data is is vulnerable to errors, corruption, OS and software problems, and, most likely, user errors. I had that point hammered home the other day when I lost some files due to my own mistakes and it's made me re-think RAID for home use and my approach to backups.

I've not investigated professional backup solutions very much but the tape suggestion is a decent one. But you also need to investigate software that makes backing up convenient (automatic is probably best). Because if it's a pain to back up some people won't bother.
posted by 6550 at 2:25 PM on September 18, 2007

"How secure though is the online backups?"
Sign up with a service that lets you specify your own encryption key that the service provider never sees; that way, the data is secured before it leaves your network (just don't lose the key). And it's certainly more secure than someone carrying a hard drive around the neighbourhood (aren't you worried about that leading to mishaps and law suits?!).

Your storage requirements are very modest by today's standards, so getting a decent RAID setup and online backups will be cheap and easy. By all means also do tapes/discs for extra peace of mind, but pay a firm to collect and store the off-site copies.
posted by malevolent at 2:39 PM on September 18, 2007

So Im a little clueless on the online backing up companies. Could you point me in the direction of a few good companies who do that?
posted by Tinen at 2:44 PM on September 18, 2007

I'd use on of the disk-to-disk solutions by Revinetix.

I was on a funding team that they pitched to a couple of years ago, and they were good enough then that they ended up getting the money.

That and their customers love them. Their stuff is really good.
posted by SlyBevel at 3:56 PM on September 18, 2007

"one of," obviously.
posted by SlyBevel at 3:56 PM on September 18, 2007

This is only theory, my first hand experience is limited to small networks, and some of the theory may be wrong (please everyone, correct my mistakes!), but..

You need several layers of redundancy and backup:

1. Online fault tolerance (RAID-1 in your server, and etc.). This is to preserve uptime. A drive failure won't close down your office, and won't cause everyone to be PO'd because they just lost the last 5 hours of work (as well as more critical things, like lost transactions since the last backup).
2. Local backup. This is for files that are written over, or deleted accidentally, or to track changes over time - RAID is useless for this purpose.
3. Remote backup. Because a catastrophic failure, like a big fire, or whatever, can happen.

On top of that, you have to be careful not to rely on any one piece of media - a single drive, a single tape - because your backup media can fail, and if it does you don't want to have to go too far back in time. So, you rotate media into circulation, and back out of circulation again, and never put consecutive backups on the same media. Hard drives are perfectly adequate backup media, as long as you treat them the same as any other backup media with regard to rotation, off site storage, and etc.

If you are doing differential backups (only the stuff that has changed), rotating backup media gets complicated, but I presume good backup software can figure it out (something to look into though, don't quote me :P).

If you go with an online backup solution, I'd still do periodic backup to hard drive (once a week? once a month?). You never know when a company will flake, and hard drives are cheap..

Finally, with regard to media aging.. There are three issues: actual physical degradation of the media, hardware compatibility, and file format compatibility (software compatibility). I don't have a good feel for how much degradation a hard drive suffers when it sits in a drawer for ten years. CD/DVD don't age particularly well.. Hardware compatibility was a big deal at one time - in the 80s some formats were only readable for a few years, and to some extent that extended into the 90s (think zip disc) - but today, as long as you don't just jump on the new big thing, it isn't a problem. HD-DVD drives still read CD and DVD, and though IDE ports will soon disappear from new computers, availability of external IDE to USB enclosures will be good for many years. Software compatibility is the real problem, and there is no good answer. Do you keep old systems, running old OSs, around just to read old file formats? Because you will run into software that worked on Windows 2000, but not on Vista..
Even software compatibility is less of a problem than it used to be.. I don't know about Vista compatibility, but there isn't much from Windows NT (or 2000) that won't run on XP. That isn't the case for Win 3.1 to 95/98.
odinsdream's solution, "data transition from old to new media" will solve the degradation and the hardware compatibility problems, but it doesn't address the software problem at all.
posted by Chuckles at 5:11 PM on September 18, 2007

Tinen: For online storage, the Wall Street Journal guy liked Mozy some months ago. I've heard interesting things about JungleDisk, which uses Amazon S3 storage. There are other threads, and still others you can search for.
posted by Dave 9 at 5:13 PM on September 18, 2007

If your firm is generating as much paper as you claim, your storage estimates are pretty low. Figure 100kb for a black and white scan of a paper page. A "box" of paper is typically 5 reams, 500 sheets per ream. That's 30mb per box. I'm guessing that "boxes and boxes" means at least 10 boxes, and I'll take "a couple of times a week" to mean literally two.

That's 100kb/page * 500 pages/ream * 5 reams/box * 10 boxes/day * 2 days/week * 52 weeks/year = 31 gigabytes a year. And these are all low estimates, so it could be as much as twice that.

Now keep in mind that the storage you should be keeping this on is not a series of hard drives bought cheaply at retail. Those things fail all the time when used for server applications. You need an honest-to-goodness file server for this, with hardware designed for the task.

To do it right, you need a file server with a proper RAID 5 array attached to it, and I'd start with a minimum of 100gb available storage. You might be able to just attach more disks to your existing server. I'd also recommend a tape drive (LTO2 or LTO3) and at least 20 tapes, maybe more depending on how you want to rotate through them. Expect to spend at least $6-8k on this hardware, more if you buy a new server instead of using your existing one.

I would also recommend software to manage the scanning. Unfortunately, there's nothing cheap AND good. (Personally I'd say there's nothing expensive and good either.) If you don't, though, make sure you have an intelligent structure of folder and file names set up, and that people use it properly, or you'll never find anything.
posted by CrayDrygu at 5:14 PM on September 18, 2007

"odinsdream's solution, "data transition from old to new media" will solve the degradation and the hardware compatibility problems, but it doesn't address the software problem at all."

So you store the data in well-known formats such as TIFF or PDF, and don't lock it up in a proprietary database. I highly suspect that TIFF and PDF files will still be readable 10 years from now, as long as they've survived.
posted by CrayDrygu at 5:17 PM on September 18, 2007

I don't have a good feel for how much degradation a hard drive suffers when it sits in a drawer for ten years.

I read recently that if you stick a hard drive in a drawer, don't count on it lasting more than 3 or 4 years. This is mainly due to mechanical parts (like bearings?) getting stuck in place or degrading over time. The life of the platters themselves should be in the neighborhood of 10 years.

If you pay for a backup service, you don't have to worry about this sort of thing. A good provider will have RAID arrays and will be constantly replacing bad drives with new ones. The end result is that your data will get migrated to new disks gradually and you'll never notice the difference.
posted by chrisamiller at 6:07 PM on September 18, 2007

Enveloc is in the same building as me, and they have a pretty tight service. It's used heavily in the accounting and law industry
posted by nikko at 6:42 PM on September 18, 2007

RAID isnt backup. Backupexec 11 does encrypted backups. No media will 'last forever', nor will any 'online vendor'.
posted by sirliberal at 8:46 PM on September 18, 2007

I rely on backups and use them often. I don't trust tape. Tape has burned me one too many times. I only use tape to keep the Sarbanes Oxley people off my back. I never rely on it. I could care less what happens to those tapes. I don't trust DVD's. They're great for small chunks of short term backup, but they do degrade, and they are easily damaged.

I only trust spinning disks. (Note: disk must be actually spinning to get my trust. As an earlier poster pointed out, a hard drive in a drawer is not the same thing.) To do this right will be expensive. There are ways to replicate raid arrays in distant locations. But it costs.

I used to think of a good backup as something that was static, locked away in a vault somewhere. But my thinking has changed, and now I like my backups "living". In other words, I don't want to back up my data, I want there to be a mirror copy of it somewhere- constantly updated from the production data, and ready to stand in for production data at any time, with little or no efforts by me to actually have to "restore" anything should the shait hit the fan.

It may be hard to convince the boss... but "standing the test of time" may not be the best way to think about the problem. Try to find a solution that stands the test of the present, and keeps your data replicated and available 24/7, always in the moment.
posted by Area Control at 9:20 PM on September 18, 2007

Boss takes hard drives home, boss wants 30G DVDs. Boss has pointy hair and maybe even thinks of backups as something you pop into the system and navigate to the file you want just like in Windows.

You should probably educate him so that you're not spending your precious MeFive dollars asking how to avoid industry standards that work great for the government and Fortune 500 companies. You want longevity? Use DLT. Sure, people have problems with tape, but they have longevity in spades and they're much cheaper than running a RAID 1+0 box 24/7 (which is the other way about this...I'd still be running tape anyway, though). Verify and test once a month or so, you'll be fine. Add an offsiting service and you've got an expandable setup that can take the company from tiny to pretty darn large.

Furthermore, weekly 30G backups are not that big a deal. You'll wind up keeping the monthlies for history's sake. Figure 15 tapes for a year's worth of backups (more for more granularity and to keep weeklies on hand from previous months), you're looking at <500G of harddrive backup. Then get a tape drive for $1-2k, offload the monthlies to tape and send them off to the vault. You could do the whole thing for under $10k, well under if you go whitebox (or as I call them, "custom").
posted by rhizome at 9:58 PM on September 18, 2007

DAT and a grandfather, father, son rotation schedule of tapes. Not foolproof, but pretty effecetive.
posted by dg at 10:20 PM on September 18, 2007

I only trust spinning disks. (Note: disk must be actually spinning to get my trust. As an earlier poster pointed out, a hard drive in a drawer is not the same thing.)

This adds another facet to my uncertainty about hard drive reliability. Say the drive has been spinning for a couple of years, but a particular sector hasn't been touched since it was written at the beginning of that time period, how reliable is that sector? I guess in practice it is pretty reliable in comparison to the service life of the drive, but it is still worth thinking about..

now I like my backups "living". In other words, I don't want to back up my data, I want there to be a mirror copy of it somewhere- constantly updated from the production data,

Unless you are taking snapshots of the data in some way, it isn't a backup at all (i.e. from my previous comment, overwritten files vs. uptime). Perhaps I just misunderstand your point..
posted by Chuckles at 10:46 PM on September 18, 2007