Most cost-effective onsite backup solution?

March 17, 2018 8:31 AM

My photography habit has me filling up hard drive space faster than anything I've done before. I need some way of expanding storage and backing things up that doesn't cost an arm and a leg. I'd like to do it onsite; I'd rather buy stuff once than pay for a subscription forever. What's the cheapest way of doing this that's worth a damn?

Windows user, if that matters. My inclination is to just start buying external hard drives and throwing files on them as my current hard drive fills up. Is there a cheaper paradigm, or a better one that costs the same? If indeed my best bet here is to just start buying hard drives and socking them away in a closet as needed, what's a model with a good bang-to-buck ratio? I don't care all that much about how much space they take up, I'm only generating something on the order of a terabyte of data every few months.

Is it worth setting up some kind of NAS to make the process smoother? This is for stuff that I don't expect to want to access regularly, and I don't need a complete backup of my whole filesystem. I just want a place to store lots of individual files that can stay within my control and which doesn't tie me to a subscription plan.

Bonus question: what's a realistic shelf life for an external hard drive sitting in a box in a closet? I intend to do everything in duplicate, but I know at some point these drives would need to be replaced.

posted by Anticipation Of A New Lover's Arrival, The to Technology (16 answers total)

If your goal is just bulk storage for little cost, and you don’t care about the occasional drive failure, then you’ll do fine just buying drives and sticking them on a shelf. (Since you mention a photography habit, I’m guessing this is the case.) I do suggest getting a NAS that lets you swap “bare” drives in and out, rather than buying “external” drives with their own USB and power cables... mostly because the external ones sometimes use proprietary cables that are easy to lose and difficult to replace a few years later. Ask me how I know! ;P

If losing these files to a busted drive is more than an annoyance, I’d look into getting a NAS that you can put multiple drives in, and set up RAID to duplicate data across them. This would give you some amount of protection against single drive failures, but obviously cost more. You also need to keep track of groups of drives, rather than single ones, since some RAID configs stripe data across drives.

Lifespan is tricky to answer, as most hard drive reliability studies apply to internal drives, and assume the drive will be continuously plugged into a single stationary machine. From the studies I’ve seen, there seems to be a cliff after 3 years or so, with up to 10% failure rate after 3 years and 20% failure rate after 4 years. Backblaze publishes a lot of good data on this.

I haven’t seen any good comparisons to external drives, and I imagine it depends on what kind of use you subject them to. Vibration, physical movement, and power cycles are all bad for hard drives, so my best guess is that external drives that get occasional use will do the same or worse than internal. I wouldn’t plan on a lifespan of more than 3-4 years, but that may be over cautious.
posted by fencerjimmy at 9:34 AM on March 17, 2018 [2 favorites]

Response by poster: Yes, I just want a bucket I can dump files in when my hard drive fills up. I plan to have two buckets, so that if one dies the other is hopefully still good. I would probably just get one enclosure and keep the bare drives wrapped in paper in a shoebox or something, obviously with labels on them. If my house burns down, Dropbox and Adobe have my stuff in their clouds.

A NAS with RAID wouldn't be the worst thing though. It would certainly be more convenient. I guess I'm just wondering if there's a better plan out there than either of these, which I'm not thinking of.
posted by Anticipation Of A New Lover's Arrival, The at 10:17 AM on March 17, 2018

My husband set up a Drobo RAID system for us for expandable backup storage, because he has so much music and I have so many stock photos and very large digital artworks that there's no way we can keep them on our desktops.*

The Drobo is nice because when we want to expand storage, it's a simple matter of popping out one drive and replacing it with a larger one and the Drobo takes care of everything else, and we can have up to 2 drives fail without losing anything. My husband's only complaint is that the fan is a little loud, but he's extremely sensitive to sounds and I don't even notice it.


* Although we're paranoid so the most important stuff goes to offsite storage as well.
posted by telophase at 12:00 PM on March 17, 2018 [1 favorite]

Another option is Blu-Ray discs. A Blu-ray writer is under $100, and the discs seem to run about $50 / terabyte; a bit more than a hard drive.

The disadvantage is that you're limited to 50 GB/disc, or 100 if you are willing to buy the somewhat more expensive discs.
posted by Hatashran at 12:43 PM on March 17, 2018 [1 favorite]

Your question made me curious about the current state of Amazon S3/Glacier as a substitute for this sort of thing. The pricing still is much higher than buying a terabyte of hard drive space every few months, so it isn't a fit for your situation, but I thought I'd report one thing I hadn't seen before: this marketing video seems to indicate that you can download and run an "AWS Storage Gateway" virtual machine and set it up in "file gateway mode", and it will appear over the network like a NAS but actually be backed by the Amazon Cloud, except that thumbnail images are cached locally. So it's the same pricing, but sounds considerably easier to set up than when I checked into it a few years ago. (Their VM doesn't have any way to present the cheapest tier—the Glacier service—as a straight-up filesystem unfortunately, but only as a virtual tape drive device.)
posted by XMLicious at 12:52 PM on March 17, 2018

Just one note, whether it's a HD at a sibling/friend house or in the cloud keep a copy physically "off site".
posted by sammyo at 3:15 PM on March 17, 2018 [1 favorite]

This isn't the answer you want to hear, but I think you should apply some Marie Kondo principles and seriously trim down the amount of stuff you keep. Throwing away all the second and third rate shots will force you to think hard about which ones you really want to keep and why, and that process will make you a better photographer.

For the hardware, something to consider is that 20% of hard drives will fail within 4 years, while an MDisc will in theory last 1000 years.
posted by Lanark at 3:20 PM on March 17, 2018 [2 favorites]

Best answer: I get the western digital “my cloud” NAS drives with 2 drives, and set them to RAID 1 when i get them, so they’re mirrored. I would love a Drobo or something bigger but these have steadily been about $350 each for years, the storage space just keeps going up.

(I usually keep hi res jpgs of my no-question-rejected images instead of the raw files. I’ve never once had to go back into them for some reason. I don’t even cull too heavily, but with my general shooting ratio that’s about 25-50% of my files)
posted by jeweled accumulation at 3:42 PM on March 17, 2018 [2 favorites]

Response by poster: Dropbox and Adobe between them already have me covered for offsite. This is specificially about onsite storage. And of course I cull, but I do that in Lightroom and keep the original files. Currently I'm periodically deleting my culls because they fill up my hard drive, but I'd like to not do that partly because of the risk of messing up and deleting a keeper and partly just because, well, you never know.
posted by Anticipation Of A New Lover's Arrival, The at 4:40 PM on March 17, 2018

Best answer: If indeed my best bet here is to just start buying hard drives and socking them away in a closet as needed, what's a model with a good bang-to-buck ratio?

Best bet here is just keeping an eye on what the drive manufacturers are offering at the top end of their capacity range for 3.5" internal drives, and looking for drives a couple of generations behind that. Those usually end up in the cost per gigabyte sweet spot while also getting bigger every year. 3TB drives can currently be had for around 2c/GB.

Sometimes, though, you will find 3.5" USB3 external drives offered at prices below those for the same capacity in an internal drive.

In any case, the physically largest form factor is always going to get you the lowest cost per gigabyte. Don't bother buying more than say six months' worth of storage ahead of time, because the per-gigabyte costs continue to come down pretty steadily and are not particularly seasonal.

Solid state storage has historically been roughly ten times the per-gigabyte cost of spinning storage for a very long time, but that gap is actually starting to narrow somewhat; spinnies have been needing to innovate their way around physical limits at a faster rate than solid state for the past year or two. So keep an eye on that trend as well.

Also, RAID is not backup. Even RAID 1, where the controller makes two identical copies of everything, only looks like backup. It isn't. The single most likely cause of massive data loss is human error, and RAID offers no protection against that whatsoever.

Buying twice as much storage as you need to hold all your stuff, and just getting religious about making your duplicates by hand, will save you grief in the long run - as will paying at least lip service to making sure your duplicates go on different brands of drive.
posted by flabdablet at 8:36 PM on March 17, 2018 [1 favorite]

I mean exporting all of the “rejects” as jpg and getting rid of the raw files for those. So you still have SOMETHING but I’m talking about the out of focus, poorly exposed kind of rejects, not the 2nd level “eh maybe not” ones; I keep those and sometimes I do want those for personal reasons even if I’m not turning those in.
posted by jeweled accumulation at 8:44 PM on March 17, 2018

Burn DVDs. Much smaller physical storage space, way less expensive.
posted by artdrectr at 11:45 PM on March 17, 2018

Best answer: Burn DVDs. Much smaller physical storage space, way less expensive.

Things have moved on a bit since that was true. To get close to 3TB you'd need six tubs of 100 DVDs, substantially physically bigger than a 3TB 3.5" drive. At $14.62 per tub they work out to 3c/GB, which is more expensive than the drive's 2c/GB. They're also much slower to transfer data to, and the requirement to break up the data into 4.7GB chunks is inconvenient.

Bluray discs do better on all fronts. At 25GB per disc and $20.99 for a tub of 50 they work out to 1.6c/GB, but the physical size of two and a half of those tubs is still well over that of a 3TB drive. And again, though faster to burn than DVDs their transfer rate is still only about a third of that of the drive.

Writeable optical media have questionable longevity, and the less you pay for them up front the more questionable it becomes. Hard drives also have their own wide variety of data rot modes, from deterioration of the magnetic traces recorded on the disc surfaces to assorted kinds of mechanical failure; no digital storage method is really trustworthy for long term archiving. So as well as keeping duplicates at all times, it's good practice to consolidate any archives written more than about four years ago onto new media, preferably using some form of checksumming technique to make sure what you're writing to the new media is uncorrupted.

Because price per gigabyte decreases with time, it will cost you a fair bit less to buy replacement storage to do this consolidation than it did to buy the storage you're retiring afterwards, but it's still something that needs budgeting for.

Picking a storage medium that makes that transfer to new media the least revoltingly slow and painful it possibly can be gives it the best chance of actually getting done.
posted by flabdablet at 12:51 AM on March 18, 2018 [1 favorite]

Is there a cheaper paradigm, or a better one that costs the same?

Just for completeness: Ultrium backup tapes are about half the cost per GB of hard disk drives, transfer data at least as quickly, and are supposed to have a storage life of well over ten years.

Trouble is that using them requires a tape drive, and those typically cost thousands of dollars. If you're only archiving a few terabytes per year, as opposed to making daily or twice-daily multi-terabyte backups, you will never make up the cost of a tape drive from the savings on tapes and your tape drive will be obsolete before it's given you more than 1% of its rated service hours.

Overall, 3.5" hard drives really are very hard to beat.
posted by flabdablet at 8:04 AM on March 18, 2018

I have three external hard drives for my digital paintings, one of which is about ten years old. It only holds 500 GB but still works just fine. It is a Western Digital.
posted by bjgeiger at 10:23 AM on March 18, 2018

From the studies I’ve seen, there seems to be a cliff after 3 years or so, with up to 10% failure rate after 3 years and 20% failure rate after 4 years. Backblaze publishes a lot of good data on this.

The interesting thing about hard drive failures is what they call a bathtub curve. There's an initial failure rate, generally a few years of stability, and then rising failure rates at around the three year mark. They also have a year end summary for 2017 that might be helpful. Because of the bathtub curve it's unwise to try to infer too much about a "new" drive in their lineup, which they point out:

For example, the Seagate 4 TB drive, model ST4000DM005, has a annualized failure rate of 29.08%, but that is based on only 1,255 drive days and 1 (one) drive failure.

I personally distrust the first drive models at any new high capacity. I've always had the worst luck with those sorts of drives, like when I had 500 GB drives and bought my first 1.5 and 2 TB drives; the newer, bigger drives all failed within a year, but the 500 GB drives are all still fine several years later. For that reason I try to keep my backup drives in the middle of current capacity ranges. My previous backups were 2 TB bought when 4 TB drives were already common; my current backups are 4 TB instead of the 6 or 8 TB I could have bought. And now you can even buy 12 TB drives, but I won't.

For my local backups I have a two-slot drive dock that I fill with identical drives, and when they get full I put them on a shelf and plunk new drives in the dock. I don't recommend the Thermaltake USB 3.0 dual drive dock, though, and I don't know if your machine can take advantage of the Thunderbolt dock I do like. If you can find a reliable dock with the right ports for your computer, you can then buy two of any reliable drive and swap them easily.

I've also had a general plan to get a safe deposit box at the bank down the street and start rotating drives out, say, weekly, but they keep bank hours and I don't, and I haven't actually followed through on this. Since all my photos and documents are also in the cloud now, it feels less urgent.
posted by fedward at 11:40 AM on March 18, 2018 [1 favorite]