What do I do with all this data?
July 5, 2017 12:47 PM Subscribe
I have about 22TB~ of data sitting on a NAS server in our living room and I don't have a clue what the hell I need to do to manage it or save it. I'd seriously hate to lose any of the data on it. What do I need to know? What do I need to do?
My dad recently passed away, he was super into computers and built everything himself. He eventually ended up storing all of our physical media on hard drives and servers and now everything is sitting on a 22TB~ NAS server.
Unfortunately, I really don't know what I need to do to manage it. I haven't even shut it off since he passed away, I'm afraid that I'll wreck it and lose everything. I don't even know what the lights mean on it.
I understand that a NAS is basically... like a big external hard drive? Ideally, I'd like to have a back up of the data on the NAS... probably on a similar server? I don't know anything about managing data or computer hardware. I just don't have a clue where to begin.
One of my dad's friends originally agreed to help me get everything set up the way I'd like it, but flaked out. I tried contacting a computer repair guy in the city and he said it was a "big job" and that I should burn it all onto disks. That doesn't seem like a reasonable solution. So, I'm turning to mefi for some help! What do I do!?
I'd be happy to provide more information as needed, I don't have the greatest computer vocabulary!
My dad recently passed away, he was super into computers and built everything himself. He eventually ended up storing all of our physical media on hard drives and servers and now everything is sitting on a 22TB~ NAS server.
Unfortunately, I really don't know what I need to do to manage it. I haven't even shut it off since he passed away, I'm afraid that I'll wreck it and lose everything. I don't even know what the lights mean on it.
I understand that a NAS is basically... like a big external hard drive? Ideally, I'd like to have a back up of the data on the NAS... probably on a similar server? I don't know anything about managing data or computer hardware. I just don't have a clue where to begin.
One of my dad's friends originally agreed to help me get everything set up the way I'd like it, but flaked out. I tried contacting a computer repair guy in the city and he said it was a "big job" and that I should burn it all onto disks. That doesn't seem like a reasonable solution. So, I'm turning to mefi for some help! What do I do!?
I'd be happy to provide more information as needed, I don't have the greatest computer vocabulary!
22TB is... a lot of data. Like, somewhat unusually large - an order of magnitude more storage than what a reasonable high-end home PC would come with, for example. If your father wasn't actually an amateur film maker, I'm not sure how he'd fill this much space with personal data.
When you say "storing all our physical media on hard drives", do you mean he just copied all your DVDs/BluRays/CDs/etc onto it? Because if so, and you have all the original media (or don't care about losing that), I suspect the actual personal data you'd want to keep is at least an order of magnitude smaller. I suspect you need to go through what's stored on it and actually figure out what is important and what isn't.
(2TB is still a decent amount of data to archive, but it's considerably more tractable than 22TB in that it will fit on a random USB hard drive you can buy from Amazon or whatever, so at the very least you can have a second copy of the data, even if that's not the final archival)
posted by parm at 1:18 PM on July 5, 2017 [4 favorites]
When you say "storing all our physical media on hard drives", do you mean he just copied all your DVDs/BluRays/CDs/etc onto it? Because if so, and you have all the original media (or don't care about losing that), I suspect the actual personal data you'd want to keep is at least an order of magnitude smaller. I suspect you need to go through what's stored on it and actually figure out what is important and what isn't.
(2TB is still a decent amount of data to archive, but it's considerably more tractable than 22TB in that it will fit on a random USB hard drive you can buy from Amazon or whatever, so at the very least you can have a second copy of the data, even if that's not the final archival)
posted by parm at 1:18 PM on July 5, 2017 [4 favorites]
Response by poster: I can access everything right now. It's all set up through the network. They have a location and I can access them as I would any other file on my computer right now. I can see what's stored on it and add/remove data as I please.
posted by VirginiaPlain at 1:18 PM on July 5, 2017
posted by VirginiaPlain at 1:18 PM on July 5, 2017
Oh wow I didn't even look at the size. What parm said. 22TB is a ridiculous amount of data. If it's media, figure out what you want to actually keep. I migrated the data off my 3TB NAS in order to mothball it by getting a 1 year trial membership to Amazon Cloud for a cheap price, pointed the Cloud desktop client at the NAS folders I wanted to migrate, and let it do that for a couple of days. When I got a new media server spun up, I did that in reverse and cancelled my Amazon account.
Is there a reason you need a backup? If your dad built this thing, it's probably RAIDed meaning that there's already redundancy happening in case of a disc failure. What is the final destination for this stuff?
posted by soren_lorensen at 1:26 PM on July 5, 2017
Is there a reason you need a backup? If your dad built this thing, it's probably RAIDed meaning that there's already redundancy happening in case of a disc failure. What is the final destination for this stuff?
posted by soren_lorensen at 1:26 PM on July 5, 2017
When you say "storing all our physical media on hard drives", do you mean he just copied all your DVDs/BluRays/CDs/etc onto it? Because if so, and you have all the original media (or don't care about losing that), I suspect the actual personal data you'd want to keep is at least an order of magnitude smaller. I suspect you need to go through what's stored on it and actually figure out what is important and what isn't.
I think this is key. IMHO it's going to be far more trouble and expense than it's worth to back up 20TB (or whatever) of ripped (or torrented) movies and music, if that's what it ends up being. I have hundreds of BluRays/DVDs, and thousands of CDs. In practice, I watch or listen to a tiny fraction of them, and these days I very often find myself consuming them via streaming services even though I have my own purchased copies. It's just so much more convenient, and the quality differernce doesn't matter in most situations. I suspect that this will be even more true if it's someone else's collection rather than your own.
Even as a notorious data hoarder, I find it unfathomable that someone would have 22TB of actual personal data that's not this kind of media...
posted by primethyme at 1:42 PM on July 5, 2017 [3 favorites]
I think this is key. IMHO it's going to be far more trouble and expense than it's worth to back up 20TB (or whatever) of ripped (or torrented) movies and music, if that's what it ends up being. I have hundreds of BluRays/DVDs, and thousands of CDs. In practice, I watch or listen to a tiny fraction of them, and these days I very often find myself consuming them via streaming services even though I have my own purchased copies. It's just so much more convenient, and the quality differernce doesn't matter in most situations. I suspect that this will be even more true if it's someone else's collection rather than your own.
Even as a notorious data hoarder, I find it unfathomable that someone would have 22TB of actual personal data that's not this kind of media...
posted by primethyme at 1:42 PM on July 5, 2017 [3 favorites]
If this data is important to you, you may want to consider a consultant to come on-site and make sure everything gets backed up properly. If it's set up with RAID 5 or a similar variant, you can lose a single hard drive without a problem, but unless it's configured with a hot spare, if you lose two drives there will be total data loss. There's enough variables that just asking people on the internet may not be able to accurately figure out what he did and where all the data is. If he used a commercial NAS product, it might be fairly straight forward to work with but if he did some custom thing with Linux and raw disks in a homemade server, it could be messy.
It sounds like most of the content is ripped DVDs and CDs? It sounds like what you want to do is get some external hard drives and start backing up the content onto them in ways that will make sense in your mind as far as cataloging them going forward. Some of it is probably things you can let go without worry, but things like tax records or the only copy of family photos might be worth tracking down. External drives fail too, so make sure there's at least two copies of everything that's precious to you.
Is there a reason you need a backup? If your dad built this thing, it's probably RAIDed meaning that there's already redundancy happening in case of a disc failure.
RAID is protection against drive failures; it can't protect against house fires and theft or ransomware getting loose on a system and encrypting the files.
posted by Candleman at 1:43 PM on July 5, 2017 [2 favorites]
It sounds like most of the content is ripped DVDs and CDs? It sounds like what you want to do is get some external hard drives and start backing up the content onto them in ways that will make sense in your mind as far as cataloging them going forward. Some of it is probably things you can let go without worry, but things like tax records or the only copy of family photos might be worth tracking down. External drives fail too, so make sure there's at least two copies of everything that's precious to you.
Is there a reason you need a backup? If your dad built this thing, it's probably RAIDed meaning that there's already redundancy happening in case of a disc failure.
RAID is protection against drive failures; it can't protect against house fires and theft or ransomware getting loose on a system and encrypting the files.
posted by Candleman at 1:43 PM on July 5, 2017 [2 favorites]
Break the problem down into smaller, manageable, pieces. As others have already observed, it's unlikely that all 22 TB of data is actually personal files, and it doesn't make sense to worry too much about ripped or downloaded media, as that is generally easily replaceable. Spend a few hours figuring out what you have before you start catastrophizing about backing up 22TB of data, because that is a difficult task; even writable Blu-Rays are 50GB max, so you'd need hundreds of them to back up that much data. The only thing that can reasonably act as a backup for that much data is another NAS of the same size, and that will cost a couple thousand dollars, and you'll end up with the same problem you have now, except now the data lives in two devices instead of one.
Most likely, unless your dad was into something like video production, you'll find that a very small percentage of that data is stuff that you'd actually be put out about if it got lost. Things like photos and documents just don't take up that much space. Back those up onto a separate device now, perhaps onto two different devices/media or on to some cloud storage as well if you're very worried about losing it.
As for the NAS itself, they're basically ordinary computers with a bit of specialized hardware and software that emphasize storage. It should be possible to figure out what make/model of NAS it is if it's an off-the-shelf one (like Synology or QNAP NASs), but there is the possibility that it's a homebrew/custom NAS as well. Figuring that out will help you figure out what's involved in the NAS's care and feeding, though a well-designed NAS should generally power off cleanly if you press the power button, and power interruptions are unlikely to cause problems unless someone was actively writing data to the NAS at the time, and even then the extent of the problem would be that some of that written data might get lost.
posted by Aleyn at 2:10 PM on July 5, 2017
Most likely, unless your dad was into something like video production, you'll find that a very small percentage of that data is stuff that you'd actually be put out about if it got lost. Things like photos and documents just don't take up that much space. Back those up onto a separate device now, perhaps onto two different devices/media or on to some cloud storage as well if you're very worried about losing it.
As for the NAS itself, they're basically ordinary computers with a bit of specialized hardware and software that emphasize storage. It should be possible to figure out what make/model of NAS it is if it's an off-the-shelf one (like Synology or QNAP NASs), but there is the possibility that it's a homebrew/custom NAS as well. Figuring that out will help you figure out what's involved in the NAS's care and feeding, though a well-designed NAS should generally power off cleanly if you press the power button, and power interruptions are unlikely to cause problems unless someone was actively writing data to the NAS at the time, and even then the extent of the problem would be that some of that written data might get lost.
posted by Aleyn at 2:10 PM on July 5, 2017
I'm sorry for your loss. I agree with others that you should first figure out broadly what all this data is and whether you need it all. ("This terabyte is 1000 movies, this terabyte is half a million mp3s, this terabyte is 100,000 vacation photos, these 6 terabytes are disk images of every computer I've ever had," whatever.) It may turn out to be a smaller problem than you think, if a big chunk is clearly less urgent.
That said, let's assume you want it all, or you don't want to decide right now. It doesn't have to be all that hard or expensive to make a few copies of this NAS for backup. 8TB drives look like a good sweet spot. You can buy them for about $180 each.
So if you want all the data, what I would do is spend $540 for three drives. Connect each one to your computer and copy 1/3 of the data over the local network (just pick folders that add up to 7 or 8TB each -- if things don't quite add up you might need a 4th drive). Plan for this to take a long time (hours or days) and have some weird errors, but it should eventually work. Check some random files on each drive to make sure the backup worked. Put the stack of drives in another physical location, like someone else's house. You now have a second copy that you fully understand, which is good. At this point it is much less urgent to tend to the NAS, and you can wait until you find someone cheap or free to help out.
You might also want a third copy, either for extra safety or because you want a copy to use and a copy for backup. For this I would buy another set of 3 drives from a different vendor (to protect against any correlated failures in the first 3), and make a copy of your copy, which will be faster than recopying from the network.
As a bonus, if you eventually decide you want to run a second NAS for whatever reason (which it sounds like you might, from your question), you can likely "shuck" the external 8TB drives and reuse them in your new NAS.
posted by john hadron collider at 2:32 PM on July 5, 2017 [7 favorites]
That said, let's assume you want it all, or you don't want to decide right now. It doesn't have to be all that hard or expensive to make a few copies of this NAS for backup. 8TB drives look like a good sweet spot. You can buy them for about $180 each.
So if you want all the data, what I would do is spend $540 for three drives. Connect each one to your computer and copy 1/3 of the data over the local network (just pick folders that add up to 7 or 8TB each -- if things don't quite add up you might need a 4th drive). Plan for this to take a long time (hours or days) and have some weird errors, but it should eventually work. Check some random files on each drive to make sure the backup worked. Put the stack of drives in another physical location, like someone else's house. You now have a second copy that you fully understand, which is good. At this point it is much less urgent to tend to the NAS, and you can wait until you find someone cheap or free to help out.
You might also want a third copy, either for extra safety or because you want a copy to use and a copy for backup. For this I would buy another set of 3 drives from a different vendor (to protect against any correlated failures in the first 3), and make a copy of your copy, which will be faster than recopying from the network.
As a bonus, if you eventually decide you want to run a second NAS for whatever reason (which it sounds like you might, from your question), you can likely "shuck" the external 8TB drives and reuse them in your new NAS.
posted by john hadron collider at 2:32 PM on July 5, 2017 [7 favorites]
Very sorry to hear about your loss. It's so tough to have work through all these administrative things while also trying to grieve. Nthing that you should look at the data. It's almost certainly a lot of uncompressed movies. He probably ripped Blu Rays at full resolution. With movies, you're likely better off deleting the movies and TV shows and just buying disks. There might also be a lot of uncompressed music, which can also be sizable. You could probably convert music file formats and save a lot of space, just to get all of this down to a reasonable size.
posted by cnc at 3:22 PM on July 5, 2017
posted by cnc at 3:22 PM on July 5, 2017
Response by poster: Yeah, I know it's a lot of data. I'd really just like to make sure everything is backed up and safe and accessible in the future (all the data is a mix of junk like movies, music, and personal documents). I'm looking for solutions to deal with the data I currently have and go from there. I'm willing to spend the money on a solution.
posted by VirginiaPlain at 4:01 PM on July 5, 2017
posted by VirginiaPlain at 4:01 PM on July 5, 2017
I would start with a program like windirstat (Windows) or Disk Inventory X to see a breakdown of data. You might find that there is a large portion of the data that you don't need to preserve which could simplify any further solutions suggested above.
posted by Gomez_in_the_South at 6:26 PM on July 5, 2017 [3 favorites]
posted by Gomez_in_the_South at 6:26 PM on July 5, 2017 [3 favorites]
Well, are three of the 8TB drives that john hadron collider proposes too much money? If not, what he or she outlines is probably the most straightforward approach.
posted by XMLicious at 6:26 PM on July 5, 2017
posted by XMLicious at 6:26 PM on July 5, 2017
I would highly recommend going through the data first to see how much you actually want to save, before buying anything. Then start with the most valuable files. That way, you're not paying to back up and store data that you already have physical copies of, and which can easily be replaced (e.g. music and movies). Save the unique items, like photos and personal files.
Google the model of NAS you have, and read through the user manual, so you know how to turn it off and what the lights mean. It's not going to fail and delete anything because you hit the wrong button, any more than hitting the wrong button on your laptop deletes your entire hard drive. Computers don't work that way. Your data will be fine.
posted by Autumnheart at 8:04 PM on July 5, 2017
Google the model of NAS you have, and read through the user manual, so you know how to turn it off and what the lights mean. It's not going to fail and delete anything because you hit the wrong button, any more than hitting the wrong button on your laptop deletes your entire hard drive. Computers don't work that way. Your data will be fine.
posted by Autumnheart at 8:04 PM on July 5, 2017
Assuming you won't be changing this data often, you may want to look into archiving the most important parts - blurays stored in a safe will keep for 30+ years, much longer than hard drives.
Of course, i second the opinions above that you probably must cull some of it. 22Tb is a lot of stuff, and unless it's video footage, not all of it may be worth duplicating.
Personally I would burn to Blu-ray everything personal / home made / unique. If anything requires special software, be sure to burn the installers as well.
And buy 8Tb drives for the rest, and put them in a reasonably secure storage. Then make a note to transfer the data again to new drives in 7-8 years.
posted by motdiem2 at 12:17 AM on July 6, 2017
Of course, i second the opinions above that you probably must cull some of it. 22Tb is a lot of stuff, and unless it's video footage, not all of it may be worth duplicating.
Personally I would burn to Blu-ray everything personal / home made / unique. If anything requires special software, be sure to burn the installers as well.
And buy 8Tb drives for the rest, and put them in a reasonably secure storage. Then make a note to transfer the data again to new drives in 7-8 years.
posted by motdiem2 at 12:17 AM on July 6, 2017
If it's a Synology Diskstation, then you can install the Cloud Sync app which can then back everything up to a web service of your choice (eg Amazon Drive, Google Drive or my personal choice, Backblaze) for a monthly fee.
22 Tb of data is going to be very expensive to maintain (Google Drive is about $10 per month for 1 Tb!) but I imagine most of them are copies of films etc. that don't need to be backed up.
Good luck!
posted by almostwitty at 8:07 AM on July 6, 2017
22 Tb of data is going to be very expensive to maintain (Google Drive is about $10 per month for 1 Tb!) but I imagine most of them are copies of films etc. that don't need to be backed up.
Good luck!
posted by almostwitty at 8:07 AM on July 6, 2017
My biggest recco is to not turn off the NAS until you have it backed up... multiple drive systems that run hot as part of their normal operation, don't like to be turned on and off a lot. (personal anecdotal corporate experience).
People with large ripped media collections hate hearing this, but ownership is over. Unless you are a true film purist or have a terrible internet connection, you can find almost everything online streaming and you don't pay to store and take care of it.
As mentioned by others, drive space is cheap if you want to make a copy... long term, after a serious culling, you can look at cold cloud storage like Amazon Glacier, but this seems like a much simpler issue.
Taking care of data is part of what I do and unless you are willing to become semi-pro and start doing some information architecture, it gets messy real fast and 22TB of data is a lot to sift through even if it's all 8GB BluRay rips.
posted by bobdow at 10:15 AM on July 6, 2017 [1 favorite]
People with large ripped media collections hate hearing this, but ownership is over. Unless you are a true film purist or have a terrible internet connection, you can find almost everything online streaming and you don't pay to store and take care of it.
As mentioned by others, drive space is cheap if you want to make a copy... long term, after a serious culling, you can look at cold cloud storage like Amazon Glacier, but this seems like a much simpler issue.
Taking care of data is part of what I do and unless you are willing to become semi-pro and start doing some information architecture, it gets messy real fast and 22TB of data is a lot to sift through even if it's all 8GB BluRay rips.
posted by bobdow at 10:15 AM on July 6, 2017 [1 favorite]
john hadron collider's suggestion of copying everything onto a few large drives is a great (relatively) low-cost, low-expertise way to create a backup. Keep in mind that hard drives don't last forever, so if you're looking for a "forever" solution, you'll need to periodically buy new drives and repeat the process.
If you want a cloud solution, AWS Glacier would cost about $90/month to store 22 TB, and has the advantage that you can use AWS Snowball to move the data instead of spending months uploading it. But.. it would be massively expensive to restore these backups from Glacier, so there should be another layer of redundancy somewhere, like hard drives sitting in a safe deposit box.
Your NAS probably also has the ability to send you email alerts if a disk fails (in which case you need to replace it ASAP because you're temporarily running with reduced redundancy)... so you should dig into its settings and set up those alarms. I had a disk failure in my NAS a month ago and didn't notice it until last week because I hadn't set up the alarms correctly.
posted by qxntpqbbbqxl at 4:59 PM on July 6, 2017 [1 favorite]
If you want a cloud solution, AWS Glacier would cost about $90/month to store 22 TB, and has the advantage that you can use AWS Snowball to move the data instead of spending months uploading it. But.. it would be massively expensive to restore these backups from Glacier, so there should be another layer of redundancy somewhere, like hard drives sitting in a safe deposit box.
Your NAS probably also has the ability to send you email alerts if a disk fails (in which case you need to replace it ASAP because you're temporarily running with reduced redundancy)... so you should dig into its settings and set up those alarms. I had a disk failure in my NAS a month ago and didn't notice it until last week because I hadn't set up the alarms correctly.
posted by qxntpqbbbqxl at 4:59 PM on July 6, 2017 [1 favorite]
This thread is closed to new comments.
posted by soren_lorensen at 1:14 PM on July 5, 2017