Cheap Object Storage
January 2, 2020 3:45 PM Subscribe
I need object storage. This is the end of the line of a backup workflow, where you throw a large encrypted archive online (for off-site backup) with no intention of syncing, updating, or downloading (except/until emergencies). Dreamhost's "DreamObjects" offerings seem expensive. Does anyone know a cheaper way, from a solid outfit that's not going to go out of biz anytime soon?
Response by poster: AWS terrifies me. I'm not a server jockey, I'm a mere mortal. Give me FTP or give me death.
posted by Quisp Lover at 3:52 PM on January 2, 2020 [1 favorite]
posted by Quisp Lover at 3:52 PM on January 2, 2020 [1 favorite]
There is AWS SFTP, which might meet your needs, but I am admittedly a server jockey professionally, so I may be woefully underestimating the complexity of this for someone without the context I've got.
posted by lhputtgrass at 3:54 PM on January 2, 2020 [1 favorite]
posted by lhputtgrass at 3:54 PM on January 2, 2020 [1 favorite]
Response by poster: If you're unsure, that means it's approximately 8000x over my head. But thanks for disclaiming :)
I'm checking out Google Cloud Service, and looks like my first task will be to grok the diff between "Standard", "Nearline" and "Coldline" service, discovering yet again that perky branding only makes my migraine worse
posted by Quisp Lover at 3:58 PM on January 2, 2020
I'm checking out Google Cloud Service, and looks like my first task will be to grok the diff between "Standard", "Nearline" and "Coldline" service, discovering yet again that perky branding only makes my migraine worse
posted by Quisp Lover at 3:58 PM on January 2, 2020
Oh god yes, everything in the cloud space (and really all of tech) is terribly named.
posted by lhputtgrass at 4:00 PM on January 2, 2020
posted by lhputtgrass at 4:00 PM on January 2, 2020
AWS SFTP is probably overkill. There are a lot of wrappers on top of AWS S3 that may make it easier to handle. Another good option for you might be Backblaze B2. Google Cloud is probably about as difficult as AWS S3, to be honest.
The way it breaks down is often around how often you want to access the data. If you need to access it all the time, you probably want "standard." For example if you were hosting images for a website. If you don't want to access it often (as your question says), something like the Google Cloud "coldline" or AWS S3 Glacier would make sense -- it is going to be slower to access most likely, but is the cheapest option for storage long term.
disclaimer: also a "server jockey" -- eek
posted by majikstreet at 4:02 PM on January 2, 2020 [1 favorite]
The way it breaks down is often around how often you want to access the data. If you need to access it all the time, you probably want "standard." For example if you were hosting images for a website. If you don't want to access it often (as your question says), something like the Google Cloud "coldline" or AWS S3 Glacier would make sense -- it is going to be slower to access most likely, but is the cheapest option for storage long term.
disclaimer: also a "server jockey" -- eek
posted by majikstreet at 4:02 PM on January 2, 2020 [1 favorite]
Response by poster: Yes, I want the coldest of cold. I want supercold. Make me wait like 17ms or whatever for my once-every-two-years download.
posted by Quisp Lover at 4:04 PM on January 2, 2020
posted by Quisp Lover at 4:04 PM on January 2, 2020
Best answer: Just how much are you needing to store, roughly?
Digital Ocean’s Spaces might work well if you’re not looking at TBs of data.
posted by jzb at 4:08 PM on January 2, 2020
Digital Ocean’s Spaces might work well if you’re not looking at TBs of data.
posted by jzb at 4:08 PM on January 2, 2020
Response by poster: AWS and GoogleCloud are SO much cheaper than the alternatives that, since this is a one-time upload and possibly-never download - i.e. no ongoing management - it might be smartest to hire someone to set it up for me. Their cost would quickly be paid back via monthly savings.
posted by Quisp Lover at 4:48 PM on January 2, 2020
posted by Quisp Lover at 4:48 PM on January 2, 2020
Best answer: For a variety of use cases, Arq Backup makes interfacing with various cloud storage pretty painless.
posted by curoi at 5:29 PM on January 2, 2020
posted by curoi at 5:29 PM on January 2, 2020
Best answer: Setting up S3 (AWS) is only mildly hair-raising, and you can connect with e.g. Transmit (and many other FTP clients) just as though it were an FTP server. I would sign up and give it a try to see if it does what you want. It’s not going anywhere, and it has the advantage that if you need help getting set up or retrieving your files, there’s lots of folks out there who will know it.
posted by pocams at 5:40 PM on January 2, 2020
posted by pocams at 5:40 PM on January 2, 2020
Best answer: Yeah, AWS is the answer. With their coldest of cold storage, "S3 Glacier Deep Archive", 500GB would cost 50¢/month (it's $0.00099 per GB), if you're willing to put up with waiting 12 hrs to retrieve the file, and to not do so more than once or twice per year (if I have an immediate need, I'd just access my on-site backup, and if my house burns down and I lose all my drives, I’m ok waiting 12 hrs to retrieve backups from, like, the 1990s).
Thanks, all. Hope it's not creepy to mark my own reply as best answer, but I couldn't have reached this without the above help.
posted by Quisp Lover at 6:20 PM on January 2, 2020 [1 favorite]
Thanks, all. Hope it's not creepy to mark my own reply as best answer, but I couldn't have reached this without the above help.
posted by Quisp Lover at 6:20 PM on January 2, 2020 [1 favorite]
I send all my backups to Backblaze B2. It's not quite as cheap as the cheapest Glacier, but also not quite as cold.
posted by jordemort at 6:38 PM on January 2, 2020 [1 favorite]
posted by jordemort at 6:38 PM on January 2, 2020 [1 favorite]
Response by poster: One downside with Amazon's Glacier Deep Archive is the standard $.09/GB egress fee. OTOH recovering your 500GB off-site backup in case of catastrophe (i.e. your backup drives fail) for $45 above/beyond minuscule monthly cost is still a great deal.
Jordemort, do I get easier user friendliness iwth Backblaze B2 than AWS with their kooky buckets and all? It seems to be 5x the monthly fee ($.005/GB/month) but 1/10th the download charge (penny per GB)
posted by Quisp Lover at 7:13 PM on January 2, 2020
Jordemort, do I get easier user friendliness iwth Backblaze B2 than AWS with their kooky buckets and all? It seems to be 5x the monthly fee ($.005/GB/month) but 1/10th the download charge (penny per GB)
posted by Quisp Lover at 7:13 PM on January 2, 2020
Response by poster: I've found the answer. BackBlaze B2 is very easy, via a friendly web interface, if your uploaded files are all under 500MB. If you're uploading larger files, you need to do lots of geeky command line things that sound easy in the instructions but that don't always yield the promised responses (do Terminal/Unix instructions EVER yield the promised responses?).
posted by Quisp Lover at 7:50 PM on January 2, 2020 [1 favorite]
posted by Quisp Lover at 7:50 PM on January 2, 2020 [1 favorite]
Best answer: I use Arq + Backblaze B2. I consider it a good mix of inexpensive and configurable. Not as cheap as Glacier, but cheaper than AWS.
posted by reeddavid at 10:38 PM on January 2, 2020 [1 favorite]
posted by reeddavid at 10:38 PM on January 2, 2020 [1 favorite]
I use AWS all day, ever day, in my professional life, am AWS certififed, and yet I use Backblaze B2 for backups of all my personal data (have for a few years now).
I do live encrypted hourly differential backups of a entire linux server to B2 (via restic), keeping 24 hourlies, 7 dailies, 5 weeklies, and 1 monthly full backup.
On top of that, I have the usual assorted "write once" type "in case of fire" encrypted document backups to B2 directly from my home NAS (a synology, if that's of incidental interest).
ALL of that costs me less than $2 a month.
Retrieval takes seconds-to-minutes (which is mostly decryption and extraction, not slow transfer).
It's a service that just works and is dirt cheap. I've yet to encounter a downside.
posted by namewithoutwords at 11:28 AM on January 3, 2020
I do live encrypted hourly differential backups of a entire linux server to B2 (via restic), keeping 24 hourlies, 7 dailies, 5 weeklies, and 1 monthly full backup.
On top of that, I have the usual assorted "write once" type "in case of fire" encrypted document backups to B2 directly from my home NAS (a synology, if that's of incidental interest).
ALL of that costs me less than $2 a month.
Retrieval takes seconds-to-minutes (which is mostly decryption and extraction, not slow transfer).
It's a service that just works and is dirt cheap. I've yet to encounter a downside.
posted by namewithoutwords at 11:28 AM on January 3, 2020
Response by poster: nwow - with $0.005/GB/month B2 storage, that means you're occupying less than 400GB for all that?
posted by Quisp Lover at 5:44 PM on January 3, 2020
posted by Quisp Lover at 5:44 PM on January 3, 2020
late to the party — for such little data, why not just store it on a physical disk? even a small raid setup is cheaper than monthly size-based payments.
posted by =d.b= at 3:59 PM on January 4, 2020
posted by =d.b= at 3:59 PM on January 4, 2020
sure, so the physical disk can be moved somewhere else. i’m thinking of the old adage “Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.” that quote applies specifically when latency isn’t a concern (which it doesn’t seem to be here). if you don’t want to deal with rising monthly payments, eventual product sunset (it always happens. look at IBM.), strange meaningless jargon, all that, then a shoebox of hard drives in a storage closet (or several storage closets) handily sidesteps the problem.
i suppose it’s worth mentioning: if you go for a backup scheme like the one you’ve specified, “active-active” is far superior to “active-passive”. that is to say, don’t just create a backup and squirrel it away until some day in the future. periodically check that you can restore an arbitrary backup. otherwise you run the risk of needing to recover a backup and finding that the data is corrupt. in my time at google i’ve seen this lesson repeatedly and painfully relearned.
posted by =d.b= at 11:59 AM on January 5, 2020
i suppose it’s worth mentioning: if you go for a backup scheme like the one you’ve specified, “active-active” is far superior to “active-passive”. that is to say, don’t just create a backup and squirrel it away until some day in the future. periodically check that you can restore an arbitrary backup. otherwise you run the risk of needing to recover a backup and finding that the data is corrupt. in my time at google i’ve seen this lesson repeatedly and painfully relearned.
posted by =d.b= at 11:59 AM on January 5, 2020
Response by poster: So I'll drive around bimonthly, ala Mike and Jesse in Breaking Bad checking their cash stashes, to my various "storage closets" (which I'll pay to rent...btw, did you know bank safety deposit boxes exist in a regulatory blind spot, making banks de facto irresponsible for contents?), pulling out hard drives (costing $100 each) to test their worthiness?
Hey, that sounds way better/cheaper/easier than paying $36/year to B2 for cloud storage! I can't imagine why that's not the standard solution!
posted by Quisp Lover at 1:31 PM on January 5, 2020
Hey, that sounds way better/cheaper/easier than paying $36/year to B2 for cloud storage! I can't imagine why that's not the standard solution!
posted by Quisp Lover at 1:31 PM on January 5, 2020
i certainly see the appeal behind paying those companies to do the work for you. i agree that these companies will do the maintenance reliably, i’m merely pointing out that these services may not be around in 3-5 years. google has a proven track record of shutting popular services down after about 2 years; amazon and microsoft are just as apt to do so if the money isn’t good enough. all cloud product prices will definitely increase in that timespan as well. to contrast, hard drives usually have 3-5 year warranties and longer lifespans if they’re not continuously in use. if this doesn’t bother you, all the power to you. i also don’t know your requirements around offsite storage, whether that means “secure” or “not in the building”. is this a home office? is it a workplace? generally if you want a simple solution, it will cost more. if you want a cheap solution, you will have to do more work yourself.
no matter what, truly long-term storage is a very difficult prospect, and no solution is perfect. keep in mind that no one will insure your data. these companies certainly provide support in the case of data loss, but they are quite careful to exclude liability for data loss from their contracts. if your data is lost (example), they will not be held liable. so a safe deposit box or even a shoebox under the bed both have the same insurance as the cloud — none.
i am aware of some services that companies offer in order to minimize the probability of data loss, but it is quite pricy — google cloud offers “data loss protection” (i can’t vouch for nor against it) for the cost of $3/GB under a terabyte, $1/GB above. that would break most people’s budgets, and even then it’s not guaranteed to work.
as for driving around checking various caches, offsite doesn’t imply offline. that said, if you work at home it’s quite different from if you work at an office. if you have a place that you could park a synology disk station, it could solve this problem. if not, a cloud provider indeed the better approach.
so i suppose another valuable question is, what is the time frame you care about? do you envision wanting to restore something from 5 years ago, or do you only care about the past year or so? if the former, i stand by my recommendation. if the latter, cheaper is better — just expect to move your data around at least once.
(as a side note, this is indeed the standard solution behind the scenes. amazon, google, microsoft, etc. all shuttle data from live disk to a stable backup medium (eg a tape) which is physically stored elsewhere. small/medium businesses sometimes handle this problem using something like IBM’s all-in-one tape archive units. as an individual, $15 can net you a 500 GB WD Blue hard drive, but it costs more to create a decent setup for those drives).
anywho, you seem content with the solution that you’ve found, so i’ll quit commenting. best of luck.
posted by =d.b= at 8:56 PM on January 5, 2020 [2 favorites]
no matter what, truly long-term storage is a very difficult prospect, and no solution is perfect. keep in mind that no one will insure your data. these companies certainly provide support in the case of data loss, but they are quite careful to exclude liability for data loss from their contracts. if your data is lost (example), they will not be held liable. so a safe deposit box or even a shoebox under the bed both have the same insurance as the cloud — none.
i am aware of some services that companies offer in order to minimize the probability of data loss, but it is quite pricy — google cloud offers “data loss protection” (i can’t vouch for nor against it) for the cost of $3/GB under a terabyte, $1/GB above. that would break most people’s budgets, and even then it’s not guaranteed to work.
as for driving around checking various caches, offsite doesn’t imply offline. that said, if you work at home it’s quite different from if you work at an office. if you have a place that you could park a synology disk station, it could solve this problem. if not, a cloud provider indeed the better approach.
so i suppose another valuable question is, what is the time frame you care about? do you envision wanting to restore something from 5 years ago, or do you only care about the past year or so? if the former, i stand by my recommendation. if the latter, cheaper is better — just expect to move your data around at least once.
(as a side note, this is indeed the standard solution behind the scenes. amazon, google, microsoft, etc. all shuttle data from live disk to a stable backup medium (eg a tape) which is physically stored elsewhere. small/medium businesses sometimes handle this problem using something like IBM’s all-in-one tape archive units. as an individual, $15 can net you a 500 GB WD Blue hard drive, but it costs more to create a decent setup for those drives).
anywho, you seem content with the solution that you’ve found, so i’ll quit commenting. best of luck.
posted by =d.b= at 8:56 PM on January 5, 2020 [2 favorites]
« Older What's your local-favorite food or drink that's... | I need some help resolving a fight my girlfriend... Newer »
This thread is closed to new comments.
posted by lhputtgrass at 3:50 PM on January 2, 2020