backitup yourself
February 7, 2010 1:19 PM Subscribe
creating my own "cloud" backup?
(i realize there are some older questions on the topic, but software changes quickly).
i'm looking for a dropbox-style syncing option for online usage, using my own webserver (that is, NOT on my computer, but a service i pay for remotely). i currently have webDAV set up (for use with zotero, which is a major impetus behind this whole question) on said server and i'd like to start syncing dissertation manuscripts, etc. i know i could just drop files over FTP, but i'd rather something a bit more.. elegant, automated. i already back up daily on an external HD but as we all know well, cloud cloud cloud! i'm on windows7, and use XP and macs at other location so would want something that i could hopefully use on at least the windows platforms, if not all three. thanks!
(i realize there are some older questions on the topic, but software changes quickly).
i'm looking for a dropbox-style syncing option for online usage, using my own webserver (that is, NOT on my computer, but a service i pay for remotely). i currently have webDAV set up (for use with zotero, which is a major impetus behind this whole question) on said server and i'd like to start syncing dissertation manuscripts, etc. i know i could just drop files over FTP, but i'd rather something a bit more.. elegant, automated. i already back up daily on an external HD but as we all know well, cloud cloud cloud! i'm on windows7, and use XP and macs at other location so would want something that i could hopefully use on at least the windows platforms, if not all three. thanks!
I use backupninja which leverages some Python scripts and duplicity to make gpg-encrypted backups over ssh. I don't know if it works with Windows. You might want to look at Duplicity.
posted by Geckwoistmeinauto at 1:24 PM on February 7, 2010
posted by Geckwoistmeinauto at 1:24 PM on February 7, 2010
Response by poster: i should stress that i really want to use my own server, for security reasons, for economic reasons, etc. things like pogoplug, dropbox, etc are not relevant for this question: not because i don't trust them now, but just because philosophically i would like to have leverage over my own stuff.
posted by yonation at 1:35 PM on February 7, 2010
posted by yonation at 1:35 PM on February 7, 2010
If it's for something you're working on, you might want to look into something like Subversion, which is a version control system. When you make changes, you submit them and you can easily go back and compare versions from the past, as well as the latest version. There are SVN clients for windows, mac, etc.
It would be especially nice if you're working with plain text or XML files, rather then binary files like word documents.
Setting it up to work over the web requires (rather then on the file system) a bit of config-file wrangling, in my experience (You have to configure an Apache module to integrate with Web DAV). I have mine setup to used HTTPS so transfers are encrypted. I have a software project that I keep in an subversion repository and it works great. It was originally designed for software, so it's well integrated with software development tools but it should work fine with any type of file.
(Also, having the files on a single web server won't really make it 'cloud'. If you want to be fully buzzword compliant, you should look into something like using Amazon S3, but I don't know any syncing tools that use it)
posted by delmoi at 1:38 PM on February 7, 2010
It would be especially nice if you're working with plain text or XML files, rather then binary files like word documents.
Setting it up to work over the web requires (rather then on the file system) a bit of config-file wrangling, in my experience (You have to configure an Apache module to integrate with Web DAV). I have mine setup to used HTTPS so transfers are encrypted. I have a software project that I keep in an subversion repository and it works great. It was originally designed for software, so it's well integrated with software development tools but it should work fine with any type of file.
(Also, having the files on a single web server won't really make it 'cloud'. If you want to be fully buzzword compliant, you should look into something like using Amazon S3, but I don't know any syncing tools that use it)
posted by delmoi at 1:38 PM on February 7, 2010
Response by poster: delmoi: touche on the cloud. my mistake on that. to answer your question, i AM looking for something that will back up binary files as well.
posted by yonation at 1:39 PM on February 7, 2010
posted by yonation at 1:39 PM on February 7, 2010
Best answer: I think I'm in a similar boat. I have an account with rsync.net and have been very happy with it. I use a tool called Unison to do the syncing and I run it on my linux workstation and my windows laptop (via cygwin). I use the remote filesystem is the "master" copy and any time I want to work on anything, I sync whichever local machine I'm using with the master (to get the latest version), do all my stuff, and then sync the changes back up at the end of the day. Unison will take care of everything unless there is a conflict (the same file altered twice from different locations, which can happen if you're lazy about syncing) in which case you simply tell it which one to pick. By the way I've found an html-based notebook tool called Keepnote that works really well with this setup.
The only downside I've encountered is that syncing starts to take a really long time as the number of files gets really large, simply because it has to scan through all of them to check for changes. With the large number of zotero references this was becoming untenable. Fortunately zotero has syncing built-in with the new beta version, so I'm happy to let zotero handle the references and point Unison only at my notes.
Note that this isn't really a "cloud backup" per se; rather it migrates your active workspace to the cloud (or to a single server, if you don't like the word cloud). If you did several stupid things in a row you could obliterate the whole thing and upon syncing to local machines obliterate local copies as well. So don't stop with the external HD backups.
posted by PercussivePaul at 1:49 PM on February 7, 2010
The only downside I've encountered is that syncing starts to take a really long time as the number of files gets really large, simply because it has to scan through all of them to check for changes. With the large number of zotero references this was becoming untenable. Fortunately zotero has syncing built-in with the new beta version, so I'm happy to let zotero handle the references and point Unison only at my notes.
Note that this isn't really a "cloud backup" per se; rather it migrates your active workspace to the cloud (or to a single server, if you don't like the word cloud). If you did several stupid things in a row you could obliterate the whole thing and upon syncing to local machines obliterate local copies as well. So don't stop with the external HD backups.
posted by PercussivePaul at 1:49 PM on February 7, 2010
this seems to be an rsync implementation for windows (without the need for cygwin); it needs the server to be running a server process, or an unix rsync daemon, though, which is likely more than you're allowed to do on a hosting server.
I'm looking for a similar tool as well; my main perplexity would be that most, if not all, hosting plans forbid you to use the server as a file repository (unless you buy a dedicated server, which is a least as expensive as a no-limits contract with a 'proper' cloud storage service such as Mozy -- scale economy and all that, I guess).
posted by _dario at 1:58 PM on February 7, 2010
I'm looking for a similar tool as well; my main perplexity would be that most, if not all, hosting plans forbid you to use the server as a file repository (unless you buy a dedicated server, which is a least as expensive as a no-limits contract with a 'proper' cloud storage service such as Mozy -- scale economy and all that, I guess).
posted by _dario at 1:58 PM on February 7, 2010
yonation: subversion will work fine with binary files, it just won't let you compare differences the way you can with text.
posted by delmoi at 2:10 PM on February 7, 2010
posted by delmoi at 2:10 PM on February 7, 2010
Your own webserver is not the cloud and in comparison to the cloud it's not distributed, not necessarily backed up and thus not safe.
If you want a backup and versioning go for Subversion of your own server. Works fine with a dissertation.
posted by oxit at 2:18 PM on February 7, 2010
If you want a backup and versioning go for Subversion of your own server. Works fine with a dissertation.
posted by oxit at 2:18 PM on February 7, 2010
Ok, you want to back up your computer to a remote location. I did a brief analysis of the options within Ubuntu / Debian. Most are based on rsync or tar. The rsync ones use some riff on rdiff-backup, which has a windows build. As a basic UNIX tool, it also runs on OSX. The author seems to suggest BackupPC if you have multiple computers.
An important note about backup tools. Restore is just as important as backup. If you encrypt the backups for privacy and use custom storage formats and software, you better have a local backup of all the restore related tools.
posted by pwnguin at 2:39 PM on February 7, 2010 [2 favorites]
An important note about backup tools. Restore is just as important as backup. If you encrypt the backups for privacy and use custom storage formats and software, you better have a local backup of all the restore related tools.
posted by pwnguin at 2:39 PM on February 7, 2010 [2 favorites]
I love SVN and use it to store all my files. Here's a guide on how to set it up. It integrates, really, really well with Windows.
posted by geoff. at 2:59 PM on February 7, 2010
posted by geoff. at 2:59 PM on February 7, 2010
Using your own server doesn't make sense from an economic or security perspective.
Use JungleDisk and encrypt your data. Much cheaper and perfectly secure, and JungleDisk supports state of the art encryption algorithms for which you will be the only person who has the key.
Although you don't say how much data you have, or whether you want to keep revision history. If it's small and you want revision history, use you own server with a revision system like subversion / git / etc.
There's also Unison, which is comparable to rsync for your purposes, but with a nice interface.
posted by qxntpqbbbqxl at 3:37 PM on February 7, 2010
Use JungleDisk and encrypt your data. Much cheaper and perfectly secure, and JungleDisk supports state of the art encryption algorithms for which you will be the only person who has the key.
Although you don't say how much data you have, or whether you want to keep revision history. If it's small and you want revision history, use you own server with a revision system like subversion / git / etc.
There's also Unison, which is comparable to rsync for your purposes, but with a nice interface.
posted by qxntpqbbbqxl at 3:37 PM on February 7, 2010
my main perplexity would be that most, if not all, hosting plans forbid you to use the server as a file repository
Take a look at https://www.nearlyfreespeech.net/ I don't think they have a problem with storage. But as qxntpqbbbqxl suggests, as cheap as nearlyfree is there may be better options.
posted by sammyo at 3:49 PM on February 7, 2010
Take a look at https://www.nearlyfreespeech.net/ I don't think they have a problem with storage. But as qxntpqbbbqxl suggests, as cheap as nearlyfree is there may be better options.
posted by sammyo at 3:49 PM on February 7, 2010
... and I'll elaborate on cloud storage a bit.
No virtual hosting provider I'm aware of even comes close to competing with cloud storage for cost.
JungleDisk uses cloud storage (Amazon S3 or Rackspace, your choice) for storage, and as a result it's one or two orders of magnitude less expensive per gigabyte / month than a virtual hosting server will be. If you're only storing a few gigabytes, this doesn't matter, but if you have tens or hundreds a virtual server will quickly become very expensive.
There are a couple of free tools that use S3 for backup, also: Cumulus and http://search.cpan.org/~bradfitz/Brackup/brackup'>Brackup. If you're comfortable with configuring and running command line programs, I'd recommend Cumulus. (I think it also need to be compiled, but Brackup is in Perl). These can also do encryption.
posted by qxntpqbbbqxl at 4:49 PM on February 7, 2010
No virtual hosting provider I'm aware of even comes close to competing with cloud storage for cost.
JungleDisk uses cloud storage (Amazon S3 or Rackspace, your choice) for storage, and as a result it's one or two orders of magnitude less expensive per gigabyte / month than a virtual hosting server will be. If you're only storing a few gigabytes, this doesn't matter, but if you have tens or hundreds a virtual server will quickly become very expensive.
There are a couple of free tools that use S3 for backup, also: Cumulus and http://search.cpan.org/~bradfitz/Brackup/brackup'>Brackup. If you're comfortable with configuring and running command line programs, I'd recommend Cumulus. (I think it also need to be compiled, but Brackup is in Perl). These can also do encryption.
posted by qxntpqbbbqxl at 4:49 PM on February 7, 2010
Response by poster: Wow, lots of great responses! Thanks for the info guys... I'm in the process of contacting the rsync guys/gals to see how they work out; might be better in the long-term than using my own server...
posted by yonation at 8:01 PM on February 7, 2010
posted by yonation at 8:01 PM on February 7, 2010
Response by poster: A quick piggybank: those who use S3 (and the clients, like jungledisk, that connect), and those who use rsync.net, did you choose one over the other? is one better, safer, less corporatistic, etc?
posted by yonation at 8:15 PM on February 7, 2010
posted by yonation at 8:15 PM on February 7, 2010
JungleDisk uses cloud storage (Amazon S3 or Rackspace, your choice) for storage, and as a result it's one or two orders of magnitude less expensive per gigabyte / month than a virtual hosting server will be.
That's obviously not the case if you area already running a server for some reason, in which case the marginal cost of a few gigs here or there is $0.
posted by delmoi at 9:54 PM on February 7, 2010
That's obviously not the case if you area already running a server for some reason, in which case the marginal cost of a few gigs here or there is $0.
posted by delmoi at 9:54 PM on February 7, 2010
yonation: I have S3, which I use because you have to use it for some other amazon services, like EC2, but I find it a convenient way to share large files with people, easier then uploading them to my server. I use S3Fox, which is really easy to use and basically it's like 10¢ a month for the non-EC2 related files.
posted by delmoi at 9:57 PM on February 7, 2010
posted by delmoi at 9:57 PM on February 7, 2010
Zotero already has cloud based backup storage that uses S3 as it's backend.
I also want to push back a bit against the subversion love. Once you've used any of the version control systems that were designed from the ground up to be distributed you probably won't be satisfied with subversion.
posted by rdr at 1:02 AM on February 8, 2010
I also want to push back a bit against the subversion love. Once you've used any of the version control systems that were designed from the ground up to be distributed you probably won't be satisfied with subversion.
posted by rdr at 1:02 AM on February 8, 2010
I'm not sure if mozy.com meets your requirements. It is a pure backup service, and keeps several versions of your backed up files.
Files can be downloaded from the web (web-restore) but it is not instant, so does not act as a shared drive... for which there are many other services...
posted by nielm at 3:40 AM on February 8, 2010
Files can be downloaded from the web (web-restore) but it is not instant, so does not act as a shared drive... for which there are many other services...
posted by nielm at 3:40 AM on February 8, 2010
I think Dropbox is a splendid solution. I use it to share my school documents between my computers and save work that I do on other machines. Publicly sharing multiple files with non-dropbox users is a little bothersome (you can share folders with other dropbox users), but there's probably a script written to aggregate all of the individual file urls to an html document or something like that. You can get up to 5 gigs for free, through inviting other people to the service, or there is an option to pay for more space.
posted by joydivasian at 7:34 AM on February 8, 2010
posted by joydivasian at 7:34 AM on February 8, 2010
I want to second that I've been really pleased with the support from rsync.net. Everything I've ever needed from them has been delivered fast, friendly, and no bullshit. They've been around for years and I get a sense they aren't going anywhere. One of the guys has a blog here which should give you a sense of their philosophy. They have student discounts too, if you ask.
posted by PercussivePaul at 11:45 AM on February 9, 2010
posted by PercussivePaul at 11:45 AM on February 9, 2010
This thread is closed to new comments.
posted by PenDevil at 1:21 PM on February 7, 2010