Ask MetaFilter questions tagged with rsync

Amazon S3 for log backups?

Anonymous — Thu, 22 Oct 2009 22:04:16 -0800

I'm looking for a step by step recipe for log rotation from a web server to Amazon S3. Caveat: I am an utter newbie to Amazon S3 and EC2.

I currently have a backup process that works well. It's a cronjob that copies server logs on a nightly basis from an external machine that I do not control to a linux VPS I do control via scp.

The cost of storage is killing me (I keep > 1 years worth of logs) so I'd like to use Amazon S3 as the destination for the logs, while keeping this as unixy as possible.

What are the steps I need to take to do this? Baby steps, please. I'm new to Amazon's S3 and EC2.

I would prefer something like a nightly job where I run:

scp user@example.com:logfile user2@amazons3domain.com:destination

or the rsync equivalent.

I do not need more than 99% reliability. Simple is best. Installing new software on the outside machine is difficult, so a standard scp or rsync method preferred.

I can install software on one of my servers, but I presume it would not be involved in the transfer as I'd take a 3x bandwidth hit (outside machine to mine, mine to s3 x 2: upload from my side, download from theirs)

Over time, I will have more than one outside machine to back up. For now, it's just one.

Rsync to Merge Disk Images?

kristi — Sun, 30 Aug 2009 23:18:10 -0800

Should I use rsync (instead of rsnapshot) to pare down Mac disk image backups? I've used SuperDuper to make several disk image backups of a boot drive.

Let's say they're called backup-090107.dmg, backup-090114.dmg, backup-090121.dmg, and so on.

I want to merge all these into a single backup directory, RETAINING incremental backups. So if file-a.txt changed on Jan 16 2009 (between backup-090114 and backup-0090121), I'd end up with a copy of how it was on 090114 and also a copy of how it was on 090121.

I tried rdiff-backup but gave up on it due to insane slowness. (It takes SuperDuper 5-6 hours to back up 60-70 GB over USB2; rdiff-backup was taking about 16 hours to back up just the 7GB applications directory. Yikes.)

I've started looking at rsnapshot, but I don't think that's what I want for this initial merge ... although it might be what I want for any subsequent backups, once I've got the disk images merged.

So:

* am I correct in thinking that rsync is best for merging these dmgs?

* would the following commands do what I want to do - so I end up with a single copy of any file that hasn't changed, plus incremental backups of each file that HAS changed?

rsync -a --delete --numeric-ids --relative --delete-excluded --xattrs --exclude=/this/ --exclude=/that/ \
/Volumes/backup-090107 /Volumes/BackupDrive/big-old-merge-directory/backup-090107/

rsync -a --delete --numeric-ids --relative --delete-excluded --xattrs --exclude=/this/ --exclude=/that/ \
/Volumes/backup-090114 /Volumes/BackupDrive/big-old-merge-directory/backup-090114/

* should I switch to rsnapshot after merging, or continue using rsync? If I continue with rsync, would I use something like

rsync -a --delete --numeric-ids --relative --delete-excluded --xattrs --exclude=/this/ --exclude=/that/ \
/MyHardDrive /Volumes/BackupDrive/big-old-merge-directory/backup-todaysdate/

Extra bonus points for lists of directories you like to exclude, or pointers to pages that list appropriate OS X (10.4) directories to exclude. (I DO want to back up even big directories; I don't need to back up the unnecessary or dangerous - for example, I know most folks exclude .Spotlight-V100.)

Once I've merged the backups (and verified them, of course), I'll be deleting the disk images to free up space for future backups. (And I'm doing full, bootable backups to a different drive; this is just my attempt to get good incremental backups.)

Many, many thanks!

How do I stop my external hard drives from changing disk IDs when I reboot on OS X?

triolus — Fri, 14 Aug 2009 20:08:06 -0800

How do I stop my external hard drives from changing disk IDs when I reboot on OS X? I have a simple rsync script that I just made for my OS X machine to backup data on my two main drives to two drives in an external enclosure. I wanted the script to mount and unmount the drives so that they wouldn't clutter my desktop and to keep them "hidden".

The problem is that the disk ID under /dev changes everytime I reboot. Sometimes the drive mounted at /Volumes/backup1 will be at /dev/disk3s3 and sometimes it will be at /dev/disk4s3. Other times it will be at /dev/disk2s3.

Is there anyway to keep the disk ID static, or to just mount by the directory name? I'm running 10.5.8, with the latest updates. My script uses the `diskutil`command to mount and unmount the drives. Oh, and AFAIK, Time Machine only works with one drive as a backup.

Back me up, Scotty!

sprocket87 — Wed, 18 Jun 2008 16:02:02 -0800

Recommend a good tool to remotely backup/synchronize large folders to my home NAS. Here's what I have:

- Laptop with 100+ GB music library; misc documents, photos, and other data that needs backed up
- Netgear ReadyNAS Duo 750GB at home (supports pretty much everything - rsync, (S)FTP, WebDAV, etc)

Here's what I want to do:

For my MP3 library (100GB, 20k files, 4k folders) - mirror or synchronize any changes from my laptop (remote at work) to my home NAS. I've tried a TON of apps already with varying degrees of success. The major problem seems to be that to do a mirror, the apps have to perform a full "audit" scan of the destination (NAS) share to see what needs updated/copied. Problem is, with 20k files, that takes f-o-r-e-v-e-r, at least using FTP or WebDAV.

Ideally, I'd like a real-time backup app that monitors some selected folders (like My Music) and kicks off an automated upload for any changed files. This (a) makes it fully automatic and (b) [in theory] faster - doesn't have to synch each time, just upload based off a good known synch. Problem is, I've yet to find the right tool to accomplish all these things correctly.

Note: I'd be happy as a clam to use rsync, which the ReadyNAS supports, but I'm sending from a Windows client and my options are limited. I've been racking my brain trying to get DeltaCopy to work, but it keeps timing out when attempting to connect to the NAS. I have port forwarding, etc, working fine on my router, so I can't really figure out why rsync is failing.

Note 2: The ReadyNAS does have optional support for SSH and Telnet, I believe, but it involves some trickery to get working and I think it might void my warranty...

Backubuntu

tmcw — Sun, 18 May 2008 15:32:24 -0800

What's the best backup solution for (Ubuntu) Linux? I need a backup as complete as possible of a home directory of ~10+ GB, nightly, etc. I have two large external USB 2.0 drives for the purpose... I know some of the options (which I'll list out), but I'm wondering which is the community favorite / most popular / most reliable / most likely to be maintained.

What I've considered: flyback, rsync, rdiff, rsnapshot, timevault, unison...

Thanks! I've gone too long without good backups, and my home directory is becoming more and more valuable. Oh, and it absolutely must backup hidden directories, although I would guess that's standard (a lot of my data is SVN repos). Something that could back up various other files (my apache configs, site-applications, etc files are fairly vital). Maybe a separate system that would do basically a drive image on a less frequent basis?

Thanks!

How do I restore my old Mac's rsync backup to my new Mac?

autojack — Fri, 09 May 2008 16:20:19 -0800

What's the easiest way to restore my OS X 10.4 backup of my iBook laptop onto my shiny new OS X 10.5 Macbook? The catch: I made the backup using rsync, to a Linux file server. A few weeks ago, my three year old iBook G4 started running very slowly and making an ominous clicking sound. A lifetime of experience, plus the knowledge that my warranty had just expired, let me to the obvious conclusion: the hard drive was dying. I was planning to buy a new Macbook as soon as I received some money I was owed, so I decided to capture a backup, power the old laptop down, and use my Windows desktop until I got the new laptop.

Having no other readily-accessible option, I used rsync to back up the entire contents of my home directory to my Linux file server. I used some kind of Mac-specific option in rsync to save the filesystem metadata, I think. I haven't been able to find the site with the instructions I used to do this, but I haven't looked very hard yet.

I need to restore this data now, and I want to do it without breaking things on my new system, since it's got a newer OS. The main things I need are all my documents (easy to recover, really), all my Stickies notes, and all my Mail.app mail and metadata.

I suppose it's possible that I could use Apple's data migration app to cleanly copy this stuff right off the old iBook, if its hard drive will cooperate for a little bit longer. I'm doubtful that it will, though, and I'd really hate to have a half-done restore.

I don't even know where to start. I'm a Linux guy so a lot of that translates to the Mac, but there always seem to be a few Mac-specific pitfalls or footnotes that the Apple fanatics point out which save my ass in these situations. I think that's basically what I'm asking for.

Going forward, I think I'll be using Time Machine for this purpose.

What features should I add to my basic backup solution?

philomathoholic — Sat, 16 Feb 2008 16:02:29 -0800

I've rolled my own backup solution, and I want to know if I could/should be doing anything more. What I'm basically doing now is copying over everything every 10 days, and deleting anything that's over 4 backups old. Basically, here's some of the files in my directory for the laptop:

080110.bookmarks.html
080110.cyg-home.tar
080110.desktop.tar
080110.music/
080110.myjunk.tar
080110.personal.tc
080119.bookmarks.html
080119.cyg-home.tar
080119.desktop.tar
080119.music/

The date of the backup is part of the name. The music directories are about 15G each, the others are less than 1MB. I'm using Bash shell scripts, ssh and scp. (I couldn't get rsync to be much faster than scp. I was using samba, but that just kept spinning-up my drive.) This is all done on my local wireless network.

I've already got the best feature: automatic. Should I be doing monthly/weekly/daily backups? What would that entail, and how do I do that? I've got the frequency set at 10 days, but more frequent would be nice. My problem is that frequent backups create more new files, so then I have to get rid of the not-so-old files (size is limited).

Please give me your recommendations.

not every snowflake is special

nomisxid — Mon, 11 Feb 2008 11:23:52 -0800

Location & name independent rsync? I have machine A, with lots and lots of empty space.

I have machines B, C, D, E, F, ... with lots of files, some of which are duplicated between machines, but often not in the same file structure or naming convention, ala
Machine B:
/home/userX/media/Wedding01.avi
Machine C:
/var/local/storage/movies/sis-wedding.avi

Ideally, I'd like an agent I can put on Machines B-F, point it at specific folder(s), and give it an ftp account on machine A, and let it auto-magically figure out the minimum set of files that need to be backed-up, to have a complete set of the files across all the original machines. It does NOT have to remember where the files were on each machine.

Machines B-F are currently windows boxen that expose their data via windows shares, so I can point a unix client at them, if needed.

Machine A can be a unix, windows, or mac os X box, whatever is needed.

What's the best online backup for a lawyer using a Mac?

katemonster — Thu, 02 Aug 2007 13:28:19 -0800

I've been asked to find the best online backup solution for a small law firm that uses Macs and PCs. Security is obviously an issue. Any personal experience with Corevault, rsync.net, or Amerivault is particularly welcome. (I have checked the previous questions on AskMe and I've googled a fair bit, but most of the questions are pretty old and/or non-law specific, and I was hoping for some personal experience from trusty mefites.)

The main goals: online, preferably automated backup, once a day; extremely secure (encrypted by user before sending); compatible with both Mac and PC, relatively inexpensive (currently looking at about 3-5 gb); and easy for the non-technical to use.
I've researched CoreVault (endorsed by the Oklahoma Bar), Amerivault (endorsed by NY Bar), and rsync.net (endorsed by lots of people, but no bar associations I can find). Unfortunately my tech knowledge, especially about Macs, fails me when it comes to the ins and outs of online data storage.
I'm particularly concerned about choosing a company that has a long-running track record, and not getting suckered by any fly-by-nights. I've heard gushing things about Mozy, but have also heard that they're slow. I'm also concerned about the security/reliability of a free site when it comes to something this critical.

LAMP scaling and NFS. Good Idea or Not?

Freen — Fri, 30 Mar 2007 14:41:37 -0800

So, I've got myself a web application. LAMP. I need to scale it up. Can I run the php application across multiple machines loaded off a mounted NFS share on another machine? I'm running gig-e with jumbo frames on a private LAN. Our application is fairly high load, hence the need to scale it up. We can't afford a real fiber SAN, so, can we replicate the benefits of having all of our application scripts live in one place, and each of the web-servers mount the nfs share and serve the application off of the nfs share?

We've thought about other possibilities, such as rsync, etc. But we're concerned about race conditions. Particularly two servers communicating with clients and with the DB using different codebases.

other datapoints:
We use a bytecode cache, called eaccelerator. those caches would be on the local webserver, not on the NFS share.

We use memcache for stuff like sessions across machines.

Would there be significant network throughput setting up our application this way? What are the downsides? Would it outweigh the benefits of having one codebase for our application?

What can a Linux fan use for consistent note taking and archival?

a007r — Fri, 29 Dec 2006 16:35:46 -0800

I am studying in human science (psychology), and I'm starting university next month. I have been using a laptop running Linux to take notes and write assignments during the past two years or so, but I feel my current technique for note taking is unsatisfactory. My current method is this: fire up OpenOffice Writer with a blank document, start typing in bullet-list form, with a Heading1 title at the top of the document, and save the document as "yyyy-mm-dd-topic.odt" in a folder such as school/the_subject/*. Whenever a semester is over, I compress the files into a tar.gz archive and put it in an archival folder. This prevents me from having the files indexed by Tracker.

- must have: formatting controlled by a central/GLOBAL STYLESHEET, unicode, open source, runs on Linux
- nice to have: OpenDocument, drawing support, autosave, wysiwyg, ability to zoom text to disproportionate sizes (those 1280x1024 widescreen laptops strain the eyes easily)
- don't care about: spellcheck
- don't want: proprietary stuff, obscure file format, latex, a database

I have been scratching my head over this for a little while, so far I see these options: OpenOffice, plain text with Gedit or whatever, Abiword, coding XHTML by hand on the fly, using Tomboy (but it is slow), using a wiki such as PmWiki running on a local server on this laptop, none of which particularily seem to fill my needs completely. Please let me know of any other possibilities I have overlooked!

Latex (even Wyneken) are beyond my understanding, and I feel they are overkill for note taking (maybe when I end up writing a huge thesis or something...)

I want everything I write to be accessible 20 years later. Actually, I have only two big criticisms against my current OpenOffice technique: it forces me to load openoffice (which does "feel" heavier than most text processors), and the contents' style is per-document, not system-wide (like in a CSSed page collection or in Latex).
A criticism I have against PmWiki (the only wiki I tested, but it uses no database, and that is nice) is that filenames it creates don't really support non-english characters properly, and I have to concede that is a limitation of the web itself; accents mess up nicely in URLs, and pmWiki doesn't like to have them on the filesystem either; actually, the problem may lie in the fact that it feels like I have "less control" over the filesystem since it uses all those WikiWordFileNamesThings. Furthermore, editing in a wiki is not exactly WYSIWYG. You have a very easy syntax, and it has the advantage to use CSS over all your documents at once, but you cannot "visually" distinguish a header paragraph from a regular paragraph, not as easily as you would in a WYSIWYG application (you have to save to do that). Printing is also a bit tricky, and a wiki is, to a certain extent, a bit of annoying maintenance to deal with (security upgrades for example).

I like solutions that "respect" my filesystem instead of forcing me into a set style of folders, or worse, a database; I backup and synchronize notes between my laptop and my desktop over the network using Rsync scripts I wrote.

I am "more inclined" towards certain file formats so far: OpenDocument or xhtml, but feel free to suggest something else. I mean I don't quite like RTF, but if it's guaranteed to work everywhere anytime, it could be useful; or even taking notes in plain text in front of a laptop without X.org would be possible (but pretty friggin radical! :).

Actually, I'm realizing that I'm typing this in Gedit, a plain text / code editor, but that's only because I do not trust the Web (even if my browser never crashed) and don't want to lose a long post.

Thoughts? Experiences? Recommendations? Questions? :) I realize there *is* note-taking software out there such as Gournal or Jarnal if I remember correctly, but are those really the end all solution? A computer certainly does not behave like a physical paper notebook, and I'm especially interested in the "way" (or maybe the medium) the notes are taken in (if you have special techniques, I'm also interested), not really specific applications. I know this is a weird question, I will try clarifying as soon as possible if you have needs for clarification.

What's the best way to backup/restore (on a Mac)?

Colloquial Collision — Mon, 11 Dec 2006 19:55:58 -0800

What's the best way to backup/restore (on a Mac)? I'm asking this question for a friend —

“I use Subversion (svnX) to backup my Documents (which allows me to check files into and out of the repository from OSX, Windows, and Linux, it works really well). But for the stuff I don't need versioned I can't figure out a clean way to back it up...

I was using Rsync to do daily backups of my home folder /Users/ben, and one time I did have the system crash. When I brought it back up my iCal, iTunes, Address Book, and iPhoto lost all their data. So I used rsync to restore /Users/ben/Library/Application Support/iCal, / Users/ben/Library/iPhoto Library, /Users/ben/Library/Application Support/AddressBook and /Users/ben/Music/iTunes/ ...here are the results:

iTunes, everthing works but it has to rebuild the database and redownload all the artwork everytime it is launched.

Address Book: restored perfectly.

iCal: data restored, but preferences are stuck. If I set any preferences it forgets them if I close and reopen iCal, but it remembers my new appointments and todos okay.

iPhoto: I can't get it to launch, it's just stuck at loading Photos. All of the Photos and data files are there and seem to be okay.

So, I must need to restore some other files I'm not aware of, or rsync just isn't a good way to do backups.”

Thanks for the help, MetaFilter!

rsync GUI

thebwit — Tue, 19 Sep 2006 06:55:37 -0800

Any recomendations for a GUI for rsync to run on a linux server using PHP? I need to be able to install it via FTP as some of the servers that I have been asked to put this on do not allow shell.

Making a stand-alone rsync appliance.

Wild_Eep — Tue, 11 Oct 2005 11:54:56 -0800

Is there a distribution of a linux-like OS which turns hardware into an easy-to-configure rsync appliance? I have a client (web dev agency) whose president wants 'on site backups of all of the websites we host'. They had issues with less-than-clueful providers, I guess. They're not interested in switching their hosting provider.

She essentially wants to be able to point to a box in her office and say 'our backups of all of the sites we're responsible for live here'. I don't work at this office, so I'd like to be able to control this 'remote backup appliance' via a web-admin or VNC or some such. If I have no other choice but to be on-site, that's okay, but not optimal.

I have a Shuttle-size PC available to me for this project (p3, 256MB, 80GB mirrored HDDs.) My original plan was to install a flavor of linux, then make some shell scripts to do simple wget requests on a regular (perhaps daily) basis. Some research revealed that the aggregate file size to transfer was ~1.5 TB range. This office has only business-class cable modem service (5Mb down).

Now I'm thinking about implementing rsync, which I need to learn more about. The client claims to have root on their server at the ISP, so if I need to install sw to support rsync, that shouldn't be a problem.

My previous experience is with Mac OS X Server, where turning on services is as easy as clicking a button. I have enough unix experience to get around a shell, but not enough to know where all of the config files, etc. live for various services.

I need to be able to ssh into this box and set up cron scripts to fire rsync events, but if there were a web interface for this kind of thing, all the better.

Basically, I don't want to go through the effort of manually writing the scripts if there's a smarter way to achieve my goal.