best way to clone my hard drive to multiple external drives at once

April 14, 2014 2:12 PM

Hello all. Whats the best way (fast, cheap?) to clone my internal hard drive to as many external usb hard drives as possible? (up to 5 at a time, at the same time?). Can this be done in the same amount of time as it takes to clone it to one external hdd? Or would the internal bus choke on the data and take forever? Is there even software that does this? Most of the usual suspects (ghost, acronis, etc) seem designed only to clone 1:1. The default solution seems to be buying a hardware standalone hdd cloner for hundreds of bucks. Thanks for any thoughts/ideas!
posted by jak68 to Technology (11 answers total)

What is your end goal here, and what platform are you on? If you're just making data backups or something you could potentially do this with rsync for free (rsync is a Unix utility but you can get it for other platforms too), though that's going to be pretty slow if you've got a lot of files and/or are using USB 2. If you're after a bit-for-bit clone (such as a full, bootable backup of a drive or system deployment), you could maybe look at CloneZilla. That link goes directly to instructions for one source image to multiple destinations, and it does a fair amount of other stuff too (it's essentially an open-source Ghost clone). That said, if you're doing a system deployment type thing, Ghost might actually be the easiest option.. it'll broadcast the image over the network to any number of computers, though you of course have to have actual computers and not just drives in USB cases. There may be options depending on what you're using - my typical recommendation for stuff like this is Carbon Copy Cloner but that does you no good if you're not on a Mac.
posted by mrg at 2:31 PM on April 14, 2014

Can this be done in the same amount of time as it takes to clone it to one external hdd? Or would the internal bus choke on the data and take forever?

Unless these are USB 3 external drives, plugged into USB 3 ports... then i bet you could get quite a few going without bogging down the internal drive much. The only thing that will hurt your read rate is tons of seeks having to occur to grab data the other drive already got.

I'm pretty sure crashplan supports just adding more than one drive in that destinations screen, and will then grind through making identical copies to all of them. I don't think this makes 1:1 or bootable clones, just file backups.

You're right though, every other solution i thought of only wanted to write to one disk at a time. Most of the stuff that will do what you want, like clonezilla, will require you to power down your system and boot to a liveCD of some sort. Clonezilla will work too, but it's byzantine and when it throws an error it can kill a couple hours very quickly with troubleshooting. Things like the destination drive needing to have the exact same partition layout, with the partitions at least the same size even if the source drive contains way less data in its slightly larger than destination partitions. It feels like some early 90s DOS app. Very fiddly.

Oh, and clonezilla batches things. It's going to write one drive at a time, from everything i've ever seen it do. I honestly can't think of anything besides some sort of external JBOD/raid system that's going to write all the drives at once. This thread talks about it with acronis, but suggests something i know for a fact will not work, which is batching two backup tasks to run at the same time. One will bomb the other out because it won't be able to "grab" the files. A couple google dives found a lot of people suggesting things with various apps that won't work because they didn't know any better, or more often just saying it won't work.

So all that said, if someone tasked me with doing this RIGHT NOW, here's what i'd do:

1. clonezilla clone of source drive to each usb drive, one at a time, yes on copy MBR/gpt boot sector/etc. Jackhammer through any dumb partition errors or copy fails here.
2. this on windows, or some rsync type of tool on OSX or linux. If you need help setting up rsync just google around, there's a bajillion guides, and it supports syncing to more than one destination.

Doing a 100% clone every time will be too annoying to figure out. Creating a baseline and syncing is the way to go, imo.
posted by emptythought at 3:09 PM on April 14, 2014

Putting aside the question of what tool to use, yes, there are likely to be multiple bottlenecks if you try to clone your HDD to multiple HDDs over USB at the same time. There is the potential for a bottleneck if reads from your hdd aren't coordinated, potential bottlenecks if multiple devices are on the same USB bus, or if a USB port shares a channel with another active drive. Plus, USB often imposes significant CPU load.

The only way around the latter issue is to check device manager (assuming this is a windows machine) and see how the various USB ports on your machine are connected, and then connect the drives up in a manner that minimizes contention.

There may be tools that get around the former issue, but the way I'd probably try to get around it is to do this under Linux, use a 'dd' process to read the source partition and pipe it into 'tee', and then have tee redirect its output to multiple dd's to write to the different drives. I imagine though that that could be rather fragile, and might lead to interesting results if the write performance of the output drives was dramatically different.

Also, you refer to this as "your internal drive," does that mean you are going to be cloning the volume your operating system is running off of?
posted by Good Brain at 3:14 PM on April 14, 2014

Rsync and/or tools like SuperDuper! are the obvious answers here, but:

If you try to clone your disk multiple times at once, the heads will be continuously seeking back and forth to comply with the overlapping data fetch requests. I can't imagine anything worse for the health of your hard drive.

Make sure you have good backups before you try this.

(In fact, if you have good backups, clone the backups instead.)
posted by RedOrGreen at 3:21 PM on April 14, 2014

I doubt very seriously that duplicating your hard drive multiple times simultaneously will be in any way harmful to your master drive. There are far worse things to do for the health of your hard drive (e.g., dropping it off a building) than reading a lot of stuff at once.
posted by waldo at 6:54 PM on April 14, 2014

Whats the best way (fast, cheap?) to clone my internal hard drive to as many external usb hard drives as possible? (up to 5 at a time, at the same time?)

The hardware bottleneck here will be the USB root hubs and/or the USB to SATA converters, at least if you're using USB2 external drives. I've never seen a USB2 drive transfer data faster than about 25Mbytes/sec; USB3 drives seem to top out at around 60. A decent internal drive should be good for at least 100Mbytes/sec, so provided the cloning software you're using is capable of not repeatedly reading the internal drive for each write to an external, it's unlikely to be your bottleneck.

When USB-connected devices limit your data speed, they do that for all the devices connected to a given root hub. So for best speed you'd want multiple PCIE USB cards installed in your computer, with a single external USB drive attached to each one.

On the software front, you should be able to get the behaviour you want by using Gnu ddrescue in Linux, where the clone target is a five-way RAID1 (mirrored) array without superblocks, set up with mdadm. If you're using anything in the Debian family (Debian itself, Ubuntu, Mint, various others) you'll want the mdadm and gddrescue packages.

Something like

mdadm --build /dev/md0 --raid-devices=5 --level=raid1 --assume-clean /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
ddrescue --force --cluster-size=2048 /dev/sda /dev/md0 /tmp/sda.log
mdadm --stop /dev/md0

should be close to what you need. Be careful with device names; always worth checking the output of

ls -l /dev/disk/by-id

to make sure that the assigned /dev/sdX names refer to the drives you think they do.
posted by flabdablet at 4:45 AM on April 15, 2014 [1 favorite]

If you'd prefer not to use ddrescue and mdadm (which I personally would do, being rather fond of ddrescue's progress reporting and bad-block handling and its ability to resume where it left off after being interrupted) and you'd rather use the strategy suggested in that superuser.com answer, be aware that it suffers from a terrible case of useless use of cat. All you should need in any Linux less than ten years old is one process:

tee </dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde >/dev/sdf

and let the kernel block cache deal with all the buffering and unbuffering. There is absolutely no need to use dd to do I/O to block devices just because they are block devices. Reading and writing them the same way you'd do for any ordinary file usually works perfectly well.
posted by flabdablet at 5:05 AM on April 15, 2014 [1 favorite]

Also, if you're not committed to using USB for the external drives and/or installing multiple USB cards doesn't appeal, you'll get faster results with eSATA or even just using spare SATA ports on your existing mobo or a cheap SATA PCIE card. If you do that, and you're running some kind of duplication production line, lay in a little stock of SATA data cables - they tend to stop working reliably after a few tens of disconnections.
posted by flabdablet at 5:11 AM on April 15, 2014

Response by poster: Hi all - thank you so much for all this great information.
Btw the goal here wasn't backups but proper cloning (we have a couple of dozen identical computers so I wanted to slap the same hard drive image into each. I did sysprep and what not).
The platform is windows 7.
Thought I'd try to save time by copying the image to multiple usb-sata drive adapters with hdd's plugged into them. Surprised to find how difficult it was to find software for that.
posted by jak68 at 5:59 PM on April 15, 2014

Best answer: If all you actually need to do is write fresh Windows images to new computers, standalone block-for-block HD cloners or anything that emulates them are absolutely the wrong way to go about it. Pushing out a fresh Windows image will typically require writing well under 10GB to the drive - less than 2% of the capacity of a typical business machine in 2014.

The traditional way to solve the problem you have is to use Ghost, which does file-by-file copies of a single source machine to multiple target machines over the network efficiently using multicasting. If things are set up properly, you can even boot your not-yet-imaged target machines into the Ghost client off the network itself. Ghost works very well, but licensing it is expensive.

Microsoft also offers its own tools to do much the same thing (Microsoft Deployment Toolkit, Windows Automated Installation Kit) that you can use at no additional cost if you're already running Windows Server.

Personally I have found all these tools to be a complete pain in the arse, to the point where I just rolled my own imaging tools for the school I netadmin. I am not going to hand those out because I am not interested in supporting them, but I understand them and they work well enough for my purposes.

If I ever get to the point of becoming frustrated with the inefficiency of my own tools, I will be trying out both FOG and CloneZilla before expending much effort on improving them.
posted by flabdablet at 11:05 PM on April 15, 2014

I concur. The main reason I haven't bothered with adding multicast to my own cloning tools is that cloning to machine N takes much less time than unboxing, unwrapping, assembling and plugging together machine N+1.
posted by flabdablet at 11:26 PM on April 16, 2014