Big downloads on firewire external disks?

September 6, 2006 12:04 PM

OK, let's say I'd like to download this thing, but it's a 56 GB thing (well, it's a bunch of things, let's say, for educational purposes only, .par and .rar files). My hard disk is only 30 GB (about 11 GB of currently free space). But: I have a 250GB external drive (firewire) that I sometimes use to store stuff. Can I download the big file (it'll take, I guess, an entire day) straight on the external drive? Or is my Mac (G4) going to blow up for some reason? thanks!!!
posted by PenguinBukkake to technology (23 comments total)

Yes that will work fine.
posted by nathan_teske at 12:04 PM on September 6, 2006

I haven't done this on a Mac, but things worked fine on my PC with a 200GB PATA drive connected via USB2.

The only thing I'd warn about is using an external enclosure that doesn't have a fan. They can get really warm (hot is more like it), so just for safety's sake, you may want to put the drive on a non-flammable material and away from anything that may be damaged when exposed to heat.
posted by owenkun at 12:10 PM on September 6, 2006

Decompressing this "thing" may also have complications, especially if it is compressed well at all.
posted by smackfu at 12:14 PM on September 6, 2006

yes, you can download it. but 56GB ... that's gangraping the internet.
posted by krautland at 12:34 PM on September 6, 2006

650kB/sec sustained for 24 hours straight. Sounds intimidating, but I'd think some file sharers have done far worse with internal drives. I've seen home file servers that put that to shame on an outgoing basis.

Listen to owenkun, see to it the enclosure's on something that can take the expected heat of internal computer parts, hopefully your enclosure is ventilated. You could use a download manager to split the file and allow for breaks (a good idea anyway), but if your enclosure has a fan and ventilation is free, you shouldn't have any trouble.

Your greatest worries are inability to resume the file (Which at 56 GB, will naturally break transfer somewhere in the low 50s just to spite you), decompression if it's one large file, and hoping your ISP/campus/workplace/etc doesn't throw a fit.
posted by Saydur at 12:54 PM on September 6, 2006

gangraping the internet

why? that's the whole point of all-u-can eat Usenet
posted by PenguinBukkake at 1:14 PM on September 6, 2006

I use a non-ventilated aluminum enclosure and it does indeed get quite hot after some use. It feels hot on the outside partly because the aluminum does a good job in conducting the heat away from the hard drive. However, when I aim my room fan at it, it doesn't even stay warm, so things can be done if you are worried about heat.

If it's a series of RARs with PARs, you shouldnt have any trouble resuming a borked transfer, so I wouldn't worry about that. Even if you lose precious bits from one or more RAR, that's why you have the PAR files handy.
posted by utsutsu at 1:27 PM on September 6, 2006

Given you username, I don't think it's wise to ask about the contents of the file, but as for the size, it shouldn't be a problem; although I've never done it with a file bigger than 15GB.
posted by blue_beetle at 1:30 PM on September 6, 2006

Why wouldn't this work? It's just a drive, just like your internal ones.
posted by chrismear at 1:49 PM on September 6, 2006

One big difference can be whether you are downloading with bittorrent or from Usenet or something similar. With bittorrent, all of the files are incomplete and active until the transfer is completely finished. This means the external drive has to be active all the time. Basically any drive anywhere, even in a really crappy enclosure, should survive a few weeks of that treatment, although you'd be wise to aim a fan its way if it has no ventilation.

If it's a sequential download, where you get separate, complete pieces at a time, download them to your internal drive and copy them every few hours to your external drive. That'll let it spin down and cool off. Then do your PAR checking and rebuild on the external unit. That'll be the minimum possible wear.
posted by Malor at 1:55 PM on September 6, 2006

Malor: For Azureus, at least, a 99% complete torrent will have all files not affected by the missing 1% in their complete form, on disk. It may be in your temporary folder, depending on how you have your client configured, but unless the torrent consists of a single file, BitTorrent works like anything else.
posted by Coda at 2:46 PM on September 6, 2006

Save to your external drive from inside of whatever software you're using.

You'll be fine. Many many systems are on 24/7 and not have any headaches.
posted by filmgeek at 2:50 PM on September 6, 2006

Point a fan at your little external drive just to be extra extra safe.
posted by damn dirty ape at 2:54 PM on September 6, 2006

Note that some browsers attempt to download files to a temporary location (on your boot drive) and then move the file to the actual destination that you set.

This would cause complications for you, because the temporary file would fill your smaller drive before it was saved completely. You may need to use a tool that you know will write the file to the real destination as it grabs it.
posted by odinsdream at 4:38 PM on September 6, 2006

You'll be fine... Though 56 Gigs is an aweful lot to be downloading in one day... If you're downloading from Usenet, make sure that your ISP doesn't limit your usenet throughput... a buddy told me that Comcast limits his usenet speeds, hence why I don't have Comcast Cable. I would be prepared to spend a week or more downloading said files. Download them to your main drive, and move the completed ones to your external drive once you're done.

ALSO of importance... some decompression utilities will decompress to the main/system drive, then copy the files to the target directory after they're decompressed... If this happens, you'll have to decompress sections (I'm guessing there will be folders) of files at a time.

Also, don't download the par files if you don't need them...
posted by hatsix at 4:48 PM on September 6, 2006

(p.s. I'm betting you're using Usenet because of the .par files... people don't use par files on torrents because the hash checks mean that all of the files you download are correct... Though I know some... easy... news... sites allow you to download via. http/ftp, same difference, but if you do this, download the rars/pars, NOT the uncompressed files.)
posted by hatsix at 4:51 PM on September 6, 2006

Coda, if the files are stored as a group, that is, they're all in a single torrent, then BT will randomly download bits and pieces spread across all of them.

When the protocol is working properly, you won't have any complete files until almost all of them are finished; the last remaining bits will be randomly scattered throughout.

It sounds like Azureus may move files around based on their completed status, but if it works like other BT clients, you'll see that nothing moves until the torrent is almost done, and then all the files move very quickly as the last random chunks fill in.

Note that this doesn't apply to running several different torrents simultaneously; this is only for a single torrent of multiple files.
posted by Malor at 5:02 PM on September 6, 2006

Malor: Using file priorities for BT (at least with Azureus), you can download complete files ahead of other files. Very handy.

Just make sure you have your temp directory set to the external drive. The Windows TEMP path may have to be changed too. That depends on what app you use to download.
posted by devilsbrigade at 5:05 PM on September 6, 2006

if it's usenet, your speed will very probably be limited ... and how long will the usenet server retain 56 gigs of files? ... if it's a bunch of different files (and i can't imagine a binary group with 56 gigs on it!) ... download the oldest first, and make sure you grab those pars, because in that amount of downloading, you will have incomplete archives

with xnews, i can create 2 queues and download 2 things at once ... your news server may allow more ... or just one, it depends on their policy

some news admins might not be happy with someone downloading 56 gigs in one fell swoop, even at the reduced speed ... (my provider limits me to 2 queues @ around 20K-bytes a piece, which is a lot less than the speed i get on the web)
posted by pyramid termite at 9:38 PM on September 6, 2006

pyramid termite: At least 2 providers have ~70 day retention on binary groups, and they don't have speed limits you're likely to run into. As for "i can't imagine a binary group with 56 gigs on it", you do realise that a full Usenet feed runs into several TB per day, much of it carried by only a small handful of groups, right? ;)

PenguinBukkake: You might find "the" unofficial newzbin forum useful.
posted by Freaky at 6:31 AM on September 7, 2006

freaky - i don't have one of those providers and i certainly haven't seen 56 gigs on a newsgroup on mine ... my situation is likely to be more typical than yours

metafilter - my news server sucks
posted by pyramid termite at 9:42 AM on September 7, 2006

pyramid termite:

for educational purposes, you might want to check out http://easynews.com (I didn't link it on purpose.... )
posted by hatsix at 11:56 AM on September 7, 2006

If you can't use Bittorrent, use some sort of download manager (like GetRight), to support resuming, and multistreaming of files. It's the best way I've found to saturate my connection, and to make up for any dropped connections.
posted by blue_beetle at 7:01 AM on September 9, 2006