Comments on: How to download website directory?

Question: How to download website directory?

abdulf — Mon, 21 Jul 2008 18:01:03 -0800

I'm trying to download from links en masse, any software solutions?

I'm trying to download files and links from a website "directory". It sorta looks like this. I guess it's directory made with html links.

Wish I could just drag and drop the folders.

I found a website copying program but it doesn't handle spaces in the address very well. Any ideas besides having to manually click each link by hand?

By: odinsdream

odinsdream — Mon, 21 Jul 2008 18:08:33 -0800

wget will do exactly what you're looking for. Curl will also do it, but I learned wget first and so recommend it first. It has a method to create a local copy of any site to a specific depth. It will also rewrite all the links and references so that you can locally browse the resulting folder offline.

By: a robot made out of meat

a robot made out of meat — Mon, 21 Jul 2008 18:08:53 -0800

downThemAll. URLToys.

By: CunningLinguist

CunningLinguist — Mon, 21 Jul 2008 18:11:41 -0800

Downthemall rocks.

By: door2summer

door2summer — Mon, 21 Jul 2008 18:12:25 -0800

I prefer Flashget, it integrates very well with the Firefox browser and has a great depth of tools for a variety of uses.

Hope that helps

By: box

box — Mon, 21 Jul 2008 18:12:55 -0800

Nthing Downthemall and wget (or winwget).

By: fogster

fogster — Mon, 21 Jul 2008 18:37:32 -0800

I came here to recommend DownThemAll. It seems I'm not the first.

By: Effigy2000

Effigy2000 — Mon, 21 Jul 2008 19:08:38 -0800

Seconding Flashget, though to integrate it into Firefox properly, as door2summer has said, you'll want the Firefox extension Flashgot.

By: tgmayfield

tgmayfield — Mon, 21 Jul 2008 19:13:26 -0800

lftp (like wget, another *nix tool available for Windows) can do smart stuff with HTTP mirroring and recursion, wherein you download the contents of subdirectories and their subdirectories (which I don't know that downthemall supports).

wget is by far the most powerful of the two, but I always seem to have to re-read the wget man page if I've not used it for a mirror in a month or so to get all the options perfect. If all I want is to mirror a visible-in-links directory structure (such as Apache index pages), lftp is definitely the easy tool:

mkdir MyMirror
lftp http://example.com/path/to/file/listing/
mirror

By: idb

idb — Tue, 22 Jul 2008 06:14:20 -0800

Linky is another Firefox option.

By: Fin Azvandi

Fin Azvandi — Tue, 22 Jul 2008 09:03:08 -0800

If FF is not your primary browser, SiteSucker is a donation-ware Mac OS X app.

By: LuckySeven~

LuckySeven~ — Tue, 22 Jul 2008 09:21:19 -0800

Blue Crab and Speed Download are also very good.