Comments on: How to save all cached pages for a particular domain?

Question: How to save all cached pages for a particular domain?

Ø — Thu, 28 Jun 2007 22:53:10 -0800

A web site with lots of useful information recently went offline, but the cached pages are still available from Google. Is there a program (for either OS X or Windows) that will automatically save all of the cached pages on Google associated with a particular domain?

By: Blazecock Pileon

Blazecock Pileon — Thu, 28 Jun 2007 23:06:00 -0800

Have you tried wget or curl with the Wayback Machine?

By: phrontist

phrontist — Thu, 28 Jun 2007 23:11:55 -0800

wget is the way to go.

I love your nick

By: zippy

zippy — Fri, 29 Jun 2007 01:21:30 -0800

Note - the Wayback machine doesn't show new pages until many months (6 - 12) later, so you may want to check them again several months from now.

By: blag

blag — Fri, 29 Jun 2007 01:41:34 -0800

Warrick is a command-line utility for reconstructing or recovering a website when a back-up is not available. Warrick will search the Internet Archive, Google, MSN, and Yahoo for stored pages and images and will save them to your filesystem.

By: Ø

Ø — Fri, 29 Jun 2007 01:45:24 -0800

Unfortunately the Wayback Machine was blocked by the site's robots.txt. Google has it all cached, though.

I've been reading the documentation for wget (which looks really cool -- thanks for the recommendation!), but can't figure out how to get it to archive just the cached page links returned by Google. I'm probably overlooking something very obvious... any pointers would be appreciated...

By: Ø

Ø — Fri, 29 Jun 2007 01:47:53 -0800

Wow, blag! Warrick looks perfect!!!! Thank you.

By: Merdryn

Merdryn — Fri, 29 Jun 2007 08:01:51 -0800

I was wondering if there's another program out there like Warrick, or if it's the only one (sorry to derail, but really curious about this one)

By: blag

blag — Sun, 01 Jul 2007 18:09:18 -0800

Happy to help.

Merdryn: as far as I know, it's the only one.