Download secured content to read later?

July 3, 2007 1:47 PM

How can I download/cache web pages that require cookies and/or a login for later reading in Firefox (or IE). Is there a spider plugin for either browsers, or a stand alone program that will support authentication?

I'm going away to somewhere I likely won't have internet access full time. I'd love to be able to download my livejournal friends page and some other pages for reading offline. Say, the root page plus 1 level of links deep, for articles and content not on the main page.

The hard part: many of the pages I want are password or login protected. NYT premium content, private livejournal pages, etc.

IE6 used to be able to do this, but it looks like they've dropped the feature in 7. I've tried HTTrack, Getright, Flashget and wget, but can't get around the cookies/login issue.

I think I need to do this from inside IE or Firefox, so that I can take advantage of pre-existing cookies, but I can't seem to find a way. Please help.

Otherwise, is there a way to use getright, flashget or some other caching-downloader to authenticate before downloading?

This is on Vista Ultimate with FF2 or IE7.

I'd even be willing to pay for software if it does a good job, so free is not a requirement.

posted by tiamat to Computers & Internet (8 answers total)

Best answer: I love Scrapbook, a Firefox plugin that downloads pages and makes locally browsable copies. It works off the browser's current state, so if you're logged in and cookied and whatever at the moment, it will pull the page.
posted by dammitjim at 2:16 PM on July 3, 2007

Oh, it doesn't download Flash BTW, but many other binary files (pdfs, etc) are fine.
posted by dammitjim at 2:17 PM on July 3, 2007

I don't have access at the moment to my windows computer that has a copy installed but I believe that BlackWidow can handle logins. It is a pretty powerful program. There are also plugins available for it to extend it's functionality.
posted by Gomez_in_the_South at 2:50 PM on July 3, 2007

It seems you can use authentication. Also, it isn't free (shareware - $39.95) but has a fully operational trial version.
posted by Gomez_in_the_South at 2:55 PM on July 3, 2007

Best answer: Did you try using cookies in wget?
posted by anaelith at 7:55 PM on July 3, 2007

WinHTTrack does this, you need to use the form based authentication.
posted by Mitheral at 11:23 PM on July 3, 2007

I think Firefox will do what you want without adding any extra extensions.

1. Visit the about:config URL, type "cache" into the "Filter:" box, and change the browser.cache.disk.capacity value from the default 50000 to 1000000. This increases Firefox's maximum disk usage size from 50 megabytes to 1 gigabyte, which should be enough. If you later find it isn't, you can make it bigger.

2. Create a bookmarks folder containing bookmarks for all the pages you want to check.

3. When you have Internet access, right-click that bookmark folder and select "open all in tabs". If you need to do logins at this stage, do them.

4. Once all the pages have loaded, check "Work Offline" in the File menu (and quit Firefox, if you want).

You should now be able to browse to any of those pages even when you have no working Internet connection.
posted by flabdablet at 3:40 AM on July 4, 2007

Dammitjim is right: You need to use Firefox with the Scrapbook extension. It will capture any page you're visiting. It's free, it's simple, and it does even more than you are hoping for because you don't have to think about logins, cookies, etc.
posted by gum at 10:56 PM on July 4, 2007