Comments on: Is there an easy way to search through a range of web pages to find which ones are active?

Question: Is there an easy way to search through a range of web pages to find which ones are active?

neurodoc — Tue, 02 Nov 2004 11:05:22 -0800

Is there an easy way to search through a range of web pages to find which ones are active? [more inside]

I'm trying to find a job posting that is no longer openly posted at a web site (the closing date was several months ago). The url for the job posting is in the format:

http://anyplace.com/aa-1234.cfm

Is there a quick and simple way for me to check pages /aa-1000 through /aa-9999, without having to type each one?

(I did do a Google search, and while the posting I was looking for didn't come up, it did hit several pages that no longer appear on the general "Jobs Available" page.)

By: DrJohnEvans

DrJohnEvans — Tue, 02 Nov 2004 11:16:30 -0800

A Unix shell script could do this in about four lines. What OS are you running?

By: ook

ook — Tue, 02 Nov 2004 11:57:52 -0800

Nah, one line.

curl -O 'http://anyplace.com/aa-[1000-9999].cfm'

Of course that or any other brute-force solution would hammer the site with nine thousand separate page requests -- not a very nice thing to do. And you'd still have to search through the results to find whatever it is you're looking for.

Can you contact the site administrators and ask for whatever you're looking for, instead?

By: DrJohnEvans

DrJohnEvans — Tue, 02 Nov 2004 12:03:48 -0800

Okay, on the non-scripting end, have you tried a Google cache search, or possibly the Internet Archive? Either might have stored the site when your page was openly posted.

By: neurodoc

neurodoc — Tue, 02 Nov 2004 12:30:45 -0800

Thanks for the quick advice. I've actually been able to find what I was looking for a different way, but this is a general question that I've often wondered about.

FYI:

- I'm running WinXP
- I did try Google and the Archive, but neither had what I was looking for.

By: punilux

punilux — Wed, 03 Nov 2004 01:04:51 -0800

If you're using Firefox, this extension can inc / decrement numeric parts of a URL, and even run a slideshow of the pages.