Is there an easy way to search through a range of web pages to find which ones are active?

November 2, 2004 11:05 AM

Is there an easy way to search through a range of web pages to find which ones are active? [more inside]

I'm trying to find a job posting that is no longer openly posted at a web site (the closing date was several months ago). The url for the job posting is in the format:

http://anyplace.com/aa-1234.cfm

Is there a quick and simple way for me to check pages /aa-1000 through /aa-9999, without having to type each one?

(I did do a Google search, and while the posting I was looking for didn't come up, it did hit several pages that no longer appear on the general "Jobs Available" page.)

posted by neurodoc to computers & internet (5 comments total)

A Unix shell script could do this in about four lines. What OS are you running?
posted by DrJohnEvans at 11:16 AM on November 2, 2004

Nah, one line.

curl -O 'http://anyplace.com/aa-[1000-9999].cfm'

Of course that or any other brute-force solution would hammer the site with nine thousand separate page requests -- not a very nice thing to do. And you'd still have to search through the results to find whatever it is you're looking for.

Can you contact the site administrators and ask for whatever you're looking for, instead?
posted by ook at 11:57 AM on November 2, 2004

Okay, on the non-scripting end, have you tried a Google cache search, or possibly the Internet Archive? Either might have stored the site when your page was openly posted.
posted by DrJohnEvans at 12:03 PM on November 2, 2004

Thanks for the quick advice. I've actually been able to find what I was looking for a different way, but this is a general question that I've often wondered about.

FYI:

- I'm running WinXP
- I did try Google and the Archive, but neither had what I was looking for.
posted by neurodoc at 12:30 PM on November 2, 2004

If you're using Firefox, this extension can inc / decrement numeric parts of a URL, and even run a slideshow of the pages.
posted by punilux at 1:04 AM on November 3, 2004