Collecting a site's URL list, , METa Descriptions etc...
July 29, 2011 7:08 AM   Subscribe

Looking for a tool for the quick and simple 'collection' of a website's page URLs along with each URLs associated 'title,', META Description, & META Keywords.

I used to be able to do this ~ in the past couple years~ with the free version of a product whose intended purpose was to detect if a site had an analytics solution installed. It worked pretty well up to the 20 URL limit on the free version. The full version is kinda pricey & limited to 5 domains..

Years ago, like a decade or so, there was an awesome product called Metabot Pro, which not only collected this info, but had a nifty editor for making adjustments to the static HTML files.

I'm looking for something like that again.... for static HTML pages. I'm trying to automate my currently manual copy/paste routine of collecting all the static HTML page URLs (filenames), & each URLs associated 'title', META Description etc.

Can ya hope me?
posted by bricksNmortar to Computers & Internet (4 answers total)
 
I don't know of any specific tools, but I'd probably use ScraperWiki for this - it lets your write little scripts to gather data from websites. You don't need to be a programmer - I'm not, but have managed to do pretty complex scraping stuff by reading the tutorials/trial and error.
posted by jack_mo at 7:59 AM on July 29, 2011


This seems like something to write a little script for, rather than something to buy a dedicated tool for.

Do you have the static HTML files on your own computer? If so, what OS?

Or are they on the server and you have to work with them there? If so, do you have to do this through http, or do you have shell access or something?
posted by stebulus at 8:01 AM on July 29, 2011


Response by poster: stebulus - I do not have the files on my computer... I want to be able to point the tool, or whatever, to a website & let the collecting begin...

I'm going to have a look at scraperwiki & will report back.
posted by bricksNmortar at 8:09 AM on July 29, 2011


Response by poster: ya know.. I went ahead and bought the "trainee" version of the analytics solutions detector software. they have both a $699 & $69 version...

$69 to save me two hours of work was a no-brainer.

thanks for the input.
posted by bricksNmortar at 8:37 AM on July 29, 2011


« Older Will this bed bite me back?   |   Video within a still image with Final Cut. Newer »
This thread is closed to new comments.