Collecting a site's URL list, , METa Descriptions etc...
July 29, 2011 7:08 AM Subscribe
Looking for a tool for the quick and simple 'collection' of a website's page URLs along with each URLs associated 'title,', META Description, & META Keywords.
I used to be able to do this ~ in the past couple years~ with the free version of a product whose intended purpose was to detect if a site had an analytics solution installed. It worked pretty well up to the 20 URL limit on the free version. The full version is kinda pricey & limited to 5 domains..
Years ago, like a decade or so, there was an awesome product called Metabot Pro, which not only collected this info, but had a nifty editor for making adjustments to the static HTML files.
I'm looking for something like that again.... for static HTML pages. I'm trying to automate my currently manual copy/paste routine of collecting all the static HTML page URLs (filenames), & each URLs associated 'title', META Description etc.
Can ya hope me?
I used to be able to do this ~ in the past couple years~ with the free version of a product whose intended purpose was to detect if a site had an analytics solution installed. It worked pretty well up to the 20 URL limit on the free version. The full version is kinda pricey & limited to 5 domains..
Years ago, like a decade or so, there was an awesome product called Metabot Pro, which not only collected this info, but had a nifty editor for making adjustments to the static HTML files.
I'm looking for something like that again.... for static HTML pages. I'm trying to automate my currently manual copy/paste routine of collecting all the static HTML page URLs (filenames), & each URLs associated 'title', META Description etc.
Can ya hope me?
This seems like something to write a little script for, rather than something to buy a dedicated tool for.
Do you have the static HTML files on your own computer? If so, what OS?
Or are they on the server and you have to work with them there? If so, do you have to do this through http, or do you have shell access or something?
posted by stebulus at 8:01 AM on July 29, 2011
Do you have the static HTML files on your own computer? If so, what OS?
Or are they on the server and you have to work with them there? If so, do you have to do this through http, or do you have shell access or something?
posted by stebulus at 8:01 AM on July 29, 2011
Response by poster: stebulus - I do not have the files on my computer... I want to be able to point the tool, or whatever, to a website & let the collecting begin...
I'm going to have a look at scraperwiki & will report back.
posted by bricksNmortar at 8:09 AM on July 29, 2011
I'm going to have a look at scraperwiki & will report back.
posted by bricksNmortar at 8:09 AM on July 29, 2011
Response by poster: ya know.. I went ahead and bought the "trainee" version of the analytics solutions detector software. they have both a $699 & $69 version...
$69 to save me two hours of work was a no-brainer.
thanks for the input.
posted by bricksNmortar at 8:37 AM on July 29, 2011
$69 to save me two hours of work was a no-brainer.
thanks for the input.
posted by bricksNmortar at 8:37 AM on July 29, 2011
This thread is closed to new comments.
posted by jack_mo at 7:59 AM on July 29, 2011