Ask MetaFilter questions tagged with spidering

How to comprehensively spider a site

yoyoceramic — Thu, 05 Jun 2008 09:42:41 -0800

I have been charged with doing a full audit of my company's "web portal solution". This involves me going through the hundreds of pages and essentially developing an incredibly detailed sitemap showing where all pages link back and forth to. Please help me do this efficiently and accurately - I want to impress. I will add that this "Web portal solution" is indeed online, however it is password protected, and therefore I have not been able to find a web service that can automate this task. The ideal solution would create a document that has a tree-type structure, or maybe flowchart layout detailing what children URLS branch off of other parent URLS.

It gets tricky because there are several external links which do not need to be followed, and several links are just ASP pages (ie .../menupage.asp?pageid=21, .../menupage.asp?pageid=22 etc...) does this complicate things?

Is there a firefox add on that can track where I click and then create a logical, visual output of where I visited? Basically I need something to look at all the links on the page, follow those links to the sub page, then repeat this process until all links in the domain have been followed.

Any ideas?

Datamining the public web

ilike — Mon, 30 Jul 2007 01:52:51 -0800

How do i build a data warehouse that scrapes data from public websites for my own use? Tools? Tips? Hi. I would like to track apartments on a classifieds site and use the data for analyzing the inpact of diffrent things on price. What i need is a tool or scripting language that would make it easy for me to spider the website and put the data in a database. Preferable this would be an open source solution.

I am also looking for good tools for extracting information out of longer pieces of text. For example on the site i want to mine users can put in comments on every object. I would like to be able to decide if a comment is positive, negative och neither. I have seen this be done on one online art site that i cant remember the name of right now. The artist used blog post and decided the mood of the writer by what words were used.

"He's making it up as he goes along!"

benzo8 — Fri, 02 Sep 2005 20:05:52 -0800

Why is google spidering specific but non-existent pages on my blog? Over the last couple of days I've seen google's bot scanning my website, trying to access specific URLS:

www.benzo8.org crawl-66-249-66-3.googlebot.com - - [03/Sep/2005:03:03:32 +0100] "GET /summit/contact.html HTTP/1.1" 200 38112 "-

" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

www.benzo8.org crawl-66-249-66-3.googlebot.com - - [03/Sep/2005:03:03:46 +0100] "GET /pages/devonshire.html HTTP/1.1" 200 38114

"-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

www.benzo8.org crawl-66-249-66-3.googlebot.com - - [03/Sep/2005:03:03:57 +0100] "GET /pages/hampton.html HTTP/1.1" 200 38111 "-"

 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

www.benzo8.org crawl-66-249-66-3.googlebot.com - - [03/Sep/2005:03:04:09 +0100] "GET /chatsford/contact.html HTTP/1.1" 200 38115

 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Unfortunately, those URLs don't exist and they've never existed. So why is google requesting them? Is it just guessing, or what? Also, my site is set up so that requests which don't got to an existing page will go to the index page, so, if google requests these non-existent pages and gets the same content each time (ie: my index page) will it think (incorrectly) that I've set up some SEO linkfarm and lower my page-rank as a punishment?