Ask MetaFilter questions tagged with webcrawling

How do we speel check an entire web sight?

advicepig — Fri, 30 Jan 2009 09:06:08 -0800

How do we spell check (and link check) an entire web site? So I look after a number of sites and we're starting to consider how we spell check an entire web site. The site uses a content management system and a number of other applications that deliver content to the user. We have things in place to spell check in the editor window of the CMS, but we're looking at a way to crawl the site and report errors. If the toolset also did link checking and validation, we'd be even happier. Got any recommendations?

How to comprehensively spider a site

yoyoceramic — Thu, 05 Jun 2008 09:42:41 -0800

I have been charged with doing a full audit of my company's "web portal solution". This involves me going through the hundreds of pages and essentially developing an incredibly detailed sitemap showing where all pages link back and forth to. Please help me do this efficiently and accurately - I want to impress. I will add that this "Web portal solution" is indeed online, however it is password protected, and therefore I have not been able to find a web service that can automate this task. The ideal solution would create a document that has a tree-type structure, or maybe flowchart layout detailing what children URLS branch off of other parent URLS.

It gets tricky because there are several external links which do not need to be followed, and several links are just ASP pages (ie .../menupage.asp?pageid=21, .../menupage.asp?pageid=22 etc...) does this complicate things?

Is there a firefox add on that can track where I click and then create a logical, visual output of where I visited? Basically I need something to look at all the links on the page, follow those links to the sub page, then repeat this process until all links in the domain have been followed.

Any ideas?

What to do with spare download bandwidth?

kureshii — Sun, 01 Jun 2008 22:57:09 -0800

Is there anything meaningful I can do with 10Mbps worth of spare download bandwidth? My ISP recently has an offer on a 10Mbps connection, under a 2-year contract. I've been thinking about getting a separate connection, mainly because the price is good and because I want a good upload bandwidth for torrent-seeding (don't ask).

That plan comes with 1Mbps upload bandwidth which, as mentioned, will be largely used for torrent-seeding. That leaves the 10Mbps download bandwidth largely unemployed. I will still be downloading every so often, but hardly enough to put more than the occasional dent in it.

This is where I need to pick your brains. Are there any meaningful projects out there requiring spare bandwidth, and if so, how can I contribute? Is there are other way I can put it to good use?

I've also been thinking about social network-related experiments and data-logging webcrawlers that would gather historical, social and other data (of course, ideally without maiming other servers), but have yet to think of a meaningful way to turn that into useful insights or conclusions.

If anyone has ideas on what to do with that download bandwidth (that would preferably leave my upload bandwidth alone), please hit me with them =)