I have been charged with doing a full audit of my company's "web portal solution". This involves me going through the hundreds of pages and essentially developing an incredibly detailed sitemap showing where all pages link back and forth to. Please help me do this efficiently and accurately - I want to impress. [more inside]
posted by yoyoceramic on Jun 5, 2008 - 4 answers

How do i build a data warehouse that scrapes data from public websites for my own use? Tools? Tips? [more inside]
posted by ilike on Jul 30, 2007 - 15 answers

Why is google spidering specific but non-existent pages on my blog? [more inside]
posted by benzo8 on Sep 2, 2005 - 8 answers