3 posts tagged with robots.txt. (View popular tags)
Displaying 1 through 3. Subscribe: http://ask.metafilter.com/tags/robots.txt/rss RSS feed for this tag

Why have Google and Microsoft shunned my web site? And why does Google's sitemap control panel think my robots.txt file is unreachable? [more inside]
posted on Aug 27, 2007 - 7 answers

Porn sites are spamming my web stats. Make it stop. [more inside]
posted on May 16, 2007 - 3 answers

how do I make my blog be invisible to search engines? [more inside]
posted on Dec 8, 2005 - 20 answers