Ask MetaFilter questions tagged with robots.txt

How to make Google and Microsoft like me again?

JDHarper — Mon, 27 Aug 2007 11:53:47 -0800

Why have Google and Microsoft shunned my web site? And why does Google's sitemap control panel think my robots.txt file is unreachable? A few days ago, I discovered that my website has mostly disappeared from Google, and I am utterly confused as to why. In the past, if you searched for JD Harper, you'd get my personal blog at http://www.jdharper.com/wordpress as the first result. It seems that that page, as well as most of my best posts, have disappeared from the Google index. (I've tried a site search to see what's missing).

Microsoft is also excluding my web site from the search results at live.com, but Yahoo! still has my site at the top. (For now.)

When I logged in to the Google Webmaster Tools, it said that my page was included in the index, but that my sitemap contained errors. The error message that it gives me is as follows:

Network unreachable: robots.txt unreachable
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.

Under Diagnostics, under "Crawl Errors," it lists 133 "Unreachable URLs," all of which say "robots.txt unreachable," with a link to the previous error message.

My robots.txt file is located at http://www.jdharper.com/robots.txt, and it consists entirely of a pointer to my sitemap. I thought that perhaps the crawler had just tried to access robot.txt at a time when the server was down, but it's been several days and it says that it encountered the same error earlier today.

So it looks to me like Google is excluding those files from the index since it can't get to my robots.txt file. But I can't figure out why Google can't find the file.

The only thing I've tried today is changing the file permissions on robots.txt to 755, but I doubt that that will fix the problem. I'm still waiting for Google to download and check that file.

Any ideas here? What can I do to fix this?

Porn sites are spamming my web stats. Make it stop.

Milkman Dan — Wed, 16 May 2007 10:53:16 -0800

Porn sites are spamming my web stats. Make it stop. For a few months now, my referral log has been flooded by mildly amusing, but mostly annoying, porn site URLs. These sites are throwing off my data entirely, filling the log with anywhere from 1 to 200 hits each. Almost all of them are over-the-top porn URLs. I want it to stop.

I know what the phenomenon is, and I’ve read all the other AskMe threads about it. I’m not interested in diagnostics, I want to know the cure. I run WordPress, and I’m currently running the Bad Behaviour plug-in, and have been for two months, but the problem has remained. Yes, I run Google Analytics as an alternative, but I’d like to solve the problem within this stats program (Webalizer Version 2.01), too.

Clear, step-by-step instructions would be most helpful, but any suggestions would be much appreciated. Oh, and you can click here to view my referral log from May.

how do i make my blog invisible?

aussicht — Thu, 08 Dec 2005 23:43:14 -0800

how do I make my blog be invisible to search engines? hi,

i want my blog to not be something people pull up with google and the like. i put up a robots.txt file but it didn't seem to help. here is it

# Robots.txt file from http://www.searchengineworld.com
#
# Bans all robots will spider the domain

User-agent: *
Disallow: /

any ideas? thanks.