What's the best practice for authenticating Google's crawlers?
September 17, 2008 10:27 AM Subscribe
I manage a website. Some content requires authentication by password or IP. I want Google to crawl that content but not cache it, so that Google users can find it in searches but can't access it without authentication. What's the best way to do this?
In 2006 Matt Cutts recommended
doing a reverse DNS lookup to verify that a bot's name is in the googlebot.com domain, and then a forward DNS->IP lookup using that googlebot.com name (to thwart spoofers).
Is that still the best solution? How do other people manage this?
posted by futility closet to computers & internet (9 answers total) 1 user marked this as a favorite