Comments on: "He's making it up as he goes along!"

Question: "He's making it up as he goes along!"

benzo8 — Fri, 02 Sep 2005 20:05:52 -0800

Why is google spidering specific but non-existent pages on my blog? Over the last couple of days I've seen google's bot scanning my website, trying to access specific URLS:

www.benzo8.org crawl-66-249-66-3.googlebot.com - - [03/Sep/2005:03:03:32 +0100] "GET /summit/contact.html HTTP/1.1" 200 38112 "-

" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

www.benzo8.org crawl-66-249-66-3.googlebot.com - - [03/Sep/2005:03:03:46 +0100] "GET /pages/devonshire.html HTTP/1.1" 200 38114

"-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

www.benzo8.org crawl-66-249-66-3.googlebot.com - - [03/Sep/2005:03:03:57 +0100] "GET /pages/hampton.html HTTP/1.1" 200 38111 "-"

 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

www.benzo8.org crawl-66-249-66-3.googlebot.com - - [03/Sep/2005:03:04:09 +0100] "GET /chatsford/contact.html HTTP/1.1" 200 38115

 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Unfortunately, those URLs don't exist and they've never existed. So why is google requesting them? Is it just guessing, or what? Also, my site is set up so that requests which don't got to an existing page will go to the index page, so, if google requests these non-existent pages and gets the same content each time (ie: my index page) will it think (incorrectly) that I've set up some SEO linkfarm and lower my page-rank as a punishment?

By: RustyBrooks

RustyBrooks — Fri, 02 Sep 2005 20:33:53 -0800

I can only imagine that someone, somewhere, has those links on their site. It may be a mistake, or it may be something that I don't quite understand. It's about the only thing I can think of.

By: mosch

mosch — Fri, 02 Sep 2005 21:28:04 -0800

weird.

By: clord

clord — Sat, 03 Sep 2005 00:27:41 -0800

they might be using it as a test to see if you are in fact a link farm?

sorta like: "If I throw these random links at you, and you provide stuff to me, I suspect you of trying to pollute your legitimate links too"

By: cillit bang

cillit bang — Sat, 03 Sep 2005 03:32:04 -0800

I notice you're returning HTTP 200 codes. You should return 404 errors or 301 redirect codes if you want Google to realize those pages don't exist.

By: benzo8

benzo8 — Sat, 03 Sep 2005 04:43:23 -0800

Well, basically every URL that arrives at the site goes through a rewriting process, and those that don't land on a certain point in the database (ie: an extant page) will return the frontpage content, so a 200 is correct in terms of outcome, but in this particularly situation, and given what I feared and clord suggested too - I could be damaging my google page rank by doing that, if they're doing what we think they might be doing.

Don't we imagine that google would be intelligent enough to check if content they were returned when they made a "random" check was actually relevant - ie: if they did trip over a link farm, they normally build the search terms/urls into the content of the dynamic page to increase their googleness. My site will just return the same index/content page time and time again, but it will (unless entirely coincidentally) bear no relevance to the URL google's decided to test me with...

So, in short - is the consensus that google are checking for me being a link farm, and I'm currently doing nothing to disuade them of that notion?

By: cillit bang

cillit bang — Sat, 03 Sep 2005 04:46:41 -0800

Every page should have one single canonical URI and everything else should be a redirect to it. Not just to help out Google, it's also good design.

By: malevolent

malevolent — Sat, 03 Sep 2005 07:09:29 -0800

"those that don't land on a certain point in the database (ie: an extant page) will return the frontpage content"

That's a really bad idea, as you're finding out. Get a helpful error page in there and use the right status code.

Is your site set up to respond only to the correct host header, or will it treat any request to the correct IP as being for that site? If it's the latter then the requests could be either for someone who previously had that IP, or for a domain name that's been incorrectly pointed your way.

By: benzo8

benzo8 — Sat, 03 Sep 2005 07:13:44 -0800

The site is set up for virtual hosts, so will only respond to www.benzo8.org requests. In terms of the URL handling - I'm running Mambo, and that pretty much does its own thing. I've got the SEO-friendly URL modules switched on and I guess that's what handles the rewriting, but I think even without it, it defaults to the index page, so I'm gonna have to get my hands dirty with the php I guess and find out how to get it to fail gracefully rather than lazily... Thanks to all.