Does my webhost not like Google?

February 21, 2009 8:54 AM

How can I figure out if my webhost is blocking Google indexing? A site I manage is listed on Google but any references to it are at least several weeks old. More info inside.

Also I tried getting the site into the Adsense program but Google responded that our site was down or blocked. I have had trouble with the host server over aggressively blocking IP addresses for "too many bad logins" but somehow I think this is different.

When the users get blocked ive been able to submit their IP address or range to the host and 'tada' no more problems, but with Google, I've no clue what range. Or even if it's my host, maybe its my config or just Google.

http://thebehrendbeacon.com
Its a college newspaper site using Wordpress along with the SEO plugins that seem reasonable, (All in one SEO, A sitemap generator that submits to Google and others, and some others)

The robots.txt only blocks the crawlers from, non website folders, like mail, spam settings, etc.

I have tried Google webmaster tools, when I try to verify the site (HTML), it gives me an error saying it cant find the html file, or there was a DNS error.

My webhost is Mysticdreaming.net, cheap, small time, and nearly instant support. (this is really the only problem)

Any help would be appreciated

posted by tropikal to computers & internet (5 comments total)

You can check your logs and see if Googlebot pays a visit.

But a quick search shows that the site has been recently indexed.
http://www.google.com/search?client=safari&rls=en-us&q=site:thebehrendbeacon.com&ie=UTF-8&oe=UTF-8
posted by avex at 9:04 AM on February 21

your robots.txt throws a 500 (internal server error).
posted by Mach5 at 9:28 AM on February 21

Following up on mach5, it might be your wordpress install. Probably has something to do with the .htaccess file that should be there. This thread might help.
posted by gchucky at 9:43 AM on February 21

I'd try digging further into the webmaster tools issue.

Also, it doesn't explain why your site isn't being indexed, but your permalink structure isn't exactly SEO friendly.
posted by Good Brain at 11:50 AM on February 21

Thanks go out to all, I kept chipping away at it and found out that the permissions were totally wrong on my robots.txt, as well as the html file that Google webmaster tools uses to verify sites, so after fixing that and adding some more SEO friendly post titles it looks like everything's going to work out fine. Thanks.
posted by tropikal at 8:48 AM on February 27