Google, was verfolgst du mich?

March 30, 2007 12:45 PM

My website/blog has all but disappeared from Google. What happened, and what can I do to fix it?

For years my personal website (linked in my profile) has been the first hit for "Rich Lafferty", without the quotes, in Google. But suddenly it's not showing up at all. I still have PageRank (5), and it's still in there if I search for the right terms, but it doesn't appear for the obvious search.

There's a whole bunch of things that could be contributing factors: I recently redid the entire site, from a static site to a blog. I started using a sitemap. I was recently linked to by some major blogs. I've been copying my content to my LiveJournal (username "mendel") -- and since I wondered if I was hitting Google's duplicate content trap, I've just today put a robots.txt file on the LiveJournal.

Aside from disappearing from the listings, the other weird things I've noticed are that my main page is no longer cached on Google, and searches for phrases from individual posts of mine turn up multiple duplicate results: my site, my LiveJournal, my Feedburner feed, Technorati, and some of my LiveJournal friends' friend lists.

Am I just in the process of getting re-added after I changed everything around? Did I get burned by the duplicate content trap, which considered my LiveJournal to be the main copy? Are there any other red flags you see?

It's not such a big deal that I'm not #1 anymore, but since I work in high tech I'd like to figure out how to at least show up again.

posted by mendel to computers & internet (10 comments total)

I should add that my Yahoo and MSN Search results remain unchanged.
posted by mendel at 12:47 PM on March 30, 2007

I recently redid the entire site, from a static site to a blog.

Did you put 302 "CONTENT MOVED" redirects in place or mod_rewrite rules so that your static pages resolved? If not, then, you screwed your longetivity factor all to hell, and Google thinks your site was deleted and is no longer there.

It'll come back, but make sure you don't have dynamic links for your blog -- pages with a querystring (?) in them.
posted by SpecialK at 1:22 PM on March 30, 2007

Did you put 302 "CONTENT MOVED" redirects in place or mod_rewrite rules so that your static pages resolved? If not, then, you screwed your longetivity factor all to hell, and Google thinks your site was deleted and is no longer there.

The 80% of the content that stuck around either stayed at the same URL or is getting mod_rewrite'd to the new URL. (Although I see it's doing 302 redirects, not 301s, I'll fix that shortly.) But as for the main/front page, will Google have noticed that it's nothing like what was there before and treat me as if I was brand-new?
posted by mendel at 1:41 PM on March 30, 2007

How recently is recently? I've had this sort of thing happen a couple of time—my name just suddenly stops returning meaningful results for my domain—but it has always resolved itself after a couple of (nail-biting, ego-crippling) days.
posted by cortex at 1:43 PM on March 30, 2007

I did a really stupid thing once where I was trying to add some search engine meta tags to my blog -- i managed to set it to "no index, follow" on every single page, so Google was dutifully following my instructions and not indexing anything. Yahoo! and MSN were still working though, which is why it took me a while to get it. It's a longshot, but maybe you accidentally did something similar in your changes?
posted by ukdanae at 2:13 PM on March 30, 2007

Assuming you're mostly concerned about Google, the webmaster tools they offer are a great collection of resources for better understanding these issues. You can use the Sitemaps tool (linked there) to actually find out directly from Google why they removed you from the crawl. It seems like you've figured out the duplicate content issue (I work with the LiveJournal team, so let me know if you need more assistance there) but if you ever get stuck in the future, you can just go into Google Sitemaps and have it tell you what the problem is.
posted by anildash at 3:57 PM on March 30, 2007

ukdanae: At first I thought it was exactly that -- I had accidentally set Wordpress's "no bots" thing on on my blog for about five minutes on Thursday before realizing I was on the wrong control panel -- but I checked and there were no Googlebot accesses during that time. Right now I'm good there, at least.

anildash: Thanks for the webmaster tools pointer, very handy. I'm surprised Google tells so much! I wasn't removed from the crawl, though -- it shows up as crawled, it's just.. unpopular somehow. I've submitted sitemaps in there for now, at least. I am more and more convinced it was a duplicate site problem combined with the site being "new" (and now that I think about it more, I also imported all of my LiveJournal posts to my WordPress blog, which would have compounded the duplicate problem, since I'd really look like the new copy.) Thanks for the offer of help on the LJ side -- I put in some time a while ago, though, so I've got that side under control for the time being at least.

cortex: Just the last few days or so. I only noticed yesterday. Ego-cripping, nail-biting indeed! I can't imagine how this feels for people whose sites are their livelihood instead of their vanity. I probably do just need to wait it out for now.

Of course, should any potential problems jump out at anyone I'd love to hear about them.
posted by mendel at 8:23 PM on March 30, 2007

Quick update: Still only slowly reappearing. I did notice in Google Webmaster Tools that my top-pageranked URL for April has a PHPSESSID in it! Well, damn. I've mod_rewrite'd those out now, hopefully that'll help a bit too.

(You want bizarre, though? Searching for "rich lafferty" returns the champion off-road motorcyclist first. Searching for "rich lafferty motorcycle" gets my blog!)
posted by mendel at 6:02 PM on April 9, 2007

And now as mysteriously as I disappeared, I'm back up at #1. A quick list of everything I did, for those who come along to read this question later:

• Submitted sitemaps to Google
• Blocked Google's access to all of the related sites I had that might look like duplicate content via robots.txt
• Added small Google Adsense ads to my blog pages, on the hunch that Google would prefer the Adsense-displaying copy of duplicate content to the non-Adsense one
• Fixed up the page title to include my name, which I had embarrassingly neglected to do
• Removed the PHPSESSID from URLs, so Google would see one page instead of hundreds of identical ones with different session IDs

I'm pretty much convinced that it was primarily the "duplicate content" penalty, magnified by the site being redone and thus needing re-indexing. Since my LiveJournal was already high-ranking, and since it looked like there were thousands of copies of my LiveJournal's posts on my new blog, Google must have decided I was a splog stealing someone else's content.

Now I just have to hope I stay there!
posted by mendel at 8:12 AM on April 10, 2007

Oh, one other thing I realized: For ages I've been filling out comment forms on other blogs as "rich", with the URL of my LiveJournal. I've switched to filling them out as "Rich Lafferty" with the URL of my own domain. A minor thing, and I suspect most of them nofollow that anyhow, but couldn't hurt to get the last name in there, I figure.
posted by mendel at 8:25 AM on April 10, 2007