Why am I not getting indexed by Google?
December 9, 2009 10:38 PM Subscribe
Why am I not getting indexed by Google?
I've been running my site for a few months and I get about 20 or so hits a day. I've started running an automatic sitemap plugin, touched up robots.txt, submitted it to Google, signed up on Webmaster Tools, added a (non-illegal) SEO package to Wordpress (just to automate meta-tagging) and I still can't get indexed. At all. I've surfed the forums and I'm getting a lot of conflicting information.
Meanwhile, my friends who start tumblr/wordpress blogs get indexed almost immediately. Is it because I'm running it off my own domain? I read that Google indexes "large" sites faster so I guess that their individual *.tumblr.com and *.wordpress.com blogs are considered part of a much larger chunk of webspace?
My reconsideration request has been processed (as of today) and I noticed some hits from a Google IP on my hit logs (although not search-hits.)
Is there something obvious that I am missing? It is not that big a deal that I get indexed (I'm not selling anything,) but it's just really bugging the hell out of me on principle.
The website is in my profile if you need to see it for any reason.
I've been running my site for a few months and I get about 20 or so hits a day. I've started running an automatic sitemap plugin, touched up robots.txt, submitted it to Google, signed up on Webmaster Tools, added a (non-illegal) SEO package to Wordpress (just to automate meta-tagging) and I still can't get indexed. At all. I've surfed the forums and I'm getting a lot of conflicting information.
Meanwhile, my friends who start tumblr/wordpress blogs get indexed almost immediately. Is it because I'm running it off my own domain? I read that Google indexes "large" sites faster so I guess that their individual *.tumblr.com and *.wordpress.com blogs are considered part of a much larger chunk of webspace?
My reconsideration request has been processed (as of today) and I noticed some hits from a Google IP on my hit logs (although not search-hits.)
Is there something obvious that I am missing? It is not that big a deal that I get indexed (I'm not selling anything,) but it's just really bugging the hell out of me on principle.
The website is in my profile if you need to see it for any reason.
Response by poster: As far as I can figure out, being Sandboxed means that I'm just ranked v. v. low. In that case, shouldn't doing a search for site:mywebsite.com pull up something instead of nothing?
posted by griphus at 11:06 PM on December 9, 2009
posted by griphus at 11:06 PM on December 9, 2009
This is somewhat strange. What's the history of the domain? Have you owned it for a long time? Is there any chance it was being used by someone else before?
You're showing a page rank of N/A which is effectively 0, meaning you haven't been indexed yet, they don't have enough information to rank you yet, or you're banned.
While I've seen it take several months to get a page rank, that didn't keep me from being indexable pretty early on. If no one is linking inbound to you, that might affect things, but having submitted to webmaster tools should help a lot--and those tools can tell you things like how often you're being indexed, etc. Why did you submit a reconsideration request? Was there a flag on the tools that indicated you needed to?
posted by disillusioned at 11:09 PM on December 9, 2009
You're showing a page rank of N/A which is effectively 0, meaning you haven't been indexed yet, they don't have enough information to rank you yet, or you're banned.
While I've seen it take several months to get a page rank, that didn't keep me from being indexable pretty early on. If no one is linking inbound to you, that might affect things, but having submitted to webmaster tools should help a lot--and those tools can tell you things like how often you're being indexed, etc. Why did you submit a reconsideration request? Was there a flag on the tools that indicated you needed to?
posted by disillusioned at 11:09 PM on December 9, 2009
Response by poster: I've (well, the guy who hosts my site) owned the domain for quite a while (5+ years at this point,) although it has been recently re-registered. As far as I can tell, I'm the only one who has ever owned it. I've got a number of inbound links, including a few from some (moderately) popular websites. Most of my public social network profiles (twitter, last.fm, etc.) link to the site as well, and Googling the domain name brings up the actual social network profiles.
I submitted a reconsideration request because I just found it very odd that I wasn't getting indexed and thought I had done something wrong.
posted by griphus at 11:15 PM on December 9, 2009
I submitted a reconsideration request because I just found it very odd that I wasn't getting indexed and thought I had done something wrong.
posted by griphus at 11:15 PM on December 9, 2009
I got delisted before because, unbeknownst to me, my Wordpress install had been hijacked to include a particularly devious script that would redirect users away to a malicious site--but only if they arrived via Google or MSN or Yahoo!, the brilliant side effect being that it took me months to notice, since I never visit my own website from any of those sources.
Worse, the JavaScript that did this was encoded in such a manner that it was simple encoded text, so it wasn't readily apparent what it was doing or how it was doing it, but I decoded it and determined what was up.
Looking through your source code, I don't see anything similar, but it's definitely strange you haven't been picked up yet and don't appear in the index at all—something's definitely wrong and they've red-flagged you for one reason or another.
(Counter-example: I just launched a new domain and new site here that's already indexed quite highly after less than two months. More worrying, a search for your full domain yields nothing which means you've been forcibly stripped/blacklisted from the index.)
Wait for reconsideration to take effect and to see what the response comes back as--mine took about another month or so to restore my page rank after Google reindexed and determined I wasn't hijacking people away.
Good luck!
posted by disillusioned at 12:42 AM on December 10, 2009
Worse, the JavaScript that did this was encoded in such a manner that it was simple encoded text, so it wasn't readily apparent what it was doing or how it was doing it, but I decoded it and determined what was up.
Looking through your source code, I don't see anything similar, but it's definitely strange you haven't been picked up yet and don't appear in the index at all—something's definitely wrong and they've red-flagged you for one reason or another.
(Counter-example: I just launched a new domain and new site here that's already indexed quite highly after less than two months. More worrying, a search for your full domain yields nothing which means you've been forcibly stripped/blacklisted from the index.)
Wait for reconsideration to take effect and to see what the response comes back as--mine took about another month or so to restore my page rank after Google reindexed and determined I wasn't hijacking people away.
Good luck!
posted by disillusioned at 12:42 AM on December 10, 2009
It's all about the Google Webmaster Tools. My site is linked to by just about nothing, but it still is fully indexed by Google because I uploaded my sitemap.
posted by The Devil Tesla at 1:47 AM on December 10, 2009
posted by The Devil Tesla at 1:47 AM on December 10, 2009
You didn't say specifically you had installed google analytics code on your site. You said you had an automatic sitemap plugin, which is good and I think that's what you mean.
One thing you can do is go and build some backlinks. Surf to a bunch of social bookmarking sites and make an entry for your site.
It's not so much that it's going to overwhelmingly change your site ranking, but it's another quality score which can make a difference.
stumbleupon
reddit
squiddoo
del.cio.us
and so forth. There's a bunch of them. Just a matter of creating an account, and making a link to your site. There's software and websites that do mass backlink creation. Better if you do this over 2-3 days or a week.
Are you using All-in-One SEO plugin. Another very useful utility for pinpointing meta tags.
posted by diode at 2:56 AM on December 10, 2009
One thing you can do is go and build some backlinks. Surf to a bunch of social bookmarking sites and make an entry for your site.
It's not so much that it's going to overwhelmingly change your site ranking, but it's another quality score which can make a difference.
stumbleupon
squiddoo
del.cio.us
and so forth. There's a bunch of them. Just a matter of creating an account, and making a link to your site. There's software and websites that do mass backlink creation. Better if you do this over 2-3 days or a week.
Are you using All-in-One SEO plugin. Another very useful utility for pinpointing meta tags.
posted by diode at 2:56 AM on December 10, 2009
I don't have an answer, but I'm in a similar boat: I've owned the domain name for around 11 years now, PageRank of 4, linked to by various outside websites, but my inbound referrers from Google has slowly dwindled to practically none (and the only things that do come in are to three or four pages that have been static for 5 years).
The most probable thing I found which could be causing the trouble was badly-formed HTML -- didn't close my </head> tag properly, <style> info appeared below the <body> tag (the side-effects of ten years of tweaking), none of which affected how the page displayed in a browser, but it may have been damaging search engine's ability to spider the site. Last week I signed up for Google Webmaster Tools and Analytics, then fixed my HTML and uploaded a sitemap, and put the Analytics code on all the pages. Through the Webmaster tools, I found that I interchangeably use the "www." prefix; the non-www domain name itself redirects with a 301 to the same page on the "www." -- but my RSS links were all set to the non-"www." URL. Again, browsers handle it fine, but I think Google was stopping at the 301 and not redirecting. Anyhow, I don't know that any of this will fix your issue, but Google can be a fickle mistress, and it might not be the obvious thing that's causing you grief.
posted by AzraelBrown at 5:41 AM on December 10, 2009
The most probable thing I found which could be causing the trouble was badly-formed HTML -- didn't close my </head> tag properly, <style> info appeared below the <body> tag (the side-effects of ten years of tweaking), none of which affected how the page displayed in a browser, but it may have been damaging search engine's ability to spider the site. Last week I signed up for Google Webmaster Tools and Analytics, then fixed my HTML and uploaded a sitemap, and put the Analytics code on all the pages. Through the Webmaster tools, I found that I interchangeably use the "www." prefix; the non-www domain name itself redirects with a 301 to the same page on the "www." -- but my RSS links were all set to the non-"www." URL. Again, browsers handle it fine, but I think Google was stopping at the 301 and not redirecting. Anyhow, I don't know that any of this will fix your issue, but Google can be a fickle mistress, and it might not be the obvious thing that's causing you grief.
posted by AzraelBrown at 5:41 AM on December 10, 2009
Go to Google Webmaster Tools and check Site Configuration / Crawler Access. That will tell you if Google can see your sitemap and robots.txt. Possibly they can't.
Some hosts lock out crawlers during traffic spikes and forget to re-permit them. This happened to me a couple months ago -- Google was indexing nothing. I sent a note to the admins and it's okay now.
posted by futility closet at 5:48 AM on December 10, 2009
Some hosts lock out crawlers during traffic spikes and forget to re-permit them. This happened to me a couple months ago -- Google was indexing nothing. I sent a note to the admins and it's okay now.
posted by futility closet at 5:48 AM on December 10, 2009
At one point when I upgraded Wordpress, it changed the defaults to "I would like to block search engines" under privacy. I'd double-check those settings to be sure.
posted by divka at 9:46 AM on December 10, 2009
posted by divka at 9:46 AM on December 10, 2009
Response by poster: @disillusioned Yeah, this is just a basic Wordpress install with high-rated plugins installed from the source.
@The Devil Tesla Yep, I have it all set up.
@diode I did the "throw a webpage on your server and then have us see it" thing through Webmaster and it saw it. I link all my social network accounts to it, and they are what pop up when I do Google searches.
@AzraelBrown The HTML checks out, although it is just an unmodded WP install so that is no surprise. I have both mywebsite.com and www.mywebsite.com hooked into Webmaster as well.
@futility closet Yep, it crawls it regularly and there's no errors w/ robots.txt and it reads my sitemap regularly and updates it as it should.
@divka I checked the setting and it is on the public mode.
posted by griphus at 2:03 PM on December 10, 2009
@The Devil Tesla Yep, I have it all set up.
@diode I did the "throw a webpage on your server and then have us see it" thing through Webmaster and it saw it. I link all my social network accounts to it, and they are what pop up when I do Google searches.
@AzraelBrown The HTML checks out, although it is just an unmodded WP install so that is no surprise. I have both mywebsite.com and www.mywebsite.com hooked into Webmaster as well.
@futility closet Yep, it crawls it regularly and there's no errors w/ robots.txt and it reads my sitemap regularly and updates it as it should.
@divka I checked the setting and it is on the public mode.
posted by griphus at 2:03 PM on December 10, 2009
Response by poster: I just got my first Google hit! I guess it was just inadvertently blacklisted and the re-indexing request came through.
posted by griphus at 2:37 PM on December 17, 2009
posted by griphus at 2:37 PM on December 17, 2009
« Older Lyrics to the song Remembrance | I want to make Asian noodles from scratch. Can you... Newer »
This thread is closed to new comments.
posted by willnot at 11:02 PM on December 9, 2009