Who watched the Search Engine Watchmen?

May 3, 2011 1:33 PM

A friend's site disappeared from google 3 days ago. Totally, completely, utterly gone. What can she do to get it back on google?

For a friend:

I have a friend who has had a actual, proper, well-designed, relevant, dot-com website that is located at HERACTUALNAME.com since 1997. She has a huge link network, meaning that she has traded links and has been linked to by others, that she has built over a long time. She is known in her community and is generally what I would call a respected subject matter expert in her field. her site is regularly updated and follows google design guidelines, always has. I can not see any google penalties.

As of sunday, searching for HERACTUALNAME reveals everything EXCEPT her own website, and no mention anywhere of HERACTUALNAME.com appears in any of the search results.

Before Sunday, seaching for HERACTUALNAME always had HERACTUALNAME.com as the first result, and it had been for a decade.

Now, google acts as if her website and URL never existed.

What could have happened?
Who can she contact?
Can someone have done this maliciously?

Thanks in advance!

posted by sandra_s to computers & internet (31 answers total)

Google often changes/tweaks its search algorithms from time to time which affect their search results. Some people gain, some lose. :T
posted by pinksoftsoap at 1:47 PM on May 3, 2011

She should inspect her code for malicious code that may have been inserted or spam that has been included on any pages. Google has been pushing lately to remove the harmful and spammy sites from the search results.
posted by cp7 at 1:52 PM on May 3, 2011 [1 favorite]

An additional data point which may help people diagnose your problem: What blog or content management software is your friend running? Is there the possibility that, without her knowledge, someone has introduced malicious code that's inserting invisible search engine gaming or other nasty stuff into pages that she's not aware of?

These behaviors may cause Google to filter a site out of search results.
posted by straw at 1:53 PM on May 3, 2011

Google might have removed the url because it found malicious code on there. This should show when trying to visit the site in Chrome.

Does she have a webmaster tools account at Google for her url? If not she could get one and see what it says about the site.

This happened to me around 2005/6 when my site literally disappeared. I emailed over and over again, replied to all the automated replies and a few months later my site re-appeared. There was no reason ever given or for it. I wouldn't waste a second looking for help in Google Groups though.
posted by episodic at 1:53 PM on May 3, 2011

"huge link network" sounds like she may be participating in a link farm, which is a major no-no for Google.
posted by adamrice at 1:59 PM on May 3, 2011 [1 favorite]

@straw - Manually coded simple HTML site.

@episodic - opens fine in Chrome, no warnings.
posted by sandra_s at 2:00 PM on May 3, 2011

@asamrice - no, it's not a link farm in any way. That means that actual people actually link to her.
posted by sandra_s at 2:06 PM on May 3, 2011

"huge link network" sounds like she may be participating in a link farm, which is a major no-no for Google.

Or it may look like a link farm to Google's heuristics, even if your friend doesn't think it is.

Manually coded simple HTML site.

Read the raw site source to see if there's anything unusual in there. And in particular, closely examine any foreign objects included e.g. trackers, advertising, and so on.
posted by Chocolate Pickle at 2:07 PM on May 3, 2011

Google on link schemes:

However, some webmasters engage in link exchange schemes and build partner pages exclusively for the sake of cross-linking, disregarding the quality of the links, the sources, and the long-term impact it will have on their sites. This is in violation of Google's Webmaster Guidelines and can negatively impact your site's ranking in search results. Examples of link schemes can include... Excessive reciprocal links or excessive link exchanging ("Link to me and I'll link to you.")

Your friend should sign up for Webmaster Tools so she can see if there are any anomalies regarding her site. Then she can remove all the links from the exchange and request reconsideration of her page.

On preview:

it's not a link farm in any way. That means that actual people actually link to her.

Sounds like a link scheme to me. Doesn't need to involve any sort of automation. The point is that she is exchanging links for the point of exchanging links.
posted by grouse at 2:11 PM on May 3, 2011 [1 favorite]

First, is it gone (not indexed) or has it simply had its results order changed? You can do a google search like "site:example.com whatever"

Secondly, She needs to make an account with google's webmaster tools for more information. This will require her uploading a static HTML file for verification.

meaning that she has traded links and has been linked to by others

I'm guessing this is the real issue. Google is cracking down on search engine optimizers because its search results are borderline useless lately (hey another useless link to fixya, bigfix, and other content stealers with good SEO). This kind of "link to me, and i'll link to you so we up our google rating" is considered unethical by google and google will act on it.
posted by damn dirty ape at 2:18 PM on May 3, 2011

Ok, let me rephrase the link thing.

She doesn't have "traded" links. over 14 years people have linked to her because they like her content and feel she is relevant, she doesn't "Know" them. When I said she has a huge link structure, i just meant that she wasn't fly-by-night and was relevant.

Taken to the extreme, I'm sure many more, probably hundreds of thousands, people link to whitehouse.gov, but whitehouse.gov doesn't link back. Is whitehouse.gov a link farm?

See what i mean?
posted by sandra_s at 2:37 PM on May 3, 2011

Seconding Webmaster Tools (google.com/webmasters). Once you authenticate that you own a domain it will show you all the details about how her site is crawled and what Google found on there. You can also use it to submit a sitemap that helps Google know what's out there on your site. We recently had an update to our CRM system that slightly modified our sitemaps across the board. Using Webmaster tools I saw that we went from 3000 indexed pages on our domain, down to 1 (the main page) and then back up to 3000 after I resubmitted a sitemap. It's there to help you know what's happening behind the scenes as it were.
posted by msbutah at 2:48 PM on May 3, 2011

Register the site with Google webmaster tools. See if it reports back any errors or other issues. If so, fix whatever they tell you to fix. If everything looks OK, there's a link in webmaster tools you can use to submit a request to have your site reconsidered. Tell them what you told us. Then just wait. The site will likely pop back up to where it used to be within a month or two.
posted by spilon at 2:50 PM on May 3, 2011

But she has exchanged links, right? Hopefully you see the distinction, because Google would probably not filter her for being linked-to. It's the "exchange" part we're focusing in on here, and you mentioned it twice.
posted by rhizome at 2:51 PM on May 3, 2011

Webmaster tools is the correct first step in addressing this with Google. From there I believe she can find the appropriate contact links if things look wrong.

On the left of the home area, for example, I see a help link for requesting reconsideration of a site, but first she'd have to see what if anything is the problem.

It's also possible she accidentally blocked crawlers, which the tools site can show.
posted by wildcrdj at 2:53 PM on May 3, 2011 [2 favorites]

OK, I'll pass all the info along to her. Thank for the insight!

I gotta admit, the link thing makes no sense to me. But so I have it straight...

If I was a manufacturer (say Procter & Gamble) who sold my products through retail establishments (like Walmart, Kmart, Sears...) and I linked to those retail stores so people would know where to buy my products, google would de-index me if those retail stores linked back to me as a vendor so their customers would have more information on the products they sell?
posted by sandra_s at 3:00 PM on May 3, 2011

Does the site have an RSS feed? Mental Floss was removed from Google a few years ago due to some issue with their RSS feed being widely republished. Here's the story from Mental Floss and TechDirt.
posted by LaurenIpsum at 3:15 PM on May 3, 2011

If I was a manufacturer (say Procter & Gamble) who sold my products through retail establishments (like Walmart, Kmart, Sears...) and I linked to those retail stores so people would know where to buy my products, google would de-index me if those retail stores linked back to me as a vendor so their customers would have more information on the products they sell?

This is not the same thing as trading links.
posted by grouse at 3:16 PM on May 3, 2011

Never mind, I fail reading comprehension. Here's the story which involves the site having been hacked.
posted by LaurenIpsum at 3:18 PM on May 3, 2011

This sounds like what happened to a company that was profiled on NPR's morning edition today. Google updated their formulas on Feb24th and websites that may be viewed as content farms, even if they're not, have been dropped down google's rankings.

Googling "Panda Update" seems to have more information that could be helpful for your friend.
posted by saffry at 4:37 PM on May 3, 2011

If her site was removed because it was hacked by a spammer, she can email Google directly to get it restored once she's fixed the problem. If it's just that she has lots of links, she should try just emailing Google now to explain and get it restored. They're ultimately pretty good about that sort of thing.
posted by Eshkol at 4:43 PM on May 3, 2011

nth-ing webmaster tools. also look at y!'s version.
posted by kcm at 5:15 PM on May 3, 2011

It doesn't have to be "hacked by a spammer" in order to suffer demotion in the world of Google. It could simply be that enough of the sites in the link-exchange only ever link to themselves, or are (or have become) affiliiated with bad sites outside of this tea-party you've been told is going on. Google has been figuring out some of the associations that can be used to juice one's rank. Check the outbound links, do some housekeeping, and don't just assume everything is going to stay the same forever.

If I was a manufacturer (say Procter & Gamble) who sold my products through retail establishments (like Walmart, Kmart, Sears...) and I linked to those retail stores so people would know where to buy my products, google would de-index me if those retail stores linked back to me as a vendor so their customers would have more information on the products they sell?

No, and I think this is why these questions are more-easily asked by the actual person. It sounds to me like you're either being given conflicting information or they're not telling you something. If you link to a page on a domain that is suddenly starting to be used as a link farm or other odious purpose, your rank goes down and you don't get to rely on past status as a backup. It may have been Procter & Gamble before, but now it's ProctoGram and they don't care who doesn't want a proctological telegram.
posted by rhizome at 5:50 PM on May 3, 2011

Does she use blog software? It's very possible that she's been compromised and there are spammy sublinks that she doesn't even know about. I've absolutely, positively seen it happen, though I'm not at liberty to say where. But somewhere on the web, a regular dotcom now has links like

http://domainname.com/search/?q=statename-state-football-tickets-2010&images

which is an auto-generated link and the type of spam that Google is working hard to get rid of these days.
posted by bink at 5:52 PM on May 3, 2011

Well, not to "get rid of" so much as "to remove from their search results".
posted by bink at 5:54 PM on May 3, 2011

Even if she only hand-codes the HTML, spam links can be inserted on her web pages.

If her computer gets infected by a particular trojan, that trojan steals stored FTP passwords. It then uses the stored FTP passwords to log into the websites, and attach a little bit of its infected code onto the files it finds there.

(This happened to me last year, and it caused HUGE headaches. One lesson being: never save your FTP password - always type it in by hand each time.)
posted by ErikaB at 6:52 PM on May 3, 2011

I would suggest having her go through her entire file tree on the server and look for HTML files she didn't upload. My referrer logs lately have had a LOT of hotlinked images on sites which are clearly hacked. They don't mess with existing pages, they just add new HTML pages which are full of spam and other evil things, so they get the same pagerank and largely without the site owner knowing. For example, look at Google's "site" search for this example and this example. The sites' main pages (if you go to the main URL) look fine and aren't affected by the spam, but google has hundreds and hundreds of spam pages indexed in more hidden corners. I'm 90% certain that those spam pages weren't put there by the site owners -- the same format and style is common across numerous sites and has both adult and non-adult components. Seriously, I doubt this professor (SFW) intentionally added porn spam to his Biblical Israel website (NSFW).
posted by AzraelBrown at 7:29 PM on May 3, 2011

I know from personal experience that Google just made a major change. One particular post on my blog used to represent about 15% of my traffic, nearly all of it via Google. Recently (in the last month or so) nearly all of that stopped.

I don't really mind so much; it was a post I wasn't proud of, and on a few occasions I've considered deleting it just to make them all go away.

But I assume that Google just seriously revamped their rating algorithm, and that's why it stopped getting so many refers from them.
posted by Chocolate Pickle at 9:30 PM on May 3, 2011

It's already been mentioned here, but the only way to know Google's views of your site is by using the Webmaster Tools. They will answer your questions.
posted by anildash at 11:11 PM on May 3, 2011

Haven't seen this mentioned yet, so it might be worth a look. Depending on what tools she used to maintain the site, have her check the root folder for a file called robots.txt. This file, if configured a certain way, will disallow search engines from "spidering" the site.

You can check for this by simply going to http://(her actual name)/robots.txt
posted by samsara at 5:21 AM on May 4, 2011

A link farm is a page with a lot of links and very little content. If this is the front page of the actualname.com, google doesn't really care whether the intent is genuine or not, it is not a good result for them to give to people searching for some term.
posted by gjc at 6:22 AM on May 4, 2011