Search Engine Optimization?

June 23, 2004 11:46 AM

My company retained a consulting firm to improve our search engine placement, but our web site is no longer showing up in Google searches that it definitely should. I believe the consulting firm screwed up royally by adding this line of HTML to our home page:

[the more, she is inside]

My reasoning is this: the robots tag I gave does not follow proper syntax. Since the default behavior of robots is to index and follow, robots tags are unnecessary unless one wants to prevent the page in question from being indexed/followed. Therefore, if a robot encounters a robots tag that it can't understand, it will assume that the intent was to exclude the page and act accordingly.

I'm a little confused, though. A Google search of the line in question shows it to be somewhat common, although I get the impression that it is preferable to use ...content="all,index,follow" or ...content="index,follow". I'd just like a definitive answer from AskMe as to whether or not I should tear these consultants a new one over this.

posted by alphanerd to computers & internet (17 answers total)

SEO consultanting is a dark business that is frequently done by hucksters. The only good information about improving your rank is easily found and freely given out online.

If your google ranking went down for something you're paying them to make better, they've completely fucked up the job in my opinion. Imagine paying a contractor to build a new deck in your backyard and coming home to a giant hole in the ground instead.

Ask them for your money back.
posted by mathowie at 11:57 AM on June 23, 2004

ARGH!!! Here is the line:
(less than sign)meta name="robots" content="all=index,follow"(greater than sign)
posted by alphanerd at 11:57 AM on June 23, 2004

Matt: I agree wholeheartedly. I think they really screwed us over on this. They also signed us up for a shady link exchange with web pages such as this one [Warning: Ass Crack] with sites whose sole purpose is to tamper with Google results. And they greenlighted courier new as the site font (which was admittedly put in place at the initiative of someone on our end but should never have been approved), and also approved a final web site which was an aesthetic disaster. I'm livid.
posted by alphanerd at 12:07 PM on June 23, 2004

Sounds like Google has banned you. They are pretty diligent about banning the sites that participate in what they consider spamdexing. There are a lot of shady ways to try and fool google, and she knows about all of them! And once she bans you, you basically have to sell her a kidney to get back in her good graces.
posted by archimago at 12:20 PM on June 23, 2004

Yeah, it sounds like you've been banned due to links to your site being found on obvious link index spam pages like that ass crack one you just showed.

I would tell your company to consider litigation against the firm, as they're in the business of improving your rank and they likely got you banned from Google entirely, which will definitely hurt your business. I bet you can demonstrate lost income pretty easily in a court of law if no one can find your site anymore.
posted by mathowie at 12:35 PM on June 23, 2004

MetaFilter: [Warning: Ass Crack]
posted by kirkaracha at 12:37 PM on June 23, 2004

What #1 said, with some additional comments and stuff I've picked up along the way (though with a nick like alphanerd, you probably know most of this already):

-You don't mention how long your site has been out of search results, but Google has been seeing some search result changes thanks to a Page Rank/Backlink Update as of yesterday. I haven't checked the various forums in the past few hours but it still wasn't settled as of 5 hours ago.

-That metatag does seem weird 1, 2; with a decent stats program you should be able to monitor robot activity on your site and see if it's dropped off any (assuming you have access to the web stats).

-Content, content, content. Google loves content. And frankly, I would say it's even more important for your site's users.

-Above content should be marked up in valid HTML. Even better is separating your content from the style with CSS. I noticed much nicer search engine results when I switched from a liquid table layout to a properly structured HTML doc using CSS for layout.

-Links are important, but link farms, doorway pages, etc tend to be bad news.

You don't say what kind of contract (if any) you all have with this firm, but if this isn't some temporary Google burp, I echo the "get your money back" statement.

On preview: bannination can be determined if you install the Google Toolbar and look at the PageRank icon: grey tends to mean either bannination or IIRC that Google hasn't indexed that page yet.

0 page rank means that you're at least in the Google index, if not heavily rated. As I said before, Google's playing with their PageRanks right now. Three of my sites went up and a subdomain of one of those has dropped dramatically, though I'm not that concerned about it.
posted by romakimmy at 12:46 PM on June 23, 2004

Any advice on how to determine whether this is a consequence of a bad robots tag or of Google banning? It seems like the rank check can tell me that I'm not in the index, but not why. I know I could wait a few weeks and give Googlebot time to come back to the page, but it would be nice to know sooner. FWIW, it looks like the sites that are part of this link exchange program do not turn up as Google search results either, which is good from the perspective of everone else on the internet but bad from that of my company.
posted by alphanerd at 1:43 PM on June 23, 2004

PS: I don't know exactly how long we've been out of the results, but it was noticed today. I guess I could dig through the site's logs and figure out when we stopped getting referrals from Google. My manager has also noticed a downtick in orders at our online store in recent weeks, and the offending file with the bad meta tag (which also contained the first of many shady links which I discovered from our site) was last updated on June 11th, although I'm not sure what the change was or who did it. This is complicated due to the fact that (1) management at my company is not tremendously web-savvy, (2) I'm the only programmer working full time and the web site has been handled by part timers who have little to no interaction with me.
posted by alphanerd at 1:49 PM on June 23, 2004

Are you getting a grey bar on the Google toolbar for your index page? If you were previously getting referrals from Google, you were in the index and had some level of Page Rank; if you now have a grey bar it's most likely that you've been banned. As I said before, if you can, check on the Google robot activity in the past few months. It's not an acurate measure of damage by any means, but you might be able to see if Googlebot visits have slacked off noticably since June 11th.

Also handy is the Google News section at WebmasterWorld; those folks might be able to help you pinpoint the source of the problem/determine bannination status.

But until you can narrow down at which point in the past you started dropping out of search results, I wouldn't freak out just yet. That Page Rank update isn't over yet from the looks of it and it's a bit hinky. The site of mine that went down in PR has now had all internal pages go up.
posted by romakimmy at 2:27 PM on June 23, 2004

Update: It appears the last time we got a visitor via a Google search was yesterday at around 1 pm eastern time. We also have gotten people who clicked through on Google ads, and there was a visit from "Googlebot/2.1 (+http://www.google.com/bot.html)" as recently as this morning at around 7:00 AM eastern, though not to our main page. It also looked for a robots.txt file and did not find it at a little before 7:00 AM eastern today.

It has also been looking for a lot of pages from our database of specifications and getting 404s. Could this be part of the problem?
posted by alphanerd at 3:45 PM on June 23, 2004

Also, I don't know whether the Google Toolbar Test is conclusive. I tried doing this search:

site:www.konabuilder.net recycling

and got no results whatsoever, despite the fact that "recycling" occurs several times on the front page of the web site. The site does not have any robots or googlebot meta tags. From this I conclude that Google has chosen not to index this website (with good cause), yet it has a page rank of 0/10.
posted by alphanerd at 4:12 PM on June 23, 2004

Any advice on how to determine whether this is a consequence of a bad robots tag or of Google banning?

You could always try asking Google.

I think roma and matt nailed it.
posted by anathema at 4:32 PM on June 23, 2004

This is obvious perhaps, but if you're going to go after the 'consultants' I hope you're taking mucho screenshots of the proceedings...
posted by stavrosthewonderchicken at 5:40 PM on June 23, 2004

site:www.konabuilder.net recycling

Sorry I wasn't completely clear on indexing. If Google knows of page foo.html, it's been indexed. That's it. It doesn't mean that page foo.html will show up in results for a certain term,at least until Google updates it's index (formerly known as Google dances, but lately Google seems to be tweaking on an almost daily basis)

Caveat emptor: Google also serves up results based on IP. You can check various datacenters here; you can also use this to check PR, though I prefer the toolbar.

It has also been looking for a lot of pages from our database of specifications and getting 404s. Could this be part of the problem?

Yuck. It certainly isn't helping matters. Can you do some temp/permanent redirects?

Also, straight from the horse's mouth: Google gets cranky with shady SEO's
posted by romakimmy at 5:38 AM on June 24, 2004

Hi alphanerd,

I've had to get familiar with the "SEO Industry" in my current job, and I'd like to add my voice to the crowd of folks who think it's a shaaady, shady business. (Did these consultants want all their money upfront?)

In your case, I'd guess that while the robots.txt snafu might have contributed to google's difficulty in indexing your site, your site's association with konabuilder.net (and probably many other such "content and links" sites) is most likely the reason you've fallen from the index.

I did a google search for related:www.konabuilder.net and got a list of sites that all use deceptive redirects and landing pages (the technique they use is a signature of the disreputable trafficpower.com) so it seems likely that you've surrounded yourself with some bad company in google's eyes.

As for a suggestion as to how to proceed from here (beyond getting your money back), here's the recommended course of action from "GoogleGuy", who is fairly well trusted to be a software engineer at G:

The preferred way to write would be to email help [at] google.com. If you suspect that some staff member did spam on your site, or you get back the email from Google that says we were unable to assign PageRank to your site, then I would play it safe and write to webmaster [at] google.com with the subject line "reinclusion request." Describe your situation (when did this happen? Is your site completely gone or just not ranking how you expect? Did you do anything like hidden text, and what steps have you taken since, etc.

Good luck!
posted by lpqboy at 10:23 AM on June 24, 2004

Ipqboy: Thanks a lot for your advice. The shady SEO firm is indeed Traffic Power. I have already emailed google help, and just sort of posed the question innocently in hopes that they will say something to the effect of, "your company was banned for x, y, and z", and we can then use the email as leverage to recover money from Traffic Power, possibly in court.

Traffic Power has engaged in the following techniques, all of which are either explicitly mentioned in what NOT to do at search engines or are obvious enough to avoid operating under the assumption that search engines will eventually figure it out:

-Set us up with a link exchange program to doorway sites, hiding the links from view on our index page.
-Put up over a hundred or so "cloaked" pages that redirect the viewer to our main page on mouseover but contain ostensibly search engine friendly content.
-Provided a "friends" list on our web site to companies that are not in any appreciable way related to the industry that we are in.

Additionally, the overwhelmingly vast majority of sites linked to at the behest of Traffic Power are not showing up on Google searches.

Traffic Power has also been evasive with respect to my growingly aggressive attempts at resolving this. The gentleman I spoke to yesterday attributed it to a problem with Google that they were supposedly keeping quiet about because they were "sitting on an IPO" and directed me to an article at searchenginewatch.com (he sent in an email and it was over a month old, so I don't think I can provide a link) that mentioned an issue with Google's site removal feature that only affected ten or so sites, and was resolved as of May 14.

When I asked their representative today about the issue, he at first claimed that they were still developing our campaign and that was probably why things weren't working out. When I pointed out that they had deployed a large number of web pages to our site on June 11th and that the obvious assumption was that it had already been rolled out, he asked to call me back and has not as of yet.

When I asked why trafficpower.com does not show up in Google results, he said that competitors were hacking it and that they were in the process of acquiring a bunch of new domain names. When I pointed out that trafficpower.com was registered in June of 2001, he tried to pawn me off to somebody else.

PS: I'd like to note that I haven't been a part of any of the Traffic Power decision making at my company, and my involvement in what they've done for us is pretty much restricted to my discovery and investigation of the problems outlined in this thread and cursory review of the changes they advised us to make with the layout of our website, which I had several strong objections to.
posted by alphanerd at 11:45 AM on June 24, 2004