That's a great answer, Google, but...
September 20, 2010 4:51 AM Subscribe
Does Google (or other search) really work, or do I just perceive that it does?
Has anyone done research to test the effectiveness of a Google search vs. the perceived effectiveness?
This may seem like an ridiculous question. Obviously, Google has value, but I'm curious if its perceived value is much higher than its actual value. Also, does its addictive quality of quickly returning to me what I want and expect to see reinforce its importance to me?
For example, a test of it would ask people to seek answers to some mundane to esoteric questions. Interesting info would be how long they searched, what they came up with, and how they decided when they were finished, and how convinced they were that they had the answer. It would be interesting to know how their results compared with an "expert" answer.
And, I don't mean Google, specifically, but search engines in general.
Has anyone done research to test the effectiveness of a Google search vs. the perceived effectiveness?
This may seem like an ridiculous question. Obviously, Google has value, but I'm curious if its perceived value is much higher than its actual value. Also, does its addictive quality of quickly returning to me what I want and expect to see reinforce its importance to me?
For example, a test of it would ask people to seek answers to some mundane to esoteric questions. Interesting info would be how long they searched, what they came up with, and how they decided when they were finished, and how convinced they were that they had the answer. It would be interesting to know how their results compared with an "expert" answer.
And, I don't mean Google, specifically, but search engines in general.
Google search is a tool, and as with any tool, the quality of the results depends as much on the skill and knowledge of the person using it.
To give you a concrete example: I work as a technical translator, and I use Google constantly in the course of my work to look up the meaning of terms--there are a ton of resources to help out the translator with good Google-fu: not only standard online dictionaries but also specialized glossaries, print dictionaries through Google books (including technical dictionaries that cost several hundreds of dollars), bilingual user manuals, engineering texts, etc. etc.
But you need to know how to use these sources to find what you need, and you need to have a general familiarity with the subject to know if you're in the right ball park, and you need to double-check everything with more Google searching of the translated term you find to make sure it's actually used in the "target" language and isn't some BS "translatorese" from a bad translation.
I actively participate on translator web sites that have terminology help forums so that we can all help one another out with this tricky process. And I never cease to be dumbfounded by the number of paid professional translators who blindly use Google without a good understanding of the subject matter, or the target language, or both, and without knowing how or taking the time to refine and cross-check a search or interpret the results--and have the gall to suggest their "findings" to other professional translators as the definitive translation of term X!
posted by drlith at 5:18 AM on September 20, 2010 [1 favorite]
To give you a concrete example: I work as a technical translator, and I use Google constantly in the course of my work to look up the meaning of terms--there are a ton of resources to help out the translator with good Google-fu: not only standard online dictionaries but also specialized glossaries, print dictionaries through Google books (including technical dictionaries that cost several hundreds of dollars), bilingual user manuals, engineering texts, etc. etc.
But you need to know how to use these sources to find what you need, and you need to have a general familiarity with the subject to know if you're in the right ball park, and you need to double-check everything with more Google searching of the translated term you find to make sure it's actually used in the "target" language and isn't some BS "translatorese" from a bad translation.
I actively participate on translator web sites that have terminology help forums so that we can all help one another out with this tricky process. And I never cease to be dumbfounded by the number of paid professional translators who blindly use Google without a good understanding of the subject matter, or the target language, or both, and without knowing how or taking the time to refine and cross-check a search or interpret the results--and have the gall to suggest their "findings" to other professional translators as the definitive translation of term X!
posted by drlith at 5:18 AM on September 20, 2010 [1 favorite]
I seem to remember a study that used a Google skin over Yahoo (I think?) search results, and people rated it higher than a Yahoo skin over Google results, suggesting that perceived value might play a part. Can't remember the exact details though. Will try to track it down.
When studying for my MLIS I had a short assignment to compare G, Y! and Ask, and Ask produced the best results, which I wasn't expecting (this was 5 or 6 years ago, though).
posted by Infinite Jest at 5:20 AM on September 20, 2010
When studying for my MLIS I had a short assignment to compare G, Y! and Ask, and Ask produced the best results, which I wasn't expecting (this was 5 or 6 years ago, though).
posted by Infinite Jest at 5:20 AM on September 20, 2010
Before Google, searching was a royal pain, even for really simple things that we now take for granted.
Often I'd have to click through piles of results to find anything of any use. The search page I'd get would only quote the first sentence on each result page, so it was anyone's guess which results might be relevant. Sometimes I'd go clicking through one of the categorised "directories" of the Internet because I'd be more likely to find something useful that way than with a search engine. Often I would fail to find what I was looking for altogether, even though I was certain it was out there because I saw it last week.
This, all when the internet was many orders of magnitude smaller than it is now.
posted by emilyw at 5:27 AM on September 20, 2010
Often I'd have to click through piles of results to find anything of any use. The search page I'd get would only quote the first sentence on each result page, so it was anyone's guess which results might be relevant. Sometimes I'd go clicking through one of the categorised "directories" of the Internet because I'd be more likely to find something useful that way than with a search engine. Often I would fail to find what I was looking for altogether, even though I was certain it was out there because I saw it last week.
This, all when the internet was many orders of magnitude smaller than it is now.
posted by emilyw at 5:27 AM on September 20, 2010
I think Google and other search engines have participated in the TREC competitions in the past (and perhaps still are). TREC is run by NIST, a standards agency. I don't know the results of these competitions offhand though.
posted by jasonhong at 6:42 AM on September 20, 2010 [1 favorite]
posted by jasonhong at 6:42 AM on September 20, 2010 [1 favorite]
Good question, worth asking about anything whose value is hard to quantify!
But on this one, I'm with antiquark -- I think anyone who, like us, remembers switching from Altavista to Google knows that Google actually works.
posted by escabeche at 6:54 AM on September 20, 2010
But on this one, I'm with antiquark -- I think anyone who, like us, remembers switching from Altavista to Google knows that Google actually works.
posted by escabeche at 6:54 AM on September 20, 2010
The competency of the user is much more important than the tool. I wish I could find it, but someone did a study where they gave 7th (?) graders a list of five questions to look up, like 'what year did man first land on the moon' and so on -- stuff that most Metafiltarians would consider trivial to look up. I mean, you type 'moon landing', you click on the first result, and the answer is in the first paragraph of the page. But the results shocked the researchers, because the kids didn't really have any idea of what a search query meant, so they would just type something like 'facts' and then start clicking on things and just meandering for a while hoping to come across the answer by luck. This comic strip by a redditor has been making the rounds and demonstrates the problem. Here's a clip of Google doing street interviews and it's clear that these people don't even have a concept of what a browser or a search engine is. And finally let's not forget the time that a google for 'facebook login' went to somebody's blog post and people couldn't figure out that they were not on facebook.
In summary, quantifying the effectiveness of a search engine is as much about quantifying the computer literacy of the person using it as it is about the qualities of the search engine.
posted by Rhomboid at 6:56 AM on September 20, 2010 [5 favorites]
In summary, quantifying the effectiveness of a search engine is as much about quantifying the computer literacy of the person using it as it is about the qualities of the search engine.
posted by Rhomboid at 6:56 AM on September 20, 2010 [5 favorites]
What drlith said about knowing how to use a tool. One of the problems with Google is that it is word driven, but one word <> one concept.
Also, if you don't know the word for a concept things can get tough fast. For example: try to find the name for the process the immune system does to make quality tightly binding antibodies without already knowing the name of the immune system does to make quality tightly binding antibodies*. You'll be reading for a while.
In the end, a search engine can only be as good as the internet it's attached to. If everyone on Metafilter agreed to put the same bit of crap "knowledge" on line somewhere, pretty soon that would be the defacto answer from Google. And wrong.
*In case you're wondering, it's affinity maturation.
posted by Kid Charlemagne at 6:57 AM on September 20, 2010 [2 favorites]
Also, if you don't know the word for a concept things can get tough fast. For example: try to find the name for the process the immune system does to make quality tightly binding antibodies without already knowing the name of the immune system does to make quality tightly binding antibodies*. You'll be reading for a while.
In the end, a search engine can only be as good as the internet it's attached to. If everyone on Metafilter agreed to put the same bit of crap "knowledge" on line somewhere, pretty soon that would be the defacto answer from Google. And wrong.
*In case you're wondering, it's affinity maturation.
posted by Kid Charlemagne at 6:57 AM on September 20, 2010 [2 favorites]
Google is a company that does advertising and information retrieval. Accuracy is measured by computing the precision and recall of search results. The precision is the fraction of relevant documents matched divided by all documents retrieved; this is also the number of true positives over the sum of true positives and false positives. Recall is the fraction of relevant documents matched out of all relevant documents; or true positives divided by the sum of the true positives and false negatives. They are standard measures of accuracy in information retrieval systems, and higher scores are better. Google has been measured to have a precision of 0.29 and recall of 0.20 for domain-specific searches.
So yes, it works.
posted by procrastination at 6:58 AM on September 20, 2010 [4 favorites]
So yes, it works.
posted by procrastination at 6:58 AM on September 20, 2010 [4 favorites]
Google worked. I do not believe I am the only person who has noticed that it works less effectively now. (Or that the methods to make it work effectively have changed somewhat. For some reason, you now need to put quotes on both ends of a phrase for exact matches, which was irritating until I figured out that difference.)
The precision and recall stats above are 6 years old, and at the very least domain-specific search engines have improved significantly. (They are -- and should be -- much better than general search engines are.)
posted by jeather at 7:03 AM on September 20, 2010
The precision and recall stats above are 6 years old, and at the very least domain-specific search engines have improved significantly. (They are -- and should be -- much better than general search engines are.)
posted by jeather at 7:03 AM on September 20, 2010
One reason Google works as well as it does is that the internet has evolved into a more Google-friendly place.
For example: movies. If the web now was just a bigger version of the web in 1996 — full of disorganized fan sites on narrow, idiosyncratic topics — Google would be a terrible tool for finding information about movies. It's only because IMDB exists that you can google a few actors' names and a keyword and get what you want.
If you give Google all the credit, then yes, you're perceiving it as more effective than it really is. Some credit needs to go to the ecosystem of encyclopedic and well-organized sites that has sprung up around it. (Of course, to some extent the pagerank algorithm has encouraged the growth of that kind of site. It's a symbiotic thing.)
posted by nebulawindphone at 7:32 AM on September 20, 2010 [3 favorites]
For example: movies. If the web now was just a bigger version of the web in 1996 — full of disorganized fan sites on narrow, idiosyncratic topics — Google would be a terrible tool for finding information about movies. It's only because IMDB exists that you can google a few actors' names and a keyword and get what you want.
If you give Google all the credit, then yes, you're perceiving it as more effective than it really is. Some credit needs to go to the ecosystem of encyclopedic and well-organized sites that has sprung up around it. (Of course, to some extent the pagerank algorithm has encouraged the growth of that kind of site. It's a symbiotic thing.)
posted by nebulawindphone at 7:32 AM on September 20, 2010 [3 favorites]
Rhomboid says pretty much what I wanted to say re: the effectiveness of Google, but in addition -
For example, a test of it would ask people to seek answers to some mundane to esoteric questions.
I run a community on LiveJournal for helping fiction writers with factual information related to their work. One of the rules of the community is that you must at least attempt to Google for the answer to your question before posting it and that you must at least include some of your search terms in your post.
The community is here if you're curious.
We get all sorts of questions, from the mundane to the esoteric, and there are definitely some patterns to the types of questions that Google can't answer. So there's your test, done in a not-rigorous way.
For example, specific cultural information. It's relatively easy to find information about broad trends, but when you're looking for something specific--like the connotations of a name in a language you don't speak, or where people of x age in Podunksville hang out, that's just not something people put on the web in an easily Google-able form very often.
There are also patterns to people's failures. Some people get led astray by their initial assumptions. Recently someone asked a question about where a prisoner awaiting trial would be kept, and had spent a lot of time looking at state prison information. But what she was having trouble with is at least partially that the crime she described was going to be federal. Or, sometimes the information is there, but finding or understanding it requires extensive knowledge of a particular field.
Sometimes the information isn't actually there yet at all. It may exist in books, but no one has put it on the web yet.
posted by Kutsuwamushi at 8:10 AM on September 20, 2010 [1 favorite]
For example, a test of it would ask people to seek answers to some mundane to esoteric questions.
I run a community on LiveJournal for helping fiction writers with factual information related to their work. One of the rules of the community is that you must at least attempt to Google for the answer to your question before posting it and that you must at least include some of your search terms in your post.
The community is here if you're curious.
We get all sorts of questions, from the mundane to the esoteric, and there are definitely some patterns to the types of questions that Google can't answer. So there's your test, done in a not-rigorous way.
For example, specific cultural information. It's relatively easy to find information about broad trends, but when you're looking for something specific--like the connotations of a name in a language you don't speak, or where people of x age in Podunksville hang out, that's just not something people put on the web in an easily Google-able form very often.
There are also patterns to people's failures. Some people get led astray by their initial assumptions. Recently someone asked a question about where a prisoner awaiting trial would be kept, and had spent a lot of time looking at state prison information. But what she was having trouble with is at least partially that the crime she described was going to be federal. Or, sometimes the information is there, but finding or understanding it requires extensive knowledge of a particular field.
Sometimes the information isn't actually there yet at all. It may exist in books, but no one has put it on the web yet.
posted by Kutsuwamushi at 8:10 AM on September 20, 2010 [1 favorite]
The dark side of what nebularwindphone is saying (and what jeather is saying) is that when Google achieved search engine dominance, it became the search engine everyone wants to game.
The web is an eco system. We tend to be pretty isolated from the state of nature and equate "all natural" with "full of goodness and health" rather than "will try to kill and eat you". Google is soaking in it. There are probably tons of things going on behind the scene that make Google less effective but keep your search results for, oh, let's say product reviews for rechargeable drills from being nothing but untargeted marketing and blatant attention whoring.
posted by Kid Charlemagne at 8:11 AM on September 20, 2010 [1 favorite]
The web is an eco system. We tend to be pretty isolated from the state of nature and equate "all natural" with "full of goodness and health" rather than "will try to kill and eat you". Google is soaking in it. There are probably tons of things going on behind the scene that make Google less effective but keep your search results for, oh, let's say product reviews for rechargeable drills from being nothing but untargeted marketing and blatant attention whoring.
posted by Kid Charlemagne at 8:11 AM on September 20, 2010 [1 favorite]
Kid C: for fun I googled "process the immune system does to make quality tightly binding antibodies"
The first paragraph (intro, not abstract) of the second hit contains:
"The adaptive immune
response evolves immune receptors through recombination from
a limited, but still substantial, arsenal of germ line precursors
that are then optimized by class switching and affinity matura-
tion."
posted by a robot made out of meat at 10:14 AM on September 20, 2010
The first paragraph (intro, not abstract) of the second hit contains:
"The adaptive immune
response evolves immune receptors through recombination from
a limited, but still substantial, arsenal of germ line precursors
that are then optimized by class switching and affinity matura-
tion."
posted by a robot made out of meat at 10:14 AM on September 20, 2010
... and now this page is the 4th hit for "tightly binding antibodies." :-)
posted by callmejay at 11:29 AM on September 20, 2010 [1 favorite]
posted by callmejay at 11:29 AM on September 20, 2010 [1 favorite]
It's a question of Information Literacy. Any other librarians/info pros want to comment?
posted by KMH at 7:13 AM on September 21, 2010
posted by KMH at 7:13 AM on September 21, 2010
This thread is closed to new comments.
I had previously been using various search engines (altavista was the main one I used, I think), and then heard about google. I used it for a while, and was amazed at how well it worked.
Altavista, and others, kinda worked, but you usually manually sifted through a few pages of results, looking for what might be relevant. Google's first link was (and is) usually the one you wanted.
I remember my reaction well, as I emailed Google at the time to thank them for existing, with the details about.
posted by antiquark at 5:09 AM on September 20, 2010