Opinion polls

April 12, 2005 2:40 AM

How do opinion polls work? You can't be forced to answer questions, so you must choose to take part in the poll - doesn't this make the sample self selecting, and therefore useless?
posted by Orange Goblin to human relations (13 answers total)

It doesn't make it useless, but it does make it a non-probability sample.
posted by Jairus at 2:42 AM on April 12, 2005

Not just opinion polls, but lots of research is done on people who choose to be studied. That's why so much time and effort goes in to designing polls/studies, but even the best studies in the world are only really reliable when they have been repeated many times across many boundaries.

Opinions can change quickly and vary greatly, so they are difficult to follow with accuracy, but that doesn't make the data useless. For instance, if one is polling regarding the views of a political candidate, a general trend is all the pollsters are looking for. Those who vote regularly are considered more willing to report their opinions, so the sample should provide data reasonably suitable for the original purpose.

This doesn't work for things that start out with, "Americans believe...." No, we don't, that's just silly and there's no good way to measure it. Notice the language when the results are reported, you'll find it interesting if you know what to look for.
posted by spaghetti at 3:31 AM on April 12, 2005

I'm not sure if they're useless, but whenever I hear words such as "We polled a sample of 18-30 year olds..." I always mentally add on "...who have the time and inclination to answer questions about their personal beliefs to a complete stranger."
If you do that, a lot of these ridiculous poll outcomes, such as "50% of Americans think that a Haggis is a real creature that you can hunt in Scotchland" make a lot more sense.
posted by chill at 3:34 AM on April 12, 2005

If you have a list of one million phone numbers, at least a quarter of them will want to talk to you and give you their opinion on whatever it is you are calling about, be it George W. Bush or bubble gum.
It's hardly scientific, it's a numbers game.
And as talk radio has proven, everyone has an opinion, and I assume the polls are about as scientific as those on websites.
posted by willmize at 3:46 AM on April 12, 2005

My friends and I are scientists, and discussed this at length one day.

Opinion polls are mostly worthless. Just taking a quick survey among ourselves, we quickly discovered we're all pathalogical liars on polls.

Who did you vote for? We always give the fringe candidate names.

While we are a small sample, the fact that we all lie on polls would lead me to believe that there are others out there who do the same. Certainly I don't think it's a stretch to believe any poll which asks about embarrasing or illegal activities will have a much higher percentage of error than a poll about say, your favorite color.

Anyway, I'm with the OG. Polls are stupid. I've always believed that simply exist just to change the opinions of weak minded people. What 80% of people out there love orange? Why, I must like orange as well.

KFJ
posted by kungfujoe at 4:23 AM on April 12, 2005

Modern polling is the water dowsing of this new millenium. However they dress up their augery, the simple fact remains that if you want to know what 10 people think -- you gotta ask 10 people.

I've seen a national polling organization (I won't plug them) hired by my University to ascertain data about employee satisfaction -- I was compelled to participate in this charade -- and it was very evident that they were merely feeding the university back the very answers they wanted to hear, sprinkled with a dose of simple-minded common-sense platitudes (in case, no doubt, the university wanted to hire their "consulting" expertise for a small extra fee). Classic snake oil, no different than a John Edwards cold reading of your dead Aunt.

How do they get away with it? Easy. Insecurity will convince mediocre managers to try anything. Particularly if they can us it to shift blame for their own policies to something else or justify some boneheaded endeavor by pointing to the "scientific" data.

(A great example of the utter fear that compells managers to grasp at and perpetuate any fantasy in order to preserve the illusion that they have discovered a magic formula for reading the mind of the public is the curious Hollywood belief that hit movies require big-name stars with mega-salaries. This has been proven wrong soooo-oooo many times that it doesn't even deserve a moment's thought -- yet just try to put a major film together; the very first question the studios and VC men will ask is: "Which stars are attached?")

But I digress ...
posted by RavinDave at 4:59 AM on April 12, 2005

I would just like to mention that Social Statistics, as a science, recognizes and accounts for these issues that kungfujoe and RavinDave cite as reasons why polling is useless. Nonrepresentative samples, lying, etc. They know people do it. A number of opinion polls are conducted where the pollers in fact don't care at all about the questions they're asking, but are looking to see how much more likely joe is to lie about embarrasing or illegal activities than he is about his favourite colour, so as to correct for lying when polling on said subjects.
posted by Jairus at 6:28 AM on April 12, 2005

What Jairus said. Polling isn't perfect, but it is a science, and some fairly smart people have spent many years and a lot of money trying to improve it.

Unfortunately, it seems to me that, among the general public, ignorance about polling is at least as common as innumeracy.

That said, different polls are useful for different purposes (as a teenager, I worked in a phone bank for a partisan polling outfit that asked questions like 'If you knew Candidate X was a wife-swapping tax cheat, would you be more likely or less likely to vote for him?'), and some polls are more useful than others.
posted by box at 7:48 AM on April 12, 2005

In particular, whenever I see "High schoolers believe... (some weird thing)" or "High schoolers think...(some blatantly incorrect fact)" I always compare their percentages to the number of high schoolers who would either a.) Answer "C" to every question or b.) Answer the most improbable answer to every question and it usually comes out even.
posted by dagnyscott at 8:15 AM on April 12, 2005

It's called response bias, and you can correct for it (in large samples anyway) if you have a good theoretical and empirical understanding of who is more and less likely to respond to a given kind of poll. Mostly you just have to weight people in line with their propensity to participate to correct back to the ``actual' distribution.

Brehm's The Phantom Respondents discusses it at book length, and he's a mensch so you should buy it.
posted by ROU_Xenophobe at 8:53 AM on April 12, 2005

...doesn't this make the sample self selecting, and therefore useless?
On the contrary, it may be very useful if the function of polls is not to measure, but to influence. Polls are the laugh track of the political sit-com.
posted by weapons-grade pandemonium at 9:18 AM on April 12, 2005

While opinion polls may not be uber-scientific, it is nonetheless true that certain polling results do, in fact, foreshadow the results of some elections conducted later on.

Selection bias is a very important constraint in polling, and reputable places do many things to try to overcome it. For instance, they try to make surveys short and convenient to respond to. Some surveys offer incentives to participate, which does increase response rates.

Distortion of responses is also an important factor limiting the accuracy of polling data. Again, there are techniques to try to discourage such responses (respondent anonymity, question wording that doesn't induce bias and "normalizes" both sides of a question, etc.)

Cognitive psychology has also made great inroads in trying to understand what goes on in forming and manufacturing opinions - for example, it is simply not the case that people will have an opinion about everything, but if asked, they will create one based on cues in the question wording. These studies can help design better questions.

There is an extensive literature on survey methodology which attempts to deal with these issues. Some big names in the literature are Sudman and Bradburn. The census bureau has some free methodological papers that you could peruse to learn about this area.
posted by jasper411 at 10:00 AM on April 12, 2005

Cognitive psychology has also made great inroads in trying to understand what goes on in forming and manufacturing opinions - for example, it is simply not the case that people will have an opinion about everything, but if asked, they will create one based on cues in the question wording. These studies can help design better questions.

Great answers here. As to the above, I'd add that I've read about studies that discovered that there is a strong human urge to try to intuit what the pollster wants/expects of a respondent and to deliver in kind.

Also, when there is an odd number of choices (strongly disagree/disagree/don't care/agree/strongly agree, or the like), pollsters may count the neutral/center response along with either of the ends of the continuum, spinning the 'results' differently.

There is a lot of science in polling, but a lot of voodoo as well. Here;s a great recent NYer article about the Zogby pollls.
posted by Miko at 1:28 PM on April 12, 2005