Polling procedure question

August 23, 2008 10:00 AM

I was talking about the polls in the presidential election yesterday and someone mentioned that the polls are skewed towards McCain because the polling companies only call people who still have landlines, or generally older people (I guess). She claimed this was for geographic demographic reasons. Is it true that polling companies do this and don't call cell phones?
posted by josher71 to law & government (5 answers total)

Yes, it's largely but not always true. Here is an in-depth look at the issue.
posted by Pants! at 10:05 AM on August 23, 2008

They're not permitted to make unsolicited calls to cell phones. It's an FCC regulation. It also applies to telemarketing.

That's because the US doesn't have "caller pays", so a telephone solicitation or a pollster who calls someone on a cell phone costs the phone owner for the minutes being used.

The fact that cell phone owners, especially those who no longer have land lines, are not included is one source of bias. There are others. Probably the most important is that the majority of people who receive this kind of call curse and hang up. Are those who are willing to participate different from those who don't? Almost certainly.

However, contra what your friend says, historically polls have skewed Democratic. The reason is because of "stratified sampling" and because the pollsters historically have included more Democrats in their samples than turn out to vote. (Zogby stands out as an aberration among pollsters because they generally include a higher proportion of Republicans in their samples. This doesn't necessarily make them more accurate, I might mention.)

If you compare poll predictions on elections to actual results over the last 30 years or so, Republican upsets are far more common than Democratic upsets. Stratified sampling using incorrect assumptions about likely voters seems to be the main reason why.
posted by Class Goat at 10:33 AM on August 23, 2008

Pew has put out a lot of work on who is in cell-phone only households and what they mean for polling. In focused on this one focused on the 2008 election, they mention that the effect on the reliability of polls is muted because they are less likely to pass through the likely voter screens that most polling places use.
posted by milkrate at 10:37 AM on August 23, 2008

Pants!'s link is an excellent explanation, and you should give it a best answer.

I work for a nonprofit health policy think-tanky place. One of our program areas is called Public Opinion and Survey Research; its director has given several talks to staff about how surveys work, and how survey methods are changing in a world where increasing numbers of households are cell phone-only. We recently released a survey of/about New Orleans residents three years after Katrina, and there's a good explanation (.pdf) of the methods they used to collect responses in an area where many people don't have phones at all or are cell phone-only.

Part of the problem is that mixed-methodology like this is quite a bit more expensive than the traditional robo-dial methods, because you have to hire actual people to dial cell phone numbers, and each number must be called N number of times (before you give up, anyway) for the try to be considered valid.
posted by rtha at 10:43 AM on August 23, 2008

Another reason why polls sometimes skew is because it turns out that participating in a poll is a social encounter for the voter.

There is a tendency to respnd with what are considered to be socially-acceptable answers rather than heartfelt ones. Since we have secret ballots, and because of this tendency, it is more common than you might think for a voter to tell a pollster one thing and to vote for a different thing.

As an extreme sample of this, there were parallel polls done during WWII among American blacks. In one poll, the pollsters (who visited in person) were white. In the other poll, the pollsters were black. They asked the same questions, but got radically different results. Almost certainly this was an example of the people being polled saying what they thought they should say, rather than what they really believed. No way to know which poll was more accurate, either. (The questions had to do with whether it was more important for the nation to beat the Axis or to work to solve problems at home. Overwhelmingly the white pollsters were told that beating the Axis was more important, but the black pollsters fond that a strong majority thought that fixing problems at home was more important. The study wasn't really concerned with either of those issues; the real reason it was done was precisely to find out how much different the answers would be for different groups of pollsters.)

That tendency seems to operate now more in favor of the Democrats overall, though it varies enormously from place to place. There is a surprisingly strong (and really quite appalling) lack of tolerance in some places for conservative voices and opinions, and people who really believe in conservative positions often lie about it socially in order not to be ostracized. But come election day, in the secret voting booth, they can really express what they think -- and then lie about it moments later to the exit-pollster.

That's part of the reason for the legendary quote from Pauline Kael about the 1972 election: "Nixon can't have won; no one I know voted for him." Actually, it's likely that a lot of people she knew did vote for Nixon, but didn't admit it publicly to people like her who would have been scandalized and offended.
posted by Class Goat at 10:51 AM on August 23, 2008 [1 favorite]