What's the methodology for conducting national political polls in the US?

August 3, 2005 8:47 PM

Are national political polls that make use of random sample populations adjusted to account for the differences in overall population distribution throughout the US (in other words, do national political pollsters typically use some methodology to appropriately weigh responses from poll respondents depending on whether they live in more densely or less densely populated regions)?
posted by all-seeing eye dog to Science & Nature (9 answers total)

Yes. I work here and we -- and I'd presume all other professional polling organizations -- use a process called weighting to turn our raw random sample into one that corresponds to the demographics of the population at large.

Not speaking specifically about our methodology -- the internal criteria are somewhat of a trade secret across the board -- In addition to regional weighting, polls are often weighted to account for sampling issues in gender, age, and race.
posted by piro at 8:54 PM on August 3, 2005

I'm guessing no. Your idea would only make sense if the pollsters took pains to ensure that they had equal proportions of respondents from each state; in other words, if they polled one Alaskan and one Californian. Then, you would need to weight the responses appropriately.

The way I'm guessing that national polls work is that they choose (for example) 300 phone numbers at random. About 30 of those will be from California; perhaps 1 or 0 will be from Wyoming. No need to adjust the statistics this way.
posted by qslack at 8:58 PM on August 3, 2005

Response by poster: so... we have a tie? maybe it'll help if i clarify the question a bit: do the big pollsters like zogby and gallup do this as a general rule? what about the major media outlet's independent polls?
posted by all-seeing eye dog at 9:16 PM on August 3, 2005

Not a tie. qslack is saying that large-sample random dialing should take care of the weighting for you.

You seem to be assuming that people in high-density areas have a greater probability of being sampled than do people in lower-density areas. Polling takes pains to make sure that this ain't so, and that all members of the target population have an equal probability of being chosen.

Random-dialing telephone polls, where all of the valid possible phone numbers in the US including the unlisted ones have an equal chance of being chosen, is a good start in many circumstances.
posted by ROU_Xenophobe at 10:13 PM on August 3, 2005

If the pollsters aren't looking to embarrass themselves, they're doing everything they can to ensure that the sample is weighted properly so that the sample population properly reflects the population at large.

In your case, the sampling would be PPS, or probability proportionate to size, a form of multistage cluster sampling. Earl Babbie in his book The Practice of Social Research gives an example like so: suppose you want to poll a city's population, but most of it lives within a small area and not much lives in the much larger area comprising the rest of the city. If you pick a household from every 10th block you'll get an unrepresentative sample because of the population distribution.

So you'd figure out how many households are per city block. Suppose there are blocks A through Z. If Block A has 100 households and Block B has only 10, then Block A should have ten times the chance of selection as Block B. So if Block A has a 1 out of 20 chance of selection out of the city blocks, then block B has 1 out of 200. If you want 5 households from each block, then if Block A is selected for sampling, a household on it has a 1/400 chance of selection (1/20 chance for the block * 5/100 for each house on it). If Block B is selected for sampling, then a household on it has the same chance of selection (1/200 chance for the block * 5/10 chance for each house on it).

So the basic idea is to weight the samples so that the sample group has a chance of selection appropriate to its proportion of the larger population, but that within each group, each unit has the same chance of selection as all the others.

As for whatever formula Zogby and the others are using to weight their samples nationwide, I have no clue. But I'd be very surprised indeed if they weren't using them.

Incidentally, Babbie starts that chapter on multistage cluster sampling by talking about a poll in 1936, conducted by the Literary Digest, which sent out over 10 million ballots (and got over 2 million responses) about who the respondents intended to vote for for U.S. President. The response was overwhelmingly in favor of Alf Landon, who'd been running against Franklin Roosevelt. The Digest was rather surprised at the end of election day. They'd skewed their sample by sending the ballots to addresses pulled from the phone book and from auto registries, but this was just after the Great Depression, when most of the U.S. population didn't have a phone or a car.

Incidentally, the Babbie book is 10th edition, the examples here from chapter 7, pp. 181 and pp. 212-213, if you'd care to read more about PPS.
posted by Tuwa at 10:26 PM on August 3, 2005

Re: random phone numbers. that would work assuming that any area code has the same probability of being chosen, and that the distribution of phones is even (e.g.:everybody has a phone).
If, for instance, more whites than blacks have phones, you’d get a built in bias.
Generally speaking, polls are adjusted for a lot of different factors, including density.
posted by signal at 10:40 PM on August 3, 2005

good point, signal; population distribution is just one factor of many they'd have to weight. It all depends on what the point of the poll is (and, therefore, what population characteristics are judged relevant to the study), but in general it is very important to guard against biasing the samples.
posted by Tuwa at 10:47 PM on August 3, 2005

qslack gets the ideal, but in practice we know a few things. Older people answer their phones more than younger people, whites more than blacks, etc. As a result, even with comparatively large samples, we have to weight. It's conceivable that a 1000 person RDD survey would underrepresent significant subpopulations, say reaching only 80 African Americans. As a result, even after legitimate sample designs, we still have to weight.
posted by piro at 5:51 AM on August 4, 2005

Response by poster: Thanks guys--I think my confusion was not taking into account the averaging affect of the randomness of the sample. I was assuming that, in a random sample, any respondent from a given region would be just as likely to be selected as any other, but obviously, if there are more potential respondents in a particular region, there's also a greater probability respondents from that region will be selected (excluding other factors that might need to be adjusted for, like phone access, etc.). Got it. Thanks!
posted by all-seeing eye dog at 2:30 PM on August 4, 2005