Is this poll valid?

June 9, 2010 12:06 PM

Is this poll bogus? A conservative friend has sent me a link to this poll, conducted by Zogby International (which appears reputable), that gives low scores on "economic enlightenment" to women, the less-wealthy, infrequent Wal-Mart shoppers, non-Nascar fans, the unmarried, union members, Obama voters, etc., and doesn't correlate it with college attendance.

The poll seems very slanted and bizarre to me but I don't know enough about polling to know if I'm right or not. Can anyone help? The following paragraph really stood out, for instance:

We think it is reasonable to maintain that if a respondent disagrees with the statement “Restrictions on housing development make housing less affordable,” the respondent betrays a lack of economic enlightenment. Challengers might say something like: “Well, not every restriction on housing development makes housing less affordable,” but such a challenger would be tendentious and churlish. Unless a statement in a questionnaire explicitly makes it a matter of 100%, by using “every,” “all,” “always,” “none,” or “never,” it is natural to understand the statement as a by-and-large statement about overall consequences. Do restrictions on housing development, by and large, make housing less affordable? Yes they do. Does free trade lead, overall, to greater unemployment? No, it does not. For someone to say the contrary is economically unenlightened.

posted by rleamon to law & government (38 answers total)

I can't answer the question about its validity, but the study is discussed by one of the authors, here: http://online.wsj.com/article/SB10001424052748703561604575282190930932412.html?mod=WSJ_Opinion_LEADTop
posted by dfriedman at 12:08 PM on June 9, 2010

Sorry, link.
posted by dfriedman at 12:09 PM on June 9, 2010

I'm not an economist, but I happened to see a reaction to this study yesterday.
posted by These Premises Are Alarmed at 12:10 PM on June 9, 2010

Analysis by Nate Silver of FiveThirtyEight.com.
posted by sharkfu at 12:12 PM on June 9, 2010 [19 favorites]

When looking for a dissection of a poll, it's methodology, etc I always check out fivethirtyeight.com
posted by iamabot at 12:15 PM on June 9, 2010

I haven't read the linked article in depth or devoted any substantial brain power to it. That said, this is my take:

Self-proclaimed smart people declare their own ideas to be the "enlightened" ideas and then declare anyone who agrees with them to be "enlightened," and then use a poll in an effort to pretend that "enlightened" is a quantifiable and objectively measurable quality, inadvertently discrediting the allegedly "enlightened" ideas by revealing that the only people in America who actually subscribe to them are idiots and a small group of axe-grindy pollsters.
posted by The World Famous at 12:16 PM on June 9, 2010 [7 favorites]

Some criticism of the study from 538, also a pretty reputable source.
posted by nangar at 12:17 PM on June 9, 2010 [1 favorite]

I'll fourth the recommendation of Nate Silver's analysis at 538. He's a stats wiz who started analyzing baseball and then moved into political polls. He analyzed the polling throughout the 2008 election and is the go-to guy for if and why a poll is crap or gold.
posted by alaijmw at 12:20 PM on June 9, 2010

I think the study was a better test of reading comprehension then of economic theory. That said of the questions that ended up being used (out of the ones asked), I don't think they were that controversial.

That said: I would expect knowledge of economics to be uncorrelated to having taken college. And I know of organizations who are trying to specifically enhance economic literacy among women, so that does not surprise me either. The rest seem like iffier conclusions.
posted by An algorithmic dog at 12:22 PM on June 9, 2010

Here is your answer:

"A number of controversial interpretive issues attend our measure, including: (1) our designation of enlightened answers; (2) an asymmetry in sometimes challenging leftist mentalities without ever specifically challenging conservative and libertarian mentalities; (3) our simple eight-question test is merely a baseline and does not gauge the heights of economic enlightenment; and (4) a concern about response bias (namely, that less intelligent people would be less likely to participate in the survey)."

First, this poll was conducted by Zogby but the two authors made up the questions and paid Zogby, in other words, it is not Zogby's poll, they just administered the poll.


Their determination of "unenlightened" appears to be based on neoliberal economic assumptions. Conservatives; regardless of education, would respond as enlightened on each of these questions thus skewing the poll. The questions, rather than being created with apolitical questions that quantitatively assess economic knowledge, lead the poll to the results that they wish to have.

Scientific rigor is not what this paper is about, it is about having a conclusion, asking a question to apply the conclusion to and writing a research paper.

A quick, final scan of the references should seal the bias of the paper and the author's lack of interest in actually measuring their results:

Hayek, Friedrich A. 1988. The Fatal Conceit: The Errors of Socialism. Chicago:
University of Chicago Press.

Hewitt, Gordon J., and Mack Mariani. 2010. Indoctrination U. Revisited:
Faculty Ideology and Changes in Student Ideology and Issue Positions,
2002-2006. Ms.

Woessner, Matthew, and April Kelly-Woessner. 2009. Left Pipeline: Why
Conservatives Don’t Get Doctorates. In The Politically Correct University:
Problems, Scope, and Reforms, ed. Robert Maranto, Richard E. Redding, and
Frederick M. Hess, 38-59. Washington, DC: American Enterprise Institute.

In the last case, why cite a study on why conservatives don't get doctorates? Does this help to answer the question? It does if you have an answer in mind.
posted by occidental at 12:23 PM on June 9, 2010 [2 favorites]

We think it is reasonable to maintain that if a respondent disagrees with the statement “Restrictions on housing development make housing less affordable,” the respondent betrays a lack of economic enlightenment. Challengers might say something like: “Well, not every restriction on housing development makes housing less affordable,” but such a challenger would be tendentious and churlish.

We think that if a respondent disagrees with our view that restrictions on housing development makes housing less affordable, we can successfully imply they are ignorant without having to explicitly state or prove it. Challengers might say something like "this is not a proper weighted, balanced poll, your questions are poorly phrased or trivial, and your answers are unsupported by research," but we will dismiss such a challenger by labeling them with big words our target audience will not admit they do not understand.
posted by davejay at 12:23 PM on June 9, 2010 [13 favorites]

Let me tell you a little story about polling.

During my first semester in grad school, I took a course in quantitative research methods. Unfortunately, my group had neither the time nor experience nor budget to conduct actual research ourselves, so we were charged with finding an existing dataset and making conclusions, hence practicing our math and paper-writing skills. I forget where we got it, but it was someplace reputable like the Pew Research Center's Internet and American Life Project.

The reason we could do this was because the dataset contained dozens of questions about everything from demographic and educational info to attitudes on science, etc. We focused on just a few of the possible questions, and we actually had to work really hard to come up with a variable that was statistically significant against something else. We made up all sorts of combinations -- using existing, valid data, mind you -- but we pretty much flailed around trying to find something that would work, because otherwise we wouldn't have anything to write a paper about. Our prof was pretty sympathetic, but we pressed on because dammit, we were NOT going to admit failure and do it half-assed!

Eventually, we finally found one. Of course, by that time it was some weird combination of education and use of the Internet correlated with attitudes of global warming, and there was really no set way to explain that without coming up with our own name for it. So we called it "global warming exigency" and that was that. (The paper, btw, was then presented at a national conference.)

So when they say "economic enlightenment," chances are that it's some combination they made up. The Zogby involvement means next to nothing; they just did the poll and made the data public. The paper's authors are required to list what went into their variables, so check it out.
posted by Madamina at 12:24 PM on June 9, 2010 [2 favorites]

We think that if a respondent disagrees with our view that restrictions on housing development makes housing less affordable, we can successfully imply they are ignorant without having to explicitly state or prove it. Challengers might say something like "this is not a proper weighted, balanced poll, your questions are poorly phrased or trivial, and your answers are unsupported by research," but we will dismiss such a challenger by labeling them with big words our target audience will not admit they do not understand.

Wait what? What does having a properly weighted and balanced poll have to do with if a given question is suitable? If the questions are trivial everyone should be getting the same answer as the author. And not every question whose answer you want to disagree with is phrased poorly.
posted by An algorithmic dog at 12:30 PM on June 9, 2010 [1 favorite]

“Well, not every restriction on housing development makes housing less affordable,” but such a challenger would be tendentious and churlish.

Tendentious, churlish, and correct.
posted by The World Famous at 12:37 PM on June 9, 2010 [7 favorites]

I used to work with pollsters. I've been out of the polling world for a couple of years, but my memory is that Zogby is a bit of a joke among other pollsters - seen as eccentric and not rigorous in their methods. This poll pretty much falls in line with that impression.
posted by lunasol at 12:53 PM on June 9, 2010

First, this poll was conducted by Zogby but the two authors made up the questions and paid Zogby, in other words, it is not Zogby's poll, they just administered the poll.

No, it was still Zogby's poll, with a sampling design created and run by Zogby, and a weighting scheme (ie, no weighting scheme) created and run by Zogby, and a way of dealing with cell-only people created and run by Zogby, and so on. The only thing that were the researchers' were the wording of those questions.
posted by ROU_Xenophobe at 1:01 PM on June 9, 2010

You're getting a biased sample of responses to this poll. In general, the paper's statements of generally-agreed economic facts are correct -- i.e., they represent the views of the economic profession (which have been surveyed pretty exhaustively), who are experts applying scientific methods -- not everything they believe is true, but they're a lot better informed than the average Ask Me troll. The challenges above are generally being made in ignorance.

The methodological error of the poll is that it disproportionately samples subject areas where liberal policies come with economic costs. Since people tend to defend their ideologies, relatively liberal respondents are giving economically wrong answers to these questions. However, we could instead frame questions where conservative policies come with economic costs, on areas like the drug war, the environment, and immigration. If we did that, we would tend to see conservative respondents disagreeing with the mainstream of the economics profession.

The larger point is that the population overall is economically illiterate, and tends to choose its favored policies based on feelings that have nothing to do with economic fact. They will then deny economic fact to defend these policies. It's quite possible that either liberals or conservatives (etc.) are worse about this, but it would take a much cleverer experimental design to reach this conclusion.
posted by grobstein at 1:03 PM on June 9, 2010 [7 favorites]

They eliminated some of the questions because, in their own words, the questions were "too vague or too narrowly factual, or because the enlightened answer is too uncertain or arguable" but left in the question "Third-world workers working for American companies overseas are being exploited". This seems pretty darned vague to me and the answer they are looking for is definitely debateable.

An what is up with "enlightened"? That seems like a term calculated to annoy. Would it have killed them to say "economically savvy" or something like that (I'd still disagree with them, but I wouldn't want to kick them in the teeth quite as hard).
posted by It's Never Lurgi at 1:06 PM on June 9, 2010

From Nate Silver:

So basically, what you're left with a number of questions in which people respond out of their ideological reference points because the questions are ambiguous, substanceless, or confusing. Klein is blaming the victims, as it were.

..But between the poorly-considered questions and the poor choice of survey partner, this amounts to junk science.

I will go one step further than Nate and say that the questions were almost certainly deliberately crafted/selected to be ones in which ideological reference points of liberals would run counter to any technically true answer.

That is, much worse than it being a poorly designed poll, it was deliberately broken in order to play on the grey ambiguities in folks' ideologies. The result being that conservatives would be more likely to get answers accidentally "correct" (when such correctness is even possible to quantify) and liberals would be more likely to get answers accidentally "incorrect".

So it's not just junk science, as Nate says. It's worse. It's intentionally deceitful propaganda.
posted by darkstar at 1:12 PM on June 9, 2010 [6 favorites]

they represent the views of the economic profession

I don't think that's completely accurate.

Take this one: "Overall, the standard of living is higher today than it was 30 years ago." It's well known that real median family income has been basically static (with noise) for 20 to 30 years.

Or take this one: "Third-world workers working for American companies overseas are being
exploited." I dunno. I could probably find you at least some economists who would agree that some third-world workers might be paid less than their marginal product (ie, exploited) and kept working by threats of violence.
posted by ROU_Xenophobe at 1:15 PM on June 9, 2010

John Zogby is a reputable pollster, and it is entirely typical for paying clients to influence the phrasing and thrust of questions (in fact, some would way that's what they're paying for).

But this is a "Zogby Interactive Poll," which is basically longhand for "shit." The problems with this methodology have been explored extensively on 538, and Zogby Interactive's consistent outlier status vis-a-vis polling composites and trends underscore that Zogby's internet methodology is really not ready for prime time.
posted by Ignatius J. Reilly at 1:16 PM on June 9, 2010

I'd have to say the poll creators are tendentious and, yes, even churlish.
posted by cftarnas at 1:16 PM on June 9, 2010

Wait what? What does having a properly weighted and balanced poll have to do with if a given question is suitable? If the questions are trivial everyone should be getting the same answer as the author. And not every question whose answer you want to disagree with is phrased poorly.

How rude of you to assume so much, and comprehend so little, of my statement!

I made no comment about questions being "suitable", I made no comment as to whether I agreed or disagreed with the poll's answers, and I am using the word "trivial" in the sense of being "of little worth or importance" rather than "commonplace; ordinary" (http://www.merriam-webster.com/dictionary/trivial).

I did, however, based my statement in large part on Nate Silver's assessment, linked above on 586.

In regard to proper weighting and balance, he said: "They took a survey using one of Zogby's internet panels, which is by far the worst polling instrument that they could have selected. The panel was not weighted and was not in balance."

In regard to the questions being poorly phrased, he said: "Other [questions] were poorly phrased...This is confusing..."

In regard to the questions being trivial, he said: "Some come closer to having a technically correct answer, but are more within the realm of trivia."

I'm normally not one to get annoyed, but I don't appreciate being misquoted, or having my statement misrepresented, for the sake of your strawman.
posted by davejay at 1:30 PM on June 9, 2010

Wow, I need to save this in case I ever do a class on research methods and want to show what NOT to do.

As others have already said, this poll and the results obtained from it are crap; those questions seemed pretty clearly designed to draw out partisan responses. Come on, rent control, free trade? Yeah, these aren't politically charged economic issues. The author really could have picked different subjects for questions about basic economics that don't have such a strong connection to partisan issues. At least they didn't include the 'poverty causes crime' question, I suppose.

I'd refer you to Damned Lies and Statistics for more information on why 90% of statistical statements are garbage, and how to evaluate those statements.

(I can't believe that this was done by someone in my university. I feel dirty)
posted by _cave at 1:40 PM on June 9, 2010 [1 favorite]

I ain't no big city statistician, but the one class in research methods I took leads me to believe that this poll is utter bullshit.
posted by nestor_makhno at 2:12 PM on June 9, 2010

The 538 critique seems basically sound to me (a polisci phd student), w/ the caveat that internet sampling is becoming more and more accepted. The big issue is in question wording and interpretation, i.e., stating a bunch of controversial economic positions and then counting disagreeing with them as lack of enlightenment. There's lots more disagreement in economics than they'd have you believe, and there are some very well-known very left-wing economists with things like Nobel prizes (Joseph Stiglitz comes to mind, also arguably Ken Arrow, who is universally recognized as one of the best 2 or 3 economists of the 20th century) who would probably disagree with many of the propositions that count toward "enlightenment" in the George Mason University economics department. (Incidentally, GMU, where the authors are located, is well-known as one of the most right-wing econ departments in the country.)
posted by paultopia at 2:35 PM on June 9, 2010 [1 favorite]

Addendum: I found this question particularly idiotic (quote from the WSJ article)

5) Third World workers working for American companies overseas are being exploited (unenlightened answer: agree).

WHAT. THE. FUCK? This builds a moral question (whether what happens to workers overseas counts as "exploitation" or not) into a poll supposedly about economics. That's just enraging. If you want to know whether something counts as exploitation, ask a philosopher, not an economist. Instead, the authors would have you believe that your position on this moral question has something to do with your understanding of economic theory. Moronic.
posted by paultopia at 2:38 PM on June 9, 2010 [3 favorites]

The only thing this poll measures is the ability of the respondent to critically filter the questions for the pollster's bias.
posted by DarlingBri at 2:54 PM on June 9, 2010

I wonder why they didn't ask the question "Tax cuts pay for themselves through economic expansion -- agree/disagree"?
posted by JackFlash at 3:00 PM on June 9, 2010 [1 favorite]

At the risk of stating the same thing as everyone else: the poll badly conflates political ideology with economic knowledge. You can get an excellent score by responding on right-wing party political lines even if you know nothing about economics, but the reverse isn't true for left-wing respondents. The point here is *even if* it were true that people on the right were generally more knowledgeable about economics, it is the job of a good survey to disentangle the two. What I find suspicious is that this wouldn't actually have been very hard to do (e.g., include questions about the conditions that produce market failures). Given this, it's either the case that the survey designers didn't think very hard about the potential confounds, or they deliberately constructed a dodgy instrument. I don't see any other possibilities.
posted by mixing at 3:08 PM on June 9, 2010

As mixing says, it conflates politics with economics. It also confuses "libertarian economics" with "economics".

Of course, most economists entangle religiosity with economics. Note the use of the word "enlightened".

So if you don't agree with the libertarian philosophy of economics, you are not enlightened.

Now, I'm not saying libertarian economics doesn't have its place- in a perfect world, everyone would be perfectly informed and be able to choose the best thing for themselves. But it's not a perfect world, clearly.

WHAT. THE. FUCK? This builds a moral question (whether what happens to workers overseas counts as "exploitation" or not) into a poll supposedly about economics. That's just enraging. If you want to know whether something counts as exploitation, ask a philosopher, not an economist. Instead, the authors would have you believe that your position on this moral question has something to do with your understanding of economic theory. Moronic.

Their position seems to be that because the workers are being paid and free to quit, they are not being exploited.
posted by gjc at 3:49 PM on June 9, 2010 [1 favorite]

Their position seems to be that because the workers are being paid and free to quit, they are not being exploited.

What workers, specifically, are you referring to? What workers, specifically, are the pollsters referring to?

Because if their position is that the more "enlightened" economic view is that every single person working for a U.S. company outside the U.S. is a) being paid, and b) free to quit, I think they're perhaps a little less enlightened than they would like to believe.
posted by The World Famous at 3:52 PM on June 9, 2010

1. Restrictions on housing development make housing less affordable. - Unenlightened: Disagree

Demand for housing is not very elastic. That's why over and under supply get people freaking out really quickly.

2. Mandatory licensing of professional services increases the prices of those
services. - Unenlightened: Disagree

Their answer is right, but not controversial. Still rather know that the doctor knows what a kidney is.

3. Overall, the standard of living is higher today than it was 30 years ago. - Unenlightened: Disagree

By some measures. But family income is steady, and it takes two working adults to achieve it now.

4. Rent control leads to housing shortages. - Unenlightened: Disagree

This is essentially the same question as #1. Restricting housing supply raises prices. Yes, but it is rarely the most important factor.

5. A company with the largest market share is a monopoly. - Unenlightened: Agree

Their answer is right, and not controversial

6. Third-world workers working for American companies overseas are being
exploited. - Unenlightened: Agree

Define "exploited." I define it as the fact that companies can cross borders to pay a lower wage, but the workers cannot generally cross borders to earn a higher wage.

7. Free trade leads to unemployment. - Unenlightened: Agree

Free trade never means free trade. If it did we wouldn't keep bitch slapping the developing world by massively subsidizing agriculture and heavily taxing sugar.

8. Minimum wage laws raise unemployment. - Unenlightened: Disagree

I am not sure how this one works out. I'm sure that unemployment would rise at first if labor became more expensive, but what about the economic advantages of having a bigger middle class with more disposable income?
posted by Nothing at 5:28 PM on June 9, 2010

Liberals believe that the economic costs are worth it due to placing a higher value on other things, like individual dignity, equality, etc. So even if something has a higher economic cost, it doesn't mean it's dumb or inherently bad. This poll uses the term "unenlightened" to mean has a different value on economic costs than conservatives.
posted by fructose at 5:31 PM on June 9, 2010

I'm no economist, but I've taken a handful of grad econ classes at one of the top econ schools in the US (mostly out of curiousity). I'm also way more conservative than the average Mefite. I actually sorta understand where this poll is coming from: economic theory has lead to some counterintuitive results, and economists are very interested in how these results have percolated into popular discussion (and ultimately political decisions). But their methodology is awful and likely to judge whether a person can recognize conservative talking points rather than have any actual comprehension of those ideas.
posted by miyabo at 9:04 PM on June 9, 2010

Incidentally, GMU, where the authors are located, is well-known as one of the most right-wing econ departments in the country.

This. I would automatically expect a conservative bias from anything coming out of that school's econ department.
posted by naoko at 9:28 PM on June 9, 2010

Here's a set of questions that I use to smoke out ideological conservatives masquerading as the economically literate:

1) Taxation causes deadweight losses and should be structured to minimise these.
Conservative answer: true
Economists answer: true

2) Estate tax typically causes less deadweight loss, so it should be high.
Conservative answer: Hell no! Don't want no death tax.
Economists answer: true
posted by atrazine at 5:03 AM on June 10, 2010 [1 favorite]

Is this poll bogus? A conservative friend has sent me a link to this poll...

Asked and answered in the first 15 words.

I'm only half-kidding.
posted by callmejay at 12:35 PM on June 10, 2010