Upgrading to Damned Lies
September 28, 2010 9:11 AM Subscribe
Please point me to quality books, articles, etc which argue both for and against the use of quantitative research techniques in the social sciences, particularly in political science and public policy.
I'm looking for the canonical arguments on this issue, as well as any effective summarize of the debate. Discussion of the treatment of the social sciences as sciences are also much appreciated. Thanks!
I'm looking for the canonical arguments on this issue, as well as any effective summarize of the debate. Discussion of the treatment of the social sciences as sciences are also much appreciated. Thanks!
Best answer: Start with the point and counterpoint. Designing Social Inquiry by King, Keohane, and Verba is a canonical text arguing for the application of quantitative thinking to qualitative research in political science. The King in that group of authors is Gary King, quantitative superstar. The book is so canonical that most people in the social sciences refer to it basically as "KKV." The counterpoint, also by political scientists, is Rethinking Social Inquiry edited by Collier & Brady (though it includes chapters by many more political scientists and sociologists). You may then want to move on to texts such as Case Studies and Theory Development in the Social Sciences by George and Bennett (political scientists), Redesigning Social Inquiry: Fuzzy Sets and Beyond and The Comparative Method, both by Charles Ragin (a quantitatively trained sociologist).
The bottom-line of this entire debate is that the use of quantitative or qualitative methods is a decision that is highly political in some circles but that method decisions should be driven by inquiry. Different methods are more appropriate for different questions; similarly, different methods are undergirded by different epistemologies. The main argument in the Collier and Brady book is that KKV's model of research is very much based in the natural sciences (particularly physics) that may be unsuited to social phenomena in which agency is prominent. Research methods are socially and politically embedded in a particular time and place and come to be taken-for-granted in a way that the underlying logic of a particular kind of inquiry becomes the only "right" way to do research and all other methodologies must respond to this logic or be perceived as less scientific or rigorous.
You might also check out Salsa Dancing into the Social Sciences: Research in an Age of Info-Glut by Kristen Luker (a sociologist) for a more conversational and "how-to" treatment of this topic.
posted by proj at 9:37 AM on September 28, 2010 [1 favorite]
The bottom-line of this entire debate is that the use of quantitative or qualitative methods is a decision that is highly political in some circles but that method decisions should be driven by inquiry. Different methods are more appropriate for different questions; similarly, different methods are undergirded by different epistemologies. The main argument in the Collier and Brady book is that KKV's model of research is very much based in the natural sciences (particularly physics) that may be unsuited to social phenomena in which agency is prominent. Research methods are socially and politically embedded in a particular time and place and come to be taken-for-granted in a way that the underlying logic of a particular kind of inquiry becomes the only "right" way to do research and all other methodologies must respond to this logic or be perceived as less scientific or rigorous.
You might also check out Salsa Dancing into the Social Sciences: Research in an Age of Info-Glut by Kristen Luker (a sociologist) for a more conversational and "how-to" treatment of this topic.
posted by proj at 9:37 AM on September 28, 2010 [1 favorite]
I can't speak for its quality, as I haven't read it, but I saw this abstract the other day:
Abbott, Andrew. "Against Narrative: A Preface to Lyrical Sociology." Sociological Theory Volume 25, Issue 1, pages 67–99, March 2007
This article develops a concept of lyrical sociology, a sociology I oppose to narrative sociology, by which I mean standard quantitative inquiry with its “narratives” of variables as well as those parts of qualitative sociology that take a narrative and explanatory approach to social life. Lyrical sociology is characterized by an engaged, nonironic stance toward its object of analysis, by specific location of both its subject and its object in social space, and by a momentaneous conception of social time. Lyrical sociology typically uses strong figuration and personification, and aims to communicate its author's emotional stance toward his or her object of study, rather than to “explain” that object. The analysis considers many examples and draws on literary criticism, the philosophy of time, and the theory of emotion. It also addresses contemporary debates in ethnography.
posted by Beardman at 9:54 AM on September 28, 2010
Abbott, Andrew. "Against Narrative: A Preface to Lyrical Sociology." Sociological Theory Volume 25, Issue 1, pages 67–99, March 2007
This article develops a concept of lyrical sociology, a sociology I oppose to narrative sociology, by which I mean standard quantitative inquiry with its “narratives” of variables as well as those parts of qualitative sociology that take a narrative and explanatory approach to social life. Lyrical sociology is characterized by an engaged, nonironic stance toward its object of analysis, by specific location of both its subject and its object in social space, and by a momentaneous conception of social time. Lyrical sociology typically uses strong figuration and personification, and aims to communicate its author's emotional stance toward his or her object of study, rather than to “explain” that object. The analysis considers many examples and draws on literary criticism, the philosophy of time, and the theory of emotion. It also addresses contemporary debates in ethnography.
posted by Beardman at 9:54 AM on September 28, 2010
Best answer: The idea to look at KKV and its opponents is basically sound, but it's going to miss the point a little bit.
KKV would argue very strongly that they're not arguing for "the application of quantitative thinking to qualitative research." They'd say that they're arguing that qualitative and quantitative research both face difficult problems of inference, and that these problems are at some level the same no matter your style of research. No matter your style of research, you still have to worry about operationalizing your theoretical variables, and you still have to worry about selecting on the dv, and you still have to worry about the range of your IVs. Thinking about these things and trying to limit the problems, they would say, are not "quantitative thinking."
That said, quant types tend to approve of KKV and squishier qual types oppose it, so there's that. And both it and the Brady/Collier book are pretty good reads (for methods books).
One thing to consider is that while proj is right that quant/qual can be a divisive and political issue, this is pretty well restricted to comparative politics. In IR, their quants and quals (and data-less formal modelers) seem to get along a lot better without constant sniping at each other. In American politics, the quants won, with the narrow exceptions of some American-political-development and some legal-oriented public law (and Dick Fenno). It's pretty much only in comparative where there are these really nasty camps. I can't speak for how things have shaken out in policy schools.
Another thing to consider is that the debate is less about "quant versus qual." Almost every quantitatively oriented scholar in the field sees nothing wrong with qualitative research, and views qualitative work as a very useful backstop to quantitative work and even fine on its own so long as it's rigorously and carefully done. Case in point, pretty much everyone (hard-ass quants included) loves Lin Ostrom's Governing the Commons to pieces, but there's not a regression to be found anywhere in there. I mean, I'm pretty firmly on one side of this divide, but from here it looks more like "Quant and qual are both okay, but hard, and since more data is better than less data you maybe ought to aim for quant" versus "Qual only and quant is inherently stupid."
posted by ROU_Xenophobe at 10:30 AM on September 28, 2010
KKV would argue very strongly that they're not arguing for "the application of quantitative thinking to qualitative research." They'd say that they're arguing that qualitative and quantitative research both face difficult problems of inference, and that these problems are at some level the same no matter your style of research. No matter your style of research, you still have to worry about operationalizing your theoretical variables, and you still have to worry about selecting on the dv, and you still have to worry about the range of your IVs. Thinking about these things and trying to limit the problems, they would say, are not "quantitative thinking."
That said, quant types tend to approve of KKV and squishier qual types oppose it, so there's that. And both it and the Brady/Collier book are pretty good reads (for methods books).
One thing to consider is that while proj is right that quant/qual can be a divisive and political issue, this is pretty well restricted to comparative politics. In IR, their quants and quals (and data-less formal modelers) seem to get along a lot better without constant sniping at each other. In American politics, the quants won, with the narrow exceptions of some American-political-development and some legal-oriented public law (and Dick Fenno). It's pretty much only in comparative where there are these really nasty camps. I can't speak for how things have shaken out in policy schools.
Another thing to consider is that the debate is less about "quant versus qual." Almost every quantitatively oriented scholar in the field sees nothing wrong with qualitative research, and views qualitative work as a very useful backstop to quantitative work and even fine on its own so long as it's rigorously and carefully done. Case in point, pretty much everyone (hard-ass quants included) loves Lin Ostrom's Governing the Commons to pieces, but there's not a regression to be found anywhere in there. I mean, I'm pretty firmly on one side of this divide, but from here it looks more like "Quant and qual are both okay, but hard, and since more data is better than less data you maybe ought to aim for quant" versus "Qual only and quant is inherently stupid."
posted by ROU_Xenophobe at 10:30 AM on September 28, 2010
Response by poster: This is all great stuff. I'm also interested particularly in arguments which address the validity of quantitative research in the social sciences per se (that is, can the results of RFTs be safely extrapolated outside their research group? Can statistical methods control for outside variables?, etc).
posted by l33tpolicywonk at 10:44 AM on September 28, 2010
posted by l33tpolicywonk at 10:44 AM on September 28, 2010
No matter your style of research, you still have to worry about operationalizing your theoretical variables, and you still have to worry about selecting on the dv, and you still have to worry about the range of your IVs. Thinking about these things and trying to limit the problems, they would say, are not "quantitative thinking."
Not to derail, but statements like these are at the heart of Brady & Collier's book and underscore my point above about the social and political embeddedness of research methods and how they become institutionalized. I don't disagree about operationalizing variables or worrying about the range of IVs, though I know that many qualitative researchers disagree with the point about selecting on the DV. KKV have chapters in the Brady & Collier book in response, though some have argued that their response demonstrates that they don't "get" the arguments being put forward in the Brady & Collier volume, i.e., they take for granted that the philosophical underpinnings of their own research methods should be the standards by which other methods are judged. Charles Ragin, a sociologist mentioned above, also makes a lot of these points by applying set theory and fuzzy set algebra to the analysis of macro phenomena.
In response to l33tpolicywonk's latest question, I'm not sure entirely what you're asking beyond basic questions about external validity and the ability for multiple regression to control for variables. If those are your basic questions, you might want to take some basic statistics courses before you get into a methodological debate about the validity of approaches.
posted by proj at 10:58 AM on September 28, 2010 [1 favorite]
Not to derail, but statements like these are at the heart of Brady & Collier's book and underscore my point above about the social and political embeddedness of research methods and how they become institutionalized. I don't disagree about operationalizing variables or worrying about the range of IVs, though I know that many qualitative researchers disagree with the point about selecting on the DV. KKV have chapters in the Brady & Collier book in response, though some have argued that their response demonstrates that they don't "get" the arguments being put forward in the Brady & Collier volume, i.e., they take for granted that the philosophical underpinnings of their own research methods should be the standards by which other methods are judged. Charles Ragin, a sociologist mentioned above, also makes a lot of these points by applying set theory and fuzzy set algebra to the analysis of macro phenomena.
In response to l33tpolicywonk's latest question, I'm not sure entirely what you're asking beyond basic questions about external validity and the ability for multiple regression to control for variables. If those are your basic questions, you might want to take some basic statistics courses before you get into a methodological debate about the validity of approaches.
posted by proj at 10:58 AM on September 28, 2010 [1 favorite]
And, if that's not what you were asking, I apologize.
posted by proj at 10:58 AM on September 28, 2010
posted by proj at 10:58 AM on September 28, 2010
I have to second that the things you're asking in the second question don't seem to really have anything to do with social science as opposed to something else. They're questions about statistical analysis in general.
posted by ROU_Xenophobe at 11:46 AM on September 28, 2010
posted by ROU_Xenophobe at 11:46 AM on September 28, 2010
A Primer for Policy Analysis by Stokey and Zekhauser is a classic, and pretty persuasive, work in favor of using all sorts of quantitative methods for analyzing policy options. It doesn't really go into the larger issue of using quantitative methods, but does show how these specific methods can be effective.
posted by lunasol at 12:13 PM on September 28, 2010
posted by lunasol at 12:13 PM on September 28, 2010
Response by poster: I appreciate that this is largely a question for statisticians. I'm interested in the commentary on applicability of those methods to social science by prospective social scientists, along the lines of Bardolph's City Journal article. Whether or not social scientists should comment on these issues is, for the time being, not as relevant to me.
posted by l33tpolicywonk at 1:58 PM on September 28, 2010
posted by l33tpolicywonk at 1:58 PM on September 28, 2010
Why? Not to be snippy. But if you're just looking for something interesting to read, that's different from looking for sources for a paper or similar, and both of those are very different from considering (more) graduate work and trying to narrow down the programs you might apply to.
posted by ROU_Xenophobe at 3:10 PM on September 28, 2010
posted by ROU_Xenophobe at 3:10 PM on September 28, 2010
Response by poster: ROU_Xenophobe: "Why?"
Primarily to better understand the viability of quantitative methods in social science and form a more informed opinion. I work in, and plan to go back to school in, public policy (specifically education) where the legitimacy of field trials to answer broad questions (i.e. do performance pay systems increase teacher quality?) is often questioned. Though the methods of particular studies are often what's under fire, there are also pundits who try to argue quantitative work has little to nothing to say about how systems interact with individuals, create incentives, etc.
I'd like to understand this discussion in the social sciences in general to see if it sheds any light on the questions about methodology in ed policy / public policy in particular. My understanding of statistical methods is elementary but growing, and my understanding of social science theory and history is fairly robust.
posted by l33tpolicywonk at 3:20 PM on September 28, 2010
Primarily to better understand the viability of quantitative methods in social science and form a more informed opinion. I work in, and plan to go back to school in, public policy (specifically education) where the legitimacy of field trials to answer broad questions (i.e. do performance pay systems increase teacher quality?) is often questioned. Though the methods of particular studies are often what's under fire, there are also pundits who try to argue quantitative work has little to nothing to say about how systems interact with individuals, create incentives, etc.
I'd like to understand this discussion in the social sciences in general to see if it sheds any light on the questions about methodology in ed policy / public policy in particular. My understanding of statistical methods is elementary but growing, and my understanding of social science theory and history is fairly robust.
posted by l33tpolicywonk at 3:20 PM on September 28, 2010
Response by poster: I should add, ROU, that if your sense (as it seems to be) is that this isn't really a debate, that quantitative research has won and that us policy wonks are just being sticklers about it because we don't like the results, that that's a welcome / valid answer too.
posted by l33tpolicywonk at 3:31 PM on September 28, 2010
posted by l33tpolicywonk at 3:31 PM on September 28, 2010
if your sense (as it seems to be) is that this isn't really a debate
That's not quite it.
My point was only that if you were considering graduate work in political science, you should know that in the field of American politics, the quants won. You don't necessarily have to do the stuff yourself, but you absolutely have to be able to read it.
I do think that if you want to end up in a position of being able to usefully critique quantitative studies, you have to know enough of the material yourself to be able to spot the particular holes in the particular study. Likewise, even if you hate the stuff, you need to be able to think and analyze in homo economicus terms in order to counter arguments that are made on that basis.
Even if you want to be a stickler because you don't like the results, the first step is knowing enough to be a stickler.
posted by ROU_Xenophobe at 5:19 PM on September 28, 2010
That's not quite it.
My point was only that if you were considering graduate work in political science, you should know that in the field of American politics, the quants won. You don't necessarily have to do the stuff yourself, but you absolutely have to be able to read it.
I do think that if you want to end up in a position of being able to usefully critique quantitative studies, you have to know enough of the material yourself to be able to spot the particular holes in the particular study. Likewise, even if you hate the stuff, you need to be able to think and analyze in homo economicus terms in order to counter arguments that are made on that basis.
Even if you want to be a stickler because you don't like the results, the first step is knowing enough to be a stickler.
posted by ROU_Xenophobe at 5:19 PM on September 28, 2010
This thread is closed to new comments.
posted by Bardolph at 9:29 AM on September 28, 2010