How can I create a qualitative assessment of policy documents?

September 2, 2008 9:11 PM

I've been asked to devise a qualitative means of analyzing multiple similar policy documents for the purpose of critically evaluating them (and hopefully publishing my results) and frankly, I'm floundering. Any advice on books, classes, or ways to learn how to approach this problem?

I'm a grad student at Cal and have a quantitative background, but need to really dive into qualitative reasoning. Honestly, I'm feeling totally overwhelmed- I've read all of the policy papers and have some general ideas of the questions I should be asking of the them through the evaluation-

does length correlate to impact/quality?
do better written/more incisive policy documents get better utilization?
how much of an impact does abysmal spelling/grammar have on the end result?

etc, etc, but I have no idea how to approach this as a research project, my adviser is busy with more important matters, and I'm reaching the end of my tether and considering buying a one-way ticket to Timbuktu where I can become a nomadic sheepherder and range the arid Sahara without again hearing the word 'dissertation'.

Any classes or books that the community can recommend or people I could talk to who would know what I should be reading to know how to devise a plan of research? I'm at Cal and in Environmental Sciences, if that is pertinent.

help? thanks.

posted by arnicae to Education (4 answers total)

Start by reading Miles & Huberman's foundation guide to qualitative data analysis (Miles, M., & Huberman, M. (1994). Qualitative Data Analysis : An Expanded Sourcebook (2nd Edition). London: Sage.) and then investigate Strauss & Corbin's grounded theory (Strauss, A. L., & Corbin, J. (1990). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. London: Sage.).

Both of these should give you some good ideas. But above all, don't fall into the trap of thinking that qualitative approaches are methodologically lax--they require tremendous amounts of time and analytical clarity.

E-mail me if you want to talk more.
posted by yellowcandy at 10:36 PM on September 2, 2008

I was going to jump in with recommending grounded theory, but yellowcandy beat me to it.

I was also going to say that I think you need to reframe your questions, because the ones you have here don't really sound like qualitative questions to me (I think of correlation as a statistical term, not an interpretive one). But I am in a completely different field (communication) so perhaps I just don't quit understand what you're trying to do with these documents.

Another possibility that strikes me is that you could do a rhetorical analysis, if your goal is to have a critical evaluation, particularly if it's possible to look at the policy documents as "arguments" of a sort.
posted by DiscourseMarker at 11:09 PM on September 2, 2008

This page has a bibliography of rhetorical analysis/criticism that looks pretty comprehensive.
posted by DiscourseMarker at 11:16 PM on September 2, 2008

On utilization - there's a tonne of literature on that (check out for example two of the big hitters Carole Weiss "the many meanings of research utilization" Public Administration Review, 1979; Kingdon "Agendas, alternatives and public policies" 1995, 2nd edn.). Basically, utilization isn't about the document itself. It's about the policy context, the links the policy-maker has, how fashionable the policy is.

If you're limited to the policy paper itself, there's a few things you can do with the way that recommendations are formulated - are they tailored to clear policy-makers? Are they in a language that can be understood? Is the evidence clearly formulated, convincing and persuasive? Is it concise? Check out this (small pdf), this [citation] and this [pdf] for overviews.
posted by YouRebelScum at 2:25 AM on September 3, 2008