Changing instruments in social research - values vs validity
June 11, 2015 2:51 AM Subscribe
I am looking for articles, case studies and examples in which researchers in the social sciences change (or consider changing, or are called on to change) research instruments (questionnaires, rating scales, assessment measures) to accommodate changes in social values. Specifically, I am interested in the discourse and debate around these changes, and the ways in which researchers attempt to manage the inherent statistical risks to internal validity instrument change entails. What should I read?
The changes could be in any field or context (academic study, internal evaluation, action research...), and for any reason. Maybe the racial categories on an intake form were last reviewed in 1967, or the baseline evaluation of a project used gender categories that weren't trans*-inclusive, or a standard assessment measure is written in language that is inaccessible and overly complex, or users of a service want a yearly questionnaire changed to better align with their values and needs. Maybe the old research instruments just plain suck, and a researcher wants to take the opportunity to create something better.
Whatever the reason for the change, am interested in how researchers have managed the tensions between maintaining validity (Wah! We can't change those forms, we won't be able to compare the new data to the old data!) and ensuring the research remains relevant and appropriate to its social context (Gah! We can't use those forms, question 3 is potentially offensive to 17% of our client base!). I'm especially interested in the discourse and debate surrounding such changes, how various sides framed their arguments, and how any changes were managed in terms of methodology and statistical analysis.
The changes could be in any field or context (academic study, internal evaluation, action research...), and for any reason. Maybe the racial categories on an intake form were last reviewed in 1967, or the baseline evaluation of a project used gender categories that weren't trans*-inclusive, or a standard assessment measure is written in language that is inaccessible and overly complex, or users of a service want a yearly questionnaire changed to better align with their values and needs. Maybe the old research instruments just plain suck, and a researcher wants to take the opportunity to create something better.
Whatever the reason for the change, am interested in how researchers have managed the tensions between maintaining validity (Wah! We can't change those forms, we won't be able to compare the new data to the old data!) and ensuring the research remains relevant and appropriate to its social context (Gah! We can't use those forms, question 3 is potentially offensive to 17% of our client base!). I'm especially interested in the discourse and debate surrounding such changes, how various sides framed their arguments, and how any changes were managed in terms of methodology and statistical analysis.
Race classifications in the Census have changed many times over its history. Here is a recent discussion; Googling will provide more details of each change.
posted by metasarah at 8:02 AM on June 11, 2015
posted by metasarah at 8:02 AM on June 11, 2015
Here's another example that might suit your needs. The NSDUH has been redesigned a couple of times. See here, here, here and here. In my experience, these days there's not a lot of discussion around redesiging a survey's demographic categories, since most Federally funded surveys are required to use certain categories in order to get through the approval process. It's not up for debate. This kind of discussion tends to be around other content areas, and like the NSDUH example reflects changes in behavior or markets, not social mores.
posted by OrangeDisk at 3:31 PM on June 11, 2015
posted by OrangeDisk at 3:31 PM on June 11, 2015
This thread is closed to new comments.
posted by OrangeDisk at 8:01 AM on June 11, 2015