How do social scientists, psychologists, etc. detect and control for people lying in response to their questionnaires?
I have no training at all in the social sciences but find some of the research absolutely fascinating.
I read a slow trickle of articles and listen to a couple of social science podcasts* which often feature interviews with the scientists who conducted the study. Every time I read or hear about a study that relies on survey or interview data, I'm struck by the fact that no-one ever mentions the possibility that the answers given were misleading.
For a few examples:
(a) In a survey about drug use and sexual activity that I completed in high school, I and almost all my peers eventually admitted to each other that we'd lied. Boys had generally exaggurated their experience (although a few regular drug-takers strongly downplayed this) and girls generally downplayed their experience. [Of course, it's possible that we were lying to each other about lying on the survey...]
(b) I recently listened to a podcast about a study on teenagers in gangs. Along with aggregate data, individual responses were quoted and discussed; neither the study author nor the interviewer mentioned the possibility that these teenage boys were exaggurating their answers to make themselves sound tough.
(c) Recently on Metafilter, we discussed a study on the development of lesbian couples' children (
here) that relied entirely on the parents' testimony. The authors didn't seem to question their testimony at any point, e.g. the possibility that lesbian parents might feel their families to be subject to a lot of scrutiny, and so be more motivated to present their kids' behaviour in a good light than hetero parents.**
It's well-established that people tend to stretch the truth to make ourselves look good and that, through selective formation and distortion of memories, we actually tend to remember events and patterns in a light that flatters our self-image and agrees with our pre-conceptions. So for any given survey response, there seem to be at least four possibilities:
1) The response is a full and accurate description of the situation
2) The respondant believes the response to be true, but their memory is biased and/or incomplete
3) The respondant believes the response to be mostly true, but with some deliberately altered details to make themselves look good
4) The response is mostly or entirely a deliberate lie
My gut instinct is that (2) and (3) are the most likely classes of response that people will give. However, all the papers I've read or interviews I've listened to seem to treat the answers as if they're (1).
So, my questions are:
(a) Is it really generally assumed that all responses are honest and accurate? How is this assumption justified?
(b) If not, how do researchers distinguish between honest-and-accurate, honest-but-wrong and dishonest answers?
(c) Is there good data indicating how certain populations tend to lie about certain responses (e.g. "take a teenage boy's reported sexual encounters and divide by three"!), or is trustworthiness just based on gut instinct?
(d) Do surveys or interviews usually include error-checking questions, and how are the sensitivity and selectivity of these deception-detecting questions validated?
*The BBC's Thinking Allowed is a great social science podcast, if you're interested.
**NB: I'm NOT saying that this is necessarily the case, as I don't have any information. I'm just using it as a recent example of a potential strong motivation for participants to lie, in a paper where the authors appear to take all the responses at face value.
Since you know what percentage of men like to drink tea (either it's common knowledge or you find it out in a separate survey), you can then calculate how many of them beat their wives, by taking into account the deviance from that number.
Sorry if my explanation is a bit sloppy, I'm also sure there's a scientific term for it, but I can't remember.
posted by cronholio at 6:07 AM on June 29, 2010