Can you answer this question (un)truthfully?

February 27, 2008 11:12 PM

(Psychology) Questionnaires designed to elicit the truth?

Over a decade ago I came across some research-paper PDF that dealt with how surveys can be designed to maximize the truthfulness of respondents' answers - I'm pretty sure it involved something like "double blind camouflage" - I think this is the concept of a question being superficially about one thing, while also being relevant to some other matter.

I've had no luck digging this paper up, or determining what field it belongs to - any pointers?

posted by unmake to science & nature (8 comments total)

Another strategy you see invoked is asking the same question with different ranges of responses—how strongly do you agree with the President's recent decision to ? (Strongly Agree, Agree, Neither, Disagree, Strongly Disagree.)

Then, further down, "do you agree with the President's recent decision to ?" If they answer Strongly Agree and No, then you know the data's not quite what you're looking for.

You might be able to play with the wording between the two in order to not raise any direct flags, but that can introduce other noise to the experiment.
posted by disillusioned at 2:04 AM on February 28, 2008

What you might be looking for is called the ipsative, or forced response method of creating a questionnaire.

More on general questionnaire design here (pdf).
posted by SteveInMaine at 3:45 AM on February 28, 2008 [1 favorite]

Marketing researchers call it funnelling. The survey first asks very general, easy-to-answer questions to put the respondent at ease, then the questions become more specific. If you have a lot of questions (30-40), break them up into subcategories and renumber them so it so the respondent isn't immediately discouraged with the volume of questions. To uncover the respondent's latent motives, use numeric rating scales:

Excellent 10 9 8 7 6 5 4 3 2 1 Poor

or a constant sum methed where the respondent must divide up a predetermined dollar amount or point total between several choices.

Also, remove the neutral midpoint (ie undecided) and force them to choose either a negative ot positive answer.

There's way more to designing questionaires than that (of course), but this is what I can remember off the top of my head. I hope it helps.
posted by PoopyDoop at 7:13 AM on February 28, 2008

Bayesian truth serum

This is not the primary source but can lead you there.
posted by dragonsi55 at 10:02 AM on February 28, 2008

I have heard of adding a random variable to obscure the results for questions unlikely to receive honest answers. Flip a coin, and if heads, mark yes, if tails answer the question normally. That way, they can still get a statistically accurate data in aggregate without any individual admitting somthing embarassing.

No reference, however
posted by jpdoane at 1:06 PM on February 28, 2008

It's called the randomized response technique. I'm looking for more source material to link you to, but it's all behind paywalls. It was a big deal awhile back and got favorable coverage in Science Magazine, of all places. But survey methodologists have concerns about it, and it isn't widely used, AFAIK.

Basically, respondents have two questions, the sensitive one and an innocuous one. They also have a randomization device and only the respondent knows which question they answered truthfully.
posted by jasper411 at 4:26 PM on February 28, 2008

In sociology we have a lot of issues with survey design. Some topics make it nearly impossible to get consistently "true" responses - sexual behavior and drug use spring to mind. There are also issues with not just how the questions are worded, but who is administering the survey, even if the results are confidential and anonymous. Someone that reminds you of your grandma handing you a survey (or asking aloud) might make you more likely to answer in a "socially acceptable way." Additionally, the questions before might influence the answer to sensitive topics.

One thing we do use a lot is the Likert scale (There are lots of other scales for sociology, Guttman comes to mind). Another tactic is to apply redundancy to your measurement instrument. So, one question might be "do you torture small animals?" (Yes, no, don't know, refuse to answer) and later you might see "Have you ever torn the wings off butterflies?" (Yes, no, don't know, refuse to answer). If your respondent answers no to the first questions and yes to the first you have...issues, perhaps with what we call "validity."

Now, we study aggregates, and not particular cases, but the individual answers are still obviously important. If you are interested in how these questions effect the work we do, you'll want to look into "research methods" and the subheading "survey design."
posted by bilabial at 1:44 PM on March 1, 2008

Questionnaires sometimes include questions that are meant to test your level of honesty. They ask questions like, "True or false: There are some people you just can't stand," or "True or false: I've sometimes pretended to be sick to get out of something." These statements are true for almost everyone—people who answer false are presumed to be lying to try to make themselves look good.
posted by incandescentman at 10:20 AM on March 25, 2008