Discouraging people from lying on a pre-screening survey?
November 9, 2014 8:56 AM Subscribe
How do I accurately screen people on a survey when my screening criteria seem fairly obvious. How do I prevent them from lying just to pass the screening portion?
I'm trying to pre-screen survey participants to see if they have ever given money to fund a certain category of project on any crowd-sourcing website (e.g. Kickstarter, Indiegogo, etc). I'm concerned that if my criteria are too obvious, that people will simply lie and say they have in order to pass the screening portion of the survey. Does anyone have any experience or ideas on how to discourage participants from doing this? Obviously I can't prevent 100% of this, but would like to minimize it as much as possible while keeping the screening portion of the survey as simple as possible. If anyone has any other strategies in general for keeping people from satisficing on surveys, that'd be useful also.
I'm trying to pre-screen survey participants to see if they have ever given money to fund a certain category of project on any crowd-sourcing website (e.g. Kickstarter, Indiegogo, etc). I'm concerned that if my criteria are too obvious, that people will simply lie and say they have in order to pass the screening portion of the survey. Does anyone have any experience or ideas on how to discourage participants from doing this? Obviously I can't prevent 100% of this, but would like to minimize it as much as possible while keeping the screening portion of the survey as simple as possible. If anyone has any other strategies in general for keeping people from satisficing on surveys, that'd be useful also.
You'll need to carefully word your questions so that they don't broadcast your intent. For example, instead of asking "Have you every supported a project from category X", provide a list and ask "Which categories of projects have you supported?" Then it won't be obvious which category you are interested in.
This could be part of a natural progression of questions, most of which you don't care about. "Have you every supported a crowd sourced project? How often? Which websites do you use? Which categories have you supported? Have any projects succeeded/not succeeded?" Etc.
posted by rakaidan at 9:07 AM on November 9, 2014 [4 favorites]
This could be part of a natural progression of questions, most of which you don't care about. "Have you every supported a crowd sourced project? How often? Which websites do you use? Which categories have you supported? Have any projects succeeded/not succeeded?" Etc.
posted by rakaidan at 9:07 AM on November 9, 2014 [4 favorites]
Best answer: What do the people you're screening already know, or think they know, about your survey? I.e.: what was the recruitment process which led them to the point where they're even taking your screening survey in the first place?
posted by feral_goldfish at 9:33 AM on November 9, 2014 [3 favorites]
posted by feral_goldfish at 9:33 AM on November 9, 2014 [3 favorites]
Best answer: About seventy million years ago I worked in market research for a couple years. Misdirection is your friend; basically, you want to hide your objective in plain sight.
e.g.:
In the past 6 months, which of the following have you used the Internet for?
- checking email
- social media (Facebook, Twitter, etc)
- crowdfunding (IndieGoGo, Kickstarter, etc)
- blogging
- reading the news
or
In the past six months, which of the following websites have you visited?
- Facebook
- IndieGoGo
- Twitter
- Kickstarter
- CNN.com
- ....
Yet another option:
Which one of the following payment options do you most often use for transactions online?
- PayPal
- Credit card
- Visa/MC Gift Card
- Bitcoin
Then you f/u with a screen about the kinds of websites they make transactions on, bury crowdfunding in there somewhere, and get to the real questions after that. In my experience, most market research-type surveys will have 3-5 screener questions (where I worked that portion of the survey was called the 'funnel' because it went from wide open--e.g. age range a/o gender--to the actual specific screen) before getting to the meat of things.
For example, one survey I did coding/testing on was testing effectiveness of snow tire advertising (seriously). Couple of quick demographic questions first--age asked, gender assumed from voice (yes ok that's problematic but let's not go there), location preselected by phone number. Then a question about TV shows they watch, based on when the client bought media time. After that, I think it was maybe two more questions to screen for awareness of the ad, and a dependency tree based on if they had seen the ad in question. The logic being, nobody's going to conflate TV viewing habits with snow tires, so you can filter out a bunch of the noise you're worried about.
Your next step depends on whether you are offering an incentive. If you are, and someone gets filtered out, they may just hit their back button to game the system. Easiest way to avoid that is to ask a demographic question or two even if they don't qualify, and thank them for their time--they walk away thinking they gave the answers required.
Your next few questions can drill down to what you're actually asking. As rakaidan said, don't ask for a specific thing, provide a list of possible answers and ask which apply. At least where I was, asking questions that way was called 'pull,' whereas 'have you ever supported X category' was 'push.' So once you've established that they have used a crowdfunding site, then you can screen for whether they've actually funded, then a checkbox list of different categories--after someone has passed all those, then you can ask the questions you actually want to ask.
on preview: feral_goldfish raises an important point; what's your audience?
posted by feckless fecal fear mongering at 9:41 AM on November 9, 2014 [12 favorites]
e.g.:
In the past 6 months, which of the following have you used the Internet for?
- checking email
- social media (Facebook, Twitter, etc)
- crowdfunding (IndieGoGo, Kickstarter, etc)
- blogging
- reading the news
or
In the past six months, which of the following websites have you visited?
- IndieGoGo
- Kickstarter
- CNN.com
- ....
Yet another option:
Which one of the following payment options do you most often use for transactions online?
- PayPal
- Credit card
- Visa/MC Gift Card
- Bitcoin
Then you f/u with a screen about the kinds of websites they make transactions on, bury crowdfunding in there somewhere, and get to the real questions after that. In my experience, most market research-type surveys will have 3-5 screener questions (where I worked that portion of the survey was called the 'funnel' because it went from wide open--e.g. age range a/o gender--to the actual specific screen) before getting to the meat of things.
For example, one survey I did coding/testing on was testing effectiveness of snow tire advertising (seriously). Couple of quick demographic questions first--age asked, gender assumed from voice (yes ok that's problematic but let's not go there), location preselected by phone number. Then a question about TV shows they watch, based on when the client bought media time. After that, I think it was maybe two more questions to screen for awareness of the ad, and a dependency tree based on if they had seen the ad in question. The logic being, nobody's going to conflate TV viewing habits with snow tires, so you can filter out a bunch of the noise you're worried about.
Your next step depends on whether you are offering an incentive. If you are, and someone gets filtered out, they may just hit their back button to game the system. Easiest way to avoid that is to ask a demographic question or two even if they don't qualify, and thank them for their time--they walk away thinking they gave the answers required.
Your next few questions can drill down to what you're actually asking. As rakaidan said, don't ask for a specific thing, provide a list of possible answers and ask which apply. At least where I was, asking questions that way was called 'pull,' whereas 'have you ever supported X category' was 'push.' So once you've established that they have used a crowdfunding site, then you can screen for whether they've actually funded, then a checkbox list of different categories--after someone has passed all those, then you can ask the questions you actually want to ask.
on preview: feral_goldfish raises an important point; what's your audience?
posted by feckless fecal fear mongering at 9:41 AM on November 9, 2014 [12 favorites]
Best answer: I think that feral_goldfish's question is really important here to be able to give any advice about how to minimize satisficing or lying.
I feel like you have a two stage problem here. It sounds like you don't have a very well-defined sampling frame. It also sounds like you are not really looking at whether or not people fund in this category as a variable. It might be worth considering that as well. Where/how is your recruitment being conducted?
Is there an incentive or something associated with your survey that you think would make respondents want to lie in order to complete the survey?
posted by paco758 at 10:30 AM on November 9, 2014 [2 favorites]
I feel like you have a two stage problem here. It sounds like you don't have a very well-defined sampling frame. It also sounds like you are not really looking at whether or not people fund in this category as a variable. It might be worth considering that as well. Where/how is your recruitment being conducted?
Is there an incentive or something associated with your survey that you think would make respondents want to lie in order to complete the survey?
posted by paco758 at 10:30 AM on November 9, 2014 [2 favorites]
Response by poster: FFFM: Those are exactly the type of questions I needed!
Reading all your answers, I realized I was aiming for the wrong sample of people.
Instead of getting people who have funded a specific type of product on crowd funding, I need to sample people who use crowd funding for anything AND are also interested in/use the product. So coming into the survey they might know it's for people interested in "Art Books" or "Comics" or something, but I can much more easily obfuscate the crowdfunding part.
posted by Chicoreus at 12:02 PM on November 9, 2014
Reading all your answers, I realized I was aiming for the wrong sample of people.
Instead of getting people who have funded a specific type of product on crowd funding, I need to sample people who use crowd funding for anything AND are also interested in/use the product. So coming into the survey they might know it's for people interested in "Art Books" or "Comics" or something, but I can much more easily obfuscate the crowdfunding part.
posted by Chicoreus at 12:02 PM on November 9, 2014
At least according to the book "The Honest Truth about Dishonesty", explicitly reminding people about honesty and dishonesty seems to make them temporarily less likely to cheat. So something like adding a "honor code" statement at the top of the survey, saying that they pledge to answer the questions on the survey truthfully, and requiring them to sign it (or something) to indicate that they agree to the pledge, might help somewhat.
posted by Flunkie at 1:04 PM on November 9, 2014 [1 favorite]
posted by Flunkie at 1:04 PM on November 9, 2014 [1 favorite]
« Older How to re-create a poor-quality microfilm scan | Help me maximize a last-minute trip to Disneyland. Newer »
This thread is closed to new comments.
posted by alex1965 at 9:00 AM on November 9, 2014 [10 favorites]