What is this type of survey called?
June 24, 2018 8:39 AM Subscribe
I'm interested in performing a survey to get people to rank 10–15 items. I would like them to be presented these items in randomized pairs, choosing the one they think has a greater value of the variable I'm interested in. This would be repeated over and over until a stable ranking is achieved. What is this kind of survey called?
I initially thought "discrete choice experiment" but that doesn't actually seem right.
Side questions:
(1) In what sorts of situations is this kind of survey preferable over presenting the entire list to the respondents and just asking them to rank them?
(2) Is there a way to convert these rankings into some kind of arbitrary 1–10 scale?
I initially thought "discrete choice experiment" but that doesn't actually seem right.
Side questions:
(1) In what sorts of situations is this kind of survey preferable over presenting the entire list to the respondents and just asking them to rank them?
(2) Is there a way to convert these rankings into some kind of arbitrary 1–10 scale?
Best answer: There’s a website called All Our Ideas (can’t link, walking) that facilitates exactly this - you enter in a bunch of terms and it presents participants with two at a time to rank. I call it forced choice but there are probably other terms.
posted by OrangeVelour at 9:14 AM on June 24, 2018
posted by OrangeVelour at 9:14 AM on June 24, 2018
I would normally call this a two-alternative forced choice (2AFC) experiment.
I agree with supercres that it has the advantage of being less cognitively demanding, and thus potentially less noisy. However, pairwise comparisons don't always reflect the same information as full-list rankings. A common example is movie popcorn prices and purchasing choices. Say a small is $3, a medium is $7.50, and a large is $8. Given a menu with small vs. large or small vs. medium, people may choose small because large and medium are too expensive. For medium vs. large, people will usually pick large because it's a better deal (only 50 cents for much more popcorn!). Given that small "wins" in its two pairwise comparisons, you'd expect it to be the most popular option when all three options are presented together, but in fact people typically end up picking the large size for the full menu. (I made up these dollar amounts but the principle generally holds.)
Depending on the nature of the variable you're trying to investigate, it may be more or less likely that this kind of phenomenon will occur. And if it does occur, then depending on the exact question you are interested in answering, a 2AFC or a full ranked ordering survey may be the more appropriate method. Without more detail it's impossible to know which is going to be a better choice for you. Note also that if you design your 2AFC survey to keep asking questions until a stable ranking is found, but there is an actual intransitivity in your subjects' responses, your survey may never actually end.
As for converting the pairwise comparisons into a global ranking, there are a variety of ways to do that. You can take a nonparametric approach, in which case you should look into rank statistics, or a parametric approach, in which case you should look into logistic regression.
posted by biogeo at 12:45 PM on June 24, 2018 [1 favorite]
I agree with supercres that it has the advantage of being less cognitively demanding, and thus potentially less noisy. However, pairwise comparisons don't always reflect the same information as full-list rankings. A common example is movie popcorn prices and purchasing choices. Say a small is $3, a medium is $7.50, and a large is $8. Given a menu with small vs. large or small vs. medium, people may choose small because large and medium are too expensive. For medium vs. large, people will usually pick large because it's a better deal (only 50 cents for much more popcorn!). Given that small "wins" in its two pairwise comparisons, you'd expect it to be the most popular option when all three options are presented together, but in fact people typically end up picking the large size for the full menu. (I made up these dollar amounts but the principle generally holds.)
Depending on the nature of the variable you're trying to investigate, it may be more or less likely that this kind of phenomenon will occur. And if it does occur, then depending on the exact question you are interested in answering, a 2AFC or a full ranked ordering survey may be the more appropriate method. Without more detail it's impossible to know which is going to be a better choice for you. Note also that if you design your 2AFC survey to keep asking questions until a stable ranking is found, but there is an actual intransitivity in your subjects' responses, your survey may never actually end.
As for converting the pairwise comparisons into a global ranking, there are a variety of ways to do that. You can take a nonparametric approach, in which case you should look into rank statistics, or a parametric approach, in which case you should look into logistic regression.
posted by biogeo at 12:45 PM on June 24, 2018 [1 favorite]
The other classic way to go from pairwise to a global ranking is the Elo rating.
posted by Homeboy Trouble at 3:31 PM on June 24, 2018
posted by Homeboy Trouble at 3:31 PM on June 24, 2018
This thread is closed to new comments.
posted by supercres at 9:11 AM on June 24, 2018 [2 favorites]