Analyzing ranking data

March 6, 2006 10:26 AM

StatsFilter: How to analyze a set of ranking data?

I ran a survey. I have a set of rankings: 10 items where each respondent uniquely ranked each item 1-10. But... I don't really know what to do with it. How do I generate a group ranking based on the individual rankings? Total the ranking vales for each item? Average them? Also, any references to a text that describes how to analyze this kind of data are welcome.

posted by GuyZero to science & nature (8 comments total)

It sorta depends what you want to know - you can work out standard deviations to tell you how consistent your rankers are on each item or group of items. Means will tell you stuff, medians will tell you slightly different stuff...

If you want to find out how related the rankings are you can use correlation tests - this can be used to determine whether people or groups of people give similar answers to questions. For ranked data you want Spearman's Rho (wikipedia) or Kendall's Tau (wikipedia). Both of these are testable for significance, if you need that sort of thing.

"Nonparametric statistics for the behavioural sciences" is a book which covers all sorts of tests, but it's a library visit rather than a book anyone in their right mind would want to purchase.
posted by handee at 10:37 AM on March 6, 2006

Check this similar Askme thread.
posted by mbd1mbd1 at 10:57 AM on March 6, 2006

I saw that thread, thanks. But I only have one set of data and I don't need to correlate it with anything else. My main desire is to translate a set of individual preferences into a group preference in a statistically valid way.
posted by GuyZero at 11:05 AM on March 6, 2006

To half-answer my own question, anyone know how to do a one-dimensional least-squares fit in Excel?
posted by GuyZero at 11:17 AM on March 6, 2006

To further answer my own question, please don't mock me for taking half an hour to realize that a one-dimensional least-squares is equivalent to the average. *sigh*
posted by GuyZero at 11:28 AM on March 6, 2006

You can use an election method like the Condorcet method to generate a group ranking. Assuming there are no ties or "circular ties," the Condorcet method, the group ranking has the following property: If A is ranked before B in the group ranking, then a majority of the individuals also rank A before B.

Other election methods like Single Transferable Vote may produce different results; it can be useful to try more than one and then analyze any differences in the results.
posted by mbrubeck at 12:41 PM on March 6, 2006

This is what you seem to be asking:

If 10 people each have their own individual preferences, what are their group preferences? What is the correct way to aggregate their preferences?

It turns out that this is a fundamentally insoluble problem -- there is no globally correct way to aggregate individual preferences; every possible way is terrible in at least one respect. Ken Arrow got a Nobel for showing this.

But that's at a more philosophical or technical level. For your purposes, I would do two things:

(1) Average ranking
(2) Borda count. This runs sort of like a car-racing championship. Each alternative get N-1 points for being at the top of someone's preference ordering, N-2 for being in second place, etc, and 0 for being in last place. Then the winner is the alternative with the most points.

Do both; neither should be difficult to do in Excel. If they come out the same, great. If not, it might well depend on the particulars of the things that people are ranking what sort of method makes most sense.
posted by ROU_Xenophobe at 12:57 PM on March 6, 2006

If my math is right, the average ranking and the Borda count will produce the same ranking, as long as every individual has uniquely ranked every item.
posted by mbrubeck at 2:22 PM on March 6, 2006