Statistical analysis of votes for a certain outcome
July 10, 2013 3:58 PM Subscribe
We have a group of six people with 55 different options. Each member of the party has to vote for each option under 8 different analyses i.e. appearance, distinctness, etc. The options are quality weighted.
The scores by each member are between 0 and 10.
So each option can score a maximum of 8 x 10 x 6.
The output required is the options ranked.
I would like to have a complex set of analyses that looks at:
1. Ranking by total
2. Average position within each persons ranking
But fear I am missing some huge statistical opportunities and concepts to establish the truest ranking from a purist perspective. One that maybe correlates the average score each person gives with there subjective view point (i.e if the give an average score of 3 that should weight no less than a person who gives an average score of 7).
My question is, how can I analyse this data quickly and complexly.
The scores by each member are between 0 and 10.
So each option can score a maximum of 8 x 10 x 6.
The output required is the options ranked.
I would like to have a complex set of analyses that looks at:
1. Ranking by total
2. Average position within each persons ranking
But fear I am missing some huge statistical opportunities and concepts to establish the truest ranking from a purist perspective. One that maybe correlates the average score each person gives with there subjective view point (i.e if the give an average score of 3 that should weight no less than a person who gives an average score of 7).
My question is, how can I analyse this data quickly and complexly.
Best answer: So each judge is making 55 x 8 scores from 0 to 10, is that right?
One thing you could do is scale and center each judge's scores. To do that you'd need to calculate the mean and the standard deviation of their scores. To center, subtract the mean from all scores, and then to scale, divide by the standard deviation. What this would do is convert all the scores to numbers like -0.1, 0, 0.5, 1.4, -0.3, etc. They can be read as "a score of (e.g.) 1.3 means this judge thought this option was above average by 1.3 standard deviations". If there are 8 categories for each of the 55 options, you might scale and center each of the 8 categories (for each judge) independently of one another before combining them, to correct for judges having differing behaviors when they score the 8 categories.
Subtracting the mean centers all the scores at zero and correct for the problem of some judges giving lower scores (on average) than others.
Dividing by the standard deviation rescales the scores and corrects for the problem of some judges having more tightly bunched scores than others. In other words, if one judge has most scores bunched around 3, 3.1, 2.9, 3.4, and then throws out a 6, that three-point jump is more important than a 3-point jump for a judge who scores 2, 7, 3, 1, 10, etc.
As for doing the math, there's quite a learning curve, but once you're proficient it's very simple in R or other statistical software (try RStudio, it's free).
posted by PercussivePaul at 4:28 PM on July 10, 2013 [1 favorite]
One thing you could do is scale and center each judge's scores. To do that you'd need to calculate the mean and the standard deviation of their scores. To center, subtract the mean from all scores, and then to scale, divide by the standard deviation. What this would do is convert all the scores to numbers like -0.1, 0, 0.5, 1.4, -0.3, etc. They can be read as "a score of (e.g.) 1.3 means this judge thought this option was above average by 1.3 standard deviations". If there are 8 categories for each of the 55 options, you might scale and center each of the 8 categories (for each judge) independently of one another before combining them, to correct for judges having differing behaviors when they score the 8 categories.
Subtracting the mean centers all the scores at zero and correct for the problem of some judges giving lower scores (on average) than others.
Dividing by the standard deviation rescales the scores and corrects for the problem of some judges having more tightly bunched scores than others. In other words, if one judge has most scores bunched around 3, 3.1, 2.9, 3.4, and then throws out a 6, that three-point jump is more important than a 3-point jump for a judge who scores 2, 7, 3, 1, 10, etc.
As for doing the math, there's quite a learning curve, but once you're proficient it's very simple in R or other statistical software (try RStudio, it's free).
posted by PercussivePaul at 4:28 PM on July 10, 2013 [1 favorite]
Best answer: This is not the type of problem to which Arrow's theorem applies, if that's reassuring.
If it were me, I would just straight up assign each of the 55 options the average of the 48 scores it received (6 raters x 8 criteria.) Anything beyond that starts to feel like complexity for complexity's sake. I wouldn't object if you rescaled to fixed standard deviation as PercussivePaul suggests, but I don't think I'd feel the results were any more "true" or "pure" than the simple average.
posted by escabeche at 7:56 PM on July 10, 2013 [2 favorites]
If it were me, I would just straight up assign each of the 55 options the average of the 48 scores it received (6 raters x 8 criteria.) Anything beyond that starts to feel like complexity for complexity's sake. I wouldn't object if you rescaled to fixed standard deviation as PercussivePaul suggests, but I don't think I'd feel the results were any more "true" or "pure" than the simple average.
posted by escabeche at 7:56 PM on July 10, 2013 [2 favorites]
You could try making some plots, too (scatter plots and histograms). Also, looking at correlations between characteristics can give you an idea of what it is that the judges are rating (maybe appearance and distinctness mean almost the same thing to the judges).
If you want fancier analyses, you could also do a factor analysis on the scores and could even compare patterns of ratings across judges. I don't think this is really necessary, and unless you fancy learning a lot of statistics, I would stick with standard descriptive statistics (means, medians, standard deviations, and correlations) and some plots. Hope this helps!
posted by zscore at 6:09 AM on July 11, 2013
If you want fancier analyses, you could also do a factor analysis on the scores and could even compare patterns of ratings across judges. I don't think this is really necessary, and unless you fancy learning a lot of statistics, I would stick with standard descriptive statistics (means, medians, standard deviations, and correlations) and some plots. Hope this helps!
posted by zscore at 6:09 AM on July 11, 2013
This thread is closed to new comments.
posted by Ashlyth at 4:08 PM on July 10, 2013 [1 favorite]