What's the fairest way to rank entries to a creative contest?

April 29, 2008 5:13 PM

I ran a creative contest. 17 people submitted. 8 judges each chose their top three, in order of preference. What is the fairest way to rank each entry?
posted by stokast to Grab Bag (15 answers total)

Is there some reason you're shying away from the obvious: 3 points for each first place vote, 2 for each second place vote, 1 point for each third place vote?
posted by dersins at 5:15 PM on April 29, 2008

Assign 3 points to each judge's #1 pick, 2 to each #2 pick, and 1 to each #3 pick. Add up the points for each submission and you'll hopefully have no ties. If there are ties, consider giving firsts 4 points. If that doesn't help, perhaps allow for ties.
posted by pmbuko at 5:19 PM on April 29, 2008

Response by poster: Because I'm leery of solutions that seem too easy and have never been good at math? :)
posted by stokast at 5:21 PM on April 29, 2008

Response by poster: Is that really the best way to do it?
posted by stokast at 5:22 PM on April 29, 2008

Are you looking for a mathematical proof of fairness in case one (or more) of your entrants complains about the scoring system?
posted by pmbuko at 5:29 PM on April 29, 2008

Best answer: Essentially, you need a Voting system. dersins and pmbuko have proposed something kind of like the Borda count system. You could also use preferential voting, but it's probably not right for your needs.

I don't think you're going to be able to rank all 17 properly using your current votes. You'll be able to pick winners, but you'll probably have a group of submissions that can't be ranked due to not having any votes.
posted by zamboni at 5:44 PM on April 29, 2008

For what it's worth, the points system that both pmbuko and I suggested is essentially the same as is commonly used to rank things like college basketball teams and the various Most Valuable Player awards across professional sports.
posted by dersins at 6:01 PM on April 29, 2008

Response by poster: Thanks zamboni... Those articles are great. And I'm not worried about ranking all of them necessarily, just finding a proper way to rank each of the ones people voted for.

Thanks, too, to dersins and pmbuko. I'm off to do some counting.
posted by stokast at 6:17 PM on April 29, 2008

I hate to be a downer, but there is no fair way to count the votes.
posted by Dec One at 6:19 PM on April 29, 2008 [1 favorite]

When I took my turn at the helm of RipRap -- the annual literary journal of Cal State University Long Beach -- I used a combination of dersins' and pmbuko's scoring system and editorial capriciousness.

We totaled all the scores and I gave the editors the opportunity to wrangle over points and to horsetrade. The editors managed to settle all the ties without too much input from me (other than setting the ground rules).

It worked out pretty well. I was happy with the system I'd set-up until I learned why Editor So-and-So was so damn determined to see Kinda Marginal Poem make the final cut. It belonged to that editor of course, contributed under a pseudonym (well outside the rules we had agreed to -- and well outside the bounds of decent behavior).
posted by notyou at 6:25 PM on April 29, 2008

Seconding the notion that you have a flawed premise if you assume there is a single "best way" of voting. See also Condorcet's paradox. But good on you for trying to be as fair as possible!
posted by chinston at 8:14 PM on April 29, 2008

I do this sort of thing A LOT in my dayjob.

We would assign points as mentioned by dersins. The highest-ranked candidates would be considered the shortlist. Shortlisted candidates are then reconsidered by the committee and again ranked/scored. (Without comparison to the lesser candidates, the dynamics can change. Or some just reaffirm their original score, which is also fine.)

For the number of candidates you've got, the top two or three would then be discussed, with a re-reminder of criteria, then a vote taken to confirm the decision emerging from discussion.
posted by desuetude at 8:33 PM on April 29, 2008

And though it's too late to make any difference in this contest, I need to point out that you now have no options that are immune from the "unfair" label, because you know the judges' scoring and can be accused of having picked one weighting method or another after the fact to guarantee that your favorite wins. (I'm not accusing you of this myself; if that were your intent you wouldn't be asking the question.) To my mind, picking one scoring method versus another is less important than deciding on that method before the judging has occurred, announcing it ahead of time, and sticking with it.

So what you really need is a method with a lengthy Wikipedia article to give it some weight as a "standard" method, as insulation against this charge.
posted by range at 8:48 PM on April 29, 2008

Best answer: It's too late to help you this time, but the single most important criteria for having a fair judging algorithm is that People Know What It Is In Advance. (And that you stick with the announced criteria.)

I've run a few minor creative contests of little consequence, and people will accept almost anything as the algorithm, as long as it makes even a tiny bit of sense, and they know how it works, and you don't have a chance to change it after the scores come in.

I've started to favor giving the weighting to the judges -- give them all a set number of points to distribute to their favorites in any way they see fit. If you give them all 10 points, for example, some of them will give 4 to 1st, 3 to 2nd, 2 to 1st and 1 to 4th. Others will give the story they think is best 6 points and 2 each to a couple of other stories they liked. Some will give 10 stories they liked each a point.

It eliminates the need for the judges to determine a strict ranking of things, which is kind of strange when comparing creative things anyway, and but gives them more ability to call out wide gaps in quality. If one entry is far and away the best in their minds, and then there are a group of 3 with some merit and all the rest kind of suck, they can reflect that in their scoring in a way that 3 for 1st, 2 for 2nd, 1 for 3rd doesn't really do.
posted by jacquilynne at 8:18 AM on April 30, 2008

Response by poster: Thanks all. This is really interesting stuff.
posted by stokast at 3:34 PM on April 30, 2008