It's a piece of cake to bake a pretty cake

June 14, 2012 12:45 PM

How do I score my fantasy cake contest in the most fair manner?

I run a large online cake contest (link in profile) every year that attracts about 300-500 entries.

It's special in that it has a panel of judges who review and cast their ballots (which was the topic of this previous Ask of mine) for their favorites, instead of being a Facebook popularity contest.

This year, I'm introducing a fantasy component: Any user will be able to sign in and rank their top ten at the end of the contest. The closest match to the judge's aggregate-scored top ten will win a prize.

My question is: how do I score the entries?

Naturally, if someone gets all 10 right and in order, then I'll simply look at the timestamp for when they "locked" their entry and call the earliest the winner.

But: what if no one gets all ten? Do I more heavily weight people who got the first three places correct? Do I assign a point for every cake they had that's also in the top ten, and then a bonus for getting the position right? Do I assign more points if they're closer in rank?

(Perhaps if you had a cake for 5th place and it won 4th, you get more points than if you had it for 10th?)

Or do I just look at number of matches in position and work off that and a timestamp?

I'm inclined to think that someone who got all of them off by one place should be higher ranked than someone who got exactly one in position and no others...

What are your thoughts? Something that can be relatively easily implemented would be helpful... Anything from suggestions on criteria to an entire approach would be nice.

Also wondering about if two cakes tie for a position in judges points how I approach that on the fantasy side... I guess I allow them to be considered matches for either of the two places they straddle?

My rough idea:

Each cake that you have that is also on the final scoresheet is worth a point.
If you match first place, you get 7 bonus points.
If you match second place, you get 6 bonus points.
If you match third place, you get 5 bonus points.
If you match any other position, you get 3 bonus points.
If you are within one of any position, you get 2 bonus points.
If you are within two of any position, you get 1 bonus point.

Does this sound like it'd work? Am I missing anything? I haven't run through any scenarios yet, but I'll put together a few and see what happens.

Thanks!

posted by disillusioned to technology (9 answers total)

The judges will score the cakes. That aggregate score will determine a rank.

The user will then rank the top ten cakes...

Use the sum of the differences in rank (between the user and the judges) to determine the winner. The smaller the difference the better the prediction... Someone that got it perfect would have a zero difference sum. Someone that got it completely backwards would have a very high sum. You could award bonus points for any arbitrary element of the prediction (like having any of the top three dead-on).

This has the advantage of being relatively straightforward to compute on a spreadsheet as well...
posted by milqman at 12:57 PM on June 14, 2012

Clarification question: Will the users rank their top 10 out of all 500 entries or only have to rank the judges' top 10?
posted by griseus at 1:01 PM on June 14, 2012

Off the top of my head, I'd probably do the scoring like this:

2 points for each cake in the top 10 plus 3 points for those in the correct position. So, you can get 5 points per cake for a maximum of 50 points.

I think that any more complicated scenarios could introduce unintended consequences. For example, in milqman's scenario, someone that matched 2 through 9 and had 1 and 10 reversed would do worse than someone that had all cakes off by 1 position.
posted by Jacob G at 1:04 PM on June 14, 2012 [1 favorite]

Yeah, sorry for the clarification: users will build their own list from the TOTAL set of entries, not just of the top ten finalists.

(So they could very possibly build an entire ballot of cakes that aren't on the final scoresheet.)
posted by disillusioned at 1:49 PM on June 14, 2012

*for the confusion. gah.
posted by disillusioned at 1:50 PM on June 14, 2012

Jacob G, what would be wrong with someone who had 1 and 10 mixed up getting a lower score than someone who had all the cakes off by one? That seems right to me.

If you otherwise like milqman's idea, I think you could use a fixed number (higher than 10) for the cakes the users incorrectly predicted would be in the top 10. The higher the fixed number, the greater the emphasis on correctly picking the top 10 vs. correctly ordering the picks that were in the top 10. E.g., if you have two user ballots x-2-3-4-5-6-7-8-9-10 and 10-2-3-4-5-6-7-8-9-1 (where the number is the judge's placement and "x" did not place in the top 10), and you decide the fixed number is 11, then the scores would be 10 and 18 respectively, so user #1 wins. If your fixed number is 25, then user #2 would win (24 pts versus 18 pts).

Personally I think it makes more sense for user #2 to win in this scenario, since there are so many entries that "top 10" versus "not top 10" seems like a bigger distinction than "tenth place" versus "first place". (Under your scoring system, user #2 loses by three points.)
posted by purplecrackers at 8:28 PM on June 14, 2012

If nobody gets the top ten all correct, I would next look at users who got the top nine, then top eight, etc. It's so much simpler than these arbitrary point systems!
posted by vasi at 2:50 AM on June 15, 2012

purplecrackers - In my example, in both cases, both participants had correctly identified the top 10 cakes. However, the "1-10 switch" had 8 cakes in the correct position and the "off by one" had 0 cakes in the correct position. Seems more fair that the person who nailed 8 of them wins the contest.
posted by Jacob G at 6:44 AM on June 15, 2012

The likelihood that anyone will nail *any* of the cakes in order when picking 10 out of a set of 500 is very limited, unless there's a huge, clear difference in the quality of cakes submitted, such that the top 10 is pretty obvious. It's more likely you'll be trying to break ties among 2 or 3 people who picked 5 of the top 10 (regardless of position on their list) than you are to be breaking ties among people who picked all 10 of the top 10, even if they were in entirely the wrong order.

I would do this:

Have people submit a list of 10 cakes they think will be the top 10. No need to order them.

Rank people by how many of the actual top 10 appear on their list. If you have a tie for the winner -- say a few people all got 6 of the 10 -- then give them all a rank based on the actual finishing positions of the cakes they selected. So, if the 6 they right were 1, 2, 4, 5, 7 and 9, they'd get a score of 28, while someone who picked 1, 2, 3, 6, 8 and 10 would have a score of 30. Lowest score wins.

If you still have a tie, factor in the finishing positions of the 4 cakes that weren't in their top 10 in the same way, so that lowest score still wins.

If you really, really want to order them, then I think I'd go with a straight sum of the difference between where they ranked each of their 10 cakes and where the judges ranked them. But that'll mean somebody who got the first 9 cakes bang on and then picked cake 472 for their 10th position (for a total differential of 462) will potentially lose to someone who got none of the first 10 cakes right, but instead picked cakes 11-20 (because they'd be off by an average of 10 points, for a total differential of about 100).
posted by jacquilynne at 6:58 AM on June 16, 2012