Contests and Competitions: Mitigating Uneven Heats

June 19, 2024 7:33 AM

I'm involved in a large-scale competition structured in heats. There's an inherent unfairness in this. I'm wondering if anyone can suggest solutions to mitigate it.

Contest structure: Competitors are randomly assigned to heats of 8-10. They are scored on performance against specific qualitative criteria (not against others in their heat). The top scoring competitors in each heat go on to to a next round of competition, eventually resulting in an overall winner.

The problem: Because of random assignment, some heats end up containing a stronger group of competitors than others. That means that a competitor who might have placed first or second in a weaker heat will place middle or last in a stronger heat. Therefore, the winners of weaker heats go on to next round, while stronger competitors never get out of the first round.

Assume:

There's no way to pre-screen the competitors to assign them into heats of even quality

Random assignment to heats is required by the structure

There is no way to have them enter the heats with weights or handicaps on their initial standing - everyone shows up on an even playing field

there is not time or capacity to add "loser's heats" where there can be a second attempt to qualify

I appreciate that this may be intractable, but am grateful for any ideas. Happy to answer questions, but I'm leaving the content and style of competition neutral so we can focus on structure.

posted by Miko to Sports, Hobbies, & Recreation (15 answers total)

If the score is independent of how others in the same heat perform, why can't you just take the top X scorers from all heats regardless of which heat they were in or how they compared to others in their heat?

So if you had 100 people competing in 10 heats of 10, the people moving on to the next round would be the top 10 scorers out of the whole group of 100, who would not each have to be from different heats. Is there a reason this doesn't work? Do you want/need to have an award given out for each heat?
posted by Redstart at 7:45 AM on June 19 [15 favorites]

If the competitors aren't compared to each other, why not just do all the heats, record everyone's scores, then sort the list to find the top X performers, who advance to the next round?

Generally speaking many competitions have unfairness for the sake of excitement or practicality. And we haven't even defined which kind of fairness we're trying to optimize! Do we want everyone to have an equal chance of winning, despite their objective skill, or do we want to set it up so the most skilled dominate?
posted by mrgoldenbrown at 7:55 AM on June 19 [1 favorite]

Can you add wildcard spots to the finals?

As an example, in the Jeopardy TOC they have (had? I don't know if they still do this now) five first round games, and the winner of each game goes to the semi-finals. But they need five semi-finalists, so in addition to the first round winners, they move on the top 4 scorers among the non-winners.

So if you're currently advancing the top 2 in each heat, advance the winner of each heat, plus the highest scoring competitors among non-winners until you have filled out your next round.
posted by jacquilynne at 8:04 AM on June 19 [5 favorites]

The usual approach is to take the top x from each heat plus the y fastest losers. But that means you need fewer heats than you have places in the next round which might not be do-able. This looks like it's the same as what jacquilynne is proposing. It's the way it's often done in track and field athletics. You'd need to translate the 'fastest loser' metric into your existing scoring system but perhaps that's do-able.
posted by plonkee at 8:46 AM on June 19

Long after the edit window, just noting that I meant 'But they need NINE semi-finalists' not five semi-finalists.
posted by jacquilynne at 8:57 AM on June 19

I've been involved with competitions that used jacquilynne's method, where times got compared to the whole field, and that worked fine. If time permitted, they might have two races and average the times to determine who went to which final. (This helped even out issues that couldn't be helped, like changing weather between heats.)
posted by mersen at 9:08 AM on June 19 [1 favorite]

Response by poster: why can't you just take the top X scorers from all heats regardless of which heat they were in or how they compared to others in their heat

This makes sense to me. Our challenge here is that instead of numerical scores or times, our scoring system is a rubric with seven categories, and a 5-point Likert scale for each category. Because there are only 5 percentiles for each category, the likelihood of many duplicate scores is very high. So we'd need a way to distinguish between duplicate scores.

An alternative is to go to point-based scoring, but there's resistance to that.
posted by Miko at 11:21 AM on June 19

How do you pick a winner in a heat if your scoring is a 7 digit string with entries of 5 possible scores? You're already converting to a linear ranking somehow. Is it just a sum, and you hope that the heats are random enough that there's no ties?

A few things to consider: you can put weights on the categories. This can effectively spread out your summed scores. But you'd have to agree on the weighting and also publicize it to the players, who then might prioritize that category. Which could be good or bad, hard to say.

I really like radar charts/aka spiderweb charts for this kind of medium-high dimensional data. You can do all sorts of stuff with that visualization, like quickly identify who is well-rounded, who excels at each part the most, etc. You can take the area of the polygon as a linear metric of how good the entry is at covering all the bases. And you can combine this visualization with the weighting idea too.
posted by SaltySalticid at 11:38 AM on June 19

Do you need to have heats at all? It sounds like the competition is really based on scores not heats, so why not just let each competitor do their thing three times, average their score and then rank everyone? If there are ties, then you could allow those tied competitors to go one more round to break the tie.

Or if that's too many individual events, then have everyone go one round, drop out the bottom half of the field, have the top half go another round, drop half again and have the final top group go a third round. If competitors are competing concurrently in these heats, they can still do so, they just won't be competing specifically against the people in their heat.
posted by ssg at 11:57 AM on June 19 [3 favorites]

Random assignment to heats is required by the structure

If the random assignment to heats is fixed, what actually is allowed to be changed? I see suggestions so far to change (or, really, ignore) the heat structure, either with or without also changing the scoring system. But aside from major structural changes like that, I'm unsure of what properties of this competition are considered open for modification.
posted by mhum at 1:11 PM on June 19

If you do just rank everyone regardless of heat, worth considering if the endeavor is one where performance is likely to be heavily influenced by those around you - for instance, being neck to neck in a race often will make folks swim faster than being way out ahead, so you might well want to include the winner from each heat even if they had a slower time than the second or third place person from the best heat. Different types of fairness, basically - an argument for "best from each heat + X overall top competitors".
posted by Lady Li at 4:48 PM on June 19 [1 favorite]

The tennis grand slams have this problem as well. What they want to avoid is two stars meeting in an early match, for the same reasons you describe: it's not fair to the stars, or to the lower-ranked players who deserve a chance to play a Federer or a Williams. They solve it through 'seeding', a system ranking players based on their past performance and world ranking, so that higher-ranked players are distributed evenly and fairly, and ideally don't meet before a later round match or final. There is some element of randomness; that within a rank cohort the players don't choose who they play, but it's designed to obtain fairness, not a random distribution.

If random distribution is required and no pre-screening is possible, it's really very hard to solve your problem. You may have to consider evening out some other way, with a partial round-robin or by increasing the number (best of 3?) of heats.
posted by Fiasco da Gama at 5:22 PM on June 19

Or, make the first round a genuinely random distribution, but use it as a qualification round, without an elimination component, to distribute strong and weak competitors.
posted by Fiasco da Gama at 5:28 PM on June 19

Because there are only 5 percentiles for each category, the likelihood of many duplicate scores is very high. So we'd need a way to distinguish between duplicate scores.

What are you doing now to distinguish between duplicate scores? If duplicate scores are that likely, you must be running into them already. What happens if two people in a heat are tied for the top score? Do both people advance? Does the judge adjust the scoring before it's announced to make sure there isn't a tie? Do you apply some tie-breaking criteria?
posted by Redstart at 11:56 AM on June 20

Response by poster: In the case of a tie on the scoring rubric, the judges confer and decide on the final ranking. That component is ultimately subjective.

One of the commitments is that we don’t have more than two rounds, prelims and finals. Doing more than that would exhaust both competitors and judges (who are volunteers).

The seed idea is interesting. Doing a pre-screen to seed strong entries evenly has been floated as an idea.

Taking the top half of each would help if we could manage the resulting number of final rounds.

I agree that within the parameters of the event, this is really difficult to solve. I’ve wondered if it’s intractable, and the answers are helping me see which commitments/constraints are cashing the issue. Thanks for the analyses.
posted by Miko at 4:36 AM on June 21