I cho-choose you

July 9, 2008 4:56 AM

I am a member of an academic committee that handles award nominations. Our job is to chose which of many internal applicants may apply for an external award. Often such awards accept just 1 or 2 applicants from a single organization. What is the most transparent and fair process for making such decisions? Alternatively, what are problems you've faced on similar committees?

In terms of transparency, I am looking for a process that allows us to say ahead of time "here are the rules by which we will make all decisions" and then afterwards "here are the decisions we made in this particular case and why". The people who are submitting applications are my peers, that is, I can't simply say "I'm the boss, that's why". I'm also concerned with bruised egos from saying, "X is better than you."

I don't need suggestions for a full process from me-fi, just some rules or vague suggestions for what you think is important. For example, "If it is the last year that someone is eligible for an award, they receive a preference" as some awards are early career awards.

Thanks for you help.

posted by about_time to work & money (8 comments total)

I like the one that you've thought of for yourself.

Also, making all of the discussion public afterward should help. It should force you to be fair and diplomatic if you know that anyone can read your reasons later.

If it really comes down to it, I would not be against a coin flip/dice roll/etc. But that should be saved as a last resort that should only be used if you really can't make a decision between people.

Depending on the award, the rules part might be taken care of for you. Look at the award criteria, and that's what you're looking for in the applicants.

Also, saying something that says you're picking people who have the best chance of winning the award would be a good idea. Because in the end you're not really saying that X is better than Y, just that X has a better chance of winning than Y.
posted by theichibun at 5:48 AM on July 9, 2008

(1) Divorce the application from the applicant. Just as brilliant students can hand in crappy work, brilliant colleagues can turn in crappy applications. Clearly state that the applications will be judged on their merits alone, without regard for previous record or accomplishments. This ensures that the best applications pass, and will also help avoid bruised egos, because you are saying "X's application is better than yours" rather than "X is better than you." In this respect, I would avoid giving preference to those in the last year of eligibility. [NOTE: disregard this if you have solid information that the external award does take the applicant's record into account].

(2) Make sure that the committee clearly identifies the selection criteria ahead of time, communicates these to applicants, and then sticks by them when making the selection. Ultimately, the criteria themselves are less important than the consistency with which they are applied. The biggest problem that I have run into in similar situations is that the committee abandons the criteria after the fact (e.g., "well, this application doesn't expressly meet the criteria that we set out, but its really cool, so let's give it a pass"). This can lead some applicants to be justifiably upset if they are passed over for something that didn't meet the criteria.

(3) Link the scoring to the criteria. For example, if you have 5 criteria, rank each on a ten-point scale for a total of 50 points, so that the strengths and weaknesses can be easily compared and clearly communicated to applicants.

(4) Link your criteria to the external award's criteria, if available, and communicate this to your applicants. Stipulate that your selection is intended to choose the applications most likely to win the award, not necessarily the one's that you think are 'best.'
So if the external award values novelty, then you should too; if it values timeliness, then you should to - even if that is not what you would prefer on your own.
posted by googly at 5:57 AM on July 9, 2008 [3 favorites]

I'm on the other end. I work for an organization that has several honors requiring institutional nominations.

Strictly-sticking to clearly-stated criteria is the most important transparency factor. Print out their criteria and stick to it. If you have questions, call the granting/awarding agency for clarification. We're pretty disappointed when we get institutional nominations that are for individuals who clearly conduct excellent research, but really are not appropriate for the particular honor.

Use a score. Industry-standard scoring can help take the heat off of you. For instance, we use a scoring system similar to NIH.

Sorry to contradict you, theichibun, but no, no coin-flips. What you need is a Selection Chairperson, who has final say. Providing applicants with constructive criticism from the review/ranking process is nice, but distributing the selection discussion is dicey -- the selection should be able to be a frank discussion not intended for the applicant's ears.

No need to repeat googly's excellent advice -- all of it is right on.
posted by desuetude at 6:33 AM on July 9, 2008

googly's got it. The only thing I would add, which might mitigate the "but I deserved to get it" sentiment of non-winning applicants, would be to do a one-page letter to each of them, outlining the strengths of their application, and where the committee felt they needed improvement. Yes, it's more work and an extra step, but many applicants really appreciate the feedback, especially if there's a chance they'll apply again the following year.
posted by LN at 6:49 AM on July 9, 2008

There is only one standard you should care about, and that is nominating the person or people who you believe are most likely to receive the award. There's no sense in any other nomination. You should not link your criteria to the award committee's, you simply use theirs as you think they will use it.

A bonus to people in their last year of eligibility is a bad idea unless the external group uses that, or you suspect that they pay attention to such matters informally. If you want to reward people in their last year of eligibility, create an internal award where that's a criterion.

If the award itself is based in part on someone's overall reputation and record, which is another way of saying that the reviewers aren't blinded, then you should absolutely not pass over bad applications from stellar people. Instead, you should choose the project from the person that is most likely to appeal to the committee, and if the application is bad funnel it through your grant-writing department for improvement before sending it out.

The simplest thing in general will be single-blind reviewers. Pass each out to three people, along with the criteria who are credibly not-entirely-incompetent at the relevant field. Ask them to rank them by perceived likelihood of success, and do a Borda count.
posted by ROU_Xenophobe at 7:34 AM on July 9, 2008

Can you anonymize the submissions, so you don't know the submitter until after the evaluation? That would help remove the politics.
posted by jenkinsEar at 7:36 AM on July 9, 2008

Thanks to everyone for the great suggestions! This is a great discussion. There are several points I had not thought of. Publishing criteria ahead of time and sticking to it seems to me to be the most important point.

JenkinsEar, unfortunately, it is not possible to anonymize submissions as a CV and publication record is typically part of the package. Plus my world is small enough that we could easily pick out who did it.
posted by about_time at 2:33 PM on July 9, 2008

If necessary, remind them that this is not a popularity contest -- submissions which are not chosen don't suck, they're just not right FOR THIS AWARD. A good Selection Committee Chairperson is an enormous blessing -- if the discussion starts getting too biased, a strong Chair will tactfully restate the purpose of the award and get it back on track.

Invoke your high and mighty principles, too. Peer review is the gold standard for a reason.

We don't usually give preference toward those nearing the end of eligibility -- actually, just the opposite, as those who are on the cusp of ineligibility naturally have more than a bit of advantage. If the aim is to, for example, reward outstanding work by a postdoc, then a senior postdoc/research associate who's 7 years out from her Ph.D. will certainly have a more impressive track record of publications...but is this person really the spirit of the award? That said, we wouldn't bias against such a person if they really have the most outstanding application.

Okay, I can go on for ages, I do this sort of thing a lot. Feel free to keep replying or MeMail me if you want to pick my brain on processes.
posted by desuetude at 6:57 PM on July 9, 2008