Help me remember something I once read about estimation

April 21, 2007 5:44 PM

WisdomOfCrowdsFilter: I read a theory smewhere that there's a peculiar thing that happens when you ask a group of people to estimate something. As I remember it, there's something about the magic number of 50%, but I'm not wholly sure how this works. The idea was basically that when at least 50% of the individuals were estimating properly in some way, the average estimate would eventually converge on the true value, but when less then 50% of the individuals estimated properly, widely divergent values would emerge. Essentially, more then 50% or _something_ produced good estimates, less then 50% produced estimates that were substantially worse than those produced by just one individual. I've spent some time with Google and Wikipedia on Estimation and Delphi methods, but to no avial. Any mifites know what I'm talking about?
posted by zachlipton to science & nature (19 comments total)

The Birthday Paradox: in a group of 23 (or more) randomly chosen people, the probability is more than 50% that some pair of them will have the same birthday.
posted by googly at 5:56 PM on April 21, 2007

Not exactly your description, but sounded like it might be a candidate.
posted by googly at 5:57 PM on April 21, 2007

Maybe it's in James Surowiecki's The Wisdom of Crowds? Or, failing that, the work of Malcolm Gladwell?
posted by box at 6:07 PM on April 21, 2007

One anecdote from Wisdom of the Crowds sounds similar but I don't remember the specifics. This CNN article had a short description::

In 1968, when the submarine Scorpion went down in the north Atlantic, the navy had only the most general idea of where it was.

Yet, using the expertise of various experts in diverse disciplines, and combining them through mathematical formulas, a naval officer managed to determine its resting place within 220 yards.
posted by sexymofo at 6:25 PM on April 21, 2007

This has nothing to do with the birthday paradox.

I believe you're talking about the Condorcet Jury Theorem.
posted by Wolfdog at 6:43 PM on April 21, 2007 [1 favorite]

It doesn't sound like the Scorpion incident is at all related, either. Mathematical formulas and a panel of experts do not a random crowd make.
posted by odinsdream at 6:47 PM on April 21, 2007

I seem to remember a science fiction story in which something like this figured; the scheme was called Oracle if I remember correctly. (Of course, Googling for that would be hopeless.)
posted by kindall at 7:02 PM on April 21, 2007

Don't know about the actual question, but kindall is probably thinking of the Delphi Pool in John Brunner's "Shockwave Rider."
posted by dersins at 7:39 PM on April 21, 2007

I second Wolfdog -- you are very likely to be remembering an approximation of the jury theorem. Though some among us may share a birthday.
posted by Clyde Mnestra at 7:46 PM on April 21, 2007

Yeah, could well be the Brunner.

Of course, even if I'd remembered it was called Delphi, that wouldn't be any more help in Googling it.
posted by kindall at 8:03 PM on April 21, 2007

I second the Wisdom of Crowds thing. The example I always heard was asking people to guess number of jelly beans in a jar.
posted by eggplantia5 at 8:24 PM on April 21, 2007

If the "Wisdom of Crowds" theory holds true, would a group of people taking the same exam collectively answer every question correctly?
posted by realblanka at 10:36 PM on April 21, 2007

Could be a result of the 'diversity prediction theorem': Collective error = average individual error – prediction diversity
posted by MetaMonkey at 10:44 PM on April 21, 2007

Forgot the link: pdf, google cache
posted by MetaMonkey at 10:48 PM on April 21, 2007

I had to think about this for a while, then woke up with it. Bayesian analysis was used to help thhe U.S. Navy recover an atomic weapon off the coast of Spain. Estimates were aggregated, outliers thrown out, and the resulting predictions were tightly clustered.

They found it. Can't remember the source.
posted by Phred182 at 6:10 AM on April 22, 2007

Sorry Sexymofo, it was the Scorpion, as you say. A brief explanation of the Bayesian analysis appears in Blind Man's Bluff, p.64.
posted by Phred182 at 6:14 AM on April 22, 2007

Thank you everyone. I am pretty sure that what I am remembering involved some form of Condorcet's Jury Theorem, although I certainly appreciate all the pointers to all the other cool stuff this post dredged up. Now, off to read more about the Jury Theorem!
posted by zachlipton at 7:45 AM on April 22, 2007

It also sounds like some version of the "least squares" may relate to what you remember. You use least squares when you have a lot of observations, each of which will contain some error, but when those errors are "averaged out," the large set of observations tends toward accuracy. An example is when a bunch of astronomers give their reports for the positions of stars.
posted by cgc373 at 4:52 AM on April 23, 2007

As I understand "least squares," nothing about it turns on having more than individual involved. Here, we can keep churning out independent estimations, but the jury already returned Condorcet -- or at least that's what the foreman heard.
posted by Clyde Mnestra at 8:20 AM on April 23, 2007