How do I interpret statistics from an ESP test?

December 21, 2005 6:41 PM

How do I interpret the statistical results of an ESP test? At this ESP testing site there's a tabulation of the daily results of people from across the world attempting to guess which of 5 cards is a target card. In the results for 12/20/2005 the top scorer had 13 hits in 25 trials for a hit rate of 52%, random chance would be 0.20. Now the website says the odds against this are 2708 to 1. First question: is this correct? Second question: if it is correct, then how weird is this statistically speaking given that there were a total of 19,886 trials by 189 users.

Incidentally in 19,886 trials there were 4008 hits for a hit rate of .202 which is not far from random. But for some reason the website states that this is "overall odds of 2.4 to 1" . Shouldn't the overall odds be about 1 to 1?

I've taken an undergrad course in Stats and another in Probability but I'm rusty (to let you know what expertise level you're addressing your answer to).

Thanks for any help!

posted by storybored to science & nature (22 comments total)

I'm not speaking to the statistical implications other than to say out of 20k trials, someone is bound to have gotten at least 50%+ (look at the bell curve).
posted by kcm at 6:55 PM on December 21, 2005

If the odds are truly 2708 to 1, and there have been 19886 trials, the expected ocurrence of the 52% hit rate is 7.3, so the odd thing is that only one person got it.
posted by signal at 6:58 PM on December 21, 2005

From the description, it sounds like a "trial" is one person making one guess, so there are really 189 trials, presumably of varying lengths.

There would be about a 7% chance of seeing an event with probability 0.000369 at least once in 189 trials. Or, to put it differently, they should have a day like this about once a fortnight, assuming the rates are consistent from day to day.

Furlongs do not enter into it.
posted by ROU_Xenophobe at 7:08 PM on December 21, 2005

ROU_Xeno, how exactly did you get your answer? (man, my probability is *really* rusty).
posted by storybored at 7:23 PM on December 21, 2005

Probability of someone doing it = 1 - Probability of no one doing it
Probability of no one doing it = (1 - 0.000369) ^ 189 = 0.933
so 1 - 0.933 = 0.067ish
posted by milkrate at 7:39 PM on December 21, 2005

I got my answer from the binomial distribution, using stata for convenience (the bitesti command).

Milkrate's answer is correct for the degenerate case of only looking for at least one "hit," but for the general problem you need to go to the cumulative binomial distribution looking for at-least-x hits in n trials with an independent probability-of-a-hit of p in each trial.

You can do it by hand if you really want to using a cumbersome but not really difficult formula, or you can do it on a computer easily enough.
posted by ROU_Xenophobe at 8:04 PM on December 21, 2005

I'm not totally sure what kind of answer you're looking for... I cannot answer with any heavy statistics, but I can tell you that I'm pretty sure that's terribly insignificant data.

The only thing I can add to this discussion is about what you said: "random chance would be 0.20." I just completed the course Research Methods in Psychology and I assume that value is the same as the significance level. The significance (p-value) represents the probability that you obtained your results purely by chance. Researchers often aim to get LESS than 5% or sometimes 1% chance of that, to ensure their results are significant and reliable and can be reproduced in duplicate trials. For most researchers (in psychology anyway), if they obtained a p-value of .20 (which means there is a TWENTY percent chance that their results were obtained merely by chance alone) they would throw out the results, considering them insignificant. Some will throw out results if p=.055, let alone p=0.20, and ESPECIALLY if that's the result for only one person!!

Personally, I'd be very skeptical of that. Perhaps the person running the site did no actual stat calculations, but merely psychically envisioned that "2708 to 1" number. :)
posted by mojabunni at 9:42 PM on December 21, 2005

This reminds me a bit of the sports-betting scam. Pick X people, say, 1024. Mail all of them a recommendation for a sports game. Send half of them a rec for team A and half a rec for team B. Obviously, you will be right for half of them. Send THOSE 512 people another rec, half for team C and half for team D. You'll be right half the time. Keep going. Eventually you'll reach a pool of people for whom you have been right several times in a row. You can tailor this by how many people you select initially. With 1024 people, you will be able to have given the last guy 10 correct guesses in a row. Now, you ask him for money to give him the next pick. Why wouldn't he pay you? You have a *100%* success rate over 10 trials. If you chose more people initiall you could make it a 100% success rate over any number of trials you like.
posted by RustyBrooks at 9:44 PM on December 21, 2005

Another thing to think about is something I've realized from playing poker. The longer and longer I play, the more likely that freaky things will happen to me. I'm a pretty tight player so I usually only play about 1 in 4 hands, or 25%. So I have minimum criteria for hand selection and generally over time, that criteria means I will play on average one in 4 hands. Very often I will go 10 hands without a playable hand. Less often, but still often, I will go 20 hands. I've gone as long as *50 hands in a row* with no playable hands. That was a very frustrating day. No doubt, I will someday go 100 hands in a row. Someone watching might conclude that I was the worlds least lucky man. They might spout statistics about how the odds of such a thing happening were X to 1. But it has happened and will happen again. Any pool of trials that is sufficiently large will have lots of freaky events in it.
posted by RustyBrooks at 9:49 PM on December 21, 2005

(By the way, when I play online I track such things. To be able to visually see that you are having a very unlikey series of bad cards can be comforting. Also, when you're hitting good cards, it can control your enthusiasm for you to see that you are getting lucky.)
posted by RustyBrooks at 9:50 PM on December 21, 2005

_{The only thing I can add to this discussion is about what you said: "random chance would be 0.20." I just completed the course Research Methods in Psychology and I assume that value is the same as the significance level. The significance (p-value) represents the probability that you obtained your results purely by chance. Researchers often aim to get LESS than 5% or sometimes 1% chance of that, to ensure their results are significant and reliable and can be reproduced in duplicate trials. For most researchers (in psychology anyway), if they obtained a p-value of .20 (which means there is a TWENTY percent chance that their results were obtained merely by chance alone) they would throw out the results, considering them insignificant. Some will throw out results if p=.055, let alone p=0.20, and ESPECIALLY if that's the result for only one person!!}

uhhh... you're reading way too much into what he said. The chance of a person guessing the correct card out of the five is 1/5, or 0.2. The 20% figure quoted is just that, and has nothing to do with anything you just said.
posted by delmoi at 10:47 PM on December 21, 2005

Anyway, to understand what's going on you have to take each individual trial separately.

If you want a metaphor, imagine a huge pool of 'trials' that already exist. Each trial is a five-sided object where each side has an equal probability of coming up. However, only one of the sides is labeled 'winner'

When a user does a 'trial' they are in essence throwing one of these dice.

So the question is, if a user takes twenty five tests, how likely is it that they win 13 times?

What you want to do is count the number of possible combinations of trials that have thirteen 'wins'.

Imagine each side of the trial has a number on it. In that case, each 'game' could be represented as a string of random numbers like this:

132131235345134123512341.

Now let's say that five is the winning side. In this set of 25 throws, the person won three times, probably because I only have four fingers to moosh the number-row with :P

There are 5²⁵ possible games that involve 25 turns. The probability of any one game is 1/5²⁵, so the probability of getting 13 right is x/5²⁵, where x is the number of possible games where 13 of the trials are winners.

Figuring out exactly that requires assorted combinations and permutations that I'm way too tired to figure out at the moment.
posted by delmoi at 11:17 PM on December 21, 2005

Anyway, in a computer simulation of 10,000,000 25-trial games, the computer 'guessed' the answer correctly 13 of the 25 trials 2944 times.

Why? Couldn't tell you...

Good thing pascal didn't have a computer! :P
posted by delmoi at 11:40 PM on December 21, 2005

One last thing to think about:

There were 19,886 trials by 189 users, which is an average of 102 trials per user. Our superstar psychic only did 25 trials so the probability that someone would have gotten 13/25 is dependant on the number of people who did exactly 25 trials.

If we assume that there were 19886/25 = 785 users today who each did exactly 25 trials, uh then I would say that there would be about 2.7 people getting 13/25.

Now I really do need to go to sleep.
posted by delmoi at 11:51 PM on December 21, 2005

and of course, the more trials you do, the less likely you'll be able to keep up your 52% l33tness.
posted by delmoi at 11:53 PM on December 21, 2005

The formula for calculating the probability of getting exactly k hits out of n trials, when the probability of getting a hit from a single trial is p, is given by (p^k)* ((1-p)^(n-k)) * the binomial coefficient (n; k). Using this formula where k is 13, n is 25, and p is .2 gives a value of about .000293. The odds of getting exactly 13 hits from 25 trials are thus about 3515 to 1. However, adding to this the possibilities of getting 14, 15, . . . 25 hits from 25 shows that the odds of getting at least 13 hits from 25 trials are 2708.67 to 1. So the website (forgiving the rounding error) is right on that score.

Milkrate has the right approach to the second question. If there were 189 runs of 25 trials, you would expect somebody to get at least 13 6.7% of the time. So it's unusual, but not outlandish.
posted by muhonnin at 1:48 AM on December 22, 2005

RustyBrooks writes "This reminds me a bit of the sports-betting scam."

That's pretty slick.
posted by Mitheral at 7:30 AM on December 22, 2005

I'm realising that I was slightly unclear just above. What I meant to say was that if you had many batches of 189 runs, each consisting of 25 trials, you would expect 6.7% of those batches to contain one run where a person got at least 13 of 25 hits.
posted by muhonnin at 8:42 AM on December 22, 2005

Could a computer program actually be a good measurement for ESP, though? If you were using actual cards, the value would be unquestionable -- there would be a real answer that someone would actually know (who is showing you the back of the card), not a randomly selected card.
posted by vanoakenfold at 9:58 AM on December 22, 2005

Wow great answers! Thanks to everyone who posted.

Special thanks to Milkrate, Muhonnin and ROU_Xenophobe for helping me recover some of my long-buried probability chops.

Mojabunni, the concept of p-values is an interesting one i hope to read up on.

Oh and here's the Epilog.

Taking the answer that the probability is around 7% means that there will a 52% chance of this weird thing happening within ten days. Soooooo... I went back through the data at gotpsi.com and checked each day's top scorer.

Lo and behold on Dec 10th (eleven days before the Dec 20th incident) user Cmumaugh scored 13 hits out of 25 tries!

This is so neat.

Thought: one reason why people tend to believe in paranormal phenomena is that weird things happen to them. But this little exercise shows Weird = normal.
posted by storybored at 10:23 AM on December 22, 2005

Vanoakenfield, it all depends on what the theoretical underpinning is of ESP you are relying on.

If we take the standard definition of clairvoyance which is the awareness of events/information outside of the normal senses, then computer-run experiments are valid. Not only that, the ones on the Internet are great for running large scale experiments.

If you require that there be a human intermediary in your definition of ESP, then clearly computer experiments don't apply. Sadly though, i don't think it makes much difference, because there have not been any repeatable real-life experiments that have succeeded either AFAIK. (although it's possible that could change at any time).
posted by storybored at 10:34 AM on December 22, 2005

Also Delmoi, i hope you got the use of your fifth finger back :-)
posted by storybored at 10:36 AM on December 22, 2005