YANMD, but seriously - did I get dumber?

July 18, 2016 3:06 PM

Four years ago, I underwent a thorough evaluation for ADHD and possible learning disabilities. Part of the evaluation was the WAIS-IV, which measures IQ. I was asked to undergo this evaluation again recently, and the IQ result was ten points lower. They tell me it's nothing to worry about, but I panicked! Can someone decipher these results and/or tell me why or why not to panic?

Age: mid-20s.

I was in an accident where I might have suffered a mild concussion a few years ago, but that's about all I can think of. I have 1-3 drinks once a week, and have never used tobacco or any illegal substances. I was for a while prescribed stimulants for the ADHD (had to go off them due to side effects).

posted by Seeking Direction to Health & Fitness (17 answers total)

Your IQ isn't static, and a single assessment of it is not a good baseline. You'd have to take a series of tests over time to get anything resembling an accurate measurement.

So, no, you didn't get dumber. You took two measurements of a value that is variable over time.
posted by late afternoon dreaming hotel at 3:12 PM on July 18, 2016 [7 favorites]

I don't think you should worry, regardless. But the mostly likely explanation here is noise in the measurement, which is why tests have a margin of error. This, doesn't give the margin of error may be 5 points, at least for people with a score of 110 (depending on the form of the function, it's possible the margin of error differs at different points along the scale). According to that book, they shouldn't be giving people results without margins of errors. Can you check your paperwork and see if you got yours?
posted by If only I had a penguin... at 3:33 PM on July 18, 2016

Testing scores can fluctuate for a variety of reasons, such as being in an unfamiliar environment or under stress currently.

They come out with new tests every few years in part because IQ tests are strongly culturally biased. This makes them questionable to begin with. Taking an IQ test in a second language or from a foreign country will tend to hurt your scores.

If you seriously want some idea of what your IQ is, you should be assessed by a professional. I tests are merely tools that can be useful to a trained professional. Assessment can be meaningful. Testing alone is much less meaningful.

Most IQ tests top out around 140 or 145. Anything above, say, about 130 suggests you have reached or exceeded the limits of what the test is capable of measuring.

Alfred Binet gets credited with developing the first intelligence test. The truth is that his test was not intended to evaluate intelligence. It was intended to evaluate school readiness for French children at a time when children born in the country often did not have a birth certificate. So, unlike our current school norms, it did not work to say "If you were born between x and Y dates, you should be enrolled in kindergarten." Additionally, there was substantial cultural disconnect between city kids and country kids. So he was asked to develop a test to help determine school readiness so that country kids were not being thrown to the wolves due to cultural differences plus inadequate information about their age.

The validity of IQ scores gets pretty hotly debated in some circles. But even those who value them will tell you that "best practices" indicates they are tools, not very meaningful in a stand alone scenario.

I suggest you not worry about it.
posted by Michele in California at 3:34 PM on July 18, 2016 [1 favorite]

Not at all. I used to be part of a research group that administered IQ tests. We kept the same participants in our study over many years and would re-do IQ tests every X years (I forget, exactly). No one expected the tests to be exactly the same score every time. Little things change - a different test administrator may give the test slightly differently (they shouldn't, of course, but things do happen), on one day you might be more or less tired, more or less well-fed, more or less distracted by stuff in your personal life. You might mis-read a couple of questions, or the test itself might have changed since you last took it. You might be in a new age bracket now where the test is scored differently (I don't know that this applies to the WAIS in particular, but for some kinds of tests it might).

I couldn't tell you off the top of my head how much variation we'd have to see in a re-administration before going "huh, wait, something is really not right here", but it would definitely have been much more than 10 points. You're well within the margin of error for expected fluctuations in scores over time on re-testing.
posted by Stacey at 3:38 PM on July 18, 2016 [5 favorites]

To clarify the issue of noise:

Did you guess at ANY of the questions either time? If so, getting one or two more guesses right the first time could easily be 10 points (depending on how the test and wrong answers are scored). Did you accidentally click A instead of B, just because stupid fingers? Did you have a brain fart and mix-up counter clockwise and clockwise in some relevant situation? Did you have a cup of coffee before the first test that made your thinking half a second faster?

Anything other than your IQ, insofar as that even exists as a fixed thing, that can make you get a question right or wrong is noise. Surely you can see that there are a lot of things beyond your IQ that could make you get a question right or wrong.

Finally, this isn't noise, but IQs are usually artificially distributed. So an 87% (or whatever) on the test could result in an IQ of 120 or 125 or something else, depending on how other people score on the test. If the test was recalibrated on a new population of test-takers, your score could change even if you gave the exact same answer.
posted by If only I had a penguin... at 3:38 PM on July 18, 2016 [1 favorite]

Now, whenever you hear about somebody's IQ, you can mentally add (or subtract) 10-20 points. Because that's how meaningful it is.
posted by amtho at 3:55 PM on July 18, 2016 [2 favorites]

I took one when I was ten and scored 122, then practiced for a few weeks and got 148 (this was part of a school thing, I wasn't that much of a loser). I am quite sure I didn't gain 26 IQ points in the space of a month, which leads me to think the tests are a load of bollocks. Even ten year old me was a bit sceptical when I got my results back.

You can definitely improve your score by practising so presumably the converse is also true and you can worsen your score by not practising too. I doubt I would get 148 if I went in to the exam cold today. Maybe you were using your verbal reasoning skills more back then?

I mean, my maths skills are way worse than they were when I was 18 because I haven't used most of them for twenty years. I have no doubt I could revise and get back up to speed since I knew it before. The fact I'm rusty says nothing about my underlying intelligence, and I'm really not convinced that IQ tests do either. They are testing how good you are at the tests.
posted by tinkletown at 4:03 PM on July 18, 2016 [2 favorites]

Not all IQ tests score the same.

The boiling point of water in Celsius is 100 degrees, which is the same as 212 degrees in Farenheit; calling it one or the other doesn't make it hotter or colder, both are the boiling point of water. Similarly, different scores on different IQ tests don't mean more or less brainpower, just different scoring.
posted by easily confused at 4:12 PM on July 18, 2016 [4 favorites]

For every IQ score there is a confidence interval, which means that you can say with whatever percent confidence [90 or 95%] the score will fall in the same range if the person is retested. It is likely that the confidence intervals of your scores overlap. The score on an IQ test should never be used as the only piece of information to make a decision [Interviews, observations, questionnaires, assessments of more specific abilities, etc]. 10 points is not a huge difference.
posted by the twistinside at 4:19 PM on July 18, 2016

10 points is not a huge difference.

I don't know where people are getting their info from. The scores on a IQ are standardized. 10 points is a large amount in a test that is normed to have 100 as the mean score and 15 pts representing a standard deviation. 10 points is enough to bump you up or down to a next general category (the difference between high-average to superior)

The real issue though is how its test-retest holds up and it is reported to be .70 to .90 for that. So this variability of your results is well within that range. Also that retest reliability result is low enough to question the psychomentric validity of the scale altogether.
posted by srboisvert at 4:37 PM on July 18, 2016

The real issue though is how its test-retest holds up and it is reported to be .70 to .90 for that. So this variability of your results is well within that range.

Test-retest is measured as a correlation. Translating this into a variability range (i.e. margin of error/confidence interval) can't be done without knowing the standard deviations of the test and retest distributions. Furthermore, it's a correlation, not a measure of whether anything changes. So if everyone's score changes a similar amount, one could have a test-retest correlation of 1.0, even though everyone's score changed. So if for the second test everyone scores 30 points lower or everyone scores 5% higher than the first test, then those outcomes would produce a test-retest correlation of 1. So what you want here isn't the test-retest correlation, but the margin of error/confidence interval.

Incidentally, somoeone mentioned above the confidence intervals for the two tests overlapping. If they do, this doesn't necessarily mean that the results aren't statistically significantly different from one another. You can't go from this set of facts:

Fact 1: There's at least a 5% chance that the score for test 1 should have been 5 points or more lower.
Fact 2: There's at least a 5% chnace that the score for test 2 should have been 5 points or more higher.

To the conclusion: "There's at least a 5% chance that test 1 was 5 points too low and test 2 was 5 pts to high in other words, they were actually the same" The "and" there means you're looking for the joint probability of the two things. What are the chances they were both true at the same time. To calculate that, you need the formula for the linear combination of variances, which requires the variance/standard deviation of the test.

I would assume a change that small is noise, especially since as srboisvert points out the tests are normed (i.e. are mathematically forced to fit a given distribution), which means your answers to the test or the number of answers you got right may not even have changed, but the norming population may have changed.
posted by If only I had a penguin... at 5:04 PM on July 18, 2016 [2 favorites]

I believe age also figures into the results, so you may have move that far in four years (15-20% of your lifetime!) to have an affect on your score.

Also, lots of "quantitative" numbers in your life vary somewhat -- consider it like your height: when you wake up in the morning, you might be a little bit taller than after walking around all day; weigh yourself after having a long sweaty run vs right after dinner; heck, if you travel fast enough your age may vary a little bit off the agreed-upon norm. IQ isn't a written-in-stone, measure-it-like-blood-sugar-to-detect-problems sort of number, it's a method for categorization.
posted by AzraelBrown at 9:17 AM on July 19, 2016

The SEM (standard error of measurement) for the WAIS-IV is about 2.6 points, which means that 68% of the time, a person's measured score will be within + or - 2.6 points (e.g. a 5.2 point range) assuming their "true intelligence" hasn't changed (and a lot of other assumptions).

The 95% confidence interval jumps to roughly double that (e.g. a 10.4 point range).

In other words, if you had two IQ test scores that were 10 points apart, and you wanted to be 95% sure that they actually represent true differences, you couldn't make this argument statistically.

If this is the only thing, I wouldn't worry about it.

If, however, this is just one of many congruent symptoms (memory troubles, bad work performance, social problems due to cognitive difficulties, etc.) then I'd look into it further.
posted by soylent00FF00 at 5:49 PM on July 19, 2016 [1 favorite]

The 95% confidence interval jumps to roughly double that (e.g. a 10.4 point range).

In other words, if you had two IQ test scores that were 10 points apart, and you wanted to be 95% sure that they actually represent true differences, you couldn't make this argument statistically.

Ummm..no... Yes, if that's the standard error then the 95% margin of error is a 10 pt range. However, that doesn't mean "it could be 10 points less or 10 points more." The confidence interval is symmetric around the point estimate. So it's possible the "true value" (insofar as that exists, because even for an individual IQ is surely stochastic) for the first test is really 5 points higher. It's possible the really 5 points lower. While this does look like they could be the same (if 1 is points higher and the other is 5% lower, doesn't that make them the same?), it's not actually accurate to conclude that.

Remember that the first true score could be 5 points higher, sure, or it could be 5 points lower. Or 2 points higher. Or 1.5454 points lower or 7.2343 points higher. So the question is what is the probability that *BOTH* the first score was at least 5 points too high AND the second was at least 5 points too low? overlapping confidence intervals don't tell you that. You can calculate it, but not with the info given here.

But again, I still vote noise or statistical artifact created by the way the score distribution is manipulated.
posted by If only I had a penguin... at 7:03 PM on July 19, 2016

@Penguin makes two good points:

First, that I doubled the distribution by considering plus or minus 10 points - that is wrong. The 95% confidence interval would be roughly plus or minus 5 points, so a 10 point difference would be outside it, but would still be inside the 99% CI (at a sigma of roughly 3.0).

The second argument is whether "the probability that these 2 measurements were within X points of each other" is that same as "the probability that all such measurements are within X points of the true score".

My hunch is that the probabilities are similar, but I also know that when you compare difference scores, your reliability goes down - which actually biases the results closer to what I (erroneously) calculated.

My source says that the SEM of the difference is the square root of the sum of the square of the difference scores for each test - in this case the same test, so SEMdiff = sqrt(SEM*SEM + SEM*SEM) = sqrt(6.76+6.76) = sqrt(13.52) = 3.68.

So using SEMdiff, we calculate the 95% confidence interval as +/- 1.9 * 3.68 = +/- 7 points, roughly. This suggests a difference score of 10 points is still unusual, but somewhere between a 95% and 99% occurrence (e.g. happening less than 5% but more than 1% of the time or in other words, less than one-in-twenty but more common than one in one hundred).

And, there's the "publication bias" aspect as well - it's probably not likely that MeFites who have taken an IQ test twice and got roughly the same score are posting to ask about the small difference.

I would mostly stand by my original advice, saying a 10 point difference is unusual but not very, and to only worry about it if it's congruent with other symptoms.
posted by soylent00FF00 at 7:04 PM on July 20, 2016

The null hypothesis here is that the Test1-test2=0. For that you need the standard error of the difference between Test1-test2 (i.e. if you took a bunch of people with two tests, subtracted one score from the other and calculated the standard error of that distribution, that's the number you need).

My source says that the SEM of the difference is the square root of the sum of the square of the difference scores for each test - in this case the same test, so SEMdiff = sqrt(SEM*SEM + SEM*SEM) = sqrt(6.76+6.76) = sqrt(13.52) = 3.68.

The formula for finding that (Well the variance, not the standard error, but as you point out, the variance is just the square of the standard error) is the I linked above (more). I'm not sure what your source is, but any formula without the covariance is not calculating the right thing. This formula requires the covariance between the tests (presumably in this case it would be calculated from test-retest data).

Though the "kind of big but not so big that I'd assume it was a real difference" conclusion I support. Just not the math. We don't have the numbers to the math. All we have are eyeball tests. And all this assumes nothing about the norming of the data has changed anything. Given that your age will have changed, especially, and that IQ tests are normed, I would just assume this was a big issue.
posted by If only I had a penguin... at 7:31 PM on July 20, 2016

Well, one can argue about the theoretical math (which I am willing to admit I may not have done correctly) or, we can use empirical data.

From this article (paywalled unfortunately)
Table 1 gives the "Frequency of Different Magnitudes of Gain or Loss on ... Full Scale IQ... for 119 adults in the WAIS-R Standardization sample". Unfortunately, this is at a 5 to 7 week retest time, so given practice effects the average score went UP by 6 points, so that doesn't directly answer the question, however...

If we assume that practice effects reduce with time (which they do) and assume that after 4 years the effect would be zero (plausible, but not a sure thing), then...

Table 1, offset by 6 points (so that +6 is now 0) gives us 2 out of 119 people whose scores went down by 10 points or more on the re-test, which is pretty close to my prior claim that it's more than 1% but less than 5%. I wouldn't trust this analysis however, as it's very sensitive to that 6 point adjustment, assumes that test-retest variance is not time-dependent, assumes the WAIS-R is the same as the WAIS-IV, etc.

But kind of interesting to look at it from theory and then from data.
posted by soylent00FF00 at 12:18 PM on July 21, 2016 [1 favorite]