Use statistics to find cheaters
November 14, 2011 4:08 AM Subscribe
What would be a good statistical approach to find manipulated test results?
Let's say we have 100 schools with a total of ~10'000 students who have filled out a test. Now you want to find out if in some of the clusters, the answers were manipulated by the teachers to improve the scores for their school.
What would be a good statistical approach for this?
Manipulated tests would have a high number of similar, high scores within a school. So I thought the average intercorrelation of scores within a test for each cluster might give me some insights. I'm not sure if Cronbach's Alpha or another internal consistency score would help here.
What do you think?
Let's say we have 100 schools with a total of ~10'000 students who have filled out a test. Now you want to find out if in some of the clusters, the answers were manipulated by the teachers to improve the scores for their school.
What would be a good statistical approach for this?
Manipulated tests would have a high number of similar, high scores within a school. So I thought the average intercorrelation of scores within a test for each cluster might give me some insights. I'm not sure if Cronbach's Alpha or another internal consistency score would help here.
What do you think?
Benford's Law and the Decreasing Reliability of Accounting Data for US Firms
http://econerdfood.blogspot.com/2011/10/benfords-law-and-decreasing-reliability.html
posted by yoyo_nyc at 4:11 AM on November 14, 2011
http://econerdfood.blogspot.com/2011/10/benfords-law-and-decreasing-reliability.html
posted by yoyo_nyc at 4:11 AM on November 14, 2011
There was a segment on this exact thing in the Freakonomics movie (it's on Netflix Instant). I don't think they went into detail on the statistical analysis, but there may be something helpful there.
posted by ella wren at 4:29 AM on November 14, 2011
posted by ella wren at 4:29 AM on November 14, 2011
Response by poster: Benford's Law is fascinating! But how would you apply it to ordinal or non-decimal scales (e.g. grades that go from 1 to 6)?
posted by lord_yo at 4:39 AM on November 14, 2011
posted by lord_yo at 4:39 AM on November 14, 2011
Benford's law (or analysis of final digits) can help you red-flag numbers that were made up from whole cloth, but scores you get by cheating (e.g. correcting answers on the ScanTron sheets before scoring) are going to be just as Benfordy as the real ones.
posted by escabeche at 4:45 AM on November 14, 2011
posted by escabeche at 4:45 AM on November 14, 2011
One thing to look for in cases like this is unusually high erasure rates. If you're doing this for real and need guidance, you might start by contacting one of the academic experts in the subject cited in the linked article.
posted by escabeche at 4:51 AM on November 14, 2011
posted by escabeche at 4:51 AM on November 14, 2011
Here are links to PDFs of the official report on the cheating scandal in Atlanta schools. Part one in particular goes into some detail about the methods they used to detect the cheating. Erasure analysis played a big role.
posted by TedW at 5:19 AM on November 14, 2011
posted by TedW at 5:19 AM on November 14, 2011
Just using the graded items, your suggestions are both confounded by actual teaching. I know, crazy, but it exists. Some strategies use odd within-test variation, like runs of correct answers, doing much better on harder questions than easier ones in another place, or unusual spatial patterns within class. These all depend very much on the test being constructed to make it work. For example if the end of the test has a run of 15 questions about a particular topic regarded as hard, someone who spent a lot of class time on that would get flagged as cheating.
posted by a robot made out of meat at 5:45 AM on November 14, 2011
posted by a robot made out of meat at 5:45 AM on November 14, 2011
I guess what I'm suggesting is that you could, in theory, do factor analysis within-classes and look for extreme variation in the structure. If teachers are telling you theanswers to some questions or correcting a few at random, they should become decorrelated from the questions to which they are otherwise most similar. Having the precision youneed for an application seems unlikely, and really distinguishing it from variation in teaching focus might be very hard. Another kind of cheating which seems very hard to detect would be cheat-sheets. Having the pythagorean theorem up onthe board is the same as having just stressed it in class (in terms of outcomes).
posted by a robot made out of meat at 5:55 AM on November 14, 2011
posted by a robot made out of meat at 5:55 AM on November 14, 2011
Do you have any money to spend? Because of the high stakes, I recommend consulting professionals in the field of test security and psychometrics, rather than going it alone. In North American, there are professionals who can do the data analysis for you, or at least act as consultants about the techniques to use.
For example, Caveon Security is a big name in North American.
If you are determined to go it alone, google searches on "cheating detection psychometrics" or similar get a lot of hits for books and articles about detecting cheating for large scale testing.
posted by Ladysin at 8:37 AM on November 14, 2011
For example, Caveon Security is a big name in North American.
If you are determined to go it alone, google searches on "cheating detection psychometrics" or similar get a lot of hits for books and articles about detecting cheating for large scale testing.
posted by Ladysin at 8:37 AM on November 14, 2011
« Older How do I alter the sleeves on a sweater? | Which one of these track and field records will be... Newer »
This thread is closed to new comments.
posted by yoyo_nyc at 4:10 AM on November 14, 2011