How can we stop people from stealing exams?

December 12, 2006 2:29 PM

How can I design an exam setting that will discourage cheating?

The basics: 500+ students, every seat in lecture hall is taken. They have flip up desks. We have 4 versions of the exam: A, B, C and D, each are numbered and the number matches up with the bubblesheet. Kids bring their own bubblesheets. There are 7 TAs with an average of 70 students each. We force the students to sit in rows specific to their TA. TAs pass out exams row by row, watching that the rows are in order of A, B, C, D, A, B, etc. When finished, the students hand their exam sheet and bubblesheet directly to their TA. All TAs roam the hall and a TA is at every exit. This happens for both the midterm and the final.

Yet somehow, every few terms, an exam gets out. Changing the questions on a regular basis is out of the question as the exams take place every 10 weeks and there is only so much that is tested. We have to do multiple choice based on the size of the class.

We know that people are sent in specifically to steal exams, but hopefully if a TA doesn't recognize the student s/he will ask to see his/her ID. We don't know how exams still get out! We only find out later that some frat or sorority has one because students will e-mail the TAs saying "I'm working with some sample questions..." and paraphrase an exam question perfectly.

Is there any way that we can design the exam setting so that exams won't get out?

PS, the exam is on Friday, but we'd appreciate the information for the future as well.

posted by k8t to education (70 comments total)

Have a sign-in table. Each TA has a list with all of their students. No way to get a test if you don't sign in with your TA.
posted by ThePinkSuperhero at 2:33 PM on December 12, 2006 [1 favorite]

Give TAs fake exams? Seems like the TAs are the weak links in your security setup. Presumably one of them is distributing the exam.
posted by GuyZero at 2:34 PM on December 12, 2006

Number the exam sheets and note each student's name and exam #. At the end of the test each student has to return the exam and get the number checked off the master sheet.
posted by holyrood at 2:37 PM on December 12, 2006

And, with TPS's suggestion, have each student sign her name on the test (itself) as she hands it back in.
posted by Alt F4 at 2:37 PM on December 12, 2006

TPS, a good idea, but how do you organize 500+ people to sign in?

TAs wouldn't screw around with this - we're stuck here for 5-8 years of our lives and if someone did that, it would have major implications. Plus, the TAs don't see the exam until 5 minutes before the exam starts.
posted by k8t at 2:38 PM on December 12, 2006

Are students allowed to have phones in class (or do you check)? If so, it could very well be that students are taking pictures of exam pages (can be discretely done with a phone) and compiling a list later. Students from a frat could easily distribute this task and later combine the list.

Just a thought.
posted by special-k at 2:38 PM on December 12, 2006

Holyrood, another good idea, but how does the TA write down 70+ students names and not end up taking a really long time?
posted by k8t at 2:38 PM on December 12, 2006

There's an important piece of information missing here. So if there are exactly 514 students, then there are exactly 514 copies of the exam? If not, where are the other copies kept? It seems also that a student could grab more than one if they are just informally passed out.
posted by vacapinta at 2:41 PM on December 12, 2006

Have each student hand his Student ID in as collateral when he gets the test, and he only gets the ID back when he hands in the test and the bubblesheet.
posted by Alt F4 at 2:41 PM on December 12, 2006

We make the exact number of tests as there are students, but as you can imagine, there are kids who miss the exam because the overslept or whatever, so the ones that sneak-in bank of that. I'd say that from the class of about 500, 30+ kids take the test with accomodation and 10-15+ oversleep.
posted by k8t at 2:43 PM on December 12, 2006

I frequently TA a class of 700 students. We print labels with student ID, name and TA's name and seal each exam. students need an ID to pick up (easy to do and doesn't take too much time). So this will prevent students not in the class from trying. Also, if a student doesn't return a test, you will know because all exams have names on them...
posted by special-k at 2:45 PM on December 12, 2006

Change the orders of the answers and the questions. Make a really big "bank" of questions that you choose from for every exam. Make the bank big enough and the questions thorough enough that so that even if a student did have access to the questions, the only way to be prepared to answer all of them would be to understand the material.
Perhaps hand out practice tests to give cheaters less of an edge.
posted by martinX's bellbottoms at 2:45 PM on December 12, 2006

Where are you getting the exam questions from? Are you using any from the official textbook teacher aides? As a student, I always used to take the time to log into the textbook's website, since they frequently had "practice" questions that were later verbatim on the exam. It was an extra leg-up that some of your students might be aware of.
posted by xo at 2:51 PM on December 12, 2006

Not to be snarky, but in the business world having access to the test before it's given would be considered a competitive advantage and you'd probably get a bonus for it.

Having said that, I would look into building a test question database. Each time a new question is written put it into a database and then on the morning of the test have a query written that could knock out four (or more) tests that would then be printed and distributed that day.

Other ideas:

1. Replace the TAs with your school's ROTC and tell them that security is of up most importance.

2. Have the tests printed/copied at an off site location, in another city perhaps and ask them to guarantee confidentiality and deliver them on the day of the test.

3. Pull and 'Apple' and give each of the separate TAs bogus tests with 'signal' questions in it. These questions should be phrased in a way that will confuse the students prompting them to ask for clarification. Then collect all the emails from students that the TAs get and cross check with the bogus questions.

4. Fake a high-profile 'bust' to scare people from leaking the tests.
posted by DragonBoy at 2:51 PM on December 12, 2006

Have each student hand his Student ID in as collateral when he gets the test, and he only gets the ID back when he hands in the test and the bubblesheet. And then maybe during the exam, one of the TAs can mark them off a roll, so no student can try to take the exam twice.
posted by b33j at 2:51 PM on December 12, 2006

Here's how I would do it:

1. When you print the exams, do a merge from your spreadsheet containing the student names so you have 500 exams, each with a single student's name printed on it.

2. When you hand out the exams, have TA's check IDs to make sure they match the name on the test.

3. Students have to sign their exam sheet and return it with their bubblesheet in order to get their test graded.

This way, only authorized students get copies of the exams, and you can be fairly sure you will get all the exams back that you hand out--and if you don't, you know who the cheater is.
posted by jtfowl0 at 2:52 PM on December 12, 2006

I think special-k has it right- print the names on the exams, have sign-in be as simple as find the table for your surname (a-k, l-r, s-z), hand over or show your ID, get your sealed test.
posted by ThePinkSuperhero at 2:53 PM on December 12, 2006 [1 favorite]

Lots of great suggestions above. I especially like holyrood and special-k. But it seems from what you're describing as though your biggest problem is that there's an appalling culture of cheating on your campus. Trying to prevent cheating is all well and good, but I think you also need some very harsh punishments for cheaters as a deterrent.

I would go to your department head and/or school dean and see if you can work out a way to punish non-students who attempt to sneak into your class to steal the exam in the same way you would punish a student you caught cheating. I think the punishment should be suspension or expulsion from school, but at the very least, attempted cheating, even for a class in which the student is not enrolled, should be noted on academic transcripts and other official records. Let the kids in your class know that if their frat brothers or anyone else attempts to steal your exams, heads will roll. Coupled with some of the security strategies outlined above, I think a few really harsh punishments will get people's attention.

Your school needs to take a stand on this now, because this isn't just your class's problem; it's the university's problem.
posted by decathecting at 2:55 PM on December 12, 2006

jtfow10: That's impossible. We (as TAs) can never print that many copies (a 10 page exam would require over 5000+ sheets to be printed). We usually photocopy from a master. My previous suggestion (with a label) is similar to what you are suggesting.
posted by special-k at 2:56 PM on December 12, 2006

- 90% of the questions result from lecture, 10% from the book. Most kids miss the book questions.

- Having a large question database is tough. With only 10 weeks of instruction and being an intro course, other than rotating the 10% book questions, there are only so many things that you can ask about. Writing test questions is harder than one would think.

- The prof copies the test on the departmental copy machine the night before the exam and is very conscious about encrypting her Word documents, locking doors, etc.

- Busting the kids at the frat or sorority isn't good. The kids may or may not know that the test was stolen a few terms back AND they aren't the ones that stole it.

- Whenever we discover that a test got out, we put it up on the online reserves so that everyone has access to it.
posted by k8t at 2:57 PM on December 12, 2006

The students bring their own bubblesheets? Is there anything preventing them from bringing in two bubblesheets, noting the exam version ABCD, writing the correct answers on both, and only handing in one? Not that this is the problem you're experiencing, but it could also be a problem.

Fundamentally, though, with four versions of the test, it is possible to reconstruct them from memory even without phones or cameras. Changing the test regularly is the only way to get around this; apparently not rewriting the whole thing every term, but at least shuffling things between ABCD once a year. Changing the details of at least one question per version per term, in such a way that a regurgitator will be tripped up in a telltale way, but an honest student will be able to find the right answer. A couple of cases of this will do a good job of tainting the sneaked-out copies.

If changing the test even this much is not worth the time for 500+ tuition-paying students, though, it seems like some departmental priorities could use at least a little reevaluation: a student would get in trouble for turning in the same final paper class after class, term after term, but your department sees that same recycling from itself as non-negotiable.

Ban cameras/phones and someone will wear Steve Mann video-spectacles. Frisk students and lose your soul. Show a little integrity by changing the test, and you (your department) support the principles of your academic environment.
posted by xueexueg at 2:58 PM on December 12, 2006

Decathecting, you make a good point, but what seems to happen is when a TA or a prof reports a student for academic dishonesty of any sort that there is a hearing and generally the tables get turned on the TA or prof. It is a huge ordeal, as the system assumes that the student is innocent.

Most TAs and profs instead try to take things into their own hands. So no one around here reports anything. And yes, there is a HUGE culture of cheating. I am shocked at the amount of plagarism, papers bought off of the internet, etc.

For our essays, for example, we have students analyze a brand new TV show or commercial to try to reduce plagarism.
posted by k8t at 3:00 PM on December 12, 2006

Oh yeah, ABCD are the same questions reordered - both questions and answers.

There are over 100 questions on the midterm and final. Can you imagine having to rewrite 200 questions every 10 weeks four times a year - 800 new questions?!?!
posted by k8t at 3:03 PM on December 12, 2006

k8t: I teach at a UC too (so we are also on the same 10 week schedule). For us, we've never had the problem of people stealing exams (but then we actually hand them back when graded). Also, we prefer essay questions (so that's fewer questions and more choices for future quarters). We also have a small multiple choice section (like 20 questions) but that's so easy to change each time. Is it really that hard to have a different exam 3 times a year?
posted by special-k at 3:07 PM on December 12, 2006

I don't know why, but it doesn't seem that you see it as quite so serious as a lot of the people responding.

Personally, I think that busting the kids in a frat who were doing it would be well worth it, as a step towards the whole 'reduce cheating culture' that was mentioned earlier. They might not be the ones that stole it, but they're still cheating by using it. If you feel bad for the few innocent flowers who really think they're using a legit sample, start next semester by announcing that the only legit sample questions are those on your online reserves, and you are cracking down on people using stolen tests. Anyone found to be studying off a stolen test will be considered to have deliberately cheated unless they turn themselves in. Now they have a way to check what they're using, and you have motivation for them to let you know what's been stolen.
posted by jacalata at 3:09 PM on December 12, 2006

"...some frat or sorority has one because students will e-mail the TAs saying 'I'm working with some sample questions...’”; "Students from a frat could easily distribute this task and later combine the list."

Christ, can we get past the stereotype - I'm assuming that you think all your Asian students are mathematical geniuses as well.

With a class of 500+ I'm assuming that under 10% are in a Fraternity or Sorority - what about the other 90%?

Sorry for the rant, whenever I see comments like that it hits the wrong nerve.

With that said (phew - I feel much better), I second special-k's suggestion. Printing off and sticking labels for exams shouldn't take but a minute or two per TA. Students should have to pick up their exam from their TA. For the most part TAs should be able to recognize and know the names for most their students. For the ones they don't, checking ID should only take a second.

At the end of the exam students turn in their answer sheet along with the questions to their TA. Simply have the TAs cross-reference between exams that were picked up and exams that weren't returned.

Also, you state that the exams are given every 10-weeks, so I assume the questions are usually fairly similar... It doesn't take much to remember the basic 'gist' of a question, what it was asking, how it asked it, what concept was being tested and how the concept was presented. Everyone talks about tests after the test is done... 'What did you get for the question that asked about X?'

Perhaps it has gotten to the point that it is common knowledge that you test on X, Y and Z. Word of mouth may be the reason your questions are getting out:

"Hey did you take physics with Professor X last term?"
"Yeah, he always asks about X, Y and Z on the final."
posted by ASM at 3:11 PM on December 12, 2006

I agree that using student ID as collateral coupled with individually labelled exams is perhaps the most effective way of ensuring the exams are all handed back. But, as has been mentioned, a cameraphone can be used.

One trick that I've used to prevent certain types of cheating (looking at another person's answers) is to photocopy the exams on different colored paper which isn't correlated with exam form. So, you have 4 colors of each form. This makes it hard to figure out who else has your form (if you just go by color you will probably bomb the test).

My favorite sneaky trick, which may or may not be particularly effective, but it sure is fun: Get a young looking stooge to pretend to be a student taking the test. About 10 or 15 minutes into the exam have one TA loudly "catch" the stooge cheating. Make a big thing of having the professor drag the perpetrator out. This at least convinces students that there is a possibility of being caught cheating.

It really does come down to creating new test items. Having written many exams in my day, I know it is hard and tedious, but it can be done. Often, you can slightly change the deep structure of existing questions, while significantly changing the surface features.
posted by i love cheese at 3:13 PM on December 12, 2006

"...some frat or sorority has one because students will e-mail the TAs saying 'I'm working with some sample questions...’”; "Students from a frat could easily distribute this task and later combine the list." Christ, can we get past the stereotype - I'm assuming that you think all your Asian students are mathematical geniuses as well.

Umm... the Greeks are the ones who live in larges houses together and have years of files of copies of things. It isn't me stereotyping, but every time an exam has been found to be leaked (10-15 times with this prof), it was from a Greek source.
posted by k8t at 3:14 PM on December 12, 2006

Write a number on each scantron and exam.

If a scantron comes back with no matching exam, they get a zero at a minimum. If you want, consider it cheating and fail them for the course. This would require being diligent about your own storage of them.

Alternatively, just have them write on the exam itself. Grading 70 multiple-choice exams shouldn't take a TA more than a hour or two.
posted by ROU_Xenophobe at 3:20 PM on December 12, 2006

Your task is almost hopeless.

They really don't need to steal an exam, and your account has not convinced me you have solid evidence that they have-- plenty of students have good enough memories to recall the majority of questions even from a fairly large exam if they do it immediately after taking it, especially if they sit around in a group and prompt each other. Your group of TAs could try it yourselves if you don't believe me.

I know you are not to blame for this, and that you have very limited power to change the situation, nor would I risk trying to change it, if I were you, but your department seems to be fostering a very adversarial relationship with students. If I had walked into an exam in my major and the the TAs were acting like you all are, I would have walked right back out, dropped the class, and tried to make an appointment with the head of the department to tell him or her he or she had a real problem on his or her hands.
posted by jamjam at 3:21 PM on December 12, 2006

JamJam, every quarter at least 5 kids rat out their friends for having copies of the exam. We also find paper copies of the exam on occasion.

This doesn't occur as much in the upper division, but is a real problem in the intro courses.

Speaking with TAs in other departments, they have similar problems.
posted by k8t at 3:25 PM on December 12, 2006

I don't know how your course works after the test, but if it's anything like tests that I've taken the problem might be there. For example, someone asks to see their test after it's been marked (maybe they're not happy with their grade, or just want to know where they went wrong), and then they photograph it with their camera phone. Where I am they give you your test and let you hang out in a separate room (usually adjacent to the prof's office) with it before giving it back.
posted by teem at 3:28 PM on December 12, 2006

Essay tests! 500 students divided by 7 TAs equals 70 exams each to grade. Not a big deal twice a quarter, and a good preparation for what most of us in academia actually do. Bring the TAs around a big table, crank up some punk music, and have a grading party. You'll be done in a few hours.

Using only multiple choice tests sends the message to the students that you can't be bothered to find out if they learned anything or not. That message is a big part of the reason for the culture of cheating.
posted by LarryC at 3:30 PM on December 12, 2006

Teem, here we sit with them while they look at their test after it is marked. They can only look at it in the TA's office and we keep the exams locked up in a hard-to-access room.
posted by k8t at 3:30 PM on December 12, 2006

Seems to me that if your question set covers the material so thoroughly that it's impractical to write extra questions, then there are only two ways that somebody could benefit from prior access to a multiple-choice test:

1. Research and memorize all of the answers to all of the questions. This strikes me as "study" rather than "cheating".

2. Generate a pre-completed bubblesheet that matches the exam you're going to take, and bring that with you to the exam.

Now, it seems to me that if kids are supposed to bring their own bubblesheets, and if the bubblesheets are numbered to match the exams so the student knows ahead of time which exam they're going to get, then method 2 becomes trivially easy to use. So I don't understand why the bubblesheets don't get handed out with the exams.

If you made this procedural change, it would become feasible for every single exam to be printed with the questions in a unique order (assuming the departmental photocopier is also a printer, which most high-end ones are). Knowing that the answer to question 23 is E would then be completely useless.
posted by flabdablet at 3:37 PM on December 12, 2006

Make a large booklet of questions (you only have to do this once). Write a program that prints an individualised sheet for each student with a list of the question numbers that they have to answer, and prints an answer sheet that you hold back to compare against their script. This can be automated at various levels of detail.

I'm not convinced it would really matter if students had a copy of the question bank in advance (see e.g. the written component of the British driving test as an example). There is some evidence (can't find the reference to hand) that students retaking an exam perform just as well/badly when they are presented with the same questions on the resit as on the first exam, and when presented with different ones.

On another point, someone earlier asked how you get 500 students to "sign in" on a sensible timescale. We handle this by having students leave their student card (a plastic card with name, student number and photo) on their desk and walk round during the exam checking that (a) each student is registered for the current exam and (b) the face on the person behind the desk matches the photo on the student card.
posted by Jabberwocky at 3:40 PM on December 12, 2006

TAs pass out exams row by row, watching that the rows are in order of A, B, C, D, A, B, etc.

This isn't about stealing copies, but it is about cheating. Seating kids in this pattern means that a kid with test A is going to be directly behind another test-A kid. Even if the classroom is flat, this will lead directly to cheating -- and I'm guessing that a lecture hall that fits over 500 students has its seats on different levels, which will only make it easier to see over the shoulder of the guy in front of you.
posted by booksandlibretti at 3:47 PM on December 12, 2006

In my uni in Australia, to manage the large number of students (200-300 maybe more) taking an exam, they split up the students by last name so there is about 20 or so in a class taking the exam in a class. (Everyone takes it at the same time.)

Each class has a sign-in sheet, held by the invigilator, and they all have to bring in Student IDs to cross-check. They only are allowed out when they return both the bubble sheets (provided by uni) and the exam questions, and sign out.

Sounds like that could work for you.
posted by divabat at 3:51 PM on December 12, 2006

There's a solution, but it takes a stance opposite to the one in your premise: don't use multiple choice. I've taught classes with more than 500 students and have used short answer or even short essay questions. It's possible (and even efficient) to do this, as long as you're clear about indicators and outcomes.

Plus, to be honest, I don't really think multiple choice exams are frequently very good. Lots and lots of people think they're great at writing them, but almost nobody is. With a short answer question, students can not only demonstrate partial comprehension, they can approach and answer a question accurately in many different ways. I think it's just better pedagogy to avoid multiple choice tests and use discursive assessments as much as possible.

If you've got 7 TAs for the course, they'll each have to grade 71-ish exams, and that's not too massive an expectation, to be honest.

Another benefit that this has is that you can use one exam for the whole class, and it really doesn't matter as much if an exam gets out-- answering the question still demands that a student be able to demonstrate understanding. Plus, if you're at all suspicious of the TAs, you can wait until the exam has been given before you talk about grading rubrics with the TAs.
posted by yellowcandy at 3:54 PM on December 12, 2006

there are only two ways that somebody could benefit from prior access to a multiple-choice test:

1. Research and memorize all of the answers to all of the questions. This strikes me as "study" rather than "cheating".

2. Generate a pre-completed bubblesheet that matches the exam you're going to take, and bring that with you to the exam.

Are you trying to prevent 1 or 2?
If your test questions are good, and cover all the material, then I don't see what's so bad about #1. Just publish the whole question bank so that no one has an unfair advantage.
Randomizing question and answer orders for every sitting of the test will completely prevent #2.
posted by martinX's bellbottoms at 3:54 PM on December 12, 2006

Speaking as someone who regularly teaches medium-sized (100--200) intro courses:

If I had walked into an exam in my major and the the TAs were acting like you all are, I would have walked right back out, dropped the class

That would be fine with me. For every one of you, there's another 3 students who registered too late to get into the course. If it's a required course, you have to take it sometime, somewhere, and I really don't give the slightest shit whether it's from me or someone else.

And, frankly, it wouldn't bother me at all if you took the intro course over the summer from a local community college -- that would mean I get to teach more interesting upper-division courses more often.

and tried to make an appointment with the head of the department to tell him or her he or she had a real problem on his or her hands.

I'm sure he or she would take that under advisement and get on it Real Soon Now.
posted by ROU_Xenophobe at 3:55 PM on December 12, 2006

Just because people know what is on the exams doesn't mean they are stealing them. If you are not worried about actually answering the questions it is fairly easy to remember a lot of them in the necessary time for the exam. Look at people who get really crappy grades on the exam...they may be there merely to memorize.
posted by frieze at 4:00 PM on December 12, 2006

That said, there is another way to deal with this basic problem:

First, assume that no matter what you do, copies will leak out one way or another, or groups of students will reconstruct them more-or-less accurately from memory.

Since you can't stop it, the real problem is that some people have a comparative, unearned advantage because their fraternity or sorority has a test bank.

So make copies of old exams available to everyone, right on the course's web page. Then nobody has an unfair advantage.

If you expect that grades would be unacceptably high doing this, then ramp the difficulty or specificity of the questions, since you know that any diligent student will be studying the old exams. Likewise, take a few semesters to build a bank of questions that's large enough that no more than 20% or so of an exam will be a repeat of last time's.
posted by ROU_Xenophobe at 4:03 PM on December 12, 2006 [1 favorite]

Are you sure that the actual exam booklets are being stolen? What if a student just snuck in blank paper on his person, copied all the questions, and then snuck out with that copy? Do you allow students to use their own scratch paper? It seems to me that the only way you will ever be sure that no test questions ever leave the room is by allowing the student to bring in nothing from the outside except a pencil (no bookbags, calculators, etc), and using only the materials that you provide, which includes scratch paper. And even then, someone could sneak out with a copy they made by hand stashed in their underpants or whatnot unless you actually count the number of sheets of scratch paper that they return as they are leaving. And even THEN, there are people that are crazy enough to just memorize the test, if they are paid enough money by a frat/club or other incentive. I think your strategy should be more of randomization of the test from a large bank of questions than trying to prevent leaks.
posted by Rhomboid at 4:11 PM on December 12, 2006

How long have you been using the same questions? And just how specific is the paraphrasing when someone asks for clarification? You only need a few people remember a dozen or so questions with any degree of detail and you've got your whole question database in two terms. In high school when I had a test in an earlier period than a friend of mine, I would always get grilled on the questions and could normally cover most of what was on the test without much problem (especially the "gotchas." The stuff you might not think to study, so if those are the questions you're seeing come back at you, I wouldn't be surprised at all.) By all means the suggestions above are great, but if you've been asking the same stuff for years someone's figured it out and might be taking notes.

Shit. I just read the last part of ASM's comment. So what she said.
posted by Cyrano at 4:11 PM on December 12, 2006

solution for the future: what Jabberwocky and ROU_Xenophobe said. word by word. I've seen it in action, it works.

Create >1000 questions database, make it public (questions, and MC answers and the right answer), on the web. The exam questions will be from the database, but the MC answers are in random order each time (and you still have 4 different types of exams to prevent cheating during the exam). If a student is able to answer 1000 questions and remember the right answers, I say s/he deserves an A.

However, you have to convince the prof to do it; so here it is the selling story: It will take about 1 year, basically 1 new exam each time. The prof should ask the TA to create about 1/2 of the questions (10 questions / TA * 7 TAs *8 times a year ~ 500 questions). Do this and there are no more worries about new exams, cheating, etc. The students will have no other incentive but to study. As always, electronic devices still need to be banned.
posted by MzB at 4:43 PM on December 12, 2006 [1 favorite]

"There are over 100 questions on the midterm and final. Can you imagine having to rewrite 200 questions every 10 weeks four times a year - 800 new questions?!?!"

This is what TAs are for. And 100 questions? How long are these exams? We used to submit 7 questions each - 4 TAs - every week. Over ten weeks, the prof would have have 280 questions to choose from. Some would be repeats, but you'd probably get enough out of it to have at least some of them be new each year. Admittedly, we only had 50 question midterms and exams, because we only had 50 minutes to give it in, but you might try this as part of your anti-cheating measures.

We also have a 'bags and all devices in the front of the room' policy, and the ID policy.
posted by cobaltnine at 5:17 PM on December 12, 2006

I think it would help if you could articulate why it's a problem that exams get out. I don't doubt that it is a problem, but I can think of several reasons why it might be a problem for you, and some of them will favor different solutions.

Are you worried that some students have more material to study from than others? Do you suspect that students are actually memorizing 200+ questions in order to cheat? Are you just frustrated that the "leaked" exams let students know what material they can safely ignore?
posted by nebulawindphone at 5:43 PM on December 12, 2006

Well, that's me told. Now I'll be up all night crying.

k8t, why is this your problem? This should be the prof's problem. If he or she isn't doing anything to combat it, that's his or her choice. As a grad student, you have enough to worry about on a day to day basis without taking on other people's concerns. If you're getting this concern passed on to you by the prof -- and it shouldn't be -- take a minute to figure out what the minimal acceptable response is, do that, and get back to work.
posted by ROU_Xenophobe at 5:58 PM on December 12, 2006

Another idea - just don't worry about it.

I gave an exam yesterday, a 20 problem multiple choice final for 85 sophomore electrical engineering students. Last week, I gave them a "study guide", which was identical to the final. Same problems, same numbers, same answers, except that the choices weren't given. I didn't tell them ahead of time that it was the same, but a number of them had strong suspicions. They had a formula sheet with them during the exam. Result: the good students did well, the less good students did less well. Average score was 61%.

I've done this experiment a number of times. It really doesn't matter if they have the exam ahead of time. I think it lowers their stress level, since they know exactly what to study.

What problem are you trying to solve?

On preview, I see some others have made a similar suggestion. Try it!
posted by Wet Spot at 6:11 PM on December 12, 2006

ROU, she asked us if we had any bright ideas, that's all. :)
posted by k8t at 6:27 PM on December 12, 2006

This whole thing gets me riled up. As a (good, honest) student, I would have resented being treated like a criminal. Why not cater to the good ones instead of the cheaters? Scrap the multiple versions. Scrap the gestapo TA force. Let the cheaters cheat and shoot themselves in the foot in the long run.

Frankly, the only thing that would make me consider cheating is being treated like a cheater.
posted by SampleSize at 6:40 PM on December 12, 2006

I once gave a HS chem exam in which I made all of the correct answers "B". Students who were sure of their knowledge did fine. Other who weren't, well.......

I got a lto of flack for it,a nd re-administered the exam....
posted by pgoes at 6:52 PM on December 12, 2006

I once gave a HS chem exam in which I made all of the correct answers "B".

I've done that too, it was hilarious. Even more fun was to make the answer to each question the same as the problem number. But the winner was when every answer on the test was "666". One guy was in tears, said he thought his calculator was possessed.
posted by Wet Spot at 7:02 PM on December 12, 2006 [4 favorites]

Giving out a superset of the exam questions ahead of time (here are 30 questions, you'll be asked to answer 8 on the exam) is the method I favor. Works best with short-answer and short-essay questions.

Whether this can work for your course is subject-dependent, of course.

If one of you (the prof or the TAs) is going to substantially revise the curriculum of one of the department's core courses -- eg by creating a monster question bank that the course can draw on in the future -- the department should pay you for the extra time it will take. It might be the kind of thing a knowledgable person can do with a few extra hours a day, for a week. If the problem is getting to be as bad as it sounds, this might be worth doing. It will stop this form of cheating, and as a bonus will illustrate that a student really has to know the material to do well so dishonesty/cutting corners won't help.
posted by LobsterMitten at 7:03 PM on December 12, 2006

As a (good, honest) student, I would have resented being treated like a criminal.

Sure, but the point isn't to avoid resentment. In these big intro courses, the modal student probably resents being required to take the course in the first place anyway.

Why not cater to the good ones instead of the cheaters?

Because in most schools, the predictable consequence of that is widespread cheating.

If this sort of thing really bothers you, either go to a liberal arts college where there just aren't large courses, or go to a school with a somewhat-functional honor system like UVa or William & Mary.
posted by ROU_Xenophobe at 7:35 PM on December 12, 2006

If you're still doing paper based exams, I'd really question how effective your testing measurements are. More and more commercial organizations are moving to computer adaptive testing, and other automated strategies as a means of both making more accurate measurements of student knowledge, and of protecting the test. If I were student at your institution, and you handed me a paper test in 2007, I'd immediately question the value of your instructional methodolgy, and your ability to measure my knowledge accurately or fairly.
posted by paulsc at 8:01 PM on December 12, 2006

If you're still doing paper based exams, I'd really question how effective your testing measurements are.

As effective as they can be given several constraints:

(1) The university isn't going to spend millions upon millions putting a terminal of some sort into every seat.

(2) The university doesn't have however many tens of thousands to hundreds of thousands of dollars it takes to code up an adaptive test for each of several hundred to a few thousand courses, for a grand total into the tens of millions of dollars.

(3) Adaptive testing seems ill-suited for college courses since there's no extant mass of hundreds of thousands to millions of people who've taken the same course or test to use as a baseline. Different courses or even different sections of the same course have different emphases and different goals about what you ought to know.

If I were student at your institution, and you handed me a paper test in 2007, I'd immediately question the value of your instructional methodolgy, and your ability to measure my knowledge accurately or fairly.

That would be your business, but you'd be profoundly out of touch with the realities of higher education.
posted by ROU_Xenophobe at 8:34 PM on December 12, 2006

paulsc, I am currently studying software engineering, and the only computer based test I've ever taken was a practical programming test. (Once). Oh wait, and a multiple choice piece of crap that had the wrong answers programmed in, and then the prof got shitty when the entire class of 700 emailed her to say so. Paper is still the cheapest and easiest way to give a test, especially with the relatively small numbers of students you get in any single university class. (The LSAT or SAT is a whole other level of magnitude, so the cost/benefit is different).

On preview: I'm with ROU_Xenophobe. You don't seem to realise that commercial testing organisations are a completely different game to university.
posted by jacalata at 8:37 PM on December 12, 2006

I do student judiciary work at my school and we see a lot of cases of cheating in large lecture halls with similar exam formats. The layout of your test sounds wonderful, but the sign-in/out sheet is such a useful tool for these types of tests. It helps keep track of externalities and exams real well.

Highly recommended
posted by stratastar at 8:37 PM on December 12, 2006

paulsc: wouldn't that require 500 computer terminals or some such for k8t's large class?
posted by Rumple at 8:54 PM on December 12, 2006

k8t, each TA has a list of their 70 students. Student walks in, goes to TA, TA checks name off list, hands out test.

Repeat as they leave.

They did this at UM-- you never had this happen? Must be a communications thing, this was how I took every comm. test in undergrad.
posted by holyrood at 9:23 PM on December 12, 2006

I've noticed a lot of people recommending essay tests, as opposed to bubble-sheet multiple choice ones; however it might not be very practical to grade such a large number of essays.

What about using "short answer" tests, however, as a sort of middle ground, between essay and scantron-type exams? You could use tests for which students are be required to write a term, equation, number, or some other very short objective response as an answer to each question, instead of choosing from a list of multiple-choices. Short answer exams can be graded pretty quickly (spread out among TA's and relative to essay exams, anyhow), and yet discourage students from "studying" for the exam by simply memorizing a series of letters. Perhaps a short answer test would be easy to grade, without allowing cheaters to ace the exam through entirely meaningless memorization.

Just a thought.
posted by washburn at 9:30 PM on December 12, 2006

"That would be your business, but you'd be profoundly out of touch with the realities of higher education."
posted by ROU_Xenophobe at 11:34 PM EST on December 12

"...You don't seem to realise that commercial testing organisations are a completely different game to university."
posted by jacalata at 11:37 PM EST on December 12

"...paulsc: wouldn't that require 500 computer terminals or some such for k8t's large class?"
posted by Rumple at 11:54 PM EST on December 12

Geez, kids. I got my BS in 2004, and the last paper test I took was in 2002. I know of few credible 4 year institutions that don't require incoming freshman to have a PC, and specify varioius study aids and paper style formats in such a way that student without access to a laptop PC can do the course work. I'm charmed if you all are attending Hogwarts, and writing your exams in quill pen and ink.

Lotsa luck with all that out in the commercial world.
posted by paulsc at 10:46 PM on December 12, 2006

"...make copies of old exams available to everyone, right on the course's web page..."

This is exactly what we do at the University I teach at in London. Some courses have a decades worth of past exams.

If, as per the original comment, "there is only so much that is tested" and "every few terms, an exam gets out", then you should logically assume that someone, somewhere has all possible exam questions.

I'd argue that for fairness past exams should be public.
posted by Mutant at 11:05 PM on December 12, 2006 [1 favorite]

And before I forget, all the College Board's CLEP exams that I took in 2003 and 2004 for 24 hours total credit towards my degree were CBT adaptive exams. Adaptives are great because they quickly put the focus of the testing at the level at which the student is demonstrating competency, and consequently, they can very accurately measure a student's comprehension of tested material, to a finer degree of grade discrimination and repeatability, than can a list of broadly applicable questions, many of which are bound to be below comprehension level of better students.

But the OP's question was "Is there any way that we can design the exam setting so that exams won't get out?" To which my answer is "Yes, make it an adaptive CBT, and each student will be adaptively examined, according to his/her responses, with an individualized test that truly measures his/her comprehension of the material, is diagnostic for deficiencies, and is essentially unique."
posted by paulsc at 11:33 PM on December 12, 2006

If, as I suspect, this isn't the only huge lecture course at the school that has difficulty testing, your institution might want to look into establishing centralized testing services, as my alma mater did.

Here's how it works: Profs are still free, if they wish, to give tests during the lecture hour and grade them themselves, using TA grunt work and/or a departmental scanner. But, especially for the larger classes, their department could also be allocated (or maybe buy) the requisite number of seat-hours from Testing Services. (All of the huge-lecture general-education classes took advantage of centralized testing, but so did various smaller classes down to about 50 students.) For these, students are given a range of days during which the test will be available, and the prof submits the test to the campus testing center. This is a dedicated facility with its own staff. At a minimum, they provide a secured environment and proctors; if the test is multiple-choice, they can also provide automated grading and reporting, and as appropriate they also do the grunt work of assembling several versions of a test (from question pools submitted by the professor or department, and to their specifications) and tracking which student gets which test.

Here's the procedure: I have a test to take in the testing center. On any of the valid days, I check their website to see how crowded it is (they have a webcam on the line and an automated listing of large tests that are on their last day). Finding the wait acceptable and my own preparation adequate, I report to the center. When I walk up to the counter, I have my pencils, pen, scratch paper, allowed support materials, and student ID in hand, and everything else closed in a bag (either my backpack or a cloth drawstring bag provided by the center). My phone or other electronuisance is off. First off, the counter worker checks that any scratch paper I'm carrying is blank, and stamps it as legit. Next he/she scans my student ID, pulling up a list of courses I'm enrolled in that are testing that day, and, if needed, asks which one I want. Worker then goes to pull one of the tests from the files (or, if they're all signed out, print one on demand), and scans the barcode on the test.

Thereupon, the computer creates a record associating my ID number, which copy of the test I received, which version it was, and when I began taking it. It also displays on-screen the requirements of the test (open/closed book, open/closed notes, calculator dis/allowed and what kind (four-function, scientific, graphing), time limit if any), of which the counter worker reminds me. Finally, it causes the printing of a bubble sheet (if applicable) or cover sheet with my name, the course and test numbers, the timestamp, those same requirements in plaintext, and a machine-readable unique random number (either a barcode on the cover sheet, or a reserved series of bubbles). A staff member takes it from the printer, calls my name, and hands it to me. (Security hole: If I arranged with someone else to be there at the same time, and it were busy enough, he might be able to take my test, and I his, without anyone noticing the change-up.) I walk into the testing hall with it.

The testing hall contains hundreds of desks, most of them at any given time occupied. They are arranged in rows that make it difficult to check out someone else's test without being really obvious, and proctors prowl it continuously. The cover sheet or bubble sheet must be kept visible at all times, and bags must be under the seat, where they're good and awkward to get at. Rummaging through a bag, using a cell phone, pulling stuff out of pockets, conferring with anyone, or other dodgy behavior will draw the attention of a proctor; they also check cover sheets to make sure that those who are using calculators etc. are legit, and may check scratch paper for the legitimizing stamp. And it's all the harder to cheat when the people around you are all testing for different courses than you are. You can only walk to the water fountain so many times before you look suspicious, and if no one else from your course is along the way you are outta luck.

On completing the test, used scratch paper goes into a recycle box via a narrow slot, unused scratch paper goes into an open reuse box, and the test and bubble sheet are taken to the exit counter. (If there's a line at the exit counter, it's customary to let the students cut to the front who are near the end of a timed test. Oh, and the exit counter is also the place to cache a test, with your picture ID, if you need to visit the non-public, testers-only toilet.) The test is received by another employee, who scans the barcode of the cover sheet and/or puts the bubble sheet through its scanner, stopping the timer if the test is timed, and creating a record that the test has been turned in. (The actual test sheet can be inspected for markings and checked back in at leisure.) This employee also watches for testers carrying scratch paper out.

By the time the tester is at the bottom of the stairs, any machine-scorable sections have been scored, and the raw score and percentage are on a monitor next to the last five digits of the student's ID number. Long before the tester is home, the list of questions missed has been posted to the student's intranet account. The letter of the correct answer is given so that the prof can later provide copies of the test for review; even if the test is to be re-used, this is no breach since neither the number of the question nor the order of the answers is necessarily the same next term.

There are certainly still weaknesses in the physical and information security of this setup, but they are reduced. It helps that the staff are familiar with the issues and practiced in dealing with them, because it's not a sideline to TAing, but the main focus of their work. It doesn't take very many staffers at a time; the school had 30,000 students, several hundred seats in the testing hall, and except during finals I never saw more than seven counter workers and a matching number of proctors on the job. (They work their whole staff during finals, and bring extra computers to open satellite locations in other campus buildings.) And all of this hard security worked alongside soft security; we had a pretty-much-functional honor code and a culture that strongly emphasized integrity. (Caution: Hard security and soft security can poison one another if either is too heavy-handed. You get the resentment thing and/or the complacency thing going. Balance is crucial.)

And paper exams are vastly superior to computer ones. Ease of use, use of space, portability, crash resistance and recovery, convenience of handling, maintenance ... paper, paper, paper; win, win, win. First I saw of that testing center was in 1999, and they were just starting to use a few center-owned laptops to try out the administration of computerized tests; as of mid-2006, they only had one smallish classroom dedicated to computers, whereas the entire second floor of the building was used for paper tests. I call that a reflection of relative value in relative demand; paulsc might call it the inertia of fossilized fuddy-duddies, and those fuddy-duddies might reply that not everything is improved by electrification. I'm not old enough for fuddy-duddihood, but I prefer a paper test any day.
posted by eritain at 2:23 AM on December 13, 2006

paulsc's suggestion is kind of the high-tech version of "create a large bank of questions, and vary which questions each student gets." (That is, objections about $$ practicality aside, someone still has to write a ton of questions.)
posted by LobsterMitten at 7:10 AM on December 13, 2006

Geez, kids. I got my BS in 2004, and the last paper test I took was in 2002.

And the rest were the adaptive tests you describe, specific only to each course and not merely purchased versions of national commercially-available tests that might or might not have fit the specific curriculum you were taught, and not simply take-home essay/problem exams that happened to be done on your computer?

I know of few credible 4 year institutions that don't require incoming freshman to have a PC

You're shitting me, right?

You're suggesting that, as a security measure, a school should administer adapative computer testing on the student's own machine, where there might be keyloggers and screenshot-grabbers, where there's a wireless card talking to God only knows who, and where there might be any number of other nefarious bits of software installed.

The only ways you can possibly make this work as a security measure are to:

(1) Require all students to use the same model PC and forcibly re-image it before the start of the exam. This would be mildly unpopular.

(2) Use the university's machines, not the students'.

It's also interesting to note that in your world,
Harvard, Princeton, Yale, Stanford, Duke, Berkeley, Virginia, Michigan, and even Wisconsin that you list above are not credible schools, at least not at the undergraduate level.

And even if they did require computers, fine. All they need now is a few tens to hundreds of thousands of dollars for each course to code the software for the tests and develop the required baseline of responses.

Lotsa luck with all that out in the commercial world.

I'm not sure what you mean by this. Do you think that your average professor has a sideline developing commercial tests? They don't, except perhaps for a very small number of highly technical people in education schools. This is a narrow and specialized skill-set.
posted by ROU_Xenophobe at 7:27 AM on December 13, 2006

Don't be lazy. Make up a new exam each time. Ok, I know you are not being lazy, it's beyond your control, but it's very helpful for students to get their exam back after to review at their leisure (ie not in the prof's office), especially for midterms.

It's not very hard to have a large 'question pool' and odds are students are not going to have access to more then perhaps 2 or 3 old exams. If the class is so identical each year that you are using the same finals, then it wouldn't be outrageous to suggest that you invest the time to make up a very large question pool. This would make sure each final has only a handful of questions that appeared on finals in the last 3 years and will last you as long as the course remains the same.

Many of my undergrad science courses are structured like this, and we are given the previous final as study material before the exam. Maybe 5-10 out of 50 of the questions are the same, and we are ready for those, big deal. Nobody is trading the exams underground because we are already provided with a recent old final.

You are never going to get rid of kids who have access to a stolen copy of an old final or at the very least an older sibling with a good memory. You can still level the playing field for everyone else.
posted by The Wig at 5:53 PM on December 13, 2006