Join 3,512 readers in helping fund MetaFilter (Hide)


a remote exam under exam conditions
April 28, 2011 8:01 PM   Subscribe

How to record something digitally in a way that verifies it has not been 'shopped

This is an experiment in remote education.

I want to film myself taking an exam as a way of verifying that the exam was done under exam conditions, without needing someone physically there to.. what's the word I'm after... invigilate the exam.

There are many problems to solve here and I am happy to hear your suggestions regarding any of the problems you imagine and possible solutions to them, but to maintain a bit of focus i want to concentrate on verifying the authenticity of the recording.

I have been imagining a version of the classic fitenss magazine before and after photos.

(where someone stands holding a copy of last months generic fitness magazine and their distended bare belly and then another photo the following month of them holding this months issue and their now rock hard abs after just one month of generic fitness magazine's one-month-rock-hard-abs-program)

So here's my current imagining:

The exam will be made available to me at a specific time*

The I will complete the exam so the work is done in view of the camera.

In the field of view of the recording there is a screen which is showing a random sequence of QR codes.

These QR codes are generated by a server which is recording the time each code was served.

Each frame of the video is then sent back to the server that makes a hash from the frame and the time using a secret key.

There will be a time lag between when the QR code was served, and when the frame containing the image of that code was hashed but it will be short.

If the video is edited, the correspondence between the frames, the times, and the QR codes will be broken.

So any trickery would have to be happening real time or it would be revealed.

In addition there will need to be a similar mechanism to verify the audio recording.

When I am finished the exam I will show all the work to the camera and place it in a provided envelope and close the security seals all in front of the camera.

The recording, the qr codes, the times and the hashes will be permanently stored on the server.

Does a system - or parts of a system - like this already exist?
If not, do you have any implementation suggestions?

*
Ok here's the extra peripheral stuff I don't want to be the focus of the question because otherwise the question just gets to big

I'm not sure how to deliver the exam, probably via the screen showing qr codes, where gestures in front of the camera provide a simple interface for navigating the text so that the I do not need to be provided with any kind of controller. Ideally I would do all the work on a whiteboard so the work is visible as it is done, but I don't know if this is a practical condition to impose on students.

Perhaps a convex mirror would need to be at the front of the room, but then what about the roof? What about messages sent by an accomplice using lasers shining on the floor? the list of potential exploits is long. Once the system is set up I'm going to try cheat it, and put the videos online and see if the public can accurately detect the cheats, and if not revise the set up.
posted by compound eye to Computers & Internet (21 answers total) 2 users marked this as a favorite
 
Why not just have CNN playing in the background while you're taking the exam? You might argue that you could edit the CNN feed into the video after the fact, but if so couldn't you also just edit the whole series of QR codes into place after the fact?
posted by alms at 8:08 PM on April 28, 2011


In security, we would ask: what's your threat model? Does the cost of preventing an attack equal the probability of a successful attack times the cost of that attack?

The easiest way to do what you want would be to just have a human being watch the video feed and occasionally ask the test-taker to raise his hand (to verify that it's real-time). Oh, and we need a really high quality video feed so you could see eg. a scrap of paper or a Bluetooth headset. Let's call that Plan A.

This prevents one attack: a student taking the exam early and then pretending to take it for the camera (Attack 1). It does not prevent an accomplice giving answers to the student off camera (Attack 2).

The system with the QR codes is an elaboration of Plan A, so let's call it Plan B. Plan B would be incredibly expensive to develop, and it would be vulnerable to a new attack: an ingenious student using real-time video editing software to overlay the current QR code on an old video feed (Attack 3).

Ok, so now we have to prevent Attacks 2 and 3. I think Attack 2 will require some kind of specially designed room. Or maybe a 360-degree camera (they do make those, they're not even too expensive). Maybe Attack 3 is too unlikely to worry about.

In real life the right way to do this would be to send the student to the nearest Prometric location, which will happily give them any test you want and supervise the student taking it...for a fee of course.
posted by miyabo at 8:18 PM on April 28, 2011 [1 favorite]


nice idea alms.

The qr code would be served at time A,
then the frame would be hashed by the server at time B.

This verifies that the frame was made between time A and B

If there is a contiguos sequence of frames with matching sequence of verified times, then it makes it impossible (i think) to have edited the video afterwards.

Neither the exam or the video could have been done in advance because the exam wasn't available in advance, and the video shows the work being done.
posted by compound eye at 8:19 PM on April 28, 2011


Why not just have CNN playing in the background while you're taking the exam?
An exploiter could simply record whatever CNN is playing at the appropriate test-taking time, and then play the recording back while actually taking the test.
You might argue that you could edit the CNN feed into the video after the fact,
The video wouldn't need to be edited at all.
but if so couldn't you also just edit the whole series of QR codes into place after the fact?
Presumably the point of sending the video back to the server in real time and generating a hash from a frame and the current time is so as to be able to later verify that the frame really was recorded at the time that it purports.
posted by Flunkie at 8:20 PM on April 28, 2011


This seems extremely delicate: prone to having the video be "stalled" waiting for more QR codes if the connection between the server and the video station is interrupted or lossy. What are the assumptions about the minimum amount of time necessary to alter a frame from its original composition? And why wouldn't I just modify everything EXCEPT the QR code assuming I could do the modification fast enough?

Basically here what's happening is that the QR code acts as a verification that any frame containing the code was recorded in the interval between its generation and when the frame was received. If your assumption is that it's impossible to perform frame modification fast enough (in that interval), I think that holds.
posted by axiom at 8:23 PM on April 28, 2011


You videotape the exam. Digitally sign the exam video file immediately. The verification of this digital signature depends upon the video bytes remaining untouched (as well as the integrity of your secret key). If the video data get 'shopped, when you try to verify the digital signature with the edited video and your secret key, you get the wrong result.
posted by Blazecock Pileon at 8:27 PM on April 28, 2011


Maybe I've misread the question, but simply taping and signing doesn't seem to suffice. It just proves that the video file wasn't tampered with after being signed. It doesn't prove that the video file was created at the appropriate time.
posted by Flunkie at 8:32 PM on April 28, 2011


If the video is being streamed to the server anyway, and if there's no concern of the test being taken too early (since it will be "made available at a specific time" - presumably at the valid start time), then I don't understand the need for all the complicated QR stuff and hashing and whatnot. The server can just record the video as it is streamed in, and it knows what time it is.
posted by Flunkie at 8:35 PM on April 28, 2011


It doesn't prove that the video file was created at the appropriate time.

If the video contains a timestamp that is part of the signed video data, then the timestamp cannot be forged or altered after the fact, without breaking the signature.
posted by Blazecock Pileon at 9:09 PM on April 28, 2011


Yeah all you need is live streaming of video of the exam-taking to a server, the exam to be made available at a set time, and there to be a set window for submission of the answers. That way you can't take the exam early, you can't take it late, and the exam-taking process was recorded. If you provide three feeds with slightly overlapping camera angles you can show 1) your face and the surrounding part of the room 2) your hands typing and 3) the screen. Toss in server-side editing of the exam answer file (a la Google Docs) and most cheating avenues are closed off.

The main attack vector I can think of is someone who is not in view of the cameras but can read the screen (e.g. either a carefully positioned person or someone using a remote screen viewing program) reading the questions, looking up answers on a separate computer, and telling you the answers. A high quality audio feed from the cameras would help with that, but there may be other ways to communicate (e.g. being fed Morse code via a machine that lightly taps on your foot). At that point you're dealing with a pretty sophisticated cheating setup, though.

And this assumes the entire exam can be taken without any breaks. Once the test-taker leaves the room all bets are off. Maybe equip them with a helmet cam and mic?
posted by jedicus at 9:15 PM on April 28, 2011


Thank you everyone for all your suggestions, even though I want to think about them for a while before I reply, I want to post this note now to convey my appreciation.
posted by compound eye at 10:35 PM on April 28, 2011


OK, forget my suggestion of how to do it,

How would you do it?
posted by compound eye at 12:26 AM on April 29, 2011


How would you do it?

I'd hire a proctor of the tester's choosing.
posted by hades at 12:30 AM on April 29, 2011 [1 favorite]


(Sorry about that previous non-answer. Here's the long one.)

If this is an experiment to see if you can set up a system that will certify that you yourself have not cheated the system, the answer is that no amount of technical wizardry will work, because the attacker and the defender are one and the same. All you have to do is compromise the system you created, have the server re-stamp-and-sign the video stream once you've made whatever edits you want, and there's no way for me (the person to whom you're trying to prove the validity of the video stream) to detect the fraud, since I'm relying on you (the trusted server) to vouch for you (the attacker). You need a trusted third party involved in the timestamping and digital signatures.

But ignoring that issue, it seems to me that there are a few distinct problems:

1: Verifying that an a/v stream was received by a trusted server at a specific time.
2: Verifying that what is depicted by the a/v stream actually took place in real-time as it was received.
3: Verifying that the a/v stream has not been tampered with since it was received.

1 and 3 are pretty easy; you don't even need the QR code stream for that. The server embeds a timestamp in the stream as it is received, and hashes each frame with a secret key. You can't create a valid hash for a subsequently-edited video frame without knowing the key, so problem solved.

I think 2 is pretty much impossible, even with the QR code stream. It is possible to combine pre-recorded video and live video in a way that's virtually undetectable. I'm pretty sure this is how, for example, Derren Brown's live TV special where he predicted a lottery drawing worked. (Possible attack: the test taker pre-records video of the empty exam room. At the time the exam is taken, he combines a portion of that video with the live video stream, inserting a pre-recorded empty corner of the room over his confederate who is in that corner during the exam.)

The fact is, if you're letting the test taker provide the a/v stream, there is _some_ attack they can come up with to cheat the test. So you'll need to have a trusted third party operate the camera(s). At which point, why not just have that third party proctor the exam?

I mean, it's a neat idea, but even if it were technically possible there would be non-technical attacks. For example, if this is distance education, how do you know that the person who appears on the video is the person registered for the class? If you know what the student looks like through some other channel, how do you know the person on the video isn't her twin sister? (These are problems for non-distance education and remotely proctored exams as well, of course.)

How much bandwidth do you have at your remote location? Can you really stream video from there with enough resolution to rule out earpieces, cheat sheets, etc? How many cameras do you require your test taker to have available for sufficient room coverage?

Realistically, there is no perfect solution. If this is really about distance education, then the goal is to make attacking the system more work than just learning the material. If this is about the generic case of proving that something on video 1) happened when you say it did and 2) actually happened as portrayed with nothing hidden, left out or inserted, that's a lot harder, maybe impossible.
posted by hades at 1:36 AM on April 29, 2011


given the two general attack classes of taking the exam at a time other than represented, and a 3rd party passing info to the examinee,

the first can be solved by closing the time window - make questions available on-demand, and stream the video back to a trusted server in realtime.

the second can't be solved absolutely, but to even get close you're going to need either a very expensive technological solution (perhaps including a faraday room) or a much cheaper human proctor.

some problems really are better solved with meat than metal.
posted by russm at 1:38 AM on April 29, 2011 [1 favorite]


Hello Hades,

make attacking the system more work than just learning the material

I see the goal as to make the a system that is as robust as having a human proctor in the room, not a perfect system, but a good enough system, and to do it with the minimum of set up requirements for the student.

So the twin exploit is not a big concern, both methods are just as vulnerable. The third party assistant exploit needs to be as risky with system as it is with a human proctor in the room.

If I throw away the goal of minimal set up requirements for the student, I was thinking the student could do all the work on a sheet of glass with a white board marker, filmed from both sides, and the qr codes could be projected on the student and the glass, from projectors offset at an angle so that the qr image is distorted across their body.

I'm trying imagine the simplest solution that can replace a human proctor, but I'm not necessarily looking for a solution that is simpler than a human proctor.
posted by compound eye at 3:05 AM on April 29, 2011


how do you know that the person who appears on the video is the person registered for the class?

the video would stay on the server as a document testifying the person had completed the exam.
If the video isn't them, we might not know. But the video will be a public document available on the server, waiting to embarrass a cheat.
posted by compound eye at 3:17 AM on April 29, 2011


I guess I'm not 100% sure what you are trying to guard against.

If you want to make sure that the person taking the exam is doing so at the correct time then make sure that they have three hours (or whatever) from receipt of the exam to finish and return it (upload to website or whatever). The QR codes can prove that the test is done in real time, but if you set the start and end then the test pretty much has to be done in real time.

Maybe I'm dense, but it seems to me that you've come up with a solution without defining the problem. A test proctor is a solution to a problem, but unless I know what you think the problems are then I can't come up with a different solution that guards against them.

Possible problems:

1. The student has looked up the answers before getting the test
2. The student looks up the answers during the test (taking a break to do it. See 4)
2a. The student looks up the answers while taking the test (without a break)
3. The student is gven the answers by someone sitting next to them
4. The student takes longer than the specified time to take the test
5. Someone else shows up and takes the test for the student
5a. The student takes the test, but at the last minute swaps in someone else's answers
posted by It's Never Lurgi at 8:55 AM on April 29, 2011


I'm trying imagine the simplest solution that can replace a human proctor, but I'm not necessarily looking for a solution that is simpler than a human proctor.

How about this:

A 20'-long cargo container is outfitted with a time-release lock, full-coverage interior video and audio recording, RF shielding and a spectrum analyzer (along with a desk, chair, whiteboard, pencils, etc). The test is placed inside the container and the whole thing is shipped to the student. If you want, there can be a time-release safe inside the container to keep the test in. The cameras are mounted inside tamper-evident boxes. If the doors open more than twice, once at the beginning and once at the end, the student fails. If the doors are open for too long, the student fails. If there is too much time between the start and end of the test, the student fails. If anything suspicious is seen on the spectrum analyzer, the student fails. If the cameras show signs of tampering, the student fails. If the locks show signs of tampering, the student fails. If the power supply shows signs of tampering, the student fails. The completed test is left inside the container, which is closed and locked and shipped back to the tester for grading.

Is "insanely expensive" a concern here?
posted by hades at 9:17 AM on April 29, 2011


Somewhat related: Nikon Image Authentication Software Cracked, Rendered Useless
posted by hades at 12:26 PM on April 29, 2011


Technological solutions are going to be easily defeated. I'd have a "virtual proctor" administer an oral exam. Have the proctor give an anti-video instruction, e.g., raise your right hand, then read off a question from a bank. I would have the tester rotate 90 degrees every question to reduce the impact of answers or notes posted out of sight. I would require hair to be pulled back so ears are exposed.

The exam would be administered via a flash-based web app that communicates back with the server to download each new question, so there'd be a flaggable time series to match against the video feed.

I think this reasonably mitigates most attack scenarios.
posted by bfranklin at 1:59 PM on April 29, 2011


« Older CarDetailingFilter: Is profess...   |  Why is the guy who appears to ... Newer »
This thread is closed to new comments.