Watson buzz-in

February 15, 2011 8:43 AM

How does Watson know when to buzz-in during this week's Jeopardy?

For a normal game, this discussion suggests an assistant enables the buzzers shortly after Trebek finishes talking. Is Watson getting fed that "ok-to-answer" signal or is it actually listening for the end of the question to buzz-in?

posted by jaimev to grab bag (29 answers total)

I don't know the answer, but I know it isn't listening to anything. It's being given a text file to read.
posted by roomthreeseventeen at 8:48 AM on February 15, 2011

From the IBM Research news blog:

"When host Alex Trebek finishes stating a clue, a human operator (who works for Jeopardy!) turns on a “Buzzer Enable” light on stage to indicate that contestants can “buzz in” and answer. At exactly the moment the “Buzzer Enable” light is activated, Watson’s system receives a signal that the buzzer is open."
posted by brainmouse at 8:50 AM on February 15, 2011 [2 favorites]

From the above link, also relevant:

The best human contestants don’t wait for, but instead anticipate when Trebek will finish reading a clue. They time their “buzz” for the instant when the last word leaves Trebek’s mouth and the “Buzzer Enable” light turns on. Watson cannot anticipate. He can only react to the enable signal. While Watson reacts at an impressive speed, humans can and do buzz in faster than his best possible reaction time.
posted by milestogo at 8:54 AM on February 15, 2011

Based on that discussion (and a friends link on FB), the contestant's buzzers don't work until the assistant enables them. Thus, some folks might know the answer before Trebek finishes, but have to wait to buzz in. (Ever see some contestants hammering on their buzzer ?).

So based on that Watson can't 'buzz in' until the assistant unlocks, and watson gets the notification 'you can buzz now'.

(FB friend's comment is you are now playing against the fastest reflex possible on a machine:

Watson does in fact have a buzz-in advantage: http://www.tnr.com/article/83337/ibm-watson-computer-jeopardy

"According to David Shepler, who is IBM’s Challenge Program Manager for the Watson project, “The buzzer is enabled when the clue is done being read, when Alex Trebek gets to that last syllable, and the guy off stage pushes a button. That’s when people can buzz in, and at the same time a signal is sent to Watson saying the same thing—telling Watson that it can buzz in if it so desires.” This is akin to playing against an opponent with near-perfect reflexes."

posted by k5.user at 8:55 AM on February 15, 2011

k5.user: "FB friend's comment is you are now playing against the fastest reflex possible on a machine"

That may or may not be true, depending on when Watson starts trying to answer the question, which is not entirely clear to me. If the buzzer signal also prompts Watson to start calculating the answer, then you have to factor in that time, which may be more or less than its "reflex advantage".
posted by mkultra at 9:02 AM on February 15, 2011

depending on when Watson starts trying to answer the question, which is not entirely clear to me


Watson starts calculating the answer as soon as he receives the question, and the question is sent to him as soon as it's displayed. It wouldn't make any sense if he didn't start thinking until after he had to answer... Also from that blog post:

"At exactly the moment that the clue is revealed on the game board, a text is sent electronically to Watson’s POWER7 chips. So, Watson receives the clue text at the same time it hits Brad Rutter’s and Ken Jennings’ retinas.... For some clues he may not complete the question answering computation in time to make the decision to buzz in."
posted by brainmouse at 9:05 AM on February 15, 2011

As I watched the show last night, I thought it was interesting that Watson seemed to be routinely beating the humans to the trigger and I wondered if its reaction time was inherently better than its human competitors, but this snippet from the blog entry linked above suggests otherwise.

The best human contestants don’t wait for, but instead anticipate when Trebek will finish reading a clue. They time their “buzz” for the instant when the last word leaves Trebek’s mouth and the “Buzzer Enable” light turns on. Watson cannot anticipate. He can only react to the enable signal. While Watson reacts at an impressive speed, humans can and do buzz in faster than his best possible reaction time.

Also, the tricky bugger knew right where that Daily Double was, didn't it? I guess the algorithm that places the Daily Double needs a little tweaking.
posted by Rock Steady at 9:06 AM on February 15, 2011

But when does he get the text file and start his algorithms? No doubt a computer can read faster than a computer can...
posted by arveale at 9:06 AM on February 15, 2011

IAAWTM, IANYWTM (I am a Watson team member, I am not your Watson team member)

The buzzer is enabled by an assistant pressing a button offstage. At that point, a light flashes for the contestants to notify them that the buzzer opened. Watson is deaf and blind, so the light is no good for him. He gets an electronic "oh hey the buzzer opened" signal. At that point, all three contestants have to physically push down the button on their buzzer to ring in - the humans with their fingers, Watson with a piston/solenoid rigged thingie.

Like humans, Watson does not wait until the buzzer opens to start thinking. Everyone's ideal, whether they're a human or a computer, is to have the answer figured out before the buzzer ever opens.
posted by badgermushroomSNAKE at 9:08 AM on February 15, 2011 [27 favorites]

Huh, I thought they would make Watson do some visual processing in order to see the lights turn on indicating it's OK to buzz in. Otherwise that's a pretty big advantage. Haven't seen yesterday's ep yet though.
posted by kmz at 9:16 AM on February 15, 2011

Of course, even with having to do visual processing it would only add a trivial amount of time to Watson's reaction time, but it would be something.
posted by kmz at 9:16 AM on February 15, 2011

If you have time to watch it, the latest episode of NOVA is all about the path to the match.
posted by Cat Pie Hurts at 9:20 AM on February 15, 2011 [4 favorites]

Of course, even with having to do visual processing it would only add a trivial amount of time to Watson's reaction time, but it would be something.

It really wouldn't. Opto-couplers would be easily placed right on the light. They'd be no slower than any other electrical signal.
posted by odinsdream at 9:25 AM on February 15, 2011

As a former contestant I can say that Watson certainly has a speed advantage if the above part about the buzzer indicator is true. Exactly when to buzz in is one of the hardest parts of the game, and it is given an indicator of when that is. Couple that with the faster than human reaction time and it's even more significant. For a truly level playing field they should have an audio processor just to determine when Trebek is done talking, then it should have to guess/predict when the buzzer is armed. It would still likely do pretty well, but the buzzer is a BIG advantage.
posted by true at 9:28 AM on February 15, 2011 [1 favorite]

The best human contestants don’t wait for, but instead anticipate when Trebek will finish reading a clue. They time their “buzz” for the instant when the last word leaves Trebek’s mouth and the “Buzzer Enable” light turns on. Watson cannot anticipate. He can only react to the enable signal. While Watson reacts at an impressive speed, humans can and do buzz in faster than his best possible reaction time.

That sounds good on paper, but anyone watching the show last night could see that, over and over again, both contestants knew the answers and were buzzing in but were getting beaten to the punch by Watson. Apparently, somehow, an assistant signaling Watson that it was ok to answer and then Watson reacting to that signal was consistently quicker than the reactions of Jennings and Rutter.
posted by iconomy at 9:31 AM on February 15, 2011 [2 favorites]

Also, as I recall the lights showing that the buzzers are armed are NOT visible to the players, or at least are activated far enough after the actual arming of the buzzers that if you wait for them you have no chance of being first. The only way to get in first is to time the end of Trebek's answer.
posted by true at 9:32 AM on February 15, 2011

Also, as I recall the lights showing that the buzzers are armed are NOT visible to the players, or at least are activated far enough after the actual arming of the buzzers that if you wait for them you have no chance of being first. The only way to get in first is to time the end of Trebek's answer.

The lights are definitely visible to the player (at least for the the last many years), but yeah, you can't wait for the lights. Buzzing in right is hugely difficult.
posted by kmz at 9:35 AM on February 15, 2011 [1 favorite]

Oh, and if you guess the buzzer arming and are too early you get locked out for half a second or so - basically an eternity. so you have to guess, but there's a huge penalty for guessing too early. Watson doesn't have to guess.

I should probably wait and get all my answers together at once, but I guess I'm just trying to time the buzzer.
posted by true at 9:37 AM on February 15, 2011 [1 favorite]

Not really fair, is it? Humans have to listen and parse. Watson should, too.
posted by theora55 at 9:43 AM on February 15, 2011

true: For a truly level playing field they should have an audio processor just to determine when Trebek is done talking, then it should have to guess/predict when the buzzer is armed.

For the record, I don't think Watson is the first contestant to be given buzzer-open notifications in a form based on what senses of his work. I believe blind contestant Eddie Timanus was notified of the buzzer opening by a tone when he played, although I'm not 100% certain of that.
posted by badgermushroomSNAKE at 9:48 AM on February 15, 2011

I thought I read somewhere that Eddie Timanus went by the sound of Alex's voice, which was one of the reasons he was so impressive. As a Jeopardy! (tm) champion, I found the buzzer the hardest part. There's a learning curve to it, and you just have to get the feel for it.

I would guess that one of Watson's greatest advantages is that he can't buzz in too early (when you buzz in too early, you get locked out for a few eternal milliseconds). I wonder if the humans will get back into their signaling device grooves a bit more on tonight's show.
posted by mskyle at 10:19 AM on February 15, 2011 [1 favorite]

...Watson certainly has a speed advantage... For a truly level playing field they should have an audio processor just to determine when Trebek is done talking, then it should have to guess/predict when the buzzer is armed...

I helped test Watson's Jeopardy! playing skills last year (one of the coolest days of my life to date). Despite any "advantage" Watson has, it's not magic. I came within a Final Jeopardy question of beating it, and it's entirely possible that I beat it to the buzzer at some point. And I'm no queen of the Jeopardy! buzzer, by any means.

You would think that Watson would totally "have an advantage" at practically any aspect of gameplay, but at least when I interacted with the technology, it wasn't as impossible to beat as you'd think.
posted by Sara C. at 10:40 AM on February 15, 2011

I would guess that one of Watson's greatest advantages is that he can't buzz in too early

Absolutely.
posted by iconomy at 10:45 AM on February 15, 2011

From what I could tell, Watson would not buzz in at all if he didn't think he had a good question. Most human players seem to try to buzz in for every answer, as this is faster than thinking about whether they know the question first, and you can't score if you don't buzz in. They have a few seconds to actually come up with the question after Alex calls on them, after all. If they respond correctly enough of the time, the occasional wrong question will not hurt them badly enough to lose. (This doesn't apply, of course, if someone else beats them to the punch and they don't know the answer; they won't buzz in in that case, but they've had plenty of time to think about it.)
posted by kindall at 11:15 AM on February 15, 2011

Relevant chat with Ken Jennings on this subject.
posted by jenfullmoon at 11:42 AM on February 15, 2011 [2 favorites]

Most human players seem to try to buzz in for every answer,

As a former Jeopardy! contestant, I would say that this is not true. One of the things I had to do in order to get good enough at the gameplay to consider trying out for the show was to get a handle on buzzer strategy. While our puny monkey brains can sometimes play tricks on us about this, it is pretty bad strategy to just buzz in on everything. You really do need to hone the skill of parsing the questions quickly and determining whether to buzz or not.

I'm not sure if the televised games are showing the audience this, but Watson has a similar threshold that seems to work in the same way. The computer comes up with four potential answers (many of which are laughably weird/incorrect/out of left field) and will not ring in if a certain percentage of certainty cannot be established*. At the testing last year, observers waiting for our turn to compete could actually watch this process on a screen - I think you can see the "thinking process" screen in some of the NOVA footage and other supporting materials, if Jeopardy! isn't showing it themselves.

*And, yes, I think it is possible for Watson to be really certain about an answer that turns out to be wrong. Again, it's not magic. The thing can be beat.
posted by Sara C. at 11:57 AM on February 15, 2011 [1 favorite]

According to the interview with Ken Jenninngs:

Q: On last night's show, I noticed you buzzing in even when you didn't know the answer right away, taking a second after Alex called on you to finish reading the question and give an answer. In your opinion, is this the only way to beat Watson?

A: Good human players do this all the time: you buzz when you see something that trips some "This looks familiar!" switch in your brain and count on dredging it out in the five seconds after Alex calls on you.

Watson can't do this: it only buzzes once it has an answer in mind and a sufficiently high confidence interval. As weird as it sounds, yes, the human brain still has a speed advantage over a 2,880-processor-core computer.
posted by kindall at 12:59 PM on February 15, 2011

While we're on the subject of advantages, I think Watson's biggest advantage is that it's playing two incredibly good human players rather than one. If you had either Jennings or Rutter playing it head to head I think they would crush it, and if you had one human against two computers I think even a relatively good player would win.
posted by true at 1:14 PM on February 15, 2011 [4 favorites]

Here's a small rule change that would temper the hair-trigger advantage for machines and humans alike: staggered buzzers.

Meaning three lights for three players, each gate programmed to open at a random interval (truly random, not pseudo) between 0s and 2s after question's end. Timeouts stagger to match.

Expected delay would be 1s per contestant per question, so it's not a handicap system, but it would permit a player's buzzing advantage to tilt outcomes only around a sweet spot of the normal distribution.
posted by Municipal Hare at 12:41 PM on February 16, 2011