IBM's DeepQA and the Jeopardy! Challenge

[edvz]

I'll jump in, because -- only speaking for myself -- I do tend to disagree with Ken.

Well hat's off to you. The best human Jeopardy players can still improve, and
they might have to in order to beat Watson.

Perhaps like chess, despite its simplicity, the game can never be played perfectly
by man or machine.

[Muskrat]

I think a bigger issue with the man/machine divide would be the psychology of the game. Although Ken an Brad are the top pros in this field, it's only human to react emotionally to things like, e.g., the other guy doubling his score on a DD, or you running an entire category. People feel things like "game momentum," and computers don't. That isn't necessarily a disadvantage -- Watson can't get a boost from adrenaline, but he also can't get flustered. Each event is probably like a new day for Watson -- "Oh, look, here's a clue... here are the three scores... here's the time left, let me start parsing this one ..."

How will the buzzer timing work? That would be an area where machine-like reflexes might be beat by actual machine reflexes.

[jzerocsk]

I don't think the wagering calculus needs to be as complex as it's being made out to be. Certainly there may be some theories that can lead to calculating the optimal score to the unit dollar for any situation but Jeopardy games rarely come down to that margin.

Sometimes the game is decided by a $1 margin but it is almost universally someone with a "tall stack" betting just enough to cover a "short stack" player's double...

Which is to say, I don't think a computer being able to "overthink" the wager is at much of an advantage over a human player with a reasonable grasp of the game's fundamental betting strategies. Heck, if it can whip out enough correct answers to always secure a runaway the betting is completely irrelevant.

[edvz]

This Nobel prize winning physicist has a four time Grammy award winning grand-daughter.

They are the three people who wrote the music and lyrics to the musical "Chess."

This member of the original Amphibians MIT Blackjack counting team has finished in the money 24 times at World Series of Poker events.

Only the Blackjack question stumped me, though I certainly read the book about the MIT team.
I read somewhere else, it was mostly fiction.

[ArtVark]

Well I'm 1/3 on those, although I'm sure they wouldn't accept "Andrew Lloyd Webber and those 2 guys from ABBA."

Um, I think you mean Tim Rice.

Andrew Webber only saw Chess once, threw up, then cried "Mamma Mia!"

Oh, Newton-John for the other, but no idear for the BJ. Carrie Henn??

Well technically, the answer to Q1 is Max Born. The Amphibian is Andy Bloch.

The point that I was trying to make is that just like the previous November 22, 1963 clue where there is an obvious hint unrelated to the subject, the key to
a clue in a subject that you may be an expert at could be a hint in a subject in which you are not an expert.

[edvz]

...The Amphibian is Andy Bloch.

The point that I was trying to make is that just like the previous November 22, 1963 clue where there is an obvious hint unrelated to the subject, the key to a clue in a subject that you may be an expert at could be a hint in a subject in which you are not an expert.

Name is familiar, but only as a team player. I don't know poker.

Right, I got that. But I don't doubt the IBM team
has crunched the J! Archive to find out how often that is true, and
incorporated their findings into Watson's wagering algorithms.

[ArtVark]

...The Amphibian is Andy Bloch.

The point that I was trying to make is that just like the previous November 22, 1963 clue where there is an obvious hint unrelated to the subject, the key to a clue in a subject that you may be an expert at could be a hint in a subject in which you are not an expert.

Name is familiar, but only as a team player. I don't know poker.

Right, I got that. But I don't doubt the IBM team
has crunched the J! Archive to find out how often that is true, and
incorporated their findings into Watson's wagering algorithms.

But the point I was trying to make is that in your examples, you would bet big on Physics and low on Constitutional Law. But you may get a physics
category, and the clue may be "This Nobel prize winning physicist was born the same year that the 17th amendment was ratified."

[Paucle]

Um, I think you mean Tim Rice.
Andrew Webber only saw Chess once, threw up, then cried "Mamma Mia!"

After that he asked, "Hey, who stole half my last name?

[rockgolf]

Suspects include Frank Lloyd Wright & David Lloyd George.

Weird anagram: Lloyd = Dolly

[Paucle]

Weird anagram: Lloyd = Dolly

you rang?
http://www.answerplus.ca/

[Sequin]

An article with some technical info:

http://www.theregister.co.uk/2010/12/20 ... _jeopardy/

[edvz]

After that he asked, "Hey, who stole half my last name?

My attorney. Also that nice joke about Ken

[ChadHahn]

Instead of a blue dot (making a return to show business return after the Anna Nichole case?) why don't they use the likeness and voice of Max Headroom?

Chad

[rick1013]

From the referenced UK Register webticle:

Humanity is rolling out Kenneth Wayne Jennings III, who has held the top rank in the game the longest (with 74 wins), and Bradford Gates Rutter, who has racked up the most cash ($3.25m) on the show during his reign earlier in the decade.

Makes Ken and Brad sound like presidential assassins or TV sidekicks ala Paul Michael Glaser and Philip Michael Thomas.

[marpocky]

A preview has shown up. It seems they're filming the real games tomorrow.

http://www.engadget.com/2011/01/13/ibms ... rdy-pract/

So, in February IBM's Watson will be in an official Jeopardy tournament-style competition with titans of trivia Ken Jennings and Brad Rutter. That competition will be taped starting tomorrow, but hopefully we'll get to know if a computer really can take down the greatest Jeopardy players of all time in "real time" as the show airs. It will be a historic event on par with Deep Blue vs. Garry Kasparov, and we'll absolutely be glued to our seats. Today IBM and Jeopardy offered a quick teaser of that match, with the three contestants knocking out three categories at lightning speed. Not a single question was answered wrongly, and at the end of the match Watson, who answers questions with a cold computer voice, telegraphing his certainty with simple color changes on his "avatar," was ahead with $4,400, Ken had $3,400, and Brad had $1,200.

Alright, a "win" for silicon for now, but without any Double Jeopardy or Final Jeopardy it's hard to tell how well Watson will do in a real match. What's clear is that he isn't dumb, and it seems like the best chance the humans will have will be buzzing in before Watson can run through his roughly three second decision process and activate his buzzer mechanically. An extra plus for the audience is a graphic that shows the three answers Watson has rated as most likely to be correct, and how certain he is of the answer he selects -- we don't know if that will make it into the actual TV version, but we certainly hope so. It's always nice to know the thought processes of your destroyer. Stand by for video of the match, along with an interview with David Gondek, an engineer on the project.

Update: Video of the match is up, check it out after the break!

Update 2: And we have the interview as well, along with a bit more on how Watson actually works.
IBM's Watson supercomputer destroys all humans in Jeopardy practice round






While Watson's ability to understand questions, buzz in, and give a correct answer might seem very human-like, the actual tech behind Watson (dubbed "DeepQA" by IBM) is very computer-ey. Watson has thousands of algorithms it runs on the questions it gets, both for comprehension and for answer formulation. The thing is, instead of running these sequentially and passing along results, Watson runs them all simultaneously and compares all the myriad results at the end, matching up a potential meaning for the question with a potential answer to the question. The algorithms are backed up by vast databases, though there's no active connection to the internet -- that seems like it would be cheating, in Jeopardy terms.

Much of the brute force of the IBM approach (and why it requires a supercomputer to run) is comparing the natural language of the questions against vast stores of literature and other info it has in its database to get a better idea of context -- it has a dictionary, but dictionary definitions of words don't go very far in Jeopardy or in regular human conversation. Watson learns over time which algorithms to trust in which situation (is this a geography question or a cute pun?), and presents its answers with a confidence level attached -- if the confidence in an answer is high enough, it buzzes in and wins Trebek Dollars.

[Enhasa]

During the practice round, it stood out that Watson consistently buzzed in before Ken and Brad when IBM's statistics showed that Watson was relatively certain of an answer. I realize that this was somewhat predictable given that a human can't electronically signal its intentions and the computer can, but it's still an enormous advantage for Watson -- one that I had actually thought that the Jeopardy people might try to mitigate in some way.

It seemed after the first category that the IBM Challenge was probably just going to become a question of whether Watson knew enough of the questions, but on the children's books, Ken was able to beat Watson to the buzzer a few times. Watson was only around 60% certain on those occasions, so it seems likely that Watson's buzzing was slightly delayed because the computer needed a little additional time to reach the >50% threshold for answering.

I'm not sure why Watson's top results weren't shown for every question. Is it that IBM doesn't want to show how off Watson's results can be when it doesn't have to? If Watson was wrong each time its results weren't shown (it appears that he was buzzing in for the most part if we're assuming that the radiating green lines are the visual indication of that, so I don't think that's the explanation), that bodes well for Ken.

It's a little foolhardy to make a prediction based on this limited and not fully explained preview, but nevertheless, I have to say I like Watson's chances based on what we've seen. Speed kills.

[rick1013]

Speed kills.

And absolute speed kills absolutely. Or somesuch. "Trey" Jennings and B-Rutt better watch out.

[GersonK]

I wonder if humanity isn't hurting our odds by putting up two champions. It looked like Ken was buzzing in just behind Brad on all the questions Brad got. Presumably most of these cases these were just human friendlier questions and Watson was not buzzing in between Ken & Brad (or maybe Watson was buzzing in early? Didn't see any sign of a lockout though). Replace Brad with Random J! Contestant and I'm thinking Ken gets a lot more of the human friendly questions and the match gets a whole lot closer.

[ADD]

I do not think Ken Jennings or anyone else should compete against Watson. I do not feel that Watson is just a computer. I think it is a form of AI that could really do some harm to people at large, especially if money can be made or if it can be used in law enforcement. I think whether Ken Jennings wins or loses, Watson and the people that built him have won. Call me what you will but, there is a serious moral and ethical problem here.

[Paucle]

Don't worry about it, ADD. It's been confirmed that Watson has been programmed with Asimov's three laws.

[geniusonwheels]

Five rules, if Watson is playing Wheel of Fortune.

[Sequin]

Don't worry about it, ADD. It's been confirmed that Watson has been programmed with Asimov's three laws.

Asimov latter added an extra law (the "zeroeth" ) which was similar to the first law, but referred to humanity, rather than "a human"

[exIBMer]

Maybe this is nothing. But I'm more than a little concerned to see IBM's YouTube videos using the exact same categories and questions that have appeared on Jeopardy over the last several days. [A December 15, 2010 video, for example, used a question (Who is William Pitt?) that appeared on the actual January 27, 2011 Jeopardy game show broadcast.]

See: http://www.youtube.com/watch?v=VjHMYuGkzlU&NR=1
See: http://www.youtube.com/watch?v=7kOEmupSHB8

I'm sure there is big money involved. But, as an outsider, I must wonder if Sony is bending their broadcast standards and practices for the benefit of this matchup.

If the January 27, 2011 show was taped after IBM filmed this particular sparring match (which was posted to YouTube on December 15, 2010), I see a huge ethical issue for Sony, and for the contestants that appeared on the January 27 show.

Even if the January 27, 2011 show was taped prior to December 15, 2010, I was under the impression that there were non-disclosure rules in place that prohibited information about a future show's questions or results from being leaked in advance of the airing of that show? [Hey, Ken! Can you tell us who won your match with Watson? Can you tell us the Final Jeopardy Question? Can you tell us one of the categories in the Double Jeopardy Round? I didn't think so.]

OK. If Jeopardy/Sony has given IBM the questions to taped (but unaired) shows, do they do the same for all contestants? If not, would that give Watson have any advantage?

I'm an ex-IBMer. When I worked for IBM, they were one of the greatest companies in the world to work for. I still have a lot of respect for IBM, and I still own IBM stock. But this cozy relationship with Jeopardy/Sony makes me very uneasy about the integrity of this tournament.

[TheConfessor]

Maybe this is nothing.

Yes, it's nothing.

But I'm more than a little concerned to see IBM's YouTube videos using the exact same categories and questions that have appeared on Jeopardy over the last several days. [A December 15, 2010 video, for example, used a question (Who is William Pitt?) that appeared on the actual January 27, 2011 Jeopardy game show broadcast.]

I'm sure there is big money involved. But, as an outsider, I must wonder if Sony is bending their broadcast standards and practices for the benefit of this matchup.

Rest assured, the exact same rigid broadcast standards and practices were followed for this match as for every other Jeopardy! match. I was there. No corners were cut.

If the January 27, 2011 show was taped after IBM filmed this particular sparring match (which was posted to YouTube on December 15, 2010), I see a huge ethical issue for Sony, and for the contestants that appeared on the January 27 show.

Even if the January 27, 2011 show was taped prior to December 15, 2010, I was under the impression that there were non-disclosure rules in place that prohibited information about a future show's questions or results from being leaked in advance of the airing of that show? [Hey, Ken! Can you tell us who won your match with Watson? Can you tell us the Final Jeopardy Question? Can you tell us one of the categories in the Double Jeopardy Round? I didn't think so.]

OK. If Jeopardy/Sony has given IBM the questions to taped (but unaired) shows, do they do the same for all contestants? If not, would that give Watson have any advantage?

Your concerns have no logical basis. There is nothing underhanded going on. Jeopardy! and IBM have been very open about their joint efforts to make this Man vs. Machine challenge successful for both parties, and entertaining to viewers. Jeopardy! gave IBM access to past show questions for use in its internal sparring matches. Obviously, the internal sparring matches would be valid only if the contestants had not seen the questions before. If there were anything devious about this, do you really think IBM would be promoting it in YouTube videos?

How would it be an advantage to Watson to see questions that are guaranteed not to be used in its televised match? That makes no sense. And yes, Jeopardy sometimes allows human contestants to see questions that were used on shows that have not yet been broadcast. The public is welcome to attend the taping of shows that often will not be broadcast for several weeks or months. You should try it sometime, it's a lot of fun. I recommend it to anyone who hopes to be a contestant, since it will make them more knowledgeable and comfortable in the studio environment when they get their chance at the podium. In any case, seeing questions from someone else's episode won't give you an advantage when you have to answer different questions in your own episode. That's true for both humans and computers.

I played eight games against Watson earlier this month. I have found it fascinating to watch the recent broadcasts to watch the differences and similarities when the same boards are played by different players. I especially enjoyed the times when a triple stumper on the show was a triple get in my sparring games, but we weren't playing for real money, so I have to take my reward wherever I can find it.

I'm an ex-IBMer. When I worked for IBM, they were one of the greatest companies in the world to work for. I still have a lot of respect for IBM, and I still own IBM stock. But this cozy relationship with Jeopardy/Sony makes me very uneasy about the integrity of this tournament.

Your first three sentences above also apply to me. Your closing sentence is unfortunate, since it totally mischaracterizes the impeccably high standards of integrity that all parties have upheld throughout this project.

[Ken Jennings]

What Ed said. There is no "cozy" relationship between IBM and Sony beyond what was necessary to collaborate on a TV show, and a very rigorous show "bible" spelled out for all participants what concessions were or were not being made to Watson.

The sparring games used a few dozen game boards that had taped but not yet aired, but I fail to see how that could produce a material advantage for Watson. For one thing, most (all?) of those games DID air between the sparring games and the televised exhibition match, so Brad and I probably saw them as well. To expand Ed's hypothetical, would a match between three human contestants be grossly unfair if one lived in LA and had attended some tapings that the other players hadn't? Because that probably happens all the time.