IBM's DeepQA and the Jeopardy! Challenge

[jbenjam]

I work at NASA GSFC, and saw this colloquium announcement today. Maybe some of you are interested in this topic (even if you can't attend the colloquium):

<begin copy-pasta>:

David Ferrucci
An Overview of DeepQA for the Jeopardy! Challenge

Wednesday, November 17, 2010
Building 3 Auditorium - 11:00 AM
(Coffee and cookies at 10:30 AM)

Computer systems that can directly and accurately answer peoples' questions over a broad domain of human knowledge have been envisioned by scientists and writers since the advent of computers themselves. Open domain question answering holds tremendous promise for facilitating informed decision making over vast volumes of natural language content. Applications in business intelligence, healthcare, customer support, enterprise knowledge management, social computing, science and government would all benefit from deep language processing. The DeepQA project (www.ibm.com/deepqa) is aimed at exploring how advancing and integrating Natural Language Processing (NLP), Information Retrieval (IR), Machine Learning (ML), and massively parallel computation and Knowledge Representation and Reasoning (KR&R) can greatly advance open-domain automatic Question Answering.

An exciting proof-point in this challenge is to develop a computer system that can successfully compete against top human players at the Jeopardy! quiz show. Attaining champion-level performance Jeopardy! requires a computer system to rapidly and accurately answer rich open-domain questions, and to predict its own performance on any given category/question. The system must deliver high degrees of precision and confidence over a very broad range of knowledge and natural language content with a 3-second response time. To do this DeepQA evidences and evaluates many competing hypotheses. A key to success is automatically learning and combining accurate confidences across an array of complex algorithms and over different dimensions of evidence. Accurate confidences are needed to know when to "buzz in" against your competitors and how much to bet. High precision and accurate confidence computations are just as critical for providing real value in business settings where helping users focus on the right content sooner and with greater confidence can make all the difference. The need for speed and high precision demands a massively parallel computing platform capable of generating, evaluating and combing 1000's of hypotheses and their associated evidence. In this talk I will introduce the audience to the Jeopardy! Challenge. I will describe our technical approach, how it is founded on a massively parallel platform using UIMA-AS and will touch on the promise of DeepQA beyond Jeopardy!.

Dr. David Ferrucci is a Research Staff Member at IBM's T.J. Watson's Research Center and where he leads the Semantic Analysis and Integration department. His research focuses on technologies for discovering knowledge in natural language and for leveraging the results in a variety of intelligent search and knowledge management solutions. He has been the Principal Investigator (PI) on several government-funded research programs on automatic question answering, intelligent systems and saleable text analytics. His team consists of 25 researchers and software engineers specializing in the areas of Natural Language Processing (NLP), Software Architecture, Information Retrieval, Machine Learning and Knowledge Representation and Reasoning (KR&R). Dr. Ferrucci, as chief architect, led the UIMA project at IBM and chaired the UIMA standards committee at OASIS. UIMA is a software framework and open standard used by industry and academia for integrating, deploying and scaling advanced text and multi-modal analytics. The UIMA framework is deployed in IBM products and has been contributed to Apache open-source to facilitate broader adoption and development. UIMA helped lay the foundation for doing large-scale, collaborative unstructured information research.

In 2007, Dr. Ferrucci took on the Jeopardy! Challenge - tasked to create a computer system that can rival human champions at the game of Jeopardy!. As the PI for the exploratory research project dubbed DeepQA, he focused on advancing automatic, open-domain question answering using massively parallel evidence based hypothesis generation and evaluation. He explored the feasibility, won the support for and has set and driven the technical agenda for the Jeopardy! Challenge. He engaged top university researchers in the field to help explore better ways to openly and collaboratively accelerate research at a workshop on the open advancement of Question Answering. By building on UIMA, on key university collaborations and by taking bold research, engineering and management steps, he led his team to integrate and advance many search, NLP and semantic technologies to deliver results that have out-performed all expectations and have demonstrated world-class performance at a task previously thought insurmountable with the current state-of-the-art. Watson, the computer system built by Ferrucci's team is now competing with top Jeopardy! players. Next steps are to demonstrate how DeepQA can help make dramatic advances for intelligent decision support in areas including medicine, government and law. Dr. Ferrucci graduated from Manhattan College with a BS in Biology and from Rensselaer Polytechnic Institute in 1994 with a PhD in Computer Science specializing in knowledge representation and reasoning. He is published in the areas of AI, KR&R, NLP and automatic question answering.


Cheers,
-Ben

[TheConfessor]

Here's a link with essentially the same information:
http://istcolloq.gsfc.nasa.gov/Fall2010 ... rucci.html

I used to work at IBM and I've seen David Ferrucci's presentation. I thought it was very worthwhile, and it seemed to change the minds of some skeptics in the audience.

[porpoise spit]

Jennings vs. Rutter vs. Watson is on like Donkey Kong, February 14-16.

[grodney]

Fun!


When is it taping??

[econgator]

I realize you may not be able to answer this, Ken, but I'll ask:

It says that the match between you, Brad, and Watson will be two games spread over three days. Are you able to tell us how they split that up?

[Ken Jennings]

Nope! Curious myself.

[thom]

Maybe it'll be presented like the Olympics, with lots of tear-jerking human interest stories surrounding the competition?

[Ken Jennings]

The computer's adorable little sister has leukemia.

[billiej]

Wow, that really bytes.

(Yes, I'm ashamed)

[rmfromfla]

Was it caused by a virus?

(BTW, article about the upcoming fireworks with picture of Ken & Brad in today's USA Today...)

[rockgolf]

It's on EW.com too.

But look at the Categories it's listed under:

Categories: 100% Pure Cheese, Celebrity Feuds, Current Affairs, Games, Geekery, Guess Who!, Guilty Pleasure, Have You Seen This?, I'm Just a Geek, News You Can Use, Shenanigans!, Things That Are Awesome!

[enielsen]

I suppose there are worse ways to make (at least) $100,000 (minus the half promised to charity).

Ken, does it bother you at all that, since IBM promised to donate 100% of their winnings to charity while you and Brad will only donate 50% (source), the computer will naturally be framed as the hero and you will be the villains everyone roots against? Will we have several promos saying that if the computer loses, hundreds of adorable children will be denied the life-saving medication they need? Or will the computer be donating to a less reputable charity, such as the Robots' Fund to Enslave Mankind? That could be an interesting twist.

[TheConfessor]

I suppose there are worse ways to make (at least) $100,000 (minus the half promised to charity).

Shouldn't that be $100,000 (plus the half promised to charity)?

[enielsen]

I suppose there are worse ways to make (at least) $100,000 (minus the half promised to charity).

Shouldn't that be $100,000 (plus the half promised to charity)?

Yes. Yes it should.

[skullturfq]

Or will the computer be donating to a less reputable charity, such as the Robots' Fund to Enslave Mankind?

I might actually donate to such a charity, merely because of the cool name.

[rockgolf]

I for one welcome our new artificially intelligent overlords.

[Sequin]

It even gets a plug on the BBC!

http://www.bbc.co.uk/news/technology-11996531

[rjmason]

I'd like to know the tape date, if it's not a huge secret.

If it is a huge secret, then somebody shoot me a Bradley Manning, thanks in advance...

[ArtVark]

One of the team names at the pub quiz that Richard (previous poster) and I were at last night was:

"Robot Jeopardy match scheduled for February -- Watson will be there too."

[Muskrat]

Looking forward to the contestant chat moments. "So, Watson, have you ever seen 'Westworld'?"

[themanwho]

I'd like to know the tape date, if it's not a huge secret.

If it is a huge secret, then somebody shoot me a Bradley Manning, thanks in advance...

I think it's been taped already. Although I have no concrete evidence that is the case.

-M

[geniusonwheels]

I am genuinely nervous about this. I consider it a challenge for the whole human race, cause you ain't gonna lose....right?

[skullturfq]

"A computer once beat me at chess. But it was no match for me at kickboxing!" -- Emo Philips

[rockgolf]

I can hear the pre-game psych-out chatter now.

"Hey guys, ever calculate pi to 750,000 digits? No? I did."

[Ken Jennings]

"Hey guys, ever calculate pi to 750,000 digits? No? I did."

Wow, you've met Brad?!

(Just wanted to try this joke out before I hear a dozen variants of it in late-night monologues...but about ME...)