Watson on Jeopardy

[Don]

So it looks like the computer is going to beat the top 2 Jeopardy pretty handily unless a miraculous comeback happens. But really, what's the point to do this? Watson gets the question input as text the moment Trebek starts reading. If it takes him 3 seconds to read it, then that should be more than enough time for a computer to search through its database, so if the search method works at all there's no way the computer would ever get it wrong. You can assume it's reflex has to be much faster than any human so what's the point? If it has to understand the language in real time maybe it'd be more interesting, but ultimately I'd think you can still just tell the computer to buzz in immediately and take the 2 seconds you have to answer to finish looking up, or just throw enough computing power such that you can find your answer in a shorter time.

To me this is as pointless as the guys who built an AI that can beat the best human player in Starcraft. The only thing surprising about that is that it hasn't been done 10 years ago (the computer uses only Mutalisks, because there are nothing that can kill a Mutalisk that it cannot outrun, and Mutalisks will beat Corsairs/Valkyries if you avoid splash damage, which the computer can do easily). Yes I know IBM's donating the result to the charity, and it has applications elsewhere and so on but to me if they're sure their system works at all, the fact that it should beat the best human Jeopardy players shouldn't need to be verified.

[Flip]

Yeah im not surprised by the results at all. In fact, i feel bad for the guys.

[Don]

Yeah im not surprised by the results at all. In fact, i feel bad for the guys.

I think even the 3rd place gets $200K so I wouldn't feel too bad about them. I'll gladly take a beating from a computer for that kind of money.

[Flip]

This could be relatively historic, though. Computers against humans in games (like the chess one) have done pretty poorly until recently. So, to answer your question, whats the point? The point is to further our ability to create AI and demonstrate it to the world in a fun way. Just doing so without a goal would be more pointless, at least in this circumstance there is a mission, which motivates the team behind it all.

[Eric]

It's all fun and games until we make a Skynet, I swear to god.

[Shrinweck]

I caught a bit of it at work. I was too far away to catch most of it but the guy on the left did pretty damn well for most of the game and into the second round but once the computer got control, well, computers tend to stay in control.

[Zeus]

One unfair advantage Watson has: it can buzz in like no human being. That's why the guys were losing

[Kupek]

If Michael Jordan trounces me in a game of basketball, it doesn't change the fact that he is really good and it took an enormous amount of effort for him to get that good.

You guys are falling into this problem: http://lesswrong.com/lw/im/hindsight_devalues_science/

That essay focuses on social science, but I think it's still relevant here. Other experts in the area did not think they could do this, and even the people involved weren't sure. It's easy to dismiss this as "yeah, it's just a big search engine" once you already know it's been done. Besides not accurately characterizing the approach Watson takes, that sentiment misses the fact that this was an open question. It's easy for you guys to say "Yeah, it's easy" now that you know for a fact that it can be done. But that's not quite fair because you don't really know how it works. This is not just a giant search engine. I've only skimmed this article, but I think that it's written at the level that non-computer scientists can appreciate the difficulty of the problem they solved: http://www.stanford.edu/class/cs124/AIM ... DeepQA.pdf

Paradoxically, people would probably be more impressed if Watson did worse and the game was more competitive. Just like you'd be more impressed to watch Michael Jordan compete against someone who would actually make him sweat.

Of course, I work at TJ Watson Research Center now, so I'm probably biased.

[Don]

The difficulty of making the project is irrelevent. It's like saying a car is very difficult to design so we should be impressed that it's a lot faster than you. You know that they clearly tested this thing ahead of time, and the nature of computer means if it can do one category well it'll pretty much always do that well whether it's playing against you or me or Ken Jennings. If you assume the computer pretty much has an insurmountable lead in reflexes, that means it only needs to know 1 out of 3 questions to win (and this assumes the human competitors each know 1/3 of the rest of the questions). There is no doubt there are categories the computer can eat human for breakfast. Assuming the categories aren't picked to be particularly difficult for the computer to interpret this result is hardly surprising. Unlike even the Deep Blue case, you already know ahead of time that no human is going to buzz faster than the computer which means as long as its search algorithm works then it doesn't matter who it's playing, even if his opponent knows the answer to every question you're still in an unbeatable position as long as you can solve 1/3 of the questions (two opponents knowing everything split the other 2/3).

It'd be more interesting if only one person can answer each question, though I suspect Watson will still win pretty easily that way.

I'm guessing you're thinking of something like the book "Andy Roddick can beat me with a Frying Pan" (actually he cannot), but human versus human don't have some inherently insurmountable difference that cannot be addressed by a handicap. When it comes to human versus computer you can't artifically handicap the computer unless you're throwing the game, like say make Deep Blue only able to look 5 moves deep (it'd be pretty much incapable of figuring out anything good), or that Watson is not allowed to buzz ahead of you even though it can do that trivially. Heck, take the Starcraft AI, are you going to handicap the computer to what, 1000 APM? It can easily do tens times that if necessary and in fact a lot of its strat wouldn't work if it can't click that fast. You might as well tell the computer 'no rush for 15 minutes' if it has to be like that for a competitive game.

[Kupek]

We should be impressed at the speed that cars can travel. The only reason we are not is they are an old technology, and we take them for granted. But I'm not impressed by the car itself, I'm impressed by the human science and engineering that is responsible for it. I mention the difficulty of the task because I'm impressed by the research accomplishment.

Again, the notion that this result is "not surprising" presupposes that we knew how to design this software. Five years ago we did not. If you had put what they had two years into the project on the show, it would have failed spectacularly.

[Flip]

[Don]

I'm not convinced that just because it hasn't been done implies it's some great feat of science. Most likely it's either a lack of firepower or practical application. Two years ago they can probably build the same thing if they spend even more money at it but it wouldn't be worth all that effort. It took 10 years for an AI to be better than the best human player in Starcraft to emerge but it wasn't because it was hard.

[Zeus]

If Michael Jordan trounces me in a game of basketball, it doesn't change the fact that he is really good and it took an enormous amount of effort for him to get that good.

You guys are falling into this problem: http://lesswrong.com/lw/im/hindsight_devalues_science/

That essay focuses on social science, but I think it's still relevant here. Other experts in the area did not think they could do this, and even the people involved weren't sure. It's easy to dismiss this as "yeah, it's just a big search engine" once you already know it's been done. Besides not accurately characterizing the approach Watson takes, that sentiment misses the fact that this was an open question. It's easy for you guys to say "Yeah, it's easy" now that you know for a fact that it can be done. But that's not quite fair because you don't really know how it works. This is not just a giant search engine. I've only skimmed this article, but I think that it's written at the level that non-computer scientists can appreciate the difficulty of the problem they solved: http://www.stanford.edu/class/cs124/AIM ... DeepQA.pdf

Paradoxically, people would probably be more impressed if Watson did worse and the game was more competitive. Just like you'd be more impressed to watch Michael Jordan compete against someone who would actually make him sweat.

Of course, I work at TJ Watson Research Center now, so I'm probably biased.

Yeah, it's impressive, very impressive. But the fact that the computer can buzz in like no human being possible could means it has a large unfair advantage. Jennings won 74 games in a row because he was better than any of his opponents at it (then lost the easiest Final Jeopardy ever). Watson's doin' the same to him and Renner. If you watch it, those two knew the answers they just couldn't buzz in.

Certainly doesn't take away from how impressive it was to be able to build a computer to understand the nuances and intricacies of language and pop culture. That really is amazing. But it still had an insurmountable advantage over its human opponents making it appear "smarter" than it really is due to the nature of the task it was given.

[Don]

Pretty sure Ken Jennings has the fastest reflexes out of the Jeopardy champions which is part of the reason why he's got that record streak. So obviously reflexes plays a huge deal in Jeopardy, and it doesn't take a rocket scientist to figure out that the computer is going to have an advantage there.

Also some of the questions are definitely not very hard to answer. One of the example they gave was like: "The patron saint for bricklayers" and that can't be very hard to parse and look up the result in a database. Yes not all categories are as straightforward as that but there's still an obvious structure to how questions in Jeopardy are framed, and again you don't have to know all of them if you're sure that you can buzz in faster most of the time.

[Shrinweck]

During the credits they were doing silly things behind the robot. All I could think was of was a Stargate-ish AI sub-routine plotting their destruction.

[Kupek]

Two words: Wolfram Alpha. Very similar task as Jeopardy. Unimpressive results.

I'm not impressed by winning the game, since that is mostly due to the buzzer. I'm impressed by Watson's ability to get the questions right. And, Don, I think you severely underestimate the difficulty of what you think is easy - which is pretty much what everyone did when computing was in its infancy. We thought this sort of thing would be easy. It's not.

[Don]

Easy and expecting the outcome are two different things. In the case of Jeopardy, since the outcome greatly depends on how fast you can buzz in, you can be almost certain the computer will win once they feel confident enough to put it on national TV no matter how hard it is to get the search stuff right. You know IBM isn't going to stage some national stuff only to have Watson go to negative 50K because it got every question wrong even though it buzzed in first. It's even worse than Deep Blue because you could be confident about how good Deep Blue was at chess, but found out that Kasparov was still better (the games aren't exactly lopsided). On the other hand if you know Watson can answer about 50% of the question correctly then there is virtually no way it will lose to any human. Note that having 2 humans is a huge advantage for Watson because you can assume Watson will easily win the questions it is better than human, but the 2 humans will have to split the questions that Watson isn't good at.

You can compare this to the Starcraft AI research. It isn't as easy as it appears (though it is indeed pretty easy), and they made a lot of mistakes along the way but once all the detail is ironed out, nobody should be surprised that a computer with basically unlimited APM can clownstomp even the best human players. That doesn't invalidate the research but it makes me wonder why they have to make a big deal out of the event. It's like having the newest super fast car is going to race against Usain Bolt. It doesn't invalidate the work needed to make the car but the showdown is nothing more than a publicity stunt.

[Tessian]

He physically pushes a buzzer like everyone else... why is that unfair? It's still all about it coming to a confident answer as quickly as possible.

And don't go linking stupid Wolfgram Alpha in comparison, last I checked that website doesn't learn from its mistakes like Watson does and WA is only useful with science type categories. Having a computer that can interpret a question that's written in natural language and lingo and respond with the correct answer in a second or two is quite amazing.

[Zeus]

He physically pushes a buzzer like everyone else... why is that unfair?

Timing and consistency. No human can be that precise and that exacting with their button presses. You'll push that split second later or earlier, not quite hard/long enough, etc. Why do you think the master of the buzzing, Jennings, couldn't buzz in ahead of Watson during that entire match?

[Tessian]

Why do you think the master of the buzzing, Jennings, couldn't buzz in ahead of Watson during that entire match?

Because he knew the answer sooner than Jennings did. That was the whole point of this research -- can we build a computer to compete in jeopardy and beat the best of the best? He buzzes in first because he knew the answer first.

[Kupek]

And don't go linking stupid Wolfgram Alpha in comparison, last I checked that website doesn't learn from its mistakes like Watson does and WA is only useful with science type categories. Having a computer that can interpret a question that's written in natural language and lingo and respond with the correct answer in a second or two is quite amazing.

Right. If you followed the back-and-forth of the thread, then it would be apparent that I linked to Wolfram Alpha to show that people have tried to address a similar problem recently, and the result is not impressive.

[Zeus]

Why do you think the master of the buzzing, Jennings, couldn't buzz in ahead of Watson during that entire match?

Because he knew the answer sooner than Jennings did. That was the whole point of this research -- can we build a computer to compete in jeopardy and beat the best of the best? He buzzes in first because he knew the answer first.

No, the entire point of the research - and IBM made a point of saying this in the 15 minute advertisements it had during the each day's airing - is to build a computer that could understand the idiosyncrasies of the human language, including such things as puns or pop culture terms. That's what the entire point of building a computer like Watson was, to be able to think like a human being could, something that's been like a science fiction goal since computers were invented. Think of it like the walking on the moon goal of computer scientists. If you really think about it, humans basically think in forms of algorithms, just extremely complex ones. This game Watson was playing was just a demonstration of such algorithms programmed into a supercomputer at work. That's what's so impressive about Watson is he was able to, for the most part, understand the questions as well as basically any human could.

If you watch the matches, Renner and Jennings knew the answers to about 99% of the questions and were trying to buzz in, they just couldn't. That's what I've been saying, that due to the nature of the task that he was given he had an unfair advantage over his human counterparts which was impossible for them to overcome. It was the nature of Jeopardy which made Watson appear more impressive that than it really was. He's not far smarter than those two. If you took away the buzzing and just asked them to each answer the questions without any buzzing, he would probably lose. But the fact that he could even get close is extremely impressive and the scientific breakthrough it represents is pretty damned important.

[Don]

You cannot buzz in before Trebek is done reading the question, so knowing the answer does not allow you to buzz in quicker. If you look at the contestants on Jeopardy it's pretty clear 99% of the time they either know the answer before they're allowed to buzz, or that they never knew the answer at all. I think you used to be able to buzz in as soon as you know the answer, but in that case it rewarded knowledge too much since the game is supposed to have a physical competitive dimension too. Also after you buzzed in you still have a few seconds to answer, which means you could just buzz in immediately and take those 3 seconds to think about the answer and that'd be pretty broken since the person who buzz in the fastest will almost always win assuming he is not significantly dumber than his opponent.

Ken Jennings has the highest reflexes out of all the Jeopardy champions, but even he wasn't as fast as the computer, and that shouldn't be surprising.

[Julius Seeker]

I expect them to have the mark 1 ready ASAP.

[SineSwiper]

He physically pushes a buzzer like everyone else... why is that unfair? It's still all about it coming to a confident answer as quickly as possible.

And don't go linking stupid Wolfgram Alpha in comparison, last I checked that website doesn't learn from its mistakes like Watson does and WA is only useful with science type categories. Having a computer that can interpret a question that's written in natural language and lingo and respond with the correct answer in a second or two is quite amazing.

Yeah, I'm more interested in the English part, rather than looking up data. It amazes me how exactly these scientists showcase this technology:

Scientist: "I think I've finally perfect my software for interpreting and reciting English!"
Boss: "Great! Maybe we can get it to play Jeopardy."
Scientist: "But, this could be used to diagnose medical problems based on a patient's description of their symptoms, or completely reform how we search the Internet and information in general."
Boss: "Fuck that! Millions of people watch Jeopardy, and we need the press! Let's get some robot guy on Jeopardy! We'll call him Watson!"

[Kupek]

It was actually the other way around. That is, the VP of software research challenged researchers to build a system that could compete on Jeopardy. In other words, the thing you find impressive was driven by the challenge to do the thing you find less impressive.

[SineSwiper]

It was actually the other way around. That is, the VP of software research challenged researchers to build a system that could compete on Jeopardy. In other words, the thing you find impressive was driven by the challenge to do the thing you find less impressive.

That is equally dumb. Why would the VP of software research spend millions of dollars to build a system that could compete on Jeopardy?

[Kupek]

Because he bet it would drive innovation in other things - the things you're impressed with.

[Zeus]

It was actually the other way around. That is, the VP of software research challenged researchers to build a system that could compete on Jeopardy. In other words, the thing you find impressive was driven by the challenge to do the thing you find less impressive.

That is equally dumb. Why would the VP of software research spend millions of dollars to build a system that could compete on Jeopardy?

Sine, I think Seek hacked your account. There's no way you would post something so short-sighted. Look into it, boy's probably got some skills you may not know about and is messin' with you for the fun of it.

In case I'm mistaken and this is actually Sine, here's a reply:

Because it's a demonstration of what can be done not only what will be done. You first have to prove a technology works before you can apply it to different situations/scenarios/tasks/business models/etc. Jeopardy is actually an excellent demonstration of the technology. Everyone understands the task and nearly everyone agrees it's an excellent measuring stick for the technology, just like going to the moon or playing chess against Kasparov were (both of which were pretty useless tasks in and of themselves, BTW) . It was really nothing more than a scientific experiment done in a public venue. And you have to at least agree that experimentation is a pretty crucial step to developing science and technology, yes?