All right. So that's supervised learning and
unsupervised learning.
That's pretty good.
The last one is reinforcement learning.
>> [SOUND].
>> Now reinforcement learning is what we both do, so
Michael does a little bit of reinforcement learning here and there.
You've got how many papers published in reinforcement learning?
>> All of them.
[LAUGH] Several.
I have several.
>> The man has like a hundred papers of reinforcement learnings and in fact he
wrote with his colleagues the great summary journal article bringing every
one up to date on what reinforcement learning was like back in 1990.
>> Yeah like 112 years ago.
>> 1992.
>> People are saying yeah we should probably somebody should write a new one
because the other ones getting a little long in the dude.
>> But there's been books written on machine learning system.
>> That's right.
>> It's a very popular field.
That's why we're both in it.
Michael tends to prove a lot of things,.
>> It is not, that is not why I'm in it.
>> What, I didn't, wait, what?
>> You said it's a very popular field and that's why we're in it.
>> No, no, no, no, no.
Did I say that?
>> That's what I heard.
>> I didn't mean to say that.
>> [SOUND] Let's run it back and see.
>> It's a very popular, yeah,
let's do that again because I did not mean to say that.
It is a very popular field.
Perhaps because you're in it Michael.
>> I don't think that's it.
When I was an undergraduate,
I thought the thing that I really want to understand.
I liked AI, I liked the whole idea of AI.
But what I really want to understand is how can you learn to be
better from experience?
And like I, I built a tic-tac-toe playing program, and like,
I want this tic-tac-toe playing program to get really good at tic-tac-toe.
because I was always interested in the most practical society
impacting problems.
>> I think that generalized pretty well to world hunger.
>> Eventually.
So so that is what got me interested in it, and
I was, I didn't even know what it was called for a long time.
So I started doing reinforcement learning, and
then discovered that it was interesting and popular.
>> Right.
Well, I certainly wouldn't suggest that we're doing the science that we're doing
because it's popular.
We're doing it because we're interested in it.
>> Yes.
>> And I'm interested in reinforcement learning because in some sense,
it kind of encapsulates all the things I happen to care about.
I come from a sort of general AI background, and I care modeling people.
I care about building smart agents that have to live in in world that
other smart agents, thousands of them, hundreds of thousand of them,
thousands of them.
Some of them might be human and
have to feel some way to predict what to do over time.
So, from a sort a technical point of view, if we can think of re,
in, supervised learning as function approximation and
unsupervised learning as, you know.
>> Concise- >> Concise,
impact description, what's the difference between something like
reinforcement learning and those two?
Supervised learning.
>> So often the way that supervised learning oh, sorry,
reinforcement learning is described is, is learning from delayed reward.
>> Mm-hm. So instead of the feedback that you get
in supervised learning which is here's what you should do.
And the feedback that you get in unsupervised learning which is
the feedback in reinforcement learning may come several steps
after the decisions that you've actually made.
>> So a good example of that, or the easy example of that would be,
actually your tic-tac-toe program, right?
So, you do something in tic-tac-toe, you put an X in the center and
then you put a, let's say, an O over here.
>> Oh.
>> And then I put an X right here.
>> Nice.
>> And then you ridiculously put an O in the center.
>> Which allows me to put an X over here and I win.
>> All right.
>> Now what's interesting about that is,
I didn't tell you what happened until the very end when I said X wins.
>> Right.
And now I know I made a mistake somewhere along the way but
I don't know exactly where.
I may have to kind of roll back the game in my mind and eventually figure out
where it is that I went off track, and what it is that I did wrong.
>> And in the full generality of reinforcement learning,
you may have never made a mistake.
It may simply be that's the way games go but
you would like to know which of the moves you made mattered.
Now, if it were a civilized learning problem, I would have put the X here,
he would have put the O there, and it would have been called that's Good.
I would have put the X here, and
when he put the O there, it would have been that's Bad, the O goes here.
>> Mm-hm.
Right. >> Or something like that.
It would have told you where he should have put the O.
But here, all he gets is eventually some kind of signal saying,
you did something well.
You did something poorly and even then it's only relative to the other
signals that you might have gotten.
>> Right, so then reinforcement learning is in some sense harder
because nobody's telling you what to do.
You have to work it out on your own.
>> Yeah it's like playing a game without knowing any of the rules.
Or at least knowing how you win or lose.
But being told every once in awhile that you've won or you've lost, okay, now-
>> Sometimes I feel like that.
>> I know man.