Okay Michael so to demonstrate this idea or
to demonstrate some of these ideas I'm going to look at grid worlds.
One a very simple grid world to try to get across a pretty
straightforward idea, at least what I hope's a pretty straightforward idea.
And then a slightly more complicated and prettier grid world in about a moment.
But first let's look at kind of a basic grid world.
So this is like our famous four by three grid world except I
added a couple of details.
I haven't drawn all the little squares and all the little tiny grids.
Because really, I just sort of want to pop up a level here.
But you start somewhere in this world, it almost doesn't matter where,
and you have to try to get here.
To the little plus one that's where all the reward in the world is.
And you have the normal actions.
You can move up, down, left, right.
And you know it's slippery.
So every once in a while when you try to move up you end up moving in
a perpendicular direction.
And the usual sort of thing that we normally do.
Now other than that, what's different about this world is that I've got walls,
like we had before.
Except these walls actually happened to coincidentally create little rooms.
So let's imagine that you're starting over here somewhere and
your goal is to eventually learn in this world how to get to, well let's go.
So I think we know how to do this.
I think we've done this before many many times we certainly did it in the four by
three grid world but I'm going to ask you a slightly different question.
I want you to think about this in a slightly different way.
If you were trying to tell a person how
you should get from this point to this point what would you tell them?
>> I'm going to say Google Maps.
>> This is not Google Maps.
This is in a GPS dead zone.
>> What? >> I know, I know it's amazing.
>> Wow I guess that's one of the real benefits of reinforcement learning
is it can work even even a GPS dead zone.
So the question is how is this?
How would you tell them to go?
Well, so, you know, we talk to agents via policies.
So I could try to give a policy.
But you didn't really get.
So that's a mapping from states to actions.
But you don't really give me a lot of details about what the states where.
So probably if I was talking to a person I would say something kind of high level
like well pass through the doorway to the east.
Then you enter a room and there's a doorway to the south.
Go through that and
then a little bit inside the doorway you'll find a great reward.
Or at least a reward.
>> Plus one is the greatest reward that anyone can receive Michael.
You know that.
>> [LAUGH] >> So right and
I like this because what you did is I gave you the actions up down left right.
And you could have said okay so move right, right, right,
right, right down right, right and then go I don't know right, right down, down,
down, down, down, down, down, down, down, down write or something but
instead you said well go to the doorway and then go to another doorway and
then once you're past the doorway, you know you'll just take a couple steps and
you'll end up where you want to be where the great reward is.
And that's perfectly reasonable in fact it's more than perfectly reasonable it's
a kind of abstraction in the same way that function approximation is
a sort of attraction of the states which you just described
is what we call temporal obstruction.
>> I see. >> And
you know that has a specific meaning but
I think to really kind of grasp what is the right way to think about it is you
came up with new actions >> That's making the action
space bigger.
>> Making the action space bigger which is a problem you would think but
each of these actions actually covers an enormous amount of ground.
So in fact, if you had said right, right, right, right, right, right,
right, right, right, down, right, right, right, right, right, right, right,
right, right, down, down, down, down, down, down, down, down, down, down,
down, down, down, down, down, down, down, right or something like that,
that's a lot of actions to have to reason over over a long period of time.
But what you actually said was, take one action, take another action, and then,
I don't know, take one or two actions.
So we managed to do with your new actions in, I don't know,
about five steps what would have otherwise taken, I don't know,
100 steps or maybe >> That's cool idea.
>> Maybe 60 steps that's right that is where the abstraction comes from and
where the temple nature comes from.
Because again, one of the things that makes reinforcement learning hard
is this delayed reward, and what's implicit in that is that you have
to reason over time that you have to take many, many steps and
because all the options there are several options at every state you're
actually getting an exponential blowup over all the paths you might take.
And here what you've done is you've traded off the price in
adding new actions with being able to skip over,
having to make a bunch of decisions along the way.
And so now I can do in five steps would have taken 100 steps and
that's going to be a gigantic exponential savings.