Alright so we've got states. So next is whats called
the model or the transition model. Sometimes people refer to
it as the transition function, and basically it is a
function of three variables. It's a state, an action, which
I haven't defined for you yet and another state. In
fact since I haven't defined what an action is for
you yet, let's skip that and actually do define actions
for you. So the third part of our model are actions.
Actions are the things that you can do in a
particular state. So what would the actions be in this world?
>> Well the four different decisions I could make were the
up, down, left and right. Though it's maybe a little confusing because
I think those were also the four possible outcomes. That's true.
Well no, there are other outcomes. You could stay where you were.
>> Right.
>> So these are actually the actions. These are the things that
when I'm in a given state I'm allowed to execute. I can
either go up, go down, go left or go right. You'll notice
that as I described before there was no option not to move. Mm.
>> But there could have been, and there could
have been other actions, like teleport, or there's anything
that you can imagine. But the point is that
your action set represents all of the things that the
agent, the robot, the person, or whatever it is
you're trying to model, is allowed to do. Now, in
its full, generalized form, we can think of the
set of actions that one can take as being a
function of state Because there's some stage you can do some
things and some stage you can't do those things. But most of
the time people just treat it as a set of actions. And
the actions that aren't allowable in them particular states have no effect.
Alright, so we understand states. They're the things that describe the world.
We understand actions. Those are the things that you can do, the
commands you can execute when your in particular states. And what a
model describes, what the transition model describes is in some sense the
rules of the game that you're playing. It's
the physics of the world. So it's a function
of three variables: a state, an action and
another state, which, by the way, could actually be
the same state. And what this function produces
is the probability That you will end up transitioning
the state s prime, given that you were in
state s, and you took action a. Got it?
>> I think so. so the s prime is where you end up, and the s, a is what
you're given. So it's, so you plug these three
in. Oh I see, and you get a probability Mm-hm.
>> But the probabilities have to add up to one if
you if you sum it up over all the s primes.
>> Right. That's exactly right. So for example if you think about the
deterministic case where there was no noise then this is a very simple model. If
I'm in the state the start state. And I take the action up,
then what's the probability, I end up in the state immediately above it?
>> Was that a 0.08?
>> No, in the, in, when we first started in the deterministic world.
>> Oh, that was probability one.
>> Right, and what would be the probability, you
ended up in the state to the right? Probability zero.
>> Right. In fact, the probability that you end up at any of the
other states is zero in the deterministic world. Now what about the case when
we were in the non-deterministic world where an action
would actually execute faithfully only 80% of the time. If
I'm in the start state and I go up, what's
the probability that I end up. In the state above.
>> That was 0.8
>> Was the probability, I end up in the state to the right.
>> That was 0.1.
>> And, what was the probability I end up where I started.
>> That was also 0.1.
>> Right, and 0 everywhere else. And, that's just sort of the way
it works. So, the model really is an important thing. And the reason,
it's important. Is it really does describe the rules of the game.
It tells you what you, what will happen if you do something in
a particular place. It captures Everything that you know about the transition
u, of the world is what you know about the rules, got it?
>> You called it physics before?
>> I called it the physics of the world.
>> Huh.
>> These are the rules that don't change.
>> But they're very different from real world physics.
>> Well, yeah, although they
don't have to be. I mean in some sense, you could argue
that a mark off decision process, what we described so far, these three
things In fact, do describe the universe, the states are, you know,
in the positions of all the atoms. The positions and velocities of all
the atoms in the universe. The transition miles as you do, take
certain positions in the world whatever they are how the state of the
universe changes in response to that. And the actions or whatever those set
of actions could be. And it can be probabilistic or it's not probabilistic.
>> It's definitely probabilistic. The transition
models are by their very definition probabilistic.
>> Gotcha.