Now, I actually snuck something important in here, I
actually snuck two things that are important in here.
The first is, the, what's called the Markovian property.
Do you remember what the Markovian property is Michael?
>> From what?
>> From, I dunno. Actually, where does the Markovian property come from?
>> I'm going to say Russia.
>> Okay, yeah, from the, from the Russian.
>> Yeah, so I have Russian ancestors, and they passed onto me this idea that.
Markov means that you don't have to
condition on anything past the most recent state.
>> That's exactly right.
>> The Markovian property is that only the present matters.
>> And they had to pass that down to me you
know, one generation at a time, because you know, Markov property.
>> Exactly right, that was very good Michael. So what does this
mean? What this means is our
transition function, which shows you the probability
you end up in some state as prime, given that you're in state
S and took action A, only depends upon the current state S. If it
also depended upon where you were 20 minutes before then, you would have
to have more S's here. And then you would be violating the Markovian property.
>> So is this like, do historians hate this?
>> Well, you know, one never learns anything from history.
>> No, you're supposed to learn from history, or you're doomed to,
I don't know, let me make something up, repeat it.
>> [LAUGH] Fair enough. Historians probably don't like
this, but there is a way for mathematicians to
convince them that they're okay with it. And
the way that, you, mathematicians convince you that you're
okay with this is to point out that
you can turn almost anything, into a Markovian Process
by simply making certain that your current state
remembers everything you need to remember from the past.
>> I see.
>> So, in general, even if something isn't
really Markovian, you need to know what you were, not only what
you're doing now, but what you were doing five minutes ago. You could
just turn your current state into what you're doing now and what you
were doing five minutes ago. The obvious problem with that, of course, is
that if you have to remember everything from the beginning of time.
You're only going to see every state once and it's going to be very difficult
to learn anything. But, that Markovian property turns out to be, turns out
to be important and it actually allows us to solve these problems in
a tractable way. But I snuck in something else, Michael.
What I snuck in is this idea about the transition
model, is that nothing ever changes. So this second property
that matters for Markov decision processes, at least for the
sets of things that we're going to be talking about in
the beginning, is that things are stationary. That is these
for the purposes of this discussion, it means that these
rules don't change. Over time. That's one notion of stationary, okay?
>> Does that mean that the agent can never leave
the start state?
>> No, the agent can leave the start state any
time it takes the action, then it gets another start state.
>> Then how is it, how is it stationary, then?
>> It's not stationary, the world is stationary. The, the transition
model is stationary, the physics are stationary, the rules don't change any.
>> The rules don't change.
>> Right.
>> I, I, okay. I see.
>> Then there's another notion of stationary
that we'll see a little bit later.
Okay, last thing to point out about
the definition of a mark on decision process,
is the notion of reward. So, reward is simply a
scalr value, that you get for being in a state. So
for example we might say, you know, this green goal
is a really good goal. And so, if you get there,
we're going to give you a dollar. This red dual,
on the other hand, is a very, very bad state. We
don't want you to end there. And so, if you end
up there we're going to take away a dollar from you.
>> What if I don't have a dollar?
>> Someone will give you the dollar.
The universe will give you the dollar.
>> [LAUGH] And then take it away?
>> Or, the universe will take it away. Even
if you don't have it. You'll have negative dollars.
>> Oh, man.
>> Okay, so Reward is very important here in a of couple
ways. Reward is one of the things that, as I'm always harping on
encompasses our domain knowledge. So, the Rewards you get from the state tells
you the usefulness of entering into that state. Now. I wrote out three
different definitions of r here, because sometimes it's very useful to think
about them differently. I've been talking about the reward you get for entering
into the state, but there's also a notion of reward that you get
for entering into a state and taking an action. There's a rewar, or,
being in a state and taking an action, there's a reward that you
could get for being in a state, taking an action, and then ending
up in another state as prime. It turns out these are all mathematically
equivalent. But often it's easier to think about one form or the other.
But for the purposes of the, you know, for the rest of this discussion,
really you can just focus on that one, the reward of the value entering
into a state. And those four things,
by themselves, along with this Markov property
and non-stationarity, defines what's called the Markov
Decision Process. Or an MDP. Got it?
>> I'm a little stuck on the, how those could be mathematically equivalent.
>> Well we'll get to that later. Would you like a little bit of intuition?
>> Sure.
>> Well, you can imagine that just as
before we were dealing with the the notion
of making a non-Markovian thing Markovian by putting
a little bit of history into your state.
You can always fold in the action that you took to be in a state or
the action that you took to get to a state, as a part of your state.
>> But that would be a different Markov Decision Process.
>> It would, but they would work out to have the same solution.
>> Oh, I see.