Alright, we're going to drill down and talk about a
specific reinforcement learning algorithm in just a moment. But
I wanted to remind everybody about, some of the
quantities that Charles introduced in the lesson last time
when he was talking about MVP's, and use them
to describe three different approaches for solving reinforcement learning
problems. So this first box here pi, maps states
to actions, and what did you call this Charles?
>> A policy.
>> That's right. And reinforcement learning
algorithms that work directly on trying to find the policy,
are called policy search algorithms. So the good thing about policy
search is, you're learning the quantity that you directly need
to use. Right? You're learning the policy. That's supposed to be
the output. So that seems like a really good thing.
Unfortunately, learning this, this function is very indirect. We don't get
direct access to, I was in this state, what actions should
I have chosen? Right? So this is, what did you call
this, Charles, the the temporal credit assignment problem, right?
>> Right.
>> So, the data doesn't tell you what action to actually choose.
>> Right. And this is why it's not exactly
like the way we've formulated supervised learning in the past.
>> Well, at least if we're trying to do policy certs,
that's right. But what we're going to do is now consider, well, maybe
that's not the quantity that we want to learn. Let's, let's think about
learning this function U, which you had said maps states to values.
So, what was this guy?
>> U is a utility.
>> Yeah.
>> Yeah the true utility of the state, sometimes I,
I think I called it the, the value of the state.
>> Yeah so, so, so sometimes it's referred to
as a value function, and learning methods, reinforcement learning
methods that target that as, as what they're trying
to learn directly, are called value function based approaches.
>> Mm-hm.
>> And, let's say that we, that we try to learn this, we're trying to learn
to map states to values, well the good news is, at
least if we're acting in the world, we're getting to see, okay
I was in some state, I took some action, duh, duh,
duh, and I can, I can observe the values that actually result
from that, and maybe use that to, to make updates. So
you can kind of imagine learning this, so that the, the learning's
not quite as indirect. How do we use this to decide how
to behave? So we, we need to turn it into a policy.
>> Mm hm.
>> And we didn't quite, we sort of talked
about how to do that, in terms of, of
looking at the Bellman Equations. We're going to, we're going to, it
turns out it's actually kind of hard to use
U directly to do that. But we're going to talk about
a different form of the value function that's going to
make that easy. So this is actually, it turns out
it'll be a relatively easy kind of argmax
operation, once we have the right kind of value function.
>> Okay.
>> So, there's some computation we have to do, and
there's some indirectness to the learning, but, you know, it's
okay. Alright, so the other quantities that you
told us about were T, which is, I
think of it as the transition function, I think you had a different name for it.
>> I just call it the transition model.
>> The, the model, right, yeah. So this is the model,
and and the, the R is the reinforcement function, the reward function.
>> Mm-hm.
>> And what do these things do, they take states and actions
and the transition function, or the
transition model, predicts next states, maybe probabilistically,
and the reward function tells you, next rewards, but just rewards. And
so this is a model, jointly, and I already mentioned this, but
we call methods that learn
this quantity, learn these, these functions and
then use them to do decisions, model based
reinforcement learners, so how do we go from T and R to something like U?
>> Well
if we had those, if we had T and we had R, and we could see the S's, and we,
and the actions we took, then we could do you
know, something like value iteration we did before the learned values.
>> Right, which value iteration was used to solve the Bellman Equations.
>> Right.
>> So that is somewhat heavy weight computation to do, but it's doable.
So, what would happens over here, is we actually have fairly direct learning.
Why is that? Because when we're trying to learn the
transition and reward function, we get state action pairs as input.
And then when we receive next state reward pairs as
output, so we can solve this as a supervised learning problem.
>> Hm.
>> So, so learning is rather direct, the usage of this
is a little bit computationally complex because you actually have to do
the planning and then the optimizing to actually develop what you're,
you know the policy doesn't come directly out of that. But but
this kind of gives you a sense of
three different ways, three different ways, places that
you can target the pieces of the MVP so that you can do reinforcement learning.
>> I like that.
>> Now, we're going to focus on this, this middle piece. Partly
because, you know, it's kind of the Goldilocks situation. It's you know, the
learning's not so indirect and the usage is not so indirect. There's
just been a tremendous amount of work focused on, on value function
based approaches. And it's, you know, remarkably simple it turns out,
if you do it right. And also very powerful. That there's
lots of ways to use these simple ideas to, to learn
hard problems. So, I think this is a good place to focus.
>> Okay. I buy that.