Let's think about what these things imply.
So the value of a state, well under the Bellman equation is this and
nothing scary here right this is all stuff that you know.
So now, the next thing we're going to do is we're going to say,
well the value of the state that we end up in since we're in average land.
We can write that as the sum over all the anchor points of how related
this state s prime is or what the wait is between s prime and that basis point.
And then whatever value we set for the basis point.
>> Sure >> All right so
here it is in the context of the whole Bellman equation.
So this is cool to write I just changed the order of summation and
kind of regrouped a little bit.
So now, we've got max over actions the reward of that action for
the state that we started.
And now, we're summing now instead of over all next states,
we're stepping over just the basis points.
Sure.
We're multiplying some quantity time's the value of the basis point.
What can we say about this value here.
If we write it let's say like this.
So we can write this as some function T of the state, the action and
the basis point.
And the sum over next states is kind of already built in here, right?
>> Yes, yes, yes.
>> [LAUGH] I gave this a name.
I called it T prime, because i wanted you to see a connection between this and
transition functions.
So, how is this like a transition function?
>> Well, choose a state, you take an action.
What basis functions will you go to next?
>> Right which you don't.
The actual MDP, you don't go to the basis function.
But the point is that
this acts like a transition function in that it's nonnegative, right?
And, when you sum up over all SBs,
over all the anchor points, it sums up to one.
>> RIght. >> Why does it sum up to one?
Because this is a convex combination, and this is a convex combination, and
so, when we combine them, we get a convex combination.
We could do the algebra for that, but i don't think we need to.
And so it acts just like a transition function.
So we can actually fold the function approximate to itself.
Into the transitions.
And all we get at the end of it is an MVP.
So why does it actually do the right thing because MVP's have
a well defined unique value function and
that you can find it using things like Q learning in value iteration.
>> That's beautiful.
>> Isn't it, though?
So, what we do is, we end up with something that is actually a reasonable
function approximator.
It's taking the value of some state, S prime, and
writing it as a combination of other states.
And then we just we turn it into, over series of steps,
into a new Markov decision process over a smaller set of states, specifically,
over just the basis states.
And at the end of the day We know that this is going to be well behaved,
we know that it's going to approximate things.
Well, we don't know that it's going to approximate things reasonably but
at least without blowing up in a convergent sort of way.
Because we know a lot of stuff about MDPs.
>> That's kind of insane.
>> It's very clever.
>> No no, it's very very clever.
So if I were going to make too fine of a point on it,
what would I end up with aA kind of interpretation here might think the MDP
are moving over is sort of basis state to basis state?
>> Yeah, I think that's a good question so
it's as if we're going from a state to some next state which is S prime.
And then, from there we're making an additional mini transition
to the basis states the SBs >> And the weights
on these put these probabilities are the combination weights,
are the convex combination weights.
Convex combination weights look a lot like probabilities.
>> Right, they do.
>> So it's as if each time we make a transition to a next state s prime
we make an additional little transition to each of the basis states, and
from there we continue.
And so, we can actually solve out What the value function is over everything
by just solving the MDP over the basis states.
And then, we can extrapolate that or
interpolate that to anything that we want.
>> I love it. And it makes some
sense because the basis states sort of represent a set of states.
So really what you're talking about is you're averaging over the set of states
that kind of get aliased through the basis functions.
I think that's beautiful.
>> Yeah, exactly.
So yeah, we fold our function approximater into the MDP itself.
>> Done.
>> Very cool. So, the good news is,
here's a value function approximator that is very well behaved.
The bad news is, it's hard to do some things with it,
because value functions aren't always very well represented.
As average or type things there's often kind of weird cliffs and
so forth in the value function that are not well captured by just
smoothing everything out.
But still this is a really nice idea.
>> Beautiful.