So here's how we relate the K-step estimators that we just
talked about with TD lambda.
So the idea is this, we're actually going to be using a weighted combination
of all of the estimators for any one of these TD lambda algorithms.
And the weights are going to look like this,
one minus lambda for the one step estimator.
Lambda times one minus lambda for the two step estimator.
Lambda squared times one minus lambda for the three step estimator.
Lambda to the K minus one times one minus lambda for the K step estimator
and then all the way down to the essentially lambda the infinity, but
of course that wouldn't, literally doesn't get any weight.
>> Wait, what's >> Except, well, let's leave it at that
for now and then we'll explain why that's an okay thing to say in a second.
>> Okay.
So, so far everything you said makes sense except you haven't told me what
you're weighting exactly.
>> Oh, so what we're going to do is when we move the value,
when we change the value of a state, we're going to
change it by all the different estimators.
All the different estimators are going to give us their own
preferences to what we should be moving the value toward.
And we're just going to take a convex combination,
a weighted combination of all of them.
Each time that we do an update for the value of a state.
>> Oh, I see.
>> Now we don't literally get to lift all these and update based on them.
But equivalently this is what we're doing.
So there's a couple things that we should probably check.
One is we should probably make sure that these weights sum up to one.
>> Mm-hm.
>> So, how do we do that?
Let's see. >> You write a summation?
Good idea.
So the summation is the all the different estimators from one
to infinity of the weight applied to that estimator,
which this formula works for all different values of K.
For K equals one it's lambda to the zero which is one times one minus lambda.
So we can now factor out the one minus lambda.
As long as lambda's less than one, we can use this as a geometric series.
And we get one over one minus lambda, and that's going to cancel with this one
minus lambda and so the weights sum up to one.
If lambda equals 1 though, what happens then?
These are all zeros and
this infinity one is one to the infinity which is just one.
So when lambda is one we put all of our wait on this last estimator.
And what happens if lambda is zero?
Sp if lambda is zero in this case,
we get a weight of one on the first estimator and zero on all the rest,
which I'll write this way, >> Mm-hm.
>> And when lambda is one we get zero for all these except for the last one.
Which let's say at least in the limit would be one.
So this is actually as we said this is giving us TD(0) and
this is giving us TD(1).
So if we just use lambda in place of one and zero,
which works, then we get td lambda.
But the important thing is for in between values of lambda, like a half,
or .3 or something like that.
We actually get weights on all of these to some degree.
And it's combining all those different estimators together.
>> And that seems okay.
Any good characterization of what that weight looks like.
So if lambda were equal to one-half, is that exactly halfway between TD(0) and
TD(1)?
>> Well, when lambda's a half, what do our weights look like?
We get a weight of a half on this one.
>> Mm-hm.
>> We get a weight of a half times a half, or a quarter, on this one.
>> Mm-hm.
>> We get a weight of a half times a half times a half, or
an eighth, on this one.
So we get this sequence of decreasing weight, exponentially decreasing weight.
Is that halfway between these two things?
I don't know.
I suppose you could define it to be.
But it is just what it is.
It's a combination of a whole bunch of estimators together.
>> Right. That eventually gets to zero.
>> Sure, but again, we're combining the outputs of all these estimators.
We're using whatever update they're suggesting and moving a little bit
in the direction of the first one and a little less in the direction of
the second one, a little bit less in the direction of the third one and so on.
So, yeah, in the limit we're not actually doing
the e infinity estimator at all, but that's okay because we're doing all
these estimators that kind of lead up to it.
So I don't know, does that help you understand TD(lambda) better?
This is the way I think about it, is to say well, it's a little bit hard to
interpret but what we're really doing is doing a weighted combination of a bunch
of step estimators instead of this sort of strange, continuous parameter.
>> Well, I guess so.
I mean, when I think about it the way you put it I guess what you're saying
is, look, you could do one step look ahead or you can do two step look ahead.
Or you can go all the way into the end of your episode which may last forever
and you can pick any of those.
Or you can say each one of those gives me some information and
I should just figure out some way to combine them.
And I can pick various ways for combining them and I know particularly
know how to turn the way I combine them into simple English sentence.
But I am finding ways to combine various ones of them and I should pick some
values of lambda that seems to work for the problem that I care about.
>> Yeah.
>> So here we go. >> All right, so
now given that we don't actually have a crisp characterization of what something
like lambda equals one or sorry, lambda equals a half, really gives us in words.
Why do we care about these intermediate values?
So I'm going to make a transition and
talk about next what happens empirically when you use TD lambda.
>> Okay.