>> Alright, so let's say something that we actually know about
this approach to action selection which is called Epsilon Greedy Exploration.
And what we know is that if the action selection if
GLIE, which means Greedy in the limit with infinite exploration, and
that basically means that we are decaying our epsilon for epsilon
Greedy. That we start off more random and over time we
get less and less random and more and more Greedy. Then
we have two things that are true. One is that Q hat
goes to Q, which is, which comes from kind of a
standard Q learning convergence result. But, we also have something cooler in
this case which is that the policy that we're following, pi
hat, is getting more and more like the optimal policy over time.
>> Hm.
>> So, not only do we learn stuff, but we use it, too. And this
is an example of the exploration exploitation
dilemma. And exploration exploitation is really, really important,
not just because they're two words that surprisingly start with
the same five letters, like unlikely letters. But also, that
it is strike, it is talking specifically about this issue.
Exploitation is about using what you know. And exploration is
about getting the data that you need so that you
can learn. And this is one particular way of doing,
it turns out there's lots of other ways of making
this trade-off and the reason it's a trade-off is because
there's only one of you. There's only one agent acting in the world and
it has these two actually somewhat conflicting
objectives. One is to take actions that it
doesn't know so much about, so it can learn about them and the other one
take, takes actions that it knows are good. So that it can get high reward.
>> Hm, that makes sense. You know, I didn't, I
never realized that exploration and exploitation share the first five letters.
>> Hm.
>> I always knew that they shared the last
five letters.
>> Oh, that's interesting too.
>> Huh. So if you take an r and turn it into an
it, you might move from exploration to exploitation. I feel like an entire
political movement could be founded on that. [LAUGH]. But I'm not sure exactly
what it would be yet. Maybe I'll work on that, before our next lesson.
>> So I have an algorithm. There, there's a standard, lemma.
>> Mm hm.
>> In reinforcement learning theory called the exploration
exploitation dilemma. Sorry, [LAUGH], no, lemma. The other kind
of lemma. The exploration exploitation lemma, which has to
do with taking actions that are either exploring or
exploiting. But I have one where you actually do teaching.
>> Mm-hm.
>> You can actually, each time you take
an action it's either going to teach the
agent something, it's going to explore or exploit. So
I call that exploration exploitation, or explore, exploit, explain.
>> Oh, nice. But you could have called it the exploration exploitation
dilemma because you used dilemma. And di means
two sometimes. And you do the two things.
>> Well, in fact, dilemma does literally
mean two. Its a choice between two things.
>> Right, so its a dilemma.
>> Nice
>> Its a lemma about two things.
>> Alright, so there is, it turns out there's a lot
of other approaches to exploration and exploitation. And some of them in
the, in the model based setting. You can do a lot
more with it, a lot more powerful things with it because you
actually can keep track of what you've learned effectively in
the environment and what you haven't, so the algorithm can
actually know what it knows and can use that information
to explore things that it doesn't know and exploit things
that it does know. Q learning doesn't really have that
distinction. It's a much harder thing to do. So, so
that's what I wanted to tell you in terms of,
you know, thinking about exploration-exploitation, does that make sense to you?
>> It does make sense to me. I think what,
the main point I got out of this, or a main
point I got out of this, other than our incredible ability
to get caught up in letters and coincidences of spelling, is
that the exploration-exploitation dilemma really is
a dilemma. It's like the fundamental
tradeoff in reinforcement learning. You have to exploit because you have
to use what you've got, but you have to learn or
otherwise you might not be able to exploit profitably. So you
have to always trade off between these things, and if you don't,
you're bound to either learn nothing or to get caught in local minima.
>> I couldn't agree with you more. In some sense, if you
think of reinforcement learning as being
the question of model learning plus planning.
There's nothing new here because model learning is well studied in the machine
learning community and planning is well
studied in the planning and scheduling community.
>> Planning.
>> And so, like, what are we adding
to this? And what we're adding is the fact
that these two processes interact with each other and depend
on each other and that's exactly the exploration, exploitation dilemma
and that's information has to go back and forth between
these two processes that other people understand and we're the glue.
>> Or the glee.
>> [LAUGH] The glee glue.
>> I like it. That's beautiful.