what is going on everybody and welcome
to part three of the reinforcement
learning as well as cue learnings and
says we're starting with tutorial series
in the last video we created a well we
got the mountain car to get up the hill
basically using cue learning which is
cool but we did it with a bunch of
parameters here or a constants that I
just set because I knew it would work
but in the real world you're gonna have
to figure these values out and the way
that you're gonna do that can't possibly
be watching every episode and seeing
does it work because you know showing
let's say even with twenty five thousand
episodes showing every two thousand
that's still not a statistically
significant sampling of how your models
doing so so we have to find a better way
and one thing that we can do just a
simple metric to track is just simply
the reward so a per episode reward
tracking system is probably enough for
most like basic operations now for
really complicated environments you
might need to do a little more but
you're not going to use basic cue
learning for a complicated environment
anyway so with that let me just show you
guys one way you could track these
metrics over time now there's a million
ways you can do this is just a simple
programming task honestly but it is
something we definitely need to do so
I'm just gonna show one example and then
after that example I'm gonna show you
guys something kind of cool with the cue
tables at the end as a just a slight
bonus so uh anyway the first thing we're
gonna do here is just come underneath
cue table also
let's change episodes let's do let's
just do two thousand episodes show every
500 episodes that'll just help us to get
through this this year so the first
thing that we're gonna do is just
underneath cue table let's just create a
couple of new a couple of new values I
meant to just zoom in a little bit there
so first of all we're gonna have episode
rewards this will just be a list that
contains each episodes reward just as a
list and then we're gonna have an
aggregate
aggregate app rewards dictionary and
this dictionary is gonna track the
episode number basically then we're
gonna have the average and then we'll
have the min and then we'll have Pepe
troubles it will have max okay so this
is this will just be a dictionary that
tracks episode number so this will just
serve as like an X Y a the x-axis for a
graph basically the average this will be
the trailing average so you almost think
of it like I'm it's not a moving average
but it's the average for any given
window so every 500 episodes for example
this will average over time so as our
model improves average should go up
minimum this is just going to be
tracking for every show every what was
the worst model we had so so so
basically it's just what's the worst
that's not her concept okay max what was
the best one so why do we want these
well the average might actually be going
up but the minimum or the worst
performing model is still in the dump
and so you might have cases where you
actually prefer that the worst model is
still somewhat decent then to have a
highest average or something like that
so this is just barely getting into it
this is you know what you're gonna
actually be looking for or trying to
optimize for is gonna vary depending on
what kind of task you're attempting to
achieve so I'm just gonna keep it fairly
simple here and just go with that
so the next thing we're going to do is
we're gonna come down into the episode
loop here and we're gonna track at this
sewed underscore reward we're gonna say
that equals zero then we're going to
come down to the iteration of steps
which is here and then we get our reward
here and what we want to do is add that
reward so episode underscore reward plus
equals reward then what we want to do
what's your deal I guess
cuz we're just not using it yeah
undefined named episode wait woah epic
oh we spree type of episode reward
episode episode reward okay coming down
here episode reward plus equals reward
phenomenal okay so that will add to
episode reward every time and then
basically at the end of the episode what
we'd like to do is append episode reward
to app rewards so we'll come to the very
end of this loop we'll come back here
and the first thing that we're going to
say is app underscore ever Awards I
don't know if my keyboards dying or what
I'm pretty sure I didn't make that typo
a per Ward's dot append although EPS for
episode rewards is a little hard to
explain with a keyboard issue but anyway
app rewards dot append we want to append
that episode reward so the total reward
at the very end what do we want to do so
then the next thing we're gonna say is
if not episode episode modulo show every
and that for just for the that's the
same as saying if episode modulo show
every double equals zero so if it
doesn't equal zero so you can kind of
shorten this just by saying if not
episode modulo show every basically it
just means every show every do this
thing okay so I can't tell you how many
times in Python I need to perform a task
like this and it's kind of unfortunate
that this is the like industry standard
it would all would be kind of nice to
have some sort of way to be like like
every like wouldn't that be a nice
statement in Python every just saying
something like that anyway make it
happen guys
if not episode show every so now what we
want to do is we actually want to build
we're gonna work on our dictionary so
basically the first thing we're gonna do
is calculate the average reward and
that's going to be equal to the sum
of our episode ro @ underscore rewards
there we go
it'll be the sum of every Ward's - the
show every : and then / and I know some
people are gonna be like why don't you
just divide my show every well in this
case some like this
- show every : just means like the last
let's say 500 but if the list is only
300 long and we do the last 500 it's
still only gonna be 300 long either way
we shouldn't run into that as a problem
but just in case I'm going to instead
say the Len of app rewards
- show every cool so that gives us our
average reward and now we're ready to
actually populate that dictionary so it
was like aggregate rewards and then
basically we're gonna have four things
that we're gonna do first of all the
episode that's just gonna be equal to or
not equal to we're gonna say append Epis
so episode and then I'm gonna take this
I'm gonna copy pasta pasta pasta average
min and Max and so here we're going to
append the average then here I'm gonna
copy and paste this upper words - show
every colon thing paste paste and this
one will be what was that min so the
minimum of that value and then here
we're going to say max cool beautiful so
now we've built this dictionary the next
thing that would be useful possibly is
to print an F string so we can just just
for this specific episode we could print
like all of these things so we could say
episode colon and then average : min :
and then max : and then we'll just copy
and paste into here come up not cut copy
please thank you thank you min max so
copy this paste and then do the same
thing for max here copy paste beautiful
okay so that'll give us some some
metrics like as it's training for us so
we can kind of see how things are going
besides just seeing the simple replay
cuz like I said that's just probably not
gonna be enough now at the very end
we're gonna graph it so at the very
tippity-top we're gonna import
matplotlib pipe lot as PL tizzle and
then we're gonna come down to the bottom
again and again we close the environment
we shouldn't be a problem so we'll do it
here peel T dot if people TWP plot and
then we want to plot basically all three
of these combinations so basically it'll
be the x will always be the episode and
then we'll do AVG and then we're gonna
say label equals AVG and then I'm gonna
copy this case paste and then we'll do
men men will do max max and then we'll
do a peel T dot legend and we'll say the
location here is 4 and then finally peel
T dot show and for the location that
just you can kind of pick where you want
the legend to go and I'm gonna put it in
4 which just means the lower right so in
theory everything should be going up
over time so hopefully the lower right
isn't in the way if it's in the way we
didn't do very well anyways so
so anyway well save that and what I want
to do is open up a command line Python
q-learning 3 we'll get that at least
started I'm gonna move this like here so
we can hopefully see the metrics up
shoot is what I was going to say list
index out of range at rewards app
rewards do we not append ever awards dot
append episode reward we are appending
we have done something stupid episode
reward equals 0 episode reward plus
equals reward at the very end up rewards
dot append episode rewards then hold on
I'm just not seeing what the issue is
here list index OH - show every okay it
I see it uh-huh so yeah coming back down
here so the issue was
- show every : simple error let's just
make sure we didn't make that anywhere
else okay I think we're good so let's
try one more time cool okay so
everything's negative 200 here not too
too shocking so while we wait for this
to do what 2000 episodes I'm gonna give
a shout out to my most recent brand new
channel members Mike Smith
AJ Santa new bhowmick and Harve
demand thank you guys for your support
welcome to the club you guys are awesome
so it looks like uh we are almost done
at 1500 wow that's stinks he like almost
made it to that flag okay hopefully at
this point we'll get a beautiful graph
wow look at that look at that was that
average or max I can't even tell the I
think that's average or that's max
rather of course it's max it's the
woman's on top I was just testing you
guys to be honest okay so we can see
here that things are improving now we
only did 2000 step
so it's no surprise that you know the
max episode was doing pretty good but
the minimum is still gonna be negative
200 like I just never made it to the
flag now
we could continue I'm just not gonna
waste y'all's time in like you know
graph is super really long one and just
wait for things to iterate through I've
actually already done it so so this is
well this is a video I was going to show
you guys and I'll still show it to you
I'll just leave it in a different tab
and instead we're gonna pop over here
and I'm gonna let's just fullscreen this
thing and I'm gonna scroll down so like
this is kind of basically what we were
just looking at and then I went a little
longer actually I'm sure what did I
change here oh this was just with an
epsilon change so one of the things that
you would use these graphs for is like
changing like how does changing the
epsilon decay value change how does the
start and stop episode number change
things all that kind of stuff and then
also you can train a model for a little
bit without an epsilon then add the
epsilon how does that change things like
you could do all kinds of stuff like
this totally up to you what you want to
do so then even after 10,000 episodes
you can see the minimum like I said it
got in the way because it didn't do very
good but there is a little tick up right
here where the minimum was at minus 200
but a and then again this was back to
changing the epsilon decaying value you
can see how that changed some things
then here I changed the discrete
observation size from 20 to 40 and we
can see here that at least the
improvement is like super linear like
here you can see it goes up and then it
kind of flatlines plateaus whereas here
it definitely is constantly improving
like to a point in continuing past or at
least we it looks like it's gonna
continue past any other point we've ever
had and for the same number of episodes
so that I'm like okay let's train it for
really long so we did 25,000 episodes
and we can see maybe at least for this
setting the sweet spots around 20,000
because we can see the the worst agents
were still at least making it to the
flag because if they don't make it to
the flag its
gonna be a minus 200 so as long as they
eventually make it to the flag that's
pretty good so for you know after about
you know just before 20000 - I don't
know maybe 22,000 or something for 2,000
episodes we found or more than 2,000
episodes I mean probably 3,000 it never
failed it was successful every single
time and then for whatever reason it
starts to come back down again so anyway
so you can kind of see how things
changed and then from here you can start
tweaking like the epsilon value or you
could reintroduce Epsilon because
chances are by this time the epsilon is
not there anymore so maybe we want to
reintroduce epsilon because maybe the
the agent is finding itself in a
position it's never been in before so
that value has like never been tweaked
right so stuff like that okay and then
here is where we start talking about
saving for queue tables so basically is
just a numpy save and then you can just
save the cue table as whatever you want
so for example just well yeah you could
save per episode but that would be a lot
of queue tables so instead what I'm
doing here is that modulo thing so you
can you could decide you know hey I want
to save so for example I'll just take
this I cut this come over here you know
basically you could save a cue table
every show every so you could paste this
in you could say okay if you had it we
don't have the cue tables directory but
I made one in the text-based version
make a cue tables directory you save the
episode number and then you save the cue
table there and you could just do this
so you could do this every episode you
can do this every show every or you
could say if not episode :
I'm sorry modulo 10 okay and then you
could instead save that so in this case
it would save the cue table every 10
episodes so it's really up to you how
you want to do that and you don't
actually need to do that but you would
do that if like if you graph select for
example after I trained this model it
was clear by 25,000 well it kind of went
back to having some failures so what I
would have wanted to do immediately
after having some of these failures be
like no I want the model
at 20,000 look that's the one I want or
I want the one at like 20 I don't know
21,000 it looks like I want that one if
I hadn't like if I only saved the cue
table at the very end too bad I now I
got a retrain I guess all the way to 20
and who knows if it's gonna look like
that again right because there's a high
degree of randomness here so so that's
one reason you might actually want to
save the cue table
fairly frequently you probably don't
want every episode but maybe every 10 or
every hundred but the other reason you
can save the cue table is I'm curious
what happened here go for a hundred I'm
not sure what changed here so could you
like what's different about this one
then this one unless it's just like
doing it again this one looks really
good though anyways I'm not sure what
changed for this picture anyway so once
you have the cue tables this is what I
did to generate graphs of the cue tables
so in this case you can see I'm gonna
say I equals 20 4999 so basically
episode 25,000 because they start at 0
then I just graph the cue table
basically so what we have here is the
cue table don't forget I made the
observation space a 40 by 40 so for
action 0 this is your your all of your
cue values basically well not Q values
but basically what I did is if the Q
value is the max Q value for each of the
possible actions I marked that that
combination with a green dot otherwise
it's a red dot so you can see here over
time I also probably could have just
kept it a green dot but whatever anyway
my initial plan actually was to create a
3d chart and then and have like cubes
basically and I started to do that and
then it actually wasn't as cool as I
thought it was gonna be and I was like
well I could just graph like each of the
actions and this is kind of what I came
up with so this is at the very end and
you can actually see here action 0 that
action 0 was I think pushed left action
1 if I recall right is just do nothing
and then action 2 is pushed right so you
can see in these kind of this cluster
here it's like this is pretty
always a push left and like some of
these that aren't pushed left or
actually probably just those are still
just left over from the random
initialization and we just haven't hit
that combination yet it's probably
what's going on there and then here
again
I can't imagine many circumstances where
you wouldn't want to be pushing one way
or the other I mean there's probably
some like as you come up like maybe you
only want to go so high up the hills or
something like when you're swinging back
the other way so maybe that's it I
really don't know but anyway
so you can see how anyway it creates
these little clusters which is kind of
cool and then here this is I equals one
so basically after one episode this is
what it looks like so obviously it's
pretty random looking because it was
randomly initialized so you basically
you go from here to here which I kind of
wanted to see happen so I have all the
queue tables so then I use the following
to basically iterate over all of the
queue tables and save the chart so I
just made a cute table charts directory
and saved every single cue table form
basically from here to here in their
graph form and then the video code is
here it's the same code I used to make
the deep dream stuff if you haven't seen
those you probably just YouTube deep
dream cent decks and you'll find it
anyway it's the same code from that and
then we get a video of the cue table
over time which is pretty darn awesome
so this is it over time and it goes kind
of quick I didn't really know because
the whole videos apparently this was
only 13 seconds but I think I did 90
frames a second or something like that
but it's not really something I want to
stare at for the longest time but it's
kind of cool
how you can kind of see how in the
middle it like it's obviously starts
random and then it starts like in the
middle area of all the cue tables and
then slowly expands out which is kind of
cool
because I guess that would be like this
is like the cart was in like a slightly
off-center position to start and so I'm
thinking it really is like left is you
know left on these graphs and then right
is probably actually right on the graphs
I could be totally mistaken but that's
almost what it looks like and then as he
continues to learn it continues to kind
of fill out that entire space but one
note is it never actually touches any of
these like fully right positions because
the victory was that I want to say like
0.5 but the action space for that x-axis
went to 0.6 or 0.7 or something like
that and I think that's why these never
get touched because the the car never
goes to those positions so it's kind of
cool to see it because you can you can
actually see what's happening in the in
the example on the graph which is just
kind of cool and also just the fact that
it slowly grows out and I think it's
again I think it's slowly growing out
because it learns to push itself further
and further by building momentum and I
just think that's super cool to see them
in in graph form it's like staring at
the matrix and seeing it seeing the
picture so anyway it's kind of cool
yeah so feel free to make your own well
and change the visualization if you want
this is just one way to do it I still
kind of want to make the 3d table thing
but I don't think I will
so anyway I think yeah that's everything
for now if you want you can go to the
text-based version editorial and look at
the video or you can make your own video
change the colors do all kinds of cool
stuff that's it for now in the next
video what we're gonna do is build our
own Q learning environment so
immediately after I learned Q learning
the first thing I want to do is make my
own environment I I didn't really intend
to do a tutorial on that just because I
didn't really see why but everybody's
asking about it and it's like so obvious
like the first thing I wanted to do is
build my own key learning environment so
I guess I shouldn't be surprised that
lots of people want to see us make our
own so so that's what I'm gonna do in
the next tutorial and the benefit there
is that you can if it's your environment
you can make certain tweaks and changes
and like slowly add complexity or remove
or you know you have like full control
over the environment whereas like with
mountain car I can't really make that
environment too much more complicated
I could go with a different environment
maybe or something like that but yet you
don't have a lot of control if you can
make your own environment plus you can
start to understand how you could take
some like real world examples or maybe a
that you want to make on your own or
whatever so anyway that's what we're
gonna be doing the next video is
actually making our own environment and
then doing cue learning on that
environment my hope is I my expectation
is I can do it all in one video so it's
gonna be the goal but so we'll see
questions comments concerns whatever
feel free to leave them below if you
want to hang out ask questions and get
hopefully probably quicker answers
probably better answers because it's
usually not me that answers hell I
joined the discord channel its discord
GG slash sent X shout out to all the
mods and helpers in there you guys are
awesome and if you like the content
subscribe otherwise I will see you guys
in the next video