what is going on everybody and welcome
to part 4 of the reinforcement learning
series in this video we're gonna be
doing is building our own cue learning
environment so the first thing I wanted
to do when I learned to learning yes it
was useful to use the open AI gym
environment but the first thing I wanted
to do is make my own environment and I
didn't really intend to make a tutorial
out of it but people were asking and
then it's kind of like well obviously
that's what I wanted to do so of course
that's what other people probably want
to do too so anyway here we are let's
make our own environment so I'm actually
just going to kind of run through the
environment that I made and explain it
as I go so well probably we're just
going to do the program linearly but if
you have questions whatever as always
comment below join the discord GG slash
syntax and feel free to ask any
questions but it should be pretty
self-explanatory it's not a super
complicated environment and just in case
I guess I'll explain it a little bit
before we get in but so first of all I
love blobs every I use blobs for like
examples of everything so if you haven't
noticed by now I'm a blob a file anyway
so I always want to make blob things and
so the idea was to make blobs and in
this case you've got a player blob of
food blob that's the objective we're
trying to achieve and then just for you
know complexity I added an enemy blob as
well so the idea is that you have this
player blob and and we'll start by
having the enemy and the food not move
it just stays stationary and they just
kind of initialize in random positions
and then the the player blob has to move
to get to that position and then later
we can add in movement it's just that
letting the things move is kind of
irrelevant to the model learning how to
actually move except in one example
which is so if the enemy is moving and
you're moving you could in theory move
into each other if the enemy is not
moving in training you you you might
learn that it's just totally fine to get
close to the enemy as long as you don't
hit the enemy but if the enemy can also
move they could collide but anyway not
really worried about that but you guys
can feel free to tinker with that anyway
let's get into it so
we're gonna be using OpenCV so the first
thing that I want to do is open up a
command prompt and let's go ahead and
pip install Python
- open CV let's make sure we have that
is it open it might be open CV - Python
open CV - Python okay I feel like I
always get that wrong anyway okay
pip install OpenCV - python if for
whatever reason python - open CV did
install for you you'll probably just
installed something really bad so don't
use it get rid of it moving along so
alright so we are also going to use
numpy but you should already have that
and then we're gonna use the Python
imaging library and I want to say that
would be a pip install pillow this is a
virtual machine and it should be clean
so if that's not what I want
we'll find out soon enough but we'll get
red pillow and then to use pillow you
import capital PIL so that should be
everything we need let's go ahead and
import import numpy as NP sorties numpy
for the array types of stuff from PIL
we're going to import import how to type
image with capital I import CV - we're
gonna import matplotlib
pipe lot as PLT we're going to import
pickle and that's to save and load our
cue table then we're going to from
matplotlib will import style just to
make our graph pretty and we'll do style
dot use ggplot and then finally we will
import time we're gonna use time purely
to set dynamic queue table filenames and
so it has like some order what's your
deal probably - under 500 is its type of
style huh okay great
so the first thing I'm gonna do is just
try to run this real quick make sure it
actually runs it
does okay so all our imports worked and
we're good to go
so so now we're gonna have some oops did
I just delete totally did okay so now
we're gonna start with some of our
constants so we're going to say size and
we'll say this is 10 so we're going to
skip the whole make a huge environment
and then boil it down to action or
observation spaces discreet observation
spaces I'm gonna skip that step I'm just
gonna make it a grid so in this case
it's going to be a 10 by 10 grid so the
player the food and the enemy will be
initialized at a random location on a 10
by 10 but we can change this as time
goes on I'll talk about how some of
those changes will impact things but it
obviously as you increase this size
especially depending on the size of your
action space as well that is going to
just exponentially explode the number of
you know possible combinations in your
cue table so anyway we'll start with a
10 by 10 that should be pretty simple
how many episodes so HM episodes will
say 25,000 we're going to say it we're
gonna give a move move underscore
penalty that's gonna be a 1 an enemy
underscore penalty so this is if we if
we hit the enemy we're gonna say is a
300 so we'll subtract that penalty
basically we're going to say a food
reward 25 this was I haven't really
decided where I want this I don't know
if I really want it to be 1 0 like we
had with the mountain car 25 I haven't
really decided I don't really know
what's the best way to what's the best
one to use so I'm just gonna throw in 25
I haven't noticed anything you know huge
to tell me one way or the other then
we're gonna have epsilon lowercase
because it's gonna change over time
we'll start at 0.9 another thing that we
could change a ton EPS decay so it's
epsilon decay zero point I went with
triple nine eight and the way I came
into this was I just made a for loop and
I literally just decayed so I just took
epsilon times or I'm sorry for you know
for I in range say 25,000 multiply the
I just kind of looked and I was like
okay that looks good so so I literally
just pulled that number out of nowhere
moving along now we'll say show
underscore every so this is just like
before how often do we want to show I'm
gonna say every 3,000 episodes if I
recall right this actually my
environment is actually a little quicker
than the mountain car one okay show
every 3,000 okay so now what we're gonna
say is start will have a variable here
start cue table table and for now we'll
say none but then it could or a file
name so if you happen to have an
existing cue table that you want to like
load in and train from that point you
would throw that right in there now you
might want to do this for a variety of
reasons you want to continue training or
or you know at least we haven't really
coded in a good way to like decay
epsilon to zero in them you know train
for a little bit and then reintroduce
epsilon we don't really have a good way
to do that so if you really wanted to do
that you could decay epsilon to zero or
train for a little bit as you know with
very low or no epsilon and then load it
in a DEP salon and continue and I've
actually found that that works really
well to like decay epsilon let it go for
a little bit and then set up salon back
to 0.9 or something indicate again and
then let it train for a while and then
decay again and keep repeating that that
that seems to actually learn pretty well
so anyway for now it's none because we
don't have a cue table but if we had one
we just passed the filename so now we'll
throw in learning underscore rates we're
gonna say zero point one discount
we'll set again zero point nine five
these two variables I really haven't
played with much so couldn't couldn't
really tell you how it's going to impact
things but now that you know how to do
analysis from the previous video feel
free to thinker with them now what I
want to do is I'm gonna give a player N
equals one we're gonna say food n these
are just numbers for labels and a keys
rather in a dictionary that's it so
clear and food and end
enemy n3 again it's just definitions for
what number of these things represent in
a dictionary so we're gonna say D equals
and then we're gonna have one colon
something well I'll just fill it in so
these will be the colors so 255 175 0
now I'm actually defining these colors
in BG or format even though I'm really
try was really trying to use RGB so
side-project is if anybody can tell me
why is it BGR because i don't think it's
like the I don't know maybe I'll figure
it out as we go but that's that was a
problem I haven't solved yet
anyway it's 255 0 so let the player be
is mostly blue some green so it's kind
of like a lightish blue food is B G so
full green so the food is green and then
the enemy will be 0 0 255 so BG R so
maximum R so it'll be very red cool so
you don't like that huh all right I'm
surprised it doesn't surprise it's
accepting my spacing over of the
dictionary I don't think that's Pepe
maybe it is must it must be if it's
accepting it ok so now what we need is
we need a blob class because really all
of these blobs are gonna have a lot of
the same attributes at least they're
gonna need to be able to move they're
gonna need a starting location like they
need to be initialized randomly they're
gonna need to be able to be moved and
then later basically our observation I
didn't really want to use because it you
would have like a huge observation space
if you needed to pass what is the
location of like I felt like it would
you have to either the problem would be
much more complex I guess if you passed
these the physical location of
everything so my plan is instead the
observation space is actually going to
be the relative position of the food and
then the relative position of the enemy
to the player that's going to be the
observation so to get that we need to be
able to
form some sort of a operation that will
produce that value so anyway we're gonna
make a blob class so that we can handle
for that I get asked all the time hey
why don't you code an object or in your
programming all the time
and it's like well I just use it where
it's at actually useful and in this case
it seems to me that making a blob class
is actually useful
so anyway class blob so define our init
method and if you don't know
object-oriented programming in Python I
would go to Python programming dotnet go
to Python fundamentals intermediate
Python and then in this series we get
into object oriented programming here
and I would strongly recommend you you
go through this if it's any if it's
confusing to you and then we also are
going to do some operator overriding and
stuff like that so you might want to
check that out if that's confusing to
you so or init method this is just gonna
run immediately so we're gonna say self
dot X is going to be equal to numpy dot
random dot R and int between zero and
the size all caps and then we're gonna
do the same thing for self dot Y so
that'll just initialize randomly it's
gonna cause a slight issue
so if well in one case the player could
spawn right on the food that would be
lame
the the other one is the player could
spawn right on the enemy that would also
be lame and finally the enemy could
spawn on the food or the food on the
enemy server the way you want to look at
it glass half full half empty kind of
thing those with that that kind of
stinks that that's a possibility I'm not
gonna handle for that because I don't
really feel like writing the code for
that define we're going to make a string
method I'm not even sure I use this for
debugging purposes but then I don't not
even sure we need it now but anyway I'm
gonna return an F string of mmm self dot
X and then we'll just say comma
self-doubt why so if we want to print
the blob it'll just print the blobs
location so now what we want to do is
I'm gonna override
define the subtraction operator here or
over I think it's operator overloading
anyway someone will correct me actually
anyway return so this just allows us to
subtract a blob from another blob so in
fact we need another blob so I'm gonna
say other so we'll Pat so when we
subtract when you do it - and then the
thing that thing will get passed here
and then in here we're going to say
return and in this case what we're gonna
say is return and then in parentheses
we're gonna say self dot X - other dot X
comma self dot Y - other dot Y cool it
looks like Pepe doesn't care either way
if you want to space here or not I'm
gonna give it a space okay so that's our
subtraction operator overloading
I think it's operator overloading I'm
gonna go with that now I'm going to
define basically this blob needs to the
player is gonna take a discrete action
right so we're gonna have an action
method but really the player is the only
one that's gonna use it but there might
be a time where you want to have many
players so I'm just gonna I'm gonna go
ahead and do this like in this like
q-learning
like we're not going to have many
players I don't think I can't really see
a good reason why I would do that but
later we might want to do that and then
so action will actually interact with
another method I'm just gonna pass for
now and we're going to call this move
method and you know it'll be self and
then probably x and y and in fact I'm
gonna set X&Y just default to false
before I forget
okay so in action we want to be able to
pass a discrete value so we'll say self
choice and then we're just going to
if choice equals zero now we're gonna
start taking some actions so we're gonna
say self dot move and we're gonna say X
equal x equals 1 y equals 1 so it's
gonna move 1 in 1 now what I'm going to
do is I'm just gonna copy this paste
this and this will be L Elif choice
equals 1 and we're gonna say negative 1
and negative 1 and then again we're just
gonna copy paste paste set this to to
set this to 3 and then negative 1 and 1
and then 1 and negative 1 and those are
gonna be all the choices possible I
wanted to do this just to keep the
action space relatively small but this
means the player can only move
diagonally now if I had thought of this
when I was coding this I wouldn't have
continued I would have actually done
each of sorry I just punched the mic I
wish I wouldn't have done that I would
have also added all of the X only in
Wyoming movements as well so it could
move up and down because to me I would
not have expected q-learning to be able
to handle for this but like I said I
actually made this environment before I
actually thought I was gonna show you
guys this environment so I already know
it's gonna work
and it works by using the wall so if it
gets to the wall and then tries to move
diagonally it it moves up or down right
or side to side depending on which wall
it's at so so it still actually works
and I'm still amazed that it learns to
use the wall because as you'll see it
has no idea about the wall it doesn't
know there's any boundary to its
environment so the fact that it learns
to use the wall is fascinating and I
don't know if it's because because it's
not smart enough it's not like deep
learning where if it it knows it's on
evens or odds and then the food is on
and off ok let's say the X is even for
the agent and then you've got like you
know like it's a exact combination that
like a deep learning model could figure
out and be like oh we need to flip this
let's go to the wall but for q-learning
it really I don't think I should be able
to learn that
because this is not how it works but it
actually does work in this model can be
trained for like almost a hundred
percent because like in theory half of
the time when it gets initialized
randomly the food should be in a
location inaccessible without using the
wall but this model actually learns to
succeed almost a hundred percent of the
time so that's mind boggling but anyway
if you want to add more choices go for
it
I'm gonna leave it this way but just
take note that it can only move
diagonally and it's amazing so now let's
code the move method and basically we
want to be able to move randomly if
that's if there's no value passed here
or move very specifically so we're gonna
say if not X then what we want to do is
we're gonna say self dot X plus equals
NP dot random dot R and int between
negative 1 & 2 so it's from negative 1
up to 2 so that just means that we'll
either do a negative 1 0 or a 1 so keep
in mind it has the ability to do zero so
this random movement can move up or down
fascinating if not X else self dot X
plus equals x then we want to do the
exact same thing for the Y so I'm
actually just gonna copy this come down
here paste so why why why and why why
though okay so now so that's it so
that's handling for X&Y the problem here
now becomes when it hits the wall so
recall our size is a 10 we're trying to
make a 10 by 10 grid so if the agent
attempts to move beyond that we need to
need a thort it's same thing with the
food if we allow the food to move
spoiler alert
I'll will turn on movement you'll get to
see it or at least I'll show you video
of it it'll take probably too long to do
it but you can train it on your own
so okay so if self dot X is less than
zero because it could be at the zero
position it just can't be less than zero
we're gonna say self dot x equals zero
no colon required there L if self dot X
self dot X is greater than the size
minus one rather so again the size is 10
but the positions are 0 through 9 so
there's 10 positions but so that's why
so if it's greater than 9 we're actually
at the 11th position so we say size
minus 1 there then what we're gonna say
is self dot x equals size minus 1 we
need a fix sighs there okay now we're
going to do the exact same thing for y
so say again y and then we'll do why why
and why fabulous so now that should be
everything we need for our blob class
which I will now refer to as blob jects
which you've altered if you've seen any
of my tutorials with blobs so now what
we want to do is either create the cue
table or load the cue table so what
we're gonna say here is if start cue
table is none so if we don't have the
start cue table we need to create one so
we're gonna say Q underscore table
equals a we're gonna just make it a
dictionary we could we could do like
something similar to before but we're
gonna do it different so all we're going
to say here is for I in range of
negative size plus 1 to size this needs
to be all caps size so so
again negative size plus 1 because we
again we need to shift and then size and
we're not doing minus one here because
all the ranges are always up to so we're
not actually including the 10 so we were
going to do that now what we're going to
do is do this three more times so we've
got one two okay because we need every
cut so think of it this way we have two
coordinates right so we have the
basically the observation space is going
to be two coordinates where the first
one is going to be the Delta to the food
so the relative difference because we're
going to subtract the blobs right we're
gonna subtract these two blobs from each
other so where we overrode here so it's
going to be the subtraction of the plate
current player about the player to the
food so that's going to return one tuple
of x and y-values and then the other one
is going to be the difference or the
subtraction or the Delta of the player
to the enemy so it'll be another tuple
so that'll be XY so it'll be like x1 y1
and then x2 y2 okay it'll look like that
that's our observation space now to get
every combination we're going to iterate
through them so we've got I I okay
we could also get fancy we could say x1
y1 x2 y2 okay that might make a little
more sense since I just showed that
other example so now we have every
combination we want to add every
combination to our table here so so the
way that we're gonna do that is I'm
gonna look at my code because I want
this because this is something I would I
didn't say it I would fudge up cue table
and then
the for the table we're actually gonna
pass the tuple so it'll be a tuple and I
changed my values so I already kind of
screwed myself here like this right so
that's gonna be the key is a tuple of
tuples so we're gonna say x1 y1 x2 y2
and let's fix this
right okay cool
so that will save that out you hmm
there you go Pepe okay so then now we
just need to initialize with random
values so what kind of we'll just
initialize the same way pretty much as
before so that's going to be equal to
and then so basically each each
observation space needs four four random
values because we have our action space
is four we have four discrete actions
surface a MV dot random dot UNIF uni
form and I'm gonna say negative 5 to 0
and then we're gonna say for I in range
for cool I think that's everything
looks good make pep eight happy except
for the line limit petition to remove
the line limit from pep eight there's
just like so many times when the line
limit is stupid and it like it would be
dumb to to not exceed the line limit
okay
so cute eh okay so we've got our key
table values that's it our cue table is
initialized now we're gonna say else
with open we basically well we're gonna
do start cue tables so it's that file
name RB as F we're gonna say now cue
table equals pickle dot load F so if we
have one pre-trained awesome will load
that or rather if we don't have one
pretty trained
fine we'll make one otherwise awesome
blowed that so then we just have our Q
code or our training or whatever you
want to call it to do so we're gonna say
for episode in range of how many
episodes do we want we're going to say
the player equals a blob the food equals
blob and enemy equals a blob easy then
we're gonna say if episode modulo show
every is equal to zero we're gonna say
we're gonna print an F string on number
episode just so we know kind of where we
where we are and then we're going to say
the epsilon also so we kind of know
where we are
epsilon wise then we are going to let's
just print one more we're going to say
whoops show every episode clips
mean and we're just gonna display the
mean episode reward so we're gonna say
NP mean and that's episode underscore
rewards negative show every : I just
don't recall making the episode rewards
I don't think we did oh now it I saw it
show up initially anyway app is oops
episode underscore re words that's just
gonna be an empty list
okay and then the other thing we'll do
is show equals true else we'll set show
equals false and this is just so we can
actually display the thing so now we're
going to get to our actual
so that's four Episode now what we want
to say is episode underscore reward duh
equals zero because we start at zero and
then for I in range we'll say 200 these
will be how many steps we take I'm hard
coding that you could later add that as
a parameter if you'd like so the first
thing is we need an observation so we're
gonna say abs equals and that's going to
be player - the food comma player - the
enemy so that's our operator overloading
in action then we're gonna have our
random movement stuff so we're just
going to say if NP dot random dot random
is greater than the epsilon as long as
that's the case we're gonna do a regular
action so we're gonna say action equals
NP arg max of the queue table at the
location of our observation otherwise if
that's not the case we're gonna say else
action equals NP random dot Rand int
between 0 and 4 again it's up to 4 so
it's 0 1 2 or a 3 so now that we've got
the action we're ready to take the
action so where is a player dot action
action so we take that action which
based on the action passed moves based
on these things that we passed and again
I just kind of wanted to separate these
because sometimes like you might want to
move a thing maybe that thing we want it
to move more than one value for whatever
reason like the movement and then the
action to move I think should be
separate for this simple example at the
stage that we're at maybe it's
unnecessary but I think it's good moving
forward anyway no one was complaining
about that and then what I'm gonna do is
I'm just gonna say maybe maybe later we
might want to say something like this
enemy dot move and then food dye move
right you might want to let them move
that would complicate things I think for
training purposes it's better to not let
them move initially Excel just like
really complicates things
I think again haven't actually tested
either way I just initially thought
no no point and then later we can we
should be able to add movement and the
agent should be able to handle for that
pretty well except for the one case that
I was explaining where what about you
know in training if the enemy can't move
it's okay to even like brush alongside
the enemy but if the enemy could move
you wouldn't want to take actions like
that'd be too dangerous because it could
move into us so now what we're going to
do is code for the actual reward process
so after we take our action what was the
results so what we're gonna say is if
player dot x equals enemy dot X and
player dot y equals enemy dot y you done
screwed up reward equals a negative
enemy penalty remember the negative
otherwise your agent is gonna love the
enemy okay so if that's the case now
we're gonna say Elif because we're gonna
assume like we definitely don't want we
we always want to avoid the enemy so if
the enemy is sitting on top of the food
or something we we'd rather just avoid
the enemy hmm so Elif player dot x
equals food on X and player dot y equals
food ly then we're gonna say reward
equals what was it food reward yeah food
reward all caps else so if those aren't
the case reward equals negative move
penalty so I think it's negative one
great now to do our Q function we need
to know we need to be able to like move
we we need to make a new observation
based on the movement so we're gonna say
here is new underscore observation
equals because we've we've made the move
so when we took that action now we've
moved so now we're going to come up here
and say the new observation is equal to
player
- food player might player - enemy and
yeah and then after that move we need to
we get we need to grab the max future
queue that's equal to NP dot the max
value of Q table for the new observation
so that gives us the maximum future
queue and then the current queue is
equal to RQ table at table at
observation and the action that we took
because it won't necessarily always be
the the maximum or the Arg max or
whatever so okay so current queue max
not ordered max anyway okay so now we
can actually calculate our queue so so
now what I would say is if reward is
equal to food reward then the new so if
reward equals food reward new underscore
Q is equal to food reward because once
we've achieved food we're done the same
thing would be true l if reward equals
enemy penalty if reward that should be
equals negative enemy penalty if L
afford equals negative penalty then
we're gonna say new Q equals negative
don't forget it
I really badly want to make that mistake
else new Q will be the Q formula so
that'll be equals 1 minus the learning
learning underscore rate 1 minus the
learning rate times the current Q
current Q plus the learning rate
times parentheses reward plus the
discount times the max future Q let's
fix our pep 8 cool
so that's our new Q function I'll zoom
out pause if you need to catch up okay
so once we've got that we want to update
the Q table so we're gonna say Q table
at Bob's action equals whatever that new
Q is okay so at that point we're
actually done with the Q learning now
what we want to do is show the
environment if we want to show it so
because we kind of want to see it over
time and then also we want to track some
of the metrics and then maybe graph or
whatever as needed so Q table ops
actually using new Q so the next thing
is if show now we want to show it now we
haven't even created the environment but
it's just a grid so it's pretty simple
environment so we're gonna say the the
base environment it's equal to NP zeros
and it's a so actually this will be a
tuple here it's a size by size so a 10
by 10 by 3 because it's RGB data and
then we're gonna say D type equals NP u
int 8 so this is just basically 0 to 256
0 to 256 by 3 so let's fix that so now
what we want to do is so it's all zeros
so this would be an all black
environment basically and then we're
just gonna mark so we're gonna say N and
we're gonna save food dot X food dot y
equals and I'm just kind of thinking now
I think actually in so in an array it's
at
the Y by X I think rather than X by X Y
as we would traditionally think of X Y
right because you've got your depth then
your with basically so I don't think in
the environment though it would really
it's gonna matter like it's not gonna
make any difference whether we flip that
or not because one it's a square and two
we're just trying to visualize so it
would just be like rotated I think right
so I don't think it actually matters
yeah
it would only matter if we were allowing
use or input but I actually think it
makes better sense to say Y X off to fix
that in the tutorial or the text-based
version anyway I think that's correct I
think it would be Y then X in any array
anyway so where is a food Y food x
equals the dictionary food and so that's
just our way of getting the color
initially I had a really great reason
for that but as long as we only have one
food it kind of makes doesn't look you
could just hard code this or you could
just say food color like so you could
just say the food color is something I
had bigger plans for the dictionary and
then it changed so anyway that would
probably make more sense but just show
must go on
so we've got food then we have the
player so we're gonna say
player player Y player X is the
player player in and then this will be
the enemy and move this mouse enemy and
and then here will be enemy dot y and
then enemy dot X okay so that'll change
that grids color and by grid I really
it's really an image now so it's a it's
an RGB image so but Italy it's only a
ten by ten so so the next thing we're
gonna do is we're gonna make this an
actual image image and we're gonna say
image equals capital I image dot from or
a and then n and then here's where I'm
saying hey this is our G frickin beat
but for whatever reason it doesn't care
it's BGR and the way I found that out is
by is yeah I'll let the player moves and
the food and the enemy doesn't move and
it's so clear it's BGR it's 100% BGR but
it's crazy anyway not n its n Vividh so
that gives us our image now what we're
gonna say is image equals image dot
resize and then we're just gonna resize
this - you can pick anything you want
I'm gonna say a 300 by 300 just so it's
like I can see it and then see v2m show
and then the title is whatever you want
it to be numpy array image and then I'm
gonna throw in some kind of shoddy
code here but that's okay everybody
accepts it if reward is equal to the
food reward or the reward is equal to a
negative enemy penalty this means the
simulation ended and we we really
screwed up so so not if we didn't get
the food but if we got the food no we
didn't screw up necessarily if we got
the food yay or we hit the enemy really
bad so if that's the case I just wanted
to pause it just for a moment longer so
I'm just gonna say if C v2 dot wait key
and in this case it's 500 milliseconds
and then this little batch of
Awesomeness oxf F equals Ord Q basically
if you hit the Q key it breaks I'm gonna
throw that in and this is just how you
let that cv2 window just refresh live
basically so if we either made it to the
food or we really screwed up by hitting
the enemy we want to run that otherwise
else
we'll just say wait key one
so it's frame-perfect frame and then at
the very end if we either get the food
or this should be a double equals should
not be an assignment yeah okay so the
frames will go really really quick
unless we hit the enemy or get the food
and then we'll just hang there for a
second just so we can catch it because
this is one millisecond so well it's not
actually gonna be 1 millisecond it's
gonna take a little longer than that but
it's gonna be quick so so that's our
show so now we can see the environment
now what we're gonna say is episode
reward plus equals reward and then if
reward is equal to the food reward or
reward is equal to the negative enemy
penalty it's over we break and then
finally in our episode episode episode
loop so just one tab over we're going to
say episode rewards uh dot append
episode reward and then we're gonna
decay the epsilon so epsilon x equals
our EP s decay and then finally we just
want to graph things at the very end so
I'm going to say moving underscore
average is equal to the number of
episode rewards uh so this will just
create a moving average and numpy ones
show every and then divide by show every
and then does this mode I don't even
know let's tab that over yeah mode is
valid and I believe that would be a
mode equals valid so if you don't have
enough to do the calculation should you
still do it or not I'm pretty sure
that's what it that's going to be but I
could be wrong anyway it's just a way
for us to make a moving average that's
totally a Google operation there because
you wanted to be like a rolling moving
average you could also do it in chunks
like we did in the previous environment
I just think a moving average is a
better idea so anyway there's a moving
average and then we're going to PLT dot
plot and then for X I'm just gonna say I
for I in range of the Len of moving
average and then for our Y we will plot
moving average beautiful and then PLT
dot y label we're gonna say reward and
then show every so the moving average
that we'll use is whatever the show
every is so if it's 100 if it's a
thousand whatever it'll be a thousand
moving average and so on
PLT e-elt dot X label we'll say this is
our episode number PLT touch show and
then finally with open and then we'll
make an F string and we're gonna say cue
table
- int value of timed time dot pickle
with open that pickle comma WB as F
pickle dot dump and we'll dump our cue
table into F hmm woo okay
coded that all the way through how many
reward on food reward Oh so this should
be a lower case lower case reward so
what are the odds that I coded this
without any other errors I don't know I
don't know we're gonna find out though
we're gonna find
let's go into here CMD Python Python
q-learning 4 come on baby
ok so it looks like we do very well
there
do we have an error are we still making
it wording invalid I think I think we're
fine
cool awesome show me my environment dang
it
so so yeah this is your enemy this is
your food this is your player so it's
running up Oh looks like we got at that
time so if you see blue and red really
quickly that means we got to the food if
you see green and red that means we ran
into the enemy at the very end so
everything's happening sometimes very
quickly so we're only on episode what
9000 yeah so he's learning quick ok he's
doing pretty good I forget how many
episodes we said to go on
probably 25,000 I think yeah he's moving
fast so anyway though the order that we
drew things the reason I made that order
was so I could figure out what happens
at the end so these are all victories so
this is pretty cool to see I'm just
waiting for the end so we could load it
in and I can show you guys you know how
we can start to kind of tinker tinker
with things I also can bring up the
text-based version of this tutorial for
some other longer simulations as well ok
so here's our graph we can see here we
started these this is basically the
rolling average of our accuracy or not
really our accuracy are rewards
basically oh we type out reward anyway
cool so you can see it definitely
learned things really quickly and again
half of the time it should not be able
to get there but it uses the wall to get
there which is super cool ok so what I'm
going to do is close out of here guess
I'll just oh oh I had to save ok so it
saves our cue table so as you can see
our cue tables like thirteen point six
megabytes as we continue to train
because we're using pickle
and a dictionary that will expand over
time and you can actually use that
expansion to know how many more random
or yeah how many more random initialized
values are being updated kind of an
interesting outlook on my flaw of using
a dictionary and pickling a dictionary
as opposed to using numpy correctly
anyway yeah so oh so so let's say okay
we like what we we think this is good so
then we could load in the cue table so
let me copy this and then rather than
none will load in the cue table itself
so now we'll load this and then now
let's just set epsilon to zero
what else would I need to change trying
to think I think we're good actually
let's say show everyone I think that's
everything I'm probably missing
something but let's run it let's see how
we look so now you're seeing it play
every time he's doing really good get it
get it get it
Oh beautiful beautiful again more ah
solved that's pretty cool okay so I
think I think he passed around so those
those worries like bouncing around
that's a perfect example of the failure
because he can't move I see he oh man
either so cool but a lot of times you
can see it actually go it specifically
it could have gone straight there but
instead are not really straight so I can
only move diagonally but it could like
zigzag right but instead it goes to the
wall and bounces off the wall and I I
just can't believe it's learned behavior
like that I just I just can't it's crazy
but anyway it has so now what I want to
do let me break this always the other
thing I wanted to show Oh let's turn
let's turn movement on and see how we do
there so let me just do this whoops this
let's save that and then run it again
and so so he died there he's surviving
now pretty well he's catching so fast
that's just crazy died there once died
again
ouch anyway yeah that I just think this
is really cool so I'm super happy that I
made the environment and I hope you guys
are as impressed as I am with this
because if I made a neural network that
was like I say a continent and it looked
at this and it tried to make these learn
to do this it would take a while and in
this trained in like I don't even know
like what three minutes I just think
that's pretty cool I mean I understand
its basic stuff but I think it's really
cool so so now we could take things
further and like we could make a bigger
environment we could do more steps or
whatever you want to do and so feel free
to tinker with the environment
the next thing or the only other thing I
want to show you guys is some of the
results of using much much larger
environment so in so we're doing a 10 by
10 if you just change that to a 20 by 20
your cue table size goes from about 15
megabytes to like 250 megabytes just for
comparison's sake and then if I figure
out where the heck I promise you there's
some cool videos and photos we just have
to get to them so let me see what this
one is this one's probably what we were
just looking at yeah and then I just
showed how to how to get it live how to
get movement blah blah blah blah nobody
cares here so here I start to actually
train for a 20 by 20 space so this is a
20 by 20 with no movement I want to say
yeah so just seeing could it actually
learn a much larger environment and
obviously it does pretty well and then I
turned on movement for both the enemy in
the food still no problem and some of
those are pretty complex where it's got
to go around the enemy or deal with the
enemy which
again it's pretty cool that it can even
do that and I almost wonder if allowing
movement makes it easier because he's
always trying to get toward he he's
always trying to get towards the food
and since the food is capable of moving
in ones like up-down left-right
not just diagonally that that might
actually even make it easier for the for
the agent let's say rather than he okay
so that's our environment I hope you
guys enjoyed if you've got questions
comments concerns whatever you think I
did something wrong feel free to let me
know I I didn't even think about all it
can only move diagonally how is it
actually getting food until I showed the
code to Daniel and he's like oh it could
only move diagonally how does it get the
food like half of the time and then I'm
like oh oh that's so cool because it
uses the wall but it has no idea about
boundaries like it doesn't know about
the wall so I'd have no idea why that
works because again it's just a cute
it's like brute force right it's just
the fact that the how do I just don't
get it I don't know how it could learn
to use the wall or if that's just random
like if the wall being there it's just
random like it hasn't actually learned
to use the wall it just happens to hit
the wall you know that could totally be
the case but anyway yeah that's pretty
cool because that's pretty complex
behavior that I'm surprised it would
learn but anyway pretty cool it does it
apparently does so that's it yeah like I
said questions comments whatever feel
free to leave them below if y'all want
to support the channel you can click
that blue join button get some shout
outs good a special rank in the discord
channel special channel hangout in and
support me in the free education and all
that kind of stuff so check that out if
you want otherwise I will see you guys
in the next video where we will be
getting into we'll probably be leaving
basic Q learning behind and moving into
DQ ends or deep hue learning so I will
see you there