what's going on everybody welcome to
part 6 of the reinforcement learning
tutorial series as well as part 2 of the
deep Q learning as well as dqn and deep
Q networks tutorials where we left off
we're basically ready to add the Train
method to our agent and then basically
incorporate our agent to the environment
and start that iterative training
process so I've switched machines I'm on
a paper space a boon to machine now and
then I've pulled up the exact versions
that I'm using also people ask a lot
like why am i on paper space it's just a
nice virtual machines in the cloud kind
of cloud desktop but also high-end GPUs
it's great for doing machine learning in
the cloud if you want I'll put a link in
the description
it's a referral link it's like a ten
dollar credit you can definitely check
them out this wasn't really meant to be
a sponsor spot and like I really do use
paper like I use the heck out of paper
space so anyway and I've got plenty of
high end GPUs locally it's just very
convenient to use paper space but
anyways I've pulled up the exact
versions of things I'm using tensorflow
2.0 is on the way
it's just not quite ready so anyway
still using tensorflow one these are the
exact versions you can feel free to
follow along on a different version it's
just if something is going wrong and you
want to match my exact versions of
things you can do this also in case you
didn't know what's not all caps please
in case you didn't know like to install
exact versions it would be like pip 3
install a tensorflow - GPU double equals
one point one three point one four
examples so that's how you install an
exact version of something ok let's go
ahead and continue so I am in actually I
think I've got it up already yeah cool
thanks Harrison any time bro so so we're
going to do is come down to our agent
class of the dqn agent and I'm just
going to go to the very bottom here I'm
gonna make a bunch of space and come up
here and now what we're gonna do is just
add the new train method so define train
and then we're going to pass here cell
terminal underscore state and then step
whatever step run okay so the first
thing we want to do is do a quick check
to see should we actually train so
recall we're gonna have this replay
memory and then from the replay memory
which should be a quite a large memory
we're gonna grab a mini batch that's
relatively small but also is a batch
size that is of decent size so typically
the neural networks feel like to Train
in batches of like 32 or 64 or something
like that so we're gonna do the same
thing here so we want 32 or 64 to be
pretty small compared to the size of our
memory so in this case I want to say our
max memory size is 50,000 and then the
least amount that we're willing to train
on is 10,000 and the reason why we want
to do that is we don't want to wind up
overfitting so we want to use this
replay memory principle I guess but what
we don't want to do is have replay
memories so small that every step is
training on the exact same thing so
we're effectively doing like you know
hundreds of epochs on the exact same
data that's not what we want so anyway
what we're gonna say is if Len of self
dot replay underscore memory is less
than the min replay memory size if
that's the case we'll just return we
actually don't wanna do anything here
otherwise if it's enough then let's go
ahead and get our mini batch which will
be a random sample of self dot replay
memory and the sample size that we want
is mini batch underscore size and we
need to do both we need to import random
and set the mini batch size so I'm going
to go up to the tippy-top here I'm going
to import random and then I'm gonna come
down here mini batch size we'll set that
to 64 and I'll go back to our bottom
here awesome so now that we've done that
we want to get our cue values so don't
forget as well that one of the things
get out of my face
let's go here python programming
net let me type the dqn
we're actually going to use some stuff
from the other one too so I just pull it
up but recall the following image so we
actually want to get so this bit is kind
of handled otherwise both learning rate
and then just some of this logic is
handled by the neural network but we
still need the reward discount in that
future value so we still want to use
this little bit so to do that we need to
know current Q values and future Q
values so so mini-batch okay so what
we're gonna say here is current
underscore states is equal to the numpad
a and then we're going to use a list
comprehension here we're gonna say
transition 0 for transition in
mini-batch okay cool and then what we
want to do here is normalize it so again
if you don't know why we're normal I
think that check out the basics tutorial
but basically it's just images so
anytime you have RGB images that's like
0 to 255 and onward you can really and
not but I had normalized the wrong word
I've probably been saying that wrong the
whole time anyway instead what we're
actually trying to do is scale it
because all the images are normalized
already pretty much but instead we're
actually trying to scale it between 0 &
1 that's just the best way for
convolutional neural networks to learn
on image data it's just useful to have
and really it's the best for any machine
learning you generally want to scale
between 0 & 1 or negative 1 and 1
anyways enough on that so current states
then what we want to grab is the current
Q's list and that is also going to be
equal to self dot model dot predict now
pay attention that's self dot model so
it's at that model that is the crazy
model soft model dot predict and we want
to predict on current States and then we
want to do the same thing with future so
we're gonna say new current States this
is after we take steps it's gonna be
equal to the number rate and again we're
gonna say transition 3 I want to say is
that right yes yeah transition 3 for
transition
in or mini-batch and again we want to do
by 255 and I'll explain these index
values in a second if you don't
understand those or you forgot it should
be up here somewhere
or is that I think we're gonna have to
define that in our environment
anyway I'll explain that in a moment
yeah so coming back a new current okay
so now what we're gonna say is future
Q's I'll explain it basically bottom
line future Q's list is equal to self
dot target model so now we're using that
target model that doesn't so crazily
change dot predict against the new
current states now we're going to
separate things out into our typical X's
and Y's so these will be our feature
sets these will be our labels or our
targets so these will be this will be
like literally the images from the game
and then this will be the action that we
decide to take which we still that's
gonna be in the environments one will
get there anyway
it'll be like up-down left-right
diagonals and all that stuff okay so now
what we're gonna say is for index and
then we're gonna have this giant tuple
and this is what is consisting in this
mini batch of things so you've got the
current state
you've got the action the reward the new
current state tub don't put the s there
and then whether or not we're done with
the environment so that's exactly where
these indexes are coming from so current
states we're grabbing the current state
from mini-batch and then down here we're
grabbing that new so 0 1 2 3 right we're
grabbing that from the mini-batch and
then we're predicting on it based on the
state itself but then with this
information what we can do is calculate
that last step of our formula from
before right we can calculate this
little bit here to create our Q's so for
index creditor to done
apparently my mouse has decided to stop
working what do we want to do the first
thing we're gonna say is if not done
then we still want to actually perform
our operations so what we're gonna say
is max future Q is equal to the NP max
of future Q's lists for whatever index
were actually at right now and then
we're gonna say the new Q is equal to
the reward plus the discount discount
which I don't think we have because it
didn't try to autocomplete that x max
future Q else we're just going to say
new Q equals reward so if we are done
then we need to set the new Q to be
whatever the reward is at the time
because there is no future or Q right
we're done okay so now we need to go up
to the top and set our discount okay so
going up to the top here let's set
discount equals 0.99 then we'll go back
to the bottom here and now what we want
to do is actually update these Q's so
recall this is how our neural network is
gonna work you've got input then you've
got the output which is your Q value so
let's imagine the scenario where we got
this out the action that we would take
given no Epsilon let's say would be this
one right 0 1 at index 1 we take action
here for because the Q value is 9.5 it
was the largest Q value but then let's
say I we don't like that action we ended
up actually degrading it a little bit we
actually want that new Q value to be 8.5
well to update that for a neural network
like in our table we just updated it
right you just updated that one value
but in the neural network we output
these four values so what we end up
having to do is we update this 9.5 to an
8.5 like in a list and then we refit the
neural network to instead be three point
two eight point five seven point two one
point three so that's the next thing
that we need to do so coming down here
what we want to say still in our for
loop we're going to say current
Q's equals the current Q's list at the
index that we're at as we're iterating
over and I miss sublime text we're
iterating over yes these two values
index and then you should be able to
guess what I meant to have here and that
was and numerate and also in Wu maybe
today is not a day for tutorials for me
cheese and then mini batch that's
probably going off of the screen here
see if I can help out here there we go
so for index Qurna state action reward
new current state done in enumerate mini
batch I guess I got stuck there because
I was explaining these and then pointing
out the transitions anyway now it's a
valid loop so if not done anybody knows
if Adam just has a built-in Pepe kind of
syntax checker let me know post comment
below because this is what comes on the
paper space machines and I just can't be
asked to always be throwing on sublime
text anyway
current cues okay so that now we have
our current cues we fixed our enumerate
now what we want to say is the current
cues for the action that we took is now
equal to that new cue value once we have
that we're ready to append to our
features and our labels so X dot append
so that would be the current state and
then Y dot append current cues so again
features labels this is the image that
we have and then current cues or the Q
values so cool so just just in case
that's not getting through X's wise all
right so now now that we've got all that
we can pop out of this for loop and then
we're going to say self dot model dot
fit and we've got another really
line here NP dot array of our X values
divided by 255 numpy array of Y and then
our batch size will be equal to exactly
what our mini batch size was and then I
think I should be valid I want to never
do this but I just I'm running off the
screen I think that was still run
because we're in the parameters here and
then we're gonna set verbose to zero
then we're going to say and make this
pet paid as well then we're going to say
shuffle is false because we've already
grabbed a random sampling we don't need
but it's no benefit to that callbacks
will be equal to that custom callback
that we wrote so self dot tensor board
and then so we're gonna do that fit if
terminal state is the well actually if
terminal state else none so we will fit
all of this if we're on our terminal
state otherwise we will fit nothing so
now what we're gonna do is come down
still in line with what we're just
writing there and we're gonna say if
terminal state if that's the case we're
going to self dot target update counter
plus equals one so if we've made it to
the end here and what we're trying to do
here is determine if it's time to update
our target model yet so the next thing
we'll just throw in right underneath
this so updating updating to the term
and if we want to update target model
yeah okay so so we'll keep adding to
that and then if self dot target our
target update counter if that is greater
than update target every so this is yet
another constant that we've got to
defined if we've hit that number or we
Vic actually if we've exceeded that
number what we want to do is self dot
target underscore model that set weights
and we just want to set the weights to
self dot model dot get weights so we'll
just we just copy over the weights from
our initial model and then self dot
target update counter needs to be reset
and we set that to zero so now let's go
ahead and update target every and I'm
trying to look and see what I win with I
think it was a five yeah 5 so we'll go
up and we'll say set update that to five
okay once we have that now we have to
bring in our environment and our blog
class so those are two things I'm just
gonna copy and paste in I don't think
there's any benefit we'll run through
them just in case
but I'm gonna come down it's a lotta ads
bro that's our modified tensor board we
already oh wait mother wrong tutorial I
was like as soon as I saw a modified
tensor board I was like oh hmm I'm just
gonna go to the bottom I guess and then
let's say where is so that's our agent
class that we've already written
modified tensor board yes so this we're
gonna grab I think we'll just grab all
the way here and we'll just paste that
above and I'll talk about that here in a
moment so I'm grabbing all of blob m and
the code that comes right after that so
defining basically everything from blob
em to modified tensor board we're gonna
grab that I'm gonna come over here
and pasta okay so we have our blob
environment we're saying the size is 10
really nothing too fancy here I don't
really think anything's been updated
there there the reset like all these
kind of methods here like everything's
been kind of turned into a method
instead if you still wanted to move in
this case we'd also have to defy think
we'd need that to be self dot enemy self
dot food
I'll just throw that in there before
before I forget that so enemies self dot
food but we're not moving them for the
basic one anyways so this is all good
whether or not we want to render it and
then get image this is just so we can
actually pull an exact image from our
environment because that's we're going
to use the image as input as opposed to
before where we were just using the
image to display it and look at it now
the image is actually the input value so
we're not doing the Delta of to the food
or to the enemy but you could you really
could it's just that I couldn't really
decide which way I wanted to do it for
this tutorial because that's actually
it's easier to learn that in this case
this model would perform way better if
we were doing that but the problem is
it's not really what we're using DQ ends
for most of the time so I decided not to
do that but you can and maybe if you
want to you should treat that like
change that like make the input be those
that was for observation things like see
if you can do that that should be a
pretty easy change if you understand
what's going on anyway so that's our
blob M then we're just kind of defining
the blob M here we're setting some
initial things and then we're setting
random seed to one just so as we do this
more and more
hopefully our results will be good to
compare to each other when we change
hyper parameters and stuff if you wanted
to train multiple models on the same
machine you can use this code as well
then finally we're gonna make this
directory called models if we need it so
just a bunch of helper stuff really
nothing specific to DQ ends so I just
didn't really
I don't really see any benefit for us to
write out blob em you're welcome so the
other thing would be let me check your
see we've got blob and we're missing
tensor board we got the D queuing agent
so the other thing we don't have is the
actual blob class so I'm also going to
take that we'll take that copy that I'm
gonna throw that down here again we've
already kind of rips I'm actually gonna
cut that I mean oh I tried
I just tried to copy it from my main
computer here so this will take our blob
class and then we'll we'll check some of
these other values but we still have
some code to write at the very end here
pasta
so this is our blob class again we
already had a blob class so I didn't
really see any point to going over this
again
the only difference here is we have this
new equals so we're just doing another
operator overloading just so this is how
we can check to see if two blobs are
over each other now so cool add that and
then here I've added all nine movement
possibilities so not only the diagonals
also don't move and then move up down
left right again not complex code don't
see any reason to belabor over that here
we do movement this keeps it in bounds I
don't really like none of this other
stuff has really changed so again
nothing with DQ ends for us to really
belabor over so cool now the last things
that we need to do is actually do our
iteration and all that so what I'm gonna
do is come to the bottom here shoot I
can't remember if we actually created
our agent or not so d cute no doesn't
look like we did
so come to the very bottom here and I'm
going to say agent equals DQ and agent
so we've got our agent now and then also
we did I believe we've already got our
environment created should be like right
after the environment if I recall right
just got a check yeah so we've got our
environment we've got our agent now
we're ready to start iterating over
things and in fact I think maybe even
before we do that let's make sure we
have all of these and ikana
I think I'm just going to copy paste
these as well and then we'll talk about
those I just want to make sure I'm not
missing something so in fact I'll just
do this so replay memory size 50,000 ok
64 5 got our name discount ok cool so
I'm gonna get rid of that and then let's
see if we've got anything new here
memory fraction I'm actually not I'm not
we're not using that at the moment but
that has to do with that code that was
commented out 20,000 episodes that
should be fine let's see here epsilon we
start at 1 that's fine this is a decent
decay for 20,000 steps minimum minimum
epsilon is fine this is for our step
aggregation show preview false we can
set that to true if we want to actually
see the visuals of everything running
okay so coming back down to the bottom
here it's a lot of scrolling need like a
paid end of page button now we're ready
to actually iterate over everything so
what we're gonna say now is for episode
in TQ TQ diem for the range of 1 to
however many episodes we decided we
wanted which was 20,000 plus 1 and then
we're gonna say ask ASCII equals true
this is for our windows fellows units
our unit is
gonna be episode episode cool so now
what we're going to say is agent dot
tensor board daunt
step is going to be equal to whatever
Epis biz episode we're on then we're
going to set the starting episode reward
at two I can type episode underscore
reward we'll set that to be a zero we're
gonna say we're on currently step number
one current state is going to be equal
to m dot reset so we're kind of matching
the syntax of hope that wasn't off the
screen what we're setting we're kind of
matching the syntax from opening I Jim
let me make some more space here before
that happens again hopefully that wasn't
off-screen I'll go back and check anyway
it wasn't really too complex of code at
least ok current state we set that to
the reset we're going to say done is
equal to false and then we're going to
iterate over all this so wow we're not
done the next thing is if if MP dot
random dot random is greater than
whatever our epsilon is so this is very
similar to everything we've done before
so we're just going to say action equals
NP arg max of agent dot get Q's get
cubes for whatever our current state is
otherwise so else action equals NP
random dot Rand int between 0 and n n
dot action space size in all caps okay
so once we've done that we're going to
continue on our theme of similarly some
text things and we're going to say new
lips new State reward and whether or not
we are done is equal to
and Don Stepp action and then we're
going to say yep I really just wonder if
I up shrub typo'd episodes somewhere
else anyway plus equals whatever the
reward is no jerk whatever the whatever
the reward is and then what were to say
is if show preview so if that is the
case and not episode modulo aggregate
steps every will do n dot render so that
just shows the environment if we want
that then every step we have to add that
to the replay memory so we're going to
say agent dot update underscore replay
memory and we're going to update that
with all those what five things current
state we've got action reward new state
new state and whether or not we're done
there so then agent dot train done and
then step and then current state current
state equals new state step plus equals
one okay so now yeah I think now we're
gonna copy the rest there's really
nothing again nothing dqn related and
this is also code that we've already
written I'm just trying to keep this as
short and applicable to what we're
hopefully trying to learn so this is
where we are now and then now all we
want to do is we're going to add the
reward to episode rewards we are going
to aggregate various stats so we're
gonna grab the average the minimum and
the maximum reward and then with those
things we're actually going to throw
those in a tensor board rather than
creating our own you know matplotlib
chart
we could use matplotlib that's totally
fine so like if you are following along
and converting this to like I don't know
some other machine learning framework
first of all how dare you but also you
could use matplotlib and then here min
reward basically what we're trying to do
is find a good reason to save a model so
if a model scored really good based on
you right now we're just throwing in one
thing but you could change it or you can
make it based on multiple multiple
things so first let me just copy this
over now so I'm going to say copy I'm
gonna come over here where we just were
pasta and cool so we add that and then
so here min reward if we scroll up to
the top that was one of the things that
we just kind of threw in I think ya so
negative 200 so this is like if the so
if the model hits an enemy it's a
negative 300 and it's taken steps so if
the model gets a negative 200 that
mostly just means it didn't get to the
food but it also didn't hit an enemy so
that's still like somewhat positive
let's say so so if the worst agent still
never hit an enemy we're saying hey
that's a good thing but you could also
say like you could take min reward you
could set it to negative 100 and instead
you could say rather than rather than
min reward being greater than or equal
to that we could actually say average
rewards so if the average reward is
greater than negative 100 or zero then
let's save that model so then we save
that model okay otherwise we're just
decaying Epsilon that's the same as what
we did before so okay so we've covered a
lot of things I did copy and paste some
things and I know someone's gonna be
like you made fun of people for copy and
get paid what I meant before was that
people were copying and pasting from
each other in this case the things I
copy and pasted was like the stuff that
we've already written or the things that
don't matter so feel free to let me know
if you like that or dislike that it
would have taken probably two more
videos for us to rewrite the
do the environment thing and to do what
was the other thing we copy and paste
well also the tents are bored the blob
em and we did we only made a couple of
changes that be even like all those
stupid elif's like come on you're
welcome
so anyways let me know if you like that
or just like that but I think that made
the most sense so now we're ready to
actually train so what I'm gonna do is
I'm gonna start trying see if we hit
errors try to debug through those errors
I'm sure I made a lot of typos and then
I'll kind of just like pause the
recording and then I'll show you
training through and then we can kind of
look at how the model is doing and stuff
like that and talk about it and you know
just really have a good time together so
so now what we're gonna do is first
let's open in terminal let's go ahead
and run dqn part two dot pi know whoa
just cuz there's no way we don't have
said of course though the one thing that
so we forgot tensorflow
so great and I think what I'm going to
do actually is because we probably are
missing a few of the other imports I'm
actually just going to copy the imports
from from here so I'm sure we're missing
a few so I'm gonna copy that up here
deep reinforcement learning is really
complicated guys there's a lot of moving
parts in case you didn't know okay so
yeah I'm trying to think if there's
anything here that might be confusing
you know I've explained all that stuff
so cool so now let's go back to where I
was I'm gonna close that cuz I don't
want that getting in my way anymore and
let's try and rerun part two yet again
okay I don't quite understand activation
linear oh it's a typo act deviation okay
so that was in our house a long time ago
guys I
I know you might watch these videos
back-to-back or think that like it was
like one day apart but it's been like
almost a week probably since I filmed
that other video I can't find it but
carve 2d find it let's go
okay activation activation what am I
missing here
Oh activation activation linear okay
cool let's try again I wonder how many
errors were actually gonna hit here step
action equals
get Q's current state did we actually
pass that's totally possible currents
get Q's for current state mm let me see
if that's actually get Q's why are we
missing
I don't think that's true get Q self
state we shouldn't he shouldn't need a
step there if I if I threw that in it's
a mistake get Q state right okay because
that's just to predict so I probably
threw in step I was maybe thinking of
train or looking at train maybe anyway
all we're trying to do there is make a
prediction and to make a prediction the
all we eat is state whether it would be
known no need for step there so let's
try again what are we on four or five
that's a lot of errors oh my goodness
DQ n has no attribute model predict I
don't see a typo there self dot model I
think probably it's self dot model dot
predict though would be my guess self
dot model underscore predict let's find
where we wrote that okay
get cubes I'm guessing that's it let me
just check let's see
def get Q's self dot model Wow even in
that that can't be right I'm just gonna
change this but actually in the in the
that's so weird in the text-based
version it even says that self dot model
dot predict that's got to be that's got
to be a mistake I mean I just can't
imagine that would be correct who knows
them sorry I'm getting pretty tired
this stuff is some dense stuff let's try
that again let me go over to the end of
this there's no way please just start
training I beg you
self dot model dot predict self dot was
that in get Q's model predict yeah it
wasn't get Q's interesting okay so it's
actually training thank the Lord
yeah interesting so that's so funny I
wonder how that how that made it in
maybe code up to this I guess cuz I just
I type it in the previous tutorial and
then I copied it to here I'm surprised
nobody picked that up in the previous
tutorial to be honest that that that
one's already been released for the
current channel members was about a
hundred people so I'm surprised nobody
noticed that one what was like what's
model predict but I guess maybe they
thought that was coming I don't know
anyway okay cool so while that's
training shout out to speaking of
channel members some of my newly
upgraded for different months channel
members I'm clicking the wrong thing
there we go fubá forty-four you honest
Gessler both of you guys on working on
month number twelve which is crazy Frank
Lloyd jr. three months payments I yeah D
five months Chris Olin four months and
the Cassie wacky for seven months thank
you all very much for your support it
allows me to do things like this that
take me so freaking long in the ad
revenue will never pay for this video so
I really really
appreciate y'all's support because this
stuff's super cool but like I said I
couldn't could not do it if I didn't
have some sort of outside support
because advertisers don't care about
these videos not enough mostly because
not enough people care about these
videos it's very difficult as a niche
channel doing it like it's already
programming as niche nobody wants to
watch programming like pretty much every
one of my viewers is like super unique
like you guys are watching this I assume
most of you for fun so you're already a
pretty rare individual so anyways it's
it's hard out there for a syntax anyway
so what I'm gonna do is I'm gonna let
this keep training I'm gonna let it I
think will let it go through 20,000
steps and we'll see where we are I might
run it again so sometimes as the model
learns and does like a cycle through
epsilon it's useful to do it again so
reset epsilon back let it decay again
and I found that to be true also with
learning rate so just with any neural
network so even if you could let it
continue training or decay it's lower so
let's say you did 20,000 steps you
decayed learning rate at a certain rate
if you want it and then did it again for
another 20,000 steps now you've got
40,000 total well if you decayed at a
certain rate that would have gotten you
to the same ending learning rate often
it's actually better to cycle it so do
20,000 reset 20,000 again it's better to
do that than to do one kind of pass
through so it kind of helps get you out
of these local traps so I'm finding the
same thing to be true for epsilon so
rather than let's say your end goal is
to do a hundred thousand steps you could
decay have a decay in such a way that it
decays smoothly all the way through to a
hundred thousand steps so it ends at a
pretty low you know
let's say less than one percent epsilon
but doesn't get there quickly it gets
there you know right on time to reach a
hundred thousand steps let's say it's I
find it's actually better or so far I've
found it's actually better to instead do
20,000 steps and do five cycles like
you'll end up with higher accuracies
higher rewards that kind of thing so
anyway kind of cool so maybe I'll get to
show you guys that here we'll see
so questions whatever if you're feeling
fuzzy it's normal ask questions either
post them in the comments or come join
us in discord GG / centex otherwise I'll
see you guys for you in a few moments
for me and probably a looks like four
hours so see you then okay a moment or
two has transpired since since they're
recording but I don't want to make this
a separate video so here are the results
of a 2 by 256 confident and basically
what I did was I did like three passes
so this would be like our first pass
through the data and you can see the
average started at negative 180 and it
kind of made its way up to negative 73
and then basically I loaded where that
model left off ran it again so just kind
of read Akay the epsilon again so there
we go
this time getting to a peak of about
negative 50 and then I ran it again and
this time got it to be an average of
about what's this one negative thirty
seven point five eight and then we could
we could try to keep continuing to do
that I just it's taking a lot of time to
train these and this isn't necessarily
like the coolest model in the world okay
it's not the coolest environment so I
don't really want to spend too much time
going over this I've already taken quite
a while to do this series like you guys
don't understand like just just doing
the last two videos has taken well over
a week probably two weeks worth of work
especially if you include what what I've
had Daniel doing as well I mean it's
just taking a lot so anyway so I'm gonna
stop it here and we're gonna move on to
another RL algorithm so but what you can
see is it's actually pretty cool some of
some of these stats like you can just
see as you continue training everything
got better like you can see on the first
past even the minimum everything except
for pretty much were word max like
pretty quickly it starts to max out but
here you can kind of see the minimum
even improves which is pretty good
because it I guess I guess we're still
not above negative 300
like if we could get Minh to be above
negative 300 that would mean we never
hit an enemy which would be really cool
anyway cool so that's one I have another
tensor board and this is a different
model and what this one is is with many
enemies as well as movement is turned on
so I had 10 enemies one food one player
and I allowed movement to be on for all
of the enemies and this was the training
I was really hoping to see something a
little better than this mostly because
over here I was kind of expecting that
we would even we would be better than
this so then I started to wonder well
you know part of the problem is if you
were to really look at this image most
of the image is like black and or just
nothingness so it's like not useful in
any way to the model so then I started
to think hmm hmm maybe that's the
problem
what if I add a lot of enemies then we'd
have like more to work with per each
convolution that happens in each window
but that didn't really solve the problem
either so and if anything it just made
it harder so you can see here it really
never did any better than like probably
the best average was negative 120 which
is pretty good I mean that's still out
most of the time it doesn't hit an enemy
at least and if it doesn't hit an enemy
I guess it didn't exhaust steps either
so I mean at some point it still
eventually on average getting to the
food but I don't really think it's it's
that great and like I said I'd rather do
a cool environment something like
Starcraft or or we could make blob and M
cooler but this is just taking way way
too long and I just don't want to
necessarily die on this hill I want to
explore some other Hills and then figure
out which one I really want to do battle
on so anyway I think that's it I think
there's anything else I really want to
say other than this if you wanted one
thing you could do is try
remember now so I have DQ and tutorial
but these are actually from a different
directory I'm sure if this is DQ n I
think it's in DQ n stuff let me check
real quick
open up DQ and stuff let's go to logs
yeah ok so so we could kind of watch the
model if we wanted so I can go up here
the last time I did this Daniel made fun
of me for this being the way that we
test or like we just watch the model but
I think it's fine
so what we'd want to do is load in the
best ish model so past three and then we
could load in well the one that's here
is actually ok so the average 25 max
0.35 is the average I like this one
better
and then the min negative whoa whoa Wow
thank you to 454 min dang so it almost
took seven hundred steps and oh this is
a negative seven this is a positive whoo
stick with the positives okay so this is
probably the one I grabbed so it's fine
so we'll load in that model we can set
render show preview to true aggregate I
think preview is when we aggregate also
load model I don't with this we didn't
even do in the tutorial but just so you
can see let me just save this this is
just in the model method so let me just
search for load model here let's just go
to next next right so if load model is
not none so if we set it to none it'll
create a fresh model otherwise if we set
something there it's gonna load the
model and then it just loads the model
okay nothing fancy going on there um and
I think that should be everything that
you would need to know as far as
changing to look from what I just
covered in the tutorial anyway so we can
load model aggregate every show preview
true okay let me then come into here
open and terminal let's do Python blob
world CNN nine moves load hi
so hopefully this trying to think if I
aggregate steps every I feel like I
might be missing something as far as
getting it to pop up every game because
that was 50 what episodes it's been just
like today's a fresh day and I meant
looked at this code let's go check so
show preview let's just copy paste
whoops
show preview and go down to the code
here ever and not episode thousand so
instead let's actually say modulo one so
we'll see every single episode thank you
very much so this is a bunch of episodes
I mean at least he's not hitting the
enemy but he's pretty stupid
this doesn't Oh epsilon is hi okay okay
let's fix that real quick sir probably
why you want a standalone script for
this but yeah you know whatever epsilon
we will just set to be zero one more
time
okay yeah he's much quicker now i
cycling through and just getting food
he's not even hitting the enemy at all I
haven't seen him hit yet that time me at
least you didn't get the food he likes
the corners boom boom boom boom okay
yeah so I mean this one's not terrible I
mean it's pretty good he really likes to
go into that corner though that's kind
of funny nice okay all right so that's
all I'm gonna do here we're gonna get
into some of the other reinforcement
learning models and like I said I'd like
to do something more complex and it kind
of bums me out that it was taking so
long but hey that's that's deep learning
for you it does take a while to train
models so I would like to do something
far more advanced but I want to figure
out which thing I want to do which which
reinforcement learning algorithm do I
want to use for that so still on the
hunt we could do some cool stuff with DQ
ends for sure but I definitely want to
check out a three-c for sure before that
maybe even neat stuff like that so
anyways if you guys something you want
me to look at let me know leave a
comment come join us in the discord
whatever otherwise it is a new day so I
will shout out some of my other channel
members Erik Gomez on five months robo
five months Alexis Wong 7 you day
Bingley and Metanoia all seven months as
well thank you guys very much for your
support it allows me to spend a week on
a stupid algorithm actually it's really
cool I like that I can do this I hope
that I can continue being able to do
tutorials like these that I will never
make the ad revenue back on so thank you
guys very much for your support till
next time