what's going on everybody and welcome to
part five of the reinforcement machine
learning series in this video in the
subsequent videos we're gonna be talking
about deep Q learning or DQ ends or deep
Q networks and to start the
prerequisites
if you don't know deep learning stop
pump the brakes
you got to learn deep learning so if you
want you can go to the homepage of
Python programming dinette click on the
machine learning here and then do this
deep learning basics with Python x flow
and chaos at least do like the first two
the first two tutorials I want to say
yeah getting it loading in your data
probably the first three especially
since we're using confidence do the
first three and then come back to this
cuz otherwise you're gonna be so lost so
okay deep cube learning basically it got
its start with this the following paper
this human level control through deep
reinforcement learning if you've ever
looked up reinforcement learning first
you found that all the tutorials suck
and hopefully not this one and then and
then you've seen the following image so
this is how the deep deep query network
or deep Q network is gonna learn so
what's gonna happen is you've got the
this input in this case it's an image
you've got some convolutional layers
they don't have to be convolutional
layers some fully connected layers and
you don't have to have those but you've
got some sort of deep neural network and
then you've got your output layer and
your output layer is going to map
directly to various actions that you
could take and it's gonna do so with a
linear output so it's a regression model
with many outputs not just one now some
people did try to do just one output per
se so it basically to have a model per
possible action but it doesn't really
work well and that's super
gonna take a really long time to train
so anyways here's another example oh
it's beautiful really beautiful you got
input values so again it doesn't have to
be an image you know this could be Delta
X Delta Y for the food Delta X Delta Y
for the enemy for example that could be
your input boom then you've got your
hidden layers again they could be
convolution they could be dense they
could be
current whatever you want there and then
here you've got your output again with
the linear activation so it's just gonna
output these scalar values well these
are your Q values they map directly to
the various actions you could take so in
this case let's say this was your output
you would take the Arg max well the bard
map or the max value here is 9.5 and the
values are you know you're basically if
we were to map this and get the index
values it would be 0 1 2 3 so Arg max
would be 1 so whatever action 1 is that
would be the move that we would make ok
so we've replaced this cue table with a
deep neural network the benefit here is
we can learn way more complex
environments first of all we can learn
more complex environments just because a
deep neural network is capable of
actually mapping that also a deep neural
network can take actions that it's never
seen before so with Q learning if a
certain scenario presented itself and it
was outside of any of the discrete
combinations that we've ever seen well
it's gonna take a random action so it
got initialized randomly a deep neural
network on the other hand is not it can
actually recognize things that are
similar but it's never seen this thing
before
and it can act accordingly so first of
all a deep neural network is going to do
better in that case it so in that way it
can solve for way more complex
environments but also as we saw as you
just even barelly increase that that
discrete size of your Q table the amount
of memory that's required to maintain
that cute a fool just just explodes
right and it both in in terms of your
observation space size or your discrete
observation space size but also in your
actions so in our case up to this point
we've only taken four I'd like to
introduce a few more actions moving
forward you know so what we're gonna do
is we're gonna keep the diagonal moves
but also allow for just straight
Cardinal like up-down left-right as well
as don't move so so we're gonna
introduce all of those as well anyway so
so for those two reasons
networks are way better the downside is
normal networks are kind of finicky so
we've got a we're gonna have to handle
for a lot of things that are finicky
about neural networks also it's gonna
take a lot longer to train so on an
identical model or an identical
environment like in the blob and where
it took our cue table minutes to fully
populate basically it's just a
brute-force operation basically so if
it's small enough you know your CPU can
handle it where it took you know minutes
for a cue table it's gonna take hours
for cue learning but the benefit is
where it takes I'm sorry it's gonna take
hours for deep cue learning but the
benefit here is for certain environments
where it takes yes a long time like
weeks for deep cue learning to learn an
environment it would require you know
petabytes of memory for a cute able to
figure it out okay so so that's that
there's your difference so really
they're gonna solve different types of
environments and honestly q-learning is
pretty much useless you can use it for
cool novel little niche things for sure
but you're not gonna find too much use
for it in the real world I don't think
we're as deep q-learning you can
actually start to apply it to really
cool things so anyways enough
jibber-jabber the last concept I want to
do before we get into a actually writing
code is right here this learned value
change so before the new cue function
basically was this whole thing where as
the neural network kind of solves for
this like learning rate and all that and
just updating values this is solved
through back propagation and fancy
things but we still want to retain the
future values because normal networks
don't care they don't give a heck about
future they care about right now what is
this thing what is this exact value they
don't care about well what does this
chain of events do other than a
recurrent neural network I suppose but
really a recurrent neural network cares
about the history not the future so
anyways in this case we still want to
retain this so we are still going to use
this and basically the way we're going
to do this is every step this agent
takes we still need update Q value so
what we're going to do is
query for the q-value we take that
action or a random one depending on
epsilon then we know we re sample the
environment figure out what our you know
what the next reward would be your and
then we can calculate this new Q value
and then do a fit operation so people
who are familiar with the neural
networks are already like wow that's a
lot of fits yep sure is also that's one
fit at a time so as you're gonna see
when we go to write this code we
actually have have to handle for that as
well because that would make for a very
unstable neural network so there's two
ways that we're handling for that but
with that I think we're just gonna jump
in the code because I think it'll make
more sense
use code coding it and covering it as we
get to those points okay so hopefully
that was enough information like I said
all the tutorials have ever seen on deep
hue learning have been terrible and
there's like so much information that's
left out that you know to get the
overarching concept honestly this
picture is enough but then when it comes
time to actually look at code and how it
really will work like can you sit down
and code it after you read someone's
tutorial my hope is you really can after
this one but otherwise I don't think a
tutorial exists for doing it so so
anyways let's get started so the first
thing we're going to do is I'm going to
create we're gonna at least hopefully
code our agent at least the model and
talk about some of the parameters and
then probably in the next tutorial we'll
do fitment and or training basically and
all that so anyway class dqn agents and
this agents let's just do define create
model first this should be fairly basic
it's just going to be a continent we
don't have to do a continent I'm just
gonna do one just so you can more easily
translate this to something else so the
first thing you should do at any time
you learn something is go try to apply
it to something else so try it and then
you'll mediate because you it might make
sense to you as you're listening to me
talk about it but then you go to try to
apply it and then suddenly it doesn't
work or you're confused or you realize
oh I don't really actually know how to
do this so anyway the first thing you
should do try to
apply it someone complained recently
about my drinking on camera saying they
don't want to hear me gulp I'm drinking
because my mouth is dry
and the most annoying thing ever is
listening to someone talk with a dry
mouth so you're welcome
jerk so anyway create model okay so
we'll start with model equals a
sequential model and now let's go ahead
and make some imports so the first thing
that I need to bring in is from careless
models let's import import sequential
and then we're gonna say from Charis
layers let's import dense drop out conv
to d-max
pool to D activation activation flatten
I think that's everything
sorry that ran over my face my bad
anyway there we go
dense dropout comp doody doody max pull
to D and actually that's max pooling to
D activation and flatten okay those are
all of the imports also that you can
always go to the text-based version as
you toriel's if code runs over my ugly
mug you can check those at office all
actually I don't even know maybe at the
very end it'll be there anyway by the
time you need it it'll be there anyways
okay so we've got that stuff the other
thing I'll go ahead and import to is
from Kaos duck call back so let's import
tensor board we need other things but I
want to cover them when we get there so
model is equal to a sequential model
then we're going to say model dot add
and then we're gonna start adding conv
to capital d let me fix that as well
cool calm 2d which will give us 256
convolutions and the convolution window
will be a 3x3 and then input underscore
shape and this is going to be equal to
it will say n dot observation space
values and then close this off and this
isn't quite yet exist we have to create
the environment so a little uncertain if
I'm going to actually rewrite the
environment so we it's converted to
object-oriented programming now I may
not actually like we might just copy and
paste that the updated environment I'll
just talk about it because otherwise
it'll take like an hour to to go for
that so anyways but we will this will
exist at some point in the near future
then once we've done that I guess let me
zoom out a little bit since we're
running a space here the next thing that
we're gonna do is we'll say model dot
add so it's our conf layer we're gonna
add an activation and the activation
here we'll use rectified linear and then
model dot add max pooling to D then
we'll use a two by two window and again
if you don't know what max pooling is or
convolutions
check out that basics tutorial because I
cover it and I also have beautifully
drawn photos if you really like my other
photo you'll love that those photos then
after the max polling we're just a model
that and and we're gonna add a drop out
layer and we'll drop out 20% and then
we're just gonna do the same thing again
so this will be two by 256 so copy-pasta
we don't need to include the environment
shape again or the input shape rather
this
so two by two fifty-six then we're gonna
say model dot add we're gonna do a
flattened here so we can pass it through
dense layers and then we'll say model
dot ad we'll throw in a dense sixty-four
layer and then finally model dot add a
dense layer and it'll be m dot action
space size and then the activation
activation will be linear and then model
dot compile and we'll say loss is MSE
for mean squared error optimizer will be
the atom optimizer with a learning rate
of 0.001 and then metrics we will track
accuracy okay so that is our model again
sample code is in there will be a link
in the description to the sample code so
if you've missed anything you can check
that out
okay so and then atom we don't actually
have atom and ported so let's go ahead
and grab that as well so from Charis
dots optimizers import atom
just for the record to if anybody's
watching this in the future this is
still tensorflow like 1.15 I don't
really know actually version I'm on but
it's not 10 serve low to yet so keep
that in mind so some things might change
by then and if it has changed check the
comments and then when it finally
actually does truly matter what version
I'm tensorflow I'm on I will let you
guys know it's also kind of expected
that you guys will know how to install
tensorflow and Karos I'm not gonna cover
that again for that go to the basics
tutorial
so anyway go model compiled okay then we
will return that model okay
so that's our model then define in it so
and then now we're gonna do the init
method for this model so we're gonna say
self dot model equals create underscore
model and then so that is going what is
your problem why are you unhappy what
what the heck am I missing Oh
self dot Karima more coffee is
definitely necessary okay self dot model
is create model so that is that's gonna
be our main model but then we need
what's going to be called our target
model
what the heck's at Dec so let's write
that first so self dot will call this
target model equals self dot create
model but then we're gonna say self dot
target model dot set underscore weights
and we want to set the weights to be
exactly the same as self dot model get
underscore weights okay so what's going
on here so we are gonna wind up having
two models here and the reason why we
want to do that is kind of there's
there's a few reasons but mostly it's
because this model is gonna be going
crazy so first of all we initial the
model itself is initialized randomly as
all neural networks are but we also are
gonna initialize with an epsilon likely
of up one so though the agent is also
going to be taking random actions
meaningless so initially this model is
going to be trying to fit to a whole
bunch of random and that's gonna be
useless and but eventually as as its
explored as it's gotten rewards it's
going to start to hopefully figure
something out but the problem is we're
doing a dot predict for every single
step this agent takes
and what we want to have is some sort of
consistency in those dot predicts that
were doing because besides doing a dot
predict every single step we're also
doing a dot fit every single step so
this model especially initially is just
going to be like all over the place as
its attempting to figure things out
randomly so the way we're going to
compensate for that is we're gonna have
two models so we've got self dot model
this is the model that we're not fitting
every step and then we have self dot
target model this model we're this is
the one that we're doing a dot predict
every step and I would even say this is
this is what we not predict against
every step and then this is the one that
gets you know gets trained every step
make note of that because you'll
probably forget so so so then what
happens is after every some number of
steps or episodes or whatever you'll
re-up date your target model so you'll
just set the weights to be equal to to
model again so this just keeps a little
bit of sanity in your predictions right
so in the predictions that you're making
this is how you'll have a little bit of
stability and consistency so your model
can can actually learn something because
it's just gonna be there's so much
randomness it's gonna be very
challenging initially and so long as
you're doing things randomly okay so
that's one way we are handling for the
chaos that is about to ensue next we're
going to have self dot replay underscore
memory and that is going to be a dequeue
or day q I don't I always forget how to
pronounce that to use that we're gonna
say from collection collections import
dequeue and if you don't know a dik-dik
you is it is a set length think of it as
ikat an array or a list that you can say
I want this list to be
sighs so we're gonna say max Len equals
replay memory size and let's go ahead
and just set that real quick we'll just
say boom and we'll set this to 50,000
also a cool little trick I recently
learned was you can use underscores in
place of commas so so Python sees this
as 50,000 but it's a little more human
readable like this number is a little
harder but then especially like once you
have like a number like this the
underscore suddenly becomes very useful
so cool
so don't do a plus though underscore
cool so that's our replay memory size so
what the heck is this
so the replay memory so we talked about
how we keep the predictions so how we
keep the agent taking consistent ish so
at least the agent is taking steps
consistent with with the model over time
okay try to think of a great way to word
that but so we've got the prediction
consistently consistency sort of under
control it's still gonna be crazy it's
at the start period but this is that
we've got that settled now we need to
handle for the fitment craziness that is
going to ensue because because we saw we
have you know we've got two models so
we've slowed down one one way of working
with that model but the other ones still
gonna be crazy so self dot model it's
still gonna be dot fit every single step
but not only is it getting a dot fit
every single step it's getting a dot fit
of one value every single step so again
assuming you're not new to normal
networks you know that we trained neural
networks with a batch and and part of
the reason why we do that is one it's
quicker but it also tends to have better
results
not necessarily the higher the batch you
use at some point the batch size gets
too big but to have some sort of batch
size used it generally will result in a
more stable model and just learning over
time so if you just fit with one thing
it's gonna adjust all those weights in
accordance to that one thing and it's
gonna get to the new thing do another
fit and it's going to adjust all the
weights in accordance to that one thing
as opposed to if you throw in a batch of
60
four things it's gonna adjust all the
weights in accordance to all 64 things
so it's not going to overfit to one
sample and then go to the next sample
over fit to that sample then go to
another set you know I'm Sam so so we
want a handle for that too and the way
that we do that is we have this replay
memory so we take this replay memory and
that can be let's say 50,000 steps okay
and then we take a random sampling of
those 50,000 steps and that's the batch
that we're gonna feed off of so train
the neural network off of so that's how
we're getting stability in both the
training Network that's getting updated
every single step but hopefully we're
smoothing that out a little bit and then
we're smoothing out the predictions
because we're also not allowing the
prediction model to be updated every
step instead it's being updated let's
say every five episodes or whatever we
intend to go with they these are all
hyper parameters or really constants
that we're gonna set at the top of our
program
okay replay memory okay so the next
thing mmm is gonna be self dot tensor
tensor board and that's going to be
equal to modified tensor board yeah
we're gonna set log underscore dirt to
be logs /var - var and we're gonna make
this an F string and we're going to
import import time if you're want to say
3.5 or younger Python you can't do F
strings you'll have to do a dot format
or whatever and so for the logs we're
gonna say model underscore name and then
we're gonna say int time dot time this
is just so we can kind of keep track of
what's what and then the next thing I
want to do I thought it was gonna piss
you at me for this anyway let's go ahead
and set model name before I forget and
for now just it's whatever you want I'm
gonna say it 256 by - you might try a 32
by two and a 64 by 2 it anyways I'm by
256 by - I'm een the neural network it's
a 256 by 2 content
so we've got tensor board done obviously
this doesn't yet exist I'll talk about
that in a moment and then finally self
dot target update counter equals zero so
we're gonna use this to track internally
when we are ready to update that target
model so this you know this thing here
that we've explained why we want to do
that so the next thing we want is I'm
not gonna write out that so go to the
text-based version of 0 and we're gonna
grab this updated this 10th this updated
tensor board class did we already pull I
think we already pulled in tensor board
but what I really want is this right
here this whole class I'm like oops I
want that comment too
I'm gonna copy that I don't really see
any benefit for us writing that but I'll
explain why we want that here in a
moment paste that in cool so what this
is doing is it's just modifying the
tensor board functionality from tensor
flow and Karos
so by default Karis wants to update this
this tensor board file this log file
every time we do a dot fit well we're
doing a dot fit uh you know up you know
about 200 times in episode and then
we're gonna do like 25 thousand episodes
so first of all I'm sorry I think I use
the word update log file it wants to
create a new log file per dot fit well
we don't want that we want just one log
file so so having this is gonna solve
quite a few things as was written by
Daniel and though basically one the i/o
was taking a lot of time so just simply
writing the new file was sometimes
taking longer than a dot fit but also
we've got all these extra files each one
was 200 kilobytes not necessary and yeah
so just a bunch of it was just not
constructive to do this I guess that's
the behavior because most neural
networks are just gonna fit once on
sort of cycle and it's not a big deal
but for deep queue learning where you're
fitting just constantly if something
needed to be done okay so that is our
modified tensor board now what we're
gonna do is go back to our DQ n agent so
that's our init method let me just kind
of clean this up a little bit I want
these kind of separated out I don't want
that separated out I don't know I can't
really do that what I want so I'll just
leave that so
then create model let's go ahead let me
see if I go hopefully well we're not
gonna finish this but let's add in one
more method so we're gonna say define
and in this one we're gonna say update
underscore replay underscore memory self
and then transition train transition can
we type so then we're gonna say self dot
replay underscore memory dot append
Trant
Wow I really want to call it transition
transition so transition is observation
is just going to be your observation
space action reward new observation
space and then whether or not it was
done so we need to do that so we can do
that new cue formula so that's that's
all we're doing there you'll hopefully
that'll probably make more sense when we
actually get to actually use self dot
update replay memory or agent update
replay memory finally yeah we'll do
we'll do this one last method and then
we'll save train for the next next
tutorial so that's gonna take a while so
define get cues so we will get Q self
terminal terminal state and then step
and then so this will be returned a self
dot model underscore predict and we're
gonna say the numpy array I think we've
imported numpy we'll grab that soon
of the state dot reshape negative one
and then we're gonna say asterisk state
dot shape all that's doing is unpacking
state and oh did I
I sure did terminal state that's for
train right now we're just passing state
here so state shape and and then we're
gonna do div whoops div by 255 and then
zero so model dapper ticket
always returns a list even though in
this case we're really only predicting
against one element it's still going to
be a list of one element urn array of
one element so we still want the zero
with element there div by 255 is because
we're passing in this RGB image data so
you can very easily normalize that data
by just simply dividing by 255 okay so
that's it let's go ahead and go up to
the very top here let me import numpy as
NP so let's say import numpy as in b let
me go ahead and save that and I think
I'm gonna save train for the next
tutorial
because that's gonna take a while so
quick shoutout to the recent we're not
recent actually these are my long-term
channel members people who have been
sticking around for a while
reginald roberts seven months luiz
fernando seven months stefan our scoot
Harwood eight months ampere hammer eight
months ever William Sam's nine months
and abbaji ten months that's quite a
while you guys thank you guys all for
your support and if you guys have any
questions comments concerns whatever
feel free to leave them below if
anything up to this point is confusing
or you want to get a little more
information again comments below or come
join us in discord GG's slash syntax if
you think I've done something wrong or
said something wrong I'm sure I have
feel free to let me know and yeah I am
hoping that you guys can get through
this tutorial series here and actually
be able to use this stuff I think right
now I just like I said I could not find
a tutorial or a information to actually
do this kind of stuff that actually made
sense or actually explained everything
so just a lot of copying and pasting
going on so my hope is that you guys can
actually get through this and know how
to do this yourself so that's my goal so
if you're having trouble getting to that
point let me know because I'll I need to
fix that so anyways I will see you guys
in the next video where we will do the
Train method and then we might only do
the Train method but hopefully we can do
the environment I'm thinking I might
just copy the environment we like we we
made some tweaks to the environment and
we made that object-oriented and so we
mmmm I might just like copy that because
the the environment honestly has not
changed we just made an object-oriented
just to make it easier so you know you
know there's like little tweaks made I
think I'm just going to talk about the
tweaks that were made and that'd be it
that way hopefully in maybe in the next
tutorial we can actually do the Train
method here talk about that deep Q
learning stuff and hopefully get a model
training by the next video so I'll think
about that
you can leave your opinion below but
probably by the time you've seen this I
will have already made my decision so
anyway yeah till next time everybody