what's going on everybody and welcome to
self-driving cars with Python Karla and
hopefully some reinforcement learning
we'll see where we left off we actually
built the environment code that we're
going to use sort of the environment
layer on top of the karla server client
code basically and so this is this is
where this is basically what translates
information to our agent which is what's
going to train our actual model now what
we want to do today is actually code
that agent but doing that brings up a
different problem that we have and that
is we want to be able to predict and
train at the same time right that's
pretty inherent to reinforcement
learning the problem is this is a
relatively complex environment that
requires a decent amount of processing
both just to display the environment but
then the actual model that we're gonna
wind up with is taking in a large amount
of data because it's taking in this
image data here so the models gonna be
large ok there's a lot of weights going
on in a lot of calculations so so we've
got that as a problem the other thing is
we want to do we really want to do this
in real time we could do this in like a
synchronous mode with Karla so it's not
running full like a full simulation
speed right it's running and waiting for
everyone to get their update and you
could say I want this to run exactly at
60 frames per second or 42 or whatever
the heck you wanted not good so so
instead what we have is this this
challenge where we want to train and
predict at the same time on something
that isn't gonna be super fast but yet
at the same time we want this to happen
at I mean ideally 90 frames a second or
60 frames a second but we're probably
not gonna see anything even remotely
close to that but we want as good as we
can get so we're going to end up using
threads so and basically that way we can
train and predict at the same time and
hopefully not have so much delay in
between doing these things so anyway
that's what we're going to be doing
today
or at least starting to do so what I'm
gonna do is I'm just gonna come down
here and then we're gonna begin some new
codes so whereas a class D Q and agent
and let me just bring this over so if
you haven't done that reinforcement
learning series I strongly recommend you
pause now and do that series if you go
to Python program init go to machine
learning then click on reinforcement
learning and then I would honestly just
do the whole thing it's not really a
long series there's only six parts but
definitely you want to do deep Q
learning and get an idea for how that
works because we're gonna be using very
very everything's gonna be very similar
just as it's been even up to this point
so definitely check that out
because you're gonna be pretty confused
otherwise but anyway here we go so
define and we're gonna have our init
method self and we're gonna start off by
saying self dot model equals self dot
create model so we're obviously gonna
have a method that creates model for now
we're not gonna worry about that
then we're gonna say self dot target
underscore model and that's gonna be the
exact same thing self dot create model
then we're gonna say self dot target
model dot set weights and we're gonna
set the weights to be self dot model dot
get weights so again if these is that
this line here is confusing to you go
back to that reinforcement learning
series okay so we want this every now
and then basically the target model is
going to update to the models so
basically with reinforcement learning as
a reminder even if you have seen it you
have two models right one model is the
one that's actually being constantly
trained the other one's the one that we
predict against we want the one that we
predict against to be relatively stable
if we're always updating that model
we're gonna get really volatile results
this can be very hard for this model to
actually get like coherent results so
instead what we do is we kind of hold
that model steady to predict against and
then we're constantly training the other
model and then after n number of
episodes or even steps or whatever you
want to go with usually its episodes
though we're going to up
date the the what the other model the
one that we're predicting against so
anyway cool target model pop up a blah
okay cool
so now we're gonna say is self dot
replay underscore memory is equal to de
Q or de Q I don't really know and then
we're gonna say max land equals and then
this is replay memory size so so again
if you don't know what that is
go back to that course but we want
basically this is the memory of previous
actions and again we pick a random set
of actions again to help with just crazy
volatility and actually keeping things
relatively sane or at least attempting
to so let's go ahead and make these
these two things since we're using them
so let's go to the top let's say from
collections import de Q nice got it and
then we're gonna say replay memory size
we're gonna set that to be 5,000 I know
I've said it before but yeah the the
underscore is is like a comma right so
that means 5,000 and it's super useful
like if you wanted to do five million
right that that's so much easier to read
then that right I can't just quickly
glance it and be like oh that's five
million or even in fee especially once
you start getting into more astronomical
numbers the underscore is super helpful
okay replay memory size that's fine
let's go ahead and make a few more of
these constants while we're here we're
gonna need replay memory size we're also
going to have we can set min replay
memory size we'll set that to be a
thousand we're gonna set mini batch size
will say that sixteen we're gonna say
prediction batch size we'll set that to
be one training batch size will be mini
batch size divided by four no remainder
that's what the double div is they're
updates
Update target every so this is how many
basically at the end of how many
episodes so basically every five
episodes will update that target model
we're going to be and we're gonna end up
using model name at some point for now
I'm gonna call this exception so that's
the model that I'm going to use you can
feel free to change it you can make your
own custom model use a different one I
don't really care and I'll throw in I'll
throw in these other things to memory
fraction equals and I'm gonna say 0.8
this is how much of your GPU are you
gonna want to use so I'm using an RT ex
Titan card in for some reason that card
and I apparently all the RT X cards have
this weird issue where they attempt to
allocate more memory than they have and
the only way for me that I've found to
overcome this is to use a memory
fraction that is less than you know 1.0
basically so don't let the card attempt
to allocate all the memory I don't know
why that is I don't know if that's
tensorflow specifically fault or is that
CUDA toolkit or kudi nn I don't know
whose fault that is but it's just a
thing that's happening right now so I
have to do this prime most people maybe
don't even have to do that at all I
don't really know so let's see so the
other things we can say min reward is
negative 202 for the model saving um
actually we might not even use min
reward I don't know anyway I think
that's the aunt that's all the stuff I'm
going to use for now
we could also do our discount epsilon
and all that stuff as well so let's go
and do that too so we're gonna set
discount discount we'll set that to 0.99
then we're gonna set let's do F assad's
how so how many episodes how many
episodes do we want to do a hundred is
way too few but I'm just gonna set it to
something reasonable for now so we've
got discount so then EPS not camps
epsilonc this is gonna change hopefully
epsilon underscore decay 0.95 just to
see it decay probably later on we're
gonna wind up with something more like
0.99 75 or nine nine nine seven five you
know something like that honestly to
check that I usually determine how many
episodes do I really want to go and then
or actually how many's yeah it depends
either episodes are steps depending on
when you're gonna decide to decay then I
just write a for loop to determine
there's room in the number there's
there's got to be like some sort of
function out there that you could just
say hey I want to do this many steps may
give me a reasonable decayed number but
for now we'll go with that and then
we're gonna say min min at sill epsilon
equals this so a tenth of a percent
then we'll have aggregate stats every
will say ten episodes and then we'll
have actually do we yeah we've already
got show preview okay so now that we've
added you know six thousand lines of
constants and starting variables let's
go back to our actual agent code so
replay memory cool the next thing we're
going to do is self dot self dot tensor
board and that's going to be a modified
tensor board I believe oh I closed it I
believe that
the this will Kyle put this in the code
as well logged or equals I think this is
I think I use this as well in the Q
learning stuff so if you've done that
series you should have this code
somewhere but I'll put it in the
text-based version two for this modified
tensor board function at some point I'll
show where to get it and until I have it
paste it somewhere reasonable I'm not
going to show it so right now I'm not
going to add this code but anyway it
just modifies tensor board to be a
little more reasonable for the task of
I'm confident it's for Q learning
because the problem is we don't need as
many updates to tensor board as tensor
board wants to do so
yeah cuz it's gonna basically want to
create a custom file per trained session
or trained loop I want to say or
something or is it it might be a perfect
fit I think is the problem I can't I
can't remember it's been too long
but anyway regardless we want to modify
it so it stops doing that nonsense and
both for speed purposes and storage
purposes anyway we're gonna at least
specify the log door and it'll be model
it's an F string in case you didn't see
that I threw that F there model name and
then - whoops - and then the int oops
int value of time time did we import
time we did great okay so that specifies
our tensor board then we're gonna say
self dot target underscore update
underscore counter we're gonna say
that's equal to zero for now and then
that basically gets updated every end of
episode then we're gonna say self dot
graph equals TF get defaults graph and
we're gonna do this in a separate thread
again we have to well we don't have to
use threading but we want to use
threading and this is we need to specify
this because we want to use this in a
separate thread and we want to do that
because we want to predict and we want
to train in different threads
self diagraph okay so now we want to do
is self dot terminate equals false this
is one of the flags that we're gonna use
for our threads and then we're gonna say
self dot last logged episode equals zero
this is just to help us keep track of
tensor board as we're doing things
relatively asynchronously and then self
dot training underscore initialized
equals false again this will be another
flag we just because we're going to be
doing stuff in separate threads we want
to have these flags to sort of keep
track of what we're doing so this will
be when we start running simulations so
now what I want to do is create or
actual model so define create model self
and in this case we could do a bunch of
things right you could you could begin
by saying model equals you know and do
the sequential model stuff and then
model dot add and then you start adding
convolutional layers or dense layers or
recurrent layers or whatever that you
want to do but what I'd like to do is
stray use exceptions so I'm going to say
base model equals at X ception we need
to import exception but for now I'm
gonna say weights is none we don't want
to start with any of their weights we're
gonna say include top false input shape
so top is that first layer since we're
going to be using a different layer we
don't want to include that top layer the
input shape will be our height by width
so input shape will be image height
image width and then it's by three and
then before I forget let me go to the
top and we want to actually import
exceptions so let's just do that I don't
really know if I want this at the end of
all my imports or two separate Karla
from chaos I think I'll do it at the end
to be honest
from Charis dot applications dot
exception we want to import exception
done easy so okay so anytime you take
one of these models like these pre-built
models from tensor flow what we need to
do is usually change the input layer as
well as change the output layer so the
way that we're going to do that is we
want to specify what do we want to use
as input what do we want to use his
output basically so pretty much this is
input and then what we want to change is
we're gonna say x equals base model and
we're going to say dot output then we're
gonna say x equals global average
pooling 2d times X we need to import
global average pooling we'll do that in
a moment and then what we're gonna say
is predictions equals and it's a dense
layer again we'll need to import that as
well it's three neurons activation
activation will be linear and that's
multiplied by X this is three because we
have three options right we can go left
straight basically we can turn left
don't turn it also go straight turn
right or actually it's not don't turn at
all its straight go straight so okay so
those are our possible predictions then
what we're going to say is the actual
model that we want to use is model which
again will also have to import where to
say inputs is equal to base model dot
input and the outputs will be equal to
predictions so basically it could have
been base model dot output but we we
want to change a few things we want to
make it a dense layer we could have made
it pretty much just a dense layer but we
wanted to add the global average pooling
here okay predictions done model dot
compile and whoops model dot compile
loss is going to be MSE for mean squared
error the optimizer we're going to use
is going to be the atom optimizer again
left import that learning rate 0.001 and
the metrics we wish to track accuracy
okay
return model and we're good so we need
to make all these imports so we'll go up
to top will say from Charis daunt layers
we're going to import dense global
average pooling in and then from Kara
step up to misers
I think import atom let me try cumin
oops make sure that's correct
optimizers yes and then the other thing
we want to grab is from Charis top
models we want to import model and I
think that's everything again we're
gonna find out if we're missing imports
and stuff that's not the most important
or most challenging thing to figure out
but I think that's good let me just look
real quick here let me fix this this
should also alert me if I'm missing
anything looks like we're good so okay
that's done now we're gonna do defined
we're gonna update replay memory self
and then with any transition and I'll
explain that in a moment so a transition
is going to basically contain all of the
information that we need to train a to
train the model basically so or
specifically a deep Q learning model so
that is going to be the current let me
just say this let me just throw in
transition equals and it's going to be
it's like a list right well it's
actually we'll call it a tuple because
it's not going to change you're going to
have
of current underscore state the reward
nope
so why have notes action reward new
state and then whether or not we're done
so this has all the information that we
need when we're going to actually like
train models and stuff like that and
update these the update the q-value so
to speak so so that's what the
transition is and that'll make we'll
need to know that even more so in a
moment but for now all we're gonna say
self dot replay underscore memory dot
append transition so we don't really
need to know what a transition is here
but very soon we will because define
train self and this is where basically
the magic happens so the first thing we
need to do is make sure we have enough
replay memory to actually train if we
don't have enough replay memory one
epsilon is likely very high we're just
doing random things anyways it's not
essential that we train on this but we
want to have enough replay memory again
so we have enough information for this
model to actually start learning
anything useful at all so the first
thing we're gonna ask is if the Len of
self dot replay memory is less than the
minimum replay memory size then we're
just going to return we don't really
want to do anything otherwise we want to
Train and the first thing we want to do
is grab our mini batch to Train on so
we're gonna say random dot sample of
self dot replay memory and the sample
size that we want is mini batch size so
that gets us our random mini batch now
all we need to do is calculate or
current States and then the future
States so that we can actually do the
training operation so current nut caps
current States uh equals an umpire array
of and we're going to say transition 0
with again that's our current state
right the zeroeth element and when we
get our future that'll be zero one two
three so transition three when we
try to get our future States so
transition
zero for transition in whatever that
mini-batch was now hmm that state that's
your observation
what's our observation it's an image
what do we want to do every time we use
images with neural networks we want to
divide by 255 so again this is just our
way of scaling the information before we
were doing the scale operation in
process image but now we're gonna add it
here so current states cool so then what
we're gonna say is with lips with self
dot graph as defaults we are going to
specify our current Q's list is self dot
model pay attention to model versus
target model self dot model dot predict
and then we want to predict based on the
current states and the batch size that
we want to use is whatever a prediction
batch size that we set I think we
actually have that set to 1 right now
yeah but later we could we could try
something bigger at least for me
personally this is a super heavy GPU CPU
and RAM task it is all of the things so
it's very challenging to run so anyway
current queues list let's go ahead and
spell that correctly while we're at it
okay so now that we've done that we want
to do this future future queues so
future queues is going to be very
similar to this so in fact I'm gonna
copy paste this could bite me in the
butt
so I guess new current states that will
be transition 3 for transition in mini
batch with self dot graph as default
that's fine change this to be future
Hugh's list not self dot model it's self
dot target under
score model and it's not current States
its new current states I think that's
good let me know if you spot an issue
capital X equals empty list lower case y
equals also empty list now what we want
to do is we're going to enumerate over
all of our batches and do our training
and basically we want to make sure we
only do our training when basically well
we need to create the Q values and we
only want to do that if we didn't
because there can be no future so we
only want to update the Q's when we have
a future state so if it's a terminal
state so ie we crashed then we actually
set the Q to be whatever our reward was
just now but if we didn't crash we
actually want to set the Q based on that
Q formula so what we're gonna say here
is for throw for index comma and then
again this will be that entire tuple
this right here that the entire
transition so for index all this stuff
in and numerate whatever mini-batch is
oh don't forget our colon here okay for
that then what we want to say we want to
make sure we're not actually done so if
not done max underscore future q equals
NP dot max future qs list for the
specific index index nu q equals reward
+ discount x max future q if we are done
though the new Q is simply equal to that
of reward again this concept has been
explained already in the reinforcement
learning series but we all including
myself sometimes new refreshers so the
next thing is
current Q's equals current Q's list for
this index and then we're gonna say
current Q's for that action that we took
new Q so this is how we're gonna update
that actual Q value and then once we've
done that X dot append current States y
dot append current Q's okay so for OH
okay this needs to be tabbed over so
that was almost a a me and tears moment
in the future that's going wrong okay so
all right now that we've done that we're
going to throw in just some code for the
tensor board so log this step equals
false so we want to make sure we're
logging only per episode if self dot
tensor board dot step is greater than
self dot last logged episode long this
step we'll set that to equal to true
self self dot last log episode equals
self dot tensor board dot step okay so
now that we've updated these values and
we've we've made like X&Y and all that
stuff what we need to do is actually fit
the x and y new current stays 255
so when we go and do that so let me pull
down to here current state I just have
to check
okay so current state sorry I just want
to make a Harper sent so current state
is always going to be here so it has not
been divided by 255 yeah I had to make
sure so what we want to do now is we're
gonna say with self dot graph dot as
[Music]
defaults we are going to say self dot
model dot fits the numpy array of x div
255 NPA of y the batch size will be
training batch size we're going to set
verbose to 0
shuffle to false would be unnecessary
callbacks we're going to set to be self
dot tensor board if log this if log two
step else none so callbacks will be self
dot tensor board if we're trying to log
that step otherwise no callback
okay so now we want to do is update that
log this step stuff so if long this step
self dot target update counter plus
equals one then if self dot target
update counter is greater greater than
update target every what we want to say
is self dot tark target underscore model
dot set weights equals whoops actually
dot set weights to self dot model dot
get weights so again it just becomes a
copy of that initial model or the the
actual model then self dot target update
counter equals zero cool okay so that's
our train method hopefully that's good I
suspect we're gonna have a lot of issues
but that's okay let me decide I think
we'll finish we can finish the the agent
code here we we don't have that much
more to do so
so first of all define get Q's excuse
for self in state so we want to pass a
state get Q values so that's just gonna
be returned self dot model dot predicts
numpy array of whatever state happens to
be passed and then we're gonna do a
reshape negative one and then state dot
shape div 255 and we are expecting that
to be a single thing so again as always
prediction predicts you pass a list you
get a list back in this case we're when
we get Q's we're just getting one at a
time so we're gonna grab the zero with
element state dot shape this will just
be our image shape this has a little bit
of
sprinkled into it and I I guess he likes
making ugly code I hope you're watching
so what we're gonna do is define train
in loop thanks to annular doing great
self and so in case you don't know what
this means
up here this is just unpacking the shape
of state which is our observation which
is height with three so four eighty six
six four this is yours going on too long
anyway it's the height width times three
or by three so anyway it's just
unpacking it's a nice succinct waited to
coat it in there but a quick glance
doesn't I yeah
anyway here Jamie on training in loop
basically so far we actually haven't
seen this threading that I was referring
to that we're gonna wind up needing to
do but this will be the first time we
really see what's what truly is going on
here but recall we want to train and
predict in different threads because we
don't want training to slow down
predictions we want to be able to make
predictions as quickly as possible and
we want to train also as quickly as
possible so the way that we're gonna do
that is a couple of ways first of all
the initially for whatever reason
tensorflow
again I don't know if it's tensor flows
fault is it and videos fault is it Cudas
fault I don't know but the very first
time you actually start training or the
first time you try to do a prediction
it's really slow compared to subsequent
predictions that's why if you want to
put something in the cloud for example
that actually is like some sort of API
that does predictions you want to make
sure you're predicting in some sort of
infinite loop you would never want to
like reinitialize everything all over
against or not just initializing and
you're not just like loading in the
weights for some reason you're also
doing some other thing that I don't know
what it is but that first prediction is
slow so so what we're gonna say is we're
just going to randomly do
a fitment and then what we're going to
do from there is do sorry I said
prediction but I meant it as well it's
it's prediction or train it's just that
first thing takes forever anyway the
first thing we're gonna do is we're just
going to do a quick fitment on nothing
and then we're gonna fit from there so
we're gonna say x equals NP random dot
uniform and we're going to size equals 1
and then we're going to do what is it m
m height M width and then 3 how come you
didn't unpack there Daniel as type as
type NP float32
then what we're going to say is y equals
NP random uniform uni uniform there we
go
size equals 1/3 and then dot as type N P
dot float32 so again we're just just
gonna make it up we just want to do a
quick fitment on something so with self
graph dot as defaults self dot model dot
fit X Y verbose Falls batch size one
great so once we've done that we will
say self dot training initialized is
equal to true and then we're going to
just initiate this while true loop that
goes on and on and on so while true if
self dot terminate terminate then we
will return and we're done otherwise
we're going to self dot train and then
we're going to throw in a quick sleep
zero point zero one and we're done
so later we'll get into training
Loup and how all that's gonna work I
think I'll save that for the next
tutorial this tutorial is quite long I
think I might have reached 40 minutes on
that one so we're done okay so if you
have questions comments concerns or the
code up to this point feel free to leave
them below you can also come chat with
us hang out in the discord discord GG /
syntax also shout out to recent channel
members who are not recent we've been
with us for many months eight months
Jean get Michael Severson and Ravi
Krishna nine months UD u de Minh Shonda
Bingley Metanoia Reginald Roberts Bill
fallens B Erica Bao tip arrow gochu Nick
se wekki if I've mispronounced your name
feel free to let me know in the comment
section and I'll try to get it right
next time anyways in the next video what
we're gonna do is I haven't fully
decided we should be able to finish all
of the code and then maybe what I'll do
is I'll show you at least where I'm at
now with this code and then maybe I'll
continue tweaking things are working on
things and kind of shared updates as we
go because I hate to be the bearer of
bad news but this is probably not good
enough for example our reward function
is probably no good we're gonna need
something a little more complicated than
what we've got we've got a lot of
variables going on here there's also
highly likely to be some sort of bug in
the code or you know some sort of
assumption that I've made that it's
incorrect there's a there's just so much
going on here that it's highly likely
this does not work out of the gate and
instead it's going to take a lot of
tweaking and coming back to it to really
figure out what's best but your feel you
can feel free to play with everything
maybe you can come up with something
better than even I can or me and Daniel
can so anyways that's it for now next
tutorial yep we'll wrap all this up and
at least show it hopefully starting to
run and hopefully train to some degree
and at least make sure that everything
is working like epsilon decay and
hopefully losses doing the thing we're
not getting like perfect accuracy or any
of the other red flags and we should
hopefully see that after training for a
few hours average reward is going up
that sort of thing we want to know that
that something is happening okay so
anyways I will see you guys in the next
video