what's going on everybody and welcome to
part 5 of the self-driving cars with
Karla tensorflow
Karros and other things basically we're
trying to do reinforcement learning here
ok that's the tutorial series where we
left off we created the car in and we
created the agent and now we're just
going to tie everything together and
actually hopefully start the training
process so with that we're gonna make a
couple of imports here real quick import
tensorflow as TF import Karros dot
back-end dot tensor flow underscore back
end as back end and then from threading
will import capital T threading with
those imports we're gonna go to the very
bottom and we're gonna start our main
dare I call it a loop but it really if
statement if name equals dunder so if
this is the script that's running we're
gonna just set will say FPS is 60 you'll
see why in a moment well it will wind up
probably this will have to be tweaked
this is a we wish kind of number we're
not gonna get 60 but anyway and then F
rewards will just prime that with a
negative 200 to begin random dot seed 1
we're gonna just set some random seeds
basically this is for repeatability
Empyrean m dot seed 1 and then the
tensorflow dot set underscore random
underscore seed so that the one as well
this is so we get as a repeatable as
possible
things so we can kind of compare in
debug so now what we want to do is most
people probably don't have to do this
but if you're running multiple things on
your GPU or you're like me and for some
reason your GPU really wants to allocate
more memory than it possibly has we're
going to do GPO underscore options also
if you run one
like multiple agents you have to do this
we're gonna say T F dot GPU options and
then we're gonna say per underscore
process I'm just or GPU underscore
memory underscore fraction and then
we're gonna say that whatever memory
fraction is so scroll up to the bottom
or the top here mine is set to eight I'm
actually gonna downgrade that to six
even for you you can probably say 1.0 or
0.9 or you know whatever especially if
you want to continue using your computer
while an agent trains you can set it
even lower than that and then you might
not even notice that you've got training
going on in the background unless you
try to play some video game or something
so anyway that's how you can tell
basically tensorflow
you know it's pretty obvious but this is
per process so basically per model each
model will in my case use sixty percent
so if you wanted to have like five
models each model would have to use less
than point to write for your memory
fraction so okay cool
coming down here that's the GPU options
now we need to set that so we're gonna
say back-end dot set underscore session
TF dot capital s session config equals
TF dot can fig per odo that's tidal
cased and then we're just gonna say GPU
options equals GPU options cool so
tensorflow now will hopefully
listen to me okay
now that we've done that we're almost
ready to start training but we need
somewhere for our models to go so we're
gonna say if not OS dot path that is is
der come on oh dear models so that path
doesn't exist
OS dot make ders models done then we're
gonna define our agent equals DQ n
agents and then we need n that's going
to be car and cool so once we have that
we're ready to
initialize the trainer source a trainer
underscore thread equals capital t
thread target will be equal to agent
trained in loop Lehmann equals true and
then trainer thread dot starts and then
Wow not Asian thought training
initialized so until that flag is is
true we're just going to time dot sleep
for a minimal amount of time basically
just wait here until we're ready to go
and then just like if you recall a
training loop here we do a model dot fit
to kind of prime the fitment so because
the first one's gonna take so long and
then we engage in our infinite loop we
kind of want to do the same thing with
the predictions so because your first
prediction will also take an
exceptionally long amount of time
so we're just going to say agent get
underscore qs n p dot ones and we're
going to say m dot M hmm
actually yeah now hold on this might be
different in my notes let me go to
tippity-top here no we do if I have well
we've got carton so m dot yeah m height
and then comma m dot M width I still see
this M versus image that has me a little
cautious at the moment I think we've got
an error there MJ image height so we've
got a couple of those image height and I
don't yeah and we don't have any
definition for those so that's that's
gonna that would have been a problem so
what we want to do is I'm gonna do ctrl
H I'm going to search for image height
and I'm going to replace that with M
height replace all
and then I'm gonna do the same thing for
width so width and that needs to be
image with M width
cool so that would have bitten me don't
worry we'll have other errors that I'll
hit I'm positive
so M height and width and then three so
again it's just gonna get some Q values
for something that is totally irrelevant
and not important in any way I totally
just dribbled you saw nothing
okay now what we want to do is basically
start iterating over however many
episodes we have so we're gonna say for
episode in TQ DM did I important to you
you know from TQ DM we're going to
import to q DM back to the bottom okay
for episode in TQ DM range one two
episodes plus one and then we're gonna
say ASCII equals true this is for our
windows brothers so it looks pretty
still and then so if you're on Linux or
something you you don't have to say
ASCII equals true and then we're gonna
say unit is episodes I don't forget what
the default unit is but at least it'll
make sense Epis so episodes okay
then what we want to do is let's say n
dots collision underscore hist
is going to be an empty list and trying
to side if I want to put this know so
this is where our collision sensor is
going to start slapping in information
the next thing we're going to say is
agent dot tensor
'don't step step he's equal to episode
so you've got epic you've got steps and
normally what would happen is every
frame would in theory be a step to
tensorflow because every every frame
will it would even be well if the batch
sign anyway it would at least every
frame be a step let me put it that way
and we don't want that that's way too
absurd so instead we're actually saying
okay every episode is a step so now
tense our board will show us basically a
plot per episode instead of intra
episode anyway can do we're gonna say
episode episode underscore reward is
equal to zero and then step equals one
let me fix that and then we're gonna say
current underscore state equals m dot
reset so that gives us our first
observation ever done will be false to
start and then Epis episode start equals
time that Tom I'm getting over a cold
and I can the brain is still a bit foggy
but okay so for every single episode we
we at least set these values we do an M
reset and now we start the actual while
true which basically is Wow not done
pretty much but just because the done
flag gets triggered we're going to do a
couple of things and then exit
immediately so we're actually just going
to like we want to dictate exactly where
we exit so anyway we're just going to
use a while true and then if done at a
specific point we break so while true if
NP dot random random is greater than
whatever the epsilon value currently is
we're going to say action equals NP dot
arrgh
axé agent get underscore cues for the
current state but if that's not the case
we are going to say action equals NP dot
random dot R and int between zero up to
three and then we're gonna say time dots
sleep 1 / FPS so so what's the deal
there so again we're not gonna actually
get 60 frames per second we're probably
gonna get like I don't know 20 to 30 on
a really nice GPU and then 4 on a bad
GPU we're really probably none you can't
run Carl on a bad GPU unfortunately but
the so when epsilon is really high
you're gonna take a lot of random
actions well taking a random action we
get to avoid the agent get queues so we
get to avoid doing a dot predict so we
get to avoid using the neural network at
all which is what's going to take time
whereas a random dot randint
this is like a nearly instant operation
it's gonna happen really quick so this
is problematic because we actually want
the agent to whether it takes a random
action or takes a model action we want
that time that it takes to be about the
same right because otherwise because
frames per second really helps the model
so the higher the frames per second the
better the model is probably going to
behave and if you take a random action
but like a split second later that is
way quicker than normal you take a model
action it's just not good we want we
definitely want the frames per second of
either one of these to be as close to as
possible so the way that you can
eventually figure out what frames per
second you're getting is at least wait
until the your memory replay is filled
up and you're actually making
predictions and then it's hat like set
your epsilon to like zero and then read
your frames per second that's how many
frames per second you'll get for
if you predict it all the time and then
you can set this value you can also
guess I mean it's probably unfair of me
to say 60 at all to even start with no
one's gonna get 60 I don't think I don't
think I would get 60 so maybe put 20 or
something like that I'm gonna put 60 Wow
I'll put I'll just stick with 20 that's
why we're not gonna get I actually might
get 20 on this machine but its profits
unlikely but anyway I'll go with that
that's fine so uh all right once we've
done that so while true we take our
action the next thing we want to do is
actually do a new state comma reward
done underscore because we don't have a
value for that
whereas a m dot step based on whatever
action we decided to take episode
underscore reward plus equals mmm
whatever the reward was then we want to
update yes I'm happy with that thank you
we want to update so we're gonna say
agent dot update underscore replay
underscore memory and we're gonna update
that with our transition basically and
that transition is current state comma
action reward new state and then done
okay so then once we've done that step
plus equals one and then finally if we
are done break cool
and then if done break and then we would
come back here Wow true and then for
actor in n dot actor underscore list
lest
list actor dot d
okay so so let me break this apart so
this is what we'll do during the episode
so we're just updating a replay memory
and then also take notes so so so here
we're we're making our actual
predictions and then we're updating the
replay memory for like the target model
and all that and then you might be
wondering well what's going on in the
target model well don't forget we are
training in loop here for that ad fold
target model so that's off doing its own
thing it's taking in information from
where well from our games where we were
actually updating the replay memory so
so basically this is doing everything at
this point that we need to do both to
play and train so predict and train at
the same time okay now we want to track
some stats but I think I'm gonna not
type that out there's one I think we've
typed it out before in two it will waste
time so we do have some stats that we
want to collect and trying to decide if
there's anything else here in the text
so I'm going to pull up our text-based
version
did I not post I must not have posted
the oneth haha I know what that is so
here I am I suppose we could take
because we need to take this and we also
need to take our tensor board script so
I'm gonna grab both of those things so
I'll put a link in the description for
both of those things and then basically
when you get there to part 5 so just in
the description there would be a link
that says like text-based version there
will be sample code and text-based
tutorial or something along those lines
the first little block is the code that
we started with but if you continue
scrolling down usually it's at the very
bottom but actually at the very bottom
I've got a play script for us so go to
the very bottom skip that first script
boom there's your second script and tell
by honest with you I'm just gonna first
will copy this bit so the actor destroy
we've already done I'm gonna come over
here pasta I'm gonna tab this over I'm a
little cautious as to why that was where
it was
that's probably why we're not actually
accepting anything at the moment so
that's why that was tabbed over so
anyway yeah let's talk about this right
before I just copy and paste and do
nothing so so but this is all code that
you've definitely seen before same thing
here because everyone watching has done
the reinforcement learning series but
basically we're just collecting
information like what was the the lowest
reward what was the highest reward and
what was the what's the average reward
so we're kind of hoping average reward
ticks up over time in this case max
rewards should also take up over time
well we got a long way to go before our
maximum reward will ever max out and
then mineral Ward this should also
hopefully go up over time so even our
worst episode improves right but
something tells me probably minimal Ward
also
so if never mineral words gonna be
really challenging because sometimes
it's just gonna have a bad spawn and
crash immediately so I don't know also
like cars can I've seen it happen where
will spawn and then like an npc if we
use NPCs drops right on top of it
there's nothing you could have done so
you'll still have to deal with that
and here again based on any metric you
want so in this case this is just as
long as the minimum like the minimum
reward is greater than some value I
actually probably would go with average
reward now but anyway you you can set
this to be whatever you want so you
could save average reward is
such-and-such and mineral ward is
greater than some other value whatever
feel free to set that however you want
it'll save the model and then before we
exit the other thing we definitely want
to do basically needed these last three
lines so so this is going to run right
for every episode that we have and then
when we are completely done we want to
clean up to some degree so I'm gonna
copy and paste in these three lines and
what's happening here so here we have
this will terminate our agent this
completes basically the threading and
then finally here
we save the model regardless of you know
whether or not we hit this so the first
time actually ran this script it
actually looked pretty good and I didn't
save it because of this this this thing
wasn't met well we're only gonna run a
hundred episodes yeah just for testing
purposes I'm gonna make sure the script
actually runs and anyway when I when I
ran it the first time I didn't save the
model and I was like dang that was
actually a pretty decent looking model
so and actually I think I saved yeah I
put both a so I'll show the tensor
boards in here and we'll talk about that
in a second but the first thing I want
to do is actually make sure this script
can run at all
so one make sure you've got Karla up and
running because if you don't have that
it's not gonna work very well
and then too we need to run the script
that we're actually working on so
examples and tutorial five dot pi is
what we're doing PI - don't forget three
point seven tutorial whoops
five dot pi okay let's run that of
course oh right
okay yeah so there's been a quite a few
errors that people have pointed out
leading up to me actually finally
recording this one so in this case we
have a positional argument is required
anyway it's mostly input shape needs to
yeah equal where that so let's do a
smaadahl input shape equals save and
let's try again
is it thread actually that was not an
expected error
hmm is it thread it's probably thread
that makes a little more sense but let
me confirm that yeah it's just thread
what a noob oh now I gotta fix threading
let me find where I actually typed
threading from oh I imported thread but
then I came down here oh did I actually
type thread correctly yes I did okay
carrying on hey we're getting some
information here it's kind of going off
screen my apologies I'm sure we'll hit
an error at some point right surely we
got a hidden error I thought there was
more errors than just this oh we never
did our modified tensor board okay so
that it's definitely in the text-based
version let me just find one exam so
from Charis callbacks tensor board I
don't remember if we brought that in or
not already kara I don't see it so pip
whoops
paste that in and then again I'm just
gonna copy and paste this again this is
yet another thing that is coming from
the reinforcement learning tutorial
series I did explain it once already
I'll explain it again but yes the reason
why we're doing this is because tensor
board by default will create a log file
per dot fit well we have a dot fit for
every frame that we process and forever
eight in it so that would just get
absurd and then every step would also
happen every fit and we don't really
want that we just want to step per
episode and yeah anyway so a custom
tensor board
object is kind of required for a
reinforcement learning at least for DQ
ends okay let's try again I'm pretty
confident we have a few more errors hoop
that won't run into let's see what
happens
taking forever there we go
okay blueprint library not defined
probably because it's self-taught
blueprint library is my guess
bloop yes self-taught blooper oh my gosh
self-taught blueprint library all right
let's try again while we wait shouts out
to recent new channel members master of
none sue-shaun - bird a lance campbell
vincent simonin Lee immense B and a name
I can't pronounce but feel free to tell
me in the comment section how to
pronounce that thank you guys very very
much for your support
you guys are awesome now will it work
this time probably not I'm sure we have
other errors I'm trying to recall all
the ones that because somebody was
following along in a no has know
somebody was following along in an I
Python notebook and was like posting the
errors in the discourse because you run
every cell right in white car n has no
attribute in height what do you mean it
doesn't have a swear it does M oh it's
under under lowercased
I'm trying to decide how I want to
handle that actually I think that's
popping up on the get queues yeah and
then is that going to continue let me do
get Q's M so it happens here get keys
currents and then on the current state
so yeah the only time we actually use
that is here and in this case it's
actually a lowercase M height and M with
all right all right all right all right
I think I think we're coming down the
homestretch of issues I think this time
it's gonna work I have faith I believe
I'm I hope while we wait I am running a
live stream you can actually see it in
the background but if you go to
twitch.tv slash Centex probably every
day for the next few days I don't know
when you'll see this video but I will be
putting up the stream I'll explain what
my reboil
in state shape
okay this one is a little more
complicated but this and this is why you
don't do unpacking I'm blaming Daniel
for this one so state dot shape I got
too busy talking about how that actually
works but what I forgot to do is we
reshape one comma state shape but in if
you did that it would unpack the values
of state dot shape but instead what I
did is I forgot the comma because of
Daniels it's all his fault and instead I
did negative one x state shape so
instead what we need to do is return
model dot return self dot model that
predicts return self dot model dot
predict Bob blah blah comma state shape
running that one more time this is this
is it I'm telling you guys it's gonna
work anyway this is a the live stream of
the agent running of course this is I
wouldn't expect that you're going to be
able to get this is currently running
eight agents and we'll talk about that
in a moment
but this is basically this has been
running now for about a day in maybe
twelve hours or something like that
it does pretty good it can it can at
least stay in the lane okay wait hold on
I hear things running I wonder if it's
oh my goodness I'm about ready to our
self dot RGB how many errors are we
gonna hit that's going to be RGB camera
yep so I'm gonna do this
pasta pasta pasta see what happened here
you don't even know how I made that
mistake okay okay okay okay I know I've
said it before but this time this is the
time it's gonna work I mean how many
more errors could we have not many I
mean I at some point there's a finite
number of errors that we could possibly
hit anyway I'm gonna run this pretty
much every day around 1:00 p.m. Central
time maybe a little earlier basically at
night we'll set the epsilon quite high
let the epsilon come down to about 10%
and then that's where rewards will
hopefully back propagate in the agent
will actually learn something and make
most of its improvements whereas over
night it's just going to be doing a lot
of exploration so I probably won't
stream it at night and I'll just stream
it kind of during the day but as you can
see I mean it's it's doing okay I mean
it learns how to do some stuff I've seen
it make some really good turns and stuff
like that like right now that what
you're looking at is 10% Epsilon
it still has a good way to go and we're
at about a hundred thousand episodes so
that should tell you uh that it will
indeed take quite a while to train a
model like this so here I'm running a
hundred episodes we are on Episode two
out of a hundred it is going to take
some time but no more errors see you
guys you can trust me when I say there
will be no more errors there will be no
more errors so okay I think what I'll do
is I'll pause it and then her in fact
hmm I don't see any reason to pause it
and because I at least have some imagery
that I can show you guys so of the model
training so this was the first one that
I trained on my own like while I was
writing the text-based tutorial
this was the tensor board that we saw
and what I'd like to do is make it big
yeah so this was our accuracy obviously
this is epsilon over time again it's
only a hundred episodes but we can
already kind of learn a couple of things
from this and I'll talk more about this
in the next tutorial but you know
accuracy for the model this is about
what you would expect the fact that it
hits 100% accuracy a couple of times is
a little bit of a red flag and it's not
good
epsilon decay I mean it did what it was
supposed to do reward average I mean it
maybe it started coming up it's too
early to tell and then all the reward
stuff so there's really not much we can
see here other than already within a
couple of episodes or within a hundred
episodes we have a loss has exploded
twice to astronomical numbers and again
in the next tutorial I'll talk more
about that in stuff but that's a bad
thing we haven't we don't want loss to
explode like that like loss can explode
- you know maybe double digits or have
it but we'd like to avoid this this is
bad this tells us something that's going
wrong and then I ended up because the
model didn't save I ended up actually
running it again and this was the image
so let's just open that again in a new
tab and this actually this you guys look
better the things we're going in the
right direction again we still peaked at
a perfect 100% accuracy red flag our
loss looked a little more under control
and at least eventually kind of came
down but again we're the the scale here
is not really a scale that we want now
it's better scale right this is like you
know up to the power of nine this one's
only up to the power of five so it's
better but not quite the scale we want
for loss so something's going on there
and again I'll talk a little bit about
that but as far as reward average and
Max and min I mean it wasn't proving but
when I go to run the model it was
actually just doing circles it would
only do one action and then you know
from there you could do a couple of
things to try to alleviate that so you
let me get soom back yeah so I want to
say this is the second yeah this is this
will be the second agent in action you
can see I'm getting 18 point 5 frames
per second but yeah you can see all he
wants to do is do turns and then here
you can see the actual these are the Q
values we can see that they are changing
and in kind of interacting with the
environment and then sometimes actually
this you know this value is higher than
this value or sometimes this one's
higher or this one's always the highest
right so this is our turn right so it
always wins but sometimes this one is
higher than this one over here so so
that does sometimes happen in this video
it didn't but it did just trust me
um and so we could like wait these or
something like that but we already knew
something was wrong with the loss
anyways so I'm gonna talk about that in
the next tutorial but if you want to
kind of tinker with your model and train
the model for a little longer try a
different neural network all that you
can use the following script here and
this would be our plate up pie basically
so whatever the name of the code that
you've been working on so far is that's
what you would put here so tutorial 5
code is just kind of my placeholder
sometimes I put dashes and I've been
doing that on the previous tutorials
obviously you can't import a module with
dashes so replace those with underscores
or whatever and then model path you'd
have to modify that to be whatever the
name of your model is when it gets saved
but then you can run this and actually
see your model playing in action
unfortunately nothing was too exciting
here um but but but that's part of it is
is you can actually see what is the
model actually doing if we use 0 epsilon
and and you'll see that oh it's terrible
so so anyway in the next video we'll
talk a little bit more about the
findings here I'd like to run this a
little longer before saying all the
changes that we made and really noting
the impacts that were made because of
those changes so it definitely like it's
it's
one real full backpropagation of the
rewards from one decay so hopefully at
by the end of today we'll kind of have a
better idea for you know how much longer
can I actually train this model before
it's probably at its max and then we can
start to decide do we want to continue
pushing Karla do we want to continue
maybe give them more actions so this
agent currently has to go full throttle
we could give them control of the
throttle or actually we've been
referring to this as Charla so it's a
she so we could give her access to
throttle so whether or not she wants to
have full throttle or no throttle I
don't think we're gonna do any sort of
pit or anything and then also maybe
access to the brake so they could slow
down so maybe do that and we might
actually find that it does better the
other thing I wouldn't mind testing is
cycling different cars stuff like that
because the dream would be to move this
to Grand Theft Auto 5 and before I can
do that we would definitely need that
camera position to be somewhat dynamic
in the training data otherwise what's
likely to occur is as soon as you change
that even just a little bit the model
just loses its mind so we definitely
want to do that might even might even be
a good idea to change field of views
just like dynamic field of view for the
camera sensor stuff like that so anyway
yeah well we'll see what happens it's
still kind of up in the air
definitely have learned quite a few
things but I think I'll save that and
package that for the next video but yeah
as you can see I mean it does pretty
good in a straight line I mean
eventually he decides he's she it's so
hard I'm so used to it being Charles but
yeah so it does a pretty good job
staying in a straight line and then I've
seen it take some turns most of the time
it's still plows right into a wall but
yeah I'll show some more examples of
things in the next video so questions
comments concerns suggestions whatever
feel free to leave those below otherwise
I will see you guys in the next video