what's going on everybody and welcome to
part 3 of the self-driving cars with
Karla tutorials in this tutorial and the
coming probably at least like two plus
more we're gonna be talking about doing
reinforcement learning specifically deep
cue learning with the Karla environment
now to do this we kind of have to change
the architecture a little bit of how
we're approaching this problem we want
to approach this problem with the with
with everything being set up to work
well with the concept of reinforcement
learning so there's a million ways that
we could do this but with the advent
basically or the introduction of open AI
into the world now known as closed AI
they had kind of set the standard for
how to approach and work with
environments to do reinforcement
learning and that that methodology kind
of has persisted and tends to be how
everybody handles reinforcement learning
whether or not they're actually using
the open AI environments themselves like
gym or of retro or now maybe is the name
and I forget the other one but like a
atari maybe anyway not important but
basically you want to have some sort of
environment that has a couple of methods
so i we're gonna basically keep all this
code here we're gonna have to get rid of
a lot of stuff because we're we're not
we can't really run this as a script
anymore we need to probably use
object-oriented programming so we are
probably done with all of this I do kind
of want to leave process image we're
going to just convert that to a method
rather than a function but that's okay
in width and the height we'll probably
not moving those so I'm just gonna
delete those mostly I just wanted this
block of text I'm determined to never
write it anyway
basically the the standard for for
reinforcement learning environments is
to have some sort of dot step method for
the environment
where you pass an action to that step
and maybe if the syntax was working it
would look better so anyways you'll have
some sort of step method where you pass
action you'll do something with that
action and then collect any information
like determine what the reward was and
stuff like that and then when you're
done you return the next observation the
reward whether or not we are done so
this will be like a flag it'll be a
boolean flag basically so this will be
true or false whether or not we've
either reached the end of the
environment like week we're successful
in what we tried to do or Ruby in at a
time or we died for whatever reason so
and then finally usually it's just you
see it as an underscore but it's just
extra info so if there happens to be
extra info this just again just kind of
the standard we generally almost always
care about observation reward and
whether or not things are done but any
environment we tend to throw this in
that way and the reason why we want to
use this standard and continue to use
this standard is there are different
reinforcement learning models basically
that you can work with and whether
someone did it with you know opening eye
gym or they did some other environment
or whatever you can easily just swap
these these models around to all kinds
of different environments and some
environments might want to return some
extra info for who knows why but maybe
and we just we just leave that there so
that is possible so some sort of
extensibility is possible but anyways
this is what we want to do so we need a
step method and then we also want a
reset method and this is either at the
very very beginning of the environment
or after we've returned a done flag if
we want to run another episode so to
speak will reset and run another episode
okay so that's what we want to do now
what we need to do is actually do it so
what I'm gonna do is just come down
again I'm just gonna leave this process
image code here because I'm gonna pretty
much need it I'm just gonna add a bunch
of self dot in front of everything so
I'm gonna leave that there
we're gonna start some new code here and
the first thing I'm going to do is we're
gonna at the top of this script we're
gonna have a bunch of like starting
variables and constants but for now I'll
just have to show preview and I'm gonna
set that to be false for now and this is
whether or not we want to display the
actual camera from Karla so you wouldn't
want to always display it one you know
if you have a worthy machine you'd want
to run many agents at the same time but
to displaying that previews gonna hog up
computing resources both CPU and or GPU
I believe OpenCV purely uses CPU so it's
gonna lock up CPU but you just you just
it's gonna use resources nonetheless and
probably we're going to be maxing things
out so we'd rather not but if we want to
use it for debugging purposes we can
turn this flag to true just to see a
couple of previews and then turn it back
off maybe or whatever anyway I'm gonna
set it to false for now but you
definitely would want to be able to see
what's happening sometimes so then we're
gonna have class and we're gonna call
this car and vironment and now I'm gonna
just set some of our initial values so
first we're gonna have show cam and for
now that'll just equal show preview we
might also have show cam show every even
if show preview is set to true maybe it
shows every you know I don't know ten
thousand episodes or some not ten ten
thousand times those four big number
hundred episodes or something like that
so show cam equals show preview for now
then we're gonna have steer underscore
amounts we're going to set that to one
point now basically we're gonna fully
you know there's three actions we can
take it'll be in zero one or a two so we
can either or actually it'll be negative
one zero or one and basically we can
either steer fully left go straight or
steer fully right but later you might
want to make this may be cumulative so
that so the steering will slowly turns
one way and then if you want to go the
other way you slowly turn the other way
depending on what kind of frames per
second we get we could probably do some
much more fancy stuff there but for now
we're just gonna do full turns every
single time then what we're gonna say is
we're going to throw in an M
and in fact I guess we'll just toss this
up here so M underscore width I totally
just deleted all this stuff with 640 m
height 480 I was wondering about how
that was getting typed in there and okay
I see now cool so we've got those and
then for now I guess we can set those to
be equal to these as well so M M
underscore width and with will set equal
to M width and then M height and
underscore height will set to M height
okay so then what we're going to do is
we are going to I think we'll just set
will say front underscore camera for now
we'll say that's none and I think that's
it that's all of our like basically
initial values that we want to set now
what we're gonna do is we're gonna
define in it and so self here and what
we're gonna say is basically all that
starting I deleted it but the starting
code to connect oh it's not too much
anyway so we need to connect to the
server so self dot client is going to be
Carla dot client and then we're gonna
connect local host port 2000 self dot
client dot set underscore timeout we're
gonna set that to be two seconds still
self dot world is equal to self dot
client get underscore world
blue blue print underscore library
library equals self dot world gets
underscore blue print underscore library
so okay we've got the blueprint library
now we want to grab our car so self dot
model underscore of 3 is equal to
blueprint libery
dot filter and we're gonna filter for
model 3 model three actually it's just
model 3
underscore their and then the zeroeth
index so with that we probably have
everything we need when we first
initialize because we just we just want
to have access to our blueprint library
we want to well at least in this case we
just grabbed for the car in fact we
might even mmm we probably I think this
should just I'll just make this self
table blueprint library because we're
gonna probably need to access that
elsewhere because we need the sensors as
well so I'll leave I'll just do self
doubleu per library okay so with that we
are ready to do our a reset method so
we're gonna say define reset and again
here just past self there's nothing
needs to be passed step is what's gonna
take in the action so at the beginning
of every you know reset environment what
do we need well we're gonna say self dot
collision underscore history that'll be
an empty list because the collision
sensor returns a list of these like
collision events basically if we have
any collision event we're going to go
ahead and reset at least that's what
we're gonna do start later I might
customize that I haven't looked too
deeply into the collision sensor the
collision sensor basically reports like
some sort of like magnitude value and
sometimes like if you just drive over
like a pretty non substantial curb even
like you're totally fine even though
that wouldn't theory probably throw the
car out of alignment it caught it says
that was a collision and I've even seen
it call a collision like if you just
simply like go up hill really quickly
like maybe the bumper scraped the floor
or the ground because you went too fast
I'm not really sure but I've seen it
register that as a collision as well so
we might want to require the magnitude
to be higher than anything right but for
now if anything is in this list so if
any collision is detected we're just
going to say hey you failed and then
self dot actor underscore list is also
going to be an empty list and again we
just want always want to
tract actors so we can clean them up at
the end so then we're gonna say self dot
transform equals random dot choice self
dot world dot get underscore map dot
gets gat get spawn points okay don't
forget your open and close that's a
method get map and then get spawn points
again method okay so we've got the
transform now we're gonna do is self dot
vehicle equals self dot world dot spawn
underscore actor actor and we want to
spawn self dot that model three from
above and then we're gonna spawn it to
self dot transform so we've got we've
just spawned an actor what do we need to
do we need to self dot actor underscore
list dot up and self dot vehicle okay so
now that we've done that we want to get
our RGB camera so that's the next thing
I'm gonna go ahead and add so we're
gonna say self to RGB underscore cam
equals self or no we're gonna use
blueprint library so self dot blueprint
library and then we're gonna say dot
find and then sensor camera RGB secretly
actually an RGB alpha camera self dot
RGB RGB cam dot set underscore attribute
and we're going to say image underscore
size underscore X is the F string of
self dot oops self dot M width
M width cool and then what I'm going to
do is copy this paste paste will do
three pastes so M so then Y so you've
got your X Y or Y is your height M
height and then this will be field of
view and then we're going to just I'm
just going to hard-code this to be you
110
and for 110 for now okay so just like
our vehicle we get the vehicle what do
we need to do well we need to specify
the transform so I'm just going to say
transform equals Carla dot transform
that's a capital T that I transform
transform because Carla did I do a car
where's the other transformer hmm okay I
don't really see it anyway
Carla dot capital T transform I'm just
curious why it wanted a lowercase that T
but I'm sure when we go to run this
we'll see errors so and then Carla dot
the current location and this is again I
believe to be a relative position so
it's just in relative to where we throw
it we want to move this so transform
then what we want to say is the self dot
sensor is equal to self dot world dot
spawn underscore actor actor E actor
self dot RGB underscore cams so it
spawns the actor and then transform and
then attach underscore two equals self
dot vehicle so now we've got this camera
on the front of our car we've got a new
actor what do we do well we append to
the actor list so copy paste and in this
case it's not self dot vehicle it is
self dot censor so once we've done that
the next thing we want to do is we want
to say self dot censor dot listen and
we'll do lambda for whoops lambda data
and then self dot process underscore
image data lambda data what have I done
oh okay lambda data self dot process
image data okay so then we need to what
we we need to convert this to be a
method inside of our environment which
we will do I'm just going to go ahead
and finish what we're doing here
and then we can we can do that pretty
quickly so once we've done that we've
got our car we've spawned the car we've
got our camera we've spawned the camera
so we've got all that and what whenever
you whenever you create this car and you
spawn a car I don't know why they did
this but they did thanks Carla people
when you spawn a card actually falls
from the sky and maybe they did that
because that was the easiest way to
overcome issues where you like you
spawned and we're clipping a little bit
like your tires maybe we're clipping and
you couldn't move I don't know I'm sure
there's a great reason for it but you
fall from the sky and when you fall from
the sky one issue is you actually can't
drive yet the other issue is when you
hit the ground sometimes the collision
sensor registers that as a collision
because you just fell from the sky and
then also initially your RGB camera
hasn't yet I don't know if it
initializes or what but it doesn't
actually start pulling in data so
sometimes as soon as you try to grab
imagery from the RET RGB camera you
cannot it just it just returns none and
this is for a variable amount of time I
couldn't figure out or like sometimes it
would immediately return imagery and
then other times it would take some time
don't know why but it's a thing so we
have to handle for that sort of nonsense
occurring so so we're listening then
what we're gonna say is self not vehicle
dot apply underscore control yes so I
looked out one more thing too is that
the duration that it takes for the car
to actually start doing things also is
variable so like the camera they're just
weird things that happen and I didn't
really discover this Daniel seem to
discover that if you just apply some
control like even if you don't do
anything if you just send in the command
to control it makes the car react
quickly more quickly anyway so I'm gonna
throw this in I
I have no words anyways Karla but you
already know how to do vehicle control
so we're just gonna say throttle equals
and we're gonna say zero point zero and
then brake equals zero point zero
definitely more research is required
here to figure out exactly how to feed
Carlo what it wants to make it act
quickly as quickly as possible because
the speed that we can run through
episodes the faster we can do that the
faster we can train these models so it's
really important that we can get this
this thing to just get going like as
quickly as possible so we'll be we'll
have to just continue to work to try to
figure out how to make it go as quickly
as possible for now time don't sleep for
seconds so considering the fact that
episodes are going to be about 10
seconds a full sleep of four seconds on
top of those ten-second episodes
stinks I really wish we could get things
rolling a little quicker so anyway Carla
people if you're listening which you're
not I'm sure but if you were how can we
do this quicker anyway now we've got the
car we've spawned the car we've wiggled
the car
well we haven't wiggled but we've sent
some commands so hopefully the cars
paying attention now the last thing that
we want to do is we're going to do the
coalition sensor call sensor equals self
dots
what was it blueprint library yeah self
dot blueprint library dot find and we
are going to attach the sensor dot other
dot collision collision great so that's
our collision sensor the next thing that
we want to do is self dot call sensor is
equal to self dot world daunt spawn
underscore actor and we're gonna spawn
coal sensor transform and then we're
going to attach underscore two equals
self dot vehicle as I write this I
wonder I wonder if transform even
matters like I would I guess I would I
would almost assume you wouldn't need to
like I don't think this is going to
matter that we've added this collision
sensor here
I just kind of realized that we were
doing that anyway the polluting sensor
works like that so I'm gonna believe it
that you probably don't have to adjust
it but we're just gonna reuse transform
okay so uh okay so we've spawned an
actor what do we need to do you guys
know the deal so we need to self dot
actor lists depend call sensor cool okay
so now that we've done that we want to
uh we're gonna say self dot call sensor
call sensor dot listen and again we're
going to lamda will say event this time
self dot collision underscore data event
so now we need this collision we'll also
need the collision data method but again
that's a really simple one and then we
in order in order for this agent to run
like basically we're done at this point
but in order for this agent to actually
run it has to return an observation well
our observation is the front-facing
camera so if after four seconds that
camera still is not ready we are going
to say wow self dot front camera is none
so it returns none until it starts
returning things we're going to say time
dot sleep 0.01 now we might we could
possibly like do all the handling for
sensors and maybe a few other bits of
information and then comment this out
that way we're doing the least amount of
waiting possible so yeah there's
definitely much more optimization that
we can do here but for now we're gonna
follow up with this we just want to make
sure things work first and then we can
start to optimize other things but
anyway well can we'll do that once
that's done it means the episode
actually has started so we're going to
say self dot episode underscore start
equals time time because we want to run
episodes for ten seconds otherwise we
would just pretty much run infinitely so
yeah I think we're just going to say
episodes are
ten seconds fixed we could later maybe
even remove episode time so the longer
it runs but the problem is agents will
sometimes find things that they can do
that you didn't expect them to figure
out so for example they might find that
they can drive in circles like that was
one of the things that we discovered and
they'll just do that infinitely long so
sometimes it can be better to actually
end it at some point but when it's first
starting out I mean it's gonna crash
constantly so so you might not even
really need to worry about this but
anyways we're gonna track star episode
lengths anyway okay so then we're gonna
issue one more of these self vehicle
controls again it just seemed like
throwing these at Carla could get things
running
I really don't we we have no definitive
testable findings here yet we're trying
to figure out how we can get things to
just work quicker but yeah I mean if
you're just trying to run a car one time
you almost don't even notice the issues
but when you're trying to run hundred
thousand plus episodes you start
realizing hmm there's strange things
happening here so anyway some time apply
control great and then we will return
self dot front underscore camera okay so
we've done that now we need to add those
two methods below here so soft on camera
so we're going to say define collision
underscore data and this will be self
and then it takes in event and then what
we'll do here is we'll just say self dot
collision underscore hist dot append any
event so if we have an event we'll just
append it to the history and then we'll
we'll handle for this in the actual step
method and then we need our process
image method so now I'm going to take
this cut move it up here pasta hey whoa
I appreciate this thank you so process
image will pass self image I equals blah
blah blah blah blah if will pass here if
self dot show cam I want to say
yeah show cam so if self dot show cam we
will show the camera dude
you were killing me okay
oh and then we need to fix this which
will be self in with self tim height are
actually height and then with see i try
to make that mistake i told you that was
gonna happen in height maybe I'm just
stupid self dot M with or both in width
can't can't rule that one out
okay I think Oh
here what we want to do if selfish okay
so we'll just do this we'll say self dot
front underscore camera equals I three
and then we'll just have to remember to
scale that down with the negative or
divided by 255 we'll do that at the end
[Music]
okay I think that's good I'm sure I've
got problems but I think we're good
there so now the last thing we need to
do is the step method well it's not the
last thing but the last thing we need to
do for a Carla environment I was tempted
to do that in the next video but I think
we'll just throw that in really really
quickly that's pretty basic so define
step and then we'll pass self action so
it basically it takes in a specific
action and then we're going to have if
action yeah I totally miss Polk in
faction is equal to zero will pass and
then we're basically going to Elif
action equals one and then we're not
actually gonna pass but if and then else
actually I'd rather this be an elephant
equals to pass so if the action so we're
gonna say the actions are zero one in
two
yeah we wouldn't want to do negative 1 1
& 1 we'd rather these be like argh maxes
of output so 0 1 2 so 0 is basically the
left one will be go straight and then 2
let's say turn right
so if action is zero we're gonna say
self dot vehicle dot apply control and
then it's Carla dot vehicle control and
then we're gonna say every time the
throttle we're gonna set that to a full
throttle and then we're gonna say steer
is equal to negative 1 times self dot
steer amount so this should say negative
1 times 1 right so steer will be a
negative 1 and then because again later
we might change to your amount to do
other things so for now this will be
fine
so that will be go left then this will
be go straight and then this one will be
go right so rather than negative 1
it will be 1 and then here in theory you
would say zero times steer amounts but
I'm just going to put in a 0 just for
easy legibility sake okay
then basically you could say as this
thing goes on you could you you
basically if it's not colliding with
something you could give it a reward but
instead what we're gonna do is I'm gonna
use speed only because I know the car
will figure out to just drive in circles
and we don't want that so we're gonna
say V for velocity equals self dot
vehicle dots get underscore velocity and
then because velocity is not really
super useful we're going to convert that
to kilometers per hour we're going to
say equals the int value of three point
six times math dot square root I might
not have an important math by now but
that's okay of the velocity x squared
hey bro plus velocity y squared plus
velocity Z
to the power dude okay cool so that
should be our km/h and I don't think I
imported math I'm sure that's why it's
angry yeah so let's go ahead of you
import math and then we'll go back down
to the bottom and okay so then what
we're gonna say is if if the Len of self
dot collision history does not equal
zero like later we could do something
much more fancy with the the events that
exist in the collision history but right
now if we've registered any collisions
we're going to say done is equal to true
and then we're gonna set the reward
equal to negative 200 again I'm just
pulling these numbers out of nowhere you
might want to make that that penalty
basically much larger maybe negative 500
negative a thousand you know I don't
know I really don't of these just making
it up as I go so L if kilometers per
hour is less than 50 then what we're
gonna say is done equals false we're
still running everything's happy and
instead we're gonna say reward is equal
to negative one so any steps that we
take any frames that go when we have not
we're not going faster than 50
kilometers per hour which honestly is
not very fast but I used 50 because to
reach 50 kilometers per hour just
driving in a circle is kind of hard so
that's why I went with 50 again just
kind of making it up as I go so it'll be
negative 1 but we didn't crash so it
doesn't need to be a huge penalty again
you also could set this to be zero I
don't know the answer to that we have to
research it to find out L if else we're
gonna say done equals false and the
reward equals 1 so if it is about 50
then we'll keep appending one to the
total reward okay so that's basically
we've got our actions handled here we've
got our calculation for a reward but the
other thing we want to do is we want to
limit
lengths of these episodes so we're gonna
say if self not episode start plus
seconds we'll do seconds per episode
episode so probably to start we'll just
say 10 seconds so we can come up to the
tippity-top here we can say seconds per
episode we want 10-second episodes if
the person is less than time time time
time then what we're going to say here
is done
equals true cool so then finally the
last thing that we're gonna do here is
return self dot front underscore camera
so observation reward done and then for
now none we don't have any extra
information that we want to pass
okay I think I'm happy with that I'm
sure I've got issues here so stay tuned
when we actually run all of this stuff
to fix any errors if you feel if you see
something feel free to comment below but
I'll find it out we've also written a
lot of code here so right now I don't
have the text-based version of this
tutorial up yet but by the time you see
it it probably will be up so so I'll
have links in the description to that so
if you want to make sure your code is
the same or whatever feel free to do
that with that shouts out to my channel
members these are people with me for one
month
Carter babban Black Hawk 3003 Jesse
Jones Payne max Phillip Wagner not Brian
Bradley and this around Kumar thank you
all very much for your support you guys
are amazing individuals okay questions
comments concerns whatever feel free to
leave them below come hang out with us
in the discord that's discord GG /
syntax if you want to support the
channel you click that beautiful blue
join button otherwise that's it for now
I will see you guys in the next video
where we will be working on the agent
itself and then hopefully after that we
can bring everything together and
actually start running the agent but
it's going to take a long long time to
do the learning okay see you guys then