what's going on everybody and welcome to
part three of our tensorflow TAS
tutorial series in this video in the
next video we're going to be talking
about incorporating tensorflow TAS into
a much more complex application so we
saw like a really basic application of
doing linear regression and doing like a
best fit line to some input data but
this is a pretty boring application so
now what we're gonna do is something a
little more exciting and that's going to
be having an AI play pong against you
before we get into it I just want to
give a quick shout-out and thanks to the
channels two most recent sponsors
Leonard M and the Golden Square thank
you guys so much for your support you're
awesome okay so first of all as I've
been saying like a million times I am
NOT a JavaScript programmer so I knew
what I wanted to do was do pong but I I
couldn't code pong and JavaScript
probably to save my life so I used the
following I put a link you shouldn't
need it all the code I have in my text
base right up basically because we've
modified it pretty significantly but
this is the original code and if you
want to learn like how to actually make
it I would go here the link is in the
text-based version in this tutorial
though if you want to check it out
and come in on down here basically at
the end of the day this is this is what
you wind up with some sort of pong
actually that's not quite it I think
they have a jsfiddle yeah where you can
actually like play pong over here so you
move the paddle back and forth and then
the computer this is like the computer
then this is the player here and you can
play pong with it so anyways what I
wanted to do is take this code and adapt
it to use tensorflow HJs to play pong so
that's what we're gonna be doing I'm
gonna get this stuff out of the way now
and we're gonna get into it
also you will definitely want to head to
the text-based version of this tutorial
part three there's going to be a
significant amount of copypasta because
what I just don't see the point of us
like writing this stuff out and then
also writing out a lot of the code
that's coming from the pong game
tutorial I the whole point is adapting
JavaScript code to include JavaScript so
this is definitely not a JavaScript
tutorial on i'm just not someone who
should be teaching javascript so so
anyway yeah so first we're gonna start
off with just a pong game dot HTML so
I'm just gonna do new call this pong
game HTML cool open that up and zoom and
Elenin and here we go so all we're gonna
have in here is the following so you've
got your main content div we're bringing
in tensorflow dot j s and then finally
we're loading this pong game J s and
that's where basically all our code is
gonna go so before we do that we
probably should understand like what
would be how would we even do this like
how will we train an AI to play pong in
the first place what kind of model are
we going to use so my plan is to
basically feed into that like that our
features for the network are just gonna
be like where is what's the balls
location what's the your paddle location
what's the enemy's paddle location and
then like all the previous things of
those so that what's the previous
location of the ball previous location
of the paddle previous location of the
enemy
basically so yeah so now what we're
gonna do is go I guess we're gonna make
another let's make another and we're
gonna call this one
pong what did we call this I think it's
like pong game Jesse yeah pong game J
yes yes and then we'll open that again
in sublime and then we're just gonna
start going through it and again I'm
just gonna be copying and pasting this
and just kind of running through the
text-based version of the tutorials so
first is just our model this actually
isn't the final model end up going with
but this will do for now so we're just
defining the model that we that we want
to use so again here the input shape
that we're defining and in fact what is
it this actually came through and I
don't even want to mention that okay I'm
just gonna delete that for now
okay yes for so
for some reason this still works and I
have no idea why why would the input
shape unless it like slices it or
something which would be nuts
anyway this actually works but I'm gonna
delete that and go with this which is
better I can't believe that made it
through that really confused me
initially I have no idea why that works
but it actually does so again the the
eight inputs not in this specific order
necessarily but where is the player's
paddle whereas the computer paddle
what's the ball X ball y previous ball X
previous ball Y previous paddle X
previous paddle Y so those are all the
things now the next major chunk of code
here is all original from the pawn code
kind of tutorial that I'm linking to
anyways so I'm just gonna paste all of
that in so again you'll have to go to
the text-based version toriel because
we're not going through all that the
next thing is now we've got some custom
code because we just pasted it in so so
this this is for the actual thing to
move so when you're playing this version
that is from the tutorial the computer
player that we're gonna replace with an
AI the way it moves is by following the
ball it just follows the ball around and
like that's it so it's not like the
greatest AI in the world but it's like
it's so much challenging to defeat but
it's actually pretty simple once you
learn the simple way to defeat it now
because we're not always gonna be using
the computer we need a way to update the
paddle for our AI so actually our ai's
logic is much much simpler so our neural
network model is going to return
basically the moves so we feed in here
we feed in those eight features pass it
through the you know all these beautiful
hidden layers and the output is three
units and it's going to be a one hot
output so the output is going to be
something like 1 0 0 it could be you
know 0 1 0 or it could be 0 0 1 and this
is corresponds to like move left move
don't move it all hold still and then
move right and then what we're going to
do is apply an Arg max to this so then
you're gonna say probably like TF dark
max for example I think it's camel cased
I forget we'll get there in a second
anyway and in this case the Arg max
would be 0 1 - right so - then what
we're gonna do is we're gonna subtract
one no matter what no matter what the
output is we subtract 1 so in this case
the Arg max would be 0 we subtract 1 so
subtracting 1 assuming it was this one
that answer would be 1 so 1 is the move
that's going to get passed to this
function here
so 1 and then we're gonna move that
paddle 4 times whatever that move is so
in this case 0 1 Arg max is 1 minus 1 is
0 times 4 still 0 and then in this case
it was 0 - 1 negative 1 times 4 minus 4
so either it moves 4 pixels left right
or nothing at all ok so that's basically
the moving logic now the next bit next
batch of code is I almost mixed bit in
batch and instead of wordy derd now yeah
so the next batch of code is just from
the or you know the pong game tutorials
so I'm not really gonna talk about that
copy-pasta okay this is just some custom
code to start collecting data for the AI
also I guess there's a bunch of like
this stuff here if you're not a
JavaScript programmer like me
this is seems to be very very similar to
self in Python where it's like just this
like abstract object the only real
difference between this and self that
I've noticed is it works in functions
so it's just like this overarching
object
I guess anyways moving on someone can
post below a much better description I'm
sure otherwise you can like Wikipedia
Stack Overflow I think there's probably
mokoro stuff too for you know this for
self or just what is this
yeah ok now we have some code for it's a
little more custom
and all we're doing here is this
basically is just saving data per frame
so at the end of the day what do we need
to train a model we need training data
so we have to collect that data so in
this case what we're going to be doing
is we're gonna play against the computer
for n games we're going to decide
however many games we want to play so
for n games we're gonna play against the
computer player and then after that
we're gonna take all the training data
hopefully balance the training data and
then train the computer player to be an
AI a deep learning AI to play against us
so in order to do that the other thing
that we need to do is we basically the
computer player is gonna learn based on
what we do so in order to make that
applicable to the computer we have to
like flip the table so that's what's
going on here speaking of which I didn't
mention it before
big thanks to Daniel cooky Ella for uh
for helping with this JavaScript code
he's actually the one that wrote most of
the JavaScript here because if I didn't
mention I'm not a JavaScript programmer
although neither is he I don't think
anyways so there's probably errors here
and we apologize profusely so now we
want to do is we need to switch like I
was saying before after playing so many
games we want to switch to that AI
player so that's what this is doing so
basically every time every time a score
is made we need to start a new turn and
then in this case if the turn is greater
than 2 so after 2 whole games we're
gonna train in AI you could change this
because obviously two games is nowhere
near enough games actually train an AI
to play against you 10 is getting there
I found some success after about 20
games the AI was like starting to learn
some stuff but you're definitely gonna
need some more stuff to get here the
other option I just think it's
unreasonable to expect players to play
against this boring computer for 20
games just to play against the AI so
what's the more likely scenario is later
on we'll load a model that's already pre
trained and then you just do transfer
learning and that makes a big difference
so anyways I'm going to stick with one
just so we can get through this in this
tutorial
in a timely manner so after every two
games we're gonna just clear out the
data so it doesn't get super bloated
you either could do this or you you
don't necessarily have to that way every
time you train your training with all of
the data that's just an example for if
you did want to clear out the data
there's a nice little function for you
to do that now we actually would after
those two games and after we actually
want to train we need to actually train
so in this case we're going to go ahead
and train and then all this is doing is
it's splitting up the data pushing it to
our X's and Y's
defining the X's and Y's as tensors and
then we train the model right here okay
so once we've done that let me come down
shoot I got lost
yes okay
so then we need to be able to actually
make the predictions themselves so let
me paste that in and again do it very
similar to what we had before as long as
we actually have training data to make a
prediction we make the prediction and
then again it's just the Arg max grab
that data via data sink - one like I was
talking about before and then and then
that gets patched at the move function x
4 and that's how many pixels we end up
moving okay and then the last bit of
code is just the pong just from pong
itself so let's go ahead and test this
real quick I'll just open with chrome
pull it over all right so now let's open
up console just in case we do hit some
sort of error and I'm actually moving
let me do this okay
so here's how you beat the computer
player all you need to do is get it to
hit to the side as quickly as possible
there we go that should probably defeat
him yeah so now we just need one more
defeat and then we should see a train
and then see how it does again it's only
two games it's probably not gonna do
very well at all it just really
shouldn't it's this is nowhere near
enough data okay that should be a score
okay so now our AI is live it's making
all its predictions oh he did his best
[Laughter]
just keep in mind wow you tried again I
should I should have made it a little
more easy on him actually yeah he's
pretty dumb so one thing we could do is
maybe not clear out the data every time
cuz now he's just training on new data
but he does already have like weights
you know so it's really it's all it sort
of retains that data in memory just not
very well come on man okay anyway there
we have an AI actually playing again if
you do it for about 20 games or so it
will be a little better but obviously I
just that's not acceptable people aren't
actually gonna play pong for 20 games to
make di whoa this is reminiscent of the
robot arm playing playing air hockey
well suddenly he actually has moved and
hits things I got him here though for
sure see ya
anyway okay so what we want to do to
improve the model is somehow at least
start with a slightly trained model so
in the in the next tutorial we're
actually going to cover a bunch of stuff
it's probably gonna be super long but
basically we're gonna cover bulk
training this model so we're just gonna
have something that just automatically
like plays the game a bunch by itself
just to computers against each other to
compile a bunch of data taking that
bunch of data not training it intends to
float is because tensor flood is is slow
so we're gonna actually train it in
Python with care us and then we're going
to output that model
convert that model to tensorflow at j/s
format and then loading that model and
then we'll play with that model and see
how that model
so that's what you guys have to look
forward to if you have any questions
comments concerns whatever feel free to
leave them below if you like this
content you can support it Python
permanet slash support otherwise I will
see you in another tutorial