what's going on everybody and welcome to
another unconventional neural networks
tutorial video in this video we're going
to be changing gears a little bit and
trying to get normal networks to do math
so even though computers are actually
really good at doing math certainly
better than humans
the problem is with neural networks
normal networks are actually kind of bad
at doing math and the reason for this is
neural networks are really good at
finding nonlinear relationships whereas
you know most basic forms of math are
actually linear relationships so normal
networks are historically kind of bad at
doing just that
but for the most part no networks are
just bad at math period so my thought in
curiosity is could we figure out a way
to get neural networks to be able to do
math pretty decently and if so then we
could continue to increase the
complexity of that math and my ending
curiosity is how well if you could ever
use a neural network to do something
like say you break some forms of
encryption or something like that that
would be a curious case so so anyways
what we're going to start with is just
really basic can we get it to do
addition so so let's go ahead and get
into that and to start we have to decide
on a way that we're going to do this so
my forward thinking is that I want I
want to eventually get to the point
where we allow for a variable number of
input and a variable number of output
and we might want to have maybe you know
something plus something sure but we
also might want to have something plus
something divided by something but we
don't want all the inputs to be
something plus something divided by
something we want to have lots of
variability here so in order to do that
the probably the best way of doing that
would be like the sequence to sequence
library and the way that I'm going to do
that is we're just going to use the what
we've been using for the chat bot so
it's just simply sequence sequence code
and rather than doing words and doing
like word embeddings and stuff like that
we're going to make it a character based
model so each character has value and
then basically the way it's read by the
mesh
is a sequence of those characters so it
could be any character in this case it's
always going to be numbers or certain
operators like divide plus and so on so
to get started what we're going to do is
head over to the github project for the
chat bot I'll put a link in the
description someone to remind me if I
forget otherwise it's just github calm
and it's Daniel kook yellow and then nmt
- chat bot go ahead and clone it or do a
git clone or whatever you need to do and
get it if for whatever reason it has
changed where is I want to make sure
it's on screen the last commit here the
one that we're using is C 5 0 3 D to C I
link to it in the description the
text-based version of the tutorial so if
things are not working the way that I'm
explaining them or whatever then code
this exact commit so once you have that
I'm gonna go ahead and extract it ok I'm
gonna pop in here and probably the first
thing you should do is open up a command
prompt and do a pip install - or I think
requirements requirements dot txt let me
just make sure that runs it sure does
I'm just gonna break it and make sure
you have all the requirements so once
you've done that you're probably ready
to go so what we're going to start with
is basically you know this all has like
we need data that basically looks like
this it has you know it creates a from
and then points to it - so just look at
this this is chat data so lovely start
to the chat da anyway that's the chat
data and it points to this which would
be the responses well there's no reason
why we can't actually say in the train
dot from there's no way though no reason
why we can't save five plus two and then
in the Train - we just say seven right
there we could do that and then the
model could just be a character level
model so we wouldn't have to necessarily
just do five plus two it could be five
plus 22 and
so on so now what I'm gonna do is I'm
gonna pop up guess well first we'll just
delete everything in here so there's no
reason for this there and then I don't
see any reason to write all this out
with you guys I just assume at this
point no one following is like still
rudimentary Python so I'm not gonna
waste y'all's time so I'm gonna call
this date I don't know math how about
yeah we'll call it math data dot pi
that's fine let me open it with sublime
text so this this will be in the
text-based version of the tutorial again
there'll be a link in the description so
you can just take it from there so I'm
gonna copy paste that in let me zoom out
a little bit okay so what's going on
here first of all we just specify how
many samples in total do we want right
now it's three hundred thousand you
could make this you know anything you
want I mean it could be three million
then this is the the largest plausible
value we're gonna deal with right now so
we'll say a hundred thousand so
basically it could be up to 100 thousand
so fifty thousand plus two and there
might be twenty five thousand plus but I
don't know seven or something like that
so it could be any number within here
and then we're using random to randomly
pick some numbers this is how many total
samples and then basically I've just got
made this little simple function here
where to generate the pair that we want
string in string out you just say hey I
want you to generate a random addition I
want you to generate a random
subtraction so then what it does is it
just creates you know like a string
that's literally just is five plus two
and then the output is also a string of
seven okay so that's all that's
happening there and then here I'm just
testing it I'm not we don't really have
to test it but I mean I guess we could
just comment this out to see and get an
idea of what you're looking at a test
pair stranger so looks like it should
just run with the basics yeah so here's
an example and so this is the input this
is the output we could run it again you
can see input output and currently it's
always add but you could also have you
know subtract for example and there you
go so then what we're going to do
is come back when you do this comment
this out again so then what we're gonna
do is we're just gonna iterate over the
samples and we're just going to simply
right now we're going to just do
addition the question is can we just do
addition if we could do addition let's
make it a little more complex let's do
all of them let's do addition
subtraction multiplication division and
then if we can do off all of those then
the next question would be can we do
more complex things like you know could
we get to the point where we could do
something like I don't know 55 divided
by 7 plus 3 minus 5 divided by this
alright couldn't we get to the point
where we could get an algorithm to do
that kind of math that's a little more
complex and then from there you could
also do it we could try to do things
like you know 5x plus 3 equals I don't
know this and then get the machine to
always solve for X right that would be
even more complex of an of or at least
likely more I'm not actually sure that
would be more complex for the neural
network to figure out but anyway those
are just some of my ideas for now we're
just going to do simple addition can we
get this to work so the other thing I
want to do I'm actually going to change
this to about 500 just so we can
actually get it more and in fact I kinda
wanna do like a thousand five hundred
should be good enough well we'll go with
500 okay so we've got math data it's
gonna run and it's gonna populate all -
all these little files and the reason
why we're doing all these is just so
that we can so it just mimics what the
chat BOTS wanted so we're just kind of
hacking that chat bot code which is
funny because that actually the chat bot
is actually quite a bit of just a hack
of the neural machine translation so
we're not actually doing any translation
either so we're kind of hacking on top
of a hack here but that's okay so that's
kind of the whole point of this series
so let me save this and let's go ahead
and run it it'll probably take a little
bit to get through everything let's just
move it aside make sure
[Music]
I guess it wouldn't matter but so
hopefully that'll be populated I think
probably I don't know three million
should go pretty quick I feel like it
should be done relatively quickly okay
there it goes cool talking about 18
seconds so now we've got this data in
new data so the next steps that we need
to do just these are just the steps of
working with this package is we need to
go into setup let's go into settings dot
pi so usage is editing with idle anyway
coming back over here so the vocab size
basically we need it to be you've got 0
through 9 so that's 10 and then we have
the addition symbol which is a plus so
that should be 11 total vocab and when
we set it to that that means it will be
character level and it will get nothing
else if you made it bigger it would add
to the vocab numbers it seems more
commonly which in this case would just
kind of be random I don't really see any
point in doing that I don't think you're
gonna get any benefit from that
then for bpe we don't want anything to
do with that Harry now so set that to
false and then the embedded tokenizer
also set that to false these are really
chat these are really language specific
things that we really don't need to be
dealing with here then for the epoch
lines trying to decide if I really want
to keep this I almost think you might
want to do like to epoch set this but I
think actually all I'm gonna do for now
I just kind of want to see in general
how this works like we might not even
finish this to like full accuracy or
anything I'm just curious if it'll work
so we'll do it maybe to epochs at that
in them and in fact let's not even do
that let's just do this and then it goes
to the one and then half so we got one
two three four five total epochs how
much data do we three million or three
hundred thousand three million that
should be good enough
we can always come back later and change
it all so I made test size 500 so we'll
change that to everything else I think
I'll leave oh well we should I change
the size of the network when I see what
I did in my notes I'm not sure I don't
even know if I did change it
let's do I mean two by five twelve
should be fine again this is just like
addition I'm gonna do like five by 128
though and then I don't think I'm gonna
mess with anything else so I'm gonna go
ahead and save settings and then once
we've done that we need to run the
prepared pie or prepare data pie you
can't run that within the subdirectory
though come back out one directory open
up command prompt let's type in Python
setup slash prepare underscore data dot
pie and we'll go ahead and create that
data basically all it does for you is
create your vocab does your embedding
stuff actually I guess because we did I
think it's just two different forms of
embedding anyways it basically just
processes all your data mostly what
we're looking for is vocab and
tokenizing our data so we'll wait for
that I guess everyone's gonna have to
sit through this so might as well sit
together I don't think anyone's gonna go
any quicker than I'm going any more tea
okay all done now once we've done that
we should be good like we should be good
to actually run the train so let's go
ahead Python trained up high of course
we don't have an empty you have to grab
that recursively argh I'm just not like
I'm grabbing it on Windows so so what
you should have done is when you do the
git clone like git clone - - recursive
right that's what you want to do
let me grab that so I didn't even do
that because basically we want just the
nmt package give me one second okay so
I'll pull up that chat bot code we're
just gonna go to this the nmt here I'm
gonna go ahead and clone that let me
pause this and then I'll drag it over
okay I just grabbed it here I'm gonna go
ahead and extract it here so it should
contain this which is the directory and
then hopefully right yeah right in here
so I'm just actually kind of just got
copy that and just move it into here or
in fact actually what we could do let's
just delete this directory and then
we'll just rename this directory to that
okay what you don't have get installs
all right
Python train dot pi take two looks good
here we go
beginning the train train uh-oh ah okay
I always get smacked by this okay so
because we're using a bi-directional
neural network we can't do what I just
attempted to do let me just go ahead and
delete everything here it needs to be an
even number so let's go into setup I
sort I get tagged by that all the time I
really should have learned by now but
anyway so let's do six okay okay all
right take three we got it this time we
can do it come on if you're unlike a
smaller GPU actually you know what this
should fit about any GPU this stuff any
pretty much anybody can follow along
with it you shouldn't use very much vram
at all right now I just want to see it
start stepping through the data and if
it does start stepping up right I kind
of want to like pause and like let it go
through a bunch of training and then
I'll start back or something like that
or unpause the recording I can't really
decide what I want to do there if I want
to just pick up a next tutorial with
with how that goes I think I'm going to
stop it here and then we'll go in the
next tutorial I'll kind of go over the
results with what we had and then figure
out what we want to do from there so
that's what I think we're going to do so
if you have questions comments concerns
something going wrong for you whatever
feel free to leave those below if you're
enjoying this content
feel free to support at Python
programming done that so lash support
otherwise I'll see you in the next
tutorial