what is going on everybody and welcome
back to the unconventional neural
networks series in this video what we're
doing is running over some of the
results from the neural network that was
trained just to do addition we're going
to talk about some statistics we're
going to see how it did and then what
we're going to be looking looking to do
moving forward so with that let's hop in
first of all how did we do so one thing
I did was I wrote a few different kind
of scripts just for comparing how we're
doing as time went on so so one thing we
we can do is inside model we should have
the train log directories so one option
we have is to just do CMD here and
induce you Python or actually sorry
tensor tensor board logged er equals
Train underscore log run that and then
we want to open this so let me just pull
that up over here so these are the
results so we can we can obviously see
how loss was doing so actually this
model could have continued to Train well
out actually I'm sorry this probably
isn't fully loaded yet we've prideand -
wait a minute
yes you can see it's slowly starting to
level out we'll just we'll count need to
wait for that - to finish processing so
one way that we can see how a model is
doing is to look at train log but part
of the issue is with with loss the loss
is really all about the weights and when
we're looking at something like a
mathematic problem or in our case like
when we're doing the the nmt chatbot
even there's a reason why we used other
metrics like blue and ppl because loss
could mean sure we're learning something
but are we learning the thing we were
hoping to learn that might be a
different question so you often want to
come up with a separate metric for that
so what I did was just manually as this
model trained I just took the output
as they came through and I just renamed
them quickly so as they were coming I
would just say at what step point was
that dev so at five K steps 10 K steps
and so on all the way to 257 K when it
finished raining after about a hundred K
I just went to bed so that's why I
didn't I didn't get anything in between
there
hopefully our tensor board is is done
now so we can kind of pull that up and
I'm just gonna refresh it so let's just
open up loss okay so so we can actually
see after about 120 K steps it looks
like I'm gonna guess that the learning
rate was taken down there as well let's
see at 120 Oh actually yeah so right
before I went to bed I recognized that
the learning rate was 1e negative 5 or
it should have been 1 negative 5 yeah
wonder why that says 1u negative 4 weird
oh this is negative 1 e negative 4 so
the middle ground here is 20 negative 5
okay anyway before I went to bed that
was gonna be the last epoch and I
thought mmm it was still only about 50%
accurate so I decided you know what
let's add some more
EEP ox and when I did that it jumped
back up so interestingly enough we were
actually kind of done learning
then I'd it to jump back up and look how
we just boom he just jumped way down in
loss so if anything looking back this is
actually the first time I'm looking at
this that jump up in learning rate
basically what that tells me is we
jumped down way too early
we shouldn't have jumped down that early
because putting it back up like we were
we were fixing a plateau right there
basically but allowing it to jump back
up again comes back down here and and
that's probably what sealed the deal so
cool I'll keep that in mind because
we're gonna make things a little more
complex here so cool so that's that's
the tensor board but that doesn't tell
us you know learning rate doesn't tell
us did it actually learn math so instead
what I used was the output dev file so
just open one of those up
let's do sublime text and we can see
okay that's basically the output from
then the model and it's because it's a
character level model the spaces aren't
actually spaces just spaces between the
outputs now the input to these was
basically would be like new data and
then from so from this we should get
this is that correct that's probably
right
anyway yeah yeah all right so yeah from
here we should get this obviously minus
the spaces and the question is is this
correct or is it incorrect now even at
five thousand steps I just I know that
this is probably about Crips I know it's
about correct because they're pretty
they're at least close they're within
the same range the same ballpark right
but not all these are correct so what I
did was I just kind of manually saved
those out put Debs
as they go so those were just that's
like our way of validating things so
every 5,000 steps what are about 5,000
steps it would go to the these testing
ones which contain samples out of so
they're out of sample testing from like
the training data itself so these should
be samples that the model has never seen
and this is how it does on those so what
I did was at least started with this
file so let me open that up and we'll
talk about that so so this one opens dev
10k but we can start with 5k even and
what it's going to do is it's going to
open up the the test file so test from
it's going to open up the output file
and that's going to open up test dot -
so what was it supposed to be and then
what we're going to do is just create a
couple of counters here we're gonna
follow pep eight oh we should put our
spaces here okay and your anyway what
we're going to do is we're going to
count basically how often were we
correct versus how many total samples
did we have in the set and and then we
can get an actual accuracy rating so
that was like the first thing I started
doing man I got a lot of Pepe Pepe
tissues here
every now in a while someone says my
coat is ugly and I say you're banned
from the channel so let's go ahead and
run this so as we can see here basically
what the output of this is what the
input was what our desired output was
and what the model said hey this is the
output of like this like the output of
forty three point eight thousand plus
ninety eight point eight basically it
should have been one forty two thousand
and some change and we said one forty
four so actually almost immediately we
can see hey this this model is at least
on the right track but at this stage in
the game to me I'm starting to think you
know is it only getting close because
it's basically brute-forcing this and
it's it's finding another example where
it was close to forty three 808 plus
something close to ninety eight thousand
got millions of samples is it just
getting close because it's just like
putting the output of what that was back
it back in that example and so at least
my line of thinking here is one that
could be the case so long as accuracy
you know actual where are you actual
accuracy doesn't improve that's one simp
that's one symptom of maybe it's just
brute force the other symptom of brute
force is doing something like we're
gonna do later which would be tracking
the total absolute difference of the
model out versus that desired out so so
so not only should accuracy be rising
over time but the total difference
between accuracy and what we intended
kind of like loss only this is much more
exact to our task we'd like to see that
going down as well over time and if that
starts going down over time we can be
relatively comfortable that no it's
actually seeming to learn some sort of
math here and then of course at the
final stage of the game if it is able to
get any accuracy so the zero point zero
means it got none of them right if it's
able to learn any accuracy and that
starts going up then we're to assume
that it's learned math in in my argument
further is that we had a hundred
thousand so it's zero to a hundred
thousand
plus zero to a hundred thousand so
that's a hundred thousand combos here a
hundred thousand combos here to get the
total number of combinations on both
sides of that plus sign that would be a
hundred thousand times 100,000 which
should come out to ten billion possible
combinations now the output isn't ten
billion possible combinations of output
but as far as inputs concerned that's
ten billion and we only fed through I
think three million so someone can feel
free to correct me sometimes I get those
these kinds of scales wrong and stuff
like that but to my understanding like
you know you could have zero to a
hundred thousand here so it's a hundred
thousand and then even here you've got
five and they've got a hundred thousand
examples of five plus what so 100
thousand times a hundred thousand ten
billion so it can't in my mind be brute
force either there but it is simple math
it's just addition okay let's see how we
did later on so that was 5k how about at
I don't remember I think I had a 15k
let's just test it real quick so even at
15k we got some yeas in there and it's
one point two percent you might be
asking how is at one point two we have
five hundred samples so that's why so
anyways we have some suddenly we're
making some accuracy now like I was
saying the other thing that you might
want to start doing is tracking the
total difference let's see if we're yeah
I'm doing that in this one so in this
one we start tracking not only the total
accuracy but also the total difference
so so starting from let's just do 5k and
then we'll jump to 25k at 5k the total
difference so the you know think of it
again it's just like the loss the total
is almost 400,000 but by the time we
jump to model after 25k steps the total
loss or the total difference I'm calling
it loss but because that's kind of what
it is in my mind for this exact problem
is not lost
anyway the total difference that it was
off was only twenty one thousand that's
a massive improvement after just a
twenty thousand steps now I know
everybody's waiting for the final
result so let me open up comparing
number three and basically what I did
here is rather than printing and you
know the actual desired out in the model
out I'm just running through well where
did it go
oh I opened it Nidal I'm so used to
using Idol let's open it insu Blaine so
in this one I just iterate over I was
really mad but I iterate over all of the
checkpoints that we have so we can kind
of see how it does over time so my
apologies I just have to fix it and then
pretty soon it's gonna get angry at me
over violating 78 anyway let's go and
run it and as we can see the correct
rate was zero here the sum of all the
difference is 400 K very quickly it
drunk jumps down all the way down to
actually after the full 257 K steps
although there's quite the gap between
103 and 257 so I went to sleep we got a
hundred percent accuracy so this model
again there's 10 billion possible
combinations the model got 100 percent
accuracy on the out-of-sample testing no
difference obviously because of hundred
percent accuracy so we can definitely
train a model to do math so this was
actually really surprising to me to get
a perfect score through that and the
other thing we can do is we can actually
run the inference I haven't actually
done this so so if we find that it's
actually not accurate I'm gonna be
pretty pretty angry on video but let's
go ahead and run it so what's the worst
that could go wrong so this is just
running inference top pie now let's just
do like a five six seven plus that I
don't even know what the answer should
be
let's bought the old trusty calculator
where's the old calculator so what do we
even put in five six seven five six
seven plus 42 14 47 81 so the models top
result was 47 81 now the way this works
is it has a thing called the beam
being with and that's how many
predictions it's gonna plausibly make
and then from there you score them
basically this is the models top choice
but you can see the model was thinking
about maybe these other things too but
it shows the top result as 47 81 that's
when we go with we could do something
super simple
oh so this one that actually gets wrong
five plus two it believed it was two
sevens actually all the way down here
I'm gonna go ahead and guess it's
probably not that great at simple math
or like small numbers because I bet
small numbers were just like super rare
that should work yeah interesting so so
with large ones it works what about 500
where us do 5,000 plus seven who would
guess that one wrong - this is really
interesting that it's getting so many
wrong yet in the sample testing it got a
perfect score I just wonder if the
things that I'm thinking to type in are
just rant you know so not likely let's
try it I'm curious does it just never
did get a single digits right I've got
the nine right so this should go over a
hundred thousand yeah I don't even know
if that's a correct answer let's check
it so let's say actually I can't even
tell I don't know if you guys can see
this yes you can okay so that would be
95 41 plus 24 - 11 119 632 so it appears
to be pretty good at large number
addition but I bet it's just so rare and
the training data that it gets small
numbers let's try 7 plus 107 plus 90 how
about 91 blah blah blah so should be 91
257 yeah so what that's interesting so
it's like little numbers that it gets
wrong hmm let's try two digits 257 wrong
did it get 72 in any of these so none of
the outputs did it think 72 so this one
has learned multiple addition with large
numbers and that's also wrong Wow how do
we get 100
my accuracy on 500 samples and yet fail
so many times through the inference I
wonder if we have a difference in how
the inference runs verses 6 that's right
huh I don't know maybe because like the
same I don't know
I can only I can only guess why why
that's acting so much different it's
only for the small numbers though for
the most part I think we missed one up
here too though right this one what
about so this should have been 852 is
that even in its got 8:51 no that's not
even in there anyway regardless probably
with more training data and more kind of
manipulation
I suspect I could I could get a model to
be perfect this one was just actually
meant to just be a simple test I never
actually intended even fully trained
this model but I was curious I'm gonna
go ahead and say this is mostly solved
it's definitely solvable I think with
some tweaking we could get it to be
perfect even on the little numbers but
now what I want to talk about is kind of
moving forward into the next challenge
so there's kind of two two challenges
I'd like to consider one would be just
simply doing so like this was just
addition which is relatively simple I
think the next task would be something a
little more complex which would be doing
all of the symbols so let me open up
math out of here there we go I'm sublime
pull this over so in this case I've just
slightly modified to choose a random
choice both in the training data and the
testing data and then just chooses adds
a motive one of those and then what it
does is create the training data and
I'll just show you a quick example from
into I don't know where it went here
yeah so it's got some subtraction
multiplication division addition it's
got all of them and then obviously all
the answer
so with with that the question becomes
now in this case I think I did ten
million samples let me pull it up again
real quick I don't underst soap so used
to it yeah ten million so that's the one
I'm gonna start running right now also
even more complex and I'm going to just
probably compare the results in the next
video all in the same video also sample
code I'll post if I forget
I plan to upload the models so the model
for that I just showed you guys just the
pure addition I'm gonna upload that
model it shouldn't be too gigantic let's
see 20 megabytes yeah so I should I'll
upload that one so you guys can come
play around with it if you want without
having to train it yourselves and then
these ones I'll probably upload if
there's anything good about them but
anyways getting back into these so I've
actually what this has or this one's
already started training and what I'm
gonna do is I'm going to show you what
they look like now so model and then
we'll come in here no it's tensor Board
tensor board log dirt train underscore
lorg trail org perfect okay so this one
is just simply one type of number
followed by one type of the for
plausible operators followed by another
number again max 100000 and what I've
got going on here is basically the graph
of not only the total so here this would
be the total difference right that that
total difference that I was showing you
before where it started off as four
hundred thousand that got lower this is
the total difference and then what I
decided was probably smartest was to
track the difference of each of these
separately so the difference of the the
multiplications right the
multiplications are gonna make much
grander numbers then say division or
subtraction especially subtraction
implausibly division like we can even
see division is is very it's already
dropping into something
reasonable whereas these are still you
know at certain exponents so anyway so
that's how this one's going up I'm only
6,000 steps in so we still have quite a
bit of training to go but that's how
that model is doing and now I'm going to
show you the most complex version and
the most complex model that we've got to
date and this one is basically the last
step before I start mucking around with
with doing encryption so and by
encryption I mean both trying to either
break encryption but also even just
produce encryption so to actually
encrypt things and decrypt things with a
neural network rather than some sort of
strict rules yes that could go wrong
very quickly anyway it's just for fun so
this is a far more complex model we can
see the the numbers are much much higher
what's the difference so let's bring up
the difference so this one creates much
much more complex mathematical equations
so let me pull up our room lost actually
I think I want to do new data and then
that's like weak well we can check it
out but and again I'll probably post
this code if I forget someone remind me
because we're running through so many
things right now it's just that if we
were to type this all out by hand
one I'm not sure there's any value to it
still but also it would just take like
ten videos before we could actually just
get to the fun part and this is really
just more about having fun so test two
let's open that one up
so this is what the results that were
attempting to draw and I don't know if I
can I can't zoom it with just a plus but
this is what we're coming from so these
are the equations that we're generating
I'll use Adams so I can zoom in
okay so so as you can see these are far
more complex it's not just one number
and another number it's many numbers so
when it's got to learn PEMDAS right and
then it's also gonna have to learn
so not just PEMDAS like the order of
operations without parentheses but we've
also thrown in parentheses to the
equations and as you can see some of
these equations get quite long so let me
also zoom in a little bit cool and so
there's a variable number of numbers
there's a variable number of operators
and types of operators and then we've
got parentheses and all that so the
question is can the model learn this
something like this it's far far more
complex now from what I can tell again
we until we start seeing like we do see
some improvement here until I start
seeing total accuracy doing something
like right now we're missing everything
until I start seeing accuracy actually
ticking up I'm still a little skeptical
that it's not just brute force and it's
not just finding similar examples so we
really need to see all of the above we
need to see totals dropping like the
totals of the multiplications dropping
like these are all differences by the
way so the difference of output versus
desired output the absolute difference
dropping multiplication dropping ads is
dropping subs is dropping seems to be
leveling out though unfortunately but
anyway this is still very it has it we
haven't even dropped learning rate yet
we're still at the first learning rate
anyway I want to see that happening of
course I want to see the learning rate
continuing to drop down let me see if I
can pull it up I think it was probably
on that other graph actually not
learning rate I'm sorry loss rate where
is loss I'm lost it must be on the
second page let's try next page please
there's our loss oh it's super messy
let's smooth that that out okay so
actually loss is doing quite well
smoothed out anyways it is dropping
still so anyway these are the two things
you guys have to look forward to two
different models are being trained
simultaneously and we will check the
results on
both of them in the next video this
video is clearly getting way too long so
anyways I'm going to cut it here if you
have questions comments concerns if I
forgot to put up one of these models
we're missing some some code or whatever
you want to see the script and it's not
in the text-based version of this
tutorial let me know I'll put it there
if you've got questions comments
concerns suggestions for improvement you
feel the desire to support head to
Python programming dinettes slash
support otherwise I'll see you in the
next tutorials where hopefully we have a
model that does these calculations and
then we can go to encryption taking over
the world that sort of thing see you
next time