what is going on everybody and welcome
back to another unconventional neural
networks tutorial in this video what
we're gonna be doing is going over the
results from our more complex
mathematical models so up to this point
we've been using a character level
sequence the sequence model to do at
least addition which we found was 100%
accurate when we did the inference
testing we noticed there was some weird
inconsistencies there what was going on
is what with the chat bot we have an
additional scoring mechanism that's like
a rule based scoring mechanism that sits
on top of the the output depending on
beam width we can we can actually have a
number of outputs from the chat bot and
then we can pick which one we want to
use and that's what we were doing with
the addition as well but with the
scoring mechanism one of the main things
with scoring as we tend to like longer
results and we penalize shorter ones so
in the model where we were doing like
five plus two or whatever that's
producing a very short result and the
network was kind of being scored
negatively for that and that's why we
had that inconsistency now we could go
through and make a difference a
different inference pie I don't really
see any need this is truly just research
phase and with inference top pie you're
only doing like one at a time whereas
the output dev is truly 500 or a hundred
out of sample tests or however many you
want to have so to me that's that's the
way to go anyways when we're just doing
research so that's why I've just decided
to keep it but that's why we had that
like strange inconsistency there with a
hundred percent accuracy and then
suddenly we're finding instances where
it's not anyway let's get into it so the
first one that we have here is the the
multiple operators so if you in case you
forget we can go into data and check it
out so we'll look at look at the from
first eye and we don't really need to
look at two but anyways so this case
it's a subtraction a multiplication we
got a division problem and the reason
why there's spaces here it's like just
being tokenized hence the character
level sequence sequence here anyway all
the different variants and we're ready
to rhumble so
I'm just going to minimize this and a
couple things all the models including
the hundred percent accurate edition
model the this model here the multiple
operators and then finally the much more
complex math model all of them I'm going
to upload to Python program internet so
you can kind of play with them I'll try
to remember to upload them with the
settings file so that you could run it
actually but if I forget someone remind
me and I'll throw the settings into it
so this is the results um basically
right up here it takes me forever to
scroll this page let me just do a tiny
bit hopefully it'll there it goes rather
than doing it via the the Python files
with the output dev what what we did
here is this time it's actually just
tied in to tensor board that so we can
track these along the way so so all of
these charts here are over time from
start to the very end of what the
difference was so obviously like with
especially with multiplication the
magnitude of being wrong on
multiplication is very much different
than the magnitude of probably being
wrong with like addition or subtraction
so we didn't want to just have a general
how how how absolute valued off are we
necessarily like we did this is math
total difference like we do track that
as well but we're kind of curious really
about these individually as well just to
see if they're all learning or what's
going wrong so in addition is one above
there's so many plots here though that
it is really really laggy so I don't
want to scroll up but anyways it's there
and then what we have here is so the
total difference but then also the total
accuracy so as you can see took a little
bit then suddenly it jumps up prior
learning rate decrease there and then
eventually just kind of leveled out
despite learning rate decreases so I
just kind of stopped the model at this
point could keep going maybe it was
gonna learn something more I'm not
really sure but I'm actually pretty
impressed with a forty five percent
accuracy that's pretty good
so anyways so that's that I don't really
want to harp too long on this one
because this one is not as cool to me
I'm pretty confident we could come up
with some sort of model that would learn
this 100% but what I really wanted to do
was try far more complex types of math
than just this because if we could solve
the far more complex math with some sort
of model then that should also solve
this problem so let's move on to the far
more complex variant here and let me
just pull down our paper space so close
that and I guess we'll just have to keep
it this way so this is the these are
like all the charts basically for the
more complex one now let me pull up an
example let me see if I can't find here
we go so let me just pull up a test from
so in fact that this so I'm gonna pull
up one that's not tokenized because that
just it's harder for us to read it so
I'm just gonna go into new data instead
test from oh man is it goofing on me man
come on
there it goes anyway okay so this is
these are the types of operations now
that we're trying to solve so not only
do we have multiple types of operators
we've got multiple lengths of sequences
although these are all five
interestingly enough but this one's
there's a at least this one's for long
and so is this anyway but we also have
like parentheses in here and different
operations in the parentheses we've got
multiple parentheses you can see there's
quite a bit of embedding going on in
this one these are cool if we could
solve this kind of math that would be
impressive so so now that was the that's
the input basically to the model this
time and obviously the output or the you
know the to file is just the answer to
these which is not that interesting it's
just the right answer this is that model
after I think we're on step like 700,000
something it looks like it's about you
know maybe 750 thousand where we at 741
so 741
in steps and we're tracking all the same
things we're tracking the ads the divs
the moles the subs and then the total
difference now the total there is a
little hard to see ESO is mole's one
option we have is to try to zoom in a
little bit the problem is these values
were just so freakin huge that they
confused the chart anyway so you can
zoom in a little bit and start to kind
of get an idea of what that graph looks
like also the total difference one I can
try to zoom in so it's so laggy with
this thing's been up for so long so
anyway you can at least get the idea
they are declining over time the total
accuracy here is a mere 1.2 percent now
in the grand scheme of things with the
fact that this is a character level
sequence sequence in theory it could be
an infinite number of outputs getting
1.2 percent accuracy
he's pretty exciting to me I actually
didn't think that this would work at all
I didn't think it would train at all
so actually 1.2 percent I'm I'm pretty
impressed the fact - that the total
difference is in steady decline and it's
definitely learning things also we can
pull up some of the comparisons so I
went ahead and just pulled in the sum of
the comparing files from before just so
that we can kind of look at what the
intended output was compared to what the
actual output was so this is just a
really basic file from one of the
previous tutorials so it's just gonna
load up output dev 5,000 so it's just a
bits all 500 of them so we can see this
is after 5,000 steps how it was doing
had zero percent accuracy on here but
interestingly enough almost immediately
we actually see that it's not super far
off like I mean well okay this one is
actually pretty far off that's actually
one of the worst ones I've seen to be
honest usually it's at least somewhere
in the right order of magnitude but
actually a few of these are quite a bit
off this is more along the lines of what
I kind of expected the best of this
model to be I just didn't I feel like I
felt like this is
very challenging for a neural network to
be able to learn all the intricacies of
how to do addition how to do division
the parentheses and like combining all
these operations that is curious to me
that it's able to do that so so anyway
so that was only 5,000 steps though as I
showed you guys we have like 700,000 so
let's just jump to let's do let's jump
to like a hundred thousand or whatever's
close to that so a hundred thousand 125
so let me do and again I'll put this
model up I'll put this model up on
Python program in net is 500 megabytes
of a model it's quite a large model so
just take that in consideration but you
could run it off a CPU you know running
live on a CPU is really no big deal it's
mostly just training on a CPU that's a
pain in the butt okay so let's run this
one okay so here we have this one still
with zero percent accurate we can see
that this one it just got way way way
wrong but a lot of these are actually
pretty close and you got to kind of hand
it to the model to an extent these are
really long decimals so so yeah but you
can already tell that it is at least
getting in the ballpark
you know 170 thousand six hundred verse
170 thousand four hundred over here 208
well I think it's like 2 million right
am i blind that's about two million
something and then this one is again
about two million something so if this
this AI was on a multiple choice test it
probably would do okay and yeah so so so
at least it's it's it's getting there
it's getting closer to things right and
then let's just go ahead and jump it all
the way to the latest which is 720 to
125 so 720 to
make it nice and big here okay so in
this case after this was 0.1 I forget if
I round it do I actually round it
possibly correct no it's not really
rounded so at least in this case it was
only 1% and then like the best it's ever
got is like 1.2 not a huge difference
but anyway we can already see here I
forget which ones which one it was
historically that seemed to always get
the magnitude wrong it was one of these
shorter one of these ones down here well
this one looks pretty bad anyway we can
continue to kind of like look at some of
these and you should just be able to see
but also if we look back at the I'll
pull it back up here in a minute but the
the tensor board log of the differences
and all that like it gets pretty good I
mean it's pretty shocking to me that
over time it's able to slowly hone in
I'm not really sure you know that first
of all this model has been training for
a week at least on paper space so so I
probably gonna stop it at some point
it's a costly costly endeavor but I've
already dropped the learning rate quite
a bit I'm not really sure I want to keep
dropping it but at the same time it is
it does appear to me to continue to keep
learning and I I'd really like to see
how far we could take this model so
we'll see but I don't think that like I
don't think we're gonna suddenly jump up
to above 5% so I think the next step is
to come up with a superior model that
we'll learn quicker than this one does
so like in the first let's say 10,000 or
100,000 steps is it is it has it made
more progress in this model if so I'll
let that one continue and we'll see
where we can get it so anyway that's
kind of my plan going forward but it
takes such a long time to keep playing
around these models so I'm just going to
kind of keep playing with them I
probably won't make an update video
every time I poke around and in the
meantime we're gonna probably jump into
a different topic entirely I'd like to
play around with sound so there's all
kinds of fun things that we can do with
generating sound so that's kind of what
I'm gonna be looking into
next if you guys have any suggestions
about what kind of model size and shape
and stuff that we should use here by all
means make the suggestion this is also
bi-directional neural network I don't
know if that's actually beneficial for
us to be doing so could try to take that
away let me just pull up the settings
for this one real quick set up settings
so yeah the vocab size 18 test size 500
these were the epochs that I set up here
yes so in this case it's a 10 by 512 so
quite the large model it might even be
the case that 10 by 512 is too big so a
lot of times you make the model too big
it's just too challenging to actually
learn so probably the next step I would
do is actually shrink the size of the
network and see if that helps us at all
so that's probably the first thing that
I'm gonna actually do but anyway I'm
actually pretty excited about this
result I think that's really cool that
it's capable of learning a pretty
complex there's not a direct like relate
there's so much going on in the input to
this and that it can slowly actually get
more and more accurate especially given
the fact that we have so many inputs and
so many plausible outputs especially
like the number of combinations that you
could have here
it can't be brute-force right as though
so the fact that it can actually learn
this style of math is very intriguing I
that's cool I mean I just I think that's
really cool and I think it would be
super cool if you could get a hundred
percent accuracy out of it or even like
50 percent
I'd be like ecstatic with finding a
model that could get even like 50
percent accuracy on this kind of an
equation that'd be pretty cool and then
eventually if I can get a good math
model going I'd like to start playing
around with weaker forms of encryption
just see one can a neural network
reliably do encryption like to make
hache or something like that that you
could actually rely on and then also
could it break less strong forms of
encryption or hashes anyway that's it
for now if you guys have suggestions I
doubt anybody else is running one of
these models but if you do and you
happen to find something that's a little
more accurate let me know anyways that's
it for now hope you guys have enjoyed
like I said we're probably get into
sound and stuff like that next that way
more people could hopefully follow along
I just think this one is clearly a very
challenging task for a neural network
whereas with sound we can definitely get
more into something that everybody can
do on their computer as well so anyways
that's what we're going to be doing in
the coming tutorials hope you guys are
enjoying this series I'm having a good
time making it questions comments
concerns whatever foe feel even below if
you like the content you want to support
the channel this is my full-time job you
can do so at Python programming and that
slash support otherwise I will see you
in another video