what is going on everybody and welcome
to part four of our fun with neural
networks tutorial series in the last
tutorial we began simplifying our input
vector here but we left off with the
understanding that hey this is a
character level generation model and
generative model and really everything
is a string so we don't actually have to
adhere to any array or any other
structure at all we can change this to
be anything we want so let's make some
of those changes and then build our
training set and then hopefully all in
the same tutorial begin training and see
what we can come up with
so to begin and actually I think I'm
going to make a slight change to the
text-based version of this tutorial
and then we'll see which one does better
because actually I have an idea of how I
want to change it so I'm actually going
to change it anyway let's get started so
so the first thing is we want to like
what it break right here we can print
what if we print string pixels
[Music]
okay string pixels so that will convert
pixels to a string and then we're gonna
rip dot replace any spaces with no
spaces run it ship it ready this is the
zoom out cuz we'll be ready to see it
what did ugly I guess that's a seven cs7
okay so there's our data set I mean that
was a hard number to recognize let's
just do another one real quick but you
could you could recognize it still okay
there's a three it's clearly a three and
now the data is even more condensed we
saved like 30% of our space actually
more than that probably half of the
space or close to half because there was
a space for everything so we just saved
a ton of space okay so what we want to
do now is begin to
ooh like so we could do the same thing
with the label but I'm not gonna waste
time with that just understanding that
that's what we're going to do so now
what I want to do is we want to build
this training set so how are we gonna do
that so if we look in the text-based
version of the tutorial basically what I
wanted to do or my idea was to take you
know this bit right here so this would
come first and then it would also the
number would come first and then would
always be followed with whatever the
class of that number is so then the the
my proposal is that we could take any
number right from the validation set
convert it just in this exact same way
feed that in as the primer and then the
output should be the one hot vector and
in fact we could even leave spaces in
that one but I'm not gonna do that I'm
gonna keep that so in the text-based
version of this tutorial
we didn't convert it out to be 2 2 to a
1 hunt vector now since then and since I
did the generative model actually did
the generative model in a slightly
different way and I'm curious if I
should do that do that method instead I
don't think it's going to make any
difference the other way be it's really
challenging I really don't know I'm
gonna try it because it worked the other
way so I'm gonna reverse it I know that
doesn't mean anything to you at the
moment but like I said I'm gonna change
the text-based version so so yeah so
basically our training data will look
just like this only this will be
condensed as well and then you just do
that maybe new line new line for the
next classification and just keep doing
that so we're gonna build a training set
that looks like that so let me let's do
I kind of want to since I already know
in our Emnes data set I'm gonna do two
things one is we're gonna try to
classify numbers and the other we're
gonna try to go the other way where we
say hey draw this number so I'm gonna
make a directory called classify heading
back into our script now what I'd like
to do is I think we'll keep all of this
but I'm actually going to I'm just going
to delete all of this code and we're
going to write out the new version that
we're going to be working with so I'm
actually going to say next batch
we'll do 10,000 samples and let's just
do it all like let's just do all the all
the samples I think it's like I can't
remember I want to say there's only
40,000 I'll just do this and we'll do
all of them whatever that number happens
to be so now what we're gonna do is I'm
just gonna fix that and zoom in scroll
down a little bit and begin so we're
gonna say with open classify slash input
text with the intention to append as F
then we're going to say for I data in
and numerate and in fact I'm turneth
yeah it enumerates batch X's what do we
want to do well we also could zip these
together and then iterate through them
I'm just doing it this way so I don't
have to do that but that would actually
be the pythonic way to do it anyway
you'll see what I mean
data equals NP dot round int batch X's
X's I and then dot as type int the label
is equal to basically the exact same
thing so copy paste don't forget to do
wise and you're gonna go now pixels is
going to be data dot reshape 28 by 28
don't forget to put that in its own a
little tuple why did you do that that's
not okay
thank you sir now we're going to say now
we convert these to a string so we're
gonna say string label is equal to
string label so mmm so this is kind of
what I want to change I think if I want
to do this or not
I wonder straightly Oh
I think what I would like to do is sorry
I'm trying to trying to decide how I
want to do this I want to say I think
I'm gonna do this NP arg max the label
and then I'm trying to remember how you
can make because you could do like MP
zeros ones like I just want to make a
long array of what the number actually
is I don't know that this is actually
gonna work the other way let me pause
this for a second while I think about
how I want to make this change okay what
I'm gonna do is yeah index underscore
value is gonna be n PETA Arg max of
whatever the label is norris a new
underscore label is equal to the MP dot
array of 100 times whatever the index
value is so to be a life sign the index
value let's say it's a to the index
value will be a two and then a hundred
times that will be just like these array
that's a hundred twos okay
then what we're gonna do is we could
leave it that way but I'm gonna issue a
dot reshape and we'll reshape it to a
ten ten by ten so that's our new label
and at this point it should be already
as this so it'll look kind of like this
only this isn't quite it yeah I'll just
do will print this out so let's just
print a new label whoops
and then we're importing just sleeps for
a very long time just so it'll pause
right there instead of iterating through
the data so in this case it was a five
so it'll just kind of look like that
okay so that's our new label so I'm
gonna use that now
so what I'm gonna say now is string
image so STR image is equal to string
pixels dot replace and then we're gonna
replace all of the spaces with no spaces
and then we're gonna do the exact same
thing to string labels so string label
string new label and we'll replace them
again in the text-based version I
actually just used the one-hot array and
we convert it and then we get rid of the
spaces but I'm curious to see the
results of doing it this way I'm just
curious
I think it's more robust so when you use
the one-hot array it only has like one
opportunity to get it right and if it
screws up one boom the whole game is
over so I'm curious if this will make it
better
or not I really don't know I don't have
the answer to that we're gonna find out
so then what we're gonna have is we're
gonna say classify underscore data so
classify data equals and then we're
gonna say I kind of want to just copy
and paste this one but okay string
formatting followed by a single new line
and then we're gonna have the
classification now I'm trying to stop if
I really want to have this or not if I
want to do because sometimes you want to
like have some sort of dictation of what
what what's your classification versus
what is your input data because they're
both going to be array values I think I
will I think I was just like encase it
in like a colon or something like that
so so yeah so we'll do newline followed
by a colon some more string formatting
another colon and then we'll just do new
line new line so then dot format and
then it's the image followed by the
labels this is a classification tasks a
string image
string label okay and then the other way
around would be you know like to do
generative you would actually you would
start with the string label followed by
the image now what I'm gonna say is F
dot right classify underscore data looks
good let's go ahead and remove the sleep
and then we'll run it and see what
happens
cool so run hopefully no errors and then
as that's running let's check the output
and make sure that's it's kind of what
we're expecting let's just make sure
that runs did we always know we're not
printing anything so inside
classification we have input text it
doesn't appear to be growing it should
grow but maybe because we opened it with
the intention do we open with the
intention right no it's a pend wonder
why it wouldn't go we probably should've
just done a hundred to see if this
actually works or not hmm I'm not
content to know if this is done or part
of me wants to pause it I'm just not
sure we should have picked like a
smaller smaller number first to test and
make sure this a even works let me pause
it I'll do a smaller number and see what
happens okay I don't know it just got
kind of funky or something on me but it
looks like it's done now I'm probably
going to reload it maybe redo it for I'm
not sure if it really finished if it
really got all the samples or not but
you can see it's it's actually quite the
large quite the large data set mine is a
hundred and twelve megabytes in size so
quite a bit larger than the other sets
that we've been using up to this point
now it looks like you know so basically
you've got the number and then you're
hoping basically to use the number as
the primer and then you're hoping that
it would produce output like this that's
the hope that's the curiosity I actually
don't know the answer to this question
because I've made a live chain
the other one sort of worked but it was
it was it was not acceptable so I'm
curious to run through this one to see
if this actually produces a better
results or not
so anyways now what we need to do is
actually train this model so what we
need to do is we're gonna just you know
it's an amnesty amnesty and classify
that's our data set and then we also
want to go ahead and we'd like to have a
pretty large sequence length I let me
think I'm gonna look at the the one that
I'd used for the generative model I want
to say it's like five hundred or
something is what I went with and again
anybody with a lower end GPU is not
going to be able to do this trying to
see what I used oh I actually used to
800 a 128 by three with the bat hello
with a batch size of okay I'm gonna try
it I'm gonna try it cuz I I've tried
that one before so so what I'm gonna run
is after I'm done bashing my glasses off
my face I'm going to so we've got the
input that types ready to go let's do it
so let's go into the root dirt with
train dot pi and let's run command right
click so trained up PI and rather than M
is draw two I'm going to do but classify
it's gonna be a 128 by three with a
twenty eight batch size in a sequence
length of 800 and that's the one I want
to Train and we'll see how that goes so
yeah I'm gonna go ahead and train that
one and I guess I'll probably come back
in the next video and what kind of data
mmm oh it's eminence dataset I was like
what and this data set like that
try that see if that works again
depending on your GPU and all that you
may or may not be able to run such a
thing like an 800 sequence length is
quite long but yeah let me just make
sure that starts to train and as long as
it does I'm gonna break here and then
I'll pick back up in the next tutorial
with the results and we'll kind of
iterate through it and test it and see
how it did I think I'll just pause it
and then see if it starts going it's a
huge data set so it's probably how it's
gonna take a while I'll get through it
all and create the vocab and all that
okay so I was actually just absolutely
blowing memory and Tom so I really
wanted to test this with at least 700
plus sequence length and I just couldn't
I didn't want to make the batch size
smaller than 28 so I launched up my
paper space with a p6000 GPU which gives
me 24 gigabytes of vram and it is now
training make it whatever size you want
and whatever size will fit normally at
this point I doubt the paper space and
give a referral link and all that but
this is prot like this is strictly
research there I I don't have very high
confidence that this will work so you
don't really need to follow along you
can just kind of see what my results are
but anyways it's off its going to take a
very long time to go through 50 epochs
I'm not really sure I'm going to wait
for all 50 but I'll probably let it
train for for quite a while I'm gonna
open in terminal here tensor board
logger equals longs and I'll probably
just watch the graph and see open link
open link one of those should work oh I
just hit cancel here please thank you
close this well that's interesting it's
a very weird hop there
oh because we've moved oh this is oh my
yeah okay that's all the same so here is
our actual loss Wow
wow that that dropped a lot really
quickly that's interesting hmm anyway
yeah okay I'm gonna stop now that's
gonna take a really long time I don't
really want to do necessarily the math
but basically we've got to do all of
those batches like two hundred and fifty
six thousand batches one second actually
okay so this will take at this rate 56.8
hours to complete so over two days of
training to get the answer there so I
doubt I'm gonna let it go for an entire
two days of training but this seems like
a pretty good place to stop I might even
leave this training and then carry on
with the next phase of this tutorial and
then that way I can actually do be doing
multiple things at once and then we can
check the results later because I don't
think many people are gonna be able to
even train this model so most people
probably aren't even going to be
following exactly along with me anyways
that's all for now if you like this
content you wanna support me in the
content you know the deal with Python
programming that slash support otherwise
if you got questions comments concerns
whatever feel free to even below
otherwise I'll see you in the next
tutorial