what is going on everybody and welcome
to part three of our experimenting with
neural networks in this tutorial we're
going to be revisiting the past with the
amnesty data set and everyone's like oh
god not M missed again
well hopefully this will be something a
little different than you're used to
with the eminence data set so first I'm
going to start just create a new
directory and I'm gonna call it amnesty
data and this data set and we'll come
into here create a new file and then
we'll call this M this data creation dot
PI cool and we'll open in sublime and
we'll get started so first of all to use
M mist we want to do from let me zoom in
a little bit from tensor flow dot
examples dot tutorials dot M inist
import input data and then we're going
to say the you know the MMS data set
itself is equal to input underscore data
read underscore data sets and then where
do we want to read those two we'll read
them to M nest
base Oh let's read it to NS data and
then not a period it was supposed to be
a you / M this data and then we're gonna
say one underscore hot equals true that
way it's just it returns like a vector
with ten options and the one that's true
is the index that's one and then the
rest are zeros now to iterate through
this all this stuff should be stuff
you've seen before if you followed mine
or anyone elses pretty much deep
learning tutorial or really just you
could use the amnesty to set for a lot
of stuff actually anyway m-miss train
dots next batch 100 so you would use
batches if you're training like a neural
network or something like that in this
case it doesn't really matter but we can
use next batch to yield ourselves the
next set of training that we want to use
in this case 100 and that's it we're
just going to use 100 that's how we're
going to decide how many Sam
in total we want to have conversely you
could pass you know like a number like
that I don't they they don't have that
many just for the record I think it's
like 40,000 and then 10 and 10 or
something like that I forget the exact
numbers but anyway so we'll just use 100
just for the record there's there's ma
strain test and then there's a dot
validate for our purposes we can
actually use train and tests because
we're not gonna actually try to test
anything during training anyway so
really you just would you would probably
want to leave validated just to use as
out of sample testing but otherwise you
can actually use both of these but we
don't have to that'll create a lot of
data plenty for us to work with so the
next thing is yeah you got batch X's
batch Y's and then we could say like the
data for the you know the input data
basically would be batch X's and then we
could say it's the zeroeth and then we
could say the late
oops the label the label is batch wise
data label batch wise so now what we
could do is we can print data and print
the label but of course you know you're
probably not going to really know what
you're looking at also yes so this is
going to download everything functions
for update it looks like there's some
sort of something's going to be outdated
pretty soon with grabbing this eminence
data just follow the deprecation warning
if you get it or the error if you get it
but otherwise I'm just gonna continue
along so anyways we can see that this is
our data for the image or the number and
then down here we can see it's a 0 we
can't really tell that that's a 0 but
it's also a 28 by 28 and this isn't
really structured that way yet so it's
just one array so it's just one long
string basically and you would need to
reshape it if you wanted to see what it
looks like for real but we'll get there
so anyways that's that now what we want
to do is well first of all there's a
there's a few things for us to think
about here but even just with a regular
neural network you know there's probably
no reason to have this kind of
precision-like we don't we don't need
that so we could we can handle for that
and we can handle for some other stuff
too but first let's just go ahead and
visualize this that way as we've changed
the precision you can see what changes
that actually makes so first we're gonna
go ahead and import Matt plot Matt plot
lib pipe lot as PLT and let's go PLT mmm
you're really making me angry
what is this happening
I don't like there we go cool import
numpy as MP are you kidding me
jerk that's really annoying anyway I
feel like if you hit tab then sure I
want you to autocomplete when I hit tab
but if I hit the enter button hmm I
don't like it no I can stop that now the
next thing is let's go ahead and just
peel TM show well well we can't even
show the data because the data is just
one long string so actually we want to
reshape it so let's say pixels Eagles
data dot reshape and then we're just
gonna reshape it to be a 28 by 28 now
PLT IM show pixels and then PLT show and
now we can kind of see what we're
working with again still kind of your
typical what you would see in any you
know in this tutorial there's your eight
lovely beautiful we can also convert to
grapes it should be so see map equals
gray and you get what we're looking for
now mmm what we want to do next is
there's really no reason to have like I
was saying before any precision like we
really just want to threshold this we
want everything to be a 0 or a 1 we want
it our goal is to get rid of as much of
the bloat as we can so in a character
like this character generation character
level generative model we want we want
to simplify it in the sense where we
have as few characters as possible right
we want to condense this data as much as
we can
so one way that we're going to do that
is by thresholded data so one way we can
do that with with with our numpy here is
basically data right here we can use
numpy dot round int so NP NPR int we
round match X's and that's going to
round it to the nearest integer so we'll
do the same thing with batch wise and
then let's run it and what we should see
is a threshold image good fruit there it
is so there's a one sure enough zero one
alright so looks good let's just do one
more just so we can get the idea of a
threshold because sometimes when you
threshold it's gonna fill in holes and
gaps and it might look kind of funky Wow
another one come on man give me
something else look at this one okay so
look at two you know maybe there was a
little hole there but anyway you get the
idea now the next thing is we still have
a lot of wasted characters in the sense
of these like the periods here right the
decimal point so what can we do there
well we could say dot as type int and
then as type int run that again and
really rather than data we should just
print
okay so here's another good example
where probably if you didn't threshold
it some of these were a little lighter
but otherwise you know you could still
tell that's a to almost actually looks
like a coiled up snake like you can like
right here there's a snake head right
but we still can't tell that that's a to
here but since you guys have already
kind of seen where we're going we could
rather than print data print the pixels
and you should be able to actually
visually see what you're looking at
we'll wait for it here okay seven so if
I just clear out the seven and we zoom
out a little bit you should be able to
tell like just by looking at this yeah
that is a seven like you can tell
without even rendering an image now so
we've simplified this quite a bit
like that's much more condensed than it
was initially but we still have things
that we could change so the next thing
is this is a generative model it takes
in a string so is there any reason for
us to adhere to any array structure and
of course the answer is is no right so
all these extra spaces here do we really
need those right like there there's no
need for that so and like I said before
with the brackets we could get rid of
them I actually kind of think they're
they probably kind of helpful to the
model so I kind of want to leave the
brackets but we definitely could get rid
of the spaces here it doesn't need to be
an array so we can convert it out of an
array and then always convert it back to
an array later and so anyways in the
next tutorial what we're going to do is
we're going to actually convert this to
string form and condense it even further
and then be on our way to actually
creating a much large training set of
condensed data and see what we can
figure out so anyways that is all for
now if you have questions comments
concerns whatever feel free to leave
them below if you want to support me in
the content this is my full-time job you
can head to Python programming net slash
support for a variety of options
otherwise I will see you all in another
video