you came back welcome to part 2 of the
deep learning with Python and pine torch
tutorials in this video we're gonna be
talking about data and really that's
gonna be it the the actual input to our
neural network now I think a lot of
times everyone wants to focus on the
neural network itself but actually the
step of acquiring data pre-processing
your data and how you are going to
iterate over your data pretty much
consumes I would say 90% of your time
and energy at least in terms of thinking
about your model obviously training time
when you're just waiting for the model
to learn stuff it can take a long time
but you the work that you are going to
be putting in this is the step where
where you're gonna be doing probably the
majority of your work so to begin what
we're gonna do is work with a kind of
toy dataset it's going to be M nest
this is a really popular data set to use
for beginners one because it's a machine
learning learn about data set like it's
really simple there's no question we can
definitely learn this we can tinker with
it it just it's a good starting data set
so that's what we're gonna use and we're
gonna use a package called torch vision
which you should already have installed
if you haven't just pip install that
should just be pip install install torch
vision like then you can get it so fun
fact I also looked tried to look up how
to change font size in Jupiter lab I
found so many responses about various
things you can do none of them worked so
just curious for my future videos by the
time you tells me it'll be over but like
what would I need to change what do I
need to modify to modify just the font
size so I'll have to zoom in
I even tried editing some of the CSS
files that dictate Jupiter lab and it
didn't work so okay anyway totally
baffled so please if you actually know a
way don't just link me to a post if you
actually know away let me know so under
the tutorial you guys got to give me you
got to give back to me sometimes okay so
what we're gonna do first is we're gonna
import torch and then we're going to
import torch vision and adjust my
microphone here the other thing we're
gonna do is from torch vision we're
going to import transforms and data sets
so let's make sure that import works at
least it does cool so what is torch
vision so torch comes with a bunch of
data sets and I can't remember now if
other data sets exist now in torch
vision that aren't vision tasks but
basically it's it's a collection of data
that is used for vision so most training
data sets with neural networks have
something to do with vision just because
vision seems to be the the big thing
that we we benchmark against I wish
there were more data sets for other
tasks I mean there's some but vision is
clearly like the the main interest that
people are working with just because
really neural networks are solving
solving vision tasks and other machine
learning algorithms just haven't been
able to do that so just in terms of just
money from like investments and business
interest
it tends to be vision tasks so that's
like a low-hanging fruit that we can do
something with right now but obviously
there are other tasks as well especially
like when it comes to like advertising
is a huge market where we want to be
able to predict what people are going to
do and funneling and all that but
anyways torch vision just has a bunch of
vision data for us and as you're gonna
see as we continue on the series is this
is basically cheating to use their their
built-in data you are like I said most
of your time is going to be spent
getting data preparing your data
formatting it in such a way that's gonna
work with a neural network right like
because you again you gotta convert
things like categories or you know words
have to be converted to numbers all that
kind of stuff so you're going to spend a
long time doing that kind of thing here
it's already done for us and then the
other thing we typically need to do is
batching which I'll talk about in a
minute and again it's going to be done
for us now I will say in the next not
the next tutorial cuz we'll probably
still be working on this but in
the next kind of model that we build the
next data set we're going to use our
completely our own data set because I
know this isn't what most people want to
do most people are going to want to
probably apply this to their own problem
we're just gonna do this so it makes
other things easier to learn and like I
said we're gonna slowly increase
complexity as we go through the series
so anyways transforms and data sets and
so now what we're gonna do is we're
going to define our two major data sets
and when it comes to just machine
learning in general you're gonna have
two different things you're gonna have a
training data set and then you're gonna
have a testing data set and it's
important that you separate these out as
soon as possible before you forget
because in order to validate your data
or your model rather you want to have
what's called out-of-sample testing data
this is the most realistic test that we
can have basically it's data that has
never been seen before by your machine
because if you use in sample data
basically the machine if it has learned
to overfit it will do really good on
your in sample data and very poorly on
your out-of-sample data so it's an
essential that we have truly
out-of-sample data meaning data that the
machine has just simply never seen
before because when you actually go to
use this model in practice it's going to
be you know out-of-sample right it's
gonna be new data that it's never seen
you you're hoping that it performs well
and with neural networks there like I
said before because you've got millions
of trainable tunable parameters it is
highly likely that if you train long
enough your machine will over fit so as
we progress through the series we'll be
talking a little bit more about how long
to Train how what stats to watch and
stuff like that but anyways you need
trained and testing data so for this I
am just going to write out this is a
bunch of stuff but basically we're just
working with this data sets stuff so
we're gonna say data sets dot capital M
NIST and then you're gonna specify where
you want the data to go I just want it
to go locally someone do open and close
parentheses and we're gonna say this is
trained equals true download equals
and then any transform so transform
equals transforms dots capital C compose
and then in here you would paste all the
transforms these are things we want to
apply to the data and I don't know why
but the data is not even though we're
grabbing it from torch vision the data
is not natively already in tensor so we
have to convert it to a tensor so the
one of the transforms that we're going
to apply here is going to be transforms
dot to tensor this one is already built
in
obviously the torch vision because we
imported it here but later on you you
can actually write like if you really
enjoy this you can write your own data
set and you can use this kind of same
syntax cuz because as you'll see it
there's a lot of things like especially
in this tutorial I won't become as
obvious but in the coming tutorials it
will start to become pretty obvious how
tedious iterating over a data set for
example can be I almost think it would
be just as tedious to convert it to one
of these like data sets as well but
anyway just know that you can do that
and then you can also write your own
kind of custom transforms but for now
we'll just use the one that's built in
here and I believe that's a valid line
so actually I'm going to go ahead and
take this copy-paste train will be set
to false we do still want to download
and I think we're good so let's go ahead
and run that real quick and you can see
it's downloading that might take a
moment
hopefully that I won't take too long and
now while that's going I think we'll
just start writing writing our next two
lines so basically this gets us our data
downloaded and kind of in a sort of
variable here the next thing we want to
do is actually load this into another
type of object that's going to help us
iterate over that data so let me write
these two lines and then I'll explain
why we're doing that because it might
not seem obvious why okay we've got our
training and testing data why do we need
to talk about how to iterate over it so
the next thing I'm just going to go
ahead and write here is I did not
think it was gonna do that's kind of
funny anyways it's done so we're gonna
say trainset so this will be our
training data set we're gonna say that
is equal to data sets dots and then em
ups to write rewrite they're grabbing
the data anyway torch dot utils dot data
dot data loader and what we want to do
is load in train so we're just gonna
load in the data that's currently added
to or is currently this train variable
here so we load that in and then we
specify what batch size we want I'll
talk about what that is here in a moment
I'm gonna say ten just so it's totally
easy for everybody to do this and then
shuffle will be equal to true and we'll
talk about shuffle as well so first let
me just copy paste this line we're gonna
do that same thing for test set test and
yes okay mmm so so we've separated these
out and first batch size this is how
many at a time do we want to pass to our
model so in theory some of these data
sets like this one's not really that big
of a data set you could probably so like
if I open up that this is what it is
m-miss right here I don't even know what
those two are but I bet if we just go to
properties we could see how it yeah so
it's only a hundred and sixteen
megabytes so definitely fits into your
RAM almost certainly would fit onto any
GPU you'd be able to you know use your
GPU with so definitely we could fit this
entire data set through our model and in
one go but that's not very practical
because with deep learning where deep
learning really starts to shine over
other traditional machine learning
algorithms is when we have huge data so
millions or like tens of millions or
hundreds of millions or even billions of
samples at some point it does get to the
point where it's more samples than you
can fit into your memories so on the one
for one reason that you have to use
batch batches of data so so rather than
doing the whole data set
we're gonna feed through ten items at a
time through our model and the model
will be optimized each step of the way
in little tiny increments but you based
on those ten samples at a time they get
fed through and in general a common
batch size is gonna be anything from I
don't know eight to sixty okay
or sixty-four everybody really likes to
use base eight numbers you're gonna see
that a lot and people ask like why this
why that
it's well first of all the base a
is just for fun like there's really no
reason for it it's just everybody tends
to do it so just expect it and then in
terms of like how many neurons per layer
it's always trial and error and we'll
talk a little bit more about how you
might like at least pick a starting one
and then how you might iterate through
this like trial and error type process
but it's a it's basically a gradient
descent operation that you're gonna be
doing yourself as well so anyway more on
that later
but so one reason we have two batch is
because our data is just going to be so
big that we probably can't any realistic
example fit it all in the GPU the second
reason is because we hope this data will
generalize so if you take your entire
data set and you pass it through the
model what's gonna happen is like as the
model starts to be to optimize all those
little weights and all those little
connections remember there's millions of
these if you pass your whole data set at
once your your machine is gonna maybe
learn some generalizations but it's also
going to probably be able to figure out
some weights that just they're just
arbitrary the machine doesn't know is
this is this you know when I tweak this
and it improves the loss right is this
is that because that's this is generally
true or is this just because this is in
sample data the machine has no way to
know that so instead what we do is if we
pass through batches at a time each time
it optimizes the optimizations that
stick around generally are going to be
the general the generalizations the
correct things that assumed those ones
stick around
and then the actual just like over
fitment cases they kind of just get
erased out when we were working with
batches so
it's not the case that you sorry
punching my mic it's not the case that
you want to have the biggest batch size
you possibly can
there really is there's always kind of a
sweet spot batch size and like I said it
usually is between 8 and 64 regardless
of like how big your your memory is but
usually is just it's just usually
sometimes you're gonna go even bigger
than that so and the reason why you'd
want almost your batch size you do want
it to be as big as possible because
generally that's going to impact how
quickly you can train through your data
but anyways a lot of information there
on patch size but it's a it's just again
there's there's not that many things
that you get to tweak as the engineer
here so that's another thing that you
get to tweak and it's another thing that
have general rules but for the most part
you're going to just trial and error
again I'd I tend to just go with the
biggest batch size I can get away with
because that helps training time finally
shuffle you definitely want to shuffle
the reason why like for example like in
in our case we're doing em inist which I
realize now I haven't even explained but
it is em nest is a hand-drawn numbers
data set so it's hand-drawn digits from
0 to 9 so it'll just be one digit and
it's a 28 by 28 image of this hand-drawn
digit so all the nines look a little bit
different but obviously they're all
nines and so on if you took a neural
network and you fed through a bunch of
zeros first the neural network would
learn to optimize and say everything as
a zero and then when you get to your
ones it would be like oh wait wait
everything's a 1 and then and so on
until you got to your nines and it would
it would end by saying everything as a
nine whereas if you shuffle again the
name of the game is generalization so we
want to do everything we can do to give
the neural network the opportunity to
learn general principles rather than
just simply figuring out little tricks
because if there is a quicker quicker
route to get to increasing or decreasing
loss really the neural network is going
to take that route and so you have to be
you have to constantly be thinking
hmmm how can I better like obfuscate
over fitment so anyway
cool so we've talked about trainset
batch size shuffle true cool so once
we've done that so again in most cases
even here like training and testing you
would have to do that
split all on your own you'd have to
shuffle your data all on your own
you'd have to somehow batch through your
data all on your own again we will be
doing that all on our own soon but first
I figure hey let's just talk about it
let's keep it as simple as possible and
then we'll slowly ratchet up the
complexity so let's go ahead and run
this cell cool and now how do we
actually iterate over this data so let's
just use trainset for examples so it's
actually super simple for us to iterate
so we're gonna say for data in Train Set
let's just print data and then we're
gonna break because we don't want to run
over all of them so just run once so
what you have here is a it's that entire
batch so it'll be it'll be ten examples
of handwritten digits and then ten
tensors of the actual output so the
first example should be a three-second
one a seven one six and so on so the way
that we can actually confirm that is by
saying we could say X comma Y so this
might be a little unclear but remember
you guys are beyond the basics so in a
for loop whatever the last temporary
variable was in the for loop can still
be accessed so in this case data so
we're just going to come down here to a
new line we're gonna say XY equals data
and then again we want to access the
zeroeth element in it and I'm trying to
think here it should be to the rib
tensor I have to I just want to make
sure I'm right here so yeah so the first
so so as you iterate over here data hat
is a tensor object but it has it's a
tensor object containing first a tensor
of tensors that is your images and then
second a tensor of tensors that are your
labels so the way that we can reference
a three for example would be data the
zero if because
as your images and then we'll say the
zeroeth image so this should be an image
of three and then we would say for the
why that would be data the first if
right so these are our tensors and we
want the zero width tensor so now let's
just print let's print y real quick and
make sure yeah okay cool
so the next thing that we can do is we
can visualize this and make sure that we
are correct so we can import mat plot
libbed up pie plot as PLT if you don't
have this just pip install mat plot plot
lib and again
you guys aren't basics i assume but you
would type that into your terminal not
into jupiter notebook anyway and then
we're gonna say PLT dot in show and we
want to m show data 0 0 now here's an
issue and it's not with my typing i'm
just gonna make a new line real quick
let us print data 0 0 boom that's one
actually that's curious
is it because let's do 0 0 dot shape
cool so as you can see here's an
immediate reason why if you just did
tutorials and only use torch vision you
would immediately out of the gate be
like wait what because as soon as you
tried to do your own data set because
notice this shape it's a 1 by 28 by 28
that is not a typical image so if you
just loaded it an image and converted it
to grayscale and then converted it to a
tensor it wouldn't be a 1 by 28 by 28 it
would be a 28 by 28 there would be no
one but a pi torch wants that one there
right so that would be just one
immediate hurdle that you'd come across
but and again this these are the things
you have to start thinking about when
you want to start feeding it through a
neural network the shaping stuff
throws a lot of people off really
quickly so I'm gonna do my best to make
it clear when we do what each number in
a shape actually means even the
tensorflow and pie tours docks are all
the same a lot of times they throw these
shapes at you and they don't explain
where did that number come from they
just like pull it out of nowhere and
it's it's really frustrating when you
have no idea how they got it and I'll
talk about that when we get there soon
one day one day we'll get there but yeah
that can make learning very difficult so
anyways this one's a pretty easy one you
would just have that one but if we want
to show an image of it we we can't PLT
da m show this we would end up with a
peel T - oh let's do it we should get an
error yeah and basically it just says
hey that's not a valid shape for an
image we actually need to make it we're
gonna say dot view because remember
that's like basically reshape but it's
not in place but it's it'll it'll work
here so we're just saying hey show the
image of that data as a 28 by 28 boom
it's a 3 so sure enough that actual
shaping works so let me think here the
next thing is trying to decide if I want
to I think we'll just talk about
iterating over our data and then
possibly balancing chances are this data
is already balanced is my guess again
this is the thing that is super
convenient but as soon as you try real
world data set it's not gonna work out
so so what is balancing so remember like
I said if the model can find a shorter
path to figuring out some - decreasing
loss the model doesn't know it really
doesn't the model doesn't have a priori
knowledge of what is the the lowest loss
could be so as that optimizer is trying
to decrease our loss it doesn't know
what is what how how good could we get
right it has no idea it's just going to
try to decrease loss as best as possible
and
generally as easy as possible so what's
going to happen if in our data set right
our data set is zeros one all the way
through nine what if let's let's make an
easier exam well looking you seven is it
doesn't really matter what if what if
sixty percent of our data set is number
three like it's threes and then the rest
are like five percent ones and two
percent nines and so on our model is
going to immediately learn that the
quickest way to decrease loss because
loss is really just measured on that
output the output from the neural
network it's going to adjust weights in
such a way that it can as quickly as
possible always protect predict a three
and then it's gonna get stuck because it
can't figure out how to get out of this
hole because it anyway out of that hole
means it's got to get a lot worse before
it can get better again and it's going
to be stuck there forever like you're
not going to be able to Train out of
that hole so wait another thing you have
to think about when you go to input data
into the neural network besides you know
shuffling all this stuff in batches and
blah blah blah you got to make sure your
data is balanced now there are ways to
get around imbalanced data sets by
modifying the weights of specific
classes when calculating loss but I have
never had that work out for me there
there's research to suggest you could
get away with that somehow it has never
worked for me you generally want your
data set to be as balanced as possible
so one way that we can kind of you know
at least confirm a data sets balance is
so for example we can just make a
counter so and then I'm gonna say
counter dict it's probably a better way
to do this actually we could just make a
list and then use counter
part of me wants to do that I can't
really decide if I want to do it this
way or that way well just I'll just
continue this way because I will
actually want a percentage that'll be
easier to read so we're gonna say zero
zero and we'll go all the way to like 9
so 1 2 3 4 5 6 7 8 9
we'll make this a 1 2 3 4 5 6 7
did I really not do math okay cool 9
okay so now what we're gonna do is we're
gonna iterate over all of this data so
for data in train set we're gonna say
the X is capital X let's just stick with
the syntax equals data so we've set
because remember data is it's not a
tuple but we can kind of treat it like
one right it has two sets of tensors so
now we've got X's and Y's stored in
variables then we're just going to say
for wine in y's counter underscore dict
and then we're going to convert it to an
int why because it's we're trying to use
a dictionary key here and then we're
just gonna say it plus equals 1 so as we
iterate over all of our data all we're
going to do is add to these counters so
we can kind of see where we are in terms
of how many samples for each number that
we have and then once we've done that we
can just print counter dict so yeah we
totally probably could have done that
from collections import counter and then
just made like a big list and then made
a counter object off that list but
anyway yeah so 5,000 basically 6000 but
almost up to almost 7,000 for a 1 and so
on one thing that we could say is oh we
the other thing to do for y let's do a
totals or total plus equals 1 and then
what I'll say down here is for I in
counter dicts print and we'll do an F
string we'll say I and then :
and then we'll say counter dict
i / however many total we have and then
we'll just multiply maybe multiply that
by a hundred where did this parenthesis
come from I have no idea when that got
introduced cool yeah okay mmm so this
gives us the percentage kind of
distribution so as you can see the
number one is by far the most pop one
not by far but it's 11% versus let's say
the lowest is 9% so in this case this is
probably balanced enough like you're
probably not gonna have any issue
especially because part of the problem
too is like so so the quickest way to an
optimization would at best if it
predicted everything is a 1 will give it
11 percent whereas random is gonna be 10
percent so chances are it's going to be
able to to deal with that kind of just
slight it's not perfectly they're not
all perfectly ten percent but the
optimizer is gonna be able to work
around that but if if one was for
example forty five percent or something
like that it probably would not be able
to work around that kind of data so okay
I think that's enough for this tutorial
in the next tutorial we'll actually
build the neural network and talk about
you know all that but honestly data is
more important than the neural network
so yeah quick shout-out to some channel
members who have been with me now for a
month we've got Liam ends B Musa Kurt
Lauren - Anika DAWs Dylan died Tim
getting's and Otto kopetsky thank you
guys very much for your support you guys
are amazing and I will see you guys in
the next video