Hello.
Welcome to a continuation of my series
on convolutional neural networks at ml5js.
The last time I recorded one of these was February 24, 2020.
It is now October 2020.
I would like to keep this mask on for the entire recording
of this video, but I cannot, because it fogs up my glasses
and I can't see anything.
And fortunately I'm in a hermetically sealed room
here by myself, where it is safe for me to take off my mask.
So I'm sorry that it took me so long
to get to continue this series.
But I am very excited to do it with you today.
I spent the last half an hour re-watching this video,
and getting myself centered to where I am
and where I want to pick up with now in this video tutorial.
The first thing I want to highlight for you
is that I made a couple errors in the previous video
when I was discussing how the resolution changes
from layer to layer within a convolutional neural network.
I was off by 1 or 2 or maybe by a power of 2 here and there.
Thank you to returnexitsuccess, who eight months ago pointed
out that that 28 by 28 image actually becomes 26
by 26, when the 3 by 3 filter is passed over it,
leaving out all the edge pixels in the new processed image.
Additionally, Luis points out here
that the resolution, the total number of pixels,
is reduced by a quarter, not by a half.
Because if the width is reduced by half and the height
is reduced by half, the pixels being
the width times the height, are reduced by one quarter.
Thank you for saying I'm awesome, too.
I mean that's nice.
That's nice.
I appreciate it.
When I watched the video again this morning,
I discovered that I promised some things at the end.
And so I'm here to deliver on that promise.
And the first thing that I'm going to do
is take the example from "Train a Neural Network With Pixels,"
which didn't use convolutional layers,
and update that example to have convolutional layers,
and just look at how the code is different,
and see if it performs differently at all.
As a reminder of what this example does, is this
takes a low resolution 10 by 10 image from the webcam,
and I've already trained the model
to recognize it as label a when I'm
standing in front the camera, and label b when I move away
from the camera.
So ultimately this kind of simple binary classification
with very clear distinctive images
works fine without the convolutional layers.
But let's try something a little bit more sophisticated.
Could we perhaps get it to recognize whether I'm
wearing my mask or no mask?
And let's add convolutional layers to this example
and see how that works.
I should also quickly mention that there
have been some updates to the ml5 library
since the last time I recorded.
And you'll want to make sure you have version, at least 0.6.0.
That's the version I'm using for this particular demonstration.
The first step for adding convolutional layers
to your ml5 neural network is to change the task.
So the task that I'm going to specify
is image classification.
I should probably point out the convolutional neural
networks are not limited to working with images.
They're super effective for lots of other kinds of data.
And I would certainly like to get into that at some point,
and look at some other examples.
But a primary use case is image classification. ml5
knows how to work with images.
So this is kind of our starting point.
So the terminology, the friendly term if you will,
to having convolutional layers in your ml5 neural network
is specifying the task as image classification.
Other thing I need to do is be much more specific
about the input data here.
So with a regular neural network in ml5,
it was just about the number of inputs.
Were there three?
Were there 104?
Whatever number you might pick based on your data.
Here I'm going to be sending it images.
So I need to tell ml5 what are the dimensions of the image,
width and height, and how many channels does the image have.
Is it an RGB or RGB alpha image?
Is it a gray-scale image?
And working in p5 generally, it's going to be--
it's up to me to specify the resolution of the image.
In this case, it's 10 by 10.
Lets up that resolution to 64, 64 by 64.
So I'm going to say in an array, 64, commas, 64, comma, 4.
Because the pixels of the images that I'm going to pass in
have red, green, blue and alpha values.
Ultimately here, the alpha information
is useless, because I don't have any transparency in the images.
So I might want to filter that out.
But I'm not going to worry about that right now.
I'm just going to let it be 64 by 64 with four channels.
I'm noticing here that it says outputs 3,
and I don't remember why I that's there.
Probably when I was doing this the last time,
I knew in my head, oh, I'm going to have three labels, three
possibilities.
But actually ml5 will quite nicely
figure out how many outputs there
are, if it's a classification problem, based on the data
itself.
So I'm just going to take that out.
And I've got inputs, image classification, and debug set
to true, because I want to see the graph of the loss
as it's going.
Now if you recall, I put almost no thought
into the data collection interface.
There is no interface.
When I press the key a, saying these images are a, label a.
When I press the key b, these images are label b.
So I'm going to keep that model, but I
do need to now adjust this addExample function.
The nice wonderful thing about the fact that I've specified it
as an image classification problem,
is ml5 knows how to work with p5 images, or raw pixel data.
Both are possible.
But I'm just going to-- since I have a p5 image,
I'm going to use the p5 image.
So rather than have to loop through all the pixels
and normalize the data myself, I could
do something much more simple.
So I'm actually going to remove all
of this processing of the pixels.
And the input itself, which is really a single input image,
I'll call it input image is, I need to make it an object.
And the property, I'll just name it image.
And the image that I want to send in is the video itself.
Then for the target, the training target,
I'm going to also just be consistent and make this
an object called label, and that will actually be the label
itself.
One of the nice things, by the way I can do a JavaScript here,
is when I'm creating these object literals,
I want to have a object with property image and value video.
Here, the property name happens to be
the variable name of the value.
So I can use an enhanced object literal.
It just makes the code a little bit shorter and cleaner.
And I can just say target equals curly brackets
with label inside.
So now I just need to change this inputs to input image.
And I now have my data that I'm using
to train the model is the video image itself, as well
as the target label, both wrapped into objects.
I also need to do the same thing in the classified video
function, because that's where I'm also
sending an image into the model for a prediction.
So I can actually just remove this entire bit of code,
and replace it with that same object literal,
and pass that in.
All right, I'm going to try to run this.
I'm really not so confident it's going to work.
And there's more that I want to say
about this, a couple of things I want to add to this example.
And then there's going to be another video where
I think it would be a really excellent demonstration
to show a use case where I've collected essentially
a database of images that I want to use to train the model.
But let's just run this and see what happens.
OK, well first of all, I'm seeing the image
being drawn by 64 by 64.
Let me change the way I'm drawing the image.
I don't think I need to draw every single pixel individually
as a rectangle here in this use case.
So I'm going to take this out, and just draw the video itself,
and stretch it out over the width and height of the canvas.
It's important to realize that I've actually still set
the image to be 64 by 64.
So that's what's actually being passed
into the machine learning model here.
But we're seeing a higher resolution version of the image
stretched out over the canvas.
I'm going to try exactly what I did before,
which is every time I press the key a, I get a new training
image with label a.
Now I'm going to step away, and give it a bunch of training
images with label b.
And then when I press t, it trains the model.
That didn't seem to work so well.
So I'm glad that happened, because this is inevitably
going to happen to you.
Something went wrong, because the loss is not going down.
In fact, the loss is above--
between 4 and 5, which are very high numbers for a loss.
So I was just doing some debugging
to try to figure this out.
And I downloaded the data that ml5 neural network
saves the data from your training data set.
And I looked at it and I thought, oh, I
forgot to normalize the data.
So all of these numbers are all pixel values between 0 and 255,
which makes sense.
That's the way pixels are stored in p5.js.
But I need them to be normalized between a range of 0 and 1
for the neural network to work.
So right before I train the model,
I need to add one line of code to normalize the data.
So I'm hoping this fixes it, but it remains to be seen.
I could have reloaded the data I saved previously,
but I'm just doing it again, just to do it again.
I saved my model, training prayer.
And then I press t.
Ah, that's a loss function I like to see.
a, b, a, b.
So there's a live chat going while I'm
recording this right now.
And the question comes up, can't you add the normalizing layer?
And so first of all, the normalizing--
normalizing the data is not a layer of the neural network.
It's like a pre-processing step.
And that could be something that ml5 just always does
by default. But there are some cases where
you don't want to normalize the data,
or you want to normalize the data in your own way.
And so that's explicitly something
you do have to call with ml5.
That's an interesting question whether or not
ml5, the library itself, should change the way it works.
But for now, I do have to call it.
So this worked.
And ultimately it's the same exact result
of what I had in the previous video.
So why are we even here?
So I would like to make the case for you,
why you might want to use the convolutional neural network
functionality in ml5 beyond just the regular neural network
stuff.
So one is that it's my suspicion here
that if I gave it a much harder problem, more complex images
with less obvious distinctive differences to classify,
that the convolutional neural networks are
going to perform better.
This was such an easy case of am I standing in front the camera
or not, it's only two classes.
So we're not really seeing a difference.
So I might try it with my mask.
That might be a slightly harder problem.
The other thing you might be wondering
is why are we even doing this when
we have this whole system about transfer learning?
I mean, after all, this is basically
exactly the examples from the Teachable Machine videos.
Well in truth, if what I wanted to do
was quickly whip up an image classifier that's
recognizing some gestures and movements,
am I in front of the camera or not in front of camera,
or a particular object; using Transfer Learning and Teachable
Machine will get me probably better, more accurate
results more quickly.
But there are a couple of reasons why you might not
want to go that route.
One is maybe you don't want your model
to be based on any pre-existing model or data set.
You don't want to use MobileNet model which
was trained on the ImageNet database
as part of what you're doing.
Also maybe the images you're using really
have nothing to do with the everyday objects
that the MobileNet model was trained on.
So for example, if you're trying to recognize
drawings, or circuits, or some kind of obscure specific design
pattern that's not something you see like scissors, and phones,
and remote controls, and coffee cups;
then that Transfer Learning approach
isn't going to really help you.
Because the data of the pre-trained model that you're
basing on does not match your current data,
and that's what I'm going to show you
in the next video, where I want to look at doodles, and shapes,
and other kinds of data that just aren't photographic images
from everyday life.
But here, let's just make the case
that this is going to work a little bit better.
And let's see if I can get it to work, do the mask and no mask.
So let's add some more specific labels here in the key press.
So if I press the key m, then I'm
going to add an example with the label mask.
And actually the labels, this is a little bit silly,
because I could just code it to like
say a message when it sees a certain label.
But the labels could be just any arbitrary string.
So the label is going to be "Nice mask!" for when
I'm wearing it, and "Keep others safe!
Wear your mask," for when I'm not wearing it.
All right, let's see if we get this to work.
So let's first do the no mask.
Now on with the mask, all right, hopefully that's enough data.
Let's train the model.
Wow, that's some wacky loss.
But it looks like it figured it out.
Hey, it likes my mask.
Oh, whoops.
I didn't really think about the design here.
OK.
I fixed the layout.
So now it will tell me to put on my mask,
if I am not wearing it.
So here I am wearing my mask sitting at my computer.
And I take it off, and it tells me to put in on.
So this is great.
Our convolutional neural network,
there's no transfer learning.
There's no base model.
This was all done in the web browser, in p5.js, with ml5.
Amazing.
All right, before I go from this video,
I want to just return to this model summary panel
in the debug view, when you're training the model.
So this is ordinarily the part that I maybe
try to stay away from a little bit, the model architecture
or the lower level details.
But I think it's really important
to make the connection between these layers and the diagrams
that I showed you in the previous two videos
about what is a convolutional neural network.
And we can see right here the convolutional layers,
the pooling layers, and then the flattening, the flat layer,
and the final output label.
And you can see that there's two outputs, because there
is a probability confidence score for mask
and one for no mask.
So that would be a higher number if there are more categories.
But this is the default architecture
that ml5 will make when you say image classification.
And it's actually possible for you to configure this yourself,
to say how many convolutional layers you want,
to say how big you want the kernel--
the filter kernel to be, how do you want to do pooling,
what is your stride; all of those parameters
that I mentioned in the previous videos are configurable here.
So if this isn't working for you or you just
want to play and experiment, let me show you how to do that.
In the ml5js reference, you'll find a section
under neural network for defining custom layers.
And this is where you can actually configure individually
the number of layers and what those layers
do in an ml5 neural network.
So this is the default set of layers
for a classification, the default set
of layers for a regression.
And I want to look down here at this default image
classification layers.
All you have to do is create an array called layers,
fill that array with objects that
include the various details for each layer.
Let me copy this to the clipboard, go back to my code,
and right here in setup, right before options,
I'm going to paste that in.
So now I have a variable that's holding onto the custom layer
configuration.
And I'm going to go down to my ml5 neural network,
and I'm going to add a property, layers, custom layers.
So here, I have the input, dimensions, the task
that I'm doing--
I want to debug it while I'm training it--
and now the custom configuration for the layers.
And I could start to experiment with this.
And you can see here in this array,
my first layer is a convolutional layer.
I want to have 8 filters, a kernel size of 5,
and a strides of 1.
I haven't talked too much about activation functions,
and what's this kernel initializer.
So these are--
I'll try to put some resources in the video's description
where you can read up more about some of these other properties
that you can experiment with.
But again, this is the size of max pooling.
What would happen if I did it 3 by 3,
and change the number of strides along the x and y-axis?
And you could see one thing.
The thing that I think that's important for me to point out
is there are two convolutional layers.
And as the resolution is decreasing,
the number of filters is increasing.
Not a blanket rule, but that's one way
to approach architecting your model.
Trial and error is your friend here, experimentation,
talking to somebody else who knows
about convolutional neural networks
or has tried it before to get some advice about what
might work well for you in your particular scenario.
I encourage you to experiment with that.
Leave your feedback and things you've tried in the comments.
And in the next video, I am going
to look at a convolutional neural network that is
trained off images of shapes--
squares, circles, and triangles; so
that I could create something where maybe I'm
drawing on a piece of paper, and the neural network guesses.
Did I just draw a circle, a square, or a triangle?
And then later, I'm also going to show you
some pre-trained conventional neural networks that
are in ml5, like DoodleNet which is trained
on a whole lot of drawings from the Google Quick Draw
data set to recognize various kinds of doodles.
And these are scenarios where using
your own convolutional neural network
really makes sense, because it's the kind of data
that we're working with.
Drawing and shapes and abstract geometry,
isn't something that the original MobileNet
model, image classification model, was trained on.
So transfer learning doesn't necessarily apply.
And also there are some reasons why
you might want to train your model from scratch with only
your own data, and you have a real control and understanding
of how that data was collected, and how that model is being
used, as opposed to a situation where you're doing transfer
learning, and basing your model off of a pre-trained model
that you might not know as much about.
OK, so I hope that next video won't take a year to come out.
But who knows what's coming next in 2020 into 2021?
I hope good things for you.
And I will see you in a future ml5 video.
If you're watching this in years into the future,
well, it's a little bit of history
for you, a history lesson.
And I don't know what I'll be doing, or what you're doing,
but I'm glad that you're here, and that I'm here with you
in this virtual mediated way.
All right.
See you soon.
Goodbye
[MUSIC PLAYING]