Hello.
I am back in the beginner's guide
to Machine Learning with ML5.
And today, I am continuing this series
and I'm going to do a few videos about something
called KNN Classification.
And the thing that I'm going to use it and apply it to,
is building your own teachable machine.
Now this is the Teachable Machine Project
from Google Creative Lab demonstrating
this idea of transfer learning.
Taking a pre-trained model and using it as a foundation
to train a new model on top of.
Now you might be thinking, didn't you
already make these videos?
There was that whole thing with the feature extractor
and classification and regression
and saving the model.
Yes, you are right.
I did already make those videos.
However, I'm going to, basically,
do the exact same thing again but with a slightly different
technique.
And that technique is called KNN Classification,
which has some advantages.
There are pros and cons to both of these techniques
and at the end we maybe even sort of wrap up and look
at why you might pick one over the other.
But one of the main advantages of KNN Classification
is it's training the model is happening
as you're adding new images, as opposed to a separate step.
And I'll get into why that is in a bit.
So this is the Teachable Machine Project.
You can go to TeachableMachineWithGoogle.com.
And what I'm going to do is--
I've already trained this.
And so if I'm just standing here in the middle,
I've trained it with just me standing here
with kind of an angry look on my face as orange.
My hand to the left as green.
My hand to the right as purple.
So what I want to do is emphasis that what you can actually do
is build a gesture based controller
for some type of game using this technique.
You could do this with the other way as well,
but I really looked at it as just a new way
to classify new images.
But here, let's think about if I'm
training a model in real-time to understand
that a hand is holding up on my left,
and my hand is holding up on my right, or I have no hands
up at all.
You could imagine I could control
a character, a pong paddle to move left or right,
and maybe up or down.
So that's what I'm going to do.
It's going to take a few videos, a bunch of things
to get through.
But that's the idea.
All right.
So you can play around with this website to get a sense of--
there's a really nice interface.
There's a ML5 version of this.
This is built with TensorFlow dot js.
ML5, which is a library, built on top of TensorFlow js
We have a similar version.
All this work was done by Yining Shi for ML5.
And Yining Shi has been teaching a class
called "Machine Learning for the Web" at ITP, NYU.
Thank you so much, to Yining, for allowing
me to be inspired by and use your materials
in these tutorials.
Hopefully, Yining will come and be a guest again in "The Coding
Train" and show some additional examples of things you
can do with KNN Classification.
I highly recommend you look through the syllabus
and examples.
And there's also a whole presentation
about KNN Classification that's Yining's slides that you
can look at.
But what I'm going to actually follow
is this article, written by Nikhil Thorat,
who is one of the creators of TensorFlow dot js,
one of the members of the research team at Google Brain
developing TensorFlow dot js, you can see here.
And this is a notebook about how all of this stuff works.
And I'm going to, basically, go through this,
but not use the code here.
This code, which is kind of the raw TensorFlow js code,
I encourage you to look at it and think about how it works.
And this might be a place where you actually
want to get more lower level into it.
But I'm going to use the new ML5 KNN Classifier
feature that is an ML5 library.
Now this feature, you're going to want
to make sure you're using at least version 0.2.1 of ML5.
So as of version 0.2.1, the KNN Classifier is part of ML5.
But I will mention, at the time of this recording
the documentation is not yet up on the website.
But, hopefully, by the time this video
is released as part of that playlist
you will find the documentation on the website, ML5js.org.
All right.
So let's start going through this article
and let's start right here at this moment of Background,
MobileNet.
Now this is MobileNet, the MobileNet model
is something that I've used in just about almost
every video up until this point.
So if you've been watching the playlist,
you're somewhat familiar to it.
MobileNet, this is our first step, load MobileNet.
And MobileNet is a Machine Learning, Image Classification
model that's been trained to recognize
1,000 classes of images.
And it was trained based on a huge database
of images, known as ImageNet.
I've said this before, but this is
a database that exists for researchers to use and try
and development machine learning algorithms with,
but it's not necessarily a model that works particularly well
in generic applications of image classification
out in the world.
Because it knows a lot about birds and dogs,
but not about a lot of other stuff
that appears in the world.
However, it is a good basis to load,
the first step one is to load this model.
It's a good basis from which to do this feature extraction
process, this transfer learning process, which we did before
and we're going to do now in a different way.
So the next step is to then create what's
called a feature extractor.
I'm going to show you in the code.
This is actually, basically, what I've done before.
So this is what MobileNet does here.
You can see it can look at an image,
it can tell you what the probability
of a particular class, like Egyptian cat, tabby cat, tiger
cat, remote control apparently, there is a almost 2%
chance that it's a remote control.
Cat, you are not a remote control to me.
All right.
So now what I have already, which is basically
from my previous examples, is a sketch
that loads the image from the webcam
and displays it in the window.
And I also, like I said, have a reference to the ML5 library
and I also have this feature extractor loaded.
So I'm using P5, I'm connecting to the webcam using
a P5 library to connect to the webcam
and draw the webcam's image.
And then I'm using ML5 to create a feature extractor
with the MobileNet model.
So now we can go back to Nikhil's article
and scroll down.
And be like, OK.
So this is actually the end result, the cat softmax.
So what does this mean?
So the MobileNet model is a neural network that
expects an input of an image.
That image comes in and it's processed over a variety
of different layers.
So the image is transformed and processed
and twisted and turned, and all of that
is the neural network process.
Which you'd have to refer to some
of my other videos and other reference materials
to understand more how this works underneath the hood.
But by the end what it gets is a big vector,
a big list of numbers, that are all probabilities.
They all add up to 100%.
And if you look at that here, that's what we're seeing here.
And we're seeing all the way over here
these classes have a very high probability.
So somewhere in here, we're getting this number, like,
0.9, maybe we're getting 0.7, maybe we're getting 0.05,
and all sorts of other numbers.
They add up to 100.
And 0.9, this particular number, is the number
that corresponds to the probability of it
being an Egyptian cat.
So this is what MobileNet would do on its own.
But this stage is called the softmax stage.
Softmax is a fancy term for normalizing the output of all
of this process to a set of probability values
that all add up to 100%.
But there are stages before.
One stage before is called the logits.
And by the way, before I recorded this video I spent,
like, a half an hour trying to figure out how I pronounce it.
Whether it's "law-gits" or "low-gits."
And a lot of places said "low-gits,"
and some other places said "law-gits."
Imma go with "law-gits."
Like logic or logical.
But you can see, this is actually
what the logits look like.
It is the unnormalized, the non-normalized output
of the neural network.
It is a layer--
I shouldn't be using my hand.
I have an eraser.
It is the layer right before.
This is called the logits.
And it's just like a lot of numbers.
And softmax is applied to get this last output, which
is good for classification.
But this layer, all of these numbers,
this is referred to as the features.
I know I explained this in a previous video,
but this is useful to give this another try.
Again, we haven't really gotten to the next step,
where something pretty different is going to happen.
This is still the same as before.
And so we can look at it.
This is what those cat logits look like.
And you can notice, there's peaks still around here.
But there's lots of other stuff going on.
And you can see, Nikhil writes, this is
a "'semantic fingerprint' of the image."
Right?
This is, basically, the essence of the image.
The boiled down, numerical essence of the image as
perceived by the MobileNet model.
So because an image itself--
I forget what the dimensions MobileNet requires
is, maybe 224 by 224 pixels.
This is a lot of numbers.
So you could say, this is the literal numeric representation
of the image, is all of the RGB values of every single pixel.
But that's too many numbers to work with and compare and try
to make sense of the image.
So this pre-trained model has already
learned how to boil it down to just 1,000 numbers.
And if we have 1,000 numbers, we can
start to compare this image to, let's say, another image.
So this is the feature extraction,
getting these features, that essence, the semantic--
what did Nikhil write?
"'Semantic fingerprint' of the image."
If I have two "semantic fingerprints"
of two images that are arrays of numbers, there's lots of math
that I could start to apply to them.
I can say, well, how similar are they?
Well, how similar are they compared to these other images?
Or what if I were to sort of move interpolate between one
image to another?
And you've seen some of these, like, artistic visualizations
of the output of neural networks as moving through,
you might hear walking through the latent space.
So this idea of interpolated between the features
of images to generate something, all of this is interrelated.
But here this now, finally, is the moment where I can say,
this is how it relates to KNN Classification.
Because KNN stands for K Nearest Neighbors.
So if we get the logits for an image,
I could say, well, what other images have I seen previously
that this semantic fingerprint is very similar to?
Then this image is within that category of those images.
That's K Nearest Neighbor.
So I'm going to wrap this up and we're
going to start implementing the code in the next video.
But let's take one more step.
Let's actually look and see with ML5
how easy it is to get those logits.
So I'm gonna come back over here.
So this is my code, right?
So in my code I have the video and I have a feature extractor.
The feature extractor, I'm just calling it "features,"
is preloaded with the MobileNet model.
So now, what I'm going to do is, I'm just
going to have "mouse pressed," which is not what I'm
going to want to do ultimately.
And I'm going to say, "features dot infer video."
So the infer function is the naming--
I'm little unsure about this naming
but maybe it'll change in the API at some point.
But this is the idea of inferring the logits
from this particular image.
So this could be a static image, but since I
have the video it's whatever the snapshot of the video
is right now.
And this could go into the logits.
This is, basically, giving me that particular logits.
And I'm going to do something and is going to look weird.
I'm gonna say, "console log logits," like,
shouldn't that show me just an array of numbers?
Not exactly, but I'm going to show it to you.
Let's see what happens.
So let me refresh this.
OK, the model is loaded.
Now I'm going to click.
And you say, what?
What?
Well this looks, kind of, dtype float32.
OK, floating point numbers, there's 1,000 of them.
What's all this nonsense?
Look at all this, this looks crazy.
Well, guess what?
We just waited, by accident or kind of on purpose
since I did it on purpose, into TF dot js territory.
Now to continue writing this code,
we don't ever have to look at the TensorFlow dot js
documentation.
But this is a moment to realize ML5 isn't some magical thing
that exists on its own.
It is operating on top of TensorFlow dot js.
Meaning it's managing something called the tensors for you.
What's a tensor?
A tensor is, basically, a fancy word for an array of numbers,
or a matrix of numbers.
And so, the logits are a tensor, it's
a one dimensional array of all those numbers.
So they are actually a tensor.
I can actually look at those values in the console
by saying, "logits dot print."
So let me do that now.
And if I click here now, you're going to see, there it is.
Now it's not printing it all out,
but you can see, oh, that looks like an array of numbers.
And another way I could do it is, I could say,
"console dot log logits data Sync."
So Data Sync is a function-- these are functions
that are part of TensorFlow dot js
and you can go back and look at my TensorFlow dot js videos
as background, where I go through them more specifically.
This is a function that gives me that data as a regular array.
So last step here would be to do this.
Click and there you go.
These are the logits.
This is the digital fingerprint of that image.
So we are now ready.
We now, hopefully, from watching this video,
you understand maybe a little bit
about what KNN Classification might be.
But that's what I'm gonna get into.
That you're loading MobileNet, the pre-trained model.
You're creating a feature extractor,
which knows how to pull out the logits of the MobileNet model
with a given input image.
And then we're passing in an image from a video.
And instead of getting the classification,
getting just that digital fingerprint.
And that semantic digital fingerprint, this
is exactly what we're going to do in the next video,
to train a KNN Classification model.
All right.
So if you want to keep watching, then
keep going in this playlist.
If the video doesn't exist yet, it's
because I'm still working on it but it'll be there soon.
Bye!
[MUSIC PLAYING]