Hello, and welcome to another ml5.js video tutorial.
In this video, I am going to attempt to classify
my drawing as a cat.
So how am I going to do this, and how come
this video is so darn short?
Well, I'm going to make use of a pre-trained model called
DoodleNet that comes as part of the ml5 library.
The DoodleNet model, which you can read more about
on the documentation page, is a pre-trained model
that classifies and labels hand-drawn sketches
from 345 categories.
Huh, I wonder what might these 345 categories be?
Well, if I scroll down and take a look at the DoodleNet data
biography, you will find out that the data set comes
from Google's "Quick, Draw!" which was a game that
was created in 2016.
And Yining Shi, who created and trained this model--
you could find out more about Yining by going to this link
here--
collected the data in 2019.
This is what the Google "Quick, Draw" data set is.
It's 50 million drawings.
I don't expect that Yining used all 50 million.
We can see 50,000 per category, so do the math--
50,000 times 345.
[BELL DING]
And you can find all of Yining's training source code
here in this DoodleNet repo itself and more information
about how the model is trained.
But guess what?
This video is actually really a follow-up to my coding
challenge about training a machine learning model
to recognize drawings of shapes--
circle, squares, and triangles.
So I did this whole process of collecting a data set,
training a model, and then deploying
that model all in the browser in a separate video.
And in this one I'm just going to quickly, basically,
do the same thing but use a more sophisticated, more robust
pre-trained model created by Yining Shi.
All right, here we go.
So this is my starter code.
What is it doing?
You can see this is very simple.
It's just creating a canvas-- it's 400 by 400.
I have a Clear button.
So when I press that Clear button,
it redraws the background, erasing the drawing.
And then the draw function, whenever the mouse is pressed,
it leaves a trail from my current mouse position
to my previous mouse position-- or maybe I
should say my previous mouse position to my current mouse
position.
Yeah, it's the same thing.
I also should note that I have imported the ml5 library.
I'm currently using version 0.6.0.
So if you want to match that version if you're
working trying to recreate this code,
that's the version I'm using.
Step one, create a variable to hold the image classifier,
and I'll call it doodleClassifier.
Step two, call imageClassifier--
give myself some more space here for the code--
And?
Indicate to ml5 that I want to load
the DoodleNet pre-trained model.
And then I need a callback for when that model is ready.
I need to make sure I have all the indentation correct.
Once the model is ready, I just want
to take the canvas itself and pass it
to the model for a prediction.
Once it's gotten that canvas and sent it
through the neural network, it will get results back--
a probability score, a confidence
score for every single one of those categories.
And I can retrieve that in a callback, which
I am naming gotResults.
Ml5 follows an error first callback pattern,
which means if something went wrong,
it will come back in the first argument as an error.
If everything went well, I'll get the results
in an object called results.
Handle the case of an error by just getting out of here
and logging the error.
And then let me just log the results.
Run this sketch.
In truth, I'm not so sure what this warning is.
These kinds-- this happened to me in my own example,
where if you ever have your image resolution not matching
what the neural network is expecting,
you could get an error.
This is just a warning, so I'm not
entirely sure what's going on there,
but I'm going to ignore it for right now.
And what I want to focus on is the results that
came back in a giant array.
Object, object, object, object, object, object, object, object,
sing the object song.
The way that this array is ordered
is whichever label has the highest confidence score
is going to come back first.
So this is just a blank, white canvas.
What does it think it is?
It's a line.
That kind of makes sense, like, the least amount of drawing,
would just be a line.
The next one is a snowman.
[LAUGHING] That also makes sense. (DRAMATICALLY) Look,
this is my art.
I call it snowman.
But, let's see-- if I draw some stuff,
I could classify it and hopefully
start to get things that make a little bit more sense.
So what I want to do is just display--
I think I'm just going to display the highest confidence.
Well, we can-- let's look at a couple of them.
Let's always look at the first two confidence scores.
So I'm going to make a variable for a div where I can draw--
not draw, but I can pass the information from the result
into an HTML element--
a DOM element.
Then when I get the results--
now, if I just wanted to show the label,
I could just pass the label in there.
But let me form something that has
a bit more information in it.
I'm going to use the backtick for a template literal, which
allows me to combine text and sort of the values of variables
in one.
And I will say results label followed
by results confidence--
this actually lets me do it on multiple lines, which
will be convenient here.
And then I also want to multiply this
by 100, number format it to only have two digits,
and then put a percent sign.
So it's kind of like the percentage confidence.
And then let's do the same for the next one.
Put the final backtick in here.
And then I probably should add a line break.
Is that how do you a line break?
I sort of know how HTML works so that I see
these on two separate lines.
Let's just see what happens here.
Model loading-- line--
why do I only see "line?"
Oh!
[LAUGHING] I spent all this time making this content variable
but I forgot to actually put it down there.
Apologies.
So content should go here.
And I'm a person who likes to use semicolons.
I'm sorry, but I just have to use the semicolons.
Here we go, model loading.
Let's see what we got.
Line 34-- oh, whoops.
I lost the line break.
There we go.
Line, 34 point percent.
No, I don't want that point percent.
Why is that showing up?
All right.
Let me give it one percentage point.
And then, of course, by the way, once I
have gotten the results--
just to get to the next step here--
I also will then want to classify it again.
So this is a way that I can continuously
loop over and over and over again
with the neural network model classifying, classifying,
classifying.
Classify the canvas, show the results.
Classify it again, show the new results.
Classify again, show the new results.
And I think now--
here we go.
Let's draw my cat.
Let me try this again.
Is there a cat even in there?
So I have a suspicion here, and this is always
really critical--
is the data set that was used to train this model
is incredibly consistent in that the line thickness of all
the drawings is very uniform.
So I think maybe I haven't really gotten that right.
I arbitrarily decided-- let me use a stroke weight of 8.
My suspicion is if I make that a little bit thicker,
I might get results that are more like what I'm hoping for.
Look, it's a cat!
It's a cat.
A nice, raccoon-like cat.
Oh, we're super-confident now.
Look at this.
Oh, all right.
So we can see these works quite well
if you happen to draw like the Google "Quick, Draw!" data set.
I think it's an important question
to ask-- if you are classifying drawings
and there's an audience that is experiencing your web
application or your project, is the Google "Quick, Draw!"
data set representative of that audience?
Are they represented in that data set?
That's kind of something that I talk a lot
about over many videos about thinking
about the sort of ethics and politics of data collection.
OK, and now for the next step.
Rather than draw onto a canvas, if I use a notepad here
with a document camera pointed at it,
can I get it to recognize my drawings on paper?
I am going to duplicate this sketch,
call it DoodleNet Video.
I'm going to add a variable called video.
I'm still going to use the canvas,
and I'll show you why in a second.
I could pass the video directly, but I'm
going to draw the video on the canvas and resize it.
And this should be exactly the same now.
As you can see, it's now seeing my notepad over here.
I'm going to start by trying to draw a strawberry. (SWEETLY)
A strawberry!
[MUSIC PLAYING]
That's sort of more like a heart.
[MUSIC PLAYING]
It's a beach again, or it's raining.
So why is this not working?
I'm not entirely sure, but let's see if I can--
even with this exact same image, kind of adjust it.
So one thing that I might think about doing
is applying some image processing to it.
So one thing that I could do very quickly is add a filter--
a threshold filter with a threshold of 0.5.
Let's try that.
A threshold filter takes any image
and converts every pixel to either black or white
based on some threshold value-- in this case 50%.
Let me try a thicker pen.
[LAUGHING]
[MUSIC PLAYING]
There we go.
It's a strawberry.
Let's draw-- I'm going to see if I can draw that cat now.
[MUSIC PLAYING]
I don't know.
It's either a cactus or a cat or maybe a hedgehog.
But you can see that this is working.
We now have the ability to take input from a camera.
I don't know to what extent this filtering is altering it.
Like, look-- right now it says it's 52.4% cat.
Let's actually try making the filter only
apply if I press the mouse, and we'll
see how that affects things.
This is what it looks like without the filter.
With the filter, you can see-- first of all,
it's much more stable.
With the threshold filter, I have a very consistent image.
But there's a lot of noise otherwise in the image.
I think it wants me-- it's like it's
telling me, it's daring me, to draw a snowman
to get that 1.8% up.
[MUSIC PLAYING]
But this is a snowman.
Look at that.
Look at that snowman.
[FANFARE]
There we go, a snowman.
All right.
Can I make it-- am I going to make it worse or better
by adding some arms?
Snowman, there we go.
OK, so I hope you enjoyed this particular video.
I would encourage you to experiment with this model.
I think there's a lot of creative possibilities in terms
of what this interaction might be applied to.
Could you play a game where you have to control
a character by drawing?
What happens when you pass images
that come from fill in the blank to this model?
I encourage you to play with that.
I encourage you to look at my shape classifier video
to train your own model that recognizes shapes and drawings
and that sort of thing.
Share it with me.
I would love to see what kinds of creative possible outcomes
come from this stuff.
OK, goodbye.
Hopefully there'll be more ml5 videos to come.
And I look forward to seeing you there.
[MUSIC PLAYING]