[MUSIC PLAYING]
JEN PERSON: The MediaPipe image classifier task
lets you perform classifications on images.
Use this task to identify what an image represents
among a set of categories defined at training time.
To get started using the image classifier,
first take a look at the available image
classification models.
You can see this list in the documentation
linked in the description.
Now, this list might change over time.
So definitely check the docs for the latest.
There are two recommended models here
with two available formats each, EfficientNet-Lite0 and
EfficientNet-Lite2.
Both models use an efficient net architecture
and were trained using ImageNet to recognize a thousand
classes like trees, animals, food, vehicles, humans,
and more.
A complete list of classes is linked in the description.
The EfficientNet-Lite0 model is recommended
because it strikes a balance between latency and accuracy.
It is both accurate enough and lightweight enough
for most use cases.
EfficientNet-Lite2 is generally more accurate
than EfficientNet-Lite0.
But it's also slower and more memory intensive.
So EfficientNet-Lite2 is appropriate for use cases
where accuracy is a higher priority than speed or size.
If your use case requires a more unique classification solution,
then you can customize a model using MediaPipe Model Maker.
I've linked to a guide in the docs.
But if you'd like a video on getting started,
let me know in the comments.
Now that you've chosen a model, install
the tasks-vision package.
You can download the package using NPM
and use a JavaScript compilation tool like webpack.
Or you can import the package using a CDN.
Before continuing with the code, there
are a couple of concepts worth noting.
First MediaPipe for the web uses WebAssembly,
or Wasm, a binary instruction format for a stack-based VM.
Now, you don't need to know about Wasm
to use MediaPipe solutions for the web.
In simplest terms, Wasm allows nonweb-based code
to run on the web.
And second, as a developer, you're
accustomed to making your code lightweight
so your website can load quickly for a great user experience.
And machine learning has come a long way
in terms of both size and performance to the point
that inference can run on all sorts of devices.
So to keep that swift experience for users,
you don't want to bundle your model or Wasm
binary into your website.
Instead, you will store them server side and provide links
when initializing your classifier.
So let's take a look at that now.
Here we have a function createImageClassifier.
First we configure our Wasm binary loading
using the FilesetResolver.forVisionTasks
method.
Then we create the image classifier
using the imageClassifier. createFromOptions method,
passing in the file set resolver you just created and the model.
You can also provide optional parameters
like the maximum number of results the model should return
when running inference and the running mode
for inference, which is either image or video.
Image is the default value.
To run classification on an image,
use the imageClassifier.classify method,
passing the image source.
This function is synchronous, which
is good to keep in mind when designing your UI.
The source can be an HTMLCanvasElement,
HTMLVideoElement, HTMLImageElement,
ImageData, or Image Bitmap.
This function returns an image classification result object.
There are a series of classifications
in order of how confident the model is that the image belongs
to the given category.
In this example, the first category
has a display name of Passer domesticus, which
apparently is a house sparrow.
I had to look that one up.
It's very cute.
The score of 0.91406 is a score out of one
for how likely the image is to belong to the Passer domesticus
class.
So that's about 91%.
The scores of all categories will add up to a total of one.
So we can definitely stop there and assume
that's what's in our image.
You can access these classification results
using classificationRe sult.classifications
So if you want to access the display
name of the first result, you would
use classificationRe sult.classifications,
grab the first result, which, of course, is the zeroth,
and then use the displayName property.
To classify frames in a video, get the current time
using performance.now.
Then get the classification result
using the imageClassifier.classifyForVideo
method, passing your video element and the current time.
And that's it.
With this code, you can get started
with image classification in your own web app.
Check out a complete code example on CodePen.
And view all the available solutions
on the MediaPipe website.
Or get more hands-on with MediaPipe solutions
in MediaPipe Studio.
All these resources are linked in the description.
As you can see, there are a lot of exciting places
to go from here.
So keep me in the loop.
Tell me what you're working on.
Tell me what you learn.
Tell me what you still want to know.
Drop a comment below or on LinkedIn.
I can't wait to see what you build.
[MUSIC PLAYING]