[MUSIC PLAYING]
KHANH LEVIET: Hi, everyone.
My name is Khanh, and I'm a developer advocate at Google.
Today I'd like to introduce you to MediaPipe,
a low-code no-code framework to build and deploy
on-device machine learning solutions across platforms.
If you are interested in integrating machine learning
into your mobile web IoT applications
in just a few lines of code, then this session is for you.
I'll start with showing you what MediaPipe is
and how you can use MediaPipe to integrate
on-device machine learning into your applications
in just a few lines of code.
Then I'll wrap up with a sneak peek of our future plans,
such as our exploration into on-device generative AI.
Let's get started.
So what is on-device machine learning?
On-device machine learning is a kind of machine learning
that runs entirely on user devices, such as smartphones
or web browsers, without sending user data
to the server for processing.
You probably have used some kind of on-device machine learning
before.
For example, many magical features in Google Pixel
are powered by on-device machine learning, such as the Magic
Eraser feature that allows you to erase photo
bombers from your photos, or the live caption
feature that transcribes audio playing
on your phone in real time.
On-device machine learning has been a difficult engineering
problem.
For example, let's take a look at a seemingly simple machine
learning task, hand gesture recognition.
It takes an image as the input and returns hand gestures
found in the image.
Here it detects a thumbs-up gesture with 63% confidence.
Although it seems simple, this task actually
requires four different machine learning
models chained together.
The first one detects hands from the image.
The second one detects the hand landmarks.
The third one creates an embedding vector
of the gestures.
And the fourth one classifies this embedding as a thumbs-up.
On top of it, there are many other things
that you need to take care of, such as how
to run the whole pipeline efficiently on GPU,
or how to deploy it on very different platforms,
such as Android and web.
The good news is MediaPipe abstracts
all of this complexity and offers you
low-code and no-code tools to get
started with on-device machine learning easily.
We do that by focusing on a specific set of common machine
learning use cases and provide easy-to-use on-device solutions
for them.
We have 14 solutions available today,
and hand gesture recognition is one of them.
MediaPipe is built on top of TensorFlow Lite, which
is Google's official framework for on-device machine
inference.
Each solution in MediaPipe contains one or more TensorFlow
Lite models.
And MediaPipe provides a pipeline
that chains the models together for you.
The MediaPipe solution for hand gesture recognitions
has already done this pipelining work for you,
so you don't need to worry about orchestrating all these models.
You only need to interact with a simple API,
fit the input image in, and get the prediction result out.
Next, let me show you the MediaPipe API in more detail.
[MUSIC PLAYING]
To integrate the hand gesture recognition solution
into your app, you will need to use a library called MediaPipe
Tasks.
This is the calling code that you need to write.
The first step is to create a gesture recognizer option.
You need to specify the model bundle that you want to use.
This model bundle contains the four TensorFlow Lite models
that are needed for this solution.
Then you define the input image.
Here I used an [? Android ?] bitmap object.
And finally, you call the recognize function
on the input image.
Here's the recognition result object.
You have a list of session objects,
each representing a potential type of gesture
found in the input image.
And if you also have a web app that you
need to integrate this solution, then no problem.
You can do it very easily.
This is the JavaScript code that you should add to your web app.
It follows the same flow as in Android.
Input the image in, detected gesture out.
Very easy, isn't it?
You may wonder what are the platforms that MediaPipe
currently supports.
Well, we already have support for Android, web, and Python.
iOS support is coming soon.
And if you wonder what solutions we provide today other
than hand gesture recognition, then you
can check them out easily in MediaPipe Studio.
And let me show you how.
MediaPipe Studio is a web app.
If you go there, you will find a playground
where you can try out all on-device machine learning
solutions from MediaPipe right in your web browser.
Of course, you can try out the hand gesture recognition
solutions in MediaPipe Studio.
You can see that it can recognize thumbs up,
thumbs down, or victory gestures in real time.
This solution can recognize single hand gestures today.
We are adding support for two-hand gestures.
And you will see them coming soon.
Let's try another solution, image segmentation.
MediaPipe offers multiple off-the-shelf models
for this solution.
One of them is a selfie segmentation model
that can differentiate between a person and the background.
It's the kind of model used in Google Meet
to blur or change your background
in video conferences.
We are excited to announce the new segmentation solutions that
can segment more general objects that it hasn't seen
during the training process.
This solution takes an image and a point of interest
as an input.
Then it gives you a segmentation mask
that distinguishes the object that's a point of interest
from the rest of the image.
For example, given this input image,
you can click on the dog or the person
to segment them in the image.
Let me show you another one, face landmark detection.
This model can detect more than 400 different landmarks
on your face.
We are excited to announce a new feature in the face landmark
detection solution that gives you
52 blendshape scores representing
different facial expressions, such as open mouth
or raising eyebrow.
This technology helps you create virtual avatars that
mimic humans' facial movement.
You can find this demo and this source code
in the video description.
KDDI is a telecommunications company in Japan
that wants to innovate the entertainment
experience with XR and 5G.
They are experimenting with this MediaPipe technology
in their product.
They use a local computer running MediaPipe
to capture a human performer's facial expression
and transfer it to a high-quality 3D
virtual avatar in real time.
Then it's combined with real-time pose detection
to recreate the whole virtual performer in 3D.
The performance is then live streamed to viewers
through Google Cloud.
MediaPipe isn't limited to just computer vision domains.
We also have text and audio solutions,
such as text classification or audio classification.
Just go to MediaPipe Studio if you
want to explore what kinds of magic
you can create with on-device machine learning.
[MUSIC PLAYING]
In MediaPipe, we put a lot of effort
in creating new on-device machine learning solutions
that just work out of the box.
However, we also know that in many cases,
you need to customize the solutions to fit your own use
cases.
Let's take the hand gesture recognition solution
as an example.
The off-the-shelf model can recognize seven different types
of gestures, as I show here.
But they don't include gestures like rock, paper, or scissors.
So if you want to view an app to let users play the rock, paper,
scissors game with a computer, then you
will need a way to customize the hand gesture recognition
solution.
The good news is that you can easily do so
with MediaPipe Model Maker.
Many of you may find this product sounds familiar.
Yes, you're right.
MediaPipe Model Maker follows the same concept
as TensorFlow Lite Model Maker, but it was built entirely
from scratch to be a local library to customize solutions
provided by MediaPipe.
Let me show you how to use Model Maker to create a custom hand
gesture recognizer that can recognize rock, paper,
and scissor gestures.
You need to start with collecting a training
data set with hands showing rock, paper,
and scissors gestures.
Once you have the data set, you can
start using Model Maker to train a custom model
to recognize these gestures.
Model Maker is a Python library, so you
can use Google Codelab to train model faster using
Codelab's free GPUs.
First, you need to import the gesture recognizer model.
Then you load the rock, paper, scissor data set and start
training the custom model.
After model training has finished,
you can use the test data set which the model hasn't
seen during the training process to check
the accuracy of the model.
And finally, you can export it to deploy on-device
using MediaPipe Tasks.
You may wonder why the exported model has a dot-task extension
but not dot-tflite.
It is because its hand gesture recognition solutions requires
multiple TensorFlow Lite models, so this dot-task file
is actually a bundle containing multiple TensorFlow Lite
models and some metadata.
After you have your custom model bundle,
you can upload it to MediaPipe Studio
to try it out right away.
You can see that the custom solutions can now
recognize rock, paper, and scissor gestures.
Deploying the new custom solutions on Android
is very easy.
You just need to replace the off-the-self model bundle
with the new custom model bundle that you have just
created using Model Maker.
The rest of the code stays the same.
We have made a rock, paper, scissors game
as a web demo using this custom model.
You can find the URL of the demo and its source code
in the video description.
Try it out and see if you can beat the computer.
I hope the rock, paper, scissors example gave you
an idea of how easy it is to view
your custom on-device machine learning solutions
using MediaPipe.
Today, you need to write some Python code
to create custom solutions.
But we're working on adding a no-code customization
tool into MediaPipe Studio.
Soon you will be able to upload your training data to Studio
to create custom solutions without writing
a single line of code.
On top of the features that are already available today,
we're actively working on some exciting new features
that will push the limits of on-device machine learning.
[MUSIC PLAYING]
Generative AI has been a hot topic
gaining significant interest since the end of last year.
Today's generative AI models, such as Imagine or LaMDA
are still gigantic, so they are not yet
suitable to run on device.
However, we found that a technique
called model distillation can be used
to distill those general purpose large models into smaller
models that can run on device and specialize
on a certain number of tasks.
We are excited to announce an experimental on-device solution
that allows you to stylize faces in images.
For example, you can use this solution
to transform your photos into cartoon style.
This solution supports customization,
which means you can prepare a few style
examples, such as animate faces and use Model Maker to create
a custom solution that can transfer
any faces to the style.
Please only use this solution on your face or faces
of people who have given you permission to do so.
Another generative AI model that we're running experiments with
is diffusion-based image generation model.
We were able to generate images from a text
prompt on higher-end Android phones within a few seconds.
We're working on reducing the model
size so that it's more suitable to be used in actual apps.
Stay tuned for more updates.
We're also doing experiments with on-device text generation
solutions.
We were able to deploy a large language model on Android that
can help you do several tasks with natural languages,
such as summing up a long conversation,
or writing a formal email from a given topic.
On-device generative AI is in its early days.
So there are still many technical challenges
we need to overcome to make them ready for production.
Stay tuned for our updates in the coming months.
So there was a lot of content that we covered today.
Before we wrap up, let's recap the most important topics.
MediaPipe provides you off-the-shelf on-device machine
learning solutions that you can easily
integrate into your mobile or web apps
with just a few lines of code.
MediaPipe Studio allows you to try out these solutions right
in your web browser so that you can come up
with ideas about how on-device machine learning can
be integrated into your apps.
Many of these solutions can be customized
using your own data set.
And MediaPipe Model Maker is a local Python library
you can use to customize MediaPipe solutions.
If you want to learn more about MediaPipe,
check out our site at developers.google.com/MediaPipe.
Thank you for watching, and don't
forget to subscribe to our TensorFlow YouTube channel
for more content about on-device machine learning.
[MUSIC PLAYING]