[MUSIC PLAYING]
PAUL RUIZ: Hey, y'all.
I'm Paul Ruiz, a developer advocate with Google.
And welcome back to "ML on Android with MediaPipe,"
a series introducing machine learning
and its application on Android with the MediaPipe framework.
In this video, you will learn how
to apply the machine learning concepts
that you learned about in the last video, which I will link
in the video description in case you missed it,
within your Android apps.
We will focus on a few tasks that we already talked about--
image classification, object detection,
and hand gesture recognition.
Each of these tasks work with still images or videos that
are opened from your user's gallery
or on frames pulled directly from their device camera.
As each of these will follow a similar coding paradigm,
let's start with image classification
for still images.
This will allow you to take an image
and identify multiple items within that image,
giving you both the name of what the ML model thinks it sees
and how confident it is in that prediction.
The first thing you'll need to do
is import the MediaPipe task of vision dependency into your app
through the Build.Gradle file.
After bringing in the dependency,
you will need to create the image classifier
object that will, as you might guess, classify images.
This will involve creating a baseOptions object that
will let you set the delegate, or hardware
that will be used for classification,
as well as the model that will be
used for image classification.
The model used will need to reside in your app's assets
folder.
Next, you will want to create a ImageClassifierOptions object.
This will let you set the score threshold, which
is the minimum amount of confidence the model needs
to have before returning a result,
the max number of results you want
to get back from the model, and the running mode.
In this example, the running mode
is set to RunningMode.Image because you
will work with a still image.
From there, you have a few different ways
that you can create the image classifier.
But for this video, you can use the createFromOptions method
to apply all of the options that you just created.
In the next step, once you have imported an image
from the gallery, which is a little outside of the scope
for this video, you can convert it to a bitmap
and then turn that bitmap into a new MP image
object using the MediaPipe BitmapImageBuilder utility.
Then you can pass that to the image classifier's
classify function, which will attempt to classify
the image synchronously.
Because this is a synchronous operation,
you will want to make sure that it is happening off
of your main UI thread.
Once the classification is complete,
you can do something with the result, which
is the label and confidence score that I mentioned earlier,
such as displaying the labels within your app.
Now that you've seen how you would make this,
let's take another look at what a finished app would look like.
You can see that we're able to open the gallery,
then wait a moment while inference happens, followed
by seeing the labels for what the model believes
it has detected.
For more detail, you can find all
of the code for this sample up on the MediaPipe sample's
GitHub repo.
Great.
So let's take a look at how you can do object detection
from a video file, which you can see in the GIF
here on the screen.
Creating the object detector is done in the same way
as creating the image classifier,
except now you will use the object detector's
version of options with a running mode set
to RunningMode.VIDEO.
The next thing to do is retrieve the URI
for a video in the user's gallery
so that it can be analyzed.
With the URI, you can retrieve the media
and get the video's duration.
You will need this duration later in the process
because each detection will need a timestamp for the frame
so the video can be processed in the correct order.
Since MediaMetadataRetriever returns
frames that are smaller than the actual dimensions
of the video file, you can also get the first frame
to find that frame's width and height.
While this isn't necessary for performing object detection,
it is necessary for displaying results back
on the screen at the proper scale.
If you don't need to display the results on the user's device,
then you can skip saving these dimensions.
But if you are interested in how we handle displaying bounding
boxes on the user's device, you can head over
to the official object detection sample
after watching to see how scaling and displaying is
performed by looking at the overlayview.kt file.
From here, you can figure out how many
frames will be analyzed based on a time
gap in between key frames.
In this case, we're using 300 milliseconds in between frames.
This example will also create a new list
that will contain all of the results
from the object detector.
Next, you will want to loop through each key frame
to perform object detection.
You can start by finding the timestamp for each key frame.
And once you have that value, you
can use the retriever to extract that frame
from the video as a bitmap.
From there, you will ensure that the bitmap
is formatted as ARGB_8888.
Then you can convert it to a MediaPipe image object.
At this point, the last thing you will need to do
is call the object detector's detectForVideo method
with that MP image and the timestamp
for the specific frame.
Once you have the result for that frame,
this example stores the result, which
contains the classification, confidence, and the information
about the bounding box in the results list
and then returns it to the app.
You will also want to make sure that you release the media
metadata retriever once you're done with it.
Just like with the image classification sample,
you can find this example on GitHub to try it out yourself.
Here we can see the object detection app
again highlighting detected objects
and displaying where they are within each key frame.
The final computer vision task for this video
is gesture recognition using the live camera
feed, which you can see here.
The gesture recognizer is a little
different than the other two examples
as it supports a minimum hand detection
confidence, a minimum hand presence confidence,
and a minimum tracking confidence that
needs to be met before a successful result can
be returned as a part of its available options.
Since you're using the camera live stream for this example,
you'll also need to set the running mode to LIVE_STREAM.
One important thing to note about this
is that unlike the previous two examples,
live stream recognition happens asynchronously.
This gives developers a huge advantage over video or image
modes when using the camera, as MediaPipe already
has the logic built into it to drop frames if they start
coming in faster than they can be processed,
while also not needing any extra overhead
like finding key frames.
To support this setup, you will need
to add a results listener and an error listener to the options
before creating your gesture recognizer object.
Once you have your gesture recognizer created,
you'll need to retrieve camera frames that can be analyzed.
This is made fairly easy through the use of CameraX.
Though a complete explanation of CameraX
is beyond the scope of this video,
you can find more information about CameraX
in the documentation linked in the video description below.
The relevant part of CameraX for on-device machine learning
is the image analyzer.
This component will run after every camera frame
and has a variety of options that
will be useful for working with your models.
This includes setting image rotation,
setting the camera ratio so that it's
closest to what the model expects,
and setting the image format.
Once the image analyzer has been called,
you can kick off recognition using the image proxy
object that is passed in from the camera.
With the image proxy object available,
you will need to convert it to an MP image for use
with MediaPipe.
There are a few different ways that you can do this,
depending on the output image format that was used,
such as RGBA or YUV.
So I'll refer to the documentation
to provide you with the latest and most accurate
information on how you can do this within your apps.
Plus, you can review the official sample
for gesture recognition to see our current recommended
implementation.
After you have the MP image object,
you can pass it to the recognizeAsync method
on the recognizer, along with the timestamp
for the current camera frame.
Any result from this operation will
be sent to the result or error listener
that you added to the options earlier so that the new content
can be displayed.
For gesture recognition, these results
include the label that the model thinks the gesture represents,
how confident it is, and a series
of landmark locations on the hand that
can be used to draw out the gesture information.
In the demo that we have on GitHub,
you can see how we're displaying landmarks from the hand
and connecting them in the UI, as well
as providing that classification for each hand gesture.
You can recognize things like thumbs-up, thumbs-down,
victory, or a closed fist, using an off-the-shelf model
that we use for our sample app.
And you've made it through the computer vision
section of this series.
That was an overview of adding machine learning capabilities
from MediaPipe for still images, videos, and a camera feed
into your Android apps, which gives you the ability
to add a lot of really cool and useful features to your apps
without a ton of development overhead.
When you do build those great apps,
please post them on Twitter and tag TensorFlow, share them
to the TensorFlow forums, or leave a comment right here
on YouTube, because we're honestly
excited to see all the cool things people make.
In the next video, you will step away
from Android for a moment to learn
how to create a new custom gesture recognition
model without having to dive into the underlying data
science.
So I'll see you there.
[MUSIC PLAYING]