[MUSIC PLAYING]
ZONGMIN SUN: Hello, everyone.
Welcome.
I'm ZongMin Sun, a software engineer on Google MLKit.
Together with my colleague, Valentin,
we're going to go behind the scenes on two of the most
popular APIs from MLKit--
text recognition V2 and pose detection.
MLKit is Google's machine learning SDK for mobile.
It brings Google's machine learning expertise
to mobile developers in a powerful and easy-to-use
package.
Make your Android and iOS apps more engaging, personalized,
and helpful with solutions that are optimized
to run on mobile devices.
Most importantly, it's free to use.
Since MLKit launched in 2018, we have created 12 unique APIs
with some of the best machine learning
models Google has to offer.
Covering core use cases in vision and natural language
processing.
We provide regular updates and new use cases.
We have moved out of beta and put quality assurance
as our top priority.
Ensuring production users are in good hands.
Each API is naturally different, but they
share a common infrastructure that provides stability
and optimal performance.
The MLKit SDK manages all the complexity
of multiple models, image processing, TensorFlow Lite
runtime, hardware acceleration, and much, much more.
Delivering a complete and easy-to-use solution.
MLKit APIs are in Kotlin and Java for Android.
Swift and Object C for iOS.
With documentation and sample code for them all.
I'm excited to announce that MLKit
has more than 40,000 monthly active apps, including
many top brands.
Text recognition is one of the most popular APIs on the MLKit.
In V1, enables recognition of text in any Latin based
character set.
It can be used to automate data entry tasks such as processing
credit cards, receipts, and business cards.
With support for real time processing,
and it's around 65 kilobytes per architecture
as an unbundled library through Google Play services.
It's a lot in a small package.
Now, I'm thrilled to introduce to you text recognition
V2 and some big improvements we've been working on.
With new pipeline models baked-in,
text recognition V2 is even more powerful.
It has all the features from V1 as well as
several new capabilities.
Support for multiple scripts beyond
Latin, deep paragraphing, which provides
better structural grouping across various layouts.
All-orientation recognition, and it's more than 30% faster.
The most exciting feature of V2 is the ability
to support multiple scripts.
In addition to Latin there are four new scripts--
Chinese, Devanagari, Japanese and Korean.
Under the hood, each one correspond
to an independent TensorFlow Lite model that
also supports Latin script.
You might not need all of them at once.
So you can choose one or more of the four options.
Together with the existing Latin only option,
there are five options available.
This allows for high flexibility and the reduced installation
size on device.
Altogether, V2 now supports 37 languages
representing around 4.9 billion people worldwide.
We are proud to dramatically expand support
to so many more people.
When we talk about text recognition, what we care about
is not only the characters, words, or lines,
but more importantly, the meaning.
Even if every line is processed accurately, breaking them apart
may not make sense.
Thanks to deep paragraphing, words and lines
are intelligently grouped into sentences and paragraphs
based on the context.
From the user's perspective, they
see not only the more meaningful results,
but also reduction of ambiguous recognition
due to the lack of context.
This is a significant quality improvement
with minimal additional cost.
Last but not least, V2 recognizes text
written in all orientations.
Here a piece of Latin text that has been rotated 90 degrees
can be recognized in V2, but not in V1.
This is especially helpful for scripts that are commonly
written vertically.
With this Chinese and Japanese scripts,
horizontal text, rotated text, vertical text,
can all be recognized.
All-orientation requires more computing resources
and is currently offered in the four new script options.
Now let's take a deeper look at how V2 works behind the scene.
Recognition lifecycle.
A recognition lifecycle consists of four main sequential steps.
Detection, script identification, recognition,
and language identification.
Detection is to detect the location of the text.
Script identification is to identify the script
of the detected text.
Then comes the recognition step, where corresponding models
are applied.
Finally it tries to determine the language it belongs to.
The recognized text is segmented into blocks, lines,
and elements.
Roughly speaking, a block is a contiguous set of text lines,
such as paragraph or column.
A line is a contiguous set of words on the same axis.
And the element is a contiguous set of alphanumeric characters
on the same axis.
Here you can see them in a descending order.
The first highlighted part is a block of text.
The second set of highlighted parts are lines of text.
Finally, the third set are elements.
With V2 there are five independent libraries
corresponding to the five options mentioned previously.
The workflows are similar across libraries
but with their own TensorFlow Lite model.
All MLKit APIs are designed to be easy to use.
Text recognition is no exception.
Now let's see how easy it is to integrate V2
with just a few lines of code and start using it today.
First of all, let's add the dependency in the gradle file.
In this demonstration there are five dependencies listed.
By name, they correspond to the five libraries
mentioned previously.
Each is independent of the others.
So we would just add the ones we need.
For example, if we only want to support Korean and Latin,
only the highlighted part is required.
Secondly, we need to create an instance of the text
recognizer.
In V1, there's only one option available, which is Latin only.
But V2 provides five different options.
The main difference between V1 and V2 APIs
is that V2 use text recognizer option
classes to choose the desired option when creating a text
recognizer instance.
For example, a Korean text recognizer option
is passed into the static function getClient
to create a text recognizer instance for Korean and Latin.
Once we have the recognizer, we also
need the input to start the recognition process.
Just like V1, the V2 APIs take an input image.
In the demonstration, it shows how to create an input
image from byteBuffer.
But note that the input image can be created
from many other common formats.
Now we have both the recognizer and the input image.
It is time to kick off the recognition.
All we need to do is to pass on the input image
to the recognizer's process function
and set up listeners for result handling.
The result is returned in the format of text blocks, which
contains information about nested blocks, lines,
and elements.
Each layer also has additional information
about corner points, bounding boxes, and recognized language.
And that's all you need to integrate text recognition V2
and start using it.
If you are interested in learning more,
we have online resources, including
detailed guides, sample apps, and API references.
Just search Google MLKit and start from there.
Next, I will hand over to my colleague, Valentin,
to talk about pose detection.
VALENTIN BAZAREVSKY: Hello.
I'm Valentin, and I will present MLKit pose detection.
When we started this project, we aimed to develop an API that
would be beyond the existing state-of-the-art solutions
and would work in real time on a device.
And today we have won.
MLKit pose detection API provides both 2D and 3D
keypoints, supports custom pose classification,
and has the industry leading quality for the most
demanded mobile use cases.
Such as yoga, dance, fitness, and sign language.
In the recent release, we added 3D keypoints,
which provides relative depth and unlock pose analysis
in 3D space in a view-invariant manner, which simplifies
pose estimation a lot.
Let's have a look at a pose topology.
We extended the existing de facto standard,
17 keypoint topology, with hand, feet,
and face keypoints, which are marked with yellow on a figure.
And now it is possible to estimate
scale and rotation for hands and feet for instance.
MLKit pose API is available in two mobile platforms--
Android and iOS in two versions, Lite and Full.
And in addition to that, the same pose solution
called BlazePose is also available in MediaPipe and TFJS
via Python and JavaScript.
Let's have a look at how to use it.
First, we should select a mode, which
can be stream mode or single image mode,
and then create a pose detector object.
Then, we process the image with the pose detector
and that's it.
If a person is present, the pose would be detected.
Now let's have a look under the hood and see how it works.
Our pose detection API is a 2-step pipeline,
consisting of a pose detector and a pose tracker.
In the first frame, we run the detector,
which locates a rough area around the person, which
we call a person box.
If a person is found for this person box area
we run the tracker model, which detects accurate locations
for all the keypoints.
For the subsequent frames, the logic is a bit different.
We estimate the person box based on the keypoints
from the previous frame.
In addition to keypoints, the pose tracker model
also predicts if a person is located in this person box.
If not, we will restart the logic around the detector
again.
A couple of words about person box.
In our case, it is oriented.
So it goes not only position and scale but also orientation,
as shown in the video.
We encode person box via two auxiliary keypoints,
which are predicted both by detector and tracker.
Recently, we added 3D pose, and it is a vital component
for such applications as motion capture, AR effects,
and angle-based pose representation.
However, the main challenge for 3D poses is data.
In contrast to 2D, it is impossible to annotate existing
photos or videos manually.
There are three main approaches which are common nowadays.
First option is to create a synthetic data set.
However, this usually introduces additional challenges
with the main transfer in realistic poses.
The second option is to collect data in the lab
by multiple cameras and sensors.
However, this approach lacks a realistic environment.
We decided to go with a third option, which
is to use the existing 2D data set, which doesn't have issues
with the realistic environment or pose articulation and fit it
with a statistical body pose model.
For this purpose, we took the generative parametric human
model-- called GHUM--
which was recently released by Google.
It is based on 3D body scans, supports
full body articulation, non-linear body shapes,
and facial expressions.
So it perfectly fits our case.
We did the fitting by optimizing pose and shape
were the next objectives.
2D keypoints alignment, body silhouette alignment,
and additional shape and pose regularization
to preserve realistic poses.
And then based on these 3D ground truth,
we trained our pose detection models.
Another use case that many developers need
is custom pose classification.
To ease the usage of our pose detection API
for pose classification, we recently released an example
in an MLKit Quickstart app and MediaPipe Colab
to prepare the data for developer-defined custom poses.
Let's have a look at how to set it up.
First we need to collect images with desired poses
from different viewpoints, lighting conditions,
and different people performing this pose.
Usually, a couple of hundreds of samples of posture
should be enough for the start.
Once collected, you just need to organize these photos
in folders so all the samples of a particular class
are located in the same folder.
Please note that the folder name will be the posture class name.
Once collected and organized, you
will need to attach this data to MediaPipe Colab
and then update the input folder and output CSV file variables.
Additionally for debugging purposes,
you can set up output folder variables
and see the keypoint visualization for your images.
Then you'll need to run the Colab, which
prepares the CSV file.
Third, and the last step, is to build the MLKit Quickstart app.
You will need to attach the generated CSV file
and update pose samples file and pose classes variables
according to your case.
And that's it.
Once the application is built, you
will have your own custom pose classification.
For more details, you can read our blog poses that we released
recently about BlazePose, which is a model behind the MLKit
pose detection API, and holistic pose tracking block, which
is a more sophisticated pipeline built on top of BlazePose
and provides 540 key points with detailed hands
and face articulation.
Thank you.
[MUSIC PLAYING]