[MUSIC PLAYING]
SHIYU HU: ML Kit is Google's machine learning
SDK for mobile developers.
And its mission is to help mobile developers
to use machine learning technology easily
with or without a machine learning background.
Internally, ML Kit is built on top of TensorFlow Lite, which
is a light virtual TensorFlow and designed for mobile devices
and other agile devices.
And ML Kit is a cross platform SDK,
so it's variable in both Android and iOS.
Since the release in 2018, ML Kit
actually get adopted by a lot of developers and users.
So recently, we're just very excited to realize
we achieved one billion monthly-use sessions.
So it's actually a great milestone for us.
OK, and then let's recap a bit about what is ML Kit softwares.
So back to 2018, ML Kit released a set
of vision-based features and the Custom Model Serving service.
The Custom Model Serving service is actually
can download or upload--
I mean update custom models dynamically
without you updating your app.
And until this year's I/O, we released
a new set of features for natural language processing.
We also extend, which is the AutoML Vision Edge
for custom model chaining--
I mean, AutoML integration with ML Kit
actually enabled the developers who do have the data,
but do not have the expertise to change their models.
So you can change your custom model
without ML Vision Edge in ML Kit.
And in the Vision category, we actually add one more API
called Object Detection Tracking.
This is actually my personal favorite feature.
Not because it's, like, we're only adding one more feature
here, but we're actually working with a material design team
to come out with some US guidelines
to help you to define the UX working flow for on-device ML.
And our Showcase app adopted this guideline.
And then we opensourced it in GitHub.
So feel free to grab it and to try it out.
OK, and ML Kit is not only adding new features,
we're actually improving existing ones.
So a couple weeks back, we releasing a new update
for the barcode scanning we called V2.
And we'll share more details very soon.
OK, so in today's talk, we not only
want to show you what ML Kit has,
but also want to show how ML Kit could help you
to solving some complex use cases in real life.
Some developers came to us, saying, OK, I
have, like, a use case.
How can I train a machine model to solve it?
Well, for a complex use case in real life,
normally we can now solve it with a single model.
We normally split that task into multiple steps.
And for every step, we'd normally
come out with a machine learning model or some module
to solve it.
And then, we'd chain them together to solve the problem.
And also, when we want to move in the whole solution
into production, we normally also need
to think about performance-- optimization.
So it's like, we really want to perform
this well in real production.
So sometimes we have to run different pipelines in parallel
so that we can hit the performance goal.
That is exactly what ML Kit is doing.
We change the model, and then we put models together
into a pipeline, and then we tune the pipeline together
to give you a good performance.
OK, in today's talk, I will show you--
we will show you two examples with how ML Kit help
you to solve these problems.
The first example is Barcode V2 we just released.
Before we're talking V2, let's see what V1 is.
Barcode V1 is one of the first versions--
I mean features we released in 2018.
And this has supporting a wide range of barcode formats.
And it can automatically detect what barcode format it is.
And it's also supporting different orientation.
It's well-received by many developers.
So until now, I think there was more than 3,000 apps actually
using this feature in the market.
Since there are so many people using this feature,
we also get a lot of feedback.
And in some scenarios, V1's not working very well
in some scenarios.
And we'll see what are the challenges here.
The first challenge is, like, OK,
some people will scan the barcode--
I mean, when the barcode is very far away.
In that case, like, the barcode's
kind of small relative to the whole image.
Then the latency gets increased.
So recently, it's like, in a barcode decoder,
we're actually trying to find the barcode starting
from every pixel.
So if the barcode is big and then find the barcode,
it will skip all the pixels covered by this barcode.
But if this barcode is small, the skipped pixels
will be much less.
That's why the sides of the barcode inside the image
will affect the latency.
That's one challenge.
Second-- some developers say, OK, some user
want to try to scan the barcode, but they're not always
straight on their barcode.
They have some angles on it.
Like, scan from another angle.
Then the barcode's actually distorted.
And then they cannot detect a barcode correctly.
OK, this is actually another assumption
we have for the decoder.
The decoder trying to find the barcode with the boundaries
horizontally and vertically.
If it's not that, then it potentially
has some failures on it.
That is also why a lot of, like, barcode scanners in the market
do have a bounding box in the screen.
And trying to let the users put the barcodes
inside the bounding box to better decode their barcode.
OK, so some other challenges we have-- like,
for example, the first one is if the image size
is very big or getting bigger, then
the latency will get increased.
It's actually the same reason we mentioned the challenge one.
So it's like, we tried more pixels.
And the pixel more, and then image latency will be higher.
And also, some, like, people try, barcode scanner
was busy background.
And the background has sometimes interfered with the barcode,
so it's confused the decoder.
OK, let's go back to see what did
V1 actually do in the pipeline.
So V1-- Barcode V1 pipeline is actually quite simple.
It's has only two stage.
So one stage, the first stage, is
trying to make the image into the grayscale--
an all black and white image.
And then send it to the decoder.
If that decoder cannot solving the problem just that we
mentioned, like the challenges there,
they cannot solve it correctly.
So how we can improve it?
So then we come out with Barcode V2--
OK, great.
So actually adding two more steps before that.
So first one is actually a new machine learning model
to try and find the barcode inside the image.
And then the second step is, like, to--
it's an algorithm module to transform the crop before they
send it to the decoder.
Let's see how it can help.
OK, for the distorted barcode, the input is distorted.
We saw it before--
well, first, the localizer will find the corner points
of this barcode.
And then we apply the perspective
transform to transform that barcode
to be [INAUDIBLE] aligned with vertical and horizontal.
And then the decoder is happy.
How would you handle the latency parts when the barcode is small
or the image size is big?
Similar with using localizer defining the box first--
and then it was just easy defining the bounding box
of the barcode.
We just crop it.
Quite simple.
And then, we [INAUDIBLE] their image with the barcode.
The rest of the image disappeared.
Similar for how to handle the big size of image--
similar story.
Since, like, we only care the part
of the image with the barcode, the rest of the image
will not affect your latency.
And the busy background will also be cropped out
with the new model--
put it into the pipeline.
OK, and then here is [INAUDIBLE] video for it.
So for the V1, it's once you angle,
the barcode disappeared-- bounding box disappeared.
And when it's moving far away, you
can see the latency get increased
from 20-ish milliseconds to 120 or 140-ish milliseconds.
So actually, the latency increase-- actually,
non-trivial.
Let's see Barcode V2.
So when scan from the angle, the barcode is still be detected.
And when you move far away from it, the latency stay--
and even smaller.
The reason is, like, the number of the pixel
gets small to be scanned.
Since, like, barcodes get smaller--
so that is how it helps.
Cool.
So in this example we show, we do add two more steps.
In the pipeline.
We're actually reducing the latency.
And also, we improve the accuracy.
This is internal benchmark results,
and there's almost two digits improvements for all expected.
And FYI, we actually-- in Barcode V1,
we do not support the non-UTF8 formats.
And then for here, we actually add one more API
that supports non-UTF8 formats.
It's actually another request from, like, GitHub.
OK, so for this example, we show you
how we're using pipeline model together
to improve, like, accuracy and the performance internally.
And then I will hand it over to Julie.
And then she will show you how you
can use the ML Kit features for building your own pipelines.
JULIE ZHOU: Thank you, Shiyu.
Hi, I'm Julie.
I'm also a software engineer at ML Kit.
Actually, the pipeline that Shiyu shared
can apply to more than barcodes.
The great news is that you don't even
have to write your own localizer.
You can use ML Kit to do this for you.
And this magic comes from ODT--
Optic Detection and Tracking.
Today, I'd like to share an example of how
we use ODT to help you to improve your user experience
together with Image Labeler.
First, let's take a look at a real life application of this.
Let's say we have a mobile app that requires the user
to upload a valid form of ID, which
can be your driver license.
And this could be something on our [INAUDIBLE] app that
will occur.
And usually, we will notice that people often don't upload
the right type of document.
And we would like to catch that earlier in the process.
Well, first, we may want to start with our Image Labeler.
Usually, an on-device image labeler
returns about 400 generic labels,
which could be used for categorizing images.
But for this particular case, looks like these labels are not
what we're looking for.
What we're looking for is to identify this basic type
of this document in the view.
And that's why we may want to train our own model
for document classification.
And this model can be easily trained in ML Kit
as well using the AutoML Vision Edge
service where you upload a data set of images
with labels like driver license, business
card, and some other random images with labels as others.
And these data sets can be as few as tens of images,
which is quite light-weighted.
And then AutoML Vision Edge will generate
an optimized TensorFlow Lite model
that can be used on-device for image classification.
OK, now we get our light-weighted custom model.
And then we're going to build our own Image
Labeler with this custom model.
Let's have a try with it.
Yeah.
And then, we're going to take a photo of the driver license,
and pass that into this Image Labeler,
and see if it will recognize my driver license.
No-- others.
It means this driver license is not correctly recognized.
But what's wrong here?
Clearly, this image is very clear and very carefully taken.
But still, the Image Labeler cannot recognize it.
There must be something wrong with the Image Labeler.
Let's take a look back at what did the model learn.
There are five images uploaded, because it's
a light-weighted model.
There are five images uploaded with the label driver license.
And, of course, we will choose very carefully
with those images with no background.
But in real life, it's difficult to take such a perfect image
with minimum background noise.
Well, this background noise will confuse Image Labeler a lot.
If you want to help the Image Labeler to recognize correctly,
we may want to crop out the background noise,
and just crop the content of the document,
and pass that into Image Labeler.
How can we do that?
We can have a try with ODT.
Internally, ODT runs two pipelines in parallel
so that they can return the real time location
of the object in the camera.
And using this location information,
we can crop out the content of the document
and just pass this little cropped image
into the Image Labeler.
Sounds like a good idea, right?
Let's see how it works.
I tried the first one.
Others-- no surprising.
And then, business card.
Good.
Now the driver license.
Awesome, right?
And that's what we get from ODT.
And actually, this pipeline is very similar to what Shiyu
introduced about Barcodes.
The only difference is that the Barcode pipeline
is kind of an internal version made especially
for the zebra detection.
While for ODT pipeline, it's external one
that you can use for a more general objects detection.
And-- actually, you can always use that in your code
with ODT with a very simple setup.
Next, I'd like to glue all this together in a code snippet.
First, we'll set up our on-device AutoML Image Labeler
instance with the custom model that we trained.
And then we'll get on-device object detector and construct
the Vision image from the--
byteBuffer from the camera.
And then we parse the image to the detector.
The detector will return the bounding box,
which is the location of the image in every frame.
And then with cropped image, using this
bounding box information, and pass this cropped image
to the Image Labeler.
In this case, it will return the exact right label
that we're looking for.
If you find this pattern helpful for you
and want to do more tricks in your app with ODT,
welcome to check the free code of our Showcase app
in the Github.
It's a fully-polished app with Google material design.
And I hope you will like it.
And then, next, I hand it over to Shiyu.
Thank you.
[APPLAUSE]
SHIYU HU: Cool.
Thank you, Julie.
So I will do a quick summary here.
So in this talk, we showed you two examples
of how ML Kit handle some, like, use cases in real life.
And in the first example, we showed you
how we do it internally.
And then, so if--
I mean, ML Kit's features can fit into a use case completely.
We will encourage you to try using
ML Kit to solve the problem.
And then you do not need to use starting from scratch.
But if ML Kit features cannot fit into your use case
completely, we encourage you to using ML Kit as a building
block to building your own pipeline.
And you can try using--
I mean, AutoML Vision Edge to train your own model, then
to differentiate your own app.
OK, and one more thing.
As mentioned in the keynotes, Android Developer Challenge
is on.
And the first topic is a machine learning challenge.
And ML Kit is one of the main tools
you can use in this challenge.
If you want to know more about details,
go check out with this website.
And then, if you want to know more about ML Kit,
this is, well, the website you can go to.
And then, if you have some questions,
feel free to reach out to us, and we are happy to help.
And then, thank you for your attention.
Thank you.
[MUSIC PLAYING]