[MUSIC PLAYING]
THOMAS EZAN: Hello, my name is Thomas.
And I'm an Android Developer Relation Engineer
working on machine learning.
Today, I'm going to talk about the latest developments
of machine learning on Android.
I will show you how you can benefit from our latest
offering.
And we will see new ways to apply
on-device machine learning as mobile devices become
more powerful.
On-device machine learning has advantages over processing data
on the cloud.
First, it enables low latency by eliminating a round trip
to the server.
It also lets you process the user's data while you keep them
on the device.
And you can provide your feature offline,
since you don't need a connection to a server
to run the inference.
Finally, by running the inference locally,
you will reduce your cloud bill.
Let's take a look at a couple of experiences
enabled by on-device ML.
First, the new Lens AR Translate experience
is a perfect example of the benefits of on-device machine
learning.
It uses a model to predict the pixels behind the removed text,
allowing it to recreate the image.
So the translated text appears seamlessly integrated
as if it was part of the original picture.
Here, the low latency enabled by running inference on the device
is key.
Second, the Android team in India
is creating a solution to digitize important documents
by scanning and extracting its key information.
It leverages ML Kit On-device Text Recognition
API and other on-device ML classification models
to classify the documents, example,
in this case, a passport, and extract key fields.
All of this processing is done on-device.
Today, you have two options to add
machine learning on Android--
first, ML Kit.
It's a great way to start.
It offers a broad range of ML use
cases packaged in simple APIs.
Now, if you have a ML experience and you created your own ML
model customized for your use case,
then you can use Android's custom ML Stack.
Built on top of TensorFlow Lite, it
enables you to deploy ML-enabled features on Android
and keep control over the inference process and the user
experience.
We have exciting updates to share for both the ML Kit
and Android's custom ML Stack.
Let's start with the ML Kit and hand it over to David.
[MUSIC PLAYING]
DAVID LLOPIS: Hi, I'm David, Product Manager ML Kit.
As Thomas said, ML Kit enables turnkey ML features in your app
without needing ML expertise.
Let's take a look at what's new for ML Kit.
Over the last year, we have updated some of our existing
APIs by improving mobile performance,
for example, we increased bar code performance by 17%
by delivering many of these APIs via Google Play Services,
so you don't have to bundle them and increase your APK size.
For example, text recognition is now
available in five scripts via Google Play Services.
And by adding new functionalities, for example,
gesture classification in digital ink,
we also constantly work on new APIs.
We launched last year, Face Mesh Embedder, a very fast model
optimized for selfie-style capture for applications
such as video conferences, AR, or selfie upload flows.
And we are working on a new document scanning API.
Let's take a closer look at some of these APIs.
For bar code, we did not only improve the performance
of our model by 17%, but we also added a new option for you
to use a UI flow provided by Google instead of you
having to build your own.
This is Google Code Scanner.
And we just moved it from beta to GA.
With Google Code Scanner, you get access
to a high quality flow powered by Google Barcode model
and a provided UI that is consistent
across the ecosystem.
This flow is delivered via Google Play Services,
which has low APK size impact--
and you can use with only a few lines of code.
Finally, this flow helps increase user privacy.
Once you trigger it, the user points out a barcode.
And the flow only returns the barcode scan value.
Given you are not processing the image or the camera,
you don't need to ask for camera permission, which
results in more privacy and safety for the user.
We also brought a major improvement
to our Text Recognition API and launched it
via Google Play Services.
Now, you can extract text from images
not only in Latin languages, but in four new scripts--
Chinese, Devanagari, Japanese, and Korean.
Together, these scripts cover over 100 languages.
We also improved the mobile latency and accuracy
and added some new features.
You can now get the confidence score, the rotation
information, and recognition at the symbol or character level
versus previously only at the word level.
This can be useful, for example, in Chinese, where
a character can be a word.
Last but not least, I wanted to talk about the new Android API
we're building called Document Scanner.
Digitizing physical documents is a very common user flow.
Users turn to their phone to digitize, store, and retrieve
important documents--
for example, scanning your lunch receipt, your vaccination card,
your children's homework.
However, there was not a Google solution for this.
And each app has had to build its own, which led
to fragmented user experiences.
That's why we've built a high quality
flow that can be used consistently across Android
with the provided UI.
This flow is delivered via Google Play Services
and is very easy to use.
You can trigger the flow.
And the user will be in control of scanning their documents
in automatic or manual mode, cropping them, adding filters,
and then sending them back to your app.
And because it's done on-device and the user
decides which files to send back to the app,
you don't need to ask for camera permission.
We will be launching this flow later this year in Q4
but wanted to give you a preview today
so that you can start thinking how to include it in your app.
With that, I'll hand it over back to Thomas.
[MUSIC PLAYING]
THOMAS EZAN: Thanks, David.
Let's now talk about how to integrate your own custom
machine learning model to your Android app.
We heard your feedback.
And we know that running custom ML inference on Android
can sometimes be challenging.
You have to deal with a vast ecosystem of devices.
And identifying the best acceleration configuration
for your model can be difficult.
So today, I'm going to talk about Android's custom ML
Stack, a suite of components to make development of custom ML
features easier and faster.
First, we integrated TensorFlow Lite in Google Play Services
as the official inference engine for Android.
Second, we invested in hardware acceleration delegates
to easily leverage GPUs.
And finally, we created a new API
to help you find the optimal hardware acceleration
configuration for your model at runtime, the Acceleration
Service.
I'm very excited to share, in detail, the progress we made.
Let's start with TensorFlow Lite in Google Play Services.
TensorFlow Lite in Google Play Services
enables the developer to use TensorFlow Lite for Android
without having to bundle it with their application.
As a benefit, the application binary size
will be reduced by up to five megabytes.
And as the new stable versions of TensorFlow Lite
are fetched by Google Play Services,
developers will benefit from performance improvements
without any code changes.
Finally, TensorFlow Lite in Google Play Services
is the official machine learning inference engine
on Android, which means that new APIs
and services from Android's custom ML Stack
will be built on top of it.
Since its launch in September last year,
we've seen strong adoption.
Today, it is serving more than one billion users monthly.
And tens of thousands of apps are relying on it.
Let's take a look at some of them.
Internally, Google is actively leveraging TensorFlow Lite
in Google Play Services to solve technical problems that
didn't have a solution before.
Have you ever ordered a ride share
that arrived on the wrong side of the street?
The smartphone's GPS accuracy tends
to drop in dense urban areas where
the device doesn't have a direct line of sight
with the satellites.
Project Bluesky used in Android 8
and above uses machine learning and 3D building representations
to model the effects of the reflection
and increase the accuracy of the GPS in urban canyons
and for example, solve the wrong side of the street problem.
TensorFlow Lite in Google Play Services
is also used by Android developers outside of Google,
such as Wise.
Wise is a UK-based company operating globally,
enabling cross-border payments.
Wise uses TensorFlow Lite with Play Services
to streamline the sign-up process.
As new users take pictures of their identity documents
to open an account, Wise's custom ML model
can evaluate the quality of the pictures
and provide instant feedback to the users to properly frame it
or to take a less blurry shot, for example.
Verifying a user's identity is one of the biggest
costs for Wise's business.
So ensuring users are verified easily and on the first attempt
not only provide a great experience,
but also decreases the costs for Wise.
The ML Kit team at Google also uses TensorFlow Lite in Google
Play Services internally.
As a consequence, many Android developers
using ML Kit SDKs, such as the Barcode Scanning API, Language
Identification, or Smart Reply are also
indirectly relying on TensorFlow Lite in Google Play Services.
Last but not least, it is also used by Stripe, a payment
infrastructure company.
They built an open-source SDK for identity verification.
Stripe runs two different image classification models,
to check the user's ID document and to validate
the quality of a selfie confirming the user's identity.
By switching to TensorFlow Lite in Google Play Services,
Stripe reduced the size of the already very well-optimized SDK
by about 1.2 megabytes.
If you want to integrate TensorFlow Lite in Google Play
Services, the API is almost the same as the bundled version
of TensorFlow Lite.
So the migration should only require minor code changes.
To learn more about it, go to the TensorFlow documentation
on tensorflow.org.
Now, let's talk about hardware acceleration.
In general, app developers don't really
need to think about which chip is going to execute their code.
Most of the processing happens on CPU,
to the exception of graphics rendering handled by the GPU.
It is different from machine learning features.
Indeed, ML inference strongly benefits
from leveraging the specialized chips, such as GPU, the DSP,
or the GPU for Pixel phones.
We call it hardware acceleration.
In testing a Pose detection model on Pixel 6,
we noticed that the inference was 180% faster on GPU than CPU
and up to 200% faster on Pixel 6 GPU,
freeing up your CPU for other features.
TensorFlow Lite delegates are integrated with TensorFlow Lite
in Google Play Services.
And they are the best way to implement hardware acceleration
in your app.
Both the GPU delegates and the [INAUDIBLE] API
delegates to support some DSP and the Pixel TPU
are available via Google Play Services.
Unlike for TensorFlow Lite and Google Play Services,
you don't have to bundle them to your application
and increase your binary size.
And you will benefit from OTA updates,
as new version of the delegates are
published without code changes.
We are also actively working with hardware partners
to bring more delegates to Google Play Services.
So to learn more about acceleration delegates
and how to integrate them to your application,
go to the Documentation page on TensorFlow.
As we saw before, hardware acceleration
can dramatically improve inference performance,
up to 10 times faster in some cases.
And as a consequence, it can significantly
improve the user experience of your ML-enabled application.
However, identifying the right hardware acceleration
configuration for each device is challenging given the variety
of hardware on Android.
The Acceleration Service is a new API
built on top of TensorFlow Lite in Google Play Services that
enables you to pick the optimal hardware acceleration
configuration for each device.
Here is how it works.
When the user opens your application for the first time,
you will call the Acceleration Service API
and pass your ML model, some sample data, and the expected
inference results.
The Acceleration Service will then
run an internal ML inference benchmark
with your model and data sample on the chips
available on-device and return a performance report
and a hardware configuration recommendation
that you can directly apply to your TensorFlow Lite
interpreter.
Today, it supports the inference on CPU and GPU.
And we will be adding support for more chips in the future.
As mentioned earlier, the Acceleration Service
is already available in beta.
So if you want to benefit from hardware acceleration
without having to worry about latency, stability variation,
we strongly recommend to give it a try.
To learn more about it and integrate it
to your application, go to the Documentation page
on tensorflow.org.
We covered a lot today.
To summarize, if you want to add ML features without having
to worry about training your own model, start with ML Kits.
If you have a custom ML model or you
want to control the inference process or the user experience,
then take a look at Android's custom ML Stack.
We can't wait to see what you'll build.
ML on Android is constantly evolving.
So to continue learning about it and stay up to date,
visit the d.android.com/ml.
Thank you.
[MUSIC PLAYING]