Hi, I’m Oli, a product manager at Android, and I’m going to talk to you about
how we're making it easier to deploy ML in your own apps.
Machine learning has brought about huge advancements
over the last decade in vision, speech and language tasks.
And on-device machine learning can bring a range of benefits
over traditional server-based approaches.
By processing data locally on the device,
you can reduce your power demands through low radio usage,
improve latency by removing round trips, and minimize exposure to private data.
Additionally, you can ensure that your features work
just as well offline as they do online.
Now, these capabilities enable great new experiences.
Things like Google Meet's new background blur
and replacement are only possible when running on-device
to provide a really responsive experience.
Live Caption enables real-time transcription of calls
without sending any of that data to servers.
The ML Kit team are providing easy access to advanced features
like real-time pose detection, offline translation,
and digital ink, all running completely on the device.
Now if you’d like to learn more about Google’s on-device machine learning products,
visit our new landing page at g.co/on-device-ml.
This page will teach you all about the benefits of on-device machine learning
and help you identify the right solution
across all of Google’s machine learning offerings.
Now, ML on Android is growing fast.
From our start a couple of years ago, we now have over 730 million devices
with Neural Networks API hardware acceleration.
ML Kit has over 500 million monthly active users.
But in speaking with development teams both inside and outside of Google,
we found a common set of challenges with deploying these features on Android today.
Many apps are size constrained.
Having to bundle and manage additional libraries
just for ML can be a significant cost.
Unlike server-based ML, the computer environment is highly heterogeneous,
resulting in significant differences in performance,
stability and accuracy from device to device.
Maximizing reach can also result in using the lowest common available APIs,
and this limits the usage of the latest and greatest features and advances in ML.
We’ve also heard time and again that there should be a common set of components
for solving these common problems, and they should all just work together.
So that's exactly what we're doing.
We're introducing some big changes to provide Android
with an updateable, fully integrated ML inference stack.
We’re going to provide all the things you need
for on-device inference on the device as standard.
No need to bundle this code in your app.
We're optimizing the integration with Android
to deliver the best performance on CPU and beyond,
including when to enable hardware acceleration.
And providing a consistent API that spans Android versions
with regular updates coming direct from Google
and made available outside of Android dessert releases.
Now we'll be making these available to all developers
to enable you to produce your own amazing ML-powered experiences.
But Google SDKs like ML Kit are being built on top as well.
If you use them, you’ll get all these benefits for free.
And with that, I’d like to introduce you to Mika
who’s going to tell you more about how each of these new components work.
Thanks, Oli.
Hi, I’m Mika. I’m a software engineer.
I’m going to talk about the key underlying technologies for ML on Android.
First, let’s talk about TensorFlow Lite on and for Android.
This isn’t strictly new, but I wanted to highlight that TensorFlow Lite
is Google’s recommended inference engine for Android.
TFLite is a production-quality library
that is used in many Google apps and for billions of users.
TFLite has a high-performance CPU backend
and supports basic hardware acceleration on Android.
What is new is TensorFlow Lite for Android.
This year, we are making TensorFlow Lite pre-installed
on Android devices with Google Play Services.
This means that you no longer have to bundle TFLite,
and you can use your APK size for the feed source your users want.
Historically, TFLite has shown about 50% improvement
on CPU performance year on year.
By using TFLite for Android, you’ll get those benefits automatically.
And we, Google, are on the hook for making sure that the updates to TFLite
don’t break your app, like we are for other Google Play Services libraries.
TFLite for Android will also integrate automatic acceleration.
By adding metadata to your model,
it will help you pick the best performing backend for your model.
You can sign up for an early access program at g.co/androidml-eap.
So let's continue discussing performance.
Using GPU to run your TFLite models can make the difference between
something that runs in real-time as part of the camera viewfinder,
versus something that you can only run on single images.
However, realizing these gains on Android is painful.
Performance, stability and accuracy all vary across chipsets and GPU drivers.
Developers who want to use GPU today need to set up experiments,
get data from real users’ devices,
and use those to build targeting for GPU-based features.
This is super complicated. I know, I’ve been there.
So you might be asking, why are you on the hook for doing this?
So we have a partial solution today.
From TensorFlow 2.3, we have a built-in allow list for GPU acceleration on Android.
The allow list is part of the GPU Delicate Maven package
and using it is as simple as instantiating allow list object
and checking whether the current device is supported by the GPU delegate.
This will tell you that the GPU delicate is safe to use
and has a baseline of performance.
What it doesn’t tell you is whether you actually get better performance
for your model with GPU on that device.
So to improve on this, we’re going to have a new feature this year.
It’s called Automatic Acceleration.
The Automatic Acceleration takes your model into account.
By providing additional metadata,
we can check then whether the model works correctly
and whether it performs better on an accelerator than a CPU.
This automatic allow listing works not just for CPU and GPU
but also for other accelerators, as we will talk about in the next section.
So about those other accelerators, let’s talk about what’s new
with the Android Neural Networks API.
Newer phones come with more chips. Digital Signal Processors, DSPs,
and Neural Processing Units, NPUs, give extra ML compute.
It’s like getting two more CPU cores to run your ML workloads on.
This is especially useful if you’re already using the GPU for other stuff like graphics.
The promise of the Neural Networks API is that you have a single API
across different Android devices that gives you access to these chips.
And Android Neural Networks API
has been production quality since Android version 10.
So if that’s the good news, what’s the bad news?
The bad news is that NNAPI has been part of the Android operating system.
The Android operating system is released once per year,
and that's not a great fit forever.
ML researchers come up with new ML models, powering new cool ML features,
but those models might only run on the newest version of Android.
And since features vary across Android versions,
you as the developer need to build targeting
or different code paths or different models for different devices.
In many cases, this actually means that NNAPI has not been worth the effort.
So what are we going to do about that?
So this, in my mind, is the cool part.
From Android operating system version 12
we’ve actually taking NNAPI out of the operating system.
We'll provide updates to NNAPI through Google Play Services.
So you get the same NNAPI spec
on different Android versions, but even more importantly,
you'll also be updating the actual device drivers
so you’ll get the same driver on different Android versions and different OEMs.
As upgradable NNAPI rolls out,
instead of having to test on thousands of configurations
you'll be able to actually test your ML on a handful of configurations.
And we are really excited to have Qualcomm as a launch partner
for shipping updateable NNAPI later this year.
So what do we get if we put all of these together?
You’ll get a pre-installed TensorFlow Lite runtime
so that you can use your APK size for the features your users want.
You'll get automatic acceleration.
We'll help you manage the complexity of the Android ecosystem.
And you’ll get one version of NNAPI and NNAPI drivers
across multiple Android versions.
You can sign up for the early access program on g.co/androidml-eap.
Thank you, thank you both from me as well as from Oli.
You’ll see some links on the screen.
Those will let you find not just the updated Android ML offering
but also Google’s overall on-device ML offering.
Those same links will also be in video description.
Thank you for watching.