[MUSIC PLAYING]
THOMAS EZAN: Hello, everyone.
I'm Thomas Ezan, and I'm an Android Developer Relations
engineer working on machine learning.
So with the rising processing power in mobile devices,
it's now possible to run machine learning features and process
sound, text, images, or even videos directly on the device,
but providing low latency for live experiences can be hard.
And so today, I'm going to talk to you about how
hardware acceleration can help.
So we will start by reviewing the chips that
are available on mobile devices that can
enable hardware acceleration.
And then, we'll look at examples, and one
in particular, which is the pose detection algorithm.
And finally, we will see how we can use hardware acceleration
for machine learning features in your own application.
So let's review the hardware that
is available on mobile devices.
So it might actually feel strange for mobile developers
to consider the hardware your application is going to run on.
Usually, when you create a traditional app,
you rarely think about the chip that will execute your code.
Most of the processing power happens
on the CPU, which is versatile and available
on any mobile devices.
But with machine learning apps, going beyond the CPU
and accelerating your machine learning
features on specialized hardware can actually
make a huge difference.
For instance, by using specialized hardware,
you can divide your inference latency by 10
and enable live experiences like the pulse detection
or augmented reality filters.
And you can also offload the CPU workload to other chips.
And finally, you can reduce the battery impacts
as some hardware consumes, usually,
less power than the CPU--
like the DSP, for instance.
So we can group those specialized hardware
in three categories.
There are GPUs, graphical processing units, DSPs,
digital signal processors, and NPUs, neural processing units.
So GPUs are often used for graphical rendering processes,
but they offer high throughput and massive prioritization,
which is very handy for machine learning processes,
especially if you want to enable real-time experiences.
And they are also capable of handling floating point
calculation, so you can reach good performance
without having to quantize your model.
And actually, the GPU is often the most popular hardware
for machine learning acceleration
for Android developers.
Then there are DSPs.
They are mainly used for analog-to-digital and
digital-to-analog conversion of voice and sounds.
They are designed to handle math-heavy processes,
and they are also optimized for energy consumption--
for low energy consumption.
And finally, there are now chips that are designed and optimized
for executing machine learning, such as NPUs and TPUs.
And so after revolutionizing machine learning and inference
in the cloud in the past few years,
those chips are now becoming available on mobile devices.
So currently, they are mostly available
on higher-end devices, but they are
becoming more and more popular.
And for example, the Google Pixel 6 and 7 series
embed the Google Tensor, which enabled the acceleration
of some key Pixel features such as mobile eraser
and on-device translation, which are available on other devices
but are optimized to take advantage of Google Tensor
chips.
So if we can summarize, CPU isn't always
the only processing unit available on a mobile device,
and depending on your model characteristics,
you can use a GPU or a DSP or, if the device has one,
use NPU/TPU chips to offload the computing load from your CPU
and accelerate inference.
So we reviewed the different specialized chips available.
Let's now zoom in and see how acceleration can be
helpful in a concrete example.
So if you want to run machine learning inference
on Android today, there are several options available.
So if you have some machine learning expertise
and you need advanced control over the user experience
or the machine learning process execution,
you can use TensorFlow Lite.
On the other hand, if you don't have machine learning
experience and you want an easy-to-use solution,
you can take a look at the ML Kit SDK.
ML Kit is actually built on top of TensorFlow Lite,
but it offers machine learning enabled
APIs that are optimized for mobile and production-ready.
So if you decide to build your own custom ML solution,
TensorFlow Lite is actually available via Google Play
services, so you don't need to bundle it with your app
and it is actually guaranteed to run the latest stable version
of TensorFlow Lite.
And besides, TensorFlow Lite in Google Play services
is now the official machine learning inference engine
on Android.
So using it in your application will
let you benefit of future APIs and Services
from Android's custom ML stack as they will be built
on top of the inference engine.
And it's already used by thousands
of apps and more than a billion users monthly.
So a few years ago, Google released
BlazePose, which is a model following
a new approach to human body pose perception,
and it works in real time in videos or static images.
And the model returns positions of body points such as knees,
shoulders, hands.
So depending on your use case and the level of expertise,
if you want to use BlazePose, today you can either download
the BlazePose Lite model and use it with TensorFlow Lite,
or you can directly use the ML Kit pose detection API
and use it--
use the API, basically, as a ready-to-use solution.
And if we actually take some time
to look at the performance of the ML Kit Pose algorithm
and look at the benefits of hardware acceleration,
we can see that when the model is executed on GPU,
we are basically seeing the inference is
running 184% faster than when we run it on the CPU.
And if we run it on a GPU, then it's even faster,
and it's basically 200 faster than the CPU.
So that's one example of the benefits
of hardware acceleration.
But there are models that are even more
optimized to take advantage of GPU acceleration,
and for instance, if you use MobileNet,
which is an image classification model,
you can see up to 10x latency improvements.
So we discussed some of the benefits of enabling hardware
acceleration for inference.
If you are using ML Kit, you really
don't need to do anything.
ML Kit is already configured to get you the best performance
you can on the current device.
But if you want more control, let's
review the steps to add GPU acceleration
for TensorFlow Lite with Google Play services
to your application.
So the first step is to add the dependencies to your Gretel--
build the Gretel file in your application.
So make, actually, sure that you migrated to TensorFlow Lite
with Google Play services.
And you can read more about it in the TensorFlow Lite
documentation on tensorflow.org.
And then, actually, like for TensorFlow Lite,
the GPU support can also be provided
via Google Play services.
And the benefits are the same.
It reduces your app size, and it also
enables you to run on the latest stable version of the GPU
supports.
And you cannot hear that we are using a package name that
starts with com.google.android.gms,,
which is for the TensorFlow Lite GPU dependencies.
And that indicates that the GPU support implementation
is provided via Google Play services.
Then, in your Kotlin code, you will
initialize the TensorFlow Lite instance with the Google Play
services Task.
And you will then call setEnableGpuDele
gateSupport(true) on your TensorFlow Lite--
on your TfLite initialization options
to enable the GPU delegates.
If the initialization is successful, in the callback,
you will call the DelegateFactory,
the addDelegateFactory, on the interpreter option.
And you will pass an instance of GpuDelegateFactory.
So the code I walk you through lets
you create a TensorFlow instance using the GpuDelegate
and enable hardware acceleration.
But in addition of just enabling it in your code,
you can also adapt your model to leverage the GPU.
So for instance, one thing you can do
is actually consider using a model that
is optimized and built to take advantage of GPU acceleration
and adapted to your specific use case.
So for example, I was just talking about model nets
that you can use for image classification.
You can also look at DeepLab for image segmentation,
or you can look at PoseNet for pose estimation.
And it's actually an alternative to BlazePose that I just
talked about before.
So it's optimized for GPU acceleration.
Keep in mind that it detects less key points
than BlazePose does.
And then, the second step you can take
is to make sure that you avoid some specific operations.
For instance, the reshape operations,
such as BATCH_TO_SPACE, for instance, you probably
want to avoid using them on the GPU.
They are quick on the CPU, but they are particularly
expensive to run on GPU.
And also, another thing to keep in mind, GPU
are designed to process images and textures
using RGBA, which is coded on four channels--
the red, green, blue and alpha channel.
And so if your model is actually using four-dimension sensors
as input, it's going to be significantly faster
to run it on GPU.
And you can learn more about how to optimize your model
for the TensorFlow Lite in the TensorFlow Lite documentation.
And for even more insights, you can actually
take a look at the TensorFlow.
Model Analyzer command line tool,
which enables you to evaluate the GPU
compatibility of your model.
And so for instance, you can see an example here of the command
line, and you need to pass gpu_compatibility equals
to true to have GPU compatibility-specific
insights.
You can learn more about it in the TensorFlow documentation,
and there is even a collab notebook
that you can use to get familiar with it.
And I talk about how to enable the GPU delegates, which
is a great way to enable acceleration.
But keep in mind that the GPU is not the only way for you
to enable hardware acceleration, and you
can take a look at the neural network
API delegates to enable inference on DSP or NPU.
Same thing-- that's also something
that is documented, very well documented,
in TensorFlow documentation.
So to fully benefit from hardware acceleration,
you need to make sure that your model is using operations that
are supported by the delegates, and you
can review the list of operations supported
for each delegate, again, in the TensorFlow Lite documentation.
And also, remember that the acceleration delegates
don't support yet the custom operations
that you created yourself.
And actually, if one of those operations in your model
are not supported by the delegate,
you will get the following error at runtime.
And so what's going to happen is that this specific operation
is going to be executed on the CPU instead.
So if your model is actually splitting
the execution between the CPU and the GPU,
the back-and-forth is likely going to significantly increase
the execution time.
So for some cases, it might be better
to either fully run it on the CPU
or to re-engineer your model to make sure
that it only uses supported operations.
And so as you can see, hardware acceleration is very much
the way to increase execution performance,
but it can be a complex task, and finding
the optimal processing unit in your model
sometimes can be difficult. So that's
why we are currently building an acceleration
service for Android.
And so the way it's going to work
is that you will be able to pass your model, your sample,
and some sample inputs, and also the expected results,
and the acceleration service will then run a benchmark
to evaluate the optimal hardware available on device
and tell you, basically, which of the CPU, DSP,
or NPU, if it's available, you should use to run inference.
And so this is something that is currently in early access.
So we are accepting applications,
and we aim to ship it publicly early next year.
And you can go to the ML Android Developer documentation
to learn more about it.
So we covered a lot.
And you can see that you can already
use hardware acceleration in your application today,
but we are actively working on making the process even easier
in the future.
So ML and in general and ML in Android is a field that is
constantly evolving. .
So if you want to continue to learn more about it and stay up
to date, visit d.android.com/ml.
Thank you.
[MUSIC PLAYING]