[MUSIC PLAYING]
ANITHA VIJAYKUMAR: TensorFlow Lite
is a lightweight machine learning
library for mobile and embedded devices.
DAN GALPIN: Hi, I'm Dan Galpin, and I'm here
with Anitha Vijaykumar.
She's a technical program manager at TensorFlow Lite.
Your background is largely in infrastructure.
How did you make the jump from infrastructure
to machine learning with TensorFlow?
ANITHA VIJAYKUMAR: So I've been doing infrastructure
for a very long time, and one day I
heard a talk from Jeff Dean on how computers can see cats.
And so this was something really profound,
that we enabled machines to actually have vision and see
something.
And so I tried a few code labs in TensorFlow,
did a few programming and machine learning,
and became interested and moved in.
DAN GALPIN: So we already have TensorFlow Mobile.
What was the motivation for developing TensorFlow Lite?
ANITHA VIJAYKUMAR: This is a great question.
TensorFlow Light should be viewed as an evolution
of TensorFlow Mobile.
TensorFlow Lite supports a subset of operations today,
and going forward, the entire mobile support
should be on TensorFlow Lite.
DAN GALPIN: How did we decide which operations to support?
ANITHA VIJAYKUMAR: So we talked to a lot of partner teams,
the mobile vision team, and then within Google, we
talked to our Android partners, and we came up
with a list of models that we wanted to support.
And then we said, we want to have
a subset of operants in the vision space,
a few models on the text, and a demo, light model
on the speech.
So TensorFlow Lite supported flavors of these three models,
and going forward, we will extend that.
DAN GALPIN: So how did we approach
the problem of improving load times by comparison
to full TensorFlow?
ANITHA VIJAYKUMAR: So we use something
called flat buffers, which is again an open source Google
initiative.
They allow for really fast load times,
and they don't need unmarshalling,
and so using these techniques helps
us to ensure that we can have very low latency
and load the models faster.
DAN GALPIN: So TensorFlow Lite uses
a model that has been trained with TensorFlow.
Now, how does a TensorFlow model actually
use in TensorFlow Lite, then?
ANITHA VIJAYKUMAR: So you start off with a regular TensorFlow
model, generate a model just like you
would do in normal TensorFlow.
So have your GraphDef checkpoints.
And you go through an additional step of freezing the graph.
This is also standard TensorFlow operation, nothing
specific to Lite at this point.
But once you have a frozen graph,
once you have a frozen model, then you convert it,
you feed it into a TensorFlow Lite converter,
and once you've converted to a TensorFlow Lite converter,
you get what is called a .lite file.
With this .lite file, it can be fed into an interpreter,
and using the interpreter, you can run your model inference
on the device with TensorFlow Lite.
DAN GALPIN: What techniques can developers use
to make their models smaller?
ANITHA VIJAYKUMAR: We use quantization, mainly.
So quantization models are-- usually,
neural networks are resistant to noise,
and so we can get better computation
and use smaller memory if we use 8-bit quantization.
At the price of accuracy, paying a price for accuracy.
So what you could do is you could
train the model with fake quantization nodes,
like we did with MobileNet, and get better results when
you actually run your inference on the device.
When we did quantization and we measured
the results, what we found was that MobileNet is about 4x
smaller than its flawed and even speed it's about 50%
faster than the flowed version of the model.
So this was really great results that we saw with quantization.
DAN GALPIN: And so in terms of developing TensorFlow Lite,
what surprised you about the process?
ANITHA VIJAYKUMAR: So when we started the development
of TensorFlow Lite, we had an audacious goal
of hitting a very small memory size of being
at least 300k or lower.
And to date, just the base interpreter
is about 70 kilobytes, and with operators loaded
is under 300 kilobytes.
So we are at least 15 times smaller
than traditional TensorFlow, and so we
are really excited about it.
DAN GALPIN: We also announced the Neural Networks
API in Android.
Now, how does TensorFlow Lite support
this kind of acceleration?
ANITHA VIJAYKUMAR: So we announced
it a couple of days back, beta versions
of Android neural network API.
And TensorFlow Lite has hooks for Neural Network API.
Using the Neural Network API, you
can leverage custom hardware accelerators on your device.
So if you have a DSP, the application developer
doesn't have to worry about it and still write his TensorFlow
Lite program, and if there's an API and a specific hardware,
your instructions can be accelerated
using that hardware.
DAN GALPIN: So if developers want
to experiment with TensorFlow Lite, how can they get started?
ANITHA VIJAYKUMAR: We would love for developers
to start experimenting with TensorFlow Lite.
The tensorflow.org has a mobile web page,
and you can see it's all revamped.
And so there's a lot of data on TensorFlow Lite over there.
You can also download the source code from GitHub.
It's under contrib/lite currently.
And there's tons of documentation.
There's a model that you can download.
There's a demo application that you can download
and get started immediately.
DAN GALPIN: Great.
Well, thank you so much, Anitha, for talking to me.
Once again, this is Dan Galpin, reporting for the Developer
Show here from GDD India.
Thanks for watching the Developer Show.
Catch up on the TL;DR and additional interviews from GDD
India right over here.
[MUSIC PLAYING]