[MUSIC PLAYING]
FRANCOIS CHOLLET: Hi, everyone.
I'm Francois, and I'll be telling you
about the integration of the Keras API into TensorFlow.
So first of all, what is Keras?
If you know about TensorFlow, you probably
also know about Keras, but just in case,
for those who don't know.
Keras is an API that makes building deep-learning models
easier and faster.
So it's a deep-learning toolbox, and it's all about ease of use,
reducing complexity, reducing cognitive load.
And if you make deep learning easier to use, what happens
is that you're also making it accessible to more people.
So the key idea with Keras is to put deep learning
into the hands of everyone.
We believe that, in the future, deep learning
will be part of every developer's toolbox.
It will not just be a tool for experts and researchers.
It will be into everyone's hands.
And this will happen with platforms like TensorFlow
and APIs like Keras.
So Keras is better understood not as a library
or as a code base.
It's more of an API spec.
And it's an API spec that has several different
implementations.
There's the Theano implementations,
which was the original Keras [AUDIO OUT] released
two years ago.
There's a TensorFlow implementation, of course.
And then there will be more implementations in the future.
For instance, Microsoft is working on a [INAUDIBLE]
implementation of Keras.
And so until now, the TensorFlow implementation
separates [INAUDIBLE] with lots of uses.
And what we are doing now is that we are bringing the Keras
API directly into the Keras project,
into the TensorFlow project.
We believe that's a big step towards making
TensorFlow and deep learning accessible to everyone.
So what's concretely happening is
that we are bringing into TensorFlow this Keras
compatibility module, tf.keras, which
is an implementation of the Keras spec
that's built from the ground up for TensorFlow.
And so as part of this move, we're
introducing into core TensorFlow new data structures
like layers, which Martin just introduced.
And also models, which are containers
for graphs of layers.
So these used to be Keras data structures,
and now they will be shared between core TensorFlow
and this tf.keras module.
So, because this Keras implementation
is built from the ground up on top
of the shell of the structures, it
is fully compatible with all of the TensorFlow functionality.
That means it's also compatible with [AUDIO OUT] features that
has the experiment API that Martin introduced.
So if you're a Keras user, here's what this means for you.
First, it gives you access to more flexibility.
You become able to very easily mix and match
pure TensorFlow functionality with Keras functionality.
So you can build more powerful, more flexible models.
And what's more, you gain access to really advanced,
really powerful features through the [INAUDIBLE]
TensorFlow training API that Martin mentioned.
You gain access to very quick and easy distributed training
for Keras models.
You gain access to training on Cloud ML, distributed
hyperparameter training, which is really, really powerful,
and you can very easily deploy your Keras modules
to production with TensorFlow serving.
If you're a TensorFlow user, what
does this change mean to you?
Basically, it gives you access to the full scope of the Keras
API to make your life easier without leaving
your existing TensorFlow workflow.
So you can just start using the Keras API
at no loss of flexibility.
You don't have to adopt all of Keras,
you can just adopt the layers you need, for instance.
And also, it gives you access to any existing piece
of open source Keras code.
You can just go on GitHub, copy some piece of Keras code,
drop it into your codebase, and, by just changing imports,
it will just work.
So this gives you access to a lot of existing open source
code that you can start leveraging into your TensorFlow
workflow.
So to make things a bit more concrete,
I will walk you through an example
of what your workflow will look like when using Keras
with TensorFlow.
And we'll be looking at a video question answering model,
and we'll be using Keras to define a model
to solve this problem, and we will
retrain it using the high-level TensorFlow training
API in a distributed setting.
So this is a problem that we'll be looking at.
We have some videos, which are sampled about four frames
per second, roughly 10 seconds per video,
so we have about 40 frames per video.
And we are asking questions about the video,
about what's going on in the video, and about the contents.
Here we have this video of a woman packing objects
into a car.
You can ask, what is the woman doing?
What's the color of her shirt?
And the answer should be a single word.
So we'll build a deep-learning model
that it will take as inputs the video as a sequence of frames,
so about 40 frames in order, and the question,
and the model should output one word, the answer.
And so if you ask this question, what is the woman doing?
She's packing.
It's actually interesting because, if you only
take one frame and try to train a [INAUDIBLE],
for instance, to answer the question,
the problem is that just from one frame,
she could also be unpacking as well, right?
And the reason you know she's actually packing
is because of the order of the frames.
If you look at the second frame, the rug is outside the car.
In the next frame, it's inside the car.
And so we expect all the planning model
to be able to leverage this order to correctly answer
this type of question.
So needless to say, this is a tremendously difficult problem.
And just a few years ago, three or four years
ago before TensorFlow, before Keras,
this would only have been accessible to a handful
of large companies or in very well-funded research labs,
especially if you wanted to make your model
train on a cluster of GPUs.
And now with TensorFlow as a platform
and with Keras as an API, we're making
these three difficult problems accessible to anyone
with just basic [INAUDIBLE] scripting capabilities.
So we are putting the full power of deep learning
into the hands of everyone.
So here's a model that you would use to solve this problem.
At the high level, it has two branches.
There's one branch that's meant to encode the video inputs
into a single vector and one branch meant
to encode the question, so a sequence of words,
into a single vector.
So we have one vector, including information
about the entire video, one vector,
including information about the entire question.
And then we concatenate these two vectors.
And this gives us one single vector
that encodes information about the entire problem,
about the combination of the video and the question.
And that's kind of the beauty and the magic of deep learning
is that you take perceptual inputs, like a video,
for instance, you take semantic meaning,
and you're embedding this perceptual input [AUDIO OUT]
And you're embedding this meaning into geometric space.
You're turning meaning into vectors
and then you can learn interesting transformations
of these geometric spaces.
So that's really the beauty of deep learning.
And so in our case, we'll be taking this vector
and coding the entire problem.
And we will be training a fully connected [INAUDIBLE] on top.
And it will end up with a softmax
over a predefined vocabulary, and so the word
with the highest probability in this vocabulary
will be the answer to the question.
So now if you want to look at a higher level of detail,
how do we turn a video into a single vector?
How do we turn a question into a vector?
So for the video, we're starting out with the video
as a sequence of frames.
So each frame is just an image.
So it's RGB, it has a height, a width, a depth, color depths.
The other sequence of frame, you use
a [INAUDIBLE] to transform each frame into a single vector.
So essentially you're using the conditional base
of a pre-trained network, and you
do some pulling to extract one vector that
encodes the visual contents of the frame.
And so what you get out of this is the video
anchored as a sequence of vectors,
where each vector encodes the visual contents of one frame.
And then you take the sequence and you run it through an LSTM.
So you probably know what an LSTM is,
but basically it's a [INAUDIBLE] type of network
that can process sequences and takes into account order.
And this LSTM will output a single vector encoding
information about the entire video.
To turn a question into a vector,
we are following a very similar process.
We represent a question as a sequence
of integers, each integer standing for a word.
And then we map each word to a word vector
via a process called embedding.
So we get out of this a sequence of word expressed
as a sequence of vectors.
We run it through a different LSTM
layer, which will encode the entire question
as a single vector.
So in Keras, here's the architecture
that we would use to implement this model,
and it's literally just a very simple translation
of the previous graph.
So for the video encoder, basically, we
are starting from the video as a 5D tensor.
The first axis will be the batch axis.
Then you have the time axis.
Then you have a 3D tensor, which encodes a frame.
And so we will apply an Inception v3 network,
pre-trained on ImageNet to every frame of the 5D tensor
to extract one vector, one feature vector, per frame.
And so out of this, we get a sequence
of visual vectors, which will run through an LSTM layer.
And for the question embedding part,
we are simply using an embedding layer to map our question.
So a sequence of integers into a sequence of vectors,
and we run that through an LSTM layer.
Finally, we use a concat operation
to bring together these two vectors,
and we stack a few dense layers.
And we end up with a softmax over a predefined vocabulary.
And you are training that with a target answer word encoded
as a one-hot vector.
So what does implementation look like?
It's very simple.
What you're seeing here is the video encoder
in just five lines.
During the first line, you're just
specifying the shape of your video input.
So it's a 5D tensor.
With a shape argument, you do not actually
specify the batch size.
So the first axis, set to none, is a time axis.
So it's set to none because we want it to be viable.
We want the small one to be able to process
videos of any number of frames.
And then you have the shape of a single frame.
So a single frame is 150 by 150 RGB image,
so it's a tensor that's 150, 150 by 3.
In the next line, what we are doing
in a single line is instantiating
an Inception v3 network that will automatically
load pre-trained weights, so train on ImageNet.
And we'll configure this network to be doing visual feature
extraction.
So essentially we'll not be including
the classifier part of Inception v3,
only in the convolutional base.
And then we'll be applying some average pulling
on top of the bottleneck layer.
You just extract one single vector, bare image input.
Then this next line, we are setting [INAUDIBLE]
to be non-trainable.
So what this means is that, during training,
we will not be updating the weights.
And why?
Simply because it's a pre-trained model.
And if you were to update its weights while training
on this new problem of question answering,
we will likely be wrecking the representations
that this model has already learned on ImageNet.
So you don't want to do that.
You just freeze it.
And so the next step is to use a time-distributed layer
to essentially take this [INAUDIBLE]
and apply it to every step of the time
axis of the video input.
So this time-distributed layer just takes this [INAUDIBLE]
and apply it to every frame.
And what comes out of this layer is simply
3D tensor, which is a sequence of visual frames,
a sequence of visual vectors.
So each vector is standing for one frame.
And finally, we run this sequence tensor
through an LSTM layer, and this gives us one single vector
encoding a [INAUDIBLE] video.
And as you can notice, when we are instantiating this Keras
LSTM layer, we only specify one parameter,
which is the number of units in the LSTM layer.
And if you think about it, that's quite remarkable,
because there are so many advanced configuration
parameters in this LSTM layer.
There are a lot of these practices you have
to follow when using LSTMs.
For instance, you have to remember
that your record on weights should be initialized using
[INAUDIBLE] initialization.
And you have to remember that the bias of the [INAUDIBLE]
should be initialized to 1.
These type of best practices that you really
need to follow if you want training to go smoothly.
And one key principle of Keras is
that best practices are included,
so every Keras layer has well-optimized default
configuration that takes into account all these best
practices.
So you can't just rely on Keras defaults to be a good defaults
and to just get your model to train out of the box
without requiring you to fine-tune every
of these hyperparameters.
So best practice is included, and you should just
care about only specifying a few key parameters,
like the number of units, and everything else should just
work right out of the box.
So for the question encoder, it's even simpler.
We just have three lines.
First, we specify the input tensor for the question.
So every question would just be a sequence
of at most 100 integers.
So we will be limited to a question at 100 words long.
And then we embed every integer into word vector.
We have this embedding layer.
Note that we use masking on this embedding layer, which
means that, for instance, if we do
[INAUDIBLE] on our questions, which
means that, if the questions are shorter than 100 words,
we will just pad the rest with zeros to get to 100.
This creates a mask, which will be
propagated to the [INAUDIBLE] layer right after that.
And so we finally encode the sequence of four vectors
into a single vector via another LSTM layer.
So note that, when we're using these pre-trained inception
network, it's absolutely fundamental
to be loading these pre-trained weights
because you're dealing with a fairly small data set.
And this set alone would not have enough data
to allow you to learn to extract interesting visual features.
So you really need to make this network actually train.
You really need to be leveraging these pre-trained weights.
And so finally this is how you end up with the answer word.
You are taking the video vector and the question vector.
You are concatenating them with just a concat operation.
And finally, you are applying a couple of dense layers,
and you're ending up with 1,000 units.
So we'll have a vocabulary of possible answers
that is just 1,000 different words.
And here's a step where you're specifying the training
configuration of your network.
So you're just instantiating a model, which
is a container for a graph of layers,
and you're instantiating them by just
specifying what are the inputs of the model, what
are the outputs.
You can add any number of inputs and any number of outputs.
And you are telling the model that it
should use the ad optimizer during training
and this last [INAUDIBLE].
You can notice that, when specifying our classification
layer with 1,000 units, we did not specify any activation.
So it's actually a purely linear layer.
The softmax activation will be included with the loss.
And so to sum up this entire code, it's just about 15 lines.
So it's very, very short.
So we are essentially turning this very complex architecture,
including loading pre-trained weights,
into just a handful of lines.
So really reducing complexity and by allowing
you to rely on different configurations for every layer,
we're already reducing cognitive load.
You don't have to care about most
of the parameters of your LSTM layer, for instance.
And the best part of it, the most magic part,
is that, because this implementation of Keras
is built from the ground up for TensorFlow,
it's fully compatible with things like estimators
and experiments.
So in just one line, you can instantiate a TensorFlow
experiment.
And this gives you access to distributed training
to train on Cloud ML and so on, and also access to TensorFlow
serving.
So in just a few lines, you can start running your experiment
as you did with your question answering model,
reading your viewer data and question data and answer data
from [INAUDIBLE] data frame.
You can start running it on [INAUDIBLE]
in just a few lines.
So what you should take away from this talk is very simple.
We are taking the Keras API.
We are bringing it into TensorFlow.
It's a new implementation of Keras built from the ground up
for TensorFlow.
And this gives you, as a Keras user, more flexibility.
You can start mixing and matching
pure TensorFlow functionality with your Keras model.
As a TensorFlow user, this gives you access to the full Keras
API to make your life easier.
And so you can expect these Keras compatibility modules
to exist as tf.contrib.keras by TF 1.1.
And finally, tf.keras in TensorFlow 1.2.
And so we believe this is a big step
towards making TensorFlow and deep learning
accessible to everyone.
So that's it for me.
I believe, next, we have some snacks.
Thank you for listening.
[MUSIC PLAYING]