[MUSIC PLAYING]
MARTIN WICKE: So Dan mentioned that things
will be tweeted out.
If you want to tweet something, use TFDevSummit and people
can see it.
A quick shout-out to all the people
who've joined us in viewing parties all around the world,
literally.
It's awesome to have you all here.
And of course, with the live stream,
that's not limited to physical presence.
OK, I think I'm all set up.
Thanks very much.
I'm Martin Wicke.
I'm going to talk about high-level APIs,
and so really--
why would you do that?
TensorFlow is perfect.
Well, so shortly after we released TensorFlow, really,
it got a lot of nice reviews.
And then some of the things people
said, well, it's really low level.
And this is my favorite quote about this.
The first thing you realize about TensorFlow--
it's a really low-level library, meaning you'll
be multiplying matrices and vectors.
Now, there's nothing wrong with multiplying
matrices and vectors.
In fact, it is one of TensorFlow's great strengths
that you can do this at a very low level.
It allows us to be very flexible.
You can easily combine TensorFlow operators to create
any algorithm you can think of.
Because they're at a very low level,
you don't have to fit them together awkwardly.
It's very clear what a matmul does.
It will always be a matmul.
It's used in many algorithms, so you can just
use it to do whatever you want.
It's also extensible, so you can recombine
these small little packets of code in arbitrary ways.
And then you can create a new algorithm
without having to change core TensorFlow, which is great.
And so a lot of, actually, our Python API
is built that way, where we have taken the op kernels or the ops
and combined them simply in Python
to create new functionality.
And as you will see, we'll use those extensively
to make high-level APIs.
It's also maintainable.
So have some history with this at Google, obviously.
And other people have probably experienced this as well.
If you have large, fused operations,
they accumulate features.
And as they accumulate features, they accumulate crud,
and things go out of fashion, and you end up
with these large, unwieldy pieces.
And so we avoid that completely by having
these small operations that are really timeless.
Nothing much was going to change to a matmul.
It's always going to be a matmul, so that's great.
However, of course, this also has disadvantages.
And it has disadvantages for users.
And one of them is that we lack higher-level primitives
in core TensorFlow if we only stay at this very low level.
And we may have all the tools you need to build your model,
but you may be working at a level of abstraction that's
actually not suitable for the kind of thinking
that you're doing.
And then also, we cannot really publish out-of-the-box
algorithms this way.
And this is why we are building a high-level API
for TensorFlow.
So starting in 1.0, we're creating this high-level API.
We're building this API by taking
components that have been built over time by other people.
Now, the idea of building high-level APIs
on top of TensorFlow is not particularly novel.
So lots of people have done it.
Much of that has happened in contrib,
in our contrib part of TensorFlow.
So there's been tf.contrib.learn,
tf.contrib.slim, SKFlow, which is
integrated in tf.contrib.learn.
And we're taking the pieces of those
and making a fully supported, high-level API
inside of core TensorFlow.
Now, what that should give you is
you will be able to iterate faster,
over model architectures or in general, things
that you may want to build with TensorFlow.
Those higher-level abstractions will
allow us to encode best practices for you,
so it's harder for you to make mistakes.
And we'll be able to provide primitives
that will make it really easy for you to write code that's
scalable right out the door, and you
don't have to re-engineer it once you realize,
oh, a single machine is not enough
or a single GPU is not enough.
And then finally, we'll prepare all
of this code to make it really easy for you to deploy.
So at the level of abstraction that we'll be talking about,
you'll be able to say, OK, export this to serving.
And you'll hear a whole talk about how
to serve models later.
Export this to serving, and it will just work.
That is just a promise.
So how do we do this?
So you've seen this before.
Rajat's talked about it.
This is a very popular illustration,
and what we're doing is, on the bottom,
you have TensorFlow that you know about-- the op
kernels, the TensorFlow engine, and then the front
ends that you're typically dealing with.
And all of the things that I'm going to be talking about today
are built on the Python front end.
So the first step is to build a higher level of abstraction
for building models, and we call this layers.
It is modeled after neural network layers,
but it's not limited to neural networks, necessarily--
and basically, larger pieces of code
that are grouped together in sensible ways.
And on top of that, we offer Estimator,
which is a class that streamlines all of the things
that you have to do in order to actually train and evaluate
a model, which is most of the work.
The actual model architecture is usually not
all that complicated.
And for simple cases, you're also
going to be able to use the Keras model.
And again, there's a talk directly
after this one about Keras.
And if you're interested in Keras,
if you're a user of Keras, you should definitely
wait for that.
There's going to be a lot more detail.
So for simple cases, you can use a Keras model.
The estimator is a little more production-oriented or for more
complex cases.
And then finally, we have now this estimator interface,
which is nice.
And we can offer you models that are completely built out
of the box, ready to go.
And there's a number of talks in the afternoon that actually
will talk about what kind of models are offered
and how they work.
OK, so let me start by introducing layers.
So you've seen this before, as well, in Dandelion's talk.
This is a convolutional network that is used for--
well, this particular one, I think, is used on [INAUDIBLE].
This is written down in a way that you may
see it written down in a paper.
This could be an illustration in a paper,
except for the color scheme.
And our goal would be that you can write this network down
in code, without switching levels of abstraction here.
So I want each of these layers in the neural network
to be represented with a single function in my code.
So if I have a convolutional layer,
I want there to be a convolutional layer function
that does the same thing, conceptually,
and that I can just use and replace
my illustration with some code.
So if I have a convolution layer,
a convolution function, if I have a pooling layer,
I make a pooling function, and so on, and so on, and so on.
I've added dropout.
That's the only difference to the one
that you've seen earlier, I think,
because dropout is better.
OK, so now, if you write your models this way, what happens
is that you avoid a whole lot of cognitive load.
You don't have to worry so much about, oh, I
have to make variables.
I have to call this function, that function, reshape.
All of this is done for you, and there's a lot of things
that you probably wouldn't have done
if you wrote it from scratch that are really useful to have.
So again, you will be able to iterate much faster
using this type of working.
And then we introduce, or we encode these best practices
for you.
And a big one is scoping.
So you've seen that the graph visualizer actually uses scopes
to group nodes in the graph together,
so it's easier to get an overview over big networks.
All of that is done for you if you use layers,
so each layer will show up as a little box.
And only if you want to inspect the actual going
on inside, which you probably don't, you have to move in.
All the layers support variable batch size.
So once you move it to serving, you
don't have to re-engineer your model because suddenly you
don't have a batch size of 32.
You have a batch size of 1.
All these kind of things are things that are easy to forget,
but we do them for you, so it's OK.
I want to mention that all of these layers
are compatible with Keras.
So if you use one of these functional layers,
that is in fact equivalent.
In fact, the implementation is shared to saying,
use a dense layer object that is in Keras style,
and apply it to the same inputs.
And you'll hear about this in the next talk,
but soon we'll be able to-- we'll
have a Keras module that is just completely
compatible with the Keras API.
And Francois will talk about this in much more detail,
so I'll leave it alone.
OK, so this now makes it really easy to build models.
But models are really only a small part of the total effort.
And the real effort is getting them to train,
and they maybe getting them to train in a distributed manner.
And that's what Estimator is for.
Estimator is an interface to abstract away
what the actual implementation looks like.
So to start, let's consider a generic model.
Any model, any machine learning model,
will look somewhat like this.
You'll have some inputs, possibly labels
if it's supervised learning.
There's some sort of model, and it produces predictions.
Now, if that is trainable, then this
looks a little bit different, where instead of predictions,
you can also produce some sort of training operations.
Often, this training operation is built on the predictions,
but it's not necessarily so.
And in TensorFlow land, what that
looks like is that you have a train
op that you call in the loop.
And each time you call it in loop, probably,
it's going to feed a mini batch or absorb a mini batch
and then update the weights of the model somehow.
And we're not going to take a position on what that op looks
like exactly, but it exists, and it's
going to be run in the loop.
Of course, if you're doing machine learning,
you also want to evaluate.
And while for small or for simple cases,
you can use the predictions to evaluate and just
feed some test data into it, in more general cases
it's very useful to have a separate evaluation graph
or evaluation op that produces your evaluation metrics.
And again, because your evaluation data set may not
fit in memory, you want to run this in a loop
and feed it mini batches.
So basically, what we're saying here
for Estimator is that every model looks like this.
And so we draw a box around it, and we call it Estimator.
And Estimator is really going to be configured using this Model
function, which effectively contains the code that you have
written anyway, which produces this training op, this eval op,
these predictions.
But it's just a single function.
You pass it to Estimator, and then
what you get in return is an interface that always stays
the same, and that you can write downstream infrastructure
against, and that takes away some of the work that you would
otherwise have to do manually and every time you
have the model.
So in particular, it encapsulates
all the information and all the necessary code
about sessions and graphs, and it has the loops inside of it
so that you don't have to write them.
Writing these loops properly is error prone,
and by using an estimator, you don't have to worry about this.
You worry about what you actually care about, which
is the details of your model.
So there's one more thing that we added,
which makes sense only in the context of TensorFlow, which
is that you can export your model to a saved model, which
is a data format that is directly usable in TensorFlow
Serving, which you'll hear about later.
OK, so this, again, it encodes best practices.
It deals with end-of-input exceptions.
It deals with workers going down in disputed setting.
It produces the right summaries at the right time,
all these things.
You can easily deploy it to serving
by simply calling the Export-Save model
and using that save model on the model server.
And it's scalable by design.
So what you get for free here is, if you have a model
that you can distribute using data parallelism,
this will simply work, and you won't have to worry about it.
So now I will tell you about some models that we have built.
And I won't tell you about the details.
You'll hear more about that in the afternoon.
And I won't bore you with more diagrams,
but let's just see some code.
So here is a more or less complete
piece of code that lets you write a linear model.
It's a very simple model, linear model.
And you see the class name here is Linear Regressor, so that is
one of our canned estimators.
It is an estimator in the is-instance sense,
but it has a slightly different interface.
Or it has the same interface, but it has a slightly different
constructor.
The only thing I have to tell this estimator is,
what imports do you want or do I want you to work on?
And for this, we have made this declarative language
to specify input data, which is Feature columns.
And here in this example, I just say,
well, this model expects a thing called Area
is going to be a real number.
And if you were to parse it out of an input
stream of some sort, let's say a TF example, the field
it has in there is square foot.
Then there is another real number, and it comes--
the name is numrooms.
And then there is an integerize feature,
which is really a string that I treat as an integer.
And I know that it's never going to be higher than 100,000.
That's a zip code.
Now, the feature integerization is a little bit--
it's not entirely straightforward
if you want to do it yourself.
So you can think of this as a one-hot vector that comes in,
although it's not implemented that way because that
would be terribly inefficient.
But again, you don't really have to worry about that.
So now, all I need to do is--
I already have an estimator here,
so I just have to call [INAUDIBLE] and predict on it.
The interface always stays the same,
which is nice because if I want to experiment
with different ones, I can simply
say, oh, let me swap that out.
Instead of a linear model, I want a DNN model, so
a Dense Neural Network.
And then all I have to do is change the class name.
And this is a more complex model, so it has,
actually, some hyper-parameters.
And in this case, I can say, oh, how many
units or how many layers do I want in this model?
And how many units should they have?
And otherwise, it really stays the same.
The only difference here-- and it's
worth pointing out that dense neural networks really
don't like getting integerized features directly
or large one-hot features directly.
So instead, I'm going to embed this.
And again, using this nice declarative language
for input specification, I can just
say, oh, make an embedding column out
of this integerized feature, and then we're done.
And this embeds it into eight dimensions,
so it's very straightforward.
Now, all I have to do is make sure
that the input functions actually
produce features of that name, and I'm good to go.
So really, the only question that remains
is, when can I have it?
So right now with 1.0, which I think
should be released as of right now or something--
I skipped the last 20 minutes of releasing this.
You have layers in core.
And the distinction between core and contrib
is really-- in core, things don't change.
Things are backwards compatible until release 2.0,
and nobody's thinking about that right now.
So if you have something in core, it's stable.
You should use it.
If you have something in contrib, the API may change.
And depending on your needs, you may or may not want to use it.
So layers are in core.
They're stable.
Estimators and canned estimators right now are still in contrib.
We anticipate that we move Estimator--
the base Estimator, the custom Estimator-- into core at 1.1.
Canned estimators will still be in contrib,
and they will move to core subsequently.
So in 1.2, we expect to have some of them in,
and then there will be more, and we'll keep building more.
And you'll hear about it in the afternoon,
how many we have already.
Traditionally, these roadmap-- these releases
have happened six to eight weeks apart.
So that's kind of the timeline you're looking at.
All right, and with that, I am done.
And the next talk will be by Francois, author of Keras,
about Keras.
[MUSIC PLAYING]