[MUSIC PLAYING]
ASHISH AGARWAL: Hi, I'm Ashish.
I'll be talking about ML Toolkit.
So let me start by saying that TensorFlow is this really
amazing piece of technology.
At Google, it's already powering systems
ranging from search ranking, ads auction, YouTube
recommendations, Translate, Photos, and many, many more.
We can train complicated models with hundreds
of billions of parameters and solve them
on platforms ranging from mobile phones to cluster of TPUs.
However, TensorFlow is this really low-level framework.
As Martin mentioned in an earlier talk,
we are working on high-level parameters like layers,
estimators, losses, and metrics to make
it easier for researchers and developers
to create custom model architectures.
However, what's still missing from TensorFlow
is algorithms that work out of the box.
What a lot of developers really want
is packaged solutions that they can quickly and easily
integrate into their workflows.
So I'm happy to tell you that now we
have a toolkit of really popular machine-learning algorithms
that we have built right on top of TensorFlow.
So starting with linear and logistic regression, k-means
and Gaussian mixture models for unsupervised clustering
of data, WALS matrix factorization,
which is a popular collaborative [INAUDIBLE] algorithm
for the computer systems, support vector machines, all
the way to state of the art algorithms
like SDCA, Stochastic Dual Coordinate Ascent,
for context optimization, random forest and decision trees,
and various architectures of deep networks.
Now, all these are available on GitHub,
and we are constantly working on improving and expanding
the site.
Next I'll highlight a few of these algorithms
to give you a better sense of the existing and upcoming
features for these algorithms.
Starting with k-means.
So here we implemented the standard layouts iterative
algorithm, along with random and k-means++ initialization.
We support both full batch and mini batch training modes,
and we also allow the user to specify distance functions
like cosine or Euclidean square distances.
Now GMMs, on the other hand, model the data
as a mixture of Gaussians.
These are much more powerful models, and also harder to
train.
For these, we use an iterative EM algorithm,
and we allow the user to choose from a combination of means,
covariances, mixture weights to train on.
Next, we have WALS, which is matrix factorization using
Weighted Alternating Least Squares.
I will be revisiting this later in my talk, as well.
So here, you are given a very sparse matrix.
For example, you could be given ratings
that users have provided on videos.
Note that the matrix is sparse, so not all the videos
are being rated by each user.
And now you want to answer questions
like, which video should you recommend next to the user,
or to find user-user similarity or video-video similarity.
Now, this is typically done by factorizing this huge matrix
into a product of two factors, two dense factors.
Note also that, as the algorithm indicates,
this loss is weighted, which allows you to, say, downgrade
things that were unrated in the original input, or to avoid
spam, or popular entries from drowning out the total loss.
So this value is highly nonconvex.
However, it turns out, that if you fix some of the factors,
there's a close analytical solution for the other factor.
And so the training works in an iterative way
where you, first, fix V, compute U,
then fix U, compute V, and go on.
Next, we have SVMs.
So these work by finding a decision boundary that
maximizes the margin.
We are working with soft margin methods using a hinge loss,
and the current implementation is of linear SVMs
with L1 and L2 regularization.
Note that SVMs become much more powerful with a nonlinear
kernel, which allows a much more complicated decision boundary.
So we are working on providing nonlinear kernels using
the kernel approximation trick.
Next, we have SDCA.
Now here, you have a convex loss function
with L1 and L2 regularization, and SDCA
uses a neat trick of transforming this last function
into a dual form.
And it turns out that, for many problems,
solving it in the dual form is much more efficient.
And it turns out that this algorithm
can power models from linear and logistic regression all
the way to SVMs.
We also have random forest and decision trees.
Random forest are [INAUDIBLE] decision trees,
and decision trees work by creating
a hierarchical partitioning of the feature space.
Currently our training method is what
we call "extremely random forest training," which
allows better parallelization and scaling,
and we are also working on having gradient
boosted decision trees, which are very, very popular also.
So I talked about the different algorithms.
Now we want to make it really, really easy for users
to use them.
All of these are exposed via very high level scikit-learn
inspired estimator APIs.
Here's an example for how when [INAUDIBLE] k-means clustering.
So you start by creating a k-means clustering object,
and here you can pass in a bunch of options,
like the number of clusters, how you want to train,
how you want to initialize, and so on.
Next, you call the fit function and pass it your input,
and that's it.
Not tons of logos behind the scenes.
It creates the graph for you.
It'll run training iterations, configure the runtime
until the training is done.
When you're finally ready, you can
start inspecting the model parameters like the clusters,
and start running inference, here finding the assignment
to clusters, and so on.
So I talked about high-level APIs,
but you also want to maintain the flexibility
and extensibility that TensorFlow promises.
So these are not opaque objects that are only
accessible via this API.
They, in fact, allow the users to inspect the graphs,
and also to be able to embed these graphs into larger
training models.
So here's a training-- here's a [? trial ?] example that
embeds k-means as a layer in a bigger, deeper network.
You start with an input, you run k-means,
and get your graph for k-means.
It also returns your training op to drive the clustering,
and the output is the input transformed to a distance
to cluster space.
Next, you feed this to your dense stack of layers,
you create your supervised loss, you
run STD to get a final DNN training op.
Now you have these two training ops,
you can join them to create a single training
op that can drive the core training of these two models.
Here's a code snippet for the same.
So you start by creating this k-means object,
and you'll inspect the training graph.
It returns you a training op and the output.
Next, you feed your output into dense layers,
and you create your model architecture as usual.
Finally, you get the training op to drive your supervised loss,
or your dense stack.
And finally, use TensorFlow's group operation to group
together these ops, so that you have a single op
to drive the co-train.
So just to recap the talk so far,
I talked to you about the different algorithms
that are available now.
I showed you examples for how to use high-level APIs to access
them, and also talked about flexibility and extensibility
of these algorithms.
Next, I want to highlight the fact
that all these algorithms are backed
by distributed implementations.
Just to give you a better sense of the complexity hidden
behind these few lines of byte and code,
I will dig deeper into the implementation
of one of the algorithms.
So, in general, the distribution architecture
is going to do the same.
You'll have a set of parameter servers, which
will store all the parameters in a sharded way.
You will have a bunch of worker replicas
that will be running training steps,
with many batches of inputs.
In each step, the worker replica would fetch some parameters.
It will run compute on the input to compute the new value
of the parameters.
And finally, it will write back these updates
to the parameter server.
So let's look at WALS again.
Just to recap, with WALS, you want
to factorize a really sparse matrix into dense factors.
And as I told you, the training algorithm
involves fixing one of the factors,
computing the other one in your training.
Now I want to be able to scale to inputs that are terabytes
in size, and you want factors that
have hundreds of millions of rows and hundreds of columns.
So how do you make this work?
So we have to carefully think about all the RAM, compute,
and info bottlenecks.
The first obvious thing is, given
the size of these factors, you will have to distribute them
across parameter server shards.
So you take a U, split them across all these shards.
Next thing, given the amount of compute involved,
you want a bunch of worker replicas.
The sharding scheme that we use for these worker replicas
is row slicing.
And what that means is, each mini batch
is a subset of rows of the input,
and we compute the corresponding rows of U.
Note, however, there's still a problem.
Each update to U depends on the full V factor.
And as I said, V could be large, it could be terabytes.
So what this entails is that you might
be copying terabytes of data to each worker in each step.
So how do we get around that?
So at this point, we use some [INAUDIBLE] magic.
It turns out, you can compute certain summaries,
let's call them G. And given G, your updates to U
only depend on a few rows of V. And these are specifically
the rows that correspond to the non-zero entries of the input.
So this is great.
What that means is that, instead of copying terabytes,
you can just copy megabytes of data per step.
At this point, we can run TensorFlow's gather operation
and get the required rows.
We then compute using highly optimized custom C++ kernels
that we wrote.
And finally, we use TensorFlow's scatter operation
to write back these updates to the parameter server shards.
So how well does this work?
We believe our implementations are really high performance.
We did some benchmarking, and on single machine,
our implementations, in general, look
on par with scikit in terms of model quality and speed,
often even working faster for moderate-sized problems,
as well.
But where TensorFlow really shines
is being able to run seamlessly across hundreds of thousands
of machines.
In fact, in many cases, we are able to train
models that are much larger than what
we had seen in [INAUDIBLE].
For example, with random forest, we
were able to train thousands of trees with billions of nodes.
And we have a paper at NIPS that talks about that.
With SDCA, we saw 10x to 50x faster convergence compared
to Google's highly optimized internal [INAUDIBLE]
implementations on logistic regression for [INAUDIBLE]
with billions of examples.
With WALS, we are able to factorize
a huge matrix, 400 million rows by 600 million columns,
into factors with 500 columns in under 12 hours.
Note that this is 50x larger than what
we could do with an early MapReduce-based implementation.
So, in summary, we have really high performance distributed
and extensible implementations of different ML algorithms
that are working out of the box in TensorFlow.
These bring together the ease of scikit
with the power and scalability of TensorFlow
and moves TensorFlow one step closer to being a more
complete ML solution.
I am hoping, at this point, you guys
are excited to get started, so here
are some links to core and documentation,
and here are a bunch of estimators
that are ready to use.
With that, thank you.
[APPLAUSE]
[MUSIC PLAYING]