HENG-TZE CHENG: So hey, everyone.
Thanks for coming to the talk.
My name is Heng-Tze.
You can call me Heng.
I work at Google Research.
And today I'm very excited to tell you all about Wide
and Deep Learning and how you can use it to combine the power
of Memorization+Generalization on TensorFlow.
At Google, we started the Wide and Deep project with a mission
to combine the power of memorization and generalization
on one unified machine learning platform for everyone.
And when we say, everyone, we really mean not
just every product team at Google,
but also, everyone in the open source community,
everyone in academia, as well as industry.
So we started working towards this mission
by jointly training Wide Linear Models on the left and Deep
Neural Networks on the right, and create an easy
to use API, high-level API, available in TensorFlow,
so users can start with simple linear models
and gradually add complexity without ever
having to leave the platform.
So the first question is, what do we really
mean by memorization and generalization?
And to answer that--
before we dive into machine learning,
let's take a step back to think about how we learn as humans.
So as little kids we start forming our knowledge
by observing everyday events.
For example, seagulls can fly or pigeons can fly.
We're very good at memorizing these maybe unrelated
boring facts very precisely.
But, as you can imagine, good memorization probably
doesn't scale well to over eight million
animal species in the world.
You're not going to see all of them
and not going to memorize all of their names
and whether they can fly or not.
So a very powerful step in our learning is generalization.
By saying, animals with wings can fly,
we're basically summarizing all of these sparse observations
into one very compact representation of knowledge,
and that's whether the animal has wings or not, right?
Using these compact rules you can generalize to many more
animals you may or may not have seen before.
However, generalization has its own limitations
because it doesn't always work on all cases.
For example, penguins, well, at least they try,
but the best they can do is some sort of a free fall
into the ocean, right?
So the real power lies in how we combine
the power of memorization and generalization.
And interestingly, this is very similar to what we observe
in the progression from a pure wide model for memorization
to a pure deep model, which gives us more power
in generalization, and finally, in combining wide and deep,
which is generalization plus memorizing exceptions,
which is saying, animals with wings can fly,
but penguins cannot fly.
So enough with the hypothetical animal examples.
Now let's come back to a real world example,
and I'll walk you through it.
In this running example we'll see
how we can apply wide, deep, and wide and deep learning
on the Google Play apps recommendation problem.
So suppose one of our favorite users,
Chris, recently installed the Priceline app.
The question we want to ask is, which other apps
should we now recommend, right?
And one way to frame this problem is--
to frame the recommendation problem as Retrieval+Ranking.
So let's start from the upper left, where I'll
walk you through a live query.
So we get a query from the users in the recommendation system
case as the user feature plus some contextual features maybe
at the time and so on.
And then our retrieval system will
retrieve the relevant items that relate
to this user and context.
And maybe, let's say, give us hundreds of candidates, right?
The ranking module will rank these candidates
and present the most promising ones, the most relevant ones,
to the user.
The user takes a look at the items on their phones.
And they can take one of the possible actions,
like, click on the app, install the app, and so on.
All of these are recorded in the logs, which the learner picks
up, and trains a new model.
And this new model will power the next iteration of ranking.
And that's a cycle of recommendation, right.
So this talk we will focus on how we use Wide and Deep
Learning to solve the ranking model of this app
recommendation problem.
And the query features, as I said,
is user and contextual features.
The item features are the app impression features which
we show to the users phones.
And the objective we're optimizing for is app installs.
So if the user installed the app the label is one,
otherwise the label is zero.
OK.
So let's get started.
Now the first probably the simplest model you can pick
is a Linear Model, or known as logistic regression,
in this case.
So the inputs are a bunch of features, right.
You can have the user features installed app as Priceline,
and impression features, which is
if we show KAYAK as impression, what's
the likelihood of install?
So the impression app, let's say, is KAYAK.
And we want to predict the probability that the user will
install the app.
Now how do we generalize memorization capability?
This can be done by adding a Crossed Feature, which
is the end operation over this binary base features.
So the cross feature can be, install app as Priceline,
and impression app as KAYAK.
If both are true, what is the likelihood of install?
And if we go into the data and we
see a lot of correlation, people who installed this also
installed that later, then you can memorize this frequent co
occurrence of these features.
So crosses help you with that.
Now you might think that, well, that's very specific, right?
How do we actually generalize to other apps that
are not Priceline or KAYAK?
Well, one way you can do it with Wide Linear Models
is to add less specific features, like app categories.
So for example, if you add a cross saying, if install app
category is travel and the impression app category
is also travel, what is the likelihood of user
installing that, right?
So once you learn this, you can apply it
to other travel apps not just Priceline or KAYAK.
Now the problem, which you might have seen already,
is that the feature space is very wide and very sparse.
There are millions of apps, billions
of users, if you cross apps with apps that's
a million times million which is a trillion combinations, right?
So most of these user impression app pairs will never
occur in your training data.
So how do you actually generalize to these unseen
pairs without doing a lot of manual feature engineering
that we did in the last slide, right?
And remember, in many cases, it might be hard for humans
to come up with these really good high level,
less specific feature crosses in the first place.
So one way to solve this to generalize more automatically
is to use Deep Neural Network with a dense embedding layer.
So starting from the bottom, you still
have your base binarized features,
which is, user install Priceline,
and let's say, impression app is Yelp now.
And let's say, you've never shown--
you happen to never have shown Yelp after people have
installed Priceline, so you cannot memorize.
Now one way you can train this is for each unique feature ID
you look up dense embedding vector
and these are initialized randomly
so initially you will make very poor predictions
and you compare the error between your prediction
and the ground truth label and you
start adjusting your embedding.
So you predict better and better.
So over time, the embeddings will
act more like automatically extracted features, high level
features, that very well describe these base features.
So combined with the hidden layers on top of it,
you can now automatically learn complex interactions
between e-space features.
Now we've talked about Wide and talked about Deep.
So when is Wide better than Deep and vice versa?
So we can look into the traffic patterns.
So on the left using the Wide Model,
you're basically learning a matrix to memorize which user
install app and impression app combinations correlates with
install and which one doesn't.
The problem is there are these question marks which means
there are no data to train on.
So if you notice that in the rows and columns
there are a lot of repetitive or similar structures
you can leverage.
So if you map all of these apps onto a lower
dimensional embedding space, on the right,
you might have that similar apps that
will be closer to each other in the embedding space.
So we can now generalize your prediction
to any app in the embedding space, right?
And in this hypothetical example on the left,
you would need 22 floats numbers to memorize these interactions,
whereas on the right, you only need 20
floats to store the embeddings.
Being able to represent the same or even richer
information using less number of parameters, meaning you
are leveraging the structure of the data better.
Now, on the other hand, there are
a lot of specific user tastes, niche apps, or just exceptions
in the app training data.
So for example, a Sheldon Cooper Bazinka app probably
wouldn't make sense to recommend unless the user has seen
the show, "Big Bang Theory."
So there are all these kind of niche combinations.
And if we use a Wide Model to memorize you can just
use five ways to memorize this diagonal matrix
in this hypothetical example, right?
Whereas, if you still try to learn
a low dimensional embedding, then you
will still need 20 floats.
And now the learner will try very
hard to push these irrelevant apps away from each other,
but find it very hard to do so in a crowded low dimensional
embedding space.
And what will happen is you will accidentally recommend
less relevant apps to the user.
And that's partly because if you have a high rank sparse
matrix like the diagonal matrix on the left
is, theoretically, very hard or impossible
to find these lower dimensional embedding that
will reconstruct the left.
OK.
So our idea is very simple.
Why not just combine the power of Wide and Deep, right?
So we not only get the combination of memorization
and generalization, and in many recommendation systems
and search ranking problems we often want
to balance relevance versus diversity,
as well, so we satisfy our users need.
So that's what we did.
We jointly train Wide Linear Model and Deep Neural Network
simultaneously by back propagating to both sides
at the same time in training.
So we connect them to the same last function and backdrop
to both sides.
Now you might notice that this is somewhat
similar to an Ensemble Model.
But in the Ensemble Model you would train two models
separately without knowing each other.
And only at prediction time you would combine
their predictions, right?
So we've been trying both.
And what we've observe is that doing a joint training,
because they know each other in the training time,
the Wide Model only needs to augment what the Deep Model
cannot learn well, and vice versa.
So you often get a much smaller model,
whereas if you train ensemble, both models
need to be fairly large and self-sufficient models
so their ensemble will make a good prediction.
And we're definitely actively working
on comparing the performance of both.
And we're happy to here what you find as well.
So this is what we got eventually
that was launched into the production of Google Play
apps recommendation.
So it's a three layer neural networks on the left.
And down below, you have your continuous feature
and categorical features that feeds into the model.
And this is augmented by a cross product transformation,
a cross between the user install app and the impression app.
And both are connected to the logistic loss
for joint training.
And we published these results in the paper, previously.
We ran online experiments on Google Play.
And we've seen that the Wide and Deep Model had a significant 4%
gain on our Wide Model, which is a very mature, highly
optimized, previous generation model that's been used.
And we also compare it with the Deep part of the Wide and Deep
Model, and it has a 1% gain on top of it.
And that's pretty significant for Google Play.
Now the good news is all these you can do today
in about 10 lines of code using the high level estimator API,
which Martin and others have talked about.
So all you have to do is, first, define the features you want
to use in your Wide Models, and configure the crosses you think
might be useful for memorizing a lot of combinations.
And the second part, you define the features
you want to use in the Deep Model,
and the embedding configurations.
And finally, define your model structure,
like, how many hidden layers, how wide are the hidden layers.
And boom, you can start training right away.
So I encourage you to go online and search for Wide and Deep
Learning to learn more about Wide and Deep
from the resources we provide.
So we've open-sourced it.
We have blog post, paper, a YouTube video, and also
TensorFlow tutorials to walk you through these.
And we found it useful for general recommendations,
search, and ranking problems, especially
if you have sparse categorical inputs with a large number
of unique feature values.
We're excited to learn if you find it useful in other use
cases as well.
And finally, I think we all want to learn
Wider and Deeper about machine learning and maybe just
everything else.
But I think the true power lies in the word, together.
So in the Wide and Deep project we
brought two pretty different modeling techniques together
on one unified platform.
And that benefited many product teams at Google.
And by open-sourcing and sharing our findings,
you and we are now able to collaborate even if we're not
in the same organization.
And together, we can get a deeper understanding
of how to combine these machine learning technologies
with a community that's wider than ever.
And I think that's a very powerful thing.
So I'd like to thank all the contributors
to the Wide and Deep Project.
And thank you very much for listening to the talk today.
Thank you.