- Okay, thanks so much.
As mentioned, my name is Kim Hazelwood
from Facebook AI research.
There I work with in an amazing set of engineers,
scientists, researchers, who are passionate about
the intersection between systems and Machine Learning.
And today, on behalf of the entire team,
I wanna talk to you about an often overlooked space
within Machine Learning and systems.
First, let's take a look at how we got here.
Over the past 20 years, we have seen
a fairly significant increase in investment
in Deep Learning.
Now a lot of that has come from fairly significant advances,
algorithmic advances, that have really driven new use cases.
Now a lot of those algorithmic advances
have been empowered by fairly fundamental breakthroughs
at the system's layer, be at hardware, software systems.
And then on top of that, we now have access
to significant amounts of data, better quality data,
as well as the tools needed to be able
to aggregate and procure that data.
So now let's also take a look at the end to end
Machine Learning Execution Flow.
So we essentially have three stages
of the Machine Learning pipeline.
We have the Data stage, we have Training
and then we have Inference.
In the Data stage, we're taking unstructured data
and structuring it into features.
And the Training stage, we're taking those features
and determining how much do we care about any given feature,
and building out better models.
Once we're happy with the model,
we may deploy that out into production or to users.
And that in it as itself produces additional data
that we can use to advance the the models
that we've developed, continuing the cycle.
So what I'd like to do is talk a little bit
about each of those three stages
and the scale at which we're, we're facing
each of these stages at Facebook.
So first, let's take a look at data.
So if we look at, over the past year,
the amount of data from our data warehouse
that is feeding ML models, that went from 30% of all data,
driving Machine Learning models, up to the current state
of 50% of all data.
And then on top of that, the amount of data
that we've had doubles every year.
So over a one year span, if you're doing the math,
this ends up being a 3X increase
in the amount of data that we're using for Machine Learning
over a single year.
Now, if we take a look at Training.
First, let's look at people, so the number of engineers
who are training models as there day job each and every day
has doubled over a 12 month span.
The amount of models that they're training
has increased by 5X over that same time duration,
and then the amount of computational capability needed
to be able to power all of that Machine Learning training
has increased by 8X over a one year span.
And then how about inference?
So each and every day at Facebook
across all the family of products,
we're doing hundreds of trillions of predictions
across all products.
These may be translations from dozens of language pairs
to other language pairs,
they may be detecting objectionable content
and removing hate speech from our platform
or detecting fraudulent accounts,
which we, which our automated systems do each and every day
catching millions of fake accounts.
So now let's talk about the Machine Learning models
that power all of that.
All of our products.
Many people have a fairly simplified view of the world
when it comes to Machine Learning,
where the focus is either on Computer Vision
or natural language processing.
So we get a lot of questions along the lines of,
"Do you care more about CNN or RNNs?"
But the reality is that at Facebook,
the picture is much more diverse
and we have a wide variety of models in use.
Some of them not even in the category of deep learning.
And, in fact, if you look across
all of the various use cases,
we have a pretty heavy focus on MLP
across all of our Ranking and Recommendation products.
So it's that Recommendation space
that I wanna talk to you about today.
So what are we using Recommendation for?
Well we're using it for all sorts of use cases.
If you take a look at your Facebook newsfeed,
we're using Recommendation to determine
from the thousands of candidate stories
we could have showed you,
which ones are you most likely to like and engage with?
When it comes to stories,
which is across many of our products,
we have lots of options that we could have shown you,
how do we choose which ones are you most likely to like?
And then Instagram itself, if you're searching,
knowing what kinds of images you'd like to see,
well, this is all powered by Recommendation Systems.
So let's talk about the Compute Footprint
that is powering all of those systems.
So if you look across all the Training at Facebook,
over half of it is Training Recommendation models.
And then if you look at the deployed models
we have in production,
over 80% of our computational resources
are being spent making predictions
for Recommendation Systems.
Meanwhile, if we take a look at the research community,
you'll see a fairly different picture.
So what we did was looked at the top system conferences
for the past five years,
and took a look at what kinds,
what areas of ML were they focusing on?
What were they designing systems for?
And there were, we noticed that Computer Vision
really took the cake.
So most people were working on developing systems
to recognize cats and dogs.
Over 82% of publications are focusing on that.
We saw 16% of publications
were focusing on natural language processing,
and then a mere 2%. were focusing on Recommendation Systems.
So what's going on?
Why is there such a disconnect between,
what is happening in reality and in production
and what researchers are focusing on in the community?
Now before we uncover that,
first let's sort of run through
what a Recommendation System's doing from a high level.
So if you think about the scale
at which we're dealing with Recommendation Systems,
we're dealing with hundreds of thousands of items
that we're trying to decide how to rank,
in order to make a prediction
about what should be the item that we show a given user.
So we start from hundreds of thousands of candidate items,
we're sending it through a Neural Recommendation Model,
which has billions of parameters.
We're then coming up with this Ranked list.
And then that list is going out to users
and the scale of users at Facebook is in the billions.
Let's take a step down
and take a look at how do those Recommendation models
work from an algorithmic standpoint.
So the thing to know about Recommendation Systems
is they're leveraging two different types of features,
they're leveraging dense and sparse features.
Dense features for all of the continuous information
that we have, and sparse features
for all of the categorical features.
And the way to think about continuous features,
is these are features,
these are questions that have a concrete answer.
And then categorical and sparse features,
are a vast number of yes or no questions
about the user or about the items
that we're trying to recommend.
So the way we represent the continuous features,
we send those through a Dense DNN MLP,
like you're expecting.
When it comes to the categorical sparse features,
there we're leveraging Embedding Tables.
And this is just a representation
of all of this sparse data.
We then need to take those two different components
and Integrate and combine them into a singular DNN.
And coming out of that singular DNN then
is a set of predictions across for each individual item.
What is the likelihood that the user will like this?
And then we can form a ranked list from that.
And this allows us to come up
with a recommendation for what the user might like
and recommend a book or an article of clothing or a photo.
So when it comes to evaluating those Recommendation models,
there's a couple things we need to keep in mind.
So number one is it's widely known
that ranking more items leads to better Recommendations.
Therefore, from a systems perspective,
what we wanna focus on is very high throughput.
We wanna be able to evaluate lots
of different candidate items all the once.
At the same time, there's a user waiting
on the other end for that Recommendation.
Therefore, we have a pretty firm latency requirement
so that they're not left waiting for that Recommendation
to the point where they don't care anymore.
So the way we evaluate and the the way we optimize
our systems is that we're focusing
on latency bounded throughput.
Now when it comes to how those Recommendation Systems
are hitting the hardware, or hitting the underlying systems,
it turns out there's some fairly unique challenges
when you focus on Recommendation Systems.
So I'm gonna talk about three of those challenges.
First, I'm gonna talk about the Embedding Tables themselves
and the stress that that introduces into the system.
Next, I'm gonna talk about how the Recommendation models
themselves have a wide variety of features,
of axes that they will vary
and how they execute on the underlying hardware
will vary, the performance will vary
depending on the features of that specific model.
And so I'm gonna talk about each of these
systems challenges in turn.
So first, let's take a look at the embeddings themselves.
So the embeddings are very large.
So these are orders of magnitude
larger than anything we're used to seeing
if we're focusing on CNNs, RNNs or fully connected layers.
So there we have to deal with pretty significant challenges
to our storage hierarchy
and how we optimize for that.
On top of that, because we're dealing with sparse data,
the computational intensity is actually quite low.
It's a lot of zeros.
It's not particularly computationally intensive
to multiply lots of different numbers by zeros.
So the overall FLOPS per byte is significantly lower
for embeddings than we're used to seeing in CNNS and RNNs.
And then finally, because the the data is so sparse,
the access patterns throughout memory
are going to also be very irregular.
And this is pretty challenging to deal with
from a systems perspective.
Next, if you take a look at the models themselves,
so if we took a look at three different
Recommendation Models that are in use at Facebook,
and it turned out those Recommendation Models
varied significantly in some of the fundamental traits
that they had.
So the across the three Recommendation models,
you would have fallen Connected layer sizes
varying significantly,
you had, the number of embedding tables
would vary, the size of those embedding tables,
how often we're accessing those embedding tables
would vary significantly across
three different Recommendation models.
Then if we start to take a look at the bottlenecks
that get introduced by each of these Recommendation models
and hone in on the specific operators that,
where all of the time is being spent.
That also was varying
across different Recommendation models,
to where you can't even just go in
and optimize for a very important operator,
because that's gonna be different
between the different Recommendation models
and a lot of that had to do with some of the characteristics
of how the Recommendation Systems were set up.
And then finally, there the sensitivity of those models,
when it came to how they would perform
on the underlying systems, this also would vary,
and it was very sensitive to any minor change
we would make like Batch Sizes would swing,
which hardware system was better optimized
for that particular model.
Therefore, it it's extremely important
that both the model itself and the underlying system
are co-designed, designed together
so that you can optimize for the specific bottlenecks
that are going to come into play
and that you define your Batch Sizes
according to what the underwriting system prefers.
Okay, so let's go back to my earlier point
about what's going on in the research community.
Why is everybody writing papers
about recognizing cats and dogs
when we have a fairly important problem on our hands
on how to design systems for Recommendation Models.
And I've already outlined a long series of challenges
that need to be solved in that domain.
So if you think back to what's different
about Computer Vision and NLP domains,
you'll get a little bit of a clue
if you take a look at how they were able to evaluate
any new ideas that they had.
So it turns out one of the key enablers to doing research
is having an agreed upon standard set
of workloads and data sets that everyone can agree upon
and can optimize for.
And in CV and NLP communities, we had those,
but in the Recommendation world,
Recommendation is extremely sensitive to the data set.
And a lot of the data sets are proprietary.
So this made it extremely challenging
to even work in the space.
So it's not that the researchers didn't think
it was an important problem.
It was that they didn't have the tools,
techniques and data that they needed.
So that's why I'm excited to talk about DLRM.
A year ago, Facebook released DLRM,
this is the Deep Learning Recommendation Model.
What this is, is a configurable model,
where you can change lots of different parameters
about the type of Recommendation that you're trying to do.
The ratio of dense to sparse features,
the widths and depths of your DNNs.
All of these can be varied so that you can get a good sense
of how sensitive are your systems
to the types of variables that will change
across Recommendation models.
And so that has been open sourced It's available on GitHub.
And then on top of that,
there's also a industry wide effort
to standardize the workloads and the datasets
needed to be able to evaluate systems in the ML community,
and that effort is called MLPerf.
This is a fairly widespread effort
across nearly 50 companies and dozens of universities.
And the latest release of MLPerf
includes the DLRM benchmark and it also includes
a non-proprietary data set, adds data set that can drive
the DLRM model.
So now we have all of the ingredients
that we need to be able to kick start research
and advances in the Recommendation community.
So hopefully, throughout this talk,
I have convinced you
that Recommendation Systems are important,
it's over 80% of what the computational cycles
at Facebook are used for.
Also, they're under-invested,
only 2% of the recent publications coming out
of the systems community were focusing
on Recommendation Systems.
And at the same time, they have a set of unique challenges,
lots of problems that need to be solved.
And so that's why it's very important
that we're remedying this underinvestment
and that we have more people working in this space.
And to be able to enable that,
we are excited that now there are standardized benchmarks,
workloads and data sets available for the community
to be able to leverage.
Okay, so thank you so much for your time
and if you'd like to learn more,
feel free to check out the Facebook research website,
where we have lots of publications,
you can download the DLRM recommendation model from GitHub,
and you can contribute to MLPerf
or compete in the MLPerf competitions.
Thanks so much.