[MUSIC]
Hello, everyone.
My name's Katharine Schmidtke.
Today, I'm going to be talking to you about
Cloud-scale Interconnect Architectures in
the context of Hardware and Software Co-design.
I'll start by describing Facebook application,
I'll talk a little bit about AI and ML,
and then I'll describe
the relevance of software and hardware co-design,
particularly for performance improvements,
and then talk about network interconnects.
Facebook is a platform
that many of you are familiar with.
You'll recognize many of these icons, so Workplace,
Instagram, Instagram TV, Oculus, Messenger, and WhatsApp.
There's over 2.7 billion monthly
active users using Facebook at the moment,
and Facebook is the biggest social network worldwide.
What do people use the platforms for?
They're sharing a lot of information,
a lot of messages being sent.
Over 65 billion messages sent on Messenger,
there's 81 billion messages sent between
businesses and customers on Messenger,
there's 3.5 billion videos that are live broadcast,
and now, 50 percent of
that data is going through some AI or ML.
I'm going to talk about some of the
applications and why we do that,
some of the different workloads later in this talk.
That all happens inside one of our data centers.
We have 15 data centers now worldwide,
and at each one of these 15 locations,
there are six or so data centers
at each of those locations.
The power footprint here is significant and
we're very concerned about energy efficiency.
We've set ourselves a goal by the end of this year
to contract for 100 percent renewable energy,
and to have data centers that are
80 percent more water
efficient than average data centers.
You can find out more about that at our website,
where we go into more detail.
A little bit about [NOISE] the data growth
that's been going on in the data centers.
This chart shows
machine-to-user traffic and machine-to-machine traffic;
the green line at the bottom is machine to user,
so that has been growing.
It doesn't really show on this scale,
but it's been growing steadily,
and quite dramatically as
the number of users increases and the content that
they are sharing increases and the amount
of time that they spend on the platforms increases.
But as you can see from the chart,
that's all completely dwarfed
by the machine-to-machine traffic.
It's about 25 times more bandwidth moving around
inside the data center compared to
how much actually goes out to an end user.
Well, a lot of it is
the machine learning that we're working on to make sure
that workers and end users is relevant and is optimized.
Of course, then there's redundancy and backup,
which also consumes bandwidth.
Let's talk about those AI/ML applications.
This is just one example,
I thought I pick one and show how much that's growing.
This is AI inference,
and this is over a year and
a half period back through just before 2018,
and you can see already
the amount of inference that we're
doing as a workload across our datacenter is increasing.
That ramp at the end has continued and
it's doubling approximately every year at the moment,
so we are on a pretty steep ramp here.
What's the workload?
What's going on in the data centers?
There are three major workloads,
and you can see here over 80 percent of
the workloads is ranking and recommendation,
and then the next one is computer vision,
recognizing images and tagging faces,
and then language to be able to do translation
into the different languages
that people are sharing and communicating with.
Recommendation models are definitely
some of the most important models.
As we looked around the type of
research that was happening typically in academia,
that research was really centered
on computer vision and language,
and not so much the piece that we're interested in.
Looking deeper into that, we discover it's
because the datasets aren't available,
and because the benchmarks
and the models aren't available.
We've shared that, we've opened
our deep learning recommendation model
and that's available on GitHub,
so we're trying to encourage research in these areas,
and we're happy to provide the tools to enable that.
This is a fairly typical machine learning flow.
Obviously, it all starts with data,
and the more data, the better,
that improves the models and the results of the models,
and then understanding the features,
so what is it that you're trying to get out?
Then it goes into a training cycle,
so this is training and evaluation,
training of the model, and then deployment,
where it goes out into
the data center on
and that's where the inference happens,
and so that's where the more
repetitive inference activity is happening.
Each one of these is growing.
In Facebook's data centers,
the amounts of data that's going
into the pipeline is growing.
Back in 2018, it was
just 30 percent of the data that was being used,
and then last year it was 50 percent,
and that's grown again since then.
The amount of training we're doing,
the activities that have grown,
we've got more training engineers,
we've got more white workflows being trained.
Then in inference, we're doing a lot more inference,
so 400 trillion predictions per day
and 6.5 billion translations happening on the site,
and on top of that,
a lot of the security precautions that we're taking to
make sure that the site is only used appropriately,
and the large part of that is removing fake accounts.
All of this machine learning
puts a burden on the infrastructure,
and I've highlighted a number of the challenges here.
There's challenges in the storage environment that goes
across the storing the data to feed into the pipeline,
and then there are network challenges
from moving the information
around both whether moving
the data in, performing the training,
and then doing the inference,
and obviously, just compute processes,
acceleration of those models,
and memory challenges to have
all the right information close to the computation.
This is a lot of strain that goes on the network.
Then add on to that,
the speed of innovation,
how fast this is changing and improving,
and the sophistication level is increasing too,
so this is another big design challenge for the team.
The only way to get ahead is to co-design;
co-design the software and the hardware.
That means designing
the model to fit what the hardware can do,
but then improving the hardware,
and then feeding that back into the software
to allow more sophisticated models.
We've done that in a number of ways.
Starting with the hardware,
we've created very flexible hardware,
and this is an example.
This is our inference platform,
we call this Yosemite.
This is the second version of Yosemite, so Yosemite V2.
The architecture is quite simple,
at least, when you draw in the diagram,
but it allows us to address
high computes and memory capacity,
with high bandwidth to move the data around
and to have the model move at a rapid pace.
There are multiple accelerators and those are
tightly coupled together with a
very high-spec PCIe switch,
and this is something we can build with a single chip,
so it gets put into a module that
allows us flexibility to
configure that module in different ways.
This is something that we've open sourced into OCP,
so this is a place for us to share
the modules that we've developed.
Then as part of the community,
we've made sure that those modules
accept accelerates from a number of different suppliers.
Here I've listed some of
the suppliers that are active in this area,
and it's a community-driven approach.
These are all parts
of OCP collaborating on this module.
Then our training fabric is called Zion,
and this is a platform that actually
has a number of different networks.
It has the compute fabric,
a CPU fabric, and then it
has the separate accelerator fabric,
and that's to get the performance
that we need for this particular training,
so that the model solves quickly.
The Zion configuration offers an aggregate of hundreds of
teraflops of bfloat16, very high capacity.
It has terabytes of DDR for very large modules,
and hundreds of gigabytes of HBM,
so that these parts in the model that
need really fast access.
To provide the flexibility,
we lead the establishment
of this accelerated this OCP accelerator module.
This is a sub-module in the platform,
that can be configured with a number of
different accelerators to provide flexibility,
and again, this has been shared with OCP,
so it's available for
anyone in the community to get access to.
I talked a lot about community and sharing,
and this is another opportunity to build community,
to build an ecosystem,
and that's in the area of chiplets.
Before we were talking about the platform
and the rack level hardware.
This is configuring
different functionality by using chiplets,
that then come together into the module.
This work is being led through the ODSA,
so there's a number of different activities
going on at the ODSA at the moment.
In defining open D2D interfaces,
so that each of these chiplets can
communicate and are interchangeable.
Reference designs which are a starting points
for new designs, and reference workflows.
I mentioned we've shared a number
of different workflows and workloads as references,
so these are allowed re-usable open practices.
Then on the right-hand side of this chart, OCP,
this is the modularity in
the different modules that we're sharing,
where these chiplets,
they go into these modules and then these modules get
built up into servers or a server rack configurations.
This will allows us to
integrate the best-in-class chiplets
for multiple vendors through these open interfaces.
This brings me to network interfaces,
so not just chiplet interfaces,
but on a much bigger scale,
and you can see today I'm
standing inside one of our data centers,
this is the wonder of Zoom.
This is looking across one of the data centers,
and this is an aisle
in one of the main switch points in the data center.
This is where the building interconnects come together,
and this is all cabled with fiber,
today it's all single-mode fiber.
These network interconnects also consume a lot of power,
so I'm going to talk a little bit about those.
Earlier, I talked about how
the bandwidth is increasing over time,
and while the power per
unit bandwidth has been decreasing,
it hasn't been decreasing at
the same rate that the bandwidth has been increasing.
That leads us to a situation
where the power consumed by the network is increasing,
and here's some data from
2012 where network looked very small,
seemed negligible, build as much of that as you like,
doesn't consume much power compared to
all the compute and storage in the data center.
Today that red section has become much more apparent,
and we're paying much more attention to it,
and then what concerns us
is where this goes in the future.
It's consuming more and more power,
as a larger fraction of
the total data center power consumption.
As I said before,
we are trying to be
very careful with the amount of energy that
we consume and make sure that that's on renewable energy,
and this just makes
the problem harder to solve if
there's this much power being consumed.
One of the ways that we're
hoping to solve this problem is
by integrating the photonics
onto too smaller form factors,
and moving the photonic interconnect
much closer to the host ASIC.
This is an IO challenge and burning this much power on
that IO is never desirable,
so by moving the photonics in close,
those yellow blocks, in really close to the ASIC,
like in this particular case,
this would be a switch ASIC,
this allows us to reduce the power,
maybe by 20 or 30 percent, so quite considerable.
Integration gets you all the benefits of size as well,
so instead of, in this picture,
it's just four that we have at the edge of a card,
by miniaturizing, we would actually go
to 128 around the switch ASIC.
Continuing on the theme of
collaboration and working together,
this obstacle integration, we're
exploring it in a couple of different forums.
We've set up an industry forum
working together with Microsoft.
We've sent up a joint design foundation,
and we've put the requirements as Facebook and
Microsoft see them for this kind of
technology up on the site.
This is the co-packaged optics website,
and we've released three documents already that
outline the performance requirements for this technology.
Then the other place that we're collaborating is
on power efficient analog coherence interconnects,
some these are longer
reach interconnects for our network,
and we're collaborating with Clint Schow at
UCSB on a program called INTREPID,
and that's funded by
ARPA-E as part of their ENLITENED program.
I'm sure you're going to hear more about
that in this workshop today.
In conclusion, there's a number
of performance drivers that
each of them are accelerating,
and we're in this catch-up mood,
and balanced with that are
the practical limitations around
energy and energy consumption,
and all the different ways that we need to be
more efficient with our use of energy.
Then the solutions to that,
I talked about co-design as a way to improve
performance and that feedback loop
between hardware and software.
I talked about chiplets as a way to get
flexibility at the chip level,
and collaboration as a way
that I see it all comes together,
and that we build and share together,
and create an ecosystem
that supports many different businesses.
That's all I had for you today.
Thank you very much for your attention,
and if you'd like to reach out
and find out more about this work,
please contact me at these locations.
Thank you. [MUSIC]