[MUSIC] Ravi Agarwal is a Technical Sourcing Manager,
and he's part of Facebook infrastructure group.
In Ravi's role, he's responsible for driving
advanced packaging architectures and
foundry for both networking
and AI/ML computer applications
to meet Facebook future workloads.
He's driving the chiplets business workstream in
the open domain-specific architecture sub-project within
the open compute project and he's working with
ecosystem partners to enable the chiplet marketplace.
Prior to Facebook, Ravi spent 12 years at
Intel Corporation in MCO and he
received his PhD duo major in
Material Science and Engineering and Polymer Science
from North Carolina State University and an
MBA from the University of California, Berkeley.
Ravi has co-authored more than 20 journal and conference
publications and he holds six years patents.
Ravi is going to talk to us today about Metadata centers
heterogeneous integration driven by
AI/ML and network applications.
Ravi, thanks very much, over to you.
Thanks, Rob. That was
quite an introduction I have heard after a long time.
Thank you so much. Hello, everyone.
Thank you for investing your time and joining
today's workshop in this talk.
I want to thank the organizing committee for giving me
this opportunity and inviting
all the great speaker earlier today,
so I think I learned a lot and I'll be
focusing today on Metadata
center heterogeneous integration,
which is primarily driven by
AIML and networking applications.
I have divided my talk into three parts.
I'll talk about the heterogeneous integration
for AIML application,
and in the networking space
some of the power challenges we're seeing,
and then disaggregation,
and then some of the chiplet work we are
driving internally and externally.
In the first half of the talk,
I'll be focusing on AIML application and how at Meta
we're approaching this from
a system level as well as from a rack level,
and how advanced packaging and interconnect can drive
co-innovation for
future technologies and platform solution.
This is a video of a typical Metadata center
which place to house or compute, storage,
memory, and AIML, and video accelerator,
which are all connected together with a network.
Here we're showing the types of workload Meta deal with,
and their relative importance.
Ranking and recommendation is
the most important workload.
That's all we recommend to users when
they interact with a family of Meta products.
Computer vision and natural language processing
is becoming increasingly important for us.
As one of the other speaker in the morning shared
AI model size are growing rapidly and
the graph here shows AI trend at Metadata center
and shows that the AI inference demand
is doubling every year.
AI inference puts pressure on our computer system with
200 trillion predictions per
day and six billions translation per day.
Now the question is, how does the accelerator
fit into our server and rack infrastructure?
Schematic here shows accelerator system level view
where a general-purpose server infrastructure is
shown on the left-hand side of
the slide and their multiple CPUs
are connected with each other to network.
This is a key piece of our infrastructure.
The section on the right shows how
Meta deployed accelerator where a number of
accelerators are fitted into [inaudible]
vectors plugged into existing data center
or Edge Server platform.
These accelerators are connected
through a PCIe switch and
go into a car called Glacier Point v2.
These GPv2 have similar power voltage as to Lake,
which are our CPUs as shown on the left.
This slide shows the component,
whether it is a CPU, accelerator,
or a GPU joined into our rack.
CPU, GPU, or accelerator are mounted on
a module which is shown in the leftmost image,
then these modules gets integrated
into a Glacier Point carrier car.
Each carrier car has 12 modules,
which they're integrated into a sled
which consists of four CPU for
general-purpose compute or two CPU
and two Glacier Point v2,
which is then followed by 16 sleds integrating
into a rack and that's how the rack is formed,
which is shown on the right-hand side of this slide.
In this section, I will cover
some of the network for our challenges and
the opportunity and how heterogeneous packaging
is helping some of those.
As AIML workload is increasing
in data center, machine-to-machine,
traffic is growing rapidly and
bandwidth demand is increasing exponentially.
Power for bandwidth requirement is
decreasing and the image on the right-hand side shows
networking is consuming a higher proportion
of data center power budget over a period of time.
As we are aware of that power allocation
in the the datacenter is typically constrained.
If networking will consume more power in future,
less power will be available for some of
the important functions such as compute and storage.
Power has become one of the biggest emerging challenges
as we have seen even in
the morning from some of
the presentation and as
Barrack and Catherine both pointed out.
Where does this power gets used?
This is a typical type
of the rack address the blue switch at 100 leaks are
these where 50 percent power is being
used for switching and 19 percent for optical link.
But when we look at carefully,
the IO between switch and
optics account for roughly 20 percent,
which are shown in the red and orange in this pie chart.
By moving these two things closer,
we can eliminate up to 20 percent and
potentially up to 50 percent with the saving,
and the real timer goes up as speed of electronics go up.
As Catherine shared earlier today the evolution of
meta-optical integration where we
started with pluggable optics,
moved to onboard optics,
and now evaluating co-packaged optics.
That way, we are bringing
more and more integration of optics and switch basic.
This is a schematic of
a co-packaged system which has switch
packaged or IC along
with eight package of optical module in this picture.
It requires several technology developmental and
some of them are forcing the industry for packaging,
thermal, and reliability.
Some of the speaker today also covered some of them.
When both switch and optical package
are mounted on a common substrate,
it will have either integrated laser
as part of optical module or field serviceable laser,
and there are several packaging challenges.
Some of them are highlighted in this slide.
This co-packaged optics will
be the largest package almost
2x the largest package of level in
the market for AIML on networking application.
Currently, the industry doesn't have
such a large organic substrate.
We are looking into alternative options
by which we can come up
with such a large co-packaged optics solution.
Since it will be a very large package,
it will require a special thermal
management solution ranging from
commonly across these several packages
or maybe individually for each of these packages.
That followed by a common heat sink.
Being a large package comes with
a lot of thermal mechanical challenges.
The warpage management of such a package will
be a critical element.
Since optical module will include
several days integration and we're exporting
packaging technology ranging from 2.5D
to 3D dice stacking.
In this case, switch will
also bring some of the new business model either to get a
switch IC or switch as a package which can
be ventrally mounted on a substrate
and create a co-package optics package.
In addition, some of the companies are also
exploring switch IC as a chiplet packaging option.
Industry momentum is building,
but it is fragmented as can be seen in this slide.
There are several examples the industry
is rising to the challenge for co-package.
How are these challenges?
Is that all of them are proprietary.
The challenge is to drive a part to
a standardized interoperable ecosystem which enable
optionality and allow mix
and match between optics and switch
ligand as we do today with for pluggable optics switches.
Now I'm covering the final part of my talk is hard.
What are some of the challenges we see for chiplet and
what are some of the opportunity we see for chiplets?
This slide shows some of the chiplet innovation
happening in the industry where disaggregation
into smaller chiplet is allowing to go
beyond the full article size of the silicon.
The schematic on the left is
showing a memory and compute disaggregation.
Last year, we had a public announcement by AMD
where they have demonstrated using DSMC,
3DC, or IC technology,
SRAM disaggregation over compute.
The schematic in the center shows
disaggregation of compute chiplet by
allowing system-level optimization of
designing a system in which compute
will go to the advanced node and memory
and IO can be in the lagging node.
This approach is being already commercialized by AMD in
the epic chip and this
delivers not only performance but at the same time,
optimize the cost by mixing the silicon node.
The schematic on the right
shows system-level disaggregation where
chiplets can be used to disaggregate between compute,
or compute and memory, or memory to memory.
We have seen an example from
Intel where disaggregation is proposed
using silicon photonics chiplet
along with acute and graphics chiplet.
To continue Meta journey of
developing an open ecosystem and addressing
challenges related to flowing off
most law by enabling heterogeneous integration,
meta actively driving development of chiplet marketplace
in ODSA as part of OCB work stream.
As you are aware of, most of
the chiplet-based product are
vertically integrated ecosystem currently.
They don't allow you to mix
different functional silicon or
silicon from different vendor.
ODSA is focusing on IO disaggregation,
core disaggregation, and system integration of chiplets.
Here,
I am showing some of the business challenges
associated with chiplet,
which are categorized into four category,
cost reduction, fast innovation
, operation, and standardization.
From a cost reduction standpoint,
we would like to use
chiplet where we can reuse the silicon so that
we don't have to go through the tape
out in developing cycle again and again.
From an innovation standpoint,
the keys can integrate
third-party IP or silicon into our existing silicon.
One of the biggest hurdle or barrier we see for
chiplet standardization is
Die-to-Die Phi or interconnect standards.
Currently, most of the chiplet-to-chiplet interface
are proprietary and ODSA
and some of the other industry
Consortia are looking into developing
more open standardized Die-to-Die Phi.
Another key complex aspect of
chiplet is addressing the Known-Good-Die.
Because when you are doing
associated desegregation on the overall system level,
as well as the common security protocol
across supply chain.
With that, I want to conclude
my presentation and leave you with
a final thought that what we belive or what I believe.
If you want to go fast, go alone.
But if you want to go further, go together.
Great. Thanks very much Ravi for the overview of
a few different areas
of the Meta infrastructure and use cases.
We had one question from the audience here who came in.
Again, it was from Chris.
Chris, did you want to speak to that question directly?
Yeah. Just your architectural diagram
that you shared showed
a XSR interface between the ASIC and the chilpet.
If you compare that to
a VSR interface which you
would have with a pluggable module,
that's about a 10 percent
power savings if you maintain the optics the same.
Is that worth such a large disruption to the paradigm?
Chris I think you're referring for the
Co-Packaged Optics schematic, right?
Yeah.
I think we looked at Co-Packaged Optics,
maybe not in more
longer than technology solution where it's not
only first-generation maybe between VSR and XSR.
Maybe benefit is not huge but as the journey grows,
we are looking at Co-Packaged Optics will
provide not only power benefit,
as well as it will be cheaper.
It will be cost optimization.
I think maybe if I can chime in here, Chris as well.
I think it depends what you're comparing to what.
Because if you're comparing
a purpose-built VSR interface to
XSR you're right, it's about 10 percent.
But as you know, in many cases,
because of generality of
the cities technologies and reuse of IP blocks,
you may not be using a purpose-built XSR interface
on either switch,
or on the optics phi.
I think as Ravi mentioned,
this is a step along the road.
The increased density, easier deployment,
and overall direction to bring optics close to
silicon is the direction that we're leaning in this area.
Thank you.
Yeah.
There's another question from the chat, Ravi.
The question is, you mentioned
two extra largest package size today.
Are you talking about greater than
150 by 150 millimeters?
There is a lot of presentation has been
shared by ecosystem partner in life.
Yes, some of them are
going all the way to 150 by 150 and larger.
You believe packages of that size are
emerging and maybe within sight?
No. I think the question was, in my presentation,
was I referring 150 by 150
as a size which I'm referring towards?
Yeah.
Did I get the question right?
Yes, you did. Thanks for clarifying.
Yes. Answer is, it could be 150 by 150 or higher.
As I said, there is lot of data has been shared in YF
where ecosystem partner has
shared what substrate sizes they are working on.
Right. Thanks very much, Ravi.
Thanks very much for your presentation.
Thanks.