For the first six decades of the semiconductor industry, manufacturers improved performance
by cramming more transistors onto a single die.
It wasn't easy, but it worked.
But now, the pathway ahead isn’t so clear.
The increasingly difficult economics of designing and fabbing a faster chip has opened the door
to new approaches for achieving better performance.
One of the more popular approaches is what AMD is doing with chiplets.
In this video we dive into how it works and why it now makes sense in this new semiconductor
world.
Today, it costs more than ever to design and fab a chip with a leading edge process node.
To scale transistor density every couple years or so.
Seemingly magical solutions like EUV lithography cost billions to develop and hundreds of millions
to deploy.
The customer has to make the economics work on that one - which means that nodes like
5 nanometers and the such can only be deployed at a titanic scale.
For instance, like mobile phone chips that sell in the millions each year.
Without that scale and without access to greater transistor densities, how then to deliver
increasingly better performance?
One thing that GPU and server CPU companies have tried is make the chip bigger.
Bigger chips mean more transistors despite lower densities.
However, total SOC size has a ceiling imposed onto it by lithographic mask sizes - generally
about 858 square millimeters.
In other words, it's impractical to try to etch a chip design that's too large.
Furthermore, large SOCs have yield concerns.
Bigger chips have a bigger chance of having errors on them.
An unrepairable manufacturing fault can ruin the whole SOC - a loss that the chip seller,
not the foundry, has to eat if it can’t find some way to use them elsewhere.
Such yield costs do not scale linearly.
In other words, doubling the number of transistors on a single chip more than doubles the cost.
Thus, the industry realized that it had to find new ways to deliver better performance
at a decent price to both the company and the end user.
In the same article in which he set forth Moore's Law, Gordon Moore had an interesting
suggestion:
> It may prove to be more economical to build large systems out of smaller functions, which
are separately packaged and interconnected.
Today's system on chips or SOCs are doing something akin to this.
I talked about SOCs and the reasons for their increased popularity in a previous video about
semiconductor design.
Modern SOCs like your Apple A15 chip integrate together a variety of different processors
onto one die.
These are little specialized pieces of hardware for specific functions you know the customer
will frequently use.
For instance, the dedicated neural network or image processor hardware for those iPhone
chips.
But like I said, an SOC puts it all onto one die.
That’s generally how you got the smoothest and fastest integration.
What Moore is talking about is to have separate, discrete chips.
That's the basis for what the industry calls heterogeneous integration or multi-die technology.
This is where you take what had previously been a monolithic SOC and partition it into
several tiny dies when you then reintegrate.
These little dies are perhaps better known in the consumer market as chiplets - in the
case of AMD.
And tiles - in the case of Intel.
The goal is to achieve better performance with faster development times at reduced costs
on yield, testing, and fabbing.
If you have an SOC with X number of transistors, splitting that into chiplets vastly lowers
the cost of the chip.
Assuming performance remains roughly the same.
This is not a new concept, obviously.
For instance, the industry explored something called Multi-Chip Modules all the way back
in the 1980s.
This is where you connect two CPUs together into a package - usually side by side.
Despite a lot of research being done in the space, MCMs never reached their full potential
in the 1990s.
The problem was that it was that an innovation from the packaging side of the industry.
Thus, vendors never really could wrangle the higher-level performance problems of integrating
multiple different dies solely through packaging.
For instance, the problems of quickly transferring data between the dies.
Furthermore, the substrates were expensive and traditional chipmaking techniques were
performing very well at the time.
So a generalized business model for the product never materialized - though niches did come
about.
And the MCM name would eventually turn into something of a black sheep later on.
Times and circumstances, of course, have since changed.
As a result, a variety of companies began re-examining the chiplet approach.
In 2010, Xilinx would be the first major semiconductor company to adopt chiplets for their FPGAs.
The Virtex-7 FPGA was fabbed on a 28nm process, splitting a larger SOC monolith into four
equal pieces.
Then recombining it so to get better yield and performance on par with that of the monolith.
A few years later in 2015, Marvell Semiconductor introduced their Modular Chip architecture
or MoChi.
This approach attempted to blunt the cost curve by modularizing your traditional SOC.
You would have a bunch of chiplets with different functions, tied together with an interconnect.
The idea has definite echoes of what was to come.
However, adoption outside of Marvell was sparse.
Years later, it seems like the MoChi architecture was only really being used internally for
the company's commercial products.
But they were still talking about it going into 2018, with a senior director saying:
> “Next year you’ll be hearing a lot more about chiplets ... [they] are a good solution
to the death of Moore’s Law.
We implemented this three years ago on a switch, and we have been re-using technology across
our product line internally.”
What AMD saw was that the industry was coming around on the chiplet approach.
And that approach is capable of creating a high-performing product at a drastically lower
cost.
The company's first product to adopt the chiplet approach would be its first-generation EPYC
CPUs, code-named Naples.
These are server chips to be used for data centers.
Server chips have different concerns than mobile or even desktop chips.
They are larger and have high performance demands - requiring more cores, memory and
larger I/O bandwidths.
Internal marketing noted that such a product needed 32-cores to be competitive in the market.
But like as we talked about earlier, a bigger chip on a leading edge node raises the risk
for lower yields because bigger chips are more likely to have gamebreaking flaws.
Using the 14nm process available to AMD at the time, this Naples chip would have been
777 square millimeters large.
This is not just very large but rather at the very edge of the lithographic reticle
limit.
If it were to be built as a monolith, the chip would have very low yields indeed and
be economically unviable.
But if you were to approximate the monolith by building four smaller 8-core chiplets and
packaging them together, this would save up to 40% on costs.
This is a drastic enough savings to overcome extra costs on the packaging side and a need
for a 10% expansion in silicon real estate.
Even if the fab makes some mistakes and the chip has up to four or less cores disabled,
a number of those can still be reused as components for a 16-core server chip.
A process internally described as "harvesting".
AMD would leverage this modularity a great deal in future products.
Of course, there is no such thing as a free lunch.
Chiplets are no be-all solution.
One of the main concerns involve the interconnects.
There are latency and power consumption issues with swapping data between the chiplets.
For the first-generation Naples chip, AMD implemented an interconnect they branded as
Infinity Fabric.
Despite the fancy-sounding name, what Infinity Fabric actually does is manage and facilitate
data transmissions in environments with limited input and output.
So it is an interface like PCI express, SATA, and USB - but modified to have a higher bandwidth.
The system is an evolution of something that AMD has used before for connecting items on
a motherboard via sockets.
For the second generation of EPYC - code-named "Rome" - AMD gained access to TSMC's 7 nanometer
process.
It allowed for a doubling in the transistor density in the core logic areas of the chip
while keeping power consumption steady.
AMD decided to leverage another benefit of chiplets.
It allows you to use a leading edge process node to fab the core part of the chip, while
keeping the non-core elements with a cheaper legacy fab you’re more familiar with.
Rome uses 8 small "Zen 2" CPU chiplets, fabbed at 7 nanometers.
They sit alongside a 14nm I/O die that handles analog functions like USB and SATA and other
system level functions.
These parts of the chip don’t need to be done at 7nm and in fact trying to convert
them would cause issues.
Stitching these 9 dies together required some work on the packaging side.
Especially since Rome needs to be upgradable from Naples, and thus has to fit in the same
socket.
This required the silicon and packaging teams to work closely together, side by side.
But the final product performs very well.
The second generation EPYC processors doubled the number of cores from 32 to 64;
And also more than doubled the number of transistors.
Despite this, the total size of the silicon only grew 18%.
Silicon from Rome would be later used to help create the third generation of Ryzen processors.
The first Ryzen series chips - Ryzen 1000 - used a single Zeppelin 8-core EPYC chiplet.
Ryzen 3000 would have one or two 7nm CPU chiplets paired with a base I/O chiplet.
Same concept as the bigger EPYC server chips, just different numbers.
Note again, the ability to reuse a CPU chiplet across multiple products for different users.
Now AMD can wait and review the market conditions before deciding whether to create more 24,
32, or 64 core EPYC server products or more Threadripper products for the workstations
or more Ryzen products for the desktop.
This modularity is critical to accumulating proper scale to pay back the high costs of
designing and fabbing 7nm wafers.
The success of the Chiplet approach is spurring a great deal of innovation, especially in
the advanced packaging world.
Big companies like Intel and TSMC are starting to offer their own packaging IPs and standards
into the market space.
But it is not a panacea.
It is a response to a certain situation occurring across the semiconductor industry.
Chiplets won’t work for everyone.
AMD just found itself in the right time and place for it.
The company is highly integrated, with the resources to design leading edge chips.
Furthermore, it plays in big consumer and business markets like video gaming and servers.
These are $100 billion industries with sufficient scale, competition and margins to justify
betting on a flexible, value-oriented offering.
All that being said.
Judging by how it appears AMD is doing in recent years, this bet seems to be paying
off extremely well.