YUFENG GUO: As machine learning has taken off
in the last few years, the increased demand
for specialized computing resources for training
and predictions has led to the development of Tensor
Processing Units, or TPUs.
But where do they come from and how do they work on the inside?
Stay tuned to find out.
[MUSIC PLAYING]
[SPACESHIP NOISES]
Welcome to "AI Adventures," where
we explore the art, science, and tools of machine learning.
My name is Yufeng Guo, and on this episode,
we're going to begin a two-part adventure
into the world of high-performance computing
at Google.
In the first episode, we'll look at the original TPU,
its design, and how it came to be
used in everything from Google Photos to Google Translate.
Then, in the second episode, we'll talk about TPU V2 and V3
and learn about how they work to enable the next generation
of machine learning.
The trouble with modern computing
is that we keep wanting more processing power.
Unfortunately, we're beginning to run into the very limits
of physics.
At least, this is very true for general purpose processors,
which is what you might find in your everyday laptop
or smartphone.
The need for more specific computational power
is what led to the first tensor processing unit to be created.
Understanding the original designs of the TPU
and why it was built that way can help us
as we look forwards towards designing
machine-learning architecture and software
systems in the future.
The first tensor processing unit was built as a PCI Express
expansion card, and it was plugged
straight into existing server racks in Google data centers.
It had a 700 megahertz clock speed
and had 40 watts of power consumption,
and it's been in production since 2015, powering things
like Search, Translate, Photos, and of course,
famously featured in the AlphaGo match in South Korea.
Not only was the TPU V1 much more performant
than the existing CPUs and GPUs of the time,
it importantly had a much higher performance per watt of energy.
These two effects combined to make the first TPU
a very effective chip.
The chip featured an architecture that
was specific for deep learning.
It utilized reduced precision, a matrix processor,
and a minimal design to reduce overhead.
Let's look at each of these now in turn.
Neural networks boil down to a rich set of matrix
multiplications and additions.
The team decided to reduce the precision of the chip,
using just 8-bit integers rather than
the conventional 32-bit floating point numbers
and using a technique called quantization that could convert
or map 32-bit floating point numbers to 8-bit integers.
So this allowed significantly more integer multiplier units
to be fit into a single chip.
So, speaking of which, let's talk
about that matrix processor.
Conventional processing systems read and write
to memory very often, sometimes after every single operation.
This is because they do general purpose processing.
So in order to support many different kinds of operations,
the hardware design must necessarily be fairly generic.
However, this leads to bottlenecks
due to all of those memory accesses back and forth.
So what if we could build a special purpose processor that
performed a much more limited set of operations,
but did so more quickly?
This is what the TPU tries to do with its matrix processor.
It uses what's called a systolic array in order
to do a large hard-wired matrix calculation without memory
access.
One analogy I've seen used to describe
this is that a CPU is like printing out
letters one by one, stamping them out,
whereas a GPU is printing out a whole line at a time,
while a TPU, it's stamping out entire pages at a time.
The core of the TPU is a huge systolic array,
performing up to 250,000 operations per clock cycle.
Most of the chip is dedicated to this matrix multiplication
followed by addition operations.
Even though the clock speed is a seemingly paltry 700 megahertz,
it is performing a ton of operations every clock cycle.
And one final note about all of this
is that it is important to remember that in order
to achieve this design, the TPU version one only
did predictions.
Now, you can't get your hands on the TPU V1
or even use it directly, but every time
you use something like Google Photos or Google Translate,
there's probably a TPU powering your search somewhere.
On the next episode of "AI Adventures,"
we'll talk about the TPU V2 and V3 and how they work
and how to get your code to run on a TPU.
Don't miss it.
Thanks for watching this episode of "Cloud AI Adventures."
And if you enjoyed it, be sure to click the button
and subscribe to get all the latest episodes right
when they come out.
If you're hungry for more TPU V1 action,
you can read the paper that the team published about the TPU.
I have linked to it in the description down below.
[MUSIC PLAYING]