[MUSIC] I aimed to focus down
rather than giving the usual TPU talk.
I'm happy in Q&A talk
more details if that's the relevant thing,
to focus on the greenness and
energy-efficient aspects of this conference
because that seems like what's front and center.
This is a particular slice through
TPU land and Google's view of things where my agenda is,
accelerators are actually green,
which may not be obvious because we're putting
liquid cooling and Mongo heat sinks on these machines.
Beyond that, I think Cloud accelerators
are greener and there's
a little bit of preview of coming attractions
from Google and papers that are coming out.
But there's surprising data
about how green you
can be depending on where you cite your data center.
I'll touch on that just a little bit in this talk.
Google builds TPUs largely for our own use.
They are also available as
Cloud TPUs to rent
with varying degrees of success about that.
I have some pretty pictures of
four-and-a-half generations of TPUs in this slide.
The one on the far left is the one I
worked very closely on the initial TPU,
which was an inference on the machine.
For its 2015 TPU,
it was pretty hot stuff with
92 Tera ops and only 75 watts of power consumption.
The subsequent machines have all been training capable.
They're all floating point enabled,
or at least these loads 16 floating point enabled,
which is actually a power hip.
Floating point is tougher to do than integer arithmetic.
That has caused us to
take a little bit of a step back on energy efficiency.
But what we got for that is the ability to
train and also the flexibility to
run in inference exactly the model that
we ran in training where the first TPU,
you had a whole bunch of
difficulty in quantization for deployment.
Where basically if you train in floating point,
which you still have to,
there's no algorithmic breakthrough
yet for training and fixed point,
then you want to deploy a fixed point device,
whether it's a mobile edge device
or the first-generation TPU.
That translation fixed point is
actually wonderful for energy savings,
but hard from algorithmic and system design perspective.
That's been a break an inference deployment for a while.
Anyway, what I've listed below the TPUs,
aside from showing the pretty pictures
and air and water cooling and so forth,
is the TeraOPS per chip.
Then the effect of watts per chip.
The bottom row of figures is
actually Tera operations per watt,
where TPUV1 is actually still
doing pretty well at one TeraOP per watt or 1.2.
The floating point machines took a step
back and are slowly crawling back
with Moore's Law continuing to deliver still
slightly better silicon processes where we're
inching back at 0.16, 0.27, and 0.78.
Tera now flops per watt.
But I think the contrast to draw is actually against
more general machines because
these aren't good specialized machines.
If I didn't show a general-purpose CPU,
but if you go back to the hassle of machine that's a
contemporary of the first TPU in 2015.
It's maybe 100 watts and one TeraOP.
That number would be 0.01 TeraOP per watt instead.
An accelerator is a huge step up from that.
That's the first part of
my thesis that accelerators are green.
CPUs are of course, responding.
Nowadays, you can buy
pretty awesome sapphire rapids machines that are,
I think in the 100 watt per socket,
100 Tera flop per socket range.
That's a great solution and that's
closer to that one number,
might even give a TPU run for his money at this point.
There has been a response from the CPU vendors.
But I think that
the energy efficiency ball
is ours to lose from the accelerator perspective.
We can build green accelerators.
For the large-scale training
that is front and center now,
the giant models where there's
a space race going on among different things called
GPT 3 and 4 and all names that I can't remember.
Rob pointed out that we're using
giant machines to train these giant models.
You've got a trillion parameters,
you're going to need a cluster of machines.
Google has been doing that since
our second-generation TPU,
our first-generation training machine.
There's a first-gen cluster with
only four racks and 256 nodes at the top of this slide.
I guess third-generation TPUs,
second-generation training machine with
eight racks, 1,024 nodes.
In the bottom of the slide, I don't have
a picture of the TPUv4 pods,
but those have 4k nodes and
another eight rows deep behind
this set of wires and additional wires, you can't see.
The wiring is pretty gnarly
for doing this giant machine training.
You can imagine the corresponding just
take the watts numbers from the previous slide
and multiply by node count.
That's just the chip watt not the overhead for
all the power supplies and the cooling systems
and the transduction from air to
water if that's going on,
actually not in a training machines.
This adds up to real power.
The scale of this is of course, warehouse scale.
I think Google touts warehouse scale computers.
This is one of our datacenters.
I forget where this one's located.
Normally, I show this slide and say it's
great that a datacenter can be pretty.
But I think in the context of power,
one of the questions is how many megawatts
does a building like this represent?
It's tens of megawatts.
I think our standard datacenter building
at Google was 20 megawatts for a while.
I think we're upping that now in building
30s and 40s and maybe even 50s.
That's a decent amount of power,
decent number of accelerators.
These pods are now a significant fraction of
the entire floor space or
the entire power budget
of one of these datacenter scale buildings.
This is industrial machinery.
I think people worried about
how large-scale heavy industry
was departing the United States,
maybe 20, 30 years ago.
It's come back in some ways.
In some ways, this is heavy industry.
Although one nice thing is that it's
relatively green heavy industry in
that we can use green power to run this.
It's not actually full-blown
industrial processes
with the big stacks some of the time.
That's maybe the second part
of my talk. I got four minutes left.
This slide is basically about, wow,
there's a lot of interest in ML and green versus
red AI about the environmental impacts
of how many cycles are we burning?
How many watts are we burning
to train these giant models?
Are they a significant fraction of
gross domestic power usage?
Are we going to melt the ice caps or
kill all the polar bears as part of this?
Some of the extrapolations are really quite dire.
The most recent one that I think is in
the dire thing sounds Malthusian to me in
the sense of Malthus who predicted
that population growth would
destroy civilization sometime in
the 19th century because exponential
growth unchecked would just,
we'd reach some terrible resource scarcity result.
If you do this
from look at the past
and just multiply out in the back of the algorithm,
it is indeed terrifying.
I think part of the thesis of
this particular article is that getting an increase
in improvement in error rate
requires exponentially invested cost.
I believe that's true for the scaling approach to models,
but I don't think it's true for
AI as a whole because I think we're
just barely at the beginning stage of
AI in terms of efficient methods.
I have other slides that I'm not sure today about.
I think we have great steam engines,
and steam engines are unbelievably
effective at replacing muscle power,
but they're not the most efficient thing.
There are electric engines
today that are way more efficient.
We understand thermodynamics,
so we can do a much better job.
But anyway, to the end of
the world predictions like there's some pretty dire stuff
about where are we going to burn in
Austria or in New York City or
whatnot as part of doing this?
This dialogue about,
are we going red or are we green is out there.
In my remaining time, I'll just do a preview.
Next month, Google and a bunch of
really senior famous people here
will be releasing a paper on archive.
It'll be coming out in
actually a computer talking about carbon footprints
of ML training and talking about how big Google scale is,
what fraction of our power budget
in fact goes to neural network training.
I thought I'd preview a little bit about that,
not give you all the results.
I've fuzzed a bunch of the figures to talk
about how do you think
about where your power is going and training.
Let me mention four different factors
and apologize to Mark Hill for
stealing his 3C's thing and renaming it the
4M's for ML energy efficiency.
The four Ms that I have that contribute to
how energy efficient you
are in training are first of all, model.
If you look at the jump for a transformer,
which is the basis of the current giant model,
revolution two primer.
Four years later, it's
a forex energy improvement for similar accuracy.
The change machines, if you buy
a really great video machine
from 2017 and compare it to
our latest TPUs or the latest TPUs,
that's more than a 10x efficiency improvement in
terms of what work you can
do for the same amount of power.
That's not just moral law,
that's also designed in the machines that we're building.
My third thing is forced fit.
I call it mechanization,
but it's actually data center deficiency
or what people labeled PUE.
Google's standard data centers at 1.1, that's basically,
you pay 10 percent over what you're actually powering
the chips with to run that whole data center.
Industry-standard is probably more like 1.6 or 1.7,
and so that delta is another factor, 1.4.
Last M I have is maps.
Where you do your work actually is
a huge delta in terms of the greenness of the power.
It turns out that Oklahoma,
where Google has a bunch of data centers,
has continuous nighttime winds,
and continuous nighttime winds
means we can use wind power,
which is entirely green.
If you look at the percentage of time
a data center is able to access green power,
it's highly variable depending on where you are.
I think it's Singapore is
terrible because it doesn't have
natural resources in the same way that Oklahoma does,
and that can vary from almost 100 percent
green to like a little 20 percent green.
Because that's what you're getting out
of your photovoltaics.
That factor is another 5x
If you multiply these altogether,
the deltas from where we were four years ago
till now are over a factor of 2015.
I feel like we should definitely
pay attention to the planet scale,
nations scale, big companies scale amount
of power we're putting into these models.
But I also think we're evolving rapidly.
We're rapidly working on the efficiency front,
and all is not lost as long as we pay
attention to these few interesting things.
Sorry, I ran over one minute but well,
the moderators like me to do one question
before going on to the next person.
Absolutely. We have time left.
I think you have a path to 250x.
We just need to find another 4x to get to the 1,000x that
John laid out earlier in the session,
but it was fantastic.
You should patent the full ems
before and what comes back after you.
[LAUGHTER] Let's see, in terms of Christians,
the window I have doesn't show
any specific questions. I don't know, Mark.
There's another window I had to
get there later. But there is Q&A.
About retraining on redundantly
, slightly different object.
Yeah. Okay. I don't know how to do the cross-company one,
I think that's a really excellent question.
I guess I have two answers.
Some of the giant models are actually designed to
train once and then redeploy many ways.
They're zero-shot models.
You train GPT-3 once that might be expensive,
but then you like,
post-fine-tune it to do
a particular translation task or Q&A task or whatnot.
That's a lot cheaper.
That's one awesome way of amortizing that.
The other thing is, you're absolutely right
about like reuse the datasets.
I wonder how many M-Nest
are going at a time inside Google and M-Nest
is small enough that maybe
it doesn't matter that we've got
everybody has their own private copy
the M-Nest and you're rerunning it on a thing.
But what about Common Crawl
or what are the other bigger datasets like,
shouldn't we build a system-level solution
that if everybody is actually training on Common Crawl,
maybe randomization
and our disk accesses should be put together?
We should group those Common Crawl things.
This is history of computer systems in general.
You want your personal workstation or do you want
a batch system that has common utility efficiency?
You could build batch training systems
within an enterprise like
Google or another Cloud provider,
and maybe it has a little bit of
scheduling of latency hookup where you
got to catch the time when
particular training run data is available on the network.
But that's a potential area for saving,
so and I think it's hot for research.
Nobody actually ever deployed
things that I've heard about.
How was the one I saw? Are there other questions?
Actually now I'm really
like 15 minutes is awfully tiny slots,
so maybe just a little pause.
Could I ask just a brief question?
Yes, please.
I think what's really interesting is
that the architectures that are
emerging as great learners are at
the same time a lot less energy efficient by this trend.
Maybe a general question for today is, of course,
we all know this famous fact that the brain does
this with an extremely small amount of energy.
That seems to be an architecture issue.
Obviously, it's capable of learning in
a very efficient way at
an extremely small energy expenditure.
That seems to be an architecture issue.
What worries me is that of course,
the trends are opposite.
It seems that we're not
really paying attention to the architecture issue
as much as we are to the percentage of learning.
That trade-off seems to just have a comment.
I mean, the trade-offs seems to
be almost neglected. Yeah.
Yeah. Lovely question.
I think what you said could apply both to
model architecture and to computer architecture.
Although, I think you are initially
motivated by model architecture.
I think to the biologically inspired thing, absolutely.
Like my brain is
a dim bulb 30 watts
or 90 watts or something like that, and yet,
I seem to be able to learn things on few examples in
ways that other machines don't
know. That is an inspiration.
It's also like terribly frustrating.
Like we really don't know how actual
biological learning works and what we're
doing with our brute-force steam engines
may not be what we're doing 50 years from now.
I would love to see those breakthroughs.
That might be another revolution
to the interplay of modeling
computer architecture though I think there is
efficiency and the sparsity is there in different forms,
although not necessarily a form that you want.
Like a bunch of
these giant models are sparsely activated.
They're a mixture of extra x-models, they have routing.
Only a tiny fraction, I think primer is eight percent
of weights get touched by a given eight model example.
That's another factor of 15 there of savings
versus our classic mode of
touch every weight with every example.
There's also co-evolution.
I mentioned that my TP-V2 was efficient energy-wise.
Transformer in some way can be
viewed as an adaptation for,
I can't do what I want to exactly
with attention or attend to things.
But if I have a TPU,
I can do that in bulk by just throwing that moles at it.
I feel like the researchers are
tuning to the machines that we're able to
build to the extent that NVIDIA gives
us awesome fine-grain for sparsity.
The researchers then pick that up and running with that.
We will get to do better efficiency.
I think that it's not over.
I think as engineers,
there's the frustrating stuff.
What can we do right now with the machines we have?
Then as scientists, what do we
wish we were doing for better learning algorithms,
let better model structure,
and better computer architecture?
All that is on the table.
Speaking optimistically, I'm really enjoying the ride.
Absolutely, yes to what you're doing.
Please do more. Like if you build
the sparse and revolution that makes
all of our machines obsolete.
Because I think Robin,
I might be a little sad about
our business model disruption,
but we would get to build cool new machines.
Keep us in business, change the algorithms.
The worst thing for a computer architect
is for the algorithms to stagnate.
But happily, that's not what's happening.
The algorithms are evolving
just as fast as the architectures.
Thank you and by the way, great Dan.
Thank you.
If I can just interject for a second and yeah,
Cliff, that was actually really good.
I think there's way more we can do about
efficient architecture and improving things but
the evolution of the neural nets is so radically fast.
I'm not kidding when I say the state of
the art changes dramatically every six months.
The evolution is so rapid,
it's really hard to spend
the energy on those optimizations.
Nonetheless, I think as an industry we
do spend a huge amount of energy on that.
I think it's hidden by the advancing.
When the demand is increasing 10x per year,
it's hard for anybody to
notice the efficiencies you're baking in.
[MUSIC]