>> Welcome to this episode of The AI Show where we
talk about GreenerAI with Will Buchanan.
[MUSIC]
>> Hi Will, welcome to the AI Show, and can you
introduce yourself and tell us what you do at Microsoft.
>> Hi, everyone. My name is Will Buchanan.
I'm a PM on the Azure Machine Learning Team,
specifically focusing on user experience
and decarbonization of AI.
>> That is great. What are
some of the problems that you're solving?
>> I've been working on something called
GreenerAI which has become
a recent hot topic in the news and it's really
specifically focused on providing
a set of tactics that data scientists
and developers can use to reduce
the carbon footprint of their machine learning workloads.
>> That is great and very timely indeed.
>> It is. We're out of time in fact.
>> Can you tell us little bit more about these services,
the GreenerAI services, and what we do?
>> Certainly. Within Azure Machine Learning
we have launched a set of
resource metrics which provide transparency into
the computational costs as well as
the energetic costs of your workloads.
Really we're trying to provide transparency
to help empower developers to make decisions
about when and where they should be running their models
and how to optimize or make
decisions which would be more environmentally friendly.
>> How can developers do that,
make more informed decisions?
>> The first step is transparency around those metrics.
We're providing those metrics to help
users understand the costs of the models.
But soon we also have a set of
recipes or samples that any practitioner can
use and start understanding the savings of their workloads.
I can walk through that in a second.
>> Great. We would like to see it.
>> Sure. Before I dive in I will give a bit of context.
>> Go ahead.
>> I've been leading an initiative called GreenerAI at
Microsoft and it has about 200 people in it across the company,
and the reason we're doing this is
because you really can't manage what you can't measure.
We're showing a set of early research and
innovation that's happening across Microsoft.
We're going to talk about some new capabilities
as well as a few different use cases and
validation tactics and then a few tips on getting started.
In terms of where we are today,
we're in a state of something known as
red AI and we've created a computational mega fauna.
You have state of the art approaches that are buying
incrementally better results with high hidden costs.
Computational costs have increased in machine learning about
300,000 times from 2012-2018.
According to an article by Wired
only about 11 percent of firms are
seeing what can be called a significant
return on investment for their workloads.
It's pretty shocking when you think about that.
Some recent studies have shown that
the GPT3 training emits as much carbon
as three round-trip transcontinental flights
from San Francisco to New York.
Emma Strubell about a year or two
ago also published a seminal paper which
compared a transformer model with
a round-trip flight or a human life or US car,
so 213 million parameter transformer model training
consumes as much carbon as five cars including gas.
As you can see in this chart,
we have surpassed what
could be seen as Moore's law and we're in more of
the extreme modern era of
computation where it's doubling about every 3.4 months.
The AI market is going to grow
about 44 percent annually through 2025.
We believe that there will be a point of
inflection where there are diminishing returns where
the marginal cost-benefit ratio
are no longer worth the additional investment.
A bit of background on AI pipelines.
It consist of both training and inference.
In training you're teaching a model of how to make
predictions based on existing data and
generally you're doing long runs and a few of
them and then when you go to scale you
infer your model and you are applying
this trained model to make predictions
from data that it has not seen before.
Generally, it has short runs and lots of them.
Emergently, there's specialized hardware such as GPU which is
optimizing your workloads there and they're often power hungry,
consuming about 300 watts on average.
As I mentioned earlier it's pretty hot topic
in the media these days.
I had a chance to chat with Timnit Gebru who
published this stochastic parrots paper
and she's really saying how it is
past time for researchers to prioritize
energy efficiency and costs so
that we can reduce environmental impact and
increase equitable access to resources.
You can think of the AI industry in
some ways as similar to the oil industry.
Once you mine and refine your data,
it can be highly lucrative and it seems that
this metaphor could extend even further.
The way that we're addressing this is
through a cost metric framework.
In terms of the costs we want to expose both monetary,
the computational runtime,
the energetic cost,
as well as utilization for both training and inference.
It's important with training because only about 12 percent of
the models that you're making actually make it into production.
For inference according to an article by NVIDIA,
it accounts for about 80-90 percent of the carbon cost of model.
The vision that we're working towards is
an operational lifecycle analysis monitoring toolkit.
You can assess these different cost frameworks
for both training and inference and
then set a set of triggers or
tools you can start to optimize your return on investment.
The first capability that we've done,
you can see in Azure Machine Learning Studio.
In this interface, you can add an opt-in metric to
see the GPU energy cost for your runs.
This helps you find the most energetic culprits
and then you can discover which ones were the most expensive.
We're also offering the same capabilities for
both utilization GPU and memory.
Next, you can see
some additional screenshots in
the monitoring tab which is in preview.
You can view a real-time plot of your GPU energy usage broken
down by node and you can also see CPU utilization,
GPU utilization, and memory,
and of course energy.
We've done a study where you perform a batched inference run
and we calculated the cost
per million adults which is a standard framework for inference.
It consumes about 13.8-kilowatt
hours and that's about half
of a US home's daily energy consumption.
>> A lot.
>> It's crazy.
>> It is.
>> If you were to convert that to greenhouse gas equivalencies,
they'd be about the same as 24 miles driven or
11 pounds of coal or a gallon of gas.
You can imagine this really adds up over time.
The analysis we did was for
the BERT model using a Tensor-flow framework on a V100
with a 32 floating-point precision and batch size of 16.
We ran it for 2.5 hours and we
calculated the throughput as well as
the energy cost and then we extrapolated that
for a million inferences.
As part of our work we've been working
with NVIDIA on something called
the Triton Inference Server and this is
specifically designed for high throughput inference workloads.
It comes with a number of benefits
such as model concurrency where you can load
multiple models or copies of that model
onto a GPU and then execute them simultaneously.
Another exciting feature is dynamic batching where you
can dynamically group together
your requests on the server-side to maximize
performance and mixed weights precision.
Due to the vanishing gradients phenomenon during your training,
a backward pass,
your models you typically trained using
floating point 32 precision.
However, given for inference, only the forward pass is necessary,
you can use a lower precision such as
16 and Triton allows use of both of these.
On a similar analysis, we found that if you
use mixed weights precision you can cut your costs by
a greater than 50 percent just by switching from
floating point 32-16 using the Triton Inference Server.
You can see the cost per
million inference is significantly less, it's almost a third,
GPU power consumption is from about 75-60 watts.
>> This is great.
>> Something else we've also been working
on and will soon be releasing,
there's work with the NVIDIA Model Analyzer.
You can analyze and develop the optimal configurations for
your model and we also charted
the relationship between power efficiency and GPU utilization.
If you use Model Analyzer to optimize
your configuration you can reduce your cost per
million by about 12.6X.
Here you see the chart with the baseline there.
The baseline is compared against ORT and CUDA as the backend
versus TRT backend optimized by Model Analyzer.
Using the Model Analyzer can automatically maximize
both your GPU utilization and
your throughput and consequently save energy.
Finally, an additional case study that we've done.
For image classification scenario for training,
we compared two different convolutional neural networks
, InceptionV3 and DenseNet.
Both of these classify images with
greater than 75 percent accuracy on your ImageNet dataset.
We compare these two models for both energy usage
and cost savings to help determine which models
give you the results you need at
the lowest costs both monetary and environmental.
We found that InceptionV3 outperforms
DenseNet in terms of accuracy coming in about 10 percent higher.
Also cost about 13 percent less and uses
20 percent less energy and about 10 percent less training time.
Really key to this and this is the foundation of
the tactic is choose the right tool for the job.
Start to think about the networks that you are
choosing and the cost that incurs for your business.
>> You're not only saving the environment
but you're also paying less which is great.
>> Yeah, I agree. It's the real business case for this work.
We're just getting started doing GreenerAI and we hope
to bring additional people in to work on this.
We'd like to work with any customers to define what
carbon awareness and energy efficiency
might need for their organization.
Specifically we'd like to point you to
the Triton Inference Server to get started doing that.
Also, we will share some recipes and samples
to help compare these cost-benefit trade-offs.
I did want to throw in a plug for
the Green Software Foundation where we're starting to define
this concept called a software carbon intensity specification
and that is currently the Alpha version is live.
If you look at Green Software Foundation on GitHub,
you will see some really exciting work there.
That concludes my presentation.
>> Great. Thank you so much.
This is really exciting.
Where else people can learn more about GreenerAI?
>> We have a blog on the tech community
which outlines some of this that I showed here,
specifically the resource metrics.
Also, I believe we're posting
the link to the Triton Inference Server.
I think that's the best starting point,
but stay tuned we'll have a lot more exciting news soon,
especially with the Green Software Foundation.
>> Thank you for joining Will,
and thank you for telling us about the GreenerAI.
We'll see you next time.
[MUSIC]