(music)
- Ever wondered what it takes to run the infrastructure
that hosts ChatGPT with over 100 million users today,
and how this infrastructure can work
for your workloads in Azure at any scale?
Well, keep watching as we break down
the AI supercomputer infrastructure built to run
ChatGPT and other large language models.
And joining us to go deep on all the details,
is Azure CTO Mark Russinovich.
Welcome back to Mechanics!
- Thanks for having me back on the show, Jeremy.
It's been a long time.
- It's been really, really too long,
so thanks for coming back.
You know, especially now,
we're seeing a major shift in technology
with AI and large language models.
Almost overnight, these have now defined
a brand new way of interacting
with your apps and data.
And this really includes Microsoft services
from things like Bing chat
to Copilot content creation experience
across the Microsoft 365 portfolio of apps,
natural language coding with GitHub Copilot
and a lot more.
And now, these large language models,
they're all running on Azure.
But for this to be possible
there had to be some significant breakthroughs
in our AI infrastructure, right?
- Yeah, well we've been working
on AI at Microsoft for decades now,
but there's certainly been a huge acceleration
in AI capability over the last 10 years,
driven largely by the rise of GPUs
and cloud scale infrastructure.
And we've contributed to that in Azure.
Over the last few years,
we've built a specialized hardware and software stack
to support the training inference
of large language models or LLMs.
Our AI supercomputer in Azure can train models
that are hundreds of billions of parameters in size.
The original GPT-3 was 175 billion parameters,
but we built larger models than that since,
for example Microsoft's Megatron-Turing
natural language model, has 530 billion parameters.
And as newer generations of LLMs are released,
they're going to be even larger.
The thing we had to support with LLMs
like GPT is their self-supervised learning,
where they learn about a language
by examining billions of pages of information
over and over again.
That directed how we approach building
a specialized hardware and software stack
that can support the efficient running models
of that massive scale.
- Right, and the challenge with something like this
is that they're super resource intensive
and also pretty expensive to run.
- An efficient infrastructure for running things
at that size is really critical.
You have to deal with failures
that are happening on a regular basis.
Every few days, and even sometimes more frequently
when you're at large scale,
something's going to fail.
Like a server's going to fail
or a network link's going to start to flap.
So you've got to be able to minimize
the rate of failures.
When one happens, you've got to be able
to diagnose it very quickly
and then you've got to be able to fix it very quickly.
A lot of things had to come together
to build a hardware and software stack
that could support that kind of scale.
We've invested a lot in our datacenter infrastructure,
with state-of-the-art hardware
that we run in Azure globally for raw compute horsepower.
One of the key things we do is cluster GPUs
with a high-bandwidth network
to provide connectivity required to efficiently train them.
We layer on top of that Azure software
platform optimizations where we have for example,
enlightened the platform and the hypervisor
to be able to access those network GPUs
so that the performance is comparable
to running on bare metal,
but with a full manageability layer on top.
As we move up the stack,
we've also developed and integrated open-source frameworks
like ONYX for model portability.
And the DeepSpeed framework that came
out of Microsoft Research that helps
componentize these models so that they can train
across interconnected GPUs at greater scale and speed.
In fact, DeepSpeed has become the defacto framework
for distributed machine learning training
- Right, let's just put this into context,
because for the OpenAI service,
the supercomputer that we built for OpenAI
back in 2020 comprises more
than 285,000 AMD InfiniBand connected CPU cores.
And separately there are 10,000 NVIDIA
V100 Tensor Core GPUs that are also InfiniBand connected.
- Right, so one of the techniques used
to train those models is called data parallelism.
Where you're actually training many, many instances
of the model at the same time on small batches of data.
And after each batch is processed, the GPUs exchange
information and then proceed to the next batch.
And that's why you require systems that are that large.
By the way, that system that we built back in 2020,
was the fifth largest supercomputer in the world
at the time and the largest supercomputer ever built
in the public cloud.
It's really the result of a collaborative effort
between Microsoft engineering, Microsoft Research,
the OpenAI organization, and NVIDIA.
- One of the more impressive things
that I've found with this is just
how we've been able to network everything together.
And I know that we've actually used InfiniBand
to achieve this, but what else did you have to do
to optimize the hardware side?
- Well, we've actually been using InfiniBand
for a decade in our HPC clusters.
It has better cost performance than ethernet
and Azure is unique compared to every other cloud for that.
It's one of the key reasons OpenAI actually came to us,
we've been battle testing this for many years.
And of course, we've since built even bigger systems
than the one we talked about publicly back in 2020.
We collaborated with NVIDIA to deliver
purpose-built AI infrastructure within NVIDIA GPUs.
The multiple GPUs required to train LLMs
don't all operate on the same board
or even the same rack.
They're spread out due to the power consumption and heat,
and we want to network and cluster
as many GPUs as we can.
And the new H100 VM series in Azure,
powered by NVIDIA H100 Tensor Core GPUs,
lets you choose one or more GPUs,
cluster up to eight GPUs per VM using NVLink,
and scale out to thousands if your demands grow,
using NVIDIA Quantum-2 InfiniBand networking.
That's up to 30x higher performance in inferencing
and 4x higher for training compared
to NVIDIA's previous A100 generation of GPUs.
These are designed specifically
for multi-tenant cloud compute,
giving us the ability to isolate customers sharing
the same server from one another,
and with that we can achieve the elastic scale that we need.
- Right, and it's also critical for something
like this in training the models of this size,
but I'd like to go into throughput.
So how much have we improved the throughput
then for our GPUs?
- Yeah, so from a throughput perspective,
starting with a single node using NVSwitch
and NVLink 4.0,
bisectional bandwidth can reach 3.6 Terabytes per second
between eight local GPUs.
And to put that in perspective, at that speed,
you could download about 80 4K videos
in less than a second. - Just wish I could do that.
(both laugh) - Everybody does.
Then as you scale out with more nodes,
using a non-blocking fat-tree network architecture,
the InfiniBand links deliver 400 Gigabits per GPU,
which is a total of 3.2 Terabytes per server.
- Okay, so we've got the raw horsepower
then in the hardware,
but now as a service provider,
where we host these models in Azure,
how would we make sure that we can run everything
as reliably as we need to?
- Well, like I mentioned, models this size,
they'll typically run for days, weeks,
in some cases months for the largest model sizes.
So, you don't want to have to restart training
from scratch and lose progress if something goes wrong,
like if a server fails or a network link starts to flap,
which will happen on a regular basis
over that kind of time period and that kind of scale.
And the way we address that in Azure
is with a containerization and global scheduler service
that we've been working on called Project Forge,
which is designed specifically to help run
Microsoft's global scale AI workloads
and maintain really high levels of utilization.
Project Forge introduces transparent checkpointing,
where it periodically saves
the state of a model incrementally,
without the model's code needing to do anything.
That way, if anything fails,
it can quickly resume for the most recent checkpoint.
We combine this with our integrated global scheduler
that pools GPU capacity from regions around the world.
So, if you need to pause a job to prioritize another one,
that allows us to migrate that pause job
to another region if necessary and available,
with minimal impact on its progress.
- So, let's you kind of load balance
all the different jobs we have at global scale.
- That's right, so for example,
we can load balance between Bing chat
or Copilot services and OpenAI.
The Global scheduler provides its single
GPU capacity management to ensure resources
are not idle with fixed clusters,
instead jobs are assigned to virtual clusters
and then those virtual clusters get mapped
to physical resources, when they're available and needed.
- And this is all really pivotal in terms
of being able to run these jobs uninterrupted,
so is this something that the people watching,
Azure customers, can use right now?
- Right now, Project Forge is being used for own services.
Like for example, if you're using GitHub Copilot,
you're using Project Forge under the hood.
But we're working on making it directly available
to customers in the near future.
I've got a Jupyter Notebook that has
a machine learning model that we pulled off of GitHub.
It's open source,
it has no model checkpointing,
as part of it.
I've connected that Jupyter Notebook
to Project Forge's special Python kernel
and I've executed the first cell,
which initializes the GPUs.
So on the right side I've SSH'd into the GPU server
where you can see the GPU's utilization
and its memory consumption.
And you can see that utilization is zero,
there's a minimal amount of memory consumption,
but now I'm going to kick off that training job
and when I do, you see the memory usage spikes,
you can see a little bit of GP utilization.
It's still pretty low though,
because this model's a tiny model compared
to the amount of capacity that's on this A100 GPU.
So we're going to train the model
which just takes a short period of time
and now the GPU is going to go idle.
And at that point Project Forge sees that it's gone idle
and automatically takes a checkpoint,
because it knows that I'm not busy using the GPUs.
Normal circumstances, I would've been hogging
the GPU at that point and preventing it
from being used for somebody else's workload.
But now that GPU can be repurposed,
you can see there's the checkpoint,
memory usage goes to zero,
GPU utilization goes to zero.
Now when I come back though,
if I've got priority,
or if the GPU is available, a GPU is available,
Project Forge can reload the checkpoint,
and so that's what I'm going to do.
And I can pick up exactly where I left off in that notebook.
So that next cell there,
depends on the training run I just did
and I can execute it and it'll succeed there,
because Project Forge is behind the scenes
connected to GPU backup with the kernel.
- And just to reiterate,
you didn't have to modify your code
or manually execute that checkpoint,
it just ran in the background?
- Completely transparently. - Perfect.
- And something that will help make that possible
is our work with our hardware partners,
AMD and NVIDIA which helped implement CRIU,
or Checkpoint/Restore In Usermode for their GPUs.
Which enables checkpointing and restoring
the memory in the GPU and making that consistent
with the checkpoints we do on this CPU.
- Right, and these are all really key areas
in terms of where our software and hardware
all work together to speed things up for greater efficiency.
So, are there other examples that you've worked on?
- Yeah, another great example of that
is what we do for the process where you want
to customize one of these foundational models
for specific domain or for a custom data set.
So for example, healthcare data or your enterprise data.
And that process of customizing it
is called fine tuning.
We can make it great at knowing how to program, for example.
- Right, it's kind of like in the movie the Matrix
where Neo learns Kung Fu.
- That's right, would you like a pill?
- Gimme the red one this time.
- You took the blue one last time.
- I did.
- I mentioned that GPT-3 model that at one point,
which has 175 billion parameters.
So, if you just fine tune that
using the basic fine tuning technique,
it would require 96 GPUs.
And each checkpoint during that fine tuning training
would be about a Terabyte in size.
To make that more efficient,
we focus only on the parts of the model
that need to improve by using a technique we developed
in Microsoft Research called Low Rank Adaptive,
or LoRA fine-tuning.
And then only that part of the model
is updated with changes.
With a model like GPT,
you would only need to fine tune
far less than 1% of the total parameters.
So instead of needing 96 GPUs,
you might only need 24.
And instead of a Terabyte size checkpoint,
it's not only 200 megabytes in size.
Because we can quickly switch LoRA fine tune parameters
on top of a loaded base foundational model,
we can also optimize serving lots of fine tune models
on top of a smaller number of GPUs.
- That's really a big part in terms
of how we've been able to apply large language models
into our own services at Microsoft.
On the other hand though,
for everyone who's watching,
how can everyone use Azure's supercomputer capabilities
for their own workloads in Azure,
even if maybe they're coming from a smaller workload or job?
- Well, our AI infrastructure supports everything
from very small jobs like you saw.
Pretty tiny job right there,
to very large ones like on OpenAI scale.
So if you just want to take advantage
of our optimized hardware infrastructure,
you can of course just use the virtual machines
and do it yourself,
taking advantage of our hardware infrastructure,
the best in class GPUs, the InfiniBand networking,
the fact that we know how to tune those networks
to maintain them with a high degree
of utilization and efficiency.
But if you want to focus on building and running models,
you can take advantage of our frameworks
and ML Ops services.
So for example, you can use our ONYX runtime
that I mentioned, or use our DeepSpeed training framework
for more efficient distributed training.
And you can use Azure Machine Learning
PaaS services that natively integrates
with our AI infrastructure to build your models.
And with AML, you can easily fine tune models
like GPT-4 and deploy the fine tune models
as managed services for your own applications to use.
- So, how are we seeing organizations today
that leverage Azure's AI Supercomputer
for their own solutions?
- Well, there are a lot of really interesting
use cases where customers are using training
and inference on top of our large scale infrastructure.
One of them that's one of my favorites is Wayve.
It's a company in the UK that is a leader
in autonomous driving technologies.
So what they're doing is specializing
in distributed AI-powered autonomous systems
for self-driving vehicles using primarily
vision-based machine learning.
And they're using our AI supercomputer
to gather, manage, and train those models
on millions of hours of driving data per year.
That's petabytes of data,
consisting of images, GPS data, and data from other sensors,
each individual training sample can be
hundreds of megabytes in size and by comparison,
a training sample for a LLM is typically
just a few K in size.
So, there are also training models
with billions of parameters and use
Azure Machine Learning and AKS
for large scale distributed deep learning
and orchestration.
And that's key because Wayve,
requires a lot of linear scaling,
optimal memory utilization and a fast GPU interconnect
with minimal latency with high throughput.
And that's where we excel.
Right, and Azure is literally the best place
to run something like this.
But let's switch gears though,
because last time you were on,
we actually covered Confidential Computing,
and this is something that you've helped actually develop
as part of the solution set in Azure.
So, there's a lot more importance here
with Confidential Computing with AI workloads, right?
- That's right.
When you talk about AI workload,
you're talking about the data you use to train them
which can be very sensitive data.
The model that you train actually can have a lot of IP.
Obviously OpenAI has a lot of IP in their models
and so do many other customers, like Wayve,
have IP in their models.
And then finally, for the consumers of those models
like people that are using Azure OpenAI,
when they send their queries in
that data is also very sensitive.
So with Confidential Computing,
you can protect all of that data end-to-end
in a trusted execution environment,
which is provided by hardware
with a hardware root of trust
that only you have access to.
That means the data in memory is protected and encrypted.
Data is only in the clear while running inside of that TE,
and it's re-encrypted and unreadable when it leaves that TE.
That prevents third parties and processes,
including Azure operators,
from accessing the data and models.
That enables confidential multi-party collaborations
in what we call confidential data cleanrooms,
where parties can combine their datasets for training
and each party is unable to access anyone else's data.
In the example of autonomous vehicles,
that would allow multiple vehicle manufacturers,
GPS providers, and AI developers to work together,
to share the data needed to build
intelligent self-driving solutions.
And confidential GPU is an area
where we're investing heavily in today
with NVIDIA and other hardware partners.
We announced at GTC recently,
that NVIDIA is coming out with confidential hopper GPUs.
- Right, so this is something that's evolving quickly
this whole space, so what's next then on the horizon?
- Everything we do to build these systems
pays dividends to anybody using Azure,
whether using AI or not.
It means they're going to have lower cost,
higher scale, more efficient data centers,
more powerful software frameworks.
Right now, in the area of AI,
we're in the process of rolling out those NVIDIA H100 GPUs
for AI and HPC, and you're going to see us make
Project Forge available for everyone to use
as you train and run your AI models
like I mentioned earlier.
And there's lots more innovation,
on an ongoing basis to always make sure
we're best in class to run and train your AI workloads.
Really such a great deep dive, and I learn so much
when you're on the show.
Thanks so much for joining us today.
And by the way if you want to leverage
Azure AI capabilities for yourself,
you can start now by using our GPU enabled compute
in Azure along with the Azure Machine Learning service.
And there are lots of options available
to get started really at any scale.
So check out aka.ms/AzureAIInfrastructure
and also keep watching Microsoft Mechanics
for all the latest updates.
Thank you so much for watching
and we'll see you next time.