(soft music)
- Hello everyone.
My name is Nidhi Chappell,
and I lead the Azure HPC and AI product team.
Today I'm excited to talk to you about our AI strategy
and what Microsoft is doing to enable AI at scale
for all of our customers.
With the exclusion of data
and availability of large cloud compute
the AI community has made significant strides in many areas
like speech recognition, computer vision,
natural language understanding.
Over the last many years,
we have seen the emergence of a new training approach
called self supervised learning,
where AI models can learn
from large amounts of unlabeled data.
These models are able to learn on their own.
For example, in the space of NLP,
the model can learn and understand language constructs
by reading immense amount of data
that is available on the internet.
The more data it reads,
the better it understands how words relate to each other
in a certain context.
Besides just the data, it is able to read
the size of these models, determines its ability to learn
and In Code the complexity of the language.
In February, Microsoft announced Turing,
which is a 17 billion parameter language model
that outpaced the state of the art
on many downstream NLP task.
In Microsoft, we are using these massive models
to build intelligence solutions
into a wide variety of products like Xbox, Bing, Office,
enabling new experiences that were not possible to allow.
What's immediately apparent is, to be successful in a world
where the requirements of training such large
state of the art AI models are growing so much faster
than the capabilities of the raw hardware available
to AI researchers on premise,
we need a dramatically different approach,
a super computing approach like no other.
And this is precisely our vision at Microsoft.
We want everyone to have access
to the same state of the art AI tools,
whether from Microsoft or from outside,
and have access to big clusters of machines,
of specialized AI accelerators
that are interconnected by high memory bandwidth networks,
both inside and outside the machines.
These AI supercomputers are what we are building in Azure
and are now available to all of our end customers.
Our Azure AI supercomputers rival
the top five supercomputers,
and are really accelerating the pace of innovation
outside of the supercomputing centers.
A point in case is OpenAI.
As you know,
we have been partnering very closely with OpenAI.
They recently released a new model called GBT-3,
which is trained on Azure supercomputer infrastructure.
GBT-3 has really pushed the limits of NLP models
with 175 billion parameters.
And by doing so has opened many new capabilities
that were not possible before.
Today I am joined by Christopher Berner,
the head of compute at OpenAI
to talk about the experiences with Azure supercomputing.
Chris, thank you for your partnership.
Can you share some insights into opening your eyes vision
for GBT-3?
- Yeah, certainly
I'm really excited about GBT-3
because I see it as a big milestone in language models.
It, of course it was one of the largest
and most capable language models that's been trained.
And one of the things that I find most exciting about it is
its performance on zero shot tasks,
meaning how well it does on a task
that has never seen before.
And it hasn't been trained to do well on.
And this is an ability that I feel like
unlocks a new way of using AI and language models.
And we've in fact started building an API around it,
which is open to developers everywhere.
By using the API developers are able to easily
integrate GPT-3 into their applications.
And it also allows us some degree of control and monitoring
of GPT-3 to make sure it's not misused.
- What do you think were some of the key challenges
that you had to overcome,
when you were designing for models at the scale?
- One of the major challenges
that we faced training GPT-3
was getting all of
the distributed training infrastructure, right?
We've been using distributed training approaches in AI
for quite a while, but for the most part,
that's been data parallel approaches
and given the sheer size of GPT-3 at 175 billion parameters,
we needed to develop new model parallel approaches
to train it.
The reason that this is required is that
even if you used only 32 bits of data
for each parameter in GPT-3,
that would be 700 gigabytes of data.
And it doesn't even count temporary variables
for things like activations or training data inputs.
- Chris, can you talk about some of
the infrastructure challenges that you had to overcome?
- Yeah, for sure.
I'm happy to talk about the infrastructure challenges,
because I think that this is an area where
the challenges are, can be a little bit surprising, even.
There's been a lot of focus in the industry
on AI accelerators in the last few years,
but I feel like the focus has been too heavily
on the chips themselves and not enough on
the cluster interconnect and the software that's required
for really pushing AI at scale to the next level.
And the reason that I say that is because
these really large language models
and really any other type of AI at scale,
they're becoming much more like a classic HPC workload,
where you have a single synchronous job
running on thousands of GPU's.
And in our particular case of GPT-3
we saw a really big benefit from switching
to an InfiniBand interconnect.
And this is something that we built in
our new Azure hybrid supercomputer.
And we got well over a 50% improvement in performance
and just out of the box.
Going and making further algorithmic improvements
is going to lead to even bigger gains.
But right out of the gate, we saw a big speed up there.
And then on the software side of things,
it's similarly very challenging
as these jobs get bigger and bigger, you run into,
you know, all of the classic
distributed systems challenges around
needing to solve fault tolerance,
issues of tail latency and stragglers and everything else
that comes along with building a distributed training system
that runs at the scale of hundreds or thousands of servers.
Beyond NLP, another area
where there are really interesting infrastructure challenges
is in our RL workloads.
For example, we have a system that we built
and used for some of our projects
like Dota and our robotics work
on solving a Rubik's cube with a robotics hand.
And our all workloads are really interesting because
they have a significantly different training paradigm
than in the kind of supervised or unsupervised
training approaches that are used in NLP and other tasks.
And the main differences that from a systems perspective,
you don't have just a single tier of GPU servers.
Instead you have the GPU servers that are doing optimization
just like in any other workload,
but you also have this tier of machines
that are running the environment.
And in the case of our robotics projects,
that might be a physics simulator,
which can be extremely compute intensive on its own.
And so when we built our hybrid supercomputer with Azure,
not only did we have 10,000 V100's,
but we also built a very large cluster of CPU servers
using the HBv2, a skew that has AMD RAM cores.
We have a quarter million CPU's there
because of the number of RL environments
that we need to run.
A single job might use 100,000 CPU cores just for running
physics simulators.
And so you have the setup where the GPU tier of machines
are doing optimization,
sending actions to the physics simulators,
and then getting back responses as to what happened,
which then drives the next iteration
of the optimization loop.
So to tie that back to what I was saying earlier
about the main challenges being in cluster interconnect
and software stack in both the case
of reinforcement learning workloads
and training these really large language models,
you need this huge distributed cluster and then have
a very challenging software system that needs to get built,
to handle all of the full tolerance
across thousands of GPU's
plus potentially 100,000 or even more CPU cores.
- Chris, can you talk about some of the examples
of use cases that are now possible for developers
using GPT-3?
- Before I jump into that all just wrap up
another thought about the infrastructure and challenges
of bringing up this hybrid supercomputer,
'cause I think that's a fun story too.
The whole cluster really came together in
less than a year's time.
And I thought that was a great show of the partnership
between Azure and OpenAI.
And how we were able to bring people
from a classic supercomputer and HPC background
together with our research scientists
and build a cluster that was so well optimized
for AI workloads.
And the result of that, GPT-3
and all of the innovative things
that our developers have come up with,
have just really amazed me.
I feel like every week are I see innovative things
that developers have done with the API.
Some of my favorites have been things like
interactive storytelling and also code generation
coming from a background as a programmer,
I find that very exciting where people have built JavaScript
and react app generators,
where you just put in the name of a component or function
and it can auto-complete dozens of lines of JavaScript code
and HTML.
So things like that just really amaze me.
- This is so exciting, Chris, what's next for OpenAI?
- I think what's next and what excites me is that
this feels like just a whole new way
of interacting with computers.
You know, it used to be that we, uh,
wrote assembly or C or whatever,
and you would have to use that to talk to a computer.
But now with these new language models,
you can just interact with them in natural language,
just talk to them in English and
the computer can then continue generating your,
poetry or homework assignment or whatever.
And I'm sure it's a much better poet than I am.
And so I see this really as
the next step in our interaction with computers
and how this will empower not only the developers
to be more productive,
but I really see this as potentially unlocking
the next 10 million or 100 million developers
and poets and writers and allowing people to do things
that they couldn't have done on their own
by having this new way of just collaborating
with a computer
and integrating it into your creative process.
- Thank you, Chris.
We are proud to show our partnership with OpenAI
and believe that AI in the right hands
has the potential to change the world for better.
And that's why we are committed to making
the same fundamental hardware used by OpenAI at scale
of thousands of GPU's available to any Azure customer.
Today, I'm excited to announce
that we now have NVIDIA A100 GPUs
widely available for all of our customers to use.
The largest A100 cluster in Azure
features more than 6,000 GPU's
each with a fully subscribed 200 gigabit per second,
InfiniBand HDR link,
which means a staggering 1.2 terabits per second
of bisexual bandwidth,
not to mention the ability to do GPU direct RDMA
at full PCIe Gen 4 speeds
paid with the latest AMD Rome processors.
Each Azure supercluster would rank easily among
the top 10 supercomputers on earth, if measured today.
Innovators like OpenAI
or whoever will be behind the next breakthrough in AI
cannot accept substitutes
as we race towards the next breakthrough in healthcare,
clean energy, education, or for that matter,
just picking a movie the whole family can agree on.
So we are here to turn the problem of doing AI at scale
into a real deployable solution.
Thank you.
(upbeat music)