- Hello everybody, thank you for having me.
My name is Sharan Chetlur.
I am a Principal Engineer working on DL products,
here at Nvidia.
In particular, I work on the NeMo Megatron framework
for large language model training and deployment.
Hello all, my name is Vijay Aski.
I'm a Director of Engineering on Azure Air Platform
in Microsoft, and today we are gonna talk
about language models at scale,
also known as foundation models.
Foundation models are a new paradigm for building AI systems
that leverage massive amounts of data and compute power.
They're built using a technique
called self-supervised learning, where the output labels
are part of the input data,
thus no separate out labels are required.
It's also known as predictive or pretext learning.
For example, you can mask some words in a sentence
and ask the model to predict them.
Foundation models are really good
at many standard ML tasks out of the box,
and they can be adapted for many downstream tasks
by fine tuning them on specific data sets
or adding task specific layers.
For example, you can use GB3 or text generation
or word for text classification
or codex models for generating test cases.
Foundation models though,
are not guaranteed to be accurate or consistent
across different domains or scenarios,
which brings in its own challenges.
The models do not have a deep understanding
of the semantics or pragmatics of natural language.
They instead, rely on statistical pattern
and superficial cues, rather than logic or reasoning.
These models inherit biases
from the training data sources, for example,
that will generate are images
that reinforce stereotypes held by the images available
for the models to train.
These models are also extremely expensive
and computationally intensive to train,
and the cost of deployment is very high.
The foundation models also bring in challenges,
such as fairness, safety, and governance.
That said, you can still a lot done with foundation models
compared to traditional ML techniques,
and experimentation is the key way
for you to get the best out of foundational model APIs.
- As Vijay said, running huge models at scale
can be a real challenge,
but one that's immensely important to the future of AI.
This is clearly where the world is headed.
To help with this, we built NeMo Megatron,
a toolkit which addresses end-to-end,
the needs of training and deploying large language models.
NeMo Megatron includes utilities and libraries
for degradation distributed training at a variety of scales.
P tuning and fast multi-node inference.
NeMo Megatron allows you to go easily from model definition
to efficient execution on thousands of GPUs,
by abstracting away the complexity
of running this family of workloads at scale.
Let's dig a little bit deeper
into where some of this complexity comes from.
We're all familiar with data parallelism,
the practice of scaling out
by partitioning the mini-batch across workers,
but LLMs are specially characterized by models
that are orders of magnitude larger
than what you can accommodate in your GPUs memory.
Data parallelism is simply not enough.
NeMo Megatron equips you with sophisticated techniques
to overcome the limitations that this problem imposes.
Broader topic as what I covered is it is just one facet
of what this new generation of AI demands.
There's so much more that goes into training
and deploying an LLM and NeMo Megatron supports all of it,
from data curation to deployment.
- That's fantastic!
In addition to all the great work that Nvidia
has done that Sharan walked through,
let's walk through the optimization approach
we have done in Microsoft, where we focused on full stack
from hardware to the user-facing Azure ML APIs
to ensure that all pieces work together seamlessly
for excellent performance and scalability.
At the bottom of the stack,
we have the latest in Nvidia GPUs
with Quantum InfiniBand interconnects,
which enables high throughput for both single node
and multi-node training scenarios.
On top of the hardware, we use Azure HPC Ubuntu
images that have been thoroughly tested
to match the performance specification
of the hardware vendor.
To ensure that you only pay for the resources that you use,
we have built the Azure ML compute fabric,
which automatically scales resources as needed,
while running a job.
It also supports training at cloud scale
with frameworks of its choice such as PyTorch,
TensorFlow and others.
We also optimize the training containers,
which we call curated environments,
to provide a fast and easy way to get started
or to customize for specific training needs.
For large distributed PyTorch workloads,
we have created the Azure Container for PyTorch,
which includes the most commonly used frameworks
and libraries for efficient large scale trainings.
In the context of large training models,
let's take a quick look at what is currently available
in Azure that is state-of-the-art,
in terms of hardware capabilities.
The current flagship hardware skew we have available
in Azure ML is the Nvidia 100 V4,
supporting Nvidia and PR architecture.
This brings in eight Nvidia A100 GPUs per node,
each with 80 GB of memory.
Within the same node, pass communication is enabled
through NVLink and NVSwitch and cross nodes
we have the 200 GB InfiniBand links.
Now let's take a look at various software optimization
Azure ML provides for the monitoring journey.
Azure ML provides many built-in training environments
that are optimized and validated against Azure hardware.
They can be used as is or can be customized
for particular training workload.
After the prototyping and training experimentation phase
using the training environmental scripts,
we have to look at scaling the model building process.
For this, Azure ML provides support
for different compute fabrics,
from the built-in compute clusters
to attach communities clusters.
We realize that the errors that come up while scaling
are more expansive to debug effects, and for this reason,
we provide multiple tools to speed up debugging experience,
like SSS support, VS code debugging
for the running containers in the cloud,
native integration for Tensorboard,
and also PyTorch and tensorflow profilers.
On the save aspect of ML debugging,
Azure ML supports ML flow tracking for model metrics,
comparisons across experiments,
and enable customers to get visibility
into hardware-proof metrics, like CPU, GPU,
and network utilization.
Having the visibility into these metrics
and the tools to be able to correct inefficiencies
is critical for large workloads
to keep the costs under control.
Some other cost saving optimization that are developed
at Azure ML are the ability to leverage low priority
and spot virtual machines
that can potentially bring down training cost by up to 80%,
making use of efficient model checkpointing
and recovery leads to much better GPU utilization.
After model training is complete,
customers have the ability to evaluate models performance
through model evaluation components and similarly,
we have the ability to assess
that the model was built responsibly,
through a built-in responsible AI dashboard.
To conclude, we can take a look at experiments
we conducted using Azure NDMV A100 V4 series machines,
combined with Nvidia NeMo Megatron,
to test GBT3 model architecture.
We tested models ranging in size
from 126 million to 530 billion parameters
on clusters of up to 175 nodes each, with HEBus.
To achieve the best results, we ran the largest model,
which has 530 billion parameters,
on 105, 140 and 175 nodes.
We measured the training time
per step in seconds in each configuration.
The lowest training time per step for this model
was around 55 second with a mini batch size of one
after the model rate steady state.
We found that as we increase the number of nodes,
the training time per global step decreased,
resulting in a faster training time.
The graph shows a comparison
between a theoretical linear speed up
and actual numbers we achieved,
as we increased the number of nodes available for training.
When we increase the number of nodes from 105 to 140,
the speed of port configuration was 1.3 times
that of the 105 nodes, with a 98.1% efficiency.
Similarly, when we increased the number of nodes to 175,
the speed up was 1.58 times that of 105 node,
with a 95% efficiency.
These results indicate that adding more nodes
leads to almost linear speed increases,
resulting in better performance
for larger or time sensitive workloads.
Exactly what we want when we are trying to train
very large scale foundation models
or any other large string scale training workloads you have.
Thank you.
- That's really great to see, Vijay.
It's amazing performance numbers.
That's wonderful to see all of the tooling
that you've built to help people for training
and deploying large language models in real life.
Thank you for having me.
I hope you all have a great Azure Open Source Day.
I look forward to seeing you all in the future.
For more references on the material that was covered
in this presentation, check out the chat.
- Thank you, Sharan,
and thank you everybody for joining today.