(bright music)
(bright music swells then drops off)
- Hi everyone, and welcome to this tutorial.
My name is Hugo Affaticati,
and I am a technical program manager
in the HPC + AI Benchmarking Team on Azure.
Today, I will explain how to run BERT,
the machine learning model for natural language processing,
with MLPerf Inference v3.0
on Azure's NC A100 v4 series.
The inference benchmark that we will use for this video
is sourced from MLPerf Suite.
ML Commons is an open engineering consortium of AI leaders
that provides unbiased evaluation of inference performance
for hardware, software, and services,
all conducted under prescribed conditions.
MLPerf Inference Benchmarks
consists of real-world compute-intensive AI workloads
to best simulate customers' needs.
I will be using
an NC96ads A100 v4 virtual machine
with four Nvidia A100 PCIe GPUs,
and 80 gigabytes of memory per GPU.
If you don't know how to set up your virtual machine
using Azure Portal or Azure CycleCloud cluster,
I invite you to start
by watching the corresponding tutorial on our channel.
Benchmarking with MLPerf Inference v3.0
consists of three main steps.
First, we will set up the environment.
Then, we'll download and preprocess the datasets.
And finally, we will run the benchmarks,
and compare the performance
to the official results from ML Commons.
Let's get started.
As you can see on this console window,
I have attached four NVMe disks to my virtual machine
for 3.5 terabytes of additional storage.
I will be using the resource_nvme folder,
where the NVMe disks are mounted,
for the dataset as well as the benchmarking code.
Azure published the code publicly on ML Commons' GitHub.
Thus, we will download it with a simple git clone command.
Once we finish cloning the code,
we'll need to create several folders for the datasets.
I am exporting the path to the data,
and creating three folders:
data, models, and preprocessed_data.
Finally, we can enter the directory specific to Azure.
The environment is now ready for the second step.
This step
consists of downloading the datasets and the models,
as well as preprocessing the data.
First, we must start the container
with the make prebuild command.
This step is expected to take up to ten minutes
the first time it runs,
but it will be instantaneous every time after.
If you were to exit the container,
you must export the path to the directory
before you run the make prebuild command again.
Once inside the container,
you can download the dataset for the benchmark BERT
with the following command.
You use the same command and replace "bert"
by retinanet, resnet50, 3d-unet, rnnt, or dlrm,
if you want to run any other benchmarks of the suite.
Then, you reproduce these steps
with make download_model to download the model,
and make preprocess_data to preprocess the dataset.
Finally, you can run the command make build
to build the binaries.
This command sets the links
to your datasets, models, and preprocessed datasets,
as well as building the necessary binaries
for your virtual machine.
Once this step is done, you will see the build folder,
and you are ready to run the benchmarks.
This final step
consists of running the benchmark in a simple command.
This make run is a two-step process
that builds the engines before running them.
You can select the benchmarks.
Here we run bert.
The scenarios can be either offline or server.
Today, I will be running offline only.
Lastly, the configuration.
It can be default accuracy for 99% accuracy,
high_accuracy for 99.9%,
as well as triton, and triton with high accuracy.
Each run takes a minimum of ten minutes
to allow the system to reach thermal equilibrium.
You can avoid it by adding a --fast flag at the end,
which is what I'm doing in this tutorial.
Lastly, your result is considered correct
only if you see the mention VALID.
Together, we obtained a result
of 12,508 samples per second
for BERT offline, default accuracy.
You can compare your results
to the official results published by Azure
on the ML Commons website.
You can find the results for Azure for all the benchmarks,
and for all previous versions
of MLPerf Inference and Training.
For example, the official results
that we published in April, 2023,
was 12,455 samples per second
for BERT offline, default accuracy.
You have now successfully run
the natural language model BERT
of MLPerf Inference v3.0 on Azure.
We hope you find this content useful.
And don't hesitate to follow this channel for more content
on how to use Microsoft's HPC + AI offerings.
Have a good one.
(bright music)