Welcome back to the oneAPI channel.​
Should I buy I very large dedicated system for Model training?​
With a GPU or Habana Gaudi?​ Or maybe I could use a combination
of many existing systems I already have, say many CPUs that are anyway idle from time to time?​
The main obstacle will probably be my
ability to run model training on multiple machines efficiently.​
And communicate data between threads or between compute nodes..​
Let me tell you about oneAPI oneCCL the collective communication library
and how it can help you scale out your applications
In previous videos we have talked about the oneAPI components.
We mentioned the multi-device multi-vendor open spec libraries..
That can now work on multiple devices, CPU, GPU and others, what we call XPUs..
And are packed into toolkits..​
For example, this is the Intel oneAPI Base Toolkit.​
You can find here everything you need to get started..​
​
Including the oneAPI oneCCL​
The collective communication library​
oneCCL can help you for multiple applications, the most recent one that is in focus is model training
A typical deep learning based model,
a neural network is structured in layers executed one after the other
And during the training process, the gradients
are computed sequentially backward with very limited opportunities for parallelism
So how can we run such a model on multiple machines
and how to easily scale it out to even more machines..
One option is to break the model into pieces, and run each portion on a different machine,​
What is called “Model Parallelism”, ​
This method is sometimes used when the model memory footprint is too large for a single device​
But the more common method is “Data parallelism”
​ “Data parallelism”
is running the same model on multiple machines, each with a subset of the data.​
The complexity here is how to update and synchronize the gradients or the weights
across each of the tasks, or workers..​
For this you need a set of operations that can
support this algorithm across multiple tasks or compute nodes​
​With oneCCL, each thread, each task is assigned a worker​
Workers can communicate between themselves
asynchronously with very low overhead and without disrupting the actual execution
The oneCCL library is supported and optimized for Intel® CPUs and GPUs.​
But the spec is open so you can implement it to any architecture.​
​
oneCCL is built on the lower level communication middleware - MPI and OFI .​
And it supports many interconnects architectures Like Cornelis Networks, InfiniBand*, and Ethernet.
There are not many operations required, and you can find the list at oneAPI.io​
Look for the spec of oneCCL​
And you got the list here..​
List of all operations..​
All-reduce allows to sum the gradients and is probably the most used one​
You can read the full description here..​
And the rest of the supported instructions
If we jump to the oneCCL github page​
you can of course download just the oneCCL library or the full oneAPI Base toolkit ​
But I believe most users would like to run Hovord or distributed pyTorch training​
So here you have all the instructions to use both.​
And how to install and allow Hovod and pyTorch to use oneCCL​
Pay attention that Hovord or pyTorch does not support oneCCL by default, ​
​ BTW, if you
download the oneAPI AI Analytics toolkit, you are getting pytorch with oneCCL in the package.
There are some great article with test cases you can read to understand the benefits​
This for example is a comparison of scaling out Resnet50 training by MXNet using Horovod, ​
the red bars are openMPI, ​
the blue ones are oneCCL, still called here MLSL.. Achieving 98% of the theoretical scaling.​
​
This is oneCCL scaling-out Facebook DLRM model and there are more test cases online.​
​
So oneCCL.​
oneAPI Collective Communications Library​
Easy to use on multiple platforms and devices, ​
Scale out your models using distributed training,​
Enjoy, subscribe​
Thank you