Hello, and welcome!
Today, we are collaborating with CodeEmporium to cover some of the techniques that allow
for training neural networks fast and efficiently.
First, let's assume that you can afford hundreds or even thousands of GPUs.
An obvious way to speed up training, in that case, would be to split your data and train
your model on all GPUs that you have at the same time.
For example, if you choose your batch size to be 8192 and distribute it to 256 GPUs,
then each GPU would get a mini-batch of 32 samples.
Each one of those GPUs keeps a copy of the model parameters and gets the same updates.
Therefore, the copies on different GPUs don't diverge from each other.
TensorFlow calls this approach a mirrored strategy.
Yet, this approach has some shortcomings.
Large batches result in smoother gradients because a large batch size reduces the variance
in the gradient.
Although this gives a more accurate estimate of the 'true' gradient, a large batch size
also reduces the stochasticity of the optimization.
Empirically, models trained with very large batch sizes get worse validation accuracies
as compared to the ones trained with smaller batch sizes, given the same number of epochs.
One of the tricks to alleviate this problem is to increase the learning rate to compensate
for the averaging effect that a large mini-batch has.
A rule of thumb that many papers have adopted is to scale the learning rate linearly.
For example, when you quadruple the batch size to distribute training over four GPUs,
multiply the learning rate by four as well.
This heuristic seems to work well for a reasonable range of batch sizes and learning rates, as
shown in this paper titled 'Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.'
Another trick is to use a learning rate warmup.
It's a common strategy to start with a relatively large learning rate and decay it over time.
In learning rate warmup, one begins with a small learning rate and increases it to a
pre-set value in a warmup phase, which is typically the first few epochs.
Then the learning rate is decayed as usual.
Both of these tricks are particularly useful for distributed training on multiple GPUs.
However, they can help with single-GPU training as well.
For example, my experience is that learning rate warmup helps stabilize harder-to-train
models, regardless of the batch size and the number of GPUs you have.
Speaking of learning rates, let's talk about some learning rate schedules that might help
speed up training.
One of them is a cyclic learning rate schedule that increases and decreases the learning
rate in a cycle, within some upper and lower bounds.
The upper bound can be reduced as the training progresses.
A variant of this idea is the one-cycle policy that increases and decreases the learning
rate in a single cycle during the entire training process.
This is somewhat similar to the learning rate warmup that we talked about earlier.
The one-cycle policy is also applied to the momentum parameter in the optimizer but in
the reverse order.
Recently, I ran into this paper titled 'Bag of Tricks for Image Classification with Convolutional
Neural Networks.'
It summarizes and evaluates many efficient training tricks, including some that we already
covered in this and previous videos.
Let's go through the ones that we haven't covered so far.
One trick that we haven't covered earlier is a simple technique called mixup, which
helps against overfitting and reduces a model's sensitivity to adversarial examples.
Essentially, mixup is a very straightforward data augmentation technique that randomly
blends input samples.
More specifically, mixup training randomly samples a pair of data points and generates
a new sample by computing a weighted average of both the inputs and outputs.
For example, for an image classification task, that would mean blending the input images
with some transparency and using the same blending parameter to compute a weighted average
of the output labels.
Another label manipulation trick is label smoothing, which is a much older technique
than mixup.
In a typical classification dataset, the labels are one-hot vectors, where a true class has
a value of one, and everything else has a value of zero.
However, a softmax function at the end of a model never outputs one-hot vectors.
This creates a gap between the distributions of the ground truth labels and model predictions.
Label smoothing shrinks that gap.
This is done by subtracting some epsilon from the true labels and adding it to the others.
This approach prevents models from being "too sure" and also acts as a regularizer.
But, keep in mind that very large values of epsilon would flatten out the labels too much.
The stronger the label smoothing is, the less information the labels retain.
In the earlier videos, we discussed the relationship between model size and error.
The error of a model on a test set is typically expected to be higher than the error on the
training set.
This is called the generalization gap and is explained by overfitting, where a model
starts memorizing the samples in the training set, losing its generalization ability.
Increasing the model size makes a model more prone to overfitting.
Therefore, beyond some point, a larger model is expected to have a larger test-to-training
error ratio.
However, an interesting phenomenon that some researchers noticed in deep models is that
as the model size increases, the test-to-training error ratio starts decreasing again after
increasing up to some certain point.
They call this the double descent phenomenon.
This behavior seems to contradict the classical concept of the bias-variance trade-off.
Nevertheless, it appears that it's a good idea not to give up too early and shrink the
model size if a model is overfitting.
Once the model size gets past the critical regime, the generalization gap might start
improving again.
The same applies to the number of training iterations, since training a model for longer
iterations can increase its effective capacity.
It's a common practice to stop training once the validation error stops improving, and
the model starts overfitting.
However, at some point, the training process may enter a regime where training longer reverses
overfitting.
Why does this happen?
I don't have a clear answer, but other researchers may have some ideas.
You can check the references below to learn more.
Finally, the fastest and easiest way you can build a deep learning model, without worrying
too much about how much data or resources you have is ... (sounds familiar) ... Transfer
Learning.
In a nutshell, transfer learning gives a model kick-start by transferring the weights from
another model, instead of training from scratch.
You can check out my earlier video for a more detailed overview of transfer learning techniques.
Now, I'm passing the mic to Ajay to talk about mixed-precision training.
Hello everyone!
Ajay Halthor from CodeEmporium here.
Hope you’ve been enjoying Leo’s talk on different ways to speed up neural net training
so far.
I will introduce yet another way to speed up training with a technique called Mixed
Precision Training.
After that, we’ll actually see how fast this really is with some PyTorch code.
Because seeing is believing.
Computers love working with numbers.
Numbers are typically represented in a 32 bit format for storing and processing data.
This extends to deep learning too.
We have a simple neural network here.
For the sake of this argument, this neural network will take an input image that contains
an animal; and it spits out the animal text.
We want to train this neural network.
Training means that we want to find the edge weights of this network such that it is able
to do this animal classification task on its own.
These weights are stored in their 32 bit format.
Training in general includes forward and backward propagation, which implies billions of multiplications
and additions of 32 bit numbers.
But do we really need 32 bits to represent a number during training?
Not really.
Gradients computed in backpropagation tend to be very low values.
So, we need a lot of bits to adequately represent them.
But we could inflate these gradients, so we wouldn't require as many bits to represent
them.
In fact, we could represent numbers with 16 bits instead.
This would reduce the memory storage for the model by half, making the training several
times faster, since we are making arithmetic operations with less bits.
The only thing we need to worry about is accuracy.
The accuracy should obviously drop significantly since we are cutting precision big time.
Or does it?
Yeah.
It drops by like 80 percent, making it completely useless.
But if we scale our gradients up correctly, we can use only 16 bits while still maintaining
accuracy.
How do we do this?
There is a paper on mixed precision that was released by Nvidia and Google that explains
this concept.
We are going to state exactly what they did.
First, maintain a master copy of actual weight parameters in the original 32 bit precision
format.
This is going to be the actual set of weights that we use in the model.
Then, we convert the 32 bit to a 16 bit precision.
Then, we perform the forward propagation step with all the arithmetic operations.
Once a forward propagation part is complete a loss is computed.
Then, scale this loss by multiplying it by some constant.
This way during the back propagation phase, the
computed gradients are also scaled up.
Perform back propagation, which computes the 16-bit gradients.
Then these gradients are used to modify the original 32 bit weights.
First, maintain a master copy of actual weight parameters in the original 32 bit precision
format.
This is going to be the actual set of weights that we use in the model.
Then, we convert the 32 bit to a 16 bit precision.
Then, we perform the forward propagation step with all the arithmetic operations.
Once a forward propagation part is complete a loss is computed.
Then, scale this loss by multiplying it by some constant.
This way during the back propagation phase, the computed gradients are also scaled up.
Perform back propagation, which computes the 16-bit gradients.
Then these gradients are used to modify the original 32 bit weights.
This is one iteration and we do this more a number of iterations.
And that’s it!
This gives rise to neural networks that can be trained much faster using less memory.
This leads to similar if not better performances in different fields of computer vision and
language processing problems.
We will leave a link to the paper below and it's really not that tough to read.
So, check it out!
We got that theory locked in, now for the code!
Right now what we're gonna do is, we're going to train a generative adversarial
network to generate images of different types of objects.
We can use the cifar10 dataset to curate these images.
I'm gonna do all of this in Google Collab so everyone can execute this without installing
a thing on your local computer.
Just go to colab.research.google.com and create a new notebook.
You can name it whatever you want.
We need to write four lines of code to get things running.
I'll copy and paste them for now.
This will clone the apex repo and it creates the setup.sh file.
Then we run this setup file.
It may take a few minutes but once it's up you're all ready to go.
On the sidebar, let's go to files to get an idea of the file system that we're dealing
with.
Let's go to apex/examples/DCGAN/main_amp.py.
This file has the code for the deep convolutional GAN with the mixed precision modeling that
we discussed.
Let's run this file passing in a few arguments.
the opt_level parameter we set it to O1 to indicate that we want to use a mixed precision.
The outf parameter indicates that I want to dump my output files into the folder outputs.
Let's run this, and the training of our generative adversarial net begins.
These iterations are pretty fast!
Let's take a look at the main_amp.py file to see what's really happening.
I'll open this in GitHub because it's just cleaner to see with the line numbers.
The first 20 lines are just imports.
OS is for specifying file paths.
Random is for creating a random seed.
This ensures that random data generated here will always be the same, so we can compare
different models.
Torch is for PyTorch, a deep learning framework that allows us to build models easily, from
accessing data sets to setting up different components of a neural network model.
We import amp for the mixed precision tools too.
Next, we include a bunch of possible arguments you can pass in, while executing the Python
file.
We just included opt_level and outf in the call.
But clearly, there's a lot to play with here.
Now we begin the construction of our generative adversarial network.
The GAN consists of two parts: a generator and a discriminator.
Generator networks generate counterfeit images while the discriminator network needs to be
able to tell real images from fake images.
These two components compete each other over many iterations.
They get better and better doing their job until the generator can make nearly realistic
images.
Back in our code, we can also pass in parameters related to our discriminator and generator
too.
Next the output directory is created.
Depending on the data set you choose the code will load that in.
We're using the cifar10 dataset, so only this chunk of code is executed.
Next, construct the generator loss, starting with the basic architecture.
This is the basic transpose convolutional neural network.
The input is some noisy and the output is a 64x64 colored image.
Then, we do the same for the discriminator architecture.
This is a convolutional neural network architecture.
The input is a64x64 image and the output is just whether the image is real or fake.
This is followed by setting up the binary cross entropy loss with logits, and setting
up the optimizers for the networks.
This defines how the networks will learn.
We have a for loop for the actual training phase over a number of epochs.
But what did we add differently for incorporating mixed precision?
Let's find that word amp followed by a dot on this page.
Looks like we only have a few matches.
So it's simple additions.
We simply initialize apex giving it the generator and discriminator networks and their optimizers.
It also has a num_losses parameter to indicate the number of losses we have in the overall
network.
We specify this so that each loss can be scaled differently.
When training this DCGAN, we start by training it with real images first.
We execute the forward propagation step and then get the loss: errorD_real.
Once complete, we scale this loss and then execute the back propagation step.
Now this is from one loss but we repeat this for the other two losses too.
Now we feed the fake images generated by our generator network and compute the second loss:
errD_fake.
Once again we scaled this loss and then start the back propagation step.
The total of these two losses is the discriminator loss, overall.
Now the third loss is the generator loss.
Compute the loss errG, scale it, then execute the back propagation step.
For every 100 iterations, we show the logs and we save the real samples from the data
set, and save a set of images that were generated by the generator.
That's about it for the code!
When executing mixed precision on the cifar10 dataset, it's twice as fast.
Pretty slick, right?
Back to you Leo!
Thank you Ajay.
Alright, that's all for today.
I hope you liked it.
Subscribe for more videos.
Check out CodeEmporium's channel.
The link is in the description.
Thanks for watching, stay tuned, and see you next time.