PRIYANKA VERGADIA: Hi.
I'm Priyanka Vergadia.
And in this video, we will see what are hyperparameters,
how to tune them, and set up a hyperparameter tuning
job in Vertex AI.
All right, so what is a hyperparameter?
Well, a training application handles three categories
of data as it trains the model.
The input data, which is also called training data,
contains the features that the model uses to make predictions.
Now, but this data is never directly
part of the model architecture itself.
The second type of data is model parameters.
These are the variables that our ML model or technique
uses to adjust the data.
For example, a deep neural network, or DNN,
is composed of processing nodes or neurons,
each with an operation performed on the data
as it travels through the network.
Now, when our DNN is trained, each node
has a weight value that tells our model how much impact it
has on the final prediction.
Now, they are what distinguishes our particular model
from other models of the same architecture trained
on different data.
Now, the third type of data is hyperparameters.
And these are the variables that govern
the training process itself.
For example, a part of designing a deep neural network
is deciding how many hidden layers
you will want between the input and the output to be,
and how many nodes each hidden layer would use.
Like here I have two, then three, then three,
but this could be any number of nodes.
These variables are not directly related to the training data,
and they are the configuration variables for your model.
Now, when it comes to model parameters,
they can change with the training job.
But hyperparameters are usually going
to stay constant, which means, in our deep neural network
model example, finding the right number of hidden layers
means choosing a value of the number of hidden layers,
training a model, evaluating it with those values,
and then trying it again with a different value.
Now, if we wanted to get the best type of parameter settings
manually, it could get pretty tedious pretty quickly.
That's where Vertex AI hyperparameter tuning comes in.
It is an automated process of tuning hyperparameter jobs
as you're running your multiple trials of the training
application with values of the chosen hyperparameters set
within the limits that you specify.
And to select hyperparameters that give the best performance,
Vertex AI uses the Bayesian optimization
for hyperparameter tuning.
If you're interested in the math behind it,
I've included a link below to learn more about it.
With that, let's jump into the console
and train a hyperparameter tuning job.
Here I am in the Google Cloud console.
First, you need to enable Compute Engine and Container
Registry APIs.
In this demo, I'm training and tuning an image classification
model trained on the horses or humans data set
from TensorFlow data sets.
We will need a notebook to containerize our training
application code.
Since I'm creating a TensorFlow model in this example,
I'm going to select a TensorFlow Enterprise 2.5
notebook without a GPU.
But you can train a model built with any framework using
custom or pre-built containers, so select
the notebook accordingly.
I already created one here, so let's just hop into that.
In the terminal, I created a folder called Horses or Humans,
and inside that folder, I've created a Docker file.
This Docker file uses deep learning container TensorFlow
Enterprise 2.5 GPU Docker image.
And the deep learning containers on Google Cloud
actually come with many common machine learning and data
science frameworks that are pre-installed.
We're also installing the Hypertune Library here,
which we will use later to report the metrics we
want to optimize on Vertex AI.
After downloading the image, the Docker file
sets up the entry point for the training code.
Then, I created the Trainer folder with task.py file
and added the code for training and tuning the model.
Let's take a deeper look at this code now.
There are a few components that are specific to using
hyperparameter tuning service.
The script imports the Hypertune Library.
We installed this library in the Docker file we created earlier.
And the function, get_args, defines a command line argument
for each hyperparameter that you want to tune.
In this example, the hyperparameters
that will be tuned are the learning rate,
the momentum value in the optimizer,
and the number of neurons in the last hidden layer of the model.
But feel free to experiment with others, as well.
The value passed in these arguments
is then used to set the corresponding parameters
in the code.
At the end of main function, the Hypertune Library
is used to define the metric you want to optimize.
In TensorFlow, declares model.fit method
returns a history object.
The history.history attribute is a record
of training loss values and metrics
values at successive epochs.
If you pass validation data to model.fit,
the history.history attribute will include validation loss
and metrics values as well.
If you want the hyperparameter tuning service to discover
the values that maximize the model's validation accuracy,
you define the metric as the last entry,
or NUM_EPOCHS as minus 1 of the val_accuracy list.
Then pass this metric to an instance of hypertune.
You can pick whatever string you like
for the hyperparameter metric tag.
I'm setting it to accuracy here.
But you will need to use the same string
again later when you kick off the hyperparameter tuning job.
Now we're ready to build the container.
Set the project ID, image URI, build the container
using the image URI.
Once the container is built, push it
to the container registry.
And then next, let's head into the Vertex AI
console and create a model to run our hyperparameter tuning
job.
To configure a training job, select No Manage Data
Set, Custom Training, give the model a name,
and then select the custom container that we just
created from our notebook.
Notice that here, I'm using Custom Training
via Custom Container on Google Cloud Container Registry.
But you can also run a hyperparameter tuning job
with the pre-built containers.
In the next step, enable hyperparameter tuning,
add learning rate, which would be double, min and max values,
and log scaling.
Another parameter, which is momentum,
type would be double, minimum and maximum values,
linear scaling.
And third hyperparameter would be
on number of neurons, discrete type, and of value,
and no scaling for this one.
Next, we need to provide the metric we want to optimize
as well as the goal.
This should be the same as our hyperparameter
metric tag we set in the training application.
It was set to accuracy, and our goal is to maximize it.
The Vertex AI hyperparameter tuning service
will run multiple trials of our training application
with these values.
You will need to put an upper bound on the number of trials
this service will run.
For demo purposes, I'm adding 15 here.
More trials generally lead to better results,
but there will be a point of diminishing returns
after which additional trials have
little or no effect on the metric
you're trying to optimize.
So it's always a best practice to start with a smaller
number of trials and get a sense of how impactful
your chosen hyperparameters are before scaling up
to a larger number of trials.
Now you'll also need to set an upper bound on the number
of parallel trials.
Here, I'm setting this to three.
Increasing the number of parallel trials
will reduce the amount of time the hyperparameter tuning
job takes to actually run.
However, it can reduce the effectiveness of the job
overall.
This is because the default tuning strategy
uses results of previous trials to inform
the assignment of values in the subsequent trials.
So if you run too many trials in parallel,
there will be trials that start without the benefit
of the results from the trial that are still running.
Now the last step is to select the search algorithm.
Now grid search is a very traditional technique
for implementing hyperparameters.
It's a brute-force, combines all the combinations.
Now random search randomly samples the search space
and evaluates the sets for specified probability
distribution.
For example, instead of trying to check all the 100,000
samples, we can check 1,000 random sample parameters.
We will select default search algorithm here,
which will use Google Vizier to perform Bayesian optimization
for hyperparameter tuning.
To learn more about how this works,
see the documentation below.
In Compute and Pricing, leave the selected region as is
and configure Worker Pool zero with a machine type,
accelerator, and disk size.
Now note here that selecting GPU is totally optional--
hyperparameter tuning job will just
take a little longer to complete if you do not use a GPU.
Then start the training.
It should take about 45 minutes to train
with the demo parameters that I've selected here.
And when it's finished, you will be
able to click on the job name and see the results
of the tuning trials.
From the results of our trial, we
can see here that setting the learning rate to 3.9,
momentum to 2.8, and neurons to 128 results in the highest
validation accuracy.
In this video, we've learned what
is hyperparameter tuning, how Vertex AI helps automate
the hyperparameter tuning process,
and, then, we launched a hyperparameter tuning job
for training code in Custom Container via the UI.
If you would like to do the same using Vertex Python SDK,
I have linked the codelab below.
Give it a try, let me know how it goes, and for any questions,
reach out to me on Twitter @pvergadia.