machine learning is all about fitting models to data the models consist of
parameters and we find the value for those through the fitting process this
process typically involves some type of iterative algorithm that minimizes the
model error that algorithm has parameters that control how it works and
those are what we call hyper parameters in deep learning we also call the
parameters that determine the layer characteristics hyper parameters today
we'll be talking about techniques for both so why do we care about hyper
parameters well it turns out that most machine learning problems are non convex
this means that depending on the values we select for the hyper parameters we
might get a completely different model by changing the values of the hyper
parameters we can find different and hopefully better models okay so we know
we have hyper parameters and we know we want to tweak them but how do we do that
some hyper parameters are continuous some are binary and others might take on
any number of discrete values this makes for a tough optimization problem it is
almost always impossible to run an exhaustive search of the hyper parameter
space since it takes too long so traditionally engineers and
researchers have used techniques for hyper parameter optimization like grid
search and random search in this example I'm using a grid search method to vary
two hyper parameters box constraint and kernel scale for an SVM model as you can
see the error of the resulting model is different for different values of the
hyper parameters after 100 trials the search has found twelve point eight and
two point six to be the most promising values for the hyper parameters recently
random search has become more popular than grid search how could that be you
might be asking wouldn't grid search do a better job of evenly exploring the
hyper parameter space let's imagine you have two hyper parameters a and B your
model is very sensitive to a but not sensitive to B if we did a three by
three grid search we would only ever evaluate three differ
values of a but if we did a random search we will probably get nine
different values of a even though some of them may be close together as a
result we have a much better chance of finding a good value for a in machine
learning we often have many hyper parameters some have a big influence
over the results and some don't so random search is typically a better
choice grid search and random search are nice because it's easy to understand
what's going on however they still require many function
evaluations they also don't take advantage of the fact that as we
evaluate more and more combinations of hyper parameters we learn how those
values affect our results for that reason you can use techniques that
create a surrogate model or an approximation of the error as a function
of the hyper parameters bayesian optimization is one such technique here
we can see an example of a Bayesian optimization algorithm running where
each dot represents a different combination of hyper parameters we can
also see the algorithm surrogate model shown here is the surface which it is
using to pick the next set of hyper parameters one other really cool thing
about Bayesian optimization is that it doesn't just look at how accurate a
model is it can also take into account how long it takes to Train there could
be sets of hyper parameters that cause the training time to increase by factors
of 100 or more and that might not be so great if we're trying to hit a deadlock
you can configure Bayesian optimization in a number of ways including expected
improvement per second which penalize as hyper parameter values that are expected
to take a very long time to train now the main reason to do hyper parameter
optimization is to improve the model and although there are other things we could
do to improve it I like to think of hyper parameter optimization as being a
low effort high compute type of approach this is in contrast to something like
feature engineering where you have higher effort to create the new features
but you need less computational time it's not always obvious which activity
is going to have the biggest impact but the nice thing about hyper parameter
optimization is it lends itself well to overnight runs
so you can sleep while your computer works that was a quick explanation of
hyper parameter optimization for more information check out the links in the
description