Hello, and welcome!
In this video, we will provide an overview of several activation functions, many of which
are included in the TensorFlow library.
Activation functions are a cornerstone of Machine Learning.
Generally speaking, a processing unit will pass its input through an activation function
to generate the output, so the choice of function ends up playing an important role.
We'll go through some of the most commonly-used functions in order to develop a more intuitive
understanding.
But before we begin, we need to import a few dependencies.
The following piece of code contains a basic function that will plot the surface of an
arbitrary activation function.
The plot includes all possible weight and bias values between -0.5 and 0.5, with a step
of 0.05.
The input, the weight, and the bias are all one-dimensional.
The input can also be passed to the function as an additional argument.
The Step function was one of the first activation functions designed for machine learning.
This function essentially acts as a limiter.
If the input exceeds a certain value, the output will be 1, otherwise the output will
be 0.
It should be easy to see that this type of output lends itself well to classification
problems, like two-class logistic regression.
There are variations like the Rectangle step function and a few others, but they aren't
commonly used.
And as it turns out, TensorFlow does not offer a step function.
Sigmoid functions are an extremely popular function family in the machine learning world.
The term comes from the 'S' shape that the plots form on the Cartesian plane.
Sigmoid functions are useful because they compress the input down into a bounded interval.
This helps when you need to combine the result with other functions, like the Step function
from before.
The most commonly-used sigmoid functions are the Logistic, Arctangent, and Hyperbolic Tangent
functions.
Logistic Function As you'd expect, the logistic function is
widely used in Logistic Regression.
You can take a look at its definition here: This function maps the input into the open
interval from 0 to 1, like so: The Arctangent and Hyperbolic Tangent functions
are both based on the tangent function, as implied by the names.
Arctangent is defined here: and it produces a sigmoid over this interval:
Keep in mind that TensorFlow does not include Arctangent.
The Hyperbolic Tangent, or TanH as it's usually called, is defined like this:
It produces a sigmoid over the open interval from -1 to 1.
TanH is used in a wide range of applications, and it's probably the most popular function
of the Sigmoid family.
Linear Units form an important category of activation function by combining the best
concepts from Step and Sigmoid.
In general, all the linear unit variations stem from something called the Rectified Linear
Unit, or ReLU for short.
ReLU is a simple function that outputs a value from 0 to infinity.
So say we have some input 'x'.
If 'x' is negative, ReLU outputs 0.
But if 'x' is non-negative, ReLU just outputs 'x'.
It may seem counterintuitive to use a basic pseudo-linear function over something more
complex like a Sigmoid.
But ReLU provides some important benefits that might not be obvious at first glance.
For example, during the initialization process of a Neural Network model, weights are distributed
at random for each unit.
ReLU will only activate approximately 50% of the time, which actually saves some processing
power.
The ReLU structure also solves the 'Vanishing Gradient' and 'Exploding Gradient' problems,
both of which are well-known issues with the training process.
And as a marginal benefit, this type of activation achieves 'Biological Plausibility', because
it's directly relatable to the common biological model of a neuron.
Different ReLU variations are optimized for certain applications, but since they're implemented
on a case-by-case basis, they're beyond the scope of this video.
If you want to learn more, you can search for 'Parametric Rectified Linear Units' or
'Exponential Linear Units'.
The TensorFlow library includes ReLU along with a few key variants.
You can take a look here.
So by now, you should have an understanding of the activation functions supported by TensorFlow.
Thank you for watching this video.