[Music]
some people say
i mispronounce value
but that is okay
at least it's okay with me
stackquest hello
i'm josh starmer and welcome to
statquest today we're going to do neural
networks part
3 the relu activation function in
action note this stat quest assumes that
you are already familiar with the main
ideas behind neural networks
if not check out the quest the link is
in the description below
in neural networks part 1 inside the
black box
we started with a simple data set that
showed
whether or not different drug dosages
were effective
against a virus the low and high dosages
were not effective
but the medium dosage was effective
then we talked about how a neural
network like this one
that uses the soft plus activation
function in the hidden layer
can fit a green squiggle to the data set
bam now let's see what happens if we
swap out the soft plus activation
function in the hidden layer
with one of the most popular activation
functions for deep learning
and convolutional neural networks the
rel u
activation function which is short for
rectified
linear unit and sounds like a robot
and as a bonus because it is common to
put an activation function before the
final
output we'll do that too bam
remember to keep the math simple let's
assume dosages go from 0
for low to 1 for high so if we plug in
the lowest
dosage 0 the connection
from the input to the top node in the
hidden layer
multiplies the dosage by 1.70
and then adds negative 0.85
and the result is an x-axis coordinate
for the activation function
so if we plug in 0 for dosage
then the x-axis coordinate for the
activation function
is negative 0.85
now we plug negative 0.85 into the relu
activation function the relu
activation function outputs whichever
value is larger
0 or the input value which in this case
is negative 0.85
and because 0 is greater than negative
0.85
the output from the relu activation
function
is zero and the corresponding y-axis
value is zero so let's put a blue dot at
zero
for when dosage equals zero
now if we increase dosage to 0.2
the x-axis coordinate for the activation
function
is negative 0.51
and again because 0 is greater than
negative 0.51
the output from the relu activation
function is 0
and the corresponding y-axis value is 0.
so let's put a blue dot at 0 for when
dosage equals 0.2
and if we increase the dosage value to
0.4
we get 0 for the y-axis coordinate
again however when dosage equals 0.6
the x-axis coordinate is 0.17
now when we plug 0.17 into the rail u
activation function the output is 0.16
because 0.17 is greater than 0
and the corresponding y-axis value is
0.17
and if we continue to increase the
dosage values all the way to one
the maximum dosage we get this bent
blue line then we multiply the y-axis
coordinates on the bent blue line
by negative forty point eight
and the new bent blue line goes off the
screen
note for those of you drawing this at
home your final bent blue line will have
a steeper slope than mine
because for the sake of clarity i am not
drawing things perfectly to scale
tiny bam now when we run
dosages through the connection to the
bottom node in the hidden layer
we get the corresponding y-axis
coordinates
that go off the screen for this
straight orange line
now we multiply the y-axis coordinates
on the straight orange line
by 2.70
and we end up with this final straight
orange line
now we add the bent blue line and the
straight orange line
together to get this green wedge
now we add the final bias term negative
16
to the y-axis coordinates on the green
wedge
lastly because we included the rail u
activation function
right in front of the output we use the
green wedge
as its input for example
the y-axis coordinate for this point on
the green wedge is negative 16
which corresponds to this x-axis
coordinate for the relu activation
function
and when we plug that into the relu
activation function
we get 0 because 0 is greater than
negative 16.
and 0 corresponds to this green dot
likewise the y-axis coordinate for this
point on the green wedge is negative 9.2
and just like before when we plug that
into the rail u activation function
we get 0
and 0 corresponds to this green dot
now in contrast to the last two points
we fed into the rel u
activation function this one is positive
0.49 and when we plug it into the relu
activation function we get 0.49
and 0.49 corresponds to this green
dot likewise the remaining positive
y-axis coordinates
stay the same and the negative values
are set to zero
and at long last we end up with this
green pointy thing
double bam thus
the relu activation function may seem
weird because it's not
curvy and the equation is really simple
but just like for any other activation
function the weights and biases on the
connection
slice them flip them and stretch them
into new shapes
which are added together to get an
entirely new shape
that fits the data triple bam
oh no it's another technical detail
alert
some of you may have noticed that the
relu activation function is bent
and not curved this means that the
derivative is not defined where the
function
is bent and that's a problem
because gradient descent which we use to
estimate the weights and biases
requires a derivative for all points
however it's not a big problem because
we can get around this by simply
defining the derivative at the bent
part to be zero or one it doesn't really
matter
small bam and now it's time for some
shameless self-promotion if you want to
review statistics and machine learning
offline check out the stack quest study
guides at statquest.org
there's something for everyone hooray
we've made it to the end of another
exciting stat quest
if you like this stat quest and want to
see more please subscribe
and if you want to support statquest
consider contributing to my patreon
campaign
becoming a channel member buying one or
two of my original songs or a t-shirt or
a hoodie or just donate
the links are in the description below
alright
until next time quest on