Ridge regression versus lasso regression
which one will survive stat quest hello
I'm Josh stormer and welcome to stat
quest today we're gonna talk about rage
versus lasso regression and the
differences are going to be visualised
note this stack quest assumes that you
are already familiar with rage and lasso
regression if not check out the quests
to show you the difference between Ridge
and lasso regression we're going to use
a very simple dataset that consists of
weight and height measurements and we'll
start by fitting this horizontal line to
the data
this horizontal line represents a
terrible fit
and we can measure how bad that fit is
by calculating residuals the difference
between the observed and predicted
values in order to compare the
horizontal line to other lines fit to
the data we will plot the sum of the
squared residuals on a graph the y-axis
on this graph is the sum of the squared
residuals and the x-axis represents
different slope values for the fitted
line
in this case the slope for the
horizontal line is zero now let's
increase the slope to zero point two and
calculate a new value for the sum of the
squared residuals now let's increase the
slope to zero point four and calculate a
new value for the sum of the squared
residuals we can keep plugging in new
values for the slope and plotting the
sum of the squared residuals or we can
just plot the curve for this equation
we can see that the best fitting line is
at the bottom of the parabola in other
words when the slope equals 0.45 we get
the lowest sum of the squared residuals
in this example we simply calculated the
sum of the squared residuals for
different slopes now let's add the ridge
regression penalty aka the l 2-norm note
if you asked me it should be called the
squared penalty since that's what it is
and for me way easier to remember
the thick blue line that we just drew
represents lambda equals zero this is
because when lambda equals zero the
penalty is equal to zero regardless of
the slope and we are left with the
original sum of squared residuals
now let's see what happens when we set
lambda equal to 10 and just like before
we'll start with a horizontal line only
this time the line is orange just like
before we can calculate the residuals
and we can calculate the sum of the
squared residuals plus lambda times the
slope squared in this case lambda equals
10 in the slope of the horizontal line
is 0 so the penalty is 0
so we plot the sum of the squared
residuals here
now let's increase the slope to 0.2 note
the residuals are smaller than before so
the sum of squared residuals is smaller
than before but now the penalty is 0.4
and that gives us this point on the
graph
now let's increase the slope to 0.4 and
the residuals are even smaller but now
the penalty is 1.6
and that gives us this point on the
graph
and like we did before we can keep
plugging in new values for the slope and
plotting the sum of the squared
residuals plus the penalty or we can
just plot the curve with a thick orange
line that represents lambda equals 10
the bottom of the parabola is where the
slope gives us the lowest sum of squared
residuals plus penalty and that
corresponds to this specific line and
when we compare that to the optimal
slope when lambda equals zero we see
that setting lambda equal to ten results
in a smaller optimal slope note we can
also see that when lambda equals ten the
lowest point in the parabola is closer
to zero than when lambda equals zero
so either way we look at it we see that
the larger value for lambda Shrunk the
optimal value for the slope
likewise the thick green line represents
lambda equals 20
we see that the minimum value is closer
to zero and the optimal slope has shrunk
some more
the purple lines represent lambda equals
40 and it shrinks the slope even more
in other words as we increase lambda for
the ridge regression penalty aka the l2
penalty aka the square penalty the
optimal slope gets closer and closer to
zero but it does not equal zero
BAM
now let's see what happens if we use the
lasso penalty aka the l1 norm or if you
asked me I'd call it the absolute value
penalty
unfortunately no one asked me
again the thick blue line represents
lambda equals zero so there is no extra
penalty
this is because when we plug lambda
equals zero into the equation the
penalty becomes zero
and we are left with the original sum of
the squared residuals
note just like before we'll keep track
of the best fitting line plus penalty in
this graph on the left
the thick orange line represents lambda
equals 10 so now we are turning on the
penalty and shrinking the slope
note when lambda equals Tim we start to
see a kink in the curve where the slope
is zero
the thick green line represents lambda
equals 20 and this kink at zero is
becoming more prominent
lastly the thick purple line represents
lambda equals 40
and now the kink at zero is super
obvious now the lowest point in the
purple curve aka the optimal slope given
the absolute value penalty when lambda
equals 40 is zero
and that means the slope of the optimal
line is zero
and that means when lambda equals 40 we
ignore weight as a variable when
predicting height double bam
in summary when we increase the Ridge
regression penalty aka the l2 penalty
aka the square penalty the optimal
slopes shift towards zero but we retain
a nice parabola shape
and even when we set lambda to something
crazy high like 400 we still end up with
an optimal value greater than zero in
contrast when we increase the lasso
penalty aka the l1 penalty aka the
absolute value penalty the optimal value
shifts towards zero but since we have a
kink at zero zero ends up being the
optimal slope BAM hooray
we've made it to the end of another
exciting stat quest if you like this
stat quest and want to see more please
subscribe and if you want to support
stack quest consider contributing to my
patreon campaign becoming a channel
member buying one or two of my original
songs or a t-shirt or a hoodie or just
donate the links are in the description
below alright until next time quest on