Overfitting is a very common issue in
the field of machine learning
and L1 and L2 regularizations are
some of the techniques that can be used
to address the overfitting issue.
In this video, we'll go over some theory
on what exactly L1 L2 regularization is
and then we'll write a python code uh
and we'll see how a model which is an
overfit
can be addressed and the accuracy can be
improved
when you use L1 and L2 regularization. We
will be using
a housing price dataset from the city
of Melbourne and we'll
first build a model using a simple
linear regression and we'll see it is
overfitting the model
and then we'll use L1 and L2
regularization and we'll see
how we address that overfitting issue
and how it improves the
score on our test set. So let's get
started!
Let's say you're trying to predict
number of matches won
based on the age. Now usually when
the player gets aged any sports person
or athlete gets aged
the matches won kind of reduces. So
you can have this kind of distribution
where
to build a model you can create a simple
linear regression model and the equation
might look like this.
So matches 1 will be theta 0 plus theta
1 and into h
so theta 0 and theta 1 are just a
constant. This is a regular
uh linear like simple linear equation.
But you see that this line is not
really accurately describing all the
data points. It is trying to find a best
fit
in terms of a straight line but you see
all these data points are kind of going
away and then if you have test data
points which are lying somewhere here,
then this is not a very very accurate
representation of our
data distribution. Then you can
build a distribution which might look
like this.
So here we are trying to draw a line
which kind of exactly passes through
all our data points and in that case
your equation
might look like this. So it's a higher
order
higher order polynomial equation where
you are trying to find out the matches
won based on the
age of a of a person. But
here uh the issue is this equation is
really complicated, the line is a zigzag
type of line which is just passing
through
all the data points. And now if you have
some general data points at the top here
again this is not generalizing the the
distribution really well. What might be
better is if you have a line like this.
So this is this is a a balance between
these two cases that we saw previously.
So you can have you know only theta 2
into age
square the line will look like a curve,
and it can generalize your data really
well so that tomorrow if new data point
comes in
this equation will be able to make a
better prediction for you.
So the first case is called under
fitting, the second case is called
overfitting, and the third case is
balanced fit.
So you kind of get an idea on over
fitting here where
if you try to run
training too much and try to fit too
much to your training dataset
then you will have issues with uh
testing dataset. When you try to predict
your new data points uh it might not do
a better prediction. So you always have
to keep a balance between
these two extreme cases.
Now now if you don't know about all
these equations and everything
please refer to my linear regression
tutorials. I had
in the same playlist I had first few
videos on linear regression,
so you should watch that that's kind of
prerequisite.
Now how do you reduce overfitting? So
here is my overfit line along with the
equation
and if in this equation I somehow
make sure that my theta 0 and theta 4 is
almost close to 0,
then I will get an equation like this: so
just think about it
if theta 3 and theta 4 is almost close
to 0,
then you're kind of ruling out this
whole factor and then
you can create this type of equation. So
the idea here
is to shrink your parameters your
parameters which is theta 0,
theta 3, theta 4, even theta 2, theta 1 if
you can
reduce this parameter if you can keep
these parameters
smaller then you can get a
better equation for your prediction
function.
Now how do we do that? We
earlier saw in our linear regression
video that
um we calculate mean squared error so
when we run training
we pass first sample and then we
calculate y
predicted on some randomly initialized
weights,
then we compare it with the truth value
and then this is how we call
calculate mean square error or MSE.
Here y predictor is actually
h theta x i where s theta x i could be
higher order polynomial equation like
this
okay, and x 1 x 2 is nothing but
it is your feature. So in our case it
will be
age of a person. If you are thinking
about housing price prediction it will
be
the size of the house. Now
just think that in your
mean square error function, so by the way
this mean square error function
we use during training and we want to
minimize the value of this
this error you know on each iteration. So
in this equation what if
I add this particular parameter? So what
is this?
So there is this lambda which is a free
parameter you can control it it's like a
tuning knob and
you are making a square of each of these
theta parameters.
So now if your theta gets bigger
this value will get bigger, the error
will be big
you know and your
model will not converge. So essentially
what you're doing is
you are penalizing higher values of
theta here.
So whenever model tries to make the
theta value higher
you are adding a penalty here. So by
adding this penalty
you're making sure your theta value
doesn't go
too high so they will remain very small
you can fine tune this thing using this
this parameter lambda here,
and if you make this bigger the theta
value will even get
smaller and smaller you know, and if you
get this smaller than theta value can be
bigger.
So this is called L2 regularization it
is called L2
because we are using a square and in L1
regularization you are using the
absolute value. So that is the only
difference between L1 and L2
that in L1 you are using absolute value
of theta parameter.
Here again if theta is bigger the error
overall error is big and
it kind of acts as a penalty so that
during your training overall the value
of theta
will remain smaller and again going back
to this equation here
when these values remain smaller, you
come up with a
simpler equation you know you don't make
it really complicated
and simpler equations are the best uh to
represent the
generic case of your prediction. All
right? So let's get into coding now.
For the coding I'm using housing price
dataset for Melbourne city, I got this
dataset from
Kaggle and we are going to build a linear
regression model. So you can see that
there are different features such as
room
uh the distance, postal code, bathroom, car
and so on and in my notebook I am going
to
as usual first import
this dataset
and just call it dataset
you know. So I imported this
into my data frame now and data frame is
looking
good. I'm going to do some exploration
of my data frame now and kind of print
out
val unique values um in data sets. So you
see like there are 351 suburbs
these many addresses and so on. Also if
you look at
the shape of the dataset,
there are 3485 237 00:08:51,360 --> 00:08:57,040 7
uh records and 21 columns in total.
Now I'm going to discard certain columns.
I just did visual observation and
discarded certain columns which I don't
think are very useful. So
for example date right so date is
it's not useful, so I will just say okay
here are the columns that are useful
and I'm just doing copy paste to save
the time on recording,
and when you uh pass these columns to
this dataset you get the
filter dataset okay and then again you
can run dataset.head
so now I have less columns okay? So if
you do data
set.shape you know I have 15 columns
instead of 21.
Now I want to check for the NA values
and you can do it by calling is NA
function
on your pandas data frame and you can do
dot sum
and it will tell you so bedroom 2
has total 8217
NA um values you know NA rows.
So now we need to handle these rows. So
I'm going to fill
some of these columns with value 0
okay and those columns are these:
so columns to fill with 0 are this.
For example if car is NA which means
you know there is no car parking
available
for that particular property,
and when you run this function. So in
your data set you're saying
all these columns fill NA with zero it
will take those NA values and it will
fill them with
zero. So now after doing this I will do
this again and you can see that for
example
property count it was three so property
count now is zero.
All the NA values are filled similarly
car
car parking there were eight thousand
seven hundred rows
with NA value now I made zero.
Now what we're going to do
is we'll look at certain other columns
such as land size and building area
and we will like this through and we
will
uh calculate the mean and we'll fill
those with the mean value.
So the way you do that is using this
function.
So you are doing landsize.filln
a
with the mean of the same column okay?
So this is a safe assumption
and after that when you do NA
you find that see there are
no values with zero. I mean there are
some prizes but
if you look at your independent features
there
that's that's basically now
curated and there are
there are only two columns region name
and this
council area. So I'm not going to care
about it too much.
Now I will uh drop
those values. So like these three so if
you have a couple of
random columns with NA value you can
drop it you know our dataset is huge
and if you drop like these
three and three six rows it's not a big
deal.
So I will just drop them and run
the same function again and you see like
none of the columns have well uh any
values now.
So now uh I have some categorical
features
which I want to convert
into dummies you know I want to do one
hot encoding. So you guys might know
about one hot encoding
if you don't. I have a one hot encoding
video.
So things like suburb for this example
right any text column that you have you
need to
do a dummy encoding and pandas provide a
very convenient API called get dummies.
So from dataset you get
this and you drop first because you want
to avoid the dummy variable trap. Again
if you don't know about dummy variable
trap
you can watch my one hot encoding video,
uh and this will kind of drop that first
column when you do a uh encoding.
So now when I look at my data set you
see everything
there is no text column, and if you look
at this council area
see council area underscore so it created
a
separate column for each of the council
area. So this is what
dummy uh or hot encoding means.
Now I want to create x and y.
So my x so I can y is basically price
and
x is you know this one
and by the way I had some prices as NA
and when I did drop NA the prices
those N values got dropped as well so my
dataset looks pretty good,
and I will do now train test split.
So train test split is this. All of these
are like standard usual methods we have
covered in previous video that's why I'm
kind of going
over it little fast you are doing
a 30% split so 30% test 70%
training and you get all these different
data sets.
Now we will use a regular linear
regression
okay? So this is how you do regular
linear regression,
and then uh when you run this it's gonna
do
training and the model is fit now. Now I
will do
the score on x test and y test.
You realize that my score comes out to
be really low 14 percent
which is very very low score. But if you
do a score on
training data set say it is 68 percent
so this is clearly overfitting your
dataset
your dataset was so much over fit that
for training samples it gave a good
accuracy. But
for test samples the data samples that
it has not seen before
it gave a horrible score.
So how do we address this? Sklearn
provides
a model called LASSO. So LASSO regression
is basically
L1 regularization. So if you do
Sklearn LASSO regression,
see this is the LASSO regression and it
is L1
regularization. So I am going to use that
model,
so I created a model imported that
and then I created a LASSO regression
object
with alpha value. I'm initializing this
alpha value to be
random you can play with these values
and see which one gives you a better
accuracy,
and I initialize few other parameters as
well
and when I now run my regression
it is fitting it with the
regularization parameter on. So if you
look at our
equation earlier. So let me open
our equation,
in that equation
L1 one regularization will add an
absolute a theta value
in your error and this is the formula
that we'll use during the training.
Simple linear regression without any
regularization will not have this
red parameter. So that's the only
difference,
now let's do a score on your test.
So I will do LASSO regression score
and you find that the accuracy improved
and just to make sure
I will also do a score
on x test and on training as well.
So you see training and test both are
giving very good
not very good but 67 percent accuracy
which is
compared to 13 percent it improved to 66
percent. You can see
how much of a big difference our
regularization can make.
There is L2 regularization as well and
it's called ridge regression. So if you
do sklearn
ridge regression this one is an
L2 regularization and
here I can import that
from sklearn library as usual. I create
a regression
class object which will look like this
and call a fit method on this,
and then when I do a score on your
test dataset it looks six again it's 67
percent.
So it's pretty good and let's check on
training dataset.
Training data set is also you know it's
pretty
pretty good. So you saw that
by using ridge regression and LASSO
regression so ridge is L2,
LASSO is one your accuracy
for your not seen data samples which is
your test sample
improved a whole lot. If you are trying
to learn machine learning you can just
in youtube you can just search Code
basics machine learning tutorials,
I have the complete list of
machine learning videos here. You can go
in sequence if you're watching
regularization video if you want to
click get a fundamentals clear
initially on linear regression as etc., I
would suggest you watch tutorial two
and three I hope you like
this video if you do please give it a
thumbs up, share it with your friends
and thank you very much for watching!