Welcome everybody, you're watching mr fugu data science. Today, we're doing the
basics of maximum likelihood estimation. Now, let's think about this suppose that
we have some data. We suspect it has a given
distribution type. But, we don't know the parameters.
Well what do we do? Well this is where MLE shines.
So the parameters are found where they actually maximize the likelihood of
generating our model, from our given data. We're
trying to figure out basically, what model will best describe these data
and having a little bit of knowledge prior or
domain knowledge will really help in these circumstances. I would like to say
before we move further thank you to the recent subscribers and viewers lately i
greatly appreciate it. Now as you increase your
size of your data, if it's really large like the sample size of your
data. The variance will actually become the unbiased variance estimator.
And also you can generate a normal distribution
and normal confidence intervals, as well. So let's get into a little bit of this.
Basically, each probability distribution will have
unique parameters. And as those parameters change
a different probability distribution is generated.
The parameters for each model will fall into a family of distributions based on
the parameters you have. For example, a normal distribution has
the parameters of a mean and a standard deviation, right. But,
as those parameters change it creates a new model.
Therefore the family of distributions is generated
by varying your parameters. Let's look at some plots and see what's
going on with these particular parameters for the normal distribution.
As we have the mean constant and vary the standard deviation: you'll
see that as you increase the standard deviation
the plot will broaden out or flatten out. Now
if we vary the mean and keep constant the standard deviation you'll
have a shift: left if the mean is going to a smaller
value and to the right with a larger value.
Now in the real world you'll be given some data that's observed
and you need to find some kind of model that best describes this
all right. And you're not always in a position to transform your data
log reciprocal, square root, or whatever the transformation may be.
So if you would like to use the maximum likelihood
estimation: remember that if you have a distribution in mind
and you're looking for the parameters for your data. That's the gist of why
you're going to use this. The other thing that needs to be
mentioned: well why can't i in this circumstance just generate the
mean and the standard deviation from the data.
Well that will not necessarily give you the best approximation or estimate for
the population. That will give you what you have for
your current sample. Now, let's think about this. We're looking
at this as the likelihood of some values x which is your observed
data, and your parameters, all right. Where the L
will stand for the likelihood. And it's just some true parameters of theta
given your observed data. The right side is a function of the data
given a particular set of parameters: it's the reverse.
The semicolon is a distinction and not the same as if you use the bar
for conditional probability, where you would say if i had a bar right here
instead of the semicolon. You say given some parameters theta what is my x
and you do your probabilities based on that, right. Multiplying them across but,
the distinction is i'm letting you know that we have our
parameters here, all right. Now let's do some math for
that MLE since that's what you came for. At
least i hope you did. But, you have some nasty truth: the first
thing you do not necessarily have to have
a unique value for the MLE, and you also don't even need one to exist. But, why?
That's because of concavity think about this in terms of
if you had a function that was multimodal
like it just had a bunch of peaks and troughs, that's a situation
where you would have issues with concavity, okay. This comes up a lot in
machine learning for instance. And it's a big topic and a big deal and
there's ways to deal with this. If you have situations such as this
approximation methods would be best suited for you.
Now we're going to get into some super fun times
but not just any kind super fun happy times with our little panda friend [i meant polar bear]. So
we're going to start with our likelihood. And we're going to pull this right side and
we need to look for the instance of each x,
all right. So each sample piece of data x1, x2
all the way to xn right. And we're going to multiply all these together
for conditional probability, all right. But, now we got to do a little bit of
trickery. And the great thing is we're going to
convert all of these multiplications into additions and use the log.
And we're using a log for a few reasons one it's not going to change or affect
our distribution, the second thing is anytime you have the
opportunity to change multiplication into an addition
you're in good business when it's computing power as a side note. So
we're going to run those multiplications for each
value of x we're going to find our given theta. We're going to take the log of
this and separate everything, all right. And then we have to worry
about using some derivatives. And the reason we're going to use the
derivatives is because, we're trying to find one with the first
derivative where our peak will have that horizontal
line equal to zero. And then we're saying bazinga we're
there we're getting our little point we're able to find the best parameter
that we're looking for currently. but, then you need to use that
second derivative test because you need to see if you got a
negative slope going on there. And that's something you need because,
you got to remember concavity right. So this looks oversimplified and we're
going to get into this a little bit. But, you're going to notice something if
you have that suspected distribution you would like to replace
it: you would put it right in here, okay. Now, i need to mention
something: some really good approximation methods
if you're in a position where you either have highly parameterized
data or it's non-convex. Here's a few methods that you can look at
which are really useful and will come up quite often if you're doing any kind of
machine learning or any kind of non-parametric type of regression or
something like that you know. So let's assume that i made up some data
some fake received IQ data for people who voted
for a particular president with a weird tan and goofy looking hair.
And let's just assume that they have some really terrible IQ's,
okay. This guy over here he's probably the best one or her
who knows. But then you got some really low balls. You say oh no: what's wrong
with these people they don't seem very bright,
and you plot this out and you do three trends with two transformations.
You look at the original data to get an idea of what it looks like.
You do a log transform to see if that might clean it up.
And you do a square root and you say oh the square root's not looking too bad
it's got a long tail though on the left side, same with this one here.
But, it's a little deceiving. So let's get a little clearer picture. So
we plot this again we say okay this distribution is looking interesting.
I still notice i got this tail on the original data,
my confidence intervals here look kind of terrible.
I do that square root and that's looking you know almost the same.
Except i got this hump. That's kind of notched in here right; and say well
how should i deal with this? My data does look like it has a trend which would be
normal. But, how can we maximize our parameters for our given standard
deviation and the mean shifting. So let's figure
out what we can do with this? Sometimes you may have a distribution
that you feel may likely work with the data you
have and then later on you may reconsider and
figure out well i might have been wrong when you re-plot. Let's get into
this math once again. So we're taking our parameters with our
given data. We're throwing in the exponential
function and then for each value of x we're going
to run it through our exponential function.
And we need to find the best value of our
standard deviation and our mean. And since you're doing this for each
individual x we're also going to multiply this
through and expand this out from x1 to xn.
Take the log of it like we mentioned before, here.
Since, we're going to use the log all these multiplications
get that trusty old addition. We need to simplify our data.
So if i'm only showing the first instance of what's going
on. You're going to now take this and since you have these two which
this and this were multiplication. You're able to expand that into an addition now.
We're able to take this from this denominator
and bring it out. And now we're going to expand this once again and separate
these, okay. And we need to repeat the process
for x2 to x of n, okay. And that's what you'll end up with going
on here. Because, you have n occurrences: for all your x values
that's why these n values are here, okay. And we're basically summing up each one
of these for each individual value of x as well.
But, now we need to take that derivative. The first we'll do
is with respect to the mean. This is what we're going to use for
taking the derivative: the first two portions here
do not have the mu or the mean that's why these are cancel out to zero
and you're only left with this, okay. We gotta do a little work with the chain
rule: where we're bringing up bringing this superscript here down
it'll cancel out in here. And then we would have a
negative one that would come from this mu which changes the sign that you
noticed here. All right and that's what you're left
with. Now, to expand through this we have n occurrences of mu. And we still
have our sum. And if you notice this: the only thing
you get back is your sample mean, all right. So you're basically adding all
the values of your x and dividing by the number of values.
Okay, fine. We get our sample mean. Now, the same thing that we do for the
standard deviation: we start with this. We're finding anything that has our
trusty old friend the variance. But, what happened here? So
since you have the log. The log becomes one
over. So you have n over this. Since, this is in the denominator it
becomes a three and you have to take out the sign. So it'd
be you you're in the denominator should be
a negative 2 that comes out. This becomes a negative 3.
So that you end up getting that plus and then this just treating it as a constant.
You got to do some manipulation, you got to bring this..
There you are. You got to bring this over then you multiply by this.
Cancels out. You only have one left. Get rid of the
squared value. That's where you get the square root from and if you notice.
I put something as a distinction: this is the bias estimator,
all right. The unbiased estimator would have been
n minus one. So keep that in mind and try not to get confused, all right.
If you want to see a proof: it's in this right here.
So go on my Github and pull this code i left a pdf as well as
the Jupiter notebook. I decided to look at the
CDF to see how this looks towards a normal distribution.
So it's looking okay. That's fine. Tails are kind of long though.
And i say: can we directly compute the mean
and the standard deviation from the data. But, this doesn't really mean
like i said before: that we're representing the population of the data.
That's only with respect to the sample. Hence, the use
of the MLE. So what did i do here? I decided to plot
in red the original data. And this vertical line
here: is the mean of the original data. And the shifted new values of our
fitted MLE, okay. This plot right here though; is kind of an
issue. Because, i initially used a random variable to generate our new
fitted. These values here are your MLE values
for your mean and your standard deviation. I just
didn't shorten up the significant figures. So i decided let's
plot this again but standardize our data and not have a
random distribution created for our new fitted
data. So now i created this new data where the
original data is our green. With our green line with the long tails
and this red one is now our fitted data with the original mean
and the blue vertical line so we could see the shifting.
So this is what it would look like with the new
fitted MLE model. And if you notice these tails
look extremely short, and they're stopping at the
edge of each one of these data. I said okay, fine.
Let's see how this looks under the new CDF. And i plotted that and you see that
it does look like a normal distribution.
The hump right here and this little tail towards one are only a
reflection of what our data look like with respect to the
new mean and standard deviation. Which is basically giving us the
best model for what we're given currently. Some last thoughts:
you can't always assume your data will fit an exact model.
Approximation methods are usually a good call on real data.
And don't blindly believe everything you hear or everything you see,
alright. Please, like share and subscribe and if you subscribe
turn on that notification bell. I greatly appreciate it thank you for watching
i'll see you in the next video. Bye