CASSIE KOZYRKOV: Regression was first
used in 1805 by Legendre, who doesn't have a nice painting
of him on Wikipedia.
So I picked the one for the second person
to use it in 1807, Gauss.
So that's more than 200 years ago.
This has been around for a really long time.
And it's still a really popular standard approach
to machine learning today.
So let's have a look.
For our example, we will be making smoothies.
Every smoothie will have one cup of plain yogurt,
one secret ingredient, and a bunch of random food dyes
so that you cannot tell what is in there.
I blend it up.
I serve it to you.
And your goal now is to shout at me, what are
the calories in this smoothie?
Go.
What do I hear?
AUDIENCE: 100.
CASSIE KOZYRKOV: 100, going once, going twice.
AUDIENCE: 300.
CASSIE KOZYRKOV: 300.
AUDIENCE: 500.
CASSIE KOZYRKOV: 500.
Now, I'm doing a little test here.
I'm seeing who's got the strongest
aptitude in the audience for machine learning.
Because the machine learning attitude is not
the attitude of perfectionism, of think carefully
and get the answer right on the first go,
and don't even try it unless it's going to be perfect.
No.
The attitude is go for it, see what happens.
Get it wrong, because you're going to have
to do it over and over again.
You're going to fail, fall down over and over again.
Pick yourself up.
Try again until finally, it works.
And you have to try something to know what to do next.
And so if you're too afraid to start,
you tend not to do well in applied machine learning.
So those of you who are brave enough to just go for it, even
though you didn't know the right answer, yeah, that was what's
going to get me to move to the next slide
and give you more information.
And we'll be able to iterate towards correct together.
OK, so you've unlocked the next slide.
Turns out this was-- what was the secret ingredient?
58 delicious grams of sardines.
With 265 calories in there, lovely.
I see you don't want to drink it.
Fine.
I make you a different one.
Guess the calories now.
AUDIENCE: 80.
AUDIENCE: 400.
CASSIE KOZYRKOV: 80, 400.
AUDIENCE: 180.
CASSIE KOZYRKOV: 180.
All right, I'm hearing some more responses and rumblings.
Now, let me tell you what was in here.
50 grams of not delicious waffles that I've shown you
in the picture, no, no.
The raw stuff, waffle mix.
And this is actually in homage to a standard hazing experience
that happens on the fifth floor of the Google office
here in New York, which is most new Googlers, myself included,
I've done this.
You done this, Sam?
You walk up.
And you see this delicious smoothie.
And you try it.
And it is not a smoothie.
So standard newbie hazing experience here.
OK, so you do want to drink this one.
Fine, I make you a third one.
But before I make you guess, I'm going to give you
a little more information.
This is the 16th smoothie that I've made.
And here are the 15 previous calorie amounts.
Your first reaction to this should
be not trying to read it.
But instead, stamping your foot and saying, I'm a human.
I don't add numbers up.
That's what machines are for.
Indeed, correct.
So the machine is going to help us out and summarize them.
And we find out that the average is 237.
And the middle thing is 240.
Have a guess now as to the calories of this one.
AUDIENCE: [INAUDIBLE]
CASSIE KOZYRKOV: 237 I'm hearing, 238, 240.
Notice how your range has narrowed a whole lot,
as I give you more information.
That's going to be key to our discussion.
OK, I'll go for 237.
Ah, it turns out to be 88 grams of quiche Lorraine clocking in
at a whopping 445 calories and proving
to us that any liquid diet is beatable.
So how did we do?
We had an underestimate of 208 calories with that strategy.
Can we do better?
Well, maybe it would help to have the weight
of the secret ingredient.
So what we'll do as analysts is we'll make a little plot.
And we'll have the weight from 0 to 200 grams on the x-axis
and calories from 0 to 500 on the y-axis.
And as we look at this, we ask ourselves,
is there anything going on?
Or is weight completely useless here?
What do you say?
Completely useless?
Something to it.
And since there's something to it, maybe we'll go ahead
and we'll try it out.
Shall we?
And regression is about putting lines through stuff.
So why don't we put a line through it.
Like that line?
Good line?
We missed.
Where should we put this line?
By the way, those who forgot how lines work,
here's a handy cheat sheet.
The recipe, the model, is simply intercept
plus slope times the weight.
So we'll have two numbers here in red
that we have to find, or discover,
that determine where the line goes.
And then we pop the weight in there
and read the calories off of it.
So that's what we're dealing with, straightforward stuff.
So you don't like my line.
OK, fair enough.
How about that one?
You don't like that one.
This one?
Tough crowd.
That one?
No.
This one?
You look offended.
Why do you look offended by this?
Because let me remind you of something real quick.
This right here, this is your previous model.
And now you're offended.
But before, you were very happy to just guest 237 every time.
I think 237 and 0 times the weight
means I ignore the weight and say 237 every time.
And you were so fine to use it then.
Check your feature engineering privilege.
What is feature engineering?
That's when we create inputs that we
might use to learn from.
Stuff that might inform our solution.
And when we have a feature that we might use,
suddenly we can do better.
So that's kind of going to be the point here.
As we get information that's going
to be relevant to solving this thing,
we expect to do better and better.
And it's going to be up to us to think about what information
might be worth putting into the system in the first place.
OK, so let's do better, if we can do better.
We're going to find a line that's as close to the points
as possible.
Because you seemed offended when I wasn't doing that.
Fine.
So we better define a notion of an error.
What's an error?
It's just the point minus the line.
The truth minus what we were saying we should predict.
And because we all have type A personalities here,
we hate errors.
So we are going to minimize a function of these errors.
And that is going to be the score
for how this thing performs.
And a good score is to have fewer errors.
And we will use the RMSE here.
And this thing is very creatively named.
It's actually the recipe for calculating it backwards.
Get the error, square the error, take the mean or average,
and take the square root of that.
Root mean squared error, or RMSE.
OK.
Or you could just think of the RMSE
as the calories we're off by on average.
Not technically correct, but good enough
for government work.
We're off by about maybe five calories on average.
And that'll do.
That's fine for a morning.
OK, so now we're off by 85.
And we want to be off by fewer.
So let's see if we can just jiggle those two red parameters
there in the recipe to make things better.
What do you want to do?
Maybe you want to increase the slope a bit?
OK, that's getting better.
Good.
How about we increase the slope a little more?
Not so much better.
Maybe now we want to move the intercept down.
So let's do that.
And what we could do is we could continue
tinkering with these two numbers in red
with slow boring tinkering.
Or we could get on with it.
So what are our options?
Just play by hand with those two numbers,
which is what we were doing.
Or we could use calculus.
But who even does calculus anymore?
So old school.
Or we could just use an optimization algorithm
to get us the answer immediately.
So what does an optimization algorithm do?
It says, for this setting, you are trying-- what's your goal--
you are trying to get this function right here
to be as low as possible.
Let me figure out for you what settings, on those two
parameters that I'm allowed to play with,
give me the best possible performance here.
And then it outputs you that answer.
And that's going to be in the heart of every machine learning
algorithm.
So we're going to need to know what the scoring is,
what the goal is.
And then it's going to figure out
where to set the free parameters in the recipe based on that.
OK, great.
And we have come a long way.
I'm impressed with this.
Nice.
Let's look at quiche Lorraine though.
It's taunting us.
We did not predict quiche Lorraine well.
How did we do?
Better.
Our underestimate has gone down.
OK.
Can we do even better?
Can we?
Well, there ain't no such thing as a free lunch,
she says at Google.
To do better, we actually need something else.
So what is it going to take?
We need-- want to guess this?
AUDIENCE: More information [INAUDIBLE]..
CASSIE KOZYRKOV: More information, exactly.
That was very nicely put.
Because that says two things at once,
more information in the sense of just more of the same data,
or more information in the sense of other kinds of information
that might be helpful.
That's more features.
Features in this setting refers to variables, attributes,
and other settings.
So I'm going to be generous.
And I'm going to give you the fat percentage.
How do you feel about that?
Is that going to be helpful?
All right, so let's see.
So this is what we were working with before.
This is our form.
And we just had to muck about with these two parameters here.
And we used an optimization algorithm to set them.
And notice, who here has taken a stats course of some kind?
You probably saw regression.
It was an entirely different story in that course,
the way that you were going about it.
It was all this other stuff about what
it means and p-values values and t-tests.
None of that is relevant here.
All we are trying to do is optimize
that function to set these two numbers.
So that the difference between the calories
predicted and the actual calories
is as good as possible.
That's it.
That's all there is to it here.
Much, much simpler than what the stats course did.
So there we go.
Those are our answers.
Lovely.
Now, it's not going to be a line anymore,
if we add fat percentage.
It's a bird.
It's a plane.
Same structure, just we add another one of these right
here.
And now, the only pertinent thing for you to realize
is that there are three widgets to tinker with.
Nice.
And plotting it there, we need to add a dimension.
I can't make heads or tails of this.
So a little shout out to better plotting.
How about adding color?
OK.
Still not enough.
Because I can't see what's going on.
The color is more calories.
It gets redder.
But I can't see what's going on with this fat percentage
and weight plane.
So OK, now I can kind see what's going on.
And if you're used to looking at these things,
you kind of see the punch line that's coming.
But even if you don't see it, let's
find out what happens if we run the algorithm.
So again, our job is to get those three numbers
to be as good as possible.
We have our three options.
I pick the getting on with it.
And so we have this mosquito netting
fitted through our data for us.
What is it trying to do?
Get as close to our points as possible.
And those are the numbers that do it.
Quiche Lorraine is still taunting us.
How did we do?
We are now off by 47 calories on average.
Before it was 49.
I don't know.
I'm not feeling it.
I had to double the number of ingredients into my recipe.
And all I get for that is this measly, tiny improvement?
Maybe after all, this isn't such a promising direction to go in.
Maybe I would prefer the simpler model.
But of course, don't worry about that too much.
You can always try both of them in a new data set
and see which one is working better.
So how are we doing so far using only the mean?
There's our underestimate.
With simple linear regression, that means one ingredient.
We've improved and only a slight improvement
with multiple linear regression.
That means more than one input into the model.
All right, I'll take a question.
AUDIENCE: Can you do a model that has
fat percentage times weight?
That's actually tells you how much fat content it is.
CASSIE KOZYRKOV: Can you do a model
that does fat percentage times weight?
Absolutely.
And there's a good reason to think about doing that.
Like, the bit about the water amount
of the secret ingredient.
We didn't even cover that.
Right?
If I just dilute the secret ingredient
I can mess about with the fat percentage.
So-- but the reasoning--
that's a good one.
It's a good one.
The reasoning around whether or not
you should go ahead and try a model like that is, does it
take a long time to try it?
If no, just do it.
Even if you don't have a solid of a logical argument,
try it anyway.
You can always try something like that.
And yes, it's very, very easy to adjust the method
to input something like that.
You just make the feature by multiplying them first.
And in it goes.
Job done.
So certainly could try that.
OK, can we do even better, even better than that?
Sure.
How about if I'm really generous?
This is non-alcoholic.
And I'm going to give you the fat in grams,
the carbohydrate in grams, and the protein in grams
of our secret ingredient.
Does anyone want to spoil the punchline for us?
Who knows about nutrition?
Let's have it.
AUDIENCE: Fat times fat plus 4 times carbohydrates
plus 4 times--
CASSIE KOZYRKOV: Nine, four, and four, the magic numbers.
Because we as humanity have figured out a whole lot
about nutrition.
And we already know the answer.
Well, doesn't matter if we know the answer.
Why don't we see if we can recover that same answer
from this data set with machine learning?
But yeah, let's hope that our final answer
is going to look something like that answer.
So here we go.
This thing is now not a plane anymore.
It's now a hyperplane.
So let's go ahead and plot it.
Let's not.
There's no real reason to be plotting it here.
Because at the end of the day, the way that we check
how well we're doing, how we get that score that we're
optimizing, it happens the same way no matter what is in here.
All we need to know about is that we take what's predicted.
We compare it with the truth.
And we see how we did.
And inside this thing, however beastly it
is, there are some numbers that we are allowed to tinker with.
And we'll use an optimization algorithm
to do the tinkering for us.
And hopefully, some researcher somewhere
has written that optimization code, so we don't have to.
So that's a much better thing to spend your time on,
if you're in a research division, or research
in the industry, or research division in academia.
Good and they have.
And so we don't have to worry about it.
All we need to know is four things to tinker with.
We'll still assess the results the same way.
Doesn't matter what's there.
So let's get on with it and use that optimization algorithm,
which we don't need to know how it works.
And there you go.
How did we do?
Nine, four, four, something like that.
Looks pretty good to me.
FDA, by the way, confirms what you say.
This is a quote from their website.
Carbohydrate provides four calories
per gram, protein four, and fat nine.
So very good general knowledge.
What on earth is this 143 thing?
Well, we ask ourselves, if there is no fat, no carbs,
and no protein, if our secret ingredient was
water, what's left?
Well, there was yogurt in there every time.
Now, it turns out, our yogurt base had 145 calories.
We're predicting 143.
There's a little bit of a mismatch.
So maybe there was some error in our data.
But you know what?
Our score is getting pretty good, not perfect,
but pretty good.
We're off by four calories on average, when it used to be 47.
And this is, again, a shout out to feature engineering.
The fact that you had the knowledge about this domain
and realized that having the weights in grams
of all of these things was going to be useful,
that is the reason that you get a model that's
this good in the end.
It pays off.
You can convert your domain knowledge into good solutions.
And how did we do in the end?
Our underestimate for quiche Lorraine is only four calories.
I say, good enough.
Let's drink it.