ANNOUNCER: The following program is brought to you by Caltech.
ANNOUNCER: The following program is brought to you by Caltech.

YASER ABU-MOSTAFA: Welcome back.
YASER ABU-MOSTAFA: Welcome back.

Last time, we introduced the learning problem.
Last time, we introduced the learning problem.

And if you have an application in your domain that you wonder if machine
And if you have an application in your domain that you wonder if machine

learning is the right technique for it, we found that there are three
learning is the right technique for it, we found that there are three

criteria that you should check.
criteria that you should check.

You should ask yourself: is there a pattern to begin
You should ask yourself: is there a pattern to begin

with that we can learn?
with that we can learn?

And we realize that this condition can be intuitively met in many applications,
And we realize that this condition can be intuitively met in many applications,

even if we don't know mathematically what the pattern is.
even if we don't know mathematically what the pattern is.

The example we gave was the credit card approval.
The example we gave was the credit card approval.

There is clearly a pattern-- if someone has a particular salary, has
There is clearly a pattern-- if someone has a particular salary, has

been in a residence for so long, has that much debt, and so on, that this
been in a residence for so long, has that much debt, and so on, that this

is somewhat correlated to their credit behavior.
is somewhat correlated to their credit behavior.

And therefore, we know that the pattern exists in spite of the fact
And therefore, we know that the pattern exists in spite of the fact

that we don't know exactly what the pattern is.
that we don't know exactly what the pattern is.

The second item is that we cannot pin down the pattern mathematically, like
The second item is that we cannot pin down the pattern mathematically, like

the example I just gave.
the example I just gave.

And this is why we resort to machine learning.
And this is why we resort to machine learning.

The third one is that we have data that represents that pattern.
The third one is that we have data that represents that pattern.

In the case of the credit application, for example, there are historical
In the case of the credit application, for example, there are historical

records of previous customers, and we have the data they wrote in their
records of previous customers, and we have the data they wrote in their

application when they applied, and we have some years' worth of record of
application when they applied, and we have some years' worth of record of

their credit behavior.
their credit behavior.

So we have data that are going to enable us to correlate what they wrote in the
So we have data that are going to enable us to correlate what they wrote in the

application to their eventual credit behavior, and that is what we are
application to their eventual credit behavior, and that is what we are

going to learn from.
going to learn from.

Now, if you look at the three criteria, basically there are two that
Now, if you look at the three criteria, basically there are two that

you can do without, and one that is absolutely essential.
you can do without, and one that is absolutely essential.

What do I mean?
What do I mean?

Let's say that you don't have a pattern.
Let's say that you don't have a pattern.

Well, if you don't have a pattern, then you can try learning.
Well, if you don't have a pattern, then you can try learning.

And the only problem is that you will fail.
And the only problem is that you will fail.

That doesn't sound very encouraging.
That doesn't sound very encouraging.

But the idea here is that, when we develop the theory of learning, we will
But the idea here is that, when we develop the theory of learning, we will

realize that you can apply the technique regardless of whether there
realize that you can apply the technique regardless of whether there

is a pattern or not.
is a pattern or not.

And you are going to determine whether there's a pattern or not.
And you are going to determine whether there's a pattern or not.

So you are not going to be fooled and think, I learned, and then give the
So you are not going to be fooled and think, I learned, and then give the

system to your customer, and the customer will be disappointed.
system to your customer, and the customer will be disappointed.

There is something you can actually measure that will tell you whether you
There is something you can actually measure that will tell you whether you

learned or not.
learned or not.

So if there's no pattern, there is no harm done in trying machine learning.
So if there's no pattern, there is no harm done in trying machine learning.

The other one, also, you can do without.
The other one, also, you can do without.

Let's say that we can pin the thing down mathematically.
Let's say that we can pin the thing down mathematically.

Well, in that case, machine learning is not the recommended technique.
Well, in that case, machine learning is not the recommended technique.

It will still work.
It will still work.

It may not be the optimal technique.
It may not be the optimal technique.

If you can outright program it, and find the result perfectly, then why
If you can outright program it, and find the result perfectly, then why

bother generate examples, and try to learn, and go through all of that?
bother generate examples, and try to learn, and go through all of that?

But machine learning is not going to refuse.
But machine learning is not going to refuse.

It is going to learn, and it is going to give you a system.
It is going to learn, and it is going to give you a system.

It may not be the best system in this case, but it's a system nonetheless.
It may not be the best system in this case, but it's a system nonetheless.

The third one, I'm afraid you cannot do without.
The third one, I'm afraid you cannot do without.

You have to have data.
You have to have data.

Machine learning is about learning from data.
Machine learning is about learning from data.

And if you don't have data, there is absolutely nothing you can do.
And if you don't have data, there is absolutely nothing you can do.

So this is basically the picture about the context of machine learning.
So this is basically the picture about the context of machine learning.

Now, we went on to focus on one type, which is supervised learning.
Now, we went on to focus on one type, which is supervised learning.

And in the case of supervised learning, we have a target function.
And in the case of supervised learning, we have a target function.

The target function we are going to call f.
The target function we are going to call f.

That is our standard notation.
That is our standard notation.

And this corresponds, for example, to the credit application.
And this corresponds, for example, to the credit application.

x is your application, and f of x is whether you are a good credit risk or
x is your application, and f of x is whether you are a good credit risk or

not, for the bank.
not, for the bank.

So if you look at the target function, the main criterion about the target
So if you look at the target function, the main criterion about the target

function is that it's unknown.
function is that it's unknown.

This is a property that we are going to insist on.
This is a property that we are going to insist on.

And obviously, unknown is a very generous assumption, which means that
And obviously, unknown is a very generous assumption, which means that

you don't have to worry about what pattern you are trying to learn.
you don't have to worry about what pattern you are trying to learn.

It could be anything, and you will learn it-- if we manage to do that.
It could be anything, and you will learn it-- if we manage to do that.

There's still a question mark about that.
There's still a question mark about that.

But it's a good assumption to have, or lack of assumption, if you will,
But it's a good assumption to have, or lack of assumption, if you will,

because then you know that you don't worry about the environment that
because then you know that you don't worry about the environment that

generated the examples.
generated the examples.

You only worry about the system that you use to implement machine learning.
You only worry about the system that you use to implement machine learning.

Now, you are going to be given data.
Now, you are going to be given data.

And the reason it's called supervised learning is that you are not only
And the reason it's called supervised learning is that you are not only

given the input x's, as you can see here.
given the input x's, as you can see here.

You're also given the output--
You're also given the output--

the target outputs.
the target outputs.

So in spite of the fact that the target function is generally unknown,
So in spite of the fact that the target function is generally unknown,

it is known on the data that I give you.
it is known on the data that I give you.

This is the data that you are going to use as training examples, and that you
This is the data that you are going to use as training examples, and that you

are going to use to figure out what the target function is.
are going to use to figure out what the target function is.

So in the case of supervised learning, you have the targets
So in the case of supervised learning, you have the targets

explicitly.
explicitly.

In the other cases, you have less information than the target, and we
In the other cases, you have less information than the target, and we

talked about it-- like unsupervised learning, where you don't have
talked about it-- like unsupervised learning, where you don't have

anything, and reinforcement learning, where you have partial information,
anything, and reinforcement learning, where you have partial information,

which is just a reward or punishment for a choice of a value of y that
which is just a reward or punishment for a choice of a value of y that

may or may not be the target.
may or may not be the target.

Finally, you have the solution tools.
Finally, you have the solution tools.

These are the things that you're going to choose in order to solve the
These are the things that you're going to choose in order to solve the

problem, and they are called the learning model, as we discussed.
problem, and they are called the learning model, as we discussed.

They are the learning algorithm and the hypothesis set.
They are the learning algorithm and the hypothesis set.

And the learning algorithm will produce a hypothesis--
And the learning algorithm will produce a hypothesis--

the final hypothesis, the one that you are going to give your customer, and
the final hypothesis, the one that you are going to give your customer, and

we give the symbol g for that.
we give the symbol g for that.

And hopefully g approximates f, the actual target function,
And hopefully g approximates f, the actual target function,

which remains unknown.
which remains unknown.

And g is picked from a hypothesis set, and the general the symbol for
And g is picked from a hypothesis set, and the general the symbol for

a member of the hypothesis set is h.
a member of the hypothesis set is h.

So h is a generic hypothesis.
So h is a generic hypothesis.

The one you happen to pick, you are going to call g.
The one you happen to pick, you are going to call g.

Now, we looked at an example of a learning algorithm.
Now, we looked at an example of a learning algorithm.

First, the learning model-- the perceptron itself, which is a linear
First, the learning model-- the perceptron itself, which is a linear

function, thresholded.
function, thresholded.

That happens to be the hypothesis set.
That happens to be the hypothesis set.

And then, there is an algorithm that goes with it that chooses which
And then, there is an algorithm that goes with it that chooses which

hypothesis to report based on the data.
hypothesis to report based on the data.

And the hypothesis in this case is represented by the purple line.
And the hypothesis in this case is represented by the purple line.

Different hypotheses in the set H will result
Different hypotheses in the set H will result

in different lines.
in different lines.

Some of them are good and some of them are bad, in terms of separating
Some of them are good and some of them are bad, in terms of separating

correctly the examples which are the pluses and minuses.
correctly the examples which are the pluses and minuses.

And we found that there's a very simple rule to adjust the current
And we found that there's a very simple rule to adjust the current

hypothesis, while the algorithm is still running, in order to get a better
hypothesis, while the algorithm is still running, in order to get a better

hypothesis.
hypothesis.

And once you have all the points classified correctly, which is
And once you have all the points classified correctly, which is

guaranteed in the case of the perceptron learning algorithm if the
guaranteed in the case of the perceptron learning algorithm if the

data was linearly separable in the first place,
data was linearly separable in the first place,

then you will get there, and that will be the g that you are going to report.
then you will get there, and that will be the g that you are going to report.

Now, we ended the lecture on sort of a sad note, because after all of this
Now, we ended the lecture on sort of a sad note, because after all of this

encouragement about learning, we asked ourselves: well,
encouragement about learning, we asked ourselves: well,

can we actually learn?
can we actually learn?

So we said it's an unknown function.
So we said it's an unknown function.

Unknown function is an attractive assumption, as I said.
Unknown function is an attractive assumption, as I said.

But can we learn an unknown function, really?
But can we learn an unknown function, really?

And then we realized that if you look at it, it's really impossible.
And then we realized that if you look at it, it's really impossible.

Why is it impossible?
Why is it impossible?

Because I'm going to give you a finite data set, and I'm going to give you
Because I'm going to give you a finite data set, and I'm going to give you

the value of the function on this set.
the value of the function on this set.

Good.
Good.

Now, I'm going to ask you what is the function outside that set?
Now, I'm going to ask you what is the function outside that set?

How in the world are you going to tell what the function is outside, if the
How in the world are you going to tell what the function is outside, if the

function is genuinely unknown?
function is genuinely unknown?

Couldn't it assume any value it wants?
Couldn't it assume any value it wants?

Yes, it can.
Yes, it can.

I can give you 1000 points, a million points, and on the million-and-first point,
I can give you 1000 points, a million points, and on the million-and-first point,

still the function can behave any way it wants.
still the function can behave any way it wants.

So it doesn't look like the statement we made is feasible in terms of
So it doesn't look like the statement we made is feasible in terms of

learning, and therefore we have to do something about it.
learning, and therefore we have to do something about it.

And what we are going to do about it is the subject of this lecture.
And what we are going to do about it is the subject of this lecture.

Now, the lecture is called Is Learning Feasible?
Now, the lecture is called Is Learning Feasible?

And I am going to address this question in extreme detail from
And I am going to address this question in extreme detail from

beginning to end.
beginning to end.

This is the only topic for this lecture.
This is the only topic for this lecture.

Now, if you want an outline--
Now, if you want an outline--

it's really a logical flow.
it's really a logical flow.

But if you want to cluster it into points--
But if you want to cluster it into points--

we are going to start with a probabilistic situation, that is a very
we are going to start with a probabilistic situation, that is a very

simple probabilistic situation.
simple probabilistic situation.

It doesn't seem to relate to learning.
It doesn't seem to relate to learning.

But it will capture the idea--
But it will capture the idea--

can we say something outside the sample data that we have?
can we say something outside the sample data that we have?

So we're going to answer it in a way that is concrete, and where the
So we're going to answer it in a way that is concrete, and where the

mathematics is very friendly.
mathematics is very friendly.

And then after that, I'm going to be able to relate that probabilistic
And then after that, I'm going to be able to relate that probabilistic

situation to learning as we stated.
situation to learning as we stated.

It will take two stages.
It will take two stages.

First, I will just translate the expressions into something that
First, I will just translate the expressions into something that

relates to learning, and then we will move forward and make it correspond to
relates to learning, and then we will move forward and make it correspond to

real learning.
real learning.

That's the last one.
That's the last one.

And then after we do that, and we think we are done, we find that there is
And then after we do that, and we think we are done, we find that there is

a serious dilemma that we have.
a serious dilemma that we have.

And we will find a solution to that dilemma, and then declare victory-- that
And we will find a solution to that dilemma, and then declare victory-- that

indeed, learning is feasible in a very particular sense.
indeed, learning is feasible in a very particular sense.

So let's start with the experiment that I talked about.
So let's start with the experiment that I talked about.

Consider the following situation.
Consider the following situation.

You have a bin, and the bin has marbles.
You have a bin, and the bin has marbles.

The marbles are either red or green.
The marbles are either red or green.

That's what it looks like.
That's what it looks like.

And we are going to do an experiment with this bin.
And we are going to do an experiment with this bin.

And the experiment is to pick a sample from the bin--
And the experiment is to pick a sample from the bin--

some marbles.
some marbles.

Let's formalize what the probability distribution is.
Let's formalize what the probability distribution is.

There is a probability of picking a red marble, and let's call it mu.
There is a probability of picking a red marble, and let's call it mu.

So now you think of mu as the probability of a red marble.
So now you think of mu as the probability of a red marble.

Now, the bin is really just a visual aid to make us relate to the
Now, the bin is really just a visual aid to make us relate to the

experiment.
experiment.

You can think of this abstractly as a binary experiment--
You can think of this abstractly as a binary experiment--

two outcomes, red or green.
two outcomes, red or green.

Probability of red is mu, independently from
Probability of red is mu, independently from

one point to another.
one point to another.

If you want to stick to the bin, you can say the bin has an infinite number
If you want to stick to the bin, you can say the bin has an infinite number

of marbles and the fraction of red marbles is mu.
of marbles and the fraction of red marbles is mu.

Or maybe it has a finite number of marbles, and you are going to pick the
Or maybe it has a finite number of marbles, and you are going to pick the

marbles, but replace them.
marbles, but replace them.

But the idea now is that every time you reach in the bin, the probability
But the idea now is that every time you reach in the bin, the probability

of picking a red marble is mu.
of picking a red marble is mu.

That's the rule.
That's the rule.

Now, there's a probability of picking a green marble.
Now, there's a probability of picking a green marble.

And what might that be?
And what might that be?

That must be 1 minus mu.
That must be 1 minus mu.

So that's the setup.
So that's the setup.

Now, the value of mu is unknown to us.
Now, the value of mu is unknown to us.

So in spite of the fact that you can look at this particular bin and see
So in spite of the fact that you can look at this particular bin and see

there's less red than green, so mu must be small.
there's less red than green, so mu must be small.

and all of that.
and all of that.

You don't have that advantage in real.
You don't have that advantage in real.

The bin is opaque-- it's sitting there, and I reach for it like this.
The bin is opaque-- it's sitting there, and I reach for it like this.

So now that I declare mu is unknown, you probably see
So now that I declare mu is unknown, you probably see

where this is going.
where this is going.

Unknown is a famous word from last lecture, and that will be the link to
Unknown is a famous word from last lecture, and that will be the link to

what we have.
what we have.

Now, we pick N marbles independently.
Now, we pick N marbles independently.

Capital N. And I'm using the same notation for N, which is the
Capital N. And I'm using the same notation for N, which is the

number of data points in learning, deliberately.
number of data points in learning, deliberately.

So the sample will look like this.
So the sample will look like this.

And it will have some red and some green.
And it will have some red and some green.

It's a probabilistic situation.
It's a probabilistic situation.

And we are going to call the fraction of marbles in the sample--
And we are going to call the fraction of marbles in the sample--

this now is a probabilistic quantity.
this now is a probabilistic quantity.

mu is an unknown constant sitting there.
mu is an unknown constant sitting there.

If you pick a sample, someone else picks a sample, you will have a different
If you pick a sample, someone else picks a sample, you will have a different

frequency in sample from the other person.
frequency in sample from the other person.

And we are going to call it nu.
And we are going to call it nu.

Now, interestingly enough, nu also should appear in the figure.
Now, interestingly enough, nu also should appear in the figure.

So it says nu equals fraction of red marbles.
So it says nu equals fraction of red marbles.

So that's where it lies.
So that's where it lies.

Here is nu!
Here is nu!

For some reason that I don't understand, the app wouldn't show nu
For some reason that I don't understand, the app wouldn't show nu

in the figures.
in the figures.

So I decided maybe the app is actually a machine learning expert.
So I decided maybe the app is actually a machine learning expert.

It doesn't like things in sample.
It doesn't like things in sample.

It only likes things that are real.
It only likes things that are real.

So it knows that nu is not important.
So it knows that nu is not important.

It's not an indication.
It's not an indication.

We are really interested in knowing what's outside.
We are really interested in knowing what's outside.

So it kept the mu, but actually deleted the nu.
So it kept the mu, but actually deleted the nu.

At least that's what we are going to believe for the rest of the lecture.
At least that's what we are going to believe for the rest of the lecture.

Now, this is the bin.
Now, this is the bin.

So now, the next step is to ask ourselves the question we asked in
So now, the next step is to ask ourselves the question we asked in

machine learning.
machine learning.

Does nu, which is the sample frequency, tell us anything about mu,
Does nu, which is the sample frequency, tell us anything about mu,

which is the actual frequency in the bin that we are interested in knowing?
which is the actual frequency in the bin that we are interested in knowing?

The short answer--
The short answer--

this is to remind you what it is.
this is to remind you what it is.

The short answer is no.
The short answer is no.

Why?
Why?

Because the sample can be mostly green, while the bin is mostly red.
Because the sample can be mostly green, while the bin is mostly red.

Anybody doubts that?
Anybody doubts that?

The thing could have 90% red, and I pick 100 marbles, and all
The thing could have 90% red, and I pick 100 marbles, and all

of them happen to be green.
of them happen to be green.

This is possible, correct?
This is possible, correct?

So if I ask you what is actually mu, you really don't know from the sample.
So if I ask you what is actually mu, you really don't know from the sample.

You don't know anything about the marbles you did not pick.
You don't know anything about the marbles you did not pick.

Well, that's the short answer.
Well, that's the short answer.

The long answer is yes.
The long answer is yes.

Not because no and yes, but this is more elaborate.
Not because no and yes, but this is more elaborate.

We have to really discuss a lot in order to get there.
We have to really discuss a lot in order to get there.

So why is it yes?
So why is it yes?

Because if you know a little bit about probability, you realize that if the
Because if you know a little bit about probability, you realize that if the

sample is big enough, the sample frequency, which is nu-- the mysterious
sample is big enough, the sample frequency, which is nu-- the mysterious

disappearing quantity here-- that is likely to be close to mu.
disappearing quantity here-- that is likely to be close to mu.

Think of a presidential poll.
Think of a presidential poll.

There are maybe 100 million or more voters in the US, and you make a poll
There are maybe 100 million or more voters in the US, and you make a poll

of 3000 people.
of 3000 people.

You have 3000 marbles, so to speak.
You have 3000 marbles, so to speak.

And you look at the result in the marbles, and you tell me how the 100
And you look at the result in the marbles, and you tell me how the 100

million will vote.
million will vote.

How the heck did you know that?
How the heck did you know that?

So now the statistics come in.
So now the statistics come in.

That's where the probability plays a role.
That's where the probability plays a role.

And the main distinction between the two answers is
And the main distinction between the two answers is

possible versus probable.
possible versus probable.

In science and in engineering, you go a huge distance by settling for not
In science and in engineering, you go a huge distance by settling for not

absolutely certain, but almost certain.
absolutely certain, but almost certain.

It opens a world of possibilities, and this is one of the
It opens a world of possibilities, and this is one of the

possibilities that it opens.
possibilities that it opens.

So now we know that, from a probabilistic point of view, nu does
So now we know that, from a probabilistic point of view, nu does

tell me something about mu.
tell me something about mu.

The sample frequency tells me something about the bin.
The sample frequency tells me something about the bin.

So what does it exactly say?
So what does it exactly say?

Now we go into a mathematical formulation.
Now we go into a mathematical formulation.

In words, it says: in a big sample, nu, the sample frequency,
In words, it says: in a big sample, nu, the sample frequency,

should be close to mu, the bin frequency.
should be close to mu, the bin frequency.

So now, the symbols that go with that-- what is a big sample?
So now, the symbols that go with that-- what is a big sample?

Large N, our parameter N.
Large N, our parameter N.

And how do we say that nu is close to mu?
And how do we say that nu is close to mu?

We say that they are within epsilon.
We say that they are within epsilon.

That is our criterion.
That is our criterion.

Now, with this in mind, we are going to formalize this.
Now, with this in mind, we are going to formalize this.

The formula that I'm going to show you is a formula that is going to
The formula that I'm going to show you is a formula that is going to

stay with us for the rest of the course.
stay with us for the rest of the course.

I would like you to pay attention.
I would like you to pay attention.

And I'm going to build it gradually.
And I'm going to build it gradually.

We are going to say that the probability of something is small.
We are going to say that the probability of something is small.

So we're going to say that it's less than or equal to, and hopefully the
So we're going to say that it's less than or equal to, and hopefully the

right-hand side will be a small quantity.
right-hand side will be a small quantity.

Now if I am claiming that the probability of something is small, it
Now if I am claiming that the probability of something is small, it

must be that that thing is a bad event.
must be that that thing is a bad event.

I don't want it to happen.
I don't want it to happen.

So we have a probability of something bad happening being small.
So we have a probability of something bad happening being small.

What is a bad event in the context we are talking about?
What is a bad event in the context we are talking about?

It is that nu does not approximate mu well.
It is that nu does not approximate mu well.

They are not within epsilon of each other.
They are not within epsilon of each other.

And if you look at it, here you have nu minus mu in absolute value, so
And if you look at it, here you have nu minus mu in absolute value, so

that's the difference in absolute value.
that's the difference in absolute value.

That happens to be bigger than epsilon.
That happens to be bigger than epsilon.

So that's bad, because that tells us that they are further away from our
So that's bad, because that tells us that they are further away from our

tolerance epsilon.
tolerance epsilon.

We don't want that to happen.
We don't want that to happen.

And we would like the probability of that happening to
And we would like the probability of that happening to

be as small as possible.
be as small as possible.

Well, how small can we guarantee it?
Well, how small can we guarantee it?

Good news.
Good news.

It's e to the minus N.
It's e to the minus N.

It's a negative exponential.
It's a negative exponential.

That is great, because negative exponentials tend to die very fast.
That is great, because negative exponentials tend to die very fast.

So if you get a bigger sample, this will be diminishingly small
So if you get a bigger sample, this will be diminishingly small

probability.
probability.

So the probability of something bad happening will be very small, and we
So the probability of something bad happening will be very small, and we

can claims that, indeed, nu will be within epsilon from mu, and we will be
can claims that, indeed, nu will be within epsilon from mu, and we will be

wrong for a very minute amount of the time.
wrong for a very minute amount of the time.

But that's the good news.
But that's the good news.

Now the bad news--
Now the bad news--

ouch!
ouch!

Epsilon is our tolerance.
Epsilon is our tolerance.

If you're a very tolerant person, you say:
If you're a very tolerant person, you say:

I just want nu and mu to be within, let's say, 0.1.
I just want nu and mu to be within, let's say, 0.1.

That's not very much to ask.
That's not very much to ask.

Now, the price you pay for that is that you plug in the exponent
Now, the price you pay for that is that you plug in the exponent

not epsilon, but epsilon squared.
not epsilon, but epsilon squared.

So that becomes 0.01.
So that becomes 0.01.

0.01 will dampen N significantly, and you lose a lot of the benefit of the
0.01 will dampen N significantly, and you lose a lot of the benefit of the

negative exponential.
negative exponential.

And if you are more stringent and you say, I really want nu
And if you are more stringent and you say, I really want nu

to be close to mu.
to be close to mu.

I am not fooling around here.
I am not fooling around here.

So I am going to pick epsilon to be 10 to the minus 6.
So I am going to pick epsilon to be 10 to the minus 6.

Good for you.
Good for you.

10 to the minus 6?
10 to the minus 6?

Pay the price for it.
Pay the price for it.

You go here, and now that's 10 to the minus 12.
You go here, and now that's 10 to the minus 12.

That will completely kill any N you will ever encounter.
That will completely kill any N you will ever encounter.

So the exponent now will be around zero.
So the exponent now will be around zero.

So this probability will be around 1, if that was the final answer.
So this probability will be around 1, if that was the final answer.

That's not yet the final answer.
That's not yet the final answer.

So now, you know that the probability is less than or equal to 1.
So now, you know that the probability is less than or equal to 1.

Congratulations!
Congratulations!

You knew that already. [LAUGHTER]
You knew that already. [LAUGHTER]

Well, this is almost the formula, but it's not quite.
Well, this is almost the formula, but it's not quite.

What we need is fairly trivial.
What we need is fairly trivial.

We just put 2 here, and 2 there.
We just put 2 here, and 2 there.

Now, between you and me, I prefer the original formula
Now, between you and me, I prefer the original formula

better, without the 2's.
better, without the 2's.

However, the formula with the 2's has the distinct advantage of being: true. [LAUGHTER]
However, the formula with the 2's has the distinct advantage of being: true. [LAUGHTER]

So we have to settle for that.
So we have to settle for that.

Now that inequality is called Hoeffding's Inequality.
Now that inequality is called Hoeffding's Inequality.

It is the main inequality we are going to be using in the course.
It is the main inequality we are going to be using in the course.

You can look for the proof.
You can look for the proof.

It's a basic proof in mathematics.
It's a basic proof in mathematics.

It's not that difficult, but definitely not trivial.
It's not that difficult, but definitely not trivial.

And we are going to use it all the way-- and this is the same formula
And we are going to use it all the way-- and this is the same formula

that will get us to prove something about the VC dimension.
that will get us to prove something about the VC dimension.

If the buzzword 'VC dimension' means anything to you, it will come from
If the buzzword 'VC dimension' means anything to you, it will come from

this after a lot of derivation.
this after a lot of derivation.

So this is the building block that you have to really know cold.
So this is the building block that you have to really know cold.

Now, if you want to translate the Hoeffding Inequality into words, what
Now, if you want to translate the Hoeffding Inequality into words, what

we have been talking about is that we would like to make the
we have been talking about is that we would like to make the

statement: mu equals nu.
statement: mu equals nu.

That would be the ultimate.
That would be the ultimate.

I look at the in-sample frequency, that's the out-of-sample frequency.
I look at the in-sample frequency, that's the out-of-sample frequency.

That's the real frequency out there.
That's the real frequency out there.

But that's not the case.
But that's not the case.

We actually are making the statement mu equals nu, but we're not
We actually are making the statement mu equals nu, but we're not

making the statement--
making the statement--

we are making a PAC statement.
we are making a PAC statement.

And that stands for: this statement is probably, approximately, correct.
And that stands for: this statement is probably, approximately, correct.

Probably because of this.
Probably because of this.

This is small, so the probability of violation is small.
This is small, so the probability of violation is small.

Approximately because of this.
Approximately because of this.

We are not saying that mu equals nu.
We are not saying that mu equals nu.

We are saying that they are close to each other.
We are saying that they are close to each other.

And that theme will remain with us in learning.
And that theme will remain with us in learning.

So we put the glorified Hoeffding's Inequality at the top, and we spend
So we put the glorified Hoeffding's Inequality at the top, and we spend

a viewgraph analyzing what it means.
a viewgraph analyzing what it means.

In case you forgot what nu and mu are, I put the figure.
In case you forgot what nu and mu are, I put the figure.

So mu is the frequency within the bin.
So mu is the frequency within the bin.

This is the unknown quantity that we want to tell.
This is the unknown quantity that we want to tell.

And nu is the disappearing quantity which happens to be the frequency in
And nu is the disappearing quantity which happens to be the frequency in

the sample you have.
the sample you have.

So what about the Hoeffding Inequality?
So what about the Hoeffding Inequality?

Well, one attraction of this inequality is that it is valid for
Well, one attraction of this inequality is that it is valid for

every N, positive integer, and every epsilon which is greater than zero.
every N, positive integer, and every epsilon which is greater than zero.

Pick any tolerance you want, and for any number of examples you
Pick any tolerance you want, and for any number of examples you

want, this is true.
want, this is true.

It's not an asymptotic result.
It's not an asymptotic result.

It's a result that holds for every N and epsilon.
It's a result that holds for every N and epsilon.

That's a very attractive proposition for something that has
That's a very attractive proposition for something that has

an exponential in it.
an exponential in it.

Now, Hoeffding Inequality belongs to a large class of mathematical laws,
Now, Hoeffding Inequality belongs to a large class of mathematical laws,

which are called the Laws of Large Numbers.
which are called the Laws of Large Numbers.

So this is one law of large numbers, one form of it, and
So this is one law of large numbers, one form of it, and

there are tons of them.
there are tons of them.

This happens to be one of the friendliest, because it's not
This happens to be one of the friendliest, because it's not

asymptotic, and happens to have an exponential in it.
asymptotic, and happens to have an exponential in it.

Now, one observation here is that if you look at the left-hand side, we are
Now, one observation here is that if you look at the left-hand side, we are

computing this probability.
computing this probability.

This probability patently depends on mu.
This probability patently depends on mu.

mu appears explicitly in it, and also mu affects the probability
mu appears explicitly in it, and also mu affects the probability

distribution of nu.
distribution of nu.

Nu is the sample, in N marbles you picked.
Nu is the sample, in N marbles you picked.

That's a very simple binomial distribution.
That's a very simple binomial distribution.

You can find the probability that nu equals anything based on
You can find the probability that nu equals anything based on

the value of mu.
the value of mu.

So the probability that this quantity, which depends on mu, exceeds epsilon--
So the probability that this quantity, which depends on mu, exceeds epsilon--

the probability itself does depend on mu.
the probability itself does depend on mu.

However, we are not interested in the exact probability.
However, we are not interested in the exact probability.

We just want to bound it.
We just want to bound it.

And in this case, we are bounding it uniformly.
And in this case, we are bounding it uniformly.

As you see, the right-hand side does not have mu in it.
As you see, the right-hand side does not have mu in it.

And that gives us a great tool, because now we don't use the quantity
And that gives us a great tool, because now we don't use the quantity

that, we already declared, is unknown.
that, we already declared, is unknown.

mu is unknown.
mu is unknown.

It would be a vicious cycle if I go and say that it depends on mu,
It would be a vicious cycle if I go and say that it depends on mu,

but I don't know what mu is.
but I don't know what mu is.

Now you know uniformly, regardless of the value of mu-- mu could be anything
Now you know uniformly, regardless of the value of mu-- mu could be anything

between 0 and 1, and this will still be bounding the deviation of the
between 0 and 1, and this will still be bounding the deviation of the

sample frequency from the real frequency.
sample frequency from the real frequency.

That's a good advantage.
That's a good advantage.

Now, the other point is that there is a trade-off that you can read off the
Now, the other point is that there is a trade-off that you can read off the

inequality.
inequality.

What is the trade-off?
What is the trade-off?

The trade-off is between N and epsilon.
The trade-off is between N and epsilon.

In a typical situation, if we think of N as the number of examples that are
In a typical situation, if we think of N as the number of examples that are

given to you-- the amount of data-- in this case, the number of marbles out
given to you-- the amount of data-- in this case, the number of marbles out

of the bin,
of the bin,

N is usually dictated.
N is usually dictated.

Someone comes and gives you a certain resource of examples.
Someone comes and gives you a certain resource of examples.

Epsilon is your taste in tolerance.
Epsilon is your taste in tolerance.

You are very tolerant. You pick epsilon equals 0.5.
You are very tolerant. You pick epsilon equals 0.5.

That will be very easy to satisfy.
That will be very easy to satisfy.

And if you are very stringent, you can pick epsilon smaller and smaller.
And if you are very stringent, you can pick epsilon smaller and smaller.

Now, because they get multiplied here, the smaller the epsilon is, the bigger
Now, because they get multiplied here, the smaller the epsilon is, the bigger

than N you need in order to compensate for it and come up with the same level
than N you need in order to compensate for it and come up with the same level

of probability bound.
of probability bound.

And that makes a lot of sense.
And that makes a lot of sense.

If you have more examples, you are more sure that nu and mu will be close
If you have more examples, you are more sure that nu and mu will be close

together, even closer and closer and closer,
together, even closer and closer and closer,

as you get larger N.
as you get larger N.

So this makes sense.
So this makes sense.

Finally,
Finally,

it's a subtle point, but it's worth saying.
it's a subtle point, but it's worth saying.

We are making the statement that nu is approximately the same as mu.
We are making the statement that nu is approximately the same as mu.

And this implies that mu is approximately the same as nu.
And this implies that mu is approximately the same as nu.

What is this?
What is this?

The logic here is a little bit subtle.
The logic here is a little bit subtle.

Obviously, the statement is a tautology, but I'm just making
Obviously, the statement is a tautology, but I'm just making

a logical point, here.
a logical point, here.

When you run the experiment, you don't know what mu is.
When you run the experiment, you don't know what mu is.

mu is an unknown.
mu is an unknown.

It's a constant.
It's a constant.

The only random fellow in this entire operation is nu.
The only random fellow in this entire operation is nu.

That is what the probability is with respect to.
That is what the probability is with respect to.

You generate different samples, and you compute the probability.
You generate different samples, and you compute the probability.

This is the probabilistic thing.
This is the probabilistic thing.

This is a happy constant sitting there, albeit unknown.
This is a happy constant sitting there, albeit unknown.

Now, the way you are using the inequality is to infer mu, the sample
Now, the way you are using the inequality is to infer mu, the sample

here, from nu.
here, from nu.

That is not the cause and effect that actually takes place.
That is not the cause and effect that actually takes place.

The cause and effect is that mu affects nu, not the other way around.
The cause and effect is that mu affects nu, not the other way around.

But we are using it the other way around.
But we are using it the other way around.

Lucky for us, the form of the probability is symmetric.
Lucky for us, the form of the probability is symmetric.

Therefore, instead of saying that nu tends to be close to mu, which will
Therefore, instead of saying that nu tends to be close to mu, which will

be the accurate logical statement-- mu is there, and nu has a tendency to be
be the accurate logical statement-- mu is there, and nu has a tendency to be

close to it.
close to it.

We, instead of that, say that I know already nu, and now mu tends to
We, instead of that, say that I know already nu, and now mu tends to

be close to nu.
be close to nu.

That's the logic we are using.
That's the logic we are using.

Now, I think we understand what the bin situation is, and we know what the
Now, I think we understand what the bin situation is, and we know what the

mathematical condition that corresponds to it is.
mathematical condition that corresponds to it is.

What I'd like to do,
What I'd like to do,

I'd like to connect that to the learning problem we have.
I'd like to connect that to the learning problem we have.

In the case of a bin, the unknown quantity that we want to decipher is
In the case of a bin, the unknown quantity that we want to decipher is

a number, mu.
a number, mu.

Just unknown.
Just unknown.

What is the frequency inside the bin.
What is the frequency inside the bin.

In the learning situation that we had, the unknown quantity we would like to
In the learning situation that we had, the unknown quantity we would like to

decipher is a full-fledged function.
decipher is a full-fledged function.

It has a domain, X, that could be a 10th-order Euclidean space.
It has a domain, X, that could be a 10th-order Euclidean space.

Y could be anything.
Y could be anything.

It could be binary, like the perceptron.
It could be binary, like the perceptron.

It could be something else.
It could be something else.

That's a huge amount of information.
That's a huge amount of information.

The bin has only one number.
The bin has only one number.

This one, if you want to specify it, that's a lot of specification.
This one, if you want to specify it, that's a lot of specification.

So how am I going to be able to relate the learning problem to something that
So how am I going to be able to relate the learning problem to something that

simplistic?
simplistic?

The way we are going to do it is the following.
The way we are going to do it is the following.

Think of the bin as your input space in the learning problem.
Think of the bin as your input space in the learning problem.

That's the correspondence.
That's the correspondence.

So every marble here is a point x.
So every marble here is a point x.

That is a credit card applicant.
That is a credit card applicant.

So if you look closely at the gray thing, you will read: salary, years in
So if you look closely at the gray thing, you will read: salary, years in

residence, and whatnot.
residence, and whatnot.

You can't see it here because it's too small!
You can't see it here because it's too small!

Now the bin has all the points in the space. Therefore, this
Now the bin has all the points in the space. Therefore, this

is really the space.
is really the space.

That's the correspondence in our mind.
That's the correspondence in our mind.

Now we would like to give colors to the marbles.
Now we would like to give colors to the marbles.

So here are the colors.
So here are the colors.

There are green marbles, and they correspond to something in the
There are green marbles, and they correspond to something in the

learning problem.
learning problem.

What do they correspond to?
What do they correspond to?

They correspond to your hypothesis getting it right.
They correspond to your hypothesis getting it right.

So what does that mean?
So what does that mean?

There is a target function sitting there, right?
There is a target function sitting there, right?

You have a hypothesis.
You have a hypothesis.

The hypothesis is a full function, like the target function is.
The hypothesis is a full function, like the target function is.

You can compare the hypothesis to the target function on every point.
You can compare the hypothesis to the target function on every point.

And they either agree or disagree.
And they either agree or disagree.

If they agree, please color the corresponding point
If they agree, please color the corresponding point

in the input space--
in the input space--

Color it green.
Color it green.

Now, I'm not saying that you know which ones are green and which ones
Now, I'm not saying that you know which ones are green and which ones

are not, because you don't know the target function overall.
are not, because you don't know the target function overall.

I'm just telling you the mapping that takes an unknown target function into
I'm just telling you the mapping that takes an unknown target function into

an unknown mu.
an unknown mu.

So both of them are unknown, admittedly, but that's the
So both of them are unknown, admittedly, but that's the

correspondence that maps it.
correspondence that maps it.

And now you go, and there are some red ones.
And now you go, and there are some red ones.

And, you guessed it.
And, you guessed it.

You color the thing red if your hypothesis got the answer wrong.
You color the thing red if your hypothesis got the answer wrong.

So now I am collapsing the entire thing into just agreement and
So now I am collapsing the entire thing into just agreement and

disagreement between your hypothesis and the target function, and that's
disagreement between your hypothesis and the target function, and that's

how you get to color the bin.
how you get to color the bin.

Because of that, you have a mapping for every point, whether it's green or
Because of that, you have a mapping for every point, whether it's green or

red, according to this rule.
red, according to this rule.

Now, this will add a component to the learning problem that we
Now, this will add a component to the learning problem that we

did not have before.
did not have before.

There is a probability associated with the bin.
There is a probability associated with the bin.

There is a probability of picking a marble, and
There is a probability of picking a marble, and

independently, and all of that.
independently, and all of that.

When we talked about the learning problem, there was no probability.
When we talked about the learning problem, there was no probability.

I will just give you a sample set, and that's what you work with.
I will just give you a sample set, and that's what you work with.

So let's see what is the addition we need to do in order to adjust the
So let's see what is the addition we need to do in order to adjust the

statement of the learning problem to accommodate the new ingredient.
statement of the learning problem to accommodate the new ingredient.

And the new ingredient is important, because otherwise we cannot learn.
And the new ingredient is important, because otherwise we cannot learn.

It's not like we have the luxury of doing without it.
It's not like we have the luxury of doing without it.

So we go back to the learning diagram from last time.
So we go back to the learning diagram from last time.

Do you remember this one?
Do you remember this one?

Let me remind you.
Let me remind you.

Here is your target function, and it's unknown.
Here is your target function, and it's unknown.

And I promised you last time that it will remain unknown, and the promise
And I promised you last time that it will remain unknown, and the promise

will be fulfilled.
will be fulfilled.

We are not going to touch this box.
We are not going to touch this box.

We're just going to add another box to accommodate the probability.
We're just going to add another box to accommodate the probability.

And the target function generates the training examples.
And the target function generates the training examples.

These are the only things that the learning algorithm sees.
These are the only things that the learning algorithm sees.

It picks a hypothesis from the hypothesis set, and produces it as the
It picks a hypothesis from the hypothesis set, and produces it as the

final hypothesis, which hopefully approximates f.
final hypothesis, which hopefully approximates f.

That's the game.
That's the game.

So what is the addition we are going to do?
So what is the addition we are going to do?

In the bin analogy, this is the input space.
In the bin analogy, this is the input space.

Now the input space has a probability.
Now the input space has a probability.

So I need to apply this probability to the points from the input space that
So I need to apply this probability to the points from the input space that

are being generated.
are being generated.

I am going to introduce a probability distribution over the
I am going to introduce a probability distribution over the

input space.
input space.

Now the points in the input space-- let's say the d-dimensional
Now the points in the input space-- let's say the d-dimensional

Euclidean space--
Euclidean space--

are not just generic points now.
are not just generic points now.

There is a probability of picking one point versus the other.
There is a probability of picking one point versus the other.

And that is captured by the probability, which I'm going to call
And that is captured by the probability, which I'm going to call

capital P.
capital P.

Now the interesting thing is that I'm making no assumptions about P. P can
Now the interesting thing is that I'm making no assumptions about P. P can

be anything.
be anything.

I just want a probability.
I just want a probability.

So invoke any probability you want, and I am ready with the machinery.
So invoke any probability you want, and I am ready with the machinery.

I am not going to restrict the probability distributions over X.
I am not going to restrict the probability distributions over X.

That's number one.
That's number one.

So this is not as bad as it looks.
So this is not as bad as it looks.

Number two, I don't even need to know what P is.
Number two, I don't even need to know what P is.

Of course, the probability choice will affect the choice of the probability
Of course, the probability choice will affect the choice of the probability

of getting a green marble or a red marble, because now the probability of
of getting a green marble or a red marble, because now the probability of

different marbles changed, so it could change the value mu.
different marbles changed, so it could change the value mu.

But the good news with the Hoeffding is that I could bound the performance
But the good news with the Hoeffding is that I could bound the performance

independently of mu.
independently of mu.

So I can get away with not only any P, but with a P that I don't know, and
So I can get away with not only any P, but with a P that I don't know, and

I'll still be able to make the mathematical statement.
I'll still be able to make the mathematical statement.

So this is a very benign addition to the problem.
So this is a very benign addition to the problem.

And it will give us very high dividends, which is the
And it will give us very high dividends, which is the

feasibility of learning.
feasibility of learning.

So what do you do with the probability?
So what do you do with the probability?

You use the probability to generate the points x_1 up to x_N. So now
You use the probability to generate the points x_1 up to x_N. So now

x_1 up to x_N are assumed to be generated by that probability
x_1 up to x_N are assumed to be generated by that probability

independently.
independently.

That's the only assumption that is made.
That's the only assumption that is made.

If you make that assumption, we are in business.
If you make that assumption, we are in business.

But the good news is, as I mentioned before,
But the good news is, as I mentioned before,

we did not compromise about the target function.
we did not compromise about the target function.

You don't need to make assumptions about the function you don't know and
You don't need to make assumptions about the function you don't know and

you want to learn, which is good news.
you want to learn, which is good news.

And the addition is almost technical.
And the addition is almost technical.

That there is a probability somewhere, generating the points.
That there is a probability somewhere, generating the points.

If I know that, then I can make a statement in probability.
If I know that, then I can make a statement in probability.

Obviously, you can make that statement only to the extent that the assumption
Obviously, you can make that statement only to the extent that the assumption

is valid, and we can discuss that in later lectures when the
is valid, and we can discuss that in later lectures when the

assumption is not valid.
assumption is not valid.

So, OK.
So, OK.

Happy ending.
Happy ending.

We are done, and we now have the correspondence.
We are done, and we now have the correspondence.

Are we done?
Are we done?

Well, not quite.
Well, not quite.

Why are we not done?
Why are we not done?

Because the analogy I gave you requires a particular
Because the analogy I gave you requires a particular

hypothesis in mind.
hypothesis in mind.

I told you that the red and green marbles correspond to the agreement between h
I told you that the red and green marbles correspond to the agreement between h

and the target function.
and the target function.

So when you tell me what h is, you dictate the colors here.
So when you tell me what h is, you dictate the colors here.

All of these colors.
All of these colors.

This is green not because it's inherently green, not because of
This is green not because it's inherently green, not because of

anything inherent about the target function.
anything inherent about the target function.

It's because of the agreement between the target function and your
It's because of the agreement between the target function and your

hypothesis, h.
hypothesis, h.

That's fine, but what is the problem?
That's fine, but what is the problem?

The problem is that I know that for this h, nu generalizes to mu.
The problem is that I know that for this h, nu generalizes to mu.

You're probably saying, yeah, but h could be anything.
You're probably saying, yeah, but h could be anything.

I don't see the problem yet.
I don't see the problem yet.

Now here is the problem.
Now here is the problem.

What we have actually discussed is not learning, it's verification.
What we have actually discussed is not learning, it's verification.

The situation as I describe it--
The situation as I describe it--

you have a single bin and you have red and green marbles, and this and that,
you have a single bin and you have red and green marbles, and this and that,

corresponds to the following.
corresponds to the following.

A bank comes to my office.
A bank comes to my office.

We would like a formula for credit approval.
We would like a formula for credit approval.

And we have data.
And we have data.

So instead of actually taking the data, and searching hypotheses, and picking
So instead of actually taking the data, and searching hypotheses, and picking

one, like the perceptron learning algorithm, here is what I do that
one, like the perceptron learning algorithm, here is what I do that

corresponds to what I just described.
corresponds to what I just described.

You guys want a linear formula?
You guys want a linear formula?

OK.
OK.

I guess the salary should have a big weight.
I guess the salary should have a big weight.

Let's say 2.
Let's say 2.

The outstanding debt is negative, so that should be a weight minus 0.5.
The outstanding debt is negative, so that should be a weight minus 0.5.

And years in residence are important, but not that important.
And years in residence are important, but not that important.

So let's give them a 0.1.
So let's give them a 0.1.

And let's pick a threshold that is high, in order for
And let's pick a threshold that is high, in order for

you not to lose money.
you not to lose money.

Let's pick a threshold of 0.5.
Let's pick a threshold of 0.5.

Sitting down, improvising an h.
Sitting down, improvising an h.

Now, after I fix the h, I ask you for the data and just verify whether the h
Now, after I fix the h, I ask you for the data and just verify whether the h

I picked is good or bad.
I picked is good or bad.

That I can do with the bin, because I'm going to look at the data.
That I can do with the bin, because I'm going to look at the data.

If I miraculously agree with everything in your data, I can
If I miraculously agree with everything in your data, I can

definitely declare victory by Hoeffding.
definitely declare victory by Hoeffding.

But what are the chances that this will happen in the first place?
But what are the chances that this will happen in the first place?

I have no control over whether I will be good on the data or not.
I have no control over whether I will be good on the data or not.

The whole idea of learning is that I'm searching the space to deliberately
The whole idea of learning is that I'm searching the space to deliberately

find a hypothesis that works well on the data.
find a hypothesis that works well on the data.

In this case, I just dictated a hypothesis.
In this case, I just dictated a hypothesis.

And I was able to tell you for sure what happens out-of-sample.
And I was able to tell you for sure what happens out-of-sample.

But I have no control of what news I'm going to tell you.
But I have no control of what news I'm going to tell you.

You can come to my office.
You can come to my office.

I improvise this.
I improvise this.

I go to the data.
I go to the data.

And I tell you, I have a fantastic system.
And I tell you, I have a fantastic system.

It generalizes perfectly, and it does a terrible job.
It generalizes perfectly, and it does a terrible job.

That's what I have, because when I tested it, nu was terrible.
That's what I have, because when I tested it, nu was terrible.

So that's not what we are looking for.
So that's not what we are looking for.

What we are looking for is to make it learning.
What we are looking for is to make it learning.

So how do we do that?
So how do we do that?

No guarantee that nu will be small.
No guarantee that nu will be small.

And we need to choose the hypothesis from multiple h's.
And we need to choose the hypothesis from multiple h's.

That's the game.
That's the game.

And in that case, you are going to go for the sample, so to speak, generated
And in that case, you are going to go for the sample, so to speak, generated

by every hypothesis, and then you pick the hypothesis that is most favorable,
by every hypothesis, and then you pick the hypothesis that is most favorable,

that gives you the least error.
that gives you the least error.

So now, that doesn't look like a difficult thing.
So now, that doesn't look like a difficult thing.

It worked with one bin.
It worked with one bin.

Maybe I can have more than one bin, to accommodate the situation where I have
Maybe I can have more than one bin, to accommodate the situation where I have

more than one hypothesis.
more than one hypothesis.

It looks plausible.
It looks plausible.

So let's do that.
So let's do that.

We will just take multiple bins.
We will just take multiple bins.

So here is the first bin.
So here is the first bin.

Now you can see that this is a bad bin.
Now you can see that this is a bad bin.

So that hypothesis is terrible.
So that hypothesis is terrible.

And the sample reflects that, to some extent.
And the sample reflects that, to some extent.

But we are going to have other bins, so let's call this something.
But we are going to have other bins, so let's call this something.

So this bin corresponds to a particular h.
So this bin corresponds to a particular h.

And since we are going to have other hypotheses, we are going to call this
And since we are going to have other hypotheses, we are going to call this

h_1 in preparation for the next guy.
h_1 in preparation for the next guy.

The next guy comes in, and you have h_2.
The next guy comes in, and you have h_2.

And you have another mu_2.
And you have another mu_2.

This one looks like a good hypothesis, and it's also reflected in the sample.
This one looks like a good hypothesis, and it's also reflected in the sample.

And it's important to look at the correspondence.
And it's important to look at the correspondence.

If you look at the top red point here and the top green point here, this is
If you look at the top red point here and the top green point here, this is

the same point in the input space.
the same point in the input space.

It just was colored red here and colored green here.
It just was colored red here and colored green here.

Why did that happen?
Why did that happen?

Because the target function disagrees with this h, and the target function
Because the target function disagrees with this h, and the target function

happens to agree with this h.
happens to agree with this h.

That's what got this the color green.
That's what got this the color green.

And when you pick a sample, the sample also will have different colors,
And when you pick a sample, the sample also will have different colors,

because the colors depend on which hypothesis.
because the colors depend on which hypothesis.

And these are different hypotheses.
And these are different hypotheses.

That looks simple enough.
That looks simple enough.

So let's continue.
So let's continue.

And we can have M of them.
And we can have M of them.

I am going to consider a finite number of hypotheses, just to make the math
I am going to consider a finite number of hypotheses, just to make the math

easy for this lecture.
easy for this lecture.

And we're going to go more sophisticated when we get into the
And we're going to go more sophisticated when we get into the

theory of generalization.
theory of generalization.

So now I have this.
So now I have this.

This is good.
This is good.

I have samples, and the samples here are different.
I have samples, and the samples here are different.

And I can do the learning, and the learning now, abstractly, is to scan
And I can do the learning, and the learning now, abstractly, is to scan

these samples looking for a good sample.
these samples looking for a good sample.

And when you find a good sample, you declare victory, because of Hoeffding,
And when you find a good sample, you declare victory, because of Hoeffding,

and you say that it must be that the corresponding bin is good, and the
and you say that it must be that the corresponding bin is good, and the

corresponding bin happens to be the hypothesis you chose.
corresponding bin happens to be the hypothesis you chose.

So that is an abstraction of learning.
So that is an abstraction of learning.

That was easy enough.
That was easy enough.

Now, because this is going to stay with us, I am now going to introduce
Now, because this is going to stay with us, I am now going to introduce

the notation that will survive with us for the entire discussion of learning.
the notation that will survive with us for the entire discussion of learning.

So here is the notation.
So here is the notation.

We realize that both mu, which happens to be inside the bin,
We realize that both mu, which happens to be inside the bin,

and nu, which happens to be the sample frequency--
and nu, which happens to be the sample frequency--

in this case, the sample frequency of error-- both of them depend on h.
in this case, the sample frequency of error-- both of them depend on h.

So I'd like to give a notation that makes that explicit.
So I'd like to give a notation that makes that explicit.

The first thing,
The first thing,

I am going to call mu and nu with a descriptive name.
I am going to call mu and nu with a descriptive name.

So nu, which is the frequency in the sample you have, is in-sample.
So nu, which is the frequency in the sample you have, is in-sample.

That is a standard definition for what happens in the data that I give you.
That is a standard definition for what happens in the data that I give you.

If you perform well in-sample, it means that your error in the sample
If you perform well in-sample, it means that your error in the sample

that I give you is small.
that I give you is small.

And because it is called in-sample, we are going to denote it by E_in.
And because it is called in-sample, we are going to denote it by E_in.

I think this is worth blowing up, because it's an important one.
I think this is worth blowing up, because it's an important one.

This is our standard notation for the error that you have in-sample.
This is our standard notation for the error that you have in-sample.

Now, we go and get the other one, which happens to be mu.
Now, we go and get the other one, which happens to be mu.

And that is called out-of-sample.
And that is called out-of-sample.

So if you are in this field, I guess what matters is the out-of-sample
So if you are in this field, I guess what matters is the out-of-sample

performance.
performance.

That's the lesson.
That's the lesson.

Out-of-sample means something that you haven't seen.
Out-of-sample means something that you haven't seen.

And if you perform out-of-sample, on something that you haven't seen, then
And if you perform out-of-sample, on something that you haven't seen, then

you must have really learned.
you must have really learned.

That's the standard for it, and the name for it is E_out.
That's the standard for it, and the name for it is E_out.

With this in mind, we realize that we don't yet have the dependency on h
With this in mind, we realize that we don't yet have the dependency on h

which we need.
which we need.

So we are going to make the notation a little bit more elaborate, by calling
So we are going to make the notation a little bit more elaborate, by calling

E_in and E_out--
E_in and E_out--

calling them E_in of h, and E_out of h.
calling them E_in of h, and E_out of h.

Why is that?
Why is that?

Well, the in-sample performance-- you are trying to see the error of
Well, the in-sample performance-- you are trying to see the error of

approximating the target function by your hypothesis.
approximating the target function by your hypothesis.

That's what E_in is.
That's what E_in is.

So obviously, it depends on your hypothesis.
So obviously, it depends on your hypothesis.

So it's E_in of h.
So it's E_in of h.

Someone else picks another h, they will get another E_in of their h.
Someone else picks another h, they will get another E_in of their h.

Similarly E_out, the corresponding one is E_out of h.
Similarly E_out, the corresponding one is E_out of h.

So now, what used to be nu is now E_in of h.
So now, what used to be nu is now E_in of h.

What used to be mu, inside the bin, is E_out of h.
What used to be mu, inside the bin, is E_out of h.

Now, the Hoeffding Inequality, which we know all too well
Now, the Hoeffding Inequality, which we know all too well

by now, said that.
by now, said that.

So all I'm going to do is just replace the notation.
So all I'm going to do is just replace the notation.

And now it looks a little bit more crowded, but it's
And now it looks a little bit more crowded, but it's

exactly the same thing.
exactly the same thing.

The probability that your in-sample performance deviates from your out-of-
The probability that your in-sample performance deviates from your out-of-

sample performance by more than your prescribed tolerance is less than or
sample performance by more than your prescribed tolerance is less than or

equal to a number that is hopefully small.
equal to a number that is hopefully small.

And you can go back and forth.
And you can go back and forth.

There's nu and mu, or you can go here and you get the new notation.
There's nu and mu, or you can go here and you get the new notation.

So we're settled on the notation now.
So we're settled on the notation now.

Now, let's go for the multiple bins and use this notation.
Now, let's go for the multiple bins and use this notation.

These are the multiple bins as we left them.
These are the multiple bins as we left them.

We have the hypotheses h_1 up to h_M, and we have the mu_1 and mu_M.
We have the hypotheses h_1 up to h_M, and we have the mu_1 and mu_M.

And if you see 1, 2, M, again, this is a disappearing nu--
And if you see 1, 2, M, again, this is a disappearing nu--

the symbol that the app doesn't like.
the symbol that the app doesn't like.

But thank God we switched notations, so that
But thank God we switched notations, so that

something will appear.
something will appear.

Yeah!
Yeah!

So right now, that's what we have.
So right now, that's what we have.

Every bin has an out-of-sample performance, and out-of-
Every bin has an out-of-sample performance, and out-of-

sample is: Out. Of. Sample.
sample is: Out. Of. Sample.

So this is a sample.
So this is a sample.

What's in it is in-sample.
What's in it is in-sample.

What is not in it is out-of-sample.
What is not in it is out-of-sample.

And the out-of-sample depends on h_1 here, h_2 here, and h_M here.
And the out-of-sample depends on h_1 here, h_2 here, and h_M here.

And obviously, these quantities will be different according to the sample, and
And obviously, these quantities will be different according to the sample, and

these quantities will be different according to the ultimate performance
these quantities will be different according to the ultimate performance

of your hypothesis.
of your hypothesis.

So we solved the problem.
So we solved the problem.

It's not verification. It's not a single bin.
It's not verification. It's not a single bin.

It's real learning.
It's real learning.

I'm going to scan these.
I'm going to scan these.

So that's pretty good.
So that's pretty good.

Are we done already?
Are we done already?

Not so fast.
Not so fast.

[LAUGHING]
[LAUGHING]

What's wrong?
What's wrong?

Let me tell you what's wrong.
Let me tell you what's wrong.

The Hoeffding Inequality, that we have happily studied and declared important
The Hoeffding Inequality, that we have happily studied and declared important

and all of that, doesn't apply to multiple bins.
and all of that, doesn't apply to multiple bins.

What?
What?

You told us mathematics, and you go read the proof, and all of that.
You told us mathematics, and you go read the proof, and all of that.

Are you just pulling tricks on us?
Are you just pulling tricks on us?

What is the deal here?
What is the deal here?

And you even can complain.
And you even can complain.

We sat for 40 minutes now going from a single bin, mapping it to
We sat for 40 minutes now going from a single bin, mapping it to

the learning diagram, mapping it to multiple bins, and now you tell us
the learning diagram, mapping it to multiple bins, and now you tell us

that the main tool we developed doesn't apply.
that the main tool we developed doesn't apply.

Why doesn't it apply, and what can we do about it?
Why doesn't it apply, and what can we do about it?

Let me start by saying why it doesn't apply, and then we can go for what we
Let me start by saying why it doesn't apply, and then we can go for what we

can do about it.
can do about it.

Now, everybody has a coin.
Now, everybody has a coin.

I hope the online audience have a coin ready.
I hope the online audience have a coin ready.

I'd like to ask you to take the coin out and flip it,
I'd like to ask you to take the coin out and flip it,

let's say, five times.
let's say, five times.

And record what happens.
And record what happens.

And when you at home flip the coin five times, please,
And when you at home flip the coin five times, please,

if you happen to get all five heads in your experiment, then text us that you
if you happen to get all five heads in your experiment, then text us that you

got all five heads.
got all five heads.

If you get anything else, don't bother text us.
If you get anything else, don't bother text us.

We just want to know if someone will get five heads.
We just want to know if someone will get five heads.

Everybody is done flipping the coin.
Everybody is done flipping the coin.

Because you have been so generous and cooperative, you can keep the coin!
Because you have been so generous and cooperative, you can keep the coin!

[LAUGHTER]
[LAUGHTER]

Now, did anybody get five heads?
Now, did anybody get five heads?

All five heads?
All five heads?

Congratulations, sir.
Congratulations, sir.

You have a biased coin, right?
You have a biased coin, right?

We just argued that in-sample corresponds to out-of-sample, and we
We just argued that in-sample corresponds to out-of-sample, and we

have this Hoeffding thing, and therefore if you get five heads, it
have this Hoeffding thing, and therefore if you get five heads, it

must be that this coin gives you heads.
must be that this coin gives you heads.

We know better.
We know better.

So in the online audience, what happened?
So in the online audience, what happened?

MODERATOR: Yeah, in the online audience, there's also five heads.
MODERATOR: Yeah, in the online audience, there's also five heads.

PROFESSOR: There are lots of biased coins out there.
PROFESSOR: There are lots of biased coins out there.

Are they really biased coins?
Are they really biased coins?

No.
No.

What is the deal here?
What is the deal here?

Let's look at it.
Let's look at it.

Here, with the audience here, I didn't want to push my luck with 10 flips,
Here, with the audience here, I didn't want to push my luck with 10 flips,

because it's a live broadcast.
because it's a live broadcast.

So I said five will work.
So I said five will work.

For the analytical example, let's take 10 flips.
For the analytical example, let's take 10 flips.

Let's say you have a fair coin, which every coin is.
Let's say you have a fair coin, which every coin is.

You have a fair coin.
You have a fair coin.

And you toss it 10 times.
And you toss it 10 times.

What is the probability that you will get all 10 heads?
What is the probability that you will get all 10 heads?

Pretty easy.
Pretty easy.

One half, times one half, 10 times, and that will give
One half, times one half, 10 times, and that will give

you about 1 in 1000.
you about 1 in 1000.

No chance that you will get it--
No chance that you will get it--

not no chance, but very little chance.
not no chance, but very little chance.

Now, the second question is the one we actually ran the experiment for.
Now, the second question is the one we actually ran the experiment for.

If you toss 1000 fair coins-- it wasn't 1000 here. It's how many there.
If you toss 1000 fair coins-- it wasn't 1000 here. It's how many there.

Maybe out there is 1000.
Maybe out there is 1000.

What is the probability that some coin will give you all 10 heads?
What is the probability that some coin will give you all 10 heads?

Not difficult at all to compute.
Not difficult at all to compute.

And when you get the answer, the answer will be it's actually more
And when you get the answer, the answer will be it's actually more

likely than not.
likely than not.

So now it means that the 10 heads in this case are no indication at all of
So now it means that the 10 heads in this case are no indication at all of

the real probability.
the real probability.

That is the game we are playing.
That is the game we are playing.

Can I look at the sample and infer something about the real probability?
Can I look at the sample and infer something about the real probability?

No.
No.

In this case, you will get 10 heads, and the coin is fair.
In this case, you will get 10 heads, and the coin is fair.

Why did this happen?
Why did this happen?

This happened because you tried too hard.
This happened because you tried too hard.

Eventually what will happen is--
Eventually what will happen is--

Hoeffding applies to any one of them.
Hoeffding applies to any one of them.

But there is a probability, let's say half a percent, that you
But there is a probability, let's say half a percent, that you

will be off here.
will be off here.

Another half a percent that you will be off here.
Another half a percent that you will be off here.

If you do it often enough, and you are lucky enough that the half percents
If you do it often enough, and you are lucky enough that the half percents

are disjoint, you will end up with extremely high probability that
are disjoint, you will end up with extremely high probability that

something bad will happen, somewhere.
something bad will happen, somewhere.

That's the key.
That's the key.

So let's translate this into the learning situation.
So let's translate this into the learning situation.

Here are your coins.
Here are your coins.

And how do they correspond to the bins?
And how do they correspond to the bins?

Well, it's a binary experiment, whether you are picking a red marble
Well, it's a binary experiment, whether you are picking a red marble

or a green marble, or you are flipping a coin getting heads or tails.
or a green marble, or you are flipping a coin getting heads or tails.

It's a binary situation.
It's a binary situation.

So there's a direct correspondence.
So there's a direct correspondence.

Just get the probability of heads being mu, which is the probability of
Just get the probability of heads being mu, which is the probability of

a red marble, corresponding to them.
a red marble, corresponding to them.

So because the coins are fair,
So because the coins are fair,

actually all the bins in this case are half red, half green.
actually all the bins in this case are half red, half green.

That's really bad news for a hypothesis.
That's really bad news for a hypothesis.

The hypothesis is completely random.
The hypothesis is completely random.

Half the time it agrees with the target function.
Half the time it agrees with the target function.

Half the time it disagrees.
Half the time it disagrees.

No information at all.
No information at all.

Now you apply the learning paradigm we mentioned, and you say: let me
Now you apply the learning paradigm we mentioned, and you say: let me

generate a sample from the first hypothesis.
generate a sample from the first hypothesis.

I get this, I look at it, and I don't like that.
I get this, I look at it, and I don't like that.

It has some reds.
It has some reds.

I want really a clean hypothesis that performs perfectly--
I want really a clean hypothesis that performs perfectly--

all green.
all green.

You move on.
You move on.

And, OK.
And, OK.

This one--
This one--

even, I don't know.
even, I don't know.

This is even worse.
This is even worse.

You go on and on and on.
You go on and on and on.

And eventually, lo and behold, I have all greens.
And eventually, lo and behold, I have all greens.

Bingo.
Bingo.

I have the perfect hypothesis.
I have the perfect hypothesis.

I am going to report this to my customer, and if my customer is in
I am going to report this to my customer, and if my customer is in

financial forecasting, we are going to beat the stock market and
financial forecasting, we are going to beat the stock market and

make a lot of money.
make a lot of money.

And you start thinking about the car you are going to buy, and all of that.
And you start thinking about the car you are going to buy, and all of that.

Well, is it bingo?
Well, is it bingo?

No, it isn't.
No, it isn't.

And that is the problem.
And that is the problem.

So now, we have to find something that makes us deal with
So now, we have to find something that makes us deal with

multiple bins properly.
multiple bins properly.

Hoeffding Inequality-- if you have one experiment, it has a guarantee.
Hoeffding Inequality-- if you have one experiment, it has a guarantee.

The guarantee gets terribly diluted as you go, and we want to know exactly
The guarantee gets terribly diluted as you go, and we want to know exactly

how the dilution goes.
how the dilution goes.

So here is a simple solution.
So here is a simple solution.

This is a mathematical slide. I'll do it step-by-step.
This is a mathematical slide. I'll do it step-by-step.

There is absolutely nothing mysterious about it.
There is absolutely nothing mysterious about it.

This is the quantity we've been talking about.
This is the quantity we've been talking about.

This is the probability of a bad event.
This is the probability of a bad event.

But in this case, you realize that I'm putting g.
But in this case, you realize that I'm putting g.

Remember, g was our final hypothesis.
Remember, g was our final hypothesis.

So this corresponds to a process where you had a bunch of h's, and you picked
So this corresponds to a process where you had a bunch of h's, and you picked

one according to a criterion, that happens to be an in-sample criterion,
one according to a criterion, that happens to be an in-sample criterion,

minimizing the error there, and then you report the g as the
minimizing the error there, and then you report the g as the

one that you chose.
one that you chose.