Let's look at chapter 3, which covers norm and distance.
So first, we'll talk about norm.
Uh, norm is the norm of a vector is supposed to be a measure of its length.
Um, so we'll talk about that.
From that, we're allowed to talk about distance,
and that's another topic that we'll cover, uh,
in this- uh, in this mini-lecture.
So the norm of a vector,
uh, and this is specifically a- is called a Euclidean norm.
Um, in this course,
we're just gonna call it the norm because it's the only norm we're gonna use.
There are actually are other norms that are actually widely used.
But by far, the most widely used norm is the one we're gonna talk about,
the Euclidean norm. Here's what it is.
It says there's an n-vector x and the norm,
which is denoted with these two double bars
on either side of x. I'll say something about that.
It's supposed to generalize the absolute value for a number,
so okay, and so the notation is meant to look like it.
So here it is, is the square root of the sum of the squares.
That is the norm of a vector.
Um, and you can write that if you like as
the square root of the inner product of x with itself.
So that's- uh, that's the norm of a vector.
Um, and it's supposed to measure the size of a vector.
Um, some people say the length.
Unfortunately, the length of a vector is a little
ambiguous because it could either be n in this case,
um, or it could be the norm.
So if you say size, however,
it's probably- it probably means the,
uh, the- the- the norm.
Um, now when n is one,
so it's a one vector,
which is by the way the same as just a scalar, then, uh,
the square root of the sum of the squares is nothing but
the square root of the square of the entry,
and that's the absolute value, right?
Because when you square something,
if it's negative or positive, you get the same number.
When we take the square root,
we're taking the positive square root or the non-negative square root.
So it's a generalization of the absolute value of two, uh,
vectors, is what- what- that's the basic idea.
It's supposed to give me the length then.
Let's just do a quick example.
Uh, let's find, you know,
the- let's find the- the norm of this vector.
Well, that's gonna be the square root of 1 squared plus 0 squared plus 1 squared,
and that's the square root of two,
which is about 1.41, something like that about.
So that's the length of a vector.
Um, okay.
Now, there's a bunch of properties.
Um, some of these are simple. Uh, what is not?
I'll- I'll say a little bit about that, uh, later.
Um, so the first is, uh,
homogeneity, and what this means is the following.
It says that if I take a vector and I scale, uh,
all of its entries by Beta,
that means multiply it by the scalar vector multiplication Beta,
the norm of the result is actually the absolute value Beta times x,
uh, times the norm of x. Um,
and that tells you a lot of stuff already.
It says, for example, that the norm of x and the norm of minus x are the same,
and you can figure that out too because if you- if you
replace all the entries xi with negatives and you square them,
you're back right where you started.
So everything is- stays the same.
Um, so that's the idea.
And this one is actually relatively easy to- [NOISE] to show because what happens is,
uh, the sum of the squares,
[NOISE] each of the squares get scaled by Beta
squared but then they all come out of the square root as a Beta,
absolute value Beta I should say.
Okay. Um, the triangle inequality,
this one is- is interesting.
It's got a beautiful visual- uh,
visualization that we'll- we'll show later,
um, but we're not gonna go into it now.
I should also say it's not something that's
immediately easy to see or something like that.
It would take a paragraph or something to develop it.
It's not hard, it just involves completing the square.
I'm not gonna do it here.
Uh, you can read about it in the book or something like that,
but this is the- but this is so called triangle inequality. It's very interesting.
It says that the- the length of a sum,
uh, of two vectors is less than the sum of the lengths.
Okay. Now, I said that in English,
but it's the same as overloading, right?
So the first sum was the sum of vectors and
the second sum was the sum of two numbers, right?
So that's vector addition right there,
and that is addition of two non-negative numbers.
So, um, all right.
Uh, the next one is non-negativity.
That- that was extremely obvious because when you take the sum of the squares of x,
you get a non-negative number.
That's good because then you're gonna then gonna take the square root,
and if it were a negative number,
it wouldn't be a good thing.
Um, so- and then
of course the square root for us is define as the non-negative square root,
so that- that one's obvious.
This one's interesting, definiteness.
This says that if a- if a a norm of a vector is zero,
the only possible way that can happen is it is zero.
That I am gonna say something about,
so let's see what that means.
Suppose I told you that the norm of x is zero.
Well, that tells you that the norm of x squared is zero.
But if the norm of x squared is zero,
that's the same as x one squared plus- uh,
plus xn squared.
That sum, okay.
Now, um, if I tell you that a sum of numbers is zero,
you absolutely cannot conclude that they are all zero,
and here's a nice example of that.
Uh, one and minus one.
Their sum is zero. They're both- obviously both not zero.
Okay. However, if I tell you that a sum of non-negative numbers is zero,
then you conclude that- then you can conclude that all of them are zero.
Okay? Um, and that's easy to see.
If- if the numbers are, uh,
non-negative and one of them is positive,
then the whole sum is positive,
and if you're positive, you are not zero.
So- uh, so what this says is from this,
we can conclude that xi is equal to zero.
Okay? So that's the picture.
Um, and the triangle inequality,
we'll- we'll see later,
uh, how- how- how that works.
But, um, we'll also see a picture of it that- that gives,
you know, that explains the name.
Okay. So these are some very basic properties of the norm.
Um, so far we haven't done anything with a norm that
would be even vaguely of interest, but we'll get there.
Okay. Now, a very rel- a closely related quantity is called the RMS value,
uh, which stands for root mean square,
and so let me explain what that is.
It's- it's- it's anot- it's another measure of the size of a- of a vector.
So here, um, the mean square value of the vector is the sum- it's the sum of the squares,
right, that would be the norm squared,
divided by n. And you could see what it is.
It's the square of the entries and then the mean of them across the entries.
That's the mean square, and that is expressed in terms of a norm as norm squared x,
that's how you'd say that, divided by n. Okay?
So that- that's the mean square value.
Um, what's very commonly used is something called the
RMS value or root mean square value, um.
Uh, it's used throughout engineering, you know.
Uh, maybe, you know, in pure math.
I'd pick maybe somebody would figure out what you're talking about,
but it is used through absolutely throughout engineering and many,
many other fields, this RMS value.
Here's what it is. Well, it's literally what its name says.
It's the square root of the mean square value.
So here, that's the mean square,
you take the square root,
and you get- it's actually equal to the norm of x divided by square root of
n. So it's nothing more than the norm of x divided by square root of n. [NOISE]
Okay. Now, the reason you would use this
is that it's going to- it- it- it actually is more intuitive.
Um. So what we'll see is that the RMS value of vector gives roughly speaking,
something like the typical value of the absolute value of x_i.
I mean, it- it doesn't,
but in- in a certain way, it does, right?
So, um, uh, so here's a good example,
is take the vector of all ones, right?
Then I ask you,
what is the- what's the norm of ones?
Well, the norm of ones, uh, here,
I'll make a bold one here,
is actually the square root of n,
uh, where n is the length, and why?
Because you would- to get the norm,
you would take the sum of the squares of the entries.
All the entries are one.
That gives you n, and then you take the square root.
So you get the square root of n. Um,
so, and that's fine, um, by the way,
if I'm comparing norms of vectors always with the same dimension,
it just the square root n factor doesn't matter, right?
Um. But, if I show you a vector of one and I said, you know,
tell me about that vector,
one seems like a natural thing to say,
that's the RMS value.
Um. And the reason that's a natural thing to say is because all the entries are one.
And so one is kind of telling you the size of- of the individual entries, right?
So, um, RMS value is also useful for comparing,
uh, sizes of vectors of different lengths, right?
That- that's actually very critical, right?
We'll- we'll see several applications where you do exactly that.
Where you, you know,
you- you- you take the RMS value of one size vector and another size vector,
and now they're sort of comparable, uh, this way.
Okay, so that's RMS value,
very widely, uh, used.
You probably have seen it before.
It's the most natural way to quote, uh,
let's say the- the- to- to give an idea of the size of a vector.
Okay. Um. One property and this is gonna come up later in the course,
uh, but we'll mention it now.
Um. And that is that, that you can do the norm- you can calculate
the norm of block vectors in a very, uh, simple way.
Um. And, uh, what it is is this,
is here, here I have three vectors,
a, b, and c. Um.
This is our notation, remember that's a- a stacked vector.
Uh. So I stack a on top of b on top of c and then take the norm squared.
Well, if you think about it,
how do I take the norm squared of a- of- of- of a-
of a vector that's the sum of the squares of the entries?
So I would first take the sum of the squares of all the entries of a,
I'd add that to the sum of the squares of all the entries of b,
add that to all the entries, and you'd get, you can see this,
and you can also see that by just multiplying out the inner product like this.
Um. And that says that- and- and this is a very tricky formula to parse,
so you have to be super-duper on your toes.
Um. It says that the norm of the block vector here is equal to the following.
It is the square root of the norm squared of a,
uh, plus the norm squared of b plus the norm squared of c. Okay?
Or we can put it in super compact and honestly,
very confusing notation, but it does work.
Okay? So what this says is that the norm of a stacked vector a,
b, c, and of course, I'm- I'm showing you a stack of three,
but it could be a stack of two,
a stack a 20.
Doesn't matter, right? Um. Over here,
what it says is that's the norm of, now these are numbers,
so a, b, and c,
the norms of those, those are three numbers.
This is a three vector in here,
that- that's a three vector.
And this is the norm of the three vector.
But the norm of a three vector is
the square root of the sum of the squares of the entries,
and that's exactly what this is.
So- so we'll use these ideas that connect, uh,
norms of stacked things, uh, later.
Uh. Actually, we'll use it in the form that the nor- that the norm squared of
a stacked thing is the sum of the norm squares of the entries, right?
So this is really the one that we're gonna use later.
Um. So, okay.
Now we're gonna talk about the Chebyshev inequality.
Um. And, yeah, it's a famous inequality that says,
it basically limits the number of entries of a vector that can be,
uh, bigger than a certain size, right?
Um. And that's the- and that depends on the norm, right?
So we'll look- we'll look at how that works.
And so the story starts this way.
You say, um, let's take the entries of a vector x_1 through x_n,
and let's consider their absolute values.
And suppose I told you that k of them are bigger than equal to some number a.
A could be whatever, it doesn't- 0.6,
it doesn't- anything at all.
Okay? Um. All right, well,
if- if k of those absolute values are bigger than a,
then k of the numbers which are the squares,
are bigger than a squared.
Okay? Now, when I add up a bunch of squares and k of them are bigger than a squared,
then the sum is better- it's gotta be bigger than ka squared.
In fact, it would only be ka squared if the other numbers,
the other n minus k numbers are all zero,
otherwise this is stripped here. It's even bigger.
Okay. But this is the norm squared of x.
It's bigger than ka squared.
And so what this says is,
I simply divide it by a squared and I get this,
k is less than the norm squared of x divided by a squared.
Okay? And so what that says is, basically,
it says that the number of entries of a vector that
exceed an absolute value a number a is no more than,
uh, the norm squared of x squared,
norm squared divided by a squared.
That's it. Okay? And that's the- that's this famous Chebyshev inequality, right?
That's- that's- that's- that's what it says.
Um. And it's actually maybe clearer in terms of RMS value.
And this is the way certainly normal people think of
it in terms of the RMS value. It's this.
Um. You divide appropriately by
square root n in the right place and everything, and here's what you get.
It says I have a vector x,
it says that the fraction of the entries with absolute value x bigger than
a is no more than RMS of x divided by a squared.
So that's it.
Um. And, uh,
so let's- let's look at some special cases.
I could take a equals the RMS value squared.
Um. And it said- it says absolutely nothing.
It says the fraction of entries of a vector, uh,
where the absolute value is bigger than or equal to
the RMS value is bigger than or equal to, uh, one.
So, okay, fine.
Um. Okay. However, it gets interesting when a is bigger than RMS of x, right?
So for example, uh,
you could take five RMS,
uh, something like, you know, five times RMS of it.
And then it says- it says,
it's very- it's very specific.
It says that no more than one- than- this is now one-fifth,
no more than one-fifth squared, so one twenty-fifth.
So no more than four percent of- of the entries of
a vector can be bigger than five times the RMS value.
Okay? So that's- that's what it says.
Um. And by the way, this is kinda the idea of- of- of
justifying the idea that the RMS value of
a vector is like the typical value of the entries.
Um. The entries can be bigger than the typical value,
but they're sho- they're actually rather limited, right?
That you- if you- the number that can be
more than five times that number is less than four percent of your total vector.
So that's kinda the- tha- that's the idea.
We'll see- we'll see applications of this a bit later.