Now we're going to talk about distance.
Distance between two vectors.
Um, that's a concept that comes immediately from the idea of norm,
which is a- which is a- roughly the length of a vector.
Okay? So if I have two vectors,
let's suppose these are two vectors, or three vectors, right?
Uh, then the Euclidean distance between the- if a and b represent two positions,
it's simply the norm of a minus b. Um,
and so some people would write that this way.
So dist is the distance function, says
the distance between a and b is the norm of a minus b. Um,
and you know, the picture might be this,
that's the- that's the point a,
that's the position b.
Uh, in this case,
b minus a is the vector, uh,
shown by the blue arrow,
which in this case is 2, 1 if we
believe if- if the units on that grid are- are- are units.
Uh, and then the distance between a and b would then be b minus a.
And that's the same of course,
as a minus b, they're the same thing.
So that's- that's the distance between two vectors.
Um, and it agrees with the ordinary distance,
you know, that you learned about in your algebra class.
Uh, you know, certainly for dimensions 1,
2, and 3, it's the same.
Uh, sometimes people also, uh,
will look at the RMS deviation between a and b.
Uh, so that's the r- it's- but it's just that- it's
just the distance divided by square root n. Okay,
so that- so you automatically get distance.
Now what's new here is though,
we can now talk about the distance between 100 vectors,
namely vectors with dimension 100.
Uh, that's interesting.
You can't visualize that.
Now you can pretend to visualize that,
you can claim, you know,
in front of your friends that you can say, well,
I can visualize that there's one vector, or there's the other,
and the distance between them is 12,
I told you I can see it,
but no one can visualize 100 dimensions.
So that doesn't really make any sense.
Okay? Uh, now I can get to
the triangle inequality and I can- I can explain how it got its name.
So here's how it works.
[NOISE] Um, think of- of three positions,
a, b, and c. These are all three vectors.
Um, now the edge lengths,
and I've- I've drawn that down here.
So there's- here's a, b, and c. Um,
now what we're going to do is the edge lengths are, well,
the edge between a and b has length norm a minus b.
Uh, the length between b and c is norm b minus c, for example.
Um, and the length between of the dist- the distance between a and c,
the length of this triangle side is norm a minus c. Okay?
And so we are gonna apply the triangle inequality.
The triangle inequality says that the sum of two vectors- so first of all,
I've just- I've added b and subtracted b in
going from this equation to this equation, so nothing's changed.
It's- it's, and so I have an equal sign here.
Um, and now what I'm doing is I'm using the triangle inequality.
Triangle inequality says that the norm of
a sum is less than or equal to the sum of the norms.
So here I have norm of a minus b,
norm of b minus c, and what it says is,
I mean, it goes- goes way back,
goes to Euclid, I guess,
and basically says the following.
It says that the no- that the length of any side of
a triangle is less than or equal to the sum of the other two lengths.
So that's uh- that's what it says.
So this- that's why this is called the triangle inequality.
The- the original uh,
the- the inequality for norms. Um, okay?
[NOISE] Now, and just as- as a teaser,
we're gonna talk about
some very interesting ideas that even the norm is a very simple concept.
You can actually already see- see that it's gonna-
you can- there's hints that it could get super-duper useful.
So let me- I'll talk about the concept of a nearest neighbor.
Um, so if you have two feature vectors, for example,
um, then the norm of the difference,
which is the distance between them,
that's called the feature distance.
And so what that says is you could talk about feature,
you have the concept of two features-
two feature vectors being close or two being very far away.
So, I mean, if we go back to that very simple example, but in, er,
of regression, we had the house area as the first component,
and the second was the number of bedrooms.
And so, uh, two houses that are close would have,
well, probably the same number of bedrooms,
and similar number of thousands of square feet, right?
So that would be, in that case,
the feature distance would be small.
But a- a one bedroom,
you know, 900 square foot house, right?
Would not- would have a large feature distance to a six bedroom,
5,000 square foot house, right?
So that- that's kind of the idea.
So all of a sudden we have the distance.
Now, that's not a geometric notion of distance, right?
So that's the- that's- that's the- that's the idea.
Okay? So this brings us to this idea of a nearest neighbor.
Um, it's something you will hear about in other courses for sure.
Uh, maybe we'll say something about it too later.
But let's see, so here- if I have a list of vectors, um, then z,
j, uh, one of these vectors is- and I have another vector x,
like a query vector.
Then what it will say is that among this list of vectors,
the one that is closest to x,
we're gonna call the nearest neighbor.
Makes perfect sense, right?
So here's x in this picture,
and here's my z is. You can see the six of them.
And the distance between x and the zs,
I mean, the farthest one maybe is z1, right?
So that's the farthest distance.
Um, intermed- you know,
intermed- there's various intermediate distances,
but it looks like the winner is z3 here.
So we- you would say that z3 in this- in this set, tiny baby example here,
z3 is the nearest neighbor of x among z1 dot- dot- dot z6.
Okay? So- and it makes perfect sense.
I mean, you're basically saying of this list of vectors,
which is x closest to.
Okay? So now you might ask like, who cares?
Uh, but actually immediately this is a powerful concept in- in
sort of very basic machine learning and data analysis, already very powerful concept.
I mean, eve- even though it looks very innocent [NOISE].
Let's look at an example which hopefully will
give you a rough idea of- of some cool stuff here.
Okay? [NOISE] So, um,
what we're gonna do is we're going to look at feature vectors,
which are word count histograms.
Okay? Uh, so what we did is we took some Wikipedia articles here.
Um, I guess there's five of them,
and we have a dictionary of forty, about four-and-a-half thousand words.
That's all. Okay. [NOISE]
Um, and then what we're gonna do is,
we're gonna take for each of these five Wikipedia articles,
we take the entire article and we count the number of
words in our dictionary that appears in each of those.
So I don't know what the first entry in my dictionary is,
but I guarantee you that for these,
I have five vectors corresponding to these- these Wikipedia pages, right?
And the first entry tells you how many
times that first dictionary word appeared in those five articles.
Okay. So that's it. Now, yeah.
Don't even try to visualize, uh,
what- what a vector in- in for- with 44 or 23 dimensions is because you can't.
You can pretend you can.
And you can in fact, if you like,
you can sort of pretend that it looks like that.
Okay? But I mean,
I guess it does look like that just in 4,423 dimensions.
Okay. So, uh, let's- let's take a look at what happens now.
We're gonna calculate the pairwise distances of all, uh, of,
all 65 of these,
um, of these Wikipedia pages, okay?
And- and all I'm- all we wanna do is to see,
kind of see if it makes sense.
Okay. Well, the first one is the diagonals are easy
because Veteran's Day has a vector associated with it.
So does, uh, and- and if I say,
what's the distance between a vector and itself, it's zero.
It's the norm of zero minus zero, so it's zero.
So the diagonals are all zero, right?
Oh, and by the way, it's also as- this is symmetric in
the sense that the distance between Veterans Day and Memorial Day,
which is 0.095, is actually
the same as the distance between Memorial Day and Veterans Day.
So I mean, of course, right?
So by the way, and some people,
when they give, uh, a table like this,
they just omit the bottom half of the table because you don't need to see it,
uh, because it's the same as the top half, okay?
All right. So now let's dive a little bit deeper.
Um, and let's ask ourselves some questions like among these five vectors,
which two are the closest to each other?
Okay. So and if we just scan across all this,
it is indeed Veterans Day, Memorial's Day.
Okay? And then you'd asked that's- does that make sense?
And the answer is, yeah.
Yeah. Veterans Day, Memorial's Day are both Wikipedia pages describing holidays.
So they are actually related.
Um, in fact, they're more closely related than any of the others, right?
Um, we can also ask, uh,
what would be the second closest pair?
And look at that. It would be in fact,
the Academy Awards, and the Golden Globe Awards.
You know, by the way, when I say closest,
I mean, uh, the ones that are not the same, right?
Um, that's kind of amazing because it says that just among these five things,
the two that are closest are the two that- are the two pages that describe,
um, that- that the two pages that describe holidays, right?
The next two are ones that describe two different kinds of entertainment awards, right?
So something is happening and you can kind of imagine why,
you know, I would like, I don't know.
The- the word holiday probably appears in Veterans and Memorial or something like that.
But- but not in the super- well, okay,
not in the Golden Globe Awards or something, right?
Um, okay. Um, so it's actually kind of interesting.
Uh, so we can ask all sorts of stuff here, right?
We could have- we can say one of the most- what was the farthest distance?
And the farthest distance I guess is gonna be this,
the Golden Globe Awards and the- [LAUGHTER] and the Super Bowl.
Okay. Well, yeah.
Those seem pretty far apart.
Um, let me just say a couple of things about- about this,
because you can already do some pretty cool stuff.
Um, instead of doing with five pathetic articles like this,
we could do something like this.
We could actually- we could do this for
5,000 articles and you could have a new article that you've never seen before.
And you could ask, what is its nearest neighbor?
They- actually a very common thing to do is give me the three nearest neighbors.
Those would be the one that's closest,
the second closest, to the third closest.
So these are- these are kind of things.
Now, what's stunning about this is how something
prof- so profoundly stupid has actually quote,
figured out that, for example,
Memorial Day and Veteran's Day are related.
Um, now, let me remind you how dumb this is.
We've taken the Wikipedia page, two, two Wikipedia pages,
and we have simply counted the number of words, okay?
In the page, that obviously,
we have obscured all meaning.
It- they're no longer sentences,
they're no longer paragraphs.
Um, it just tells you how many times each word occurred.
I mean, that's all it is.
It's just really pretty simple.
Um, it doesn't understand English,
it doesn't understand anything at all.
It simply counts the words.
And when you do that, it's kinda cool.
That stuff comes up that you recognize,
like it figured out that these two things are close, right?
So, um, so we'll- we'll talk, uh,
we'll- we'll come back to this kinda stuff later in- in greater detail.
But this should just give you a little hint that even though
the concepts we're working with are embarrassed, are actually really very simple,
it's just the distance between two feature vectors.
Um, it's already interesting.
Um, I give you a couple of examples.
Uh, someone comes into an ER, right?
I take their- I'd take their feature vector.
It's a feature but you don't say it's 100 long with different things and symptoms,
attributes of the patient, all sorts of other stuff.
Okay. So I take that test, they've had results.
So I take that 100 vector or 1,000 vector, doesn't matter, I take 1,000- 1,000 vector,
and I take that and I run it against all the patients we saw in the last five years.
Okay? That's- that's a whole bunch of,
each one has a feature vector.
And then I say, give me the 10 nearest neighbors.
That means show me the patient from the past, you know,
five years who had a feature vector closest to my patient,
the second closest and so on.
Now, the closest one is interesting.
Uh, what we can do then is we can actually do immediately.
You can think of cool stuff you can do.
You can pull up that file and take a look at it and say,
"Oh- oh not cool.
Oh, boy, that- that patient developed sepsis on day 2.
That's really bad."
So maybe we better put this one on
prophylactic antibiotics right now or something like that, right?
Um, so, uh, the point is it's- it's already like pretty cool,
uh, that you can already see just with the concept of feature vector and distance.
You can already see- you should be able to
see that these practical applications are coming up, right?
I mean, I'll give you a perfect example.
Somebody says, you know,
"Go buy me a bond.
I want this bond, it's gonna be issued by this issuer and blah, blah, blah, blah."
And you're like, "There aren't any for sale."
And you go, then what this would do would say,
I would make a feature vector for bonds.
And this would someone say.
Okay. In that case,
go get me a similar bond.
And sure enough, you will find a similar bond that is for sale and you'll buy it, right?
And so- and actually a lot of things work- kind of on just- just stuff like this.