StatQuest breaks it down into bite-sized pieces hooray
Hello, I'm Josh stormer and welcome to stat quest in this StatQuest
We're going to go through principle component analysis PCA one step at a time using singular value decomposition
SVD
You'll learn about what PCA does
How it does it and how to use it to get deeper insight into your data
Let's start with a simple dataset
We've measured the transcription of two genes gene 1 and gene 2 in 6 different mice
Note if you're not into mice and genes
think of the mice as individual samples and
The genes as variables that we measure for each sample
For example the samples could be students in high school and the variables could be test scores in math and reading
Or the samples could be businesses, and the variables could be market capitalization and the number of employees
Ok now we're back to mice and genes because I'm a geneticist and I work in a genetics department if
We only measure one gene. We can plot the data on a number line
mice 1 2 & 3 have relatively high values and
mice 4 5 & 6 have relatively low values
Even though it's a simple graph it shows us that mice 1 2 & 3 are more similar to each other than they are to
mice 4 5 & 6
If we measured two genes then we can plot the data on a two-dimensional XY graph
Gene 1 is the x-axis and spans one of the two dimensions in this graph
Gene - is the y-axis and spans the other dimension
we can see that mice 1 2 & 3 cluster on the right side and
mice 4 5 & 6 cluster on the lower left-hand side
if we measured three genes we would add another axis to the graph and make it look 3d ie
three dimensional
The smaller dots have larger values for gene three and are further away
The larger dots have smaller values for gene three and are closer
If we measured for jeans however we can no longer plot the data
for jeans require four dimensions
All
So we're going to talk about how PCA can take four or more Jean measurements and thus
four or more dimensions of data and make a two dimensional PCA plot
This plot will show us that similar mice cluster together
We'll also talk about how PCA can tell us which gene or variable is the most valuable for clustering the data?
For example PCA might tell us that gene 3 is responsible for separating samples along the x axis
Lastly, we'll talk about how PCA can tell us how accurate the 2d graph is?
To understand what PCA does and how it works let's go back to the dataset that only had two genes
We'll start by plotting the data
Then we'll calculate the average measurement for gene 1 and
the average measurement for gene 2
With the average values we can calculate the center of the data
From this point on we'll focus on what happens in the graph we no longer need the original data
Now we'll shift the data so that the center is on top of the origin in the graph
Note shifting the data did not change how the data points are positioned relative to each other
this point is still the highest one and
This is still the rightmost point
Etc
Now that the data are centered on the origin
We can try to fit a line to it to do this
We start by drawing a random line that goes through the origin
Then we rotate the line until it fits the data as well as it can given that it has to go through the origin
Ultimately this line fits best
But I'm getting ahead of myself first we need to talk about how PCA decides if a fit is good or not
So let's go back to the original random line that goes through the origin
To quantify how good this line fits the data PCA projects the data onto it
And then it can either measure the distances from the data to the line and try to find the line that minimizes those distances
Or it. Can try to find the line that maximizes the distances from the projected points to the origin
If those options don't seem equivalent to you
We can build intuition by looking at how these distances shrink when the line fits better
While these distances get larger when the line fits better
Now to understand what is going on in a mathematical way, let's just consider one data point
This point is fixed and so is its distance from the origin in
Other words the distance from the point to the origin
Doesn't change when the red dotted line rotates
When we project the point onto the line
We get a right angle between the black dotted line and the red dotted line
that means that if we label the sides like this a
b and c
Then we can use the Pythagorean theorem to show how B and C are inversely related
Since a and thus a squared doesn't change
if B gets bigger
then C must get smaller
Likewise if C gets bigger, then B must get smaller
Thus PCA can either minimize the distance to the line or
Maximize the distance from the projected point to the origin
The reason I'm making such a fuss about this is that
Intuitively, it makes sense to minimize B. And the distance from the point to the line
But it's actually easier to calculate C the distance from the projected point to the origin
so PCA finds the best fitting line by
Maximizing the sum of the squared distances from the projected points to the origin
So for this line
PCA projects the data onto it and
Then measures the distance from this point to the origin
Let's call it d sub1
Note I'm going to keep track of the distance as we measure up here and
Then PCA measures the distance from this point to the origin. We'll call that D 2
Then it measures d3
d4
d5 and d6
Here are all six distances that we measured
The next thing we do is Square all of them
The distances are squared so that negative values don't cancel out positive values
Then we sum up all these squared distances and that equals the sum of the squared distances
For short. We'll call this SS distances or sum of squared distances
Now we rotate the line
project the data onto the line and
Then sum up the squared distances from the projected points to the origin
And we repeat until we end up with the line with the largest sum of square
distances between the projected points and the origin
Ultimately we end up with this line it has the largest sum of squared distances
This line is called principal component one or PC one for short
PC one has a slope of
0.25 in
Other words for every four units that we go out along the gene 1 axis
We go up one unit along the gene to access
That means that the data are mostly spread out along the gene one axis and
Only a little bit spread out along the gene to access
One way to think about PC one is in terms of a cocktail recipe
to make PC one
mix four parts gene one
with one part gene to
Pour over ice and serve
The ratio of gene 1 - gene -
Tells you that gene 1 is more important when it comes to describing how the data are spread out
Oh, No terminology alert
mathematicians call this cocktail recipe a linear combination of genes 1 & 2 I
mention this because when someone says PC 1 is a linear combination of variables
This is what they're talking about
It's no big deal
The recipe for PC one going over 4 and up 1 gets us to this point
We can solve for the length of the red line using the Pythagorean theorem the old a squared
equals B squared
plus C squared
Plugging in the numbers gives us a equals four point one two
So the length of the red line is four point one two
When you do pca with SVD the recipe for PC one is scaled so that this length equals one
All we have to do to scale the triangle so that the red line is one unit long is to divide each side by
four point one two
For those of you keeping score
Here's the math worked out that shows that all we need to do is divide all three sides by four point one two
Here are the scaled values
the new values change our recipe
But the ratio is the same we still use four times as much gene one as gene two
So now we are back to looking at the data
the best fitting line and the unit vector that we just calculated oh
No another terminology alert this one unit long vector
consisting of
0.97 parts gene one and
0.24 two parts gene two is called the singular vector or the eigenvector for PC one and
the proportions of each gene are called loading scores
Also while I'm at it
pca calls the sums of squares of the distances
for the best fit line the eigenvalue for pc 1
In the square root of the eigenvalue for pc. One is called the singular value for PC one
BAM that's a lot of terminology
Now that we've got pc1 all figured out, let's work on PC to
Because this is only a two-dimensional graph
PC 2 is simply the line through the origin that is perpendicular to PC 1 without any further
optimization that has to be done
And this means that the recipe for PC 2 is negative 1 parts gene 1 to 4 parts. Gene 2
If we scale everything so that we get a unit vector the recipe is
negative zero point two for two parts gene one and zero point nine seven parts gene -
this is the singular vector for PC - or the eigenvector for PC -
These are the loading scores for PC to
they tell us that in terms of how the values are projected onto PC -
Gene - is four times as important as gene one
Lastly the eigenvalue for pc. - is the sum of squares of the distances between the projected points and the origin
Hooray we've worked out pc1 & pc2
To draw the final PCA plot we simply rotate everything so that PC one is horizontal
Then we use the projected points to find where the samples go in the PCA plot
For example these projected points correspond a sample six
So sample six goes here
sample two goes here and
Sample one goes here etc
Double bam that's how PCA is done using singular value decomposition
Okay one last thing before we dive into a slightly more complicated example
Remember the eigenvalues
We got those by projecting the data onto the principal components
Measuring the distances to the origin then squaring and adding them together
We can convert them into variation around the origin by dividing by the sample size minus one
for the sake of this example
imagine that the variation for pc1 equals 15 and the variation for pc2 equals 3
that means that the total variation around both pcs is 15 plus 3 equals 18 and
That means PC 1 accounts for 15 divided by 18
equals zero point 8 3 or 83 percent of the total variation around the PCs
Pc2 accounts for 3/18 equals 17% of the total variation around the PCs oh
no another terminology alert a scree plot is a graphical representation of the percentages of
variation that each PC accounts for
We'll talk more about scree plot Slater
BAM
Okay now let's quickly go through a slightly more complicated example
PC a with three variables in this case that means three genes is pretty much the same as two variables
You Center the data?
You then find the best fitting line that goes through the origin
Just like before the best fitting line is PC one
But the recipe for pc1 now has three ingredients in
This case Jean 3 is the most important ingredient for pc1
You then find pc2 the next best fitting line
Given that it goes through the origin and is perpendicular to PC one
Here's the recipe for pc2
In this case gene one is the most important ingredient for PC to
Lastly we find
PC three the best fitting line that goes through the origin and is perpendicular pc1 & pc2
If we had more genes we just keep on finding more and more principal components by adding
perpendicular lines and rotating them
in
theory, there is one per gene or variable but in practice the number of PCs is either the number of variables or
the number of samples whichever is smaller
If this is confusing don't sweat it
It's not super important, and I'm going to make a separate video on this topic in the next week
Once you have all the principal components figured out you can use the eigenvalues ie the sums of squares of the distances
to determine the proportion of variation that each PC accounts for in
This case PC one accounts for 79 percent of the variation
PC to accounts for fifteen percent of the variation and
PC three accounts for six percent of the variation
Here's the scree plot
Pc1 & pc2 account for the vast majority of the variation
That means that a 2d graph using just pc1 & pc2
Would be a good approximation of this 3d graph since it would account for
94% of the variation in the data
To convert the 3d graph into a two-dimensional PCA graph
We just strip away everything, but the data and pc1 & pc2
Then project the samples onto pc1 &
Pc2
Then we rotate so that PC one is horizontal in PC two is vertical this just makes it easier to look at
Since these projected points correspond a sample for
This is where sample four goes on our new PCA plot
Etc etc etc
Double bail
To review we started with an awkward 3d graph that was kind of hard to read
Then we calculated the principal components
then with the eigenvalues for pc1 & pc2
We determined that a 2d graph would still be very informative
Lastly we used pc1 & pc2 to draw two dimensional graph with the data
If we measured for jeans per mouse we would not be able to draw a four dimensional graph of the data
wall
But that doesn't stop us from doing the pca math
Which doesn't care if we can draw a picture of it or not and looking at the screen in?
this case
Pc1 & pc2 account for 90% of the variation so we can just use those to draw two dimensional pca graph
So we project the samples onto the first two pcs
These two projected points correspond to sample two
So sample two goes here
BAM
Note if the scree plot looked like this where PC 3 and PC four account for a substantial
Amount of variation then just using the first two pcs would not create a very accurate
representation of the data
Wha-wha
However even a noisy PCA plot like this can be used to identify clusters of data
These samples are still more similar to each other than they are to the other samples
Little bam
Hooray we've made it to the end of another exciting stat quest if you liked this stack quest and want to see more please subscribe
And if you want to support stack quest please consider buying one or two of my original songs
The link to my Bandcamp page is in the lower right corner and in the description below
alright until next time quest on