Hello everyone welcome to the semicolon this tutorial is a part of the series data analytics with python and today we're going to learn about
dimensionality reduction. In a previous video about data pre-processing i had mentioned about dimensionality reduction
using pca and svd that is principal component analysis and singular value decomposition
In this video we're going to go ahead and implement them using scikit-learn and check the loss in accuracy due to
reductions in dimensions
So dimensionality reduction also helps us in visualizing data as it is tough for us humans to process data
About two or maximum three dimensions so let's get started i will go ahead and import all the necessary libraries
The main ones being pca and truncatedSVD from scikit-learn
decomposition and
For the data let's import boston housing data set which has around 18 columns and try to reduce its dimension
so let's put all the feature columns in df underscore x and the target columns in df underscore y
df_x dot describe gives little description of each column.
let's blindly put this data
on linear regression by splitting randomly into tests and train and check the accuracy so that we have our benchmark set
The accuracy turns out, to be somewhere around
72% so it is
72% with all the information present
This is a poor model but we do not care because we are trying
to observe the effect of dimensionality reduction by pca and svd on this model
So now let's declare pca and reduce our data of
18 features to 10 features, not a huge thing but let's try it anyway and
we follow the same old procedure of splitting tests and train and
fitting it into the data
And as far as the accuracy is concerned after reducing it to time ten dimensions the score turns out
To be around sixteen nine percentage
So which is
not a great reduction just by two or three percentage and
We've got half of our data removed so this is okay this is good
and if we lower the number of dimensions the score goes further down the
score becomes unbearable as we move down so
this model could roughly reduce the number of features by half
let's try doing the same thing with singular value decomposition.
The number of components we try out first is six, to compare it with this course of pca and
it turns out to be around sixty one percentage
which is around two percentage less for svd when compared to pca
So pca four six features had around 63 percentage
so
Here and for the boston data set we were successfully able to reduce the data into half by
sacrificing two to five percentage of accuracy and if you went further down our accuracy dropped even more
But, if you see the covariance matrix of this data set you can see the fields are not related by a large margin
To each other and that is why dimensionality reduction here doesn't make a lot of sense
The
dimensionality reduction in pca tries to exploit the relation or the covariance between the features and try to make
features
independent of each other that is
When plotted you'll see our 90 degree between each feature
so
It tries to make the features orthogonal
but
This is not the case here so let's load some data where we can appreciate the true power of
dimensionality reduction or the pca to be
specific. The mnist handwritten data set has around 800 columns and
most of them are redundant
There are a lot of zeros and this should be a good data set to reduce to reduce the dimension
so the random forest classifier its benchmark it on it. so the random forest classifier gives an
accuracy of our 96 percentage which is quite good but
Do we need all 800 columns because that is a huge number to train
it is okay with linear regression or random forests but as
We move to deep neural networks it is up in 800 : so let's reduce the
Number of features to 10 and repeat the same thing
because the curse of dimensionality
makes training very inefficient in some training algorithms and hence it is very good to reduce the dimensionality in
fact it's very efficient
so
with 10 features we get around 90 percentage of accuracy
So the loss in accuracy is around 6 percent when the
dimensionality is reduced by 80 times which is huge and
80 times less features and
just 6
Percent of accuracy loss it is huge but let's try to improve the accuracy and get
little more closer. so let's repeat the
experiment with 20 features and 25 features and
Let's repeat the same thing again and the accuracy turns out, to be around 94 percentage which is
very little difference from the actual from all 800 features so
we could reduce the features by 40 times by just sacrificing 2 percentage of the accuracy
now pca dot explained variance
gives the variance of each feature each new feature in decreasing order and the value is
kind of
representation of the amount of information contained in each new feature so
whenever the
whenever the variance starts to get
a little low you can start ignoring the number of columns from there so
One more application of dimensionality reduction can be in visualizing data so let's try to
visualize the
classification in this very data set from the digit 0
to all other digits, so just, for the purpose of visualization and
simplification we are just trying to classify it into 2
1 0 and other all digits so
we can reduce
For the purpose of visualization we'll have to reduce all 800
features into just 2 features because it is easier for us to visualize
only two features if we increase it then it's again tough for us to visualize so let's
start wit. let's visualize with two features before visualizing let's check out the accuracy so the accuracy
turns, out, to be around 40 percent which is very poor and
there may be lot of overlaps in visualization but anyway let's go ahead and plot each point
So the red point will represent the digit zero and the green point will represent everything else we'll plot only 5000 points
So that it's not a pain on my computer and
this plot shows a nice separation of digit 0
with other digits there's an overlap and that is because reduction and dimensionality by such a huge factor but
we can
so you can understand that dimensionality reduction can help us visualize the data as well so
there's one important point to note though, if the data is inseparable in lower dimensions it can be
separated by going into higher dimensions and that is the whole concept on which
support vector machines are based on so it's not like this is not separable we can separate it at higher dimensions
So that's it for this tutorial we could visualize the return we could reduce the data by
sacrificing very little accuracy using principal component analysis and same can be done with
svm
I'm sorry a svd so that's it for this tutorial if this helped you in any way hit the like button and
If you want to watch more of these hit the subscribe button as well thank you