Data is sacred.
Every hour of every day, a new sensor is connected to the web.
Every trip, every memory, every creation.
Every discovery joins the ever-growing web we're building together.
Amidst a world of ever growing skepticism and falsehood,
data is truth. It's transparent, provable.
Most of it is unstructured streams of raw numbers amassed at dizzying rates.
But by applying intelligence to it, we can find the patterns and connections that matter
We can find the meeting hidden in the numbers.
The economist says more for you is less for me.
But the lover knows more for you is more for me too.
And when we visualize the right data, it gives us that most precious feeling.
at the intersection of art and science. Wonder...
Hello world, it's Siraj, and today, we're going to learn how to pre-process a dataset.
Yes! Preparing data is one of the most important,
yet most overlooked parts of the machine running pipeline.
A lot of introductory tutorials just have you import a preprocessed version of a dataset,
like handwritten characters or movie ratings in just a single line of code
Real world is not that easy.
Once you've decided what problem you're trying to solve or you have a question that you want the answer to,
it's time to find the right dataset.
I want to know what happened to the plans they sent you
I stored them on Microsoft Azure.
(Choking noises)
The prediction you're deep net makes are only as good as the data you give it.
Garbage in, garbage out.
So you want to make sure your data is relevant to your problem.
There are tons of resources to find publicly available data sets,
and I've Iinked to some in the description.
The de facto standard format for data is CSV.
Most software packages out there deal with data in that format,
and you can convert your data into csv format just as easily.
There's so much we could potentially do to our data
but there are three key pre-processing steps for every data set we'll cover:
Cleaning, transformation, and reduction.
We're going to look at three different data sets and go through these steps for each one
to get them ready to be fed into a model.
So, let's start with our first.
The first dataset we'll use is music-based.
This data was collected from a game called Tag-a-Tune.
In the game, two players listen to a song
and tag it with genres and instruments that they think are relevant.
When the song is over, the player who had the most correct tags get a point.
So i would win every time!
Our dataset has 25,000 songs with the correctly labeled tags.
We want to train a model on this data so that given a new song, it will correctly classify its genre.
We'll import pandas to help parse this dataset
Then the read_csv function will let us store the data
in the two-dimensional pandas data structure known as a dataframe.
Data frames are easily modifiable, and we'll call our variable newdata.
Let's explore this data first, shall we?
We'll display the first five rows using the head function with five as a parameter,
So basically each row is numbered, has an ID and then either a 1 or a 0 next to a tag
to indicate whether or not the given MP3 has that tag.
Seems simple enough.
We can use the info function to get some more data. (Only 38 megs)
So for our cleaning step, is there anything we need to do?
Not really. Each label has a simple binary tag, it's consistent
and luckily our data does not have empty values.
(But my soul does)
We can move right on to the transformation step.
What are some modifications we can make to this data
that will make it easier for our model to understand?
Well, notice how a lot of the tags are pretty similar sounding.
Like, you know, singing, female vocals...
We can generalize these features into one feature called female.
Let's create a two-dimensional list of synonyms that we find in our data,
then we can merge them and drop all the other columns, except for the first one.
For each synonym list in our matrix, let's get the max values from each of the features
and add them all to our first synonym in our data frame object,
which will effectively merge the values into one column
Then we'll drop the rest of the features from the data frame.
Now we've got more generalized features.
Next, for the reduction step, what can we remove from this data that's not necessary?
Everything seems pretty solid so let's go ahead and put it into training, validation and testing sets
that we can feed into our model.
Notice how in this example I'm not thinking about which features to use and which not to.
Before deep learning, we had to pick the right features to use to feed our model.
But deep neural nets learn high level features from whatever features we give it.
It decides for itself what is relevant to the problem from a dataset.
Architecture engineering is the new feature engineering.
The second dataset we'll use is a collection of network connections, either labeled normal or abnormal.
The abnormal connections are intruders trying to break in.
We want to be able to classify a connection given the set of other features.
When we look at this data it seems pretty dense.
No missing values, nothing really jumps out at an outlier.
So let's skip the cleaning step and move right on to transforming it.
Our numerical features are all operating on different scales,
so we should normalize them to ensure each feature is treated equally by our model.
After storing our data into a pandas data frame,
scikit-learn has a handy sub-module called standard scalar,
which we'll import then initialize.
After that, we're ready to move on to our reduction step.
We've got a lot of features and there are probably a lot that are highly correlated
We could use a technique called dimensionality reduction
to reduce the number of features we have.
This will also let visualize our data in 2D or 3D space.
This doesn't mean that our model will be more accurate necessarily,
just that our data is easier to read.
One method of doing this is called PCA,
which stands for Porsche Club of Amer...
Wait, wrong definition!
Principle component analysis.
My data's got so many features, so squash 'em into three like little creatures.
First I'll normalize, then I'll correlation matricize.
Pull eigenvectors and values out of it's eyes [identity matrices]
Sort them. How many dimensions do I want?
I'll select that many eigens up front. (Yeah!)
Make a projection matrix from 'em and use it to turn my data three dimensional
Prod em, so I can judge 'em.
So let me summarize this process again.
Let's say we had four features and we wanted to reduce them to just two using PCA.
There are five steps to this.
The first is to normalized the data once we have it stored in a variable.
Then we want to compute a covariance matrix.
To construct this, we compute the covariance between each feature with every other feature.
So we subtract the mean from the feature matrix, calculate the transpose
and multiply it by the feature matrix minus the mean.
Then we take that whole value and divide it by the number of features minus one.
This gives us our covariance matrix
Next we'll perform eigendecomposition on it to get the eigenvectors and eigenvalues.
Eigen -- isn't it such a fun word, wouldn't you say?
Eigenvectors are the principal components of a dataset.
The give us the directions along which our transformation acts,
The eigenvalues give us the magnitude of each.
We'll sort both in descending order, then create a matrix out of them.
We'll use this matrix to transform our original feature matrix via the dot product,
We could then plot our data into 2D space and use these principal components
to replace our many features.
Let's look at one more data set, this time for airline prices
for flights between New York and Paris.
We want to predict the ticket price from just the departure date.
We've got departure and arrival dates, airports, and flight prices of 220 days before departure.
Notice how we've got quite a few missing values in our data,
so for our cleaning step, we could remove these values, fill them with zeros,
fill them with the average price across all the days, or try to predict them using a learning algorithm.
Let's go ahead and calculate the average price for each row across all days
using the mean function, and then we'll iterate through the data, and if it's null, replace it with the mean price.
Then we can smooth our data. That means finding outliers in it that we can remove.
To find these, we could run clustering or regression algorithms,
on certain values to find the outliers, and then remove them.
Or just remove them by eye.
Since our dataset is small, let's do the latter.
No need to reduce our data. This seems like a good set.
Let's break it down. There are three steps to preprocessing a dataset.
Cleaning, transformation, and reduction.
Deep learning learns the relevant features from our data
so architecture engineering is the new feature engineering.
And principal component analysis is a popular dimensionality reduction technique
that can be implemented with scikit-learn.
the winner of the coding challenge from the last video is Charles David Blot.
Charles David used just numpy to build a three-layer neural net
capable of predicting an earthquake
and he used the random search strategy to find the optimal hyperparameters for his model.
Wizard of the week.
And the runner-up is Siby-Jack Grove.
He used TensorFlow for his prediction using just three inputs.
The coding challenge for this video is
to use a dating dataset to predict if
someone gets a match based on their personality traits.
Details are in the readme. Post your GitHub link in the comments.
and I'll announce the winner next video.
Please subscribe if you want to see more videos like this.
Check out this related video, and for now,
I gotta predict if roses really smell like poo poo poo, so, thanks for watching.