- Hi, this is Jeff Heaton.
Welcome to Applications of Deep Neural Networks
with Washington University.
In this video, we are going to start a series
for this module, next five videos,
that will talk about pandas.
Now, neural networks can accept tabular data,
which is data like you might view in Microsoft Excel.
Many traditional data science problems are set up this way.
Many competitions on Kaggle are set up this way.
We will get into image and audio processing
and textual, natural language processing,
later in this course, where we won't use pandas.
But for the beginning,
when we deal with some of the traditional datasets,
pandas is very useful for dealing with datasets
that can be divided into columns.
For the latest on my AI course and projects,
click subscribe and the bell next to it
to be notified of every new video.
Neural networks, you receive a variety of data
to predict on.
One type of data that you will read is CSV files.
And this deals with tabular data.
We'll spend a few class sessions on tabular data,
a few modules,
but we'll also get very much into images
and other more advanced inputs
that neural networks can deal with,
that neural networks are particularly good at.
But for now, we're going to look at pandas
and see how to deal with tabular data.
Here, we do the pandas read_csv.
You'll see this a lot as we read different datasets.
I'm reading this from the web,
so this should work in Collab
as well as if you're running this locally.
And it prints out just a dump of the DataFrame.
The first five row of it, anyway.
You can also use the display function,
that prints it a little nicer.
So this shows you the miles per gallon dataset.
This is a classic dataset
that you see in a lot of machine learning literature.
We'll use it some, but not a great deal.
It shows the miles per gallon
for each of these various cars,
particularly cars made in the 1970s.
These values of miles per gallon,
you're trying to predict using these other values
that you see here.
This shows a way that we can print some basic statistics
on this value.
And it shows kinda how you work with some of pandas.
If you run this, it basically gives you each of the fields.
So the field is named miles per gallon.
The next field is named cylinders, and so on and so forth.
And this gives you some of the statistics.
The mean, the variance, the standard deviation, and so on.
Kinda hard to read, but we'll make it easier to read
in a moment.
Let's see how we're actually doing this, though.
We are taking the DataFrame
and we're only using the data types
that are integer and float.
That's effectively dropping the name.
Then, we get the headers of these values
and we create an empty list called fields.
And we're going to loop over each of the field
in the column.
So each column we're going to loop over,
and we're going to append to fields
a dictionary that we're just building on the fly.
And this dictionary has four elements in it.
Four entries.
Name, which is just the field that we're on.
Mean, which is the DataFrame,
this is how you take the mean of a column in a DataFrame,
.mean, .variance, and .standarddeviation.
You can do median, a whole bunch of statistical values
are available to you.
And then we print 'em out.
So this is a good example
of how you build up a list of dictionaries,
which is a very common structure to use.
This is kinda like a database table.
And this is also exactly the format
that you can put data in
to load it directly into a DataFrame.
So here we'll see that we can take this data
that we created here and turn it into a pandas DataFrame.
Very handy, because now we can display it nicely.
So we've basically created a DataFrame from scratch.
All these mean values we put in, names,
standard deviations, and variances.
So this dictionary that we were adding one by one,
those are like the rows in this particular table
that we're creating.
Now, missing values.
The reality is,
the data that you deal with as a data scientist
always has problems.
In fact, if the data's perfect,
I would almost be afraid that there's a problem
that I haven't seen.
Here, we're going to load in auto miles per gallon again.
That's that same dataset.
But we're seeing here that the NA values
are NA and question mark,
because there's a few question marks
in this particular dataset.
There's a horsepower value that's missing.
So if you run this, it will show you that,
yes, horsepower does have an NA value.
Then we fill the missing values.
We take the median.
Now, median is usually a better value
to put in for missing values than mean
because median is not that sensitive to outliers,
whereas mean is.
If you have a huge outlier value,
that's going to affect your mean,
but it's not really going to affect your median.
So we take the horsepower and we fill in the missing,
the NA values, the missing values with that median.
You can also just drop the NA values too,
if you want to do that.
That drops the entire row that has a NA value.
And now we print out that horsepower
does indeed not have an NA, because we've filled it in.
Outliers are another thing
that you potentially have to deal with.
You can remove them or deal with them in other ways.
I'll show you how to remove them here.
We are defining an outlier
as something that is a higher number,
maybe two or more standard deviations away.
Two might be a bit close, but maybe three or higher.
But this shows you how to remove,
we wanna remove a fair amount,
so we'll remove anything
that is more than two standard deviations
above or below the mean.
The first thing we do is we calculate
what our drop rows are.
So we are essentially asking for a list of the values
where the absolute value
of each of these individual values,
the name is the column
that we're calculating the outliers of,
we subtract it from the mean
and we make sure that it is not
that amount of standard deviations above.
And then we drop these rows
that we've gotten all of the indexes to,
and we use axis equals zero
because we're dealin' with rows, not columns.
We're not deleting columns.
Inplace means that it modifies the actual DataFrame
and does not return a second DataFrame,
run the function so it's in memory.
This then loads that same dataset
and creates the feature vector.
It does that by replacing the horsepower with the median.
We're gonna drop name because we don't need it.
It doesn't really matter so much for this example.
And then we look at the length of the miles per gallon
before we drop those values and after.
And we see that we went from 398 to 388.
And I show some of the data here.
You probably really can't see it dropping fields.
It's pretty easy to drop fields in pandas.
If we wanted to drop the name field, we simply do this.
And it tells you the number of columns before and after,
or the actual columns.
You can see name was there,
and now it's very handy to be able to delete
entire columns from the DataFrame.
We can also concatenate things together.
This becomes very useful
when we start to generate dummy variables
and other variables
and we need to add 'em into our DataFrame.
What this is going to do
is we're going to create, basically, a new DataFrame
that is just two of the values together.
So this is creating just name and horsepower.
So what we're doing is we're extracting horsepower,
we're extracting name,
and then we're concatenating those two together as columns.
Axis one always means columns.
And you get this nice resulting,
we're just displaying the first five rows.
But this is how you can sub-select,
one of them anyways, you can sub-select
and build up DataFrames from scratch.
Some of the assignments will want you to do
something very much like this.
So this is useful,
to be able to take these columns or series
and put them together to build entirely new DataFrame.
You can also concatenate rows together.
What this is doing is taking the first two
and the last two rows.
So we're taking DataFrame from zero to two
and then from minus two, which means two back from the end,
to the end, and we're concatenin' 'em together.
Now, this is axis zero because we're dealing with rows.
Another thing that you will often use pandas for
is training, breaking your training data
into training and validation sets, and even K-folding.
We'll get into K-folding more in a later module.
But you can basically take your data
and break it into a training-validation split
so that you have your entire dataset here
but you're taking 80% for training, 20% for validation.
You'll usually train and fit your model on the train set
and you'll evaluate that model created from the training set
on your validation set.
This is some simple code that shows how to do this.
We're basically using the mask here to create
a mask of values.
So this is just a list of trues and falses.
The trues are the ones that will be included
and falses are the ones that are not.
So since we're taking 80%, the training DataFrame
basically gets the DataFrame with the mask applied,
opposite mask.
And we're able to print out
that we have our training set of 312,
and our validation set of 86.
This is very important.
You don't send DataFrames directly onto Keras.
You need to be able to use numpy for that.
Numpy.values is what does this.
So here you can see it's basically taken the values
and created a matrix.
Now, you may not want all of these values.
Like, that name here is not numeric,
so that could cause problems.
You can pass in a list, now you often see this in Python
as a source of some confusion.
When you have the double brace for a array,
it's not a double brace,
it just means that the index that you're passing in
is a list.
So you're saying that I want all the rows
from the DataFrame, but I want just these columns.
And if you want that one now, without the name there,
now you don't have the ugly character string in there.
It's pure numeric, which is what you want to have.
We'll see more about this
when we learn how to prepare feature vectors
for neural networks.
You can save a DataFrame that you've created or modified
to a CSV file.
This allows you to view it outside of Jupyter Notebook
using Excel, or preferably something more advanced
for CSV file viewing.
You have to be careful with Excel.
It tends to modify CSV files
and will even corrupt 'em by removing leading zeros
and other things.
And it also has kinda limited support
of text encodings like UTF-8.
Just using Excel to view things can be okay,
but just be aware, it may reformat some of the data
and corrupt it.
Now, when you save it as a CSV file,
that can be useful
because that's often what I have you submit
those files to me,
in that form, for some of the assignments.
When you compete in the Kaggle competition,
you'll often submit a CSV file for that.
So I'll post exact requirements for the Kaggle competition
for this semester, when we reach that point.
The command to save a CSV file is simply this, to_csv.
And we are essentially just shuffling this DataFrame
and then writing it back.
Index equals false.
Usually you want index equals false
because that is going to tell it
not to put a row number on each row.
And when you run this, it just says Done,
and it generates the CSV at the path that we said.
Dot means the current directory.
If you're using Google Collab, which you probably are,
you should put the path to your mapped G drive up there,
and then it will output it to a place
that you can get a hold of that file.
And you can take a look at that in Google Docs
or something else.
Google Docs is a very good program for viewing CSVs.
Now, saving a DataFrame to Pickle.
Pickle is a binary file format.
So when you save it as CSV,
you're basically writing it out to a text file,
which you often read it from.
But you can run into very, very small precision problems
when you bring things back and forth between CSV file.
Usually, this is not a problem.
It's only a problem if you're trying to diff two files.
So Pickle will ensure that you get an exact rendering
of the,
of the file.
And it also stores other metadata
that is stored in the DataFrame
that simply doesn't fit into a CSV, like row numbers.
If I run this, it simply tells me that I'm done.
Again, it did this reindexing again.
We will load the Pickle file back.
It's done with this, pickle load.
Pickle dump is how you save it.
And this loads it back in.
Now, here's something important to note.
This is the file, these are all the columns and everything.
But here's how you can tell the difference
between loading a Pickle file and loading a CSV file.
Notice, the row numbers don't line up
because we reindexed it
and we didn't rebuild that index.
So that was stored in the Pickle file.
That would've been lost in the CSV file.
Now typically, you wouldn't mind necessarily losing those,
but your actual row numbers
from when it was first loaded,
these are only out of order because we shuffled it.
That is not something that would be stored into a CSV file.
Thank you for watching this video
on the introduction to pandas.
We will see how to do more advanced processing with pandas
in the other parts of this modules.
This content changes often, so subscribe to the channel
to stay up to date on this course
and other topics in artificial intelligence.