hello everyone Ken here back with a new video for you today
if you're familiar with my channel you know that my goal is to make data
science accessible to everyone with that being said I get a lot of feedback about
what tools to use you know where to start how to get a project going and I
thought it was important to make a couple videos about the fundamentals of
getting into data science this video is going to be about the fundamentals of
data exploration I'm gonna make another video about the fundamentals of data
manipulation and then a third video about some actual data analysis how to
get started there so if there's good feedback related to these I'll be making
a couple more on more advanced topics as well so I know that starting can can be
the hardest part you got to get over that initial hump to be able to hit the
ground running and start doing really cool projects on your own the goal here
again is to get you over that hump and to be able to give you the tools and the
skills to start having that autonomy start being able to implement and
integrate that curiosity that you have into these problems this video is going
to be held tutorial style I'll download an actual data set and I'll
work through some of the main and important functions that you need to be
able to analyze this data I'll be using Python and we'll be focusing primarily
on using the pandas library to work through this here so I'm going to say
show you how to you know for the most parts out of your environment use spyder
which is an IDE that will is good for beginners and I think it'll help you get
used to working with data in Python as usual if you enjoy this video please hit
that like button and if you'd like to see more videos similar to this please
subscribe and hit that notification button if you want to be alerted when
the next two videos in this series will be released all right without being said
let's get started so I have you at the end the Conda homepage and so anaconda
is a data science environment that you can download for Windows for Linux for
Mac there's plenty of great videos out there on
to install this but I'm going to share the link to this to the data source that
I use to my github to my personal information anything that you need
especially the code for this video will be in the description below if you have
any questions about anything that's going on the video please leave those in
the comment section as well so I'll let you pause here and download anaconda all
the default settings should be fine I'll also have a link to another video
about how to actually install this embedded as well if we want to look at
our data set we're going to be using this one from kaggle you have to create
an account and after you do you can download all this information this is a
good data set it's not perfectly clean so this will be really important for our
data exploration effort now after you've downloaded anaconda we can press Windows
or the command line and go to the anaconda prompt this is more practical
for work on a Windows computer than a Mac but we want to check that we have
pandas installed so pandas is a really good library it has a lot of the
fundamentals of data science in it you can load in data you can manipulate data
and you can also analyze data panda works off of data frames pandas works
off of data frames so a data frame is very similar to a excel table basically
you have rows and columns and we manipulate those rows and columns to be
able to you know generate features or to get exactly what we want so we can do
pit install and us and that will actually install pandas if you don't
have it and if you do have it it'll give you this message so I obviously already
have it now we just do type in spider SP ye er not I and this will open our IDE I
like spider because it has a little bit more of a graphic user interface so you
can see close that and right here is where you type the code right here is
where if you click variable Explorer your variables will show up and here is
the console so let's run something real quick so for
seven and if you highlight it and press ctrl under it'll run so as you see that
comes out on the console here if we make this into a variable so let's do X is
equal to seven same thing then it'll show up up here
after we've assigned it so I'm just gonna restart this real quickly perfect
now we want to go in and actually download this data so I've really I've
already downloaded it but you download it here and I recommend putting it into
a new file so I'm gonna put this into one of the files that I've been working
on here so let me go to it I'll show you what that file looks like so there we go
I have these two data files and I'm going to create a new Python file in
here in a second so I want to go and this is what's known as the working
directory so we can go in here and we want to set the working directory to
where our Python files are so you can go in the file explorer and you can see
that these are located here now we want to save this file and we'll call it
exploration too because I already have an exploration one and save that in the
same file location great now okay so the first thing we want to do is import
pandas and we can assign it like a nickname so we do as PD so that means
when we use it we don't have to type in all of all pandas it just saves us a
little bit of time all right so we import it that means the module is in
our you know our local workspace so the next thing we want to do is download the
data so a lot of people will call this DF because it's a data frame I'm just
going to call it data so we remember what it is the PD and we read we use the
function from pandas read CSV and I'm gonna go in the file explorer where
these files live and for me it's easier I just copy what's in rename and you
have to put it in quotes here and then I will control enter and run this so this
is reading in our data and we should be able to see that it right here
momentarily after we have that data read in we can start manipulating it start
exploring it and that's when the real fun takes place so as you can see one of
the nice things about spider is you can see this here in Jupiter notebook you
don't see all your variables you have to type in and explore it that way so we're
gonna go and we want to know what the shape of our data frame is so we're
going to do data that's how many rows and how many columns oops
great and so as you can see this is the number of rows and this is the number of
columns so we have about you know 450,000 rows and 22 columns if we want
to look at you know kind of a sample of the data that's there we can do data dot
head and that'll give us the first five rows of the data frame
and we can see that okay you know we have a bunch of URLs bunch of difference
a bunch of the same city in the first five in the price of cars years etc if
we want to look at more than just five we can do you know ten for example and
that'll give you ten rows same thing goes if you want to see the last few you
can just type in tail and I'll give you the bottom the bottom five so we'd
expect let's see here we have our URL Zanesville so it looks like this data is
sorted by by city which is interesting okay the next thing we want to see is
what the columns are the data columns again when you when you see it this way
you can see the column names but a lot of people you know they don't
necessarily they don't have this on their IDs so this is a quick way to see
all of the column names if you want to change them you can set data columns
equal to and then basically a list of whatever columns you want as long as
they are number now the next thing we want to do
is get an understanding of the data types that we have how many inputs for
each value for each column that we have and we do that by doing data dot info so
we can see here that for URL it looks like there are no null values because
this number is matching the size of the data frame but for example for condition
there's something funky with you know almost half of the data here we also see
this object in float and I think that's all we see so float is a decimal number
int is an integer number an object in this case is usually gonna be well it's
an object but we can convert it into string if we want to use it in a
different way so you know something that jumps out to me is that cylinders which
I would expect to be numeric as an object and so that's something we'd want
to look into further and I'll talk about that in the next video about data
manipulation so one other thing we can do just to again get familiar with this
data set is data dot describe and this gives us information on all of the
floats in ants all of the numeric data that we have so we can see the the
average price we can see the you know standard deviation of the price we can
see that the average year was around 2008 the cars and some other information
related to that here you know it's included in data dot info but data if
we'd look at d-type we can see what types do types are we can see what the
data types are and you know sometimes these give us problems so if a numeric
column should be is classified as a string we're not going to be able to you
know aggregate it or do some of the functions that are important related to
it so it's definitely something to keep that off now when you're looking through
data another really important thing is to evaluate
the null values so data isn't gonna be perfect it's not gonna be clean and you
want to get an understanding of kind of the quality of your data set so we can
start by doing data dot is null and that'll show all of the null values in
our data set if we want to do if you want to see but that by column we can do
is null dot any and that'll show which columns have null values in general so
as you can see URL city city or own price it looks like they don't have any
no values but pretty much everything else in our data set has null values you
know with that being said you know that's very common and pretty much any
dataset now if we want to look and see what percent of data has null values we
can come here and copy this you take that and then some it so this is gonna
be kind of weird but in in Python true is considered one and false is
considered zero so if we sum up all of these columns we're getting the number
of values that are null because is null sets all null values to true so we can
see that there's you know a lot in size for example
but not that many in year and so we can take that information and divide it by
data a dot shape right and so shape if we look at it we have two inputs to the
rows and column so we want to divide it by the rows and to get just that row
number we take the first entry in it and we did by putting the 0 in parentheses
so we can see now that that percent of null values for each of these so sides
you know 65% cylinders 40 percent manufacture is small so this is
something you know size is something might want to remove from our data set
because it might cause more problems than it's worth you know we could also
use that as a input variable so we could perhaps train
a model on if you know if someone has actually put the size or not it doesn't
matter what the size is because that can tell us more information as well so you
know describe up here has told us some information about the numeric data let's
look into some information about the non numeric data so if we do data dot you
know type right here we can see all of the entries for that column right that's
the exact same thing as doing data type it's the exact same thing so this is
useful this bottom one is useful if there's a space in the column header
this one for me is just faster when there isn't so if we do data dot type
and we do dot unique we can see all of the unique values for that column so we
can see that there's sedan none is not a number coupe SUV van minivan truck etc
we can also if we want to look at how many we have of each different type we
do data type value counts now we can see that while we have more sedans than
anything else a lot of SUVs a lot of pickup trucks but
very few minivans and very few off-road vehicles which you'd probably expect on
the off-road vehicles similar to what we did with the nulls we can actually take
a look at you know what percent of sedans SUVs etc are you know did the
data set to make is the data set made up of so we can take those value counts
that we had and we can take data type not know so we only want to divide it by
the nominal values so we see the nons there right and then we also want to
stop this and that should give us the percent of
sedans SUVs pick ups etc all the way down it makes sense to do these things
with a lot of the different variables that you have in the data set or the all
the different columns that you have because you really get a feel for what
variables might be useful to do further analysis on another great way to explore
data is also with graphs so we're gonna start making some graphs here a lot of
people use matplotlib but if you're just you know trying to make some very simple
graphs for you to understand and see the data better I think you can do it
straight from pandas that's very effective so we can do the same thing
new data dot let's look at year so as we can see we see the there's all these
year value z' we can actually do dot a plot and then kind evil to pissed so we
can see the distribution of all of these plots right here which is pretty cool
you can see there's a bias towards newer cars we can also the exact same thing we
do data year dot hist you don't even need the plot but it's versatile and
that you can do it either way so looking at this I don't think this tells us a
ton of information let's make it so that some of the bins so that we have more
bars rather than just this group so we can do bins equal to let's do 50 so as
you can see that graph there's a lot more in the spike kind of that looks
like near 2010 ish we can also compare these two if we run them at the same
time and you can see that there's a large volume here which is you know if
you add up all these bars and the bottom they reach that height another
interesting chart that so the histogram is probably my favorite chart to
understand data you know get understand if it's no
you gotta understand if there's weird outliers but you also can use a bar
chart for example to show really interesting information so let's take a
look at the car types again so data types and then value counts that's type
there we go now so we see all these numbers but it'd be cool if we could
visualize this and this is pretty easy to it we do plot bar that's art kind
equal to bar and there you have it you have a very simple quick visualization
that should be able to explain the makeup in this data set of the different
car types I think that's enough data exploration for today these are a lot of
the most basic concepts in the next two videos again I will discuss data
manipulation and also some further analysis some would consider the
analysis you know very related to data exploration I'll get into you know
mapping correlation within the data frame and you know evaluating what
features are related to other features thank you so much for watching please
stay tuned for the next episode and until then enjoy your data science
journey