Hello Everyone, welcome to The Semicolon.
This tutorial is a part of the series data analytics with python
and today we are going to learn about pandas
pandas are cute, very cute
but sadly they can't help us analyze data
so we use a library pandas which helps us manipulate, store
and perform lots of operations on data
pandas store data in the form of dataframes
and it makes our work of shaping data,
to fit machine learning algorithms very easy
In this tutorial, I am going to show you
to store, manipulate and
analyze data
this library comes in very handy in data exploration.
So lets get started
If you have anaconda package installed,
you can just import pandas as pd
if you don't , google pandas, download the library
install it and then import pandas as pd
import numpy as well,
now let us create a dataframe, data frame is a data structure
that stores our data in the forms of a table
using the numpy array,
we can create a matrix of random numbers
in our case 10 X 4 matrix
then specify the column names as an input
to our dataframe
and this is our dataframe,
here our column names are A, B, C, D as specified
now you can also
create dataframes using dictionary
instead of passing a matrix, pass a dictionary
with the column names and the column vectors
If you have confusion with dictionaries, you can
refer my tutorials on python basics
let us look at some basic functions
which will help us to explore the data
you can type data frame object dot
dtypes for data types of
each column
type dataframe object dot columns
for the column names
you can type dataframe object dot
values and you get all the
values available in the dataframe
as a numpy array
to have a peak at the data frame or the
top 5 values type dataframe object dot head
it returns the top 5 rows. you can also specify
the number of rows as an argument to the
head function, similarly to
look at the bottom 5 rows, you can type
dataframe object dot tail and also specify
the number of rows as an argument
you can look at the index
by typing dataframe object dot index
you can look at some quick statistics by
typing dataframe object dot describe
you can look at the count, mean, standard
deviation, 25th percentile
50th percentile, 70th percentile, minum
and the maximum value
the data which you are seeing on the screen doesn't make much sense now
because we have generated it from the random numbers
but when you do it with actual data
it does give some valuable insights about the data
sorting is an important part of data manipulation
we type dataframe object
dot sort_index
and mention the axis, axis = 0
is for sorting columns
or sorting vertically
axis = 1
is for sorting the rows or sorting horizontally
and mention the ascending attribute
true or false by passing the column name
to the parameter, we can sort the data relative to a column
let us look at how we select data
we can select a particular column
we can type dataframe object
with the column name enclosed in the square paranthesis
and that is how we select a column
you can select specific rows
or a range of rows by using the ':' operator
dataframe object
0:3
selects rows indexed at 0,1 and 2
a similar example of selecting just one row
would be dataframe objects [0:1]
a faster method to select a whole row,
would be using the loc attribute
and passing the index of the rows
to select a particular value, you can pass the column names
as a list too
you can use ':' to select all the rows
instead of mentioning a specific row
you can use dataframe object dot
at index, column names in square paranthesis
you can also use iloc to access
a whole row
now, the difference between loc and iloc
is that loc works on labels
and iloc works on position in the dataframe
Iloc wouldn't
be dependent on the index and
loc would be
let us learn how we can apply conditions to dataframe
and select some specific data
or filter data according to our conditions
you can type the conditions within the square parenthesis
and call the dataframe object
when ever the condition is true, it returns the rows which
satisfy the condition
for example df[df.A>5]
gives us all the values where
A is greater than 5
or df[df.A> 0.5] would give us all the rows
where values of A are greater than 0.5
In the real- time datasets, we have lots of missing values
there are methods to fill the missing values
let us create some missing data first
and then we can learn how we fill them
so, we create missing data
by selecting all the values of A where A
is less than 0.5, so it selects
all the values of A where A is less than 0.5
but the remaining values are left empty or missing.
now you have two choices, as to what
to do with the missing data
you can either fill the missing data with some value or
drop the whole row. So to drop the whole row
dataframe object dot dropna
to remove all the column with null values
but that is not a nice way
we can use the fillna function to fill
missing values with a specific value
there are methods to estimate what the value should be but
let's not get into that now
all the values where NAN
where Not A Number (NAN) exists
they got replaced with 1
when we used fillna function
we can take the mean values by typing
dataframe object dot mean
it would give the mean of all the columns
you can change the axis to 1
as input to the mean function and then
it would give the input of all the rows
An important feature is to read the csv file
Most of the datasets
are available in the form of a csv file
so it is important that we know how to read it.
so we use read_csv function
and pass the file name
it is pd.read_csv
this is the data I have which we will be using for
linear regression in next tutorials
we can apply all the functions which we have learned
the head or
the standard deviation
that is it for this tutorial
all the codes are available on my github account
and the links for it is mentioned in the description below
hit the like button and subscribe to the channel
as I have lot more of interesting tutorials coming up. Thank you