Welcome to Engineering Python.
This is a Python programming course for engineers.
In this video, I'll talk about how to use Pandas for data analysis.
Pandas is a Python library for data analysis.
It is usually used together with NumPy.
Before we start, we need to import numpy as np, and import pandas as pd.
First, let's create a DatetimeIndex using the date_range function in pd.
The date range starts from January 1st, 2018, 30 minutes and 40 seconds past 8 pm.
The periods is 10, the frequency or step is 2 seconds.
This is the output.
It starts from 40th second, then this is 42nd, 44th and so on.
There are a total of 10 datetime strings.
The last one is 58th second.
In this example, we create a DatetimeIndex with four days because the periods is 4 and
the frequency is one day.
Some other commonly used frequency letters are listed here.
A means yearly or annually, M means monthly, W means weekly, and so on.
Now, we create a DataFrame with a DatetimeIndex and these column names using the DataFrame
function.
A DataFrame is a 2D labeled data structure with columns of potentially different types.
We use NumPy random function to generate 4-row and 3-column randomly distributed data from
the standard normal distribution.
The index is the DatetimeIndex we just created.
It contains 4 dates.
We give each column a name.
The first column is a string A, the second is B, and the third is C. They are enclosed
in this list and assigned to columns.
This is the first DataFrame we created.
The top row is the column names.
The leftmost column is the DateTime index.
This part contains the 4-row by 3-column random numbers.
We can also give the index column a name using this statement.
This is the new DataFrame.
We can create a DataFrame using a dictionary of one or more Panda Series.
In this dictionary, one and two are the keys.
They are paired with two series.
Each series will just create a single column data.
The first series contains two numbers, 1 and 2.
The indexes are a and b.
The second series contains three strings t1, t2, and t3, the indexes are a, b, and c.
We create DataFrame 2 by merging these two series in the dictionary together.
This DataFrame has three rows and two columns.
This NaN means not a number or empty.
This is because the first series originally has only two rows.
When it's expanded to three rows, this cell will be empty.
We can see from this example that a pandas DataFrame can contain heterogeneous data.
We have numbers, an empty cell, and strings all in the same DataFrame.
This is very different from a NumPy ndarray because it can contain homogeneous data only.
That means, if one element is a number, all other elements must be numbers as well.
If one element is a string, all other elements must be strings as well.
After the DataFrame is defined, we can just type in its name to view its content.
If we only want to see the top 2 rows we can use the head function with an argument 2.
The output contains only the data of January 1st and 2nd.
This is useful when we are dealing with large datasets and we don't want to see the entire
DataFrame.
Similarly, we can use the tail function with an argument 3 to view the bottom 3 rows.
The output contains only the data of January 2, 3, and 4.
The index attribute stores all the DataFrame indexes in the leftmost column.
In this case, it's a DatetimeIndex from January 1st to January 4th.
The columns attribute stores all the column names in the top row.
In this case, it's A, B, and C. The values attribute stores the actual value.
In this case, it's the 4-row by 3-column random numbers.
The describe function shows the basic statistics of the DataFrame.
For example, count means there are 4 numbers in column A.
The mean of these four numbers.
The standard deviation.
The minimum value.
The 25th percentile.
The 50th percentile.
The 75th percentile.
The maximum value.
The numbers for columns B and C can be explained similarly.
We can use df dot capital letter T to show the transpose of the DataFrame.
This is the output.
The rows are changed to the columns and the columns are changed to the rows.
Transposing is not done in place.
So, the original DataFrame is not changed.
The sort_index function will sort the DataFrame by index.
When axis equals 0 and ascending is False, we are sorting by the DateTime in descending
order.
So, January 4th data is now on top and January 1st data is now at the bottom.
When axis equals 1 and ascending is False, we are sorting by the column names in the
descending order.
So, column C is now on the left and column A is now on the right.
We can also sort by values.
In this example, we sort the DataFrame in ascending order based on the values of column
B. Now the datetime values are January 1st, 3rd, 4th, and 2nd.
Sorting is not done in place.
So, the DataFrame is still the old one.
If now we want to select a single column A, we can use string A as the index like this.
The output includes both the DateTime and the associated A values.
We can also select a sub-DataFrame using the location function and labels.
This part defines the rows, from January 2nd to January 4th, including both.
This part defines the columns, columns B and C.
As a result, this DataFrame contains 3 rows and 2 columns.
We can also select a sub-DataFrame using the index location function and the indexes.
Remember, Python indexes always start from 0.
This part defines the rows, from first row to third row but not including the third row.
This part defines the columns, the zeroth column and the second column.
As a result, this DataFrame contains 2 rows and 2 columns.
Here I show the DataFrame df again.
We can compare column A's elements with 0.
This is an element-wise comparison.
The result shows only this number is greater than zero.
It's marked by True.
All other values are less than 0.
So, these positions are filled with False.
We can also use this comparison result as an index like this.
This is called boolean indexing.
We show the rows where A's value is positive.
This comparison will check every element in the DataFrame df and find those that are greater
than zero.
They are marked by True.
Other elements are marked by False.
When we use this comparison as the boolean index, it only returns the elements that are
greater than 0.
Others will be blocked and their positions are filled with NaN.
Here I show the DataFrame df2 again.
This DataFrame with the label two means we select column two.
We further compare these elements to see if they are in this list that contains three
strings.
The isin function works like a filter.
t1 is in this list.
So, it's True.
t2 is not in this list.
So, it's False.
t3 is in this list.
So, it's True.
Now, we use this filter as the index to find the rows that meet this requirement.
This is the result.
We can also use these indexing and slicing techniques to assign new values to some of
the numbers in a DataFrame.
We still use this df DataFrame as an example.
In this statement, we set all the elements in column B to 1.
This is the new DataFrame.
Then we set the element in the zeroth row and the second column to 100.
This is the new DataFrame.
In this statement, we set all the elements that are less than 0 in the DataFrame to their
opposite number.
In the new result, all the numbers are positive now.
Occasionally, we may have to deal with missing data or empty cells in a DataFrame.
As I mentioned before, missing data are denoted by NaN, which is short for not a number.
In this example, we can set the two elements in the zeroth and 3rd rows and the first column
to NaN.
This is the new DataFrame.
The dropna function with the dropping method how being equal to any will show a new DataFrame
by dropping the rows with any missing data.
As a result, only the two rows in the middle are displayed.
The fillna function with value being -1 will display a new DataFrame by filling missing
data with -1.
This is the result.
This is not done in place.
The original DataFrame is still this df.
The isnull function will check every element to see if it is NaN.
In this output, only these two cells are True and all others are False.
Okay, that was the basics of Pandas for data analysis.
The course materials are available on YouTube and GitHub.
You can watch the course videos in sequence.
If you like this video, please subscribe and share.
I'm Yong Wang.
Thanks for watching.