Hi this is Jeff Heaton, welcome to Applications of Deep Neural Networks,
with Washington University. In this video we're going to look at grouping, sorting,
and shuffling of a data set within Pandas. this allows you to do things like
an SQL group by where you summarize and aggregate your data to process it in
additional ways to get it ready to go into the neural network for the latest
on my AI course and projects click subscribe in the belt next to it to be
notified of every new video pandas dataframes can also be grouped sorted
and shuffled in this part we're going to look at how to do just that
shuffling a data set is very useful you want to be sure that you don't have a
number of common targets near each other thymine the worst possible thing would
be to sort your data set according to the target and then split it evenly to
get your training set and test set from it because your test set would have all
low values your training set may have all high values or something similar to
that so shuffling is a very good thing to do this command that I'm running here
takes the auto mpg data set now if you one consistent shuffling which is not
bad that's actually usually preferable because you may need to reshuffle your
data set to add columns or something like that
you probably don't want the order changing because you've already
established some benchmarks you do want it random but you want to consistently
random so you can pick a seed number here I'm choosing 40 to Hitchhiker's
Guide to the galaxy that's where that number comes from it's a fiction book
you can look at nut but this is this is basically taking the this line here
resource it this line causes these numbers down this list to be in order it
doesn't really matter if you leave the second line off notice if I run it here
those numbers now are are not put back to account between zero and
or 0 1 2 3 and so on I like to have them in order like that so that's why I
usually do that it can cause some other strange strangeness if you don't re
reset those you can also sort a data set so here we'll use the auto miles per
gallon data set and we're going to sort it we're going to sort by the name of
the car in ascending order and we can then print out the name of the first car
which is the AMC ambassador rogue ham and now you have a sorted data set
notice these numbers are again out of order so you can tell where the original
position was you can use exactly the same reset index command that you used
up here to put these back again you you may or may not care about the original
sort order before you shuffle it usually I don't and I almost always reset that
you can also group a data set grouping is very similar to the group by command
that you have in SQL it takes the data usually use one of the categorical
values and you group it according to that categorical value and you can then
count up how many values of each category you can do a average of another
column based on based on groups of that first one so let's look at this here's
the cars database again we're going to group by cylinders and report the
average miles per gallon so the mean will run this and we see that this is
the average miles per gallon so a eight-cylinder not surprisingly gets the
worst miles per gallon six is better five is even better four and then these
three cylinder ones three cylinder cars as our five cylinder cars are a bit of
outliers they're very rare so that's probably why the miles per gallon drops
on those three they might those might have actually been performance cars I
don't know I I'd have to study the data to explain
that but nonetheless this is how you very quickly go across all of these and
can get average values you might want these as a dictionary this way you can
very quickly look up in that dictionary and find the mean of any of these for
example if you wanted to know the mean of a 6-cylinder car you could run that
and it would tell you 19 now it's important to realize that that's 6
that's not an index that's a key so that's this B key here it returns the
value of 19 point nine eight you can also get a count of how many cars are at
each of these and this is somewhat important you can see why the three
cylinder cars are such outliers there's only four of them versus two hundred
four or four cylinders four cylinder at least at the time this data set was
created was the most was the most common there were also a lot of v8s back then
too I suspect this distribution would be different in in today's day and age so
this is how you can both sort and group data this is very the grouping
especially you will use this again for feature engineering because very very
often you want to group up the data set and look at how how those values are are
varying according to this for example say you had missing mpg values you could
just stick in the median of mpg or you could do I suspect this will be pretty
similar to the means mostly so you could use the medians and say you had a
missing mpg rather than just throwing in the overall median for the entire data
set you could look at the cylinders and put in the the missing value of the
median by the cylinders that that car has that gives you a much more
intelligent way of filling in missing values than simply putting in the
overall meat median you can use the median for
another or field that's very strongly correlated and in this case cylinders
and miles-per-gallon are very strongly correlated this would be another
technique that would be very good to use with the Kegel competition project that
we will have later in the semester thank you for watching this video on grouping
sorting and other techniques in pandas we're going to look in the next video
about how to use apply and map this allows you to use lambda functions to do
much more complex modifications to a panda's data frame this content change
is often so subscribe to the channel to stay up to date on this course and other
topics and artificial intelligence