Hey y'all, in this video we're going to be talking
about why your pandas data frame might be using a
bit more memory usage than you want it to, why
it's doing this, and a couple of quick solutions to fix it.
All right, so I want to start with
a baseline so we can see just how much memory
you might be wasting in your data frame. I have
this super simple code here where I'm just reading in
a CSV and then displaying the information using this memory
usage equals deep parameter which basically just says, "give me
the actual memory usage of the data frame instead of an estimate"
So if we take a look at the output,
we can see that this data frame has 14 columns.
We can see the dtypes, or data types, of
each column and if we go to the end we
can see that this data frame is using just shy
of 500 megabytes of memory. So that's our starting point
and I want to see just how low we can
get that with a couple of tips and tricks. So
my first tip is incredibly simple, but you'd be surprised
how many people still are not using it. Let's say
that I really only care about analyzing three of these
data columns. In that instance, I just wasted a bunch
of memory
because I read all 14 data columns into my data
frame. So instead what we would want to do is
specify exactly which columns we care about when we read
in the CSV. We can do that using this usecols
parameter in the read CSV method. In this case
I am specifying item type, order priority, and units sold
as the columns I'm going to read in, and I'm
not going to read any other columns in because I
don't want any unnecessary memory usage for data that I
don't care about. So if we take a look at
this output, we can see that we shrunk the memory
usage from 500 megabytes to
125 megabytes. So that's pretty intuitive, right? This isn't really
a complicated concept that I'm reading in about 1/5 of
the columns, and as a result I have about 1/5
of the memory usage, but it's so simple and so
quick that I had to mention it because it's such
an easy way to just cut down on memory right
off the bat, so you all should be using this
any chance you can get.
My next tip is a little bit more in depth
and requires some knowledge of your data set, so let's
take a little bit deeper look at this dtype information.
How does pandas even figure out what the data type
of your column is? Basically, when you call the read
CSV method, pandas will read each column in its entirety
to figure out what sort of data is in there,
and this can be pretty costly because essentially pandas has
to consume a dataset's entire memory just to figure
out what that data type could be. Now the way
to alleviate this is to pass in the dtype
parameter in the read CSV method and this will specify
exactly what those data types are, so that way pandas
won't waste a bunch of memory trying to figure it
out. So to see this in action, let's take a
look at the data viewer for the df_usecols. If
we take a look at this first column, the item
type, this really is just a bunch of strings. We
don't really know all the different types they could be,
so a string object is really the most appropriate data type.
The second column, the order priority, might also look
like a bunch of variable strings to you right off
the bat, but I know that my order priority only
ever takes the form of H, M, or L for high,
medium or low priority, so there's really a finite set
of values that this column can have. As such, that
makes it the perfect candidate for the category data type
and the category data type takes up a bunch less
memory than the string data type, so we want to
use that at any time possible. Now my last column
is units sold. Now we could just keep this as
an int 64, but I know that my units will
never top 10,000, so really int 64 gives me a
larger number space than I really need to work with.
An int 16 data type will really do the job
to making sure that we have enough memory to accommodate
for the type of data we're expecting here. So now
if I go into the notebook we can see what
that dtype parameter looks like in action. I have the
same call where I'm still using the usecols, but
now I have the addition of this dtype parameter, where
I'm saying the item type as a string type, order
priority as category, and units sold as INT 16. So
now if we run this, let's see how much we
have compressed from 125 megabytes.
So if we take a look at this, we actually
cut it down, basically in half, to 65 megabytes, which
is awesome because by just using two quick column tricks,
we went from 500 megabytes to 125 megabytes to 65
megabytes. And just in that last step from going to
something as simple as a string to a category data
type, and int 64 to int 16, we cut our
usage in half. So use these tips to stop wasting
so much memory in your data frames. There you have
it, it's really not too difficult to start
reducing your pandas data frame memory usage. We looked at
how something as simple as specifying exactly which columns you
want to import and then their corresponding data types can
significantly reduce the amount of memory used, and these were
just a couple of quick steps that you can start
to take. So tell us what tricks do you use
to help cut back on this memory usage? Let us
know in the comments. But wait, if you like this
video, then do us a favor and actually hit that
like button. While you're at it, might as well subscribe
too. We post a lot of other great content about
Python, Jupyter notebooks,
and all things vscode. Happy coding!