in this video we will look into how we can optimize
memory usage for larger data set you are usign for machine learning
or general data science jobs.
In python we often use pandas or numpys
and if you dont know how to use these libraries in the right way
you might end up using a lot of memory and we all know that data sets
for data science are so huge and sometimes your laptop has less memory
and it can consume like majority of it.
so lets look into some of the
easy tehcniques.
there is this great website for optimizing memory.
usage for pandas and numpys and so on, its called pythonspeed dot com.
it is run by Itamar.
I met Itamar at a pydata conference in New York.
I have had a conversation with him and he is a very smart engineer
and he runs this awesome blog. and he wrote an article on reducing
memory usage in pandas.
We all use pandas for our data cleaning
in machine learnign or data science jobs.
There are few simple techniques
that if you use, it will drastically reduce the memory usage.
for example you are doing some election voters data analytics.
now you have this voter.csv file.
That file might have 50 columns.
But lets say you need only 2 columns.
so oftentimes the mistake that
people make is that they read the data frame and they keep the data frame
which has 50 columns around throughout the session.
If you need only
2 columns you can use this method.
We all know that if you need to do this
your data frame will become 2 columns and your data usage will go down to 71 megabytes
to 8 megabytes.
there is another arguement called used columns and that will reduce your
memory usage by a whole lot.
Now when you are talking about numbers,
pandas and numpys has different datatypes.
even for integer numbers, lets say data 8
it just stores one byte, okay, which will take into account numbers from -128 to 127
if your numbers are bigger you can even have integer64. by default the data type
that pandas use is int64.
thats your default.
now lets say you have a column called ward number.
and ward numbers are lets say from 1 to 11.
now you dont need integer 64 for that.
integer 8 is good enough.
so here when you read the csv file in dtype what you meant is ward number is 8.
your memory usage goes down from 550 000 bytes to 68 000 bytes.
you see how less
memory integer8 is using.
similarly sometimes you have categorical column.
let's say what is party affiliation.
lets say BJP, Congress whatever.
that can be represented
as a numerical categorical data.
but if you just read that csv file into dataframe
without specifying categorical as a data type, see? it is occupying 4 million data bytes.
and the same thing if I convert it into all i have to say is my data type dtype is category
and then it will define categories for it.
so from 4 million bytes, you are 70 000 bytes.
that is so small.
then there is a sparse series.
there is something called sparse series as well.
for this particular column, most of the values are NA.
So the original series length is 68000
but most of this series is NA.
So if you drop NA you get only 13000.
When you convert this using sparse series
again you covert this seeries using stype method the data usage goes down
drastically.
so I will provide a link of this article.
so Itamar has written another
article in general.
like we talked about only pandas okay? but pandas is not the only
way to reduce memory techniques.
there are code structure guidelines.
you know.
there are guidelines to use numpys in the right way.
So all of those different techniques
are mentioned in this excellent blog and I will provide a link to this in my video description
below.
check it out. and try those in your jupiter notebook and see how much saving you get
in terms of memory.
and what happens in data science and machine learning you know the data set will
be very big and you can optimise your memory and you can even improve the performance.
anyway your machine learnign job is taking so much time so if you improve eprformance
and if you want to do memory optimization,overall your program execution will be
much faster.
alright so i hope this video was helpful.
if it was please share ti with your friends
who are struggling with memory usage in pandas.
Thank you.