hi there serdar yagalop here for infoworld at idg in this episode of smart python we're going to
look a little more deeply into the behaviors of python's memory management mechanisms
in particular how python handles memory that can't be automatically reclaimed with reference counting
in the previous video i talked about how python uses a reference counting mechanism
to track the lifetimes of objects when an object no longer has any references it's removed from
memory and the memory it used is reclaimed now there's a problem with this approach and it's
something that might only become obvious with more advanced use what happens if we have two objects
that hold references to each other even if we delete both of them they'll still linger in memory
because they each still have a reference remaining one inside the other here's an example of this
we have two lists each with the other as their last element if we delete either the names x
or y in this case we'll delete y the object they point to will still linger inside the
other object even if we delete both of these names the objects behind them remain unclaimed
you'll notice that when we print the last element of list x the y list is still inside it and the
x list is also still inside the y list as well that's what those three dots indicate
this is what's known as a cyclic reference two objects that refer to each other and in python
it's actually a common thing for objects to do this the good news is python's memory management
system also takes this into account it has a mechanism that detects cyclic references
and removes them this is what's actually referred to in python as the garbage collector
and many people myself included have colloquially referred to python's memory management system as
a whole as its garbage collection system but to be totally pedantic about it the garbage
collector is this part here it's the mechanism that detects when objects refer to each other
but are otherwise inaccessible by the rest of the program and removes them now we can't control
reference counting that is baked way too deeply into the way python works to be something we can
switch on and off but we can get insight into the garbage collection mechanisms and we can control
it in many useful ways to work with the garbage collector we import the gc module into a program
one of the things that we can do for instance is get information about how many objects are
being tracked by the garbage collector if i run gc.getcount right after starting a script
it indicates there are already a few hundred uncollected objects floating around just from
starting the script up if we force a garbage collection pass right after that by running gc
collect that number drops to zero this means a slew of objects were initialized at startup
tracked by the collector were found not to have any existing references and were cleaned up after
we forced the system to do so now you're probably wondering what are those other two numbers that
gc.getcount spit out well the answer to that is tied up with the next part of this discussion
to keep python programs running reasonably fast the garbage collector doesn't and can't run all
the time for one the vast majority of the time it doesn't have any real work to do anyway for
another each time the garbage collection process takes place nothing else in the program can run
until the collection process finishes this is actually one of the common objections
raised to languages that use garbage collected memory management like python the pauses that
are imposed by the garbage collector they won't show up in a script like this which
is way too short for those effects to manifest but they do become noticeable in bigger programs
so to that end python's garbage collector only runs every so often and it doesn't run on a
timer instead it's triggered when the number of objects tracked in the system reaches one of a
certain set of thresholds the garbage collector classifies python objects into three categories
called generations each of these generations has an object threshold count we can see this count
by using the gc.get threshold command the initial thresholds are 700 for the first generation
10 for the second generation and 10 for the third whenever a python object is created it starts its
life in the first generation when the number of objects in any generation exceeds its threshold
the garbage collector runs for any objects in a generation that aren't collected because they're
still live they still have references they get bumped into the next highest generation
so those are the three numbers that we saw before those are the object accounts for each of the
three generations of objects now because python is constantly creating and discarding objects
the lowest generation has a high default collection trigger threshold
of 700 objects generations two and three which are for long lived and for really raw lived objects
have collection triggered thresholds of ten so if an object has already survived in the
system for a fair amount of time there's a good chance it's not going away any time soon
the idea behind using these generations is to perform fewer collections as the number
of objects grows because each collection pass will become more expensive and will take longer let's
rework our earlier script those two cyclically dependent lists to show the object counts
after the first cleanup we have zero stray objects after creating the x and y lists we have five
delete x and the number goes up to six clean up that number drops back to zero
delete y the number goes up to four clean up again the number once again drops back to zero
now to expand on something i mentioned earlier if we did no manual garbage collections ourselves in
this script the thresholds for running collections would never be reached during its lifetime it
doesn't generate enough objects to trigger those thresholds but it also wouldn't matter much with
a script is small it doesn't generate enough objects for it to really become an issue and once
the script terminates all the memory used by the script gets surrendered back to the system anyway
so to that end uncollected garbage amounts to very little if your script runs and terminates quickly
but if it runs a long time and generates a lot of objects that are long lived
it may trigger many garbage collection runs and again every single time a garbage collection run
is triggered everything else in the program has to stop for programs that have lots of objects
this garbage collection pause can become really noticeable so what does all this mean for you
well first off it means you shouldn't worry too much about how memory is managed automatically
but you should think about how python's garbage collection behaviors can slow an application down
especially if it runs long has a lot of moving parts and generates many objects now in the next
video i'm going to go into more detail about how to manage and fine-tune the garbage collector's
behavior especially in long-running applications where its activity can set performance
if you like this video please leave a comment below don't forget to
subscribe to the idg tech talk channel on youtube and for more dev with serdar
and more smartpython be sure to follow me on facebook youtube and infoworld.com you