hi there Serdar Yegulalp for InfoWorld at IDG. Today I'm going to show
how Python programs can be made used multiple CPUs by way of the multi
processing module by default Python programs run in a single physical thread
that's because the default Python runtime is single threaded for the sake
of simplicity if you have a Python task that's cpu dependent and you want to use
Hardware threads to speed it up you can use the multi processing module to do
this multi processing works by creating a pool of Python interpreters typically
one interpreter per CPU in your system but it's tunable and then allowing you
to feed jobs to that pool which are then executed in parallel because launching
the process pool itself takes time you typically want to do that once at the
start of a program and then just feed it jobs as needed what I'm going to do here
is take a simple benchmark application and show you how multi processing can be
used to speed it up the benchmark reads a large CSV file taken from Google's
Ngram database about 187 megabytes and tallies some statistics from it on my
system an i7 3770k processor it takes around 11 to 12 seconds to run the
benchmark multi-processing usually works one of two ways the first way you take
many little jobs that are CPU dependent and submit them to the process pool the
second you take one big job and find a way to slice it up efficiently into many
little jobs the benchmark we have here fits in the second category it's a big
job that can be sliced up we're reading a file line-by-line and gathering
statistics about each line which we then aggregate at the end multi-processing
gives us tools to work with both kinds of jobs and so in this case we're going
to be using the imap unordered function which takes an iterable like a file
which is read line my line and parcels it out piece by piece in round robin
fashion to processes in the pool each process runs a defined function on it in
this case our simple statistics gathering function now let's run the
multi processing version of our benchmark as you can see it runs much
faster it doesn't provide an absolutely linear speed-up with
number of cores in the system but that's fine and the improvement is better than
none and it's actually harder to get an absolutely linear speed-up with more
cores in many cases anyway one important thing about multiprocessing to keep in
mind is inter process overhead I mentioned earlier how there's the
overhead of launching the process pool but if you keep that confined a program
startup it's less of an issue but the other kind of overhead you need to
remember is the overhead involved in communication between the main process
and the process pool each time the main process sends a job to the pool and each
time a job returns a result there's overhead involved so to reduce the
effects of that we want to keep the total number of round trips between main
process and pool as low as possible multi-processing has a few built-in ways
to deal with that for instance you'll notice this parameter here named chunk
size this is how many slices we send to each pool process at a time for a large
job like this you want to set chunk size quite high in this case the main process
is going to read around 16,000 lines or so at once and sends that in one big
chunk to the first process then it reads the next 16,000 lines
sends that to the next process and so on this is faster than going a line at a
time because you can bundle that many more lines in a single transaction on
the other hand if you set chunk size too high you'll end up losing performance
since the cost of sending larger chunks starts to become its own burden doing
things in a multi-process way brings other kinds of risks you need to be
conscious of for instance you always need to be aware of how a multi
processing job contends for resources in this case the main resource is the file
we're reading through and python automatically handles contention for it
since all of the read operations are in a single thread but if you have multiple
jobs in a pool all fighting to read the same file on their own or contending for
some external resource you lose most of the benefits of dividing up a job across
physical processors if you liked this video please leave a comment below don't
forget to subscribe to our IDG Tech Talk channel on YouTube and for more Python
tips follow us on Facebook YouTube and info WorldCom