foreign
here for infoworld in this episode of smart python I'm going to take a look at my Pi c a
python to C compiler that's included with the my Pi type checking system
now if you've been following my videos and why aren't you you'll remember that I've taken a look
in the past at a number of solutions for compiling python to C or to Native machine language one such
solution is cython which takes type annotated python code and converts it to C with the right
annotations and with the proper use case you can achieve major speedups over vanilla python
my Pisces has much the same idea you can take type annotated python code feed it to the my Pi
C compiler and out comes the python module written in C the type annotations can be used to speed
things up quite a bit and best of all you don't need to use any esoteric typing or annotation
syntax as cython requires my pisc works with the type annotations already present in Python
now this does come with some limitations as we will soon see but you can sometimes get double or
more performance speed UPS on code without having to do much more than just compile the module
so let's start with a simple example this one's harvested straight from my Pisces documentation
this snippet computes a Fibonacci sequence for some input it's a math heavy operation that
python isn't well suited to but it doesn't need much type decoration we just need to decorate
the function input and output so if we run this code we'll see the whole thing takes about half
a second on this particular machine using a pure python version now to compile us to see all we
need to do is use the my Pi C command line tool and provide the name of the module to compile
as long as we have a c compiler installed in this system the rest is automatic and once the
compilation process finishes we can run the newly generated C module this way by importing it from
a python stub program which I provided the command line and as you can see this new my Pi C compiled
version is something like 20 times faster than our vanilla python code and we got that speed up for
nothing all we had to do was compile our code and make sure that it had the proper type decorations
to emphasize just how much of a difference those few type annotations make I'm going
to go back to the code and I'm going to remove them from the code and recompile it
because this function is our hot loop as they say this is the part of the program
that has the most activity having the type annotations there is making the whole difference
so when we rerun this recompiled code you'll see that we do still have a speed up but it's
nowhere nearly as dramatic as when we had the type annotations this is because my Pi
C cannot make type specific optimizations for this function without those type datas
and so everything is far slower with my Pi c as with cython and other such Solutions you're
going to see the biggest speed ups when you have a few common attributes with your code
first you have a tight loop operation of some kind as we do in this code most performance issues in
Python code are in Loops like this it's because in this particular code here it's because we have
this recursive calling of the code itself and that is another thing that python tends to stumble with
number two everything in this Loop everything in this hot code as again as we say is type
annotated there's no guessing about the types of things and when the compiler has to guess
the types of things it will fall back on using generic python object behaviors which are slow
three the types of things in the loop are native python objects or built-ins and they're being
used in ways that are readily optimized for instance integer math is easy to optimize the
my Pisces documentation which I've Linked In the article description goes into details about what
operations are optimized for what built-ins number four we're not waiting on anything else that can't
be optimized because it's not actually part of our code for instance if we had a network call
or if we were reading from disk we can't wait on those things any faster in C than we can in Python
so trying to optimize things like that aren't going to help because they're out of the reach
of what my Pi C can make faster I'm going to illustrate this last Point using a benchmark
program that I've used in the past it's called the Google engram Benchmark it involves processing a
very large tab separated data file this thing is about 200 megabytes in size
and it generates statistics from it most of the work in The Benchmark is done in this
Loop here where we iterate through each line in the file and we perform some processing with it
as you can see I have some type decorations in here which won't be of use in a minute
now if we run this code AS is the whole thing will take about five seconds to complete on my
system and I've got good reason to believe that much of the time involved is just i o
operations just waiting for disk reads now we can compile this code with my Pi C since again
it has Type annotations already included for all of the elements that are in the hot Loop
but if I run this compiled version you'll see that we're only able to shave about 20 to 30 percent
off of the runtime and again that's because a big part of what we're doing involves trawling through
a text file so much of why this program is slow is because of that I O bottleneck which we really
can't do very much about in these circumstances dad 20 to 30 speed up for doing nothing more
than feeding the program to my piscesn't bad and you're getting that completely free as they say
another thing to keep in mind is how my Pisces differs from cython in one key way that the
two are different is that cython lets you use C types natively by way of your type decorations
for instance let's say I took my Fibonacci function and I rewrote it to work in cython
which I've done in this example I have to use python-specific annotations to get the most of
how cython works and that's not just for type definitions as you can see here I'm specifying
that these are going to be C integers but also for things like the function definition I want
to specify that this function needs to be a native C function and not a python function
but that said when I compile and run this version of the code you're going to see that
it runs nearly twice as fast as our my Pi C version with all of its type decorations
and that's because we can specify the use of machine native types if your problem in your
program calls for those kinds of machine native types then cython is going to be the better choice
another thing that my Pisces does not do that cython does do by Pisces does not work as
an interface between Python and C libraries and that's another job that's left a cython
so to sum up my pisc compiles python to see by using Python's standard type annotations doing
so gives you a performance boost without having to do much more than add type annotations and
use the my Pi C compiler however the speed up you get is not guaranteed some things especially if
they're i o bound May speed up by only a little but some things may speed up by as much as an
order of magnitude or more that said if your problem calls for the kinds of speed that only
native C types can unleash or you're wrapping C libraries then reach for cython instead
that's it for this episode if you liked it leave a comment below don't forget to
subscribe to the tech talk Channel on YouTube and for more smart Python and
more Dev with zodar be sure to follow me on Facebook YouTube and infoworld.com foreign