Boost up the performance of your Python code
Quick hack to improve the performance of the code with minimal changes
Performance(Speed & efficiency) is one of the key desires of every developer and an ask from the product. This small trick will help boost the performance of your code like magic.
A lot of people complain that Python is so slow and compare Python to C in the context of performance only, but they don’t compare in the context of fast development. Python development time is so fast because you don’t have to deal with pointers, memory management, etc.
This is one of the reasons Python is the most popular in most of the product based companies where in most cases, development time is more important than performance. Python has a great community and great libraries that work to achieve all the developer’s tasks. That being said, I will be talking about a package Cython in this article that will speed up your python code. Let’s dive in…
To demonstrate these let us consider the pure Python example that generates a list of KMAX prime numbers.
Timing our maxx_primes function, we see that this function takes on an average of about 98 milliseconds to run.
Now, Let’s make one simple change: We will load Cython, a module dedicated to bridging the gap between C and Python (more on this later).
Cython: C-Extensions for Python
Do you love easy syntax like Python and high performance like C? Cython is what you want. You can use Cython to write C extensions for Python. Your code will get translated into an optimized C code and will give you high performance.
Install Cython
To enable the C compiler, you need to install the Cython Library.
pip install CythonTo start the magic, we add %%cython --annotate at the top of the code which automatically converts Python code into C behind the scenes. The function maxx_primes execution will be done with C structures.
To enable support for Cython compilation, install Cython and load the Cython extension from within the Jupyter Notebook. To load the Cython extension add %load_ext cython in the top cell.
Timing the code with this minor change, we see a time reduction to almost 35 milliseconds. That’s a pretty good discount (approx 64%) for doing such a small change!
This increment in the performance may vary for different code functions
Cython’s --annotate option will help in Code analysis.
Adding the Cython extension just converts Python processes into C structures. We can still do better, Lets add some optimization to our code.
Timing this new updated code, we see it took 2.21 ms approx. This is a shocking reduction in time to 95% approx compared to the raw code.
How did this happen ?!
we have added statically-typed C variables using the cdef keyword. Using these C variables results in faster execution of the code because the variables themselves have specified types, instead of figuring the type from the assigned value which is the feature in Python. For example, in Python one could set the same variable to an integer in one line and a string in another — a = “hello world” & a = 1. In C, however, a statically-typed variable is set only to have one data type.
This inflexibility avoids the tremendous memory space overhead that is required by Python’s dynamic typing. Since a variable can take on so many possible values, more memory needs to be allocated. On the other hand, these limitations on the statically-typed variables assure that the memory space and execution process is much more efficient and this causes such a speedup in processing time.
Static type declarations
Static type declarations can be added to the Variables & function which will give better performance. You can use all C types for declarations like int, float, double, etc.
cdef is the magic word that is used to declare C variables & functions, either local or module-level.
Variable Declaration in Cython
cdef int i = 10Function Declaration in Cython
def square(int i, char *s):
...cdef int counter (unsigned long l, float f):
...
cdef declared functions will not be visible to your Python code when you import the module. To make it visible you need to use cpdef instead.
Voila… You made it!!!
Cython advantages
Besides being able to speed up your code, Cython grants several other advantages:
- Working with external C libraries can be faster
Python packages like NumPy wrap C libraries in Python interfaces to make them easy to work with. However, going back and forth between Python and C through those wrappers can slow things down. Cython lets you talk to the underlying libraries directly, without Python in the way.
- Cython can be used to obscure sensitive Python code
Python modules are trivially easy to decompile and inspect, but compiled binaries are not. When distributing a Python application to end-users, if you want to protect the modules from casual snooping, you can do so by compiling them with Cython and package the compiled binaries.
- You can use both C and Python memory management
If you use Python objects, they’re memory-managed and garbage-collected the same as in regular Python. But if you want to create and manage your own C-level structures, and use malloc/free to work with them, you can do so.
- Cython can use Python type hinting syntax
Python has a type-hinting syntax that is used mainly by linters and code checkers, rather than the CPython interpreter. Cython has its own custom syntax for code decorations, but with recent revisions of Cython, you can use Python type-hinting syntax to provide basic type hints to Cython as well.
Conclusion
There are many other methods to make Python code more efficient with Cython, the one demonstrated above is the easiest, least technical, and perhaps most high-return method. Note that examples are for illustration purposes; they can be optimized further.
