Welcome back to the oneAPI channel.
Using Pandas to load, manipulate, arrange and analyze large data?
How about running it 10,20,30 X..
And sometime even faster
with the ability to scale your existing code to larger data-sets without crashing the machine?
on a simple Intel Xeon CPU?
In minutes?
Using the same Pandas code…
In previous videos we have talked about the oneAPI components.
About the languages, tools and libraries, That can now work on multiple devices, what
we call XPUs..
And are arranged into toolkits, for example the base toolkit
But also many other toolkits
The intel AI Analytics toolkit, is one of them..
And it has many tools, resources and utilities as you can see in the dedicated video to the
AI toolkit
You can find here optimized Deep Learning frameworks
And many other python packages for data-analysis and machine learning and data preparation
Modin is part of the AI toolkit, but you can get it separately as well.
Modin supports 100% of Pandas API, and seamlessly works with the rest of Python ecosystem.
So besides the fact that you need to install the AI Analytics toolkit or Modin standalone
And call Modin instead of Pandas, your code looks exactly the same
So Modin, is Pandas on steroids if you like
If Pandas runs on 1 core of the CPU, Modin can utilize all of the cores out of the box
with no need for tunning.
Of course the more cores the better but you do not need a super computer to see the speedup,
I’ll show you in a minute
This magic of performance speedup happens because we are using the oneAPI optimized
libraries underneath and other APIs optimized exactly for speeding up these tasks on multiple
cores architecture And we are working to enable it on Intel GPUs
as well..
There are multiple ways to get Modin.
Maybe the easiest is to log in to the Intel dev-cloud.
Get access in few clicks and then, when you get the email, just click it, and open one
of the samples..
It is all installed.
You can just go ahead and use it.
The could also be found on git-hub, you can just google “Modin Git” and you’ll find
them
I chose to installed the oneAPI AI-analytics toolkit
You can see that I’m running on an Intel Xeon machine
I’m getting Panadas, and Modin, and I’ll compare their performance
I generate a synthetic array, this is about 1.5 Giga Byte array and I save it to a file
And now I’ll just read the file, first with pandas..
10 seconds..
And with Modin…
2.5 seconds
So 5x faster just to read the file..
Now let’s use applyMap to perform a lambda function on the elements of the array
24 seconds using pandas And I’ll run the same lambda function using
Modin 0.4 seconds
This is a speedup of 60 X..
And last example Just concatenate the same data-frame, the
same array 4 times, Using pandas
And using Modin, and Modin is 50 X faster
These numbers tells the full story All you have to do is to install the oneAPI
AI-analytics-toolkit and with the same pandas code you can get 10,20,30 X faster processing.
Performance boost of course varies with the size of the data and the system you use
but I think it is more then enough motivation to use the AI-Analytics toolkit and specifically
Modin.
Subscribe to the cannel to get more videos like this one
Thank you