Welcome back to the oneAPI channel.

Using Pandas to load, manipulate, arrange and analyze large data?

How about running it 10,20,30 X..

And sometime even faster

with the ability to scale your existing code to larger data-sets without crashing the machine?

on a simple Intel Xeon CPU?

In minutes?

Using the same Pandas code…

In previous videos we have talked about the oneAPI components.

About the languages, tools and libraries, That can now work on multiple devices, what

we call XPUs..

And are arranged into toolkits, for example the base toolkit

But also many other toolkits

The intel AI Analytics toolkit, is one of them..

And it has many tools, resources and utilities as you can see in the dedicated video to the

AI toolkit

You can find here optimized Deep Learning frameworks

And many other python packages for data-analysis and machine learning and data preparation

Modin is part of the AI toolkit, but you can get it separately as well.

Modin supports 100% of Pandas API, and seamlessly works with the rest of Python ecosystem.

So besides the fact that you need to install the AI Analytics toolkit or Modin standalone

And call Modin instead of Pandas, your code looks exactly the same

So Modin, is Pandas on steroids if you like

If Pandas runs on 1 core of the CPU, Modin can utilize all of the cores out of the box

with no need for tunning.

Of course the more cores the better but you do not need a super computer to see the speedup,

I’ll show you in a minute

This magic of performance speedup happens because we are using the oneAPI optimized

libraries underneath and other APIs optimized exactly for speeding up these tasks on multiple

cores architecture And we are working to enable it on Intel GPUs

as well..

There are multiple ways to get Modin.

Maybe the easiest is to log in to the Intel dev-cloud.

Get access in few clicks and then, when you get the email, just click it, and open one

of the samples..

It is all installed.

You can just go ahead and use it.

The could also be found on git-hub, you can just google “Modin Git” and you’ll find

them

I chose to installed the oneAPI AI-analytics toolkit

You can see that I’m running on an Intel Xeon machine

I’m getting Panadas, and Modin, and I’ll compare their performance

I generate a synthetic array, this is about 1.5 Giga Byte array and I save it to a file

And now I’ll just read the file, first with pandas..

10 seconds..

And with Modin…

2.5 seconds

So 5x faster just to read the file..

Now let’s use applyMap to perform a lambda function on the elements of the array

24 seconds using pandas And I’ll run the same lambda function using

Modin 0.4 seconds

This is a speedup of 60 X..

And last example Just concatenate the same data-frame, the

same array 4 times, Using pandas

And using Modin, and Modin is 50 X faster

These numbers tells the full story All you have to do is to install the oneAPI

AI-analytics-toolkit and with the same pandas code you can get 10,20,30 X faster processing.

Performance boost of course varies with the size of the data and the system you use

but I think it is more then enough motivation to use the AI-Analytics toolkit and specifically

Modin.

Subscribe to the cannel to get more videos like this one

Thank you