- And good morning, everybody.
Many of you are here today for the Apache Spark Project.
And some might even know this is a very special year
for Apache Spark.
2020 marks the 10th year anniversary,
since the projects first release,
and to celebrate that the community put together
a Spark 3.0 release,
the latest major release of the project.
Now, today we see more and more Sparks use as a computation
engine to power the Lakehouse architecture.
This is combined the very different kinds of workloads
from analytics, BI, streaming,
data science and machine learning.
But this was not always the case in the beginning.
I remember in 2013, when Spark was a three-year-old project,
we started Databricks as a company.
At the time, virtually everybody,
all of the spark users are using Spark Scala API,
and we just added the Python API and a sequel API
to the project because Spark itself was retaining Scala.
Really only, Scala was the first class API
and everybody's using it.
Now over the years,
we've done a lot more work to make Spark,
more accessible to data teams,
including two data scientists with the Python API
and two data analysts with SQL API.
So last week we took a look at the number of commands,
run on data breaks across all of our customers.
And we found something pretty astonishing,
especially when compared with the 2013 results
while Scala do a first-class API.
And it's used by most data engineers
to write and build their data,
compute intensive data engineering jobs.
You'll see the significant rise of Python and SQL
each claiming almost half of all of the commands run
on Databricks.
This is really the evidence for the Lakehouse division
because data scientists use Python,
analysts use SQL
while the data engineers are using the JVM based languages.
Every day,
spark users process exabytes of data using SQL
on Databricks alone.
And this is not just on the Databricks platform.
For example, earlier this year,
Ali-Baba won the official hundred terabyte TPC-DS benchmark
using a Spark based system.
And this is sort of the standard benchmark
for our data warehousing workloads.
But this users don't just use SQL for batch analytics.
They're also using them for streaming analytics.
Every day,
We see over 10 trillion records process on Databricks
with structured streaming,
which is a streaming project that takes in the SQL query
and automatically incrementalize it and stream it buy it.
In June Spark and AI summit.
My taste celebrate it sparks 10-year anniversary
and talked about various improvements
were made to a spark sequel and the 3.0 release.
He talked about the various performance improvements
like dynamic partition pruning, adaptive crew execution
that brought significant speed ups anywhere from two to 20 X
to the very specific SQL workloads.
But he also talked about the various user-ability features
were added to Spark SQL like NC SQL compliance.
So business analysts get more easily pour their SQL
workloads over to Spark.
In this talk,
I would like to focus on a very different aspect,
which is the Python part of Spark.
I would like to talk about Project Zen.
We launched this project earlier this year
to make Spark more Pythonic.
And it includes a collection of the features
from better API designed to better error reporting
to better documentation.
Metate briefly talked about Project Zen
and his summer Spark summit keynote
only to give you an update by showing you the progress,
we have made some of this already ship with Spark 3.0
and some of them are ongoing work.
So let's take a preview.
The first thing I want to show you,
our user defined functions or UDFs.
In Spark, it's very common to define the UDF
and lovely Spark to distribute the UDF,
the process, large amount of data across the cluster,
because Spark needs to know the type information
for all the query planner UDFs at the completion time,
you just must annotate this UDFs with their return types.
And before Project Zen,
Spark has a specific way of doing annotation.
And you can see on the screenshot here.
Now at Project Zen,
we're adopting the Python Three style type hints
that are native to Python to allocate this UDF since that.
So the code is now much more clear,
but not only that,
the users no longer need to learn a special way
of annotating this UDFs.
They can just take whatever they've learned
from the Python type hands project over.
Next is our messages.
And to be honest,
it's quite embarrassing for me to show this.
PySpark,
if you talk to a lot of the practitioners notorious
for his lends, the error messages,
when something goes wrong.
For example, if there's an error in the UDF,
it usually shows you a giant stack trace
that combines both some Python stack trace
and some Scala stack trace.
Here's an example of an error message on the screen here.
You probably can't even tell what's had gone wrong.
That's because there's actually no real error message
showing here.
It's always showing just very drawn a stack trace,
and there's only the first page.
Now let's go to the second page.
You can see at the top of the second page
there's a division by zero error,
and that is the actual error that happened on via cluster.
And then it's followed by, again,
a very giant JVM stack trace,
but that's still not yet.
Page three,
now you have a different kind of JVM stack trays
because this is the JVM stack traits for the driver.
Whereas the previous JVM stack traits is for the executor.
Yeah, all internal dispart.
But that's not yet.
Page four this longer stack traces.
Page five and finally the last page.
So six pages of very long error messages.
This six pages of very long error messages
have as we talked about earlier,
have a very long say the Python stack traces internal
to Spark logically and multiple JVM stack traces
also internal to Spark.
So you might want to ask,
why are we showing all of this?
Why don't we just show the division by zero,
along with the Python stack traces,
that's native to the Python project.
So that's exactly what we're doing with Project Zen.
When now removing the JVM stack traces
our internal Spark itself that users have no control over
and only dissipating a Python stack traces
along with the error at the very end.
So it's the very last line,
which is what you would see when you in the console,
in the notebook.
And now it makes the error really easy to spot.
We're managed to shrink six pages of error messages
into just one in order to make you more productive.
And this you're just a work in progress.
In the near future, we hope we can cut this
even the one page error message down even further
to show you what's the most critical message
you want to see.
Now, let's talk about Auto complete.
I personally rely on Auto compete in IDEs and no books
every day when thereby program.
And it's because I simply can't memorize
all the different APIs in Spark
and in all the Python ecosystem.
And I believe that's the case for almost everybody.
Before Project Zen,
Auto complete for Python leave a lot to be desired,
it lacks context,
and can show you the most important things.
So in the example, I've shown here,
it was the screenshot you can see.
So in the middle of doing a CSV parsing,
and most likely at this particular moment,
while I want to see
is the list of different pricing options for CSV files,
and you can pretty easily see the suggestions
are not that useful.
They're mostly generic Python sort of variable
was in global scope area, but what sets exists.
Was Project Zen
we're shipping a much better Auto complete
that's actually contacts aware and relevant,
and it works in notebooks as well as IDEs.
When we take a look at the new screenshot here,
you can see now there's very different kinds of CSV
parsing options that directly being suggested
versus the old one that's mostly Python global variables.
As part of the Project Zen,
we have talked about earlier,
how we can leverage the Python type pins to allocate UDFs,
but we have taken one step further and define type pins
for the entire PySpark project itself.
As a result,
now you can run static type checking
across all the coding including Sparks
and your own user defined function.
This video on the screen shows you how PyCharm
the so one of the most popular IDEs for a Python
can statically analyze the code
and tell the developer when the code has errors
without even running it,
which was not possible before without the type pins.
This, even though it looks like a pretty small convenience
feature, it really,
this is one of those features that made it possible
to scale Python based Spark development,
to much larger teams,
because the type tracking the static error checking
becomes an implicit explicit contract
between the different development teams
working on the same side of the projects.
The last thing I want to show you as part of the Project Zen
are Docs.
The existing Python Docs for PySpark
can be difficult to navigate.
For example,
the overall index page I'm showing here is
it's very long and includes a lot of different packages,
a lot of different classes.
And when you click and go into a specific package,
we show you all the classes,
the contents of all of the methods in one page,
which again is very difficult to navigate.
Sometimes you have sort of a very long documentation
just for one method and you can't see the other ones.
Sometimes it's just that it's in a way
information overloading.
Project Zen adopt the Num PyStyle Docs
to make Docs much easier to consume.
This is how the new Doc would look like is not finalized yet
because work in progress.
But you can see in the screenshot here,
we're just listing all of the most important packages
with the classes.
And we're showing a summary of all the classes.
The one sentence summary is very concise
and easy to understand what they are.
And once you click into one of the classes,
we'll show you all of the methods.
Again, was one sentence summary for each of the method.
And once you go into a specific method,
we have the entire page just for that method itself.
And we show you all the details about that method,
including a more flushed out detailed description
with the parameters, that types of parameters, examples.
This really makes it much easier to navigate
with this clean interface and can make you more productive.
Because now it's easier to look up information.
It's also easier to understand
what the Docs are talking about.
Of course, we can't talk about making Spark more Pythonic
at data and AI summit foaming or Spark summit
without talking about Koalas.
We launched the Koalas project about a year ago at Spark
and AI summit in the summer in San Francisco.
Koalas implements a Pandas APIs over Spark
to make it easier for a data scientist support
the existing code over tool Spark to run
on larger amount of data,
or to just write new code without having to learn
a complete new set of API like PySpark
by just using a Pandas API.
It's pretty impressive that in merely one year,
we're now seeing more than 850,000 downloads per month
of Koalas out of PyPi.
And on top of that on Databricks,
our customer has to importing over 1 million imports
of Koala instance each month.
So as part of the conference here today,
I'm excited to announce Koalas as 1.4 release.
This release includes a lot of the performance improvements
and a much better API coverage,
along with a lot of other usability features
to make data scientists more productive.
Now, Koala has cover off the 85% of all of the Pandas APIs.
So very likely you can
and it's actually the 85% of the most important API.
So it's very likely you can just port your Pandas code over
was minimum change needed.
On Databricks Koala pre-installed,
so you can just start using it without doing anything.
Outside of Databricks,
you can very simply install it with a PIP or conduct,
which is PIP install Koalas,
because it's been published
to all of the Python repositories.
Rather showing you more screenshots.
Caryl will give you a demo of Python Spark release Koalas.
So what better way to pass it over to you?
That's throwing you across Caryl.
- Thanks, Reynold.
So a lot of you have probably heard by now
about our source project called Koalas,
which basically puts a pandas like interface
on top of Spark,
which means you could take the single node code that you
built to process maybe hundreds or thousands of rows,
and now easily scale it to millions, billions, trillions,
or even more rows using the power of Spark.
But you might not have heard about how real life Koalas
themselves are getting more data-driven
and they want to use their new skills
to find the best home for their Koala babies
and what makes a good home, good food of course.
So let's see if we can look at some Australian weather data
to figure out where and Australia the growing conditions
are best for their favorite food Eucalyptus.
All right.
So the first thing I'm gonna do is import the libraries
I know I'm going to need,
and then I'm gonna wanna take a look
at the weather data that I know we have.
And I know it's an a Delta table somewhere,
and because I love using pandas,
I wanna use Koalas to read it in,
but I can't quite remember the function for it.
So many use Auto complete here
and there it is read underscore Delta,
and we can see that that function expects a path
of type string.
So what we're seeing here is our new support for type pins,
which makes it a lot easier to, as you're writing your code,
just simply tab over,
see what the names of the functions that you need
and see the parameters and the types for those parameters
that the functions expects.
And just go ahead and in this case,
enter the string where our Australian weather data lives.
Right.
Now, I know this is millions and millions of rows,
so I'm just gonna pull the first 10
and while that's loading,
note that type head works for all of the built in
Spark functions that you see here,
any UDFs that you defined,
you'll be able to access those as well as
list the parameters that are expected
for any of the functions, et cetera.
Great.
And now that our data's populated,
let's see what we're working with.
So here we have an area code in the leftmost column,
which looks like a pre-fixed
indicating the Australian state.
So VIC for Victoria,
WA for Western Australia and so on,
as well as a code that seems to indicate, well,
what they're calling area code
might correlate with a weather station
or something like that.
So for each of these area codes,
we get hourly data streaming in
for a bunch of different weather parameters.
This is perfect.
It's exactly what I was hoping for
because my Koala researcher friend has already built a model
to estimate health score for Eucalyptus,
given a few pre weather parameters per region.
So I've just copied her function and pasted it here.
And we can see that that function expects
a series of temperature and a series of precipitation.
Again, leveraging the new type pin support for UDFs
in this case.
And in return, it will produce a score in the, of a float,
letting us know how likely Eucalyptus is
to thrive in that area code.
Great, so I'll go ahead and register that.
And if you haven't seen this before,
what we're looking at here is called a Pandas UDF.
It's also been called a vectorized UDF,
and it's basically just a way for you to take your art
existing Python functions,
drop it into Spark and define it in such a way that Spark
knows how to efficiently and easily distribute that under
the hood in a vectorized format.
So now that we've done that,
we'll go ahead and call it on our entire weather dataset.
We're gonna group it by area code, pass it,
the temperature and precipitation columns it expects,
and now we've sorted it in descending order by score.
So we can identify the top five area codes that we might
wanna explore for our Koala babies.
So it looks like our top contenders are Queensland QLD 038,
New South Wales, 158,
QLD 5221 and NSW 103.
So these all look pretty good,
but one way to kind of narrow it down
is to figure out which of these area codes
have the most stable Eucalyptus health scores over time.
So I prepared some time series data
that we can go ahead and use our built-in visualization,
plot dot scatter, and Koalas to plot.
So we can look at all of our Eucalyptus scores by month,
which is great, that was super easy,
but it's not really clear which Eucalyptus scores correlate
with which area codes.
And I don't think there's an easy way
to get it all in one graph using that plot lib.
So I could either do this area could by area code
or I can take advantage of the new backend Plotly support
for visualization and Koalas,
we can just pass a Plotly backend parameter
into our scatter plot,
which is awesome because all of the data scientists
and Python users, I know love Plotly
for how interactive it is.
And I know that Plotly also supports passing a column
to color code your dots on a scatter plot.
So awesome.
This works like a charm on the right-hand side,
you can see our legend here,
blue for QLD 38, red for NSW 158 and so on.
And it looks like red gave us our highest score.
NSW 158 in December of 93.3,
but it also looks pretty volatile
'cause it dipped all the way down
to 87.9 in March.
If we look at blue on the other hand,
it looks like it tends to stay pretty stable
above the 90th percentile mark.
So I think that's the best choice for our Koala babies.
And we're going to go ahead and fly them over to Queensland
PTs 038.
Here we come.
So hopefully this was a fun way to highlight some of the
exciting new features that should boost productivity
for Python users and data scientists on Spark.
Again, including the type pin support,
which allows you to use Auto-complete
and look in line to get an idea of
what functions are available to you
and what types of paths to those functions.
So you don't have to
go to multiple screens to figure it out
again, just allowing you to stay in and keep coding
as well as our support for Plotly,
which just gives you a ton more flexibility
for interacting with your data.
We hope you enjoy the demo.
Thanks for watching.
- Thank you, Caryl, for the great demo.
Just to recap this talk,
we talk about how Project Zen is making Spark more Pythonic,
and to make data scientists more productive
by giving data scientists, better error messages,
better Docs, better Auto-complete,
and a better user defined function with type pins.
I don't have time to dive into all the details
of Project Zen and for that,
I will encourage you to check out huge ins target
this conference.
We're seeing Spark use more and more as the compute engine
for Lakehouse.
Data engineers are using Spark Scala API
to build data pipelines.
Data scientists are using Sparks Python API
to do a statistical modeling and machine learning
and data analysts are using Spark SQL API for analytics.
In the past few years,
the community have invested tremendous amount of energy
and effort in improving all of these different use cases
and the APIs with different personas
and the community will continue to do so.
I can't wait to see what you would be able to do
with this project.