(upbeat music)
- We have the most active developers
of any project after Hadoop MapReduce.
And at the Spark Summit today,
it's really great to have you all here.
And as Andy said, we are sold out with over 1000 people,
which is the largest Spark event yet.
- We are very happy to introduce our first product,
Databricks Cloud.
Databricks Cloud is based on Apache Spark
and includes a Databricks Platform
and a Databricks Workspace.
- GUI pioneer, Alan Kay, said,
"Make simple things simple, and complex things possible."
- The goal of the project is pretty simple.
We want to empower data scientists and engineers
and give them good tools to get the things that they need
to get done, done quickly and done without a lot of work.
- Welcome, everybody, to the first Sparks Summit
that we've ever had in Europe.
And at such a historic beautiful venue.
It's appropriate because this is a historic moment
for our community.
A community which all of you
have played an important role since the beginning.
Why we came out to the Spark Summit in the first place.
Spark is amazing.
I totally agree with that.
Meet some fellow developers,
and there's a lot more than developers here today too.
Community, technology, and business.
It's awesome to be here in New York
with a very large audience.
Summit is actually sold out.
- Thank you very much, Matay.
I am super excited to be here today
to demonstrate exactly how you can use Apache Spark 2.0
inside of my favorite platform,
Databricks Community Edition,
which as Matay just announced, is now available to everyone
in the world for free.
- We're introducing MLFlow,
which is an open source platform
for the complete machine learning life cycle.
- I'm super excited to see almost 5000 of you here today.
And you're here to exchange knowledge to share ideas
to learn about what's going on in Spark
and also its ecosystem.
And more importantly, you can take that knowledge home
and apply it to our next data pipeline applications.
We're announcing a new open source project here today,
Koalas. (audience cheering)
As the title suggests, it's a very simple thing,
very simple to understand.
It provides a Pandas API on Spark.
- That is why I'm very excited to introduce Koalas
to you today.
A unified API to run your Pandas code at scale
using Apache Spark.
As you can see here, I've already written some Pandas code
to read in a CSV file, select some columns,
and display the results.
But we can see this took almost two minutes when I ran it.
What I'm gonna do
is I'm going to copy and paste this code down below,
change the import from Databricks, import Koalas.
And now all I have to do is replace my Panda's code
with Koalas.
And so I get to keep my same code that I wrote
with single node Pandas, but now it's gonna be running
against an Apache Spark data frame,
and we can get our results much more quickly.
I can see I got them in eight seconds now.
- We are super excited to announce today
that we're actually open sourcing Databricks Delta.
(audience cheering and applauding)
With Spark 2.0, we've now added broad support for SQL.
We've also integrated it with Python,
and in some sense, it's really unifying now
SQL analytics or data warehousing and data science.
And that's really exciting
because that's where we think the future is going.
In fact, our customers were unifying these two things
for a while and actually a pattern has emerged
in which they call this the Lakehouse paradigm.
(upbeat music)