- So you've been hearing about the Lakehouse pattern
and I'd like to put this talk into perspective.
As you know, the foundation of the Lakehouse is Delta Lake,
which makes all of your data no matter what type
available at scale and high quality.
Now, in order to support data teams,
Databricks provide several capabilities
that all operate on the same consistent data.
You heard about the SQL service and (mumbles) keynote,
which provides high performance sucralose analytics
for BI and reporting.
To complete the picture, we provide
a collaborative environment
for data science and machine learning.
You heard Mate talk about
some of the latest machine learning features earlier today.
I am going to focus on the next generation
of our data science workspace,
which is the environment Data Teams use
to access our data science
and machine learning capabilities.
Now, why is this environment so important?
It's because of you.
It's because of data scientists, ML engineers,
data analysts, data engineers,
and all of the roles that make up what we call Data Teams.
And Data Teams are on the rise.
And I'd say we're just at the beginning of this trend.
In fact, three out of the top eight jobs
in LinkedIn's latest emerging jobs report
are data and AI rolls.
And we all know that there's a shortage of talent
in this space.
So making Data Teams as productive as possible
seems to be pretty important.
However, Data Teams are struggling today.
Let's look at why.
First, the tools that are available to Data Teams
have been built decades ago for software engineering,
when life was much simpler and software wasn't
as closely tied to data as it is today.
Second, because Data Teams consists of many different roles,
handoffs are necessary,
but are often slow and delayed projects.
And finally, there's so much innovation in this space,
but most open source tools are built by and for academia
and are not production ready.
As a result, when it comes to
productionizing data science projects today,
we're still dealing with a big mess.
If you've ever tried to do statistical data analysis,
maybe between a machine learning model,
you know that the tools
that are supposed to make your life easier
are still difficult to use.
The reason of course, is that existing solutions
still haven't caught up with emerging practices
in data science and machine learning.
As a result we are made to choose between three options
that are all not that great.
The first and for many the most natural option
is to just give everyone the freedom
to do whatever they want on their laptops.
Of course, data scientists love that.
You have full freedom to install anything you need
and you can move fast.
However, you're pretty far away from your data
and you'll need to down sample
and copy data onto your laptop.
Of course, you don't have to sit in compliance
to know that moving sensitive data onto your laptop
is generally a bad idea.
And the folks who are maintaining your production systems
are definitely not going to be happy
to try to reproduce your local environment.
To address some of these concerns,
some vendors take the approach to just
put the same tools you use on your laptop, into the cloud.
Essentially, they're giving you a virtual laptop.
However, just hosting (mumbles)
and giving you a virtual machine
with scikit-learn pre-installed
isn't that much of an improvement.
Sure, you no longer have to copy data onto your laptop,
but aside from security and governance,
there are no obvious benefits.
So really what you're getting are the same tools,
just with more constraints.
And finally, you may be asking yourself
why not iterate on our production infrastructure directly?
Well, unfortunately those production hardened systems
are not really ideal for exploration
and most data scientists will not be happy
if you try to make them learn Coopernetties.
So we're left with a hard choice,
full freedom of your laptop, slightly worse experience
of the same tools in the cloud with few benefits,
or a fully production hardened system
that no data scientist will want to use.
And I'd say, none of these options are that great.
But we at Databricks believe
that Data Teams really deserve better.
So with a little bit of customer obsession and engineering,
we've been able to navigate some of these trade-offs.
Our solution for modern Data Teams
starts with the premise that develop environments
need to be open and collaborative.
Our workspace follows open source standards
and provides a collaborative notebook environment
on a secure and scalable platform.
Let's start with looking at
some of its differentiated features.
Here I am in a Databricks notebook where immediately
you can see that Databricks notebooks are multi language.
Each notebook has a default language, which can be
Python, Scala, SQL, or R, but that's just the default.
Each cell can individually declare its language
and you can mix and match.
In this example, we use a SQL query
to quickly look data from Delta Lake.
This allows different members
from the data team to collaborate.
One common example, is data engineers
writing their pipelines in Scala
and data scientists then using Python
to train machine learning models.
Databricks notebooks also have built in visualizations.
So you can look at your data in a table view,
or you can switch to one of many
supported visualization types.
Here you can see how easy it is
to create a bar chart based on the data that we just loaded.
Because notebooks can become quite long,
we added the table of contents feature
that uses marketing hierarchy to allow easy navigation.
This notebook Train Keras Model,
so let's navigate to that section.
One thing you will notice
is the experiment button in the top right.
Here you can see all of the runs
that were locked using ML flow.
One common use case is to look for the best model
by sorting by a metric.
So you can easily find the best run with the lowest loss.
You'll see that we show parameters and metrics
that we're locked with this run as well.
And because of the tight integration with the notebook,
you can also go back to the exact version of the code
that created this run,
allowing you to reproduce this in the future.
If you use the TensorFlow backend for carers,
we also embed TensorBoard in the notebook
to look at the TensorFlow metrics during training.
You can already see how easy it is for data engineers,
data scientists, and ML engineers
to collaborate in this environment.
Let me highlight some of the collaborative features
in a different notebook.
Here I asked my colleague Austin
to help me out with this forecasting notebook.
Databricks notebooks can be shared,
and you can see from the co-present indicators
in the top right, that there are two users in this notebook.
This environment also provides a commenting feature
to facilitate collaboration.
And what you will see here is Austin's cursor position
in real time, as he edits the notebook.
And he'll add a new cell with more plots,
which helps me out a lot.
Now once that is done,
it is super easy to create production jobs
from your notebooks, by allowing you
to configure a schedule on which they should run.
This is very useful in cases
where you want to update forecasts or dashboards regularly.
Now finally, I kept the most strategic feature,
which has been widely requested
for our customer base, for last.
Soon, Databricks will allow users to configure
a new color scheme for their notebooks.
And maybe you can already guess what it is.
It's dark mode.
So by turning the lights off, you can go from this view
to this one.
And I know a lot of our users
are as excited about this feature as I am.
To summarize, the Databricks workspace
provides a differentiated notebook experience
for Data Teams.
I showed you, in order of appearance,
support from multiple languages, built in visualizations,
a table of contents, integrated experiment tracking
and reproducibility with ML flow, embedded TensorBoard,
collaborative features like co-presence, commenting,
and co-editing, scheduling notebooks as production jobs,
and of course, dark mode.
Now, many people think that such a collaborative platform
can only be useful for exploration and experimentation.
It has proven difficult to combine the ease of use
and collaborative features of notebooks
with the rigor of production deployments.
So let's take a look at the other capabilities,
then ideal solution for modern Data Teams need to provide.
For code versioning and CI/CD,
the industry has already figured out best practices
and they're Gid based.
So the solution has to integrate with this ecosystem
and bring those best practices to data engineering
and data science, where reproducibility
is becoming more and more important.
Finally, to reduce the time
from experimentation to production,
the same environment needs to be scaled
to production deployments, allowing you to manage
the full life cycle within one platform.
Now, as mentioned earlier, most people say
that it is impossible to provide one solution
that satisfies all of these requirements.
But at Databricks, we are really focused
on making Data Team successful.
So we've spent a lot of time and energy on this problem.
So today, for the very first time,
I'm extremely excited to announce
that we are combining all of these attributes
in one product for modern Data Teams,
and we're calling it
The Next Generation Data Science Workspace.
Now I already walked you through
our collaborative notebooks.
So let's see how we extend this environment
to support production grade deployments.
We have introduced a new concept called projects
to the left navigation panel in Databricks.
One of the most common ways
that data scientists start the work
is to clone a Git based repository.
With projects, you can use Git
to bring all of your work to Databricks,
where you can access all of your data
and use best of breed open source tools
in a secure and scalable environment.
Of course, when you do this in an enterprise environment,
there's a lot at stake.
So we have added security features
to protect your intellectual property.
Admins can configure allow lists, that specify
which Git providers and repositories users
have access to them.
The most common use case for this is to avoid code
making its way to public repositories.
Another common pitfall are unprotected secrets
checked into your code.
We detect those before you actually commit your code,
helping safeguard your credentials.
So now that you can be assured
that your code base is secured,
we want to make working with projects as easy as possible.
One aspect of this is to support a rich data format
that allows us to retain more metadata
than just exporting source files.
We have extended the MB format,
that underlies ipynb, to retain some of the metadata
from Databricks notebooks.
One example, our notebook dashboards,
that can now be stored with the ipynb file.
By using ipynb, we make sure that you can benefit
from the ecosystem around this open format.
To give you one example, most Gid providers
can render ipynb files and give you
a great code review experience.
Once you're done updating your code,
we built the easiest way we could think of
for staging your files and looking at visual diff
before you check in your code.
This makes it much easier
to decide which changes to check in,
which to revert or which still needs some more work.
Finally, to integrate all of these workflows
with your CSED systems, we provide an API surface,
so you can programmatically manage projects,
perform Git operations, and automate your deployments.
Now, this really brings production grade rigor
to your data science and machine learning teams.
To summarize, we showed the new project concept,
allowing repositor level Git integration.
Security features like allow list and secret detection,
to safeguard your intellectual property.
Support for sewing Databricks notebooks, with the NB format.
New features for staging files and visual diffing.
And the projects API, which allows programmatic integration
with your CSED system.
And I'm extremely excited to announce
that everything I showed you today
will be released within two short months.
Now, to make sure that you don't miss the public launch,
sign up at this URL to get notified.
So we've worked with many of our customers
to develop these features, and before I close,
I briefly want to share some of their success stories.
One of our retail customers observed the cultural shift
in their Data Teams.
For them, The Next Generation Workspace
really elevated the team to focus on quality
and follow best practices.
Another customer in media, found that the projects feature
helped them scale their efforts
and increase their confidence
in the production systems that they built.
Initially, they went through elaborate processes
to productionize their work,
but with projects, they could automate that process.
And finally, a customer building industrial machines
cut their time to value from weeks to just days
by streamlining the path from experimentation to production.
Now all of these companies
are reinventing themselves to be data first,
and The Next Generation Data Science Workspace on Databricks
is giving their Data Teams
the environment that they really deserve.
So in summary, this is our solution for modern Data Teams.
I showed you how we provide collaborative notebooks
based on open standards, integrate with the Git ecosystem
for collaboration reproducibility,
and provide integrations with CSED systems
for robust workflow, from experimentation to production,
all on a secure and scalable cloud platform.
Now, right after this talk another company
that has reinvented itself around data
is going to share their story.
When we think about luxury cars,
what first comes to mind is often the end product.
The sleek design, how fast it goes and so on.
But we often overlook the enormous amount of effort it takes
before that car actually rolls off the assembly line.
So next up we are going to hear
from the data team at Timler.
Who will give us a peek into how data and machine learning
are playing a critical role
to drive car production automation.
The leverage tools like ML flow,
to automate a number of complex processes
and provide insights that create production efficiencies.