[Narrator] Live from San Francisco
it's the Cube covering Sparks Summit 2017
brought to you by databricks.
>> Welcome to the Cube.
My name is David Goad and I'm your host
and we are here at Spark day two.
It's the Spark Summit and I am flanked by
a couple of consultants here from--
sorry, analysts from Wikibon.
I got to get this straight.
To my left we have Jim Kobielus
who is our lead analysist for Data Science.
Jim, welcome to the show.
>> Thanks David.
>> And we also have George Gilbert
who is the lead analyst for Big Data and Analytics.
I'll get this right eventually.
So why don't we start with Jim.
Jim just kicking off the show here today,
we wanted to get some preliminary thoughts
before we really jump into the rest of the day.
What are the big themes that we're going to hear about?
>> Yeah, today is the Enterprise day at Sparks Summit.
So Spark for the Enterprise.
Yesterday was focused on Spark,
the evolution, extension of Spark
to support for native development of deep learning
as well as speeding up Spark
to support sub-millisecond latencies.
But today it's all about Spark and the Enterprise
really what I call wrapping dev-ops around Spark,
making it more productionizable, supportable.
The databricks serverless announcement,
though it was announced yesterday,
the press release went up
they're going into some depth right now
in the key note about serverless
and really serverless is all about
providing an in cloud Spark,
essentially a sand box for teams of developers
to scale up and scale out enough resources
to do the modeling, the training, the deployment,
the iteration, the evaluation of Spark jobs
in essentially a 24 by seven multi-tenant
fully supported environment.
So it's really about driving this
continuous Spark development and iteration process
into a 24 by seven model in the Enterprise,
which is really what's happening is that
data scientists, Spark developers are becoming
an operational function that businesses are building,
strategic infrastructure around things like
recommendation engines, and e-commerce environments,
absolutely demand 24 by seven resilience Spark
team based collaboration environments,
which is really what
the serverless announcement is all about.
>> David: So getting increasing demand on
mission critical problems
so that optimization is a big deal.
>> Yeah, data science is not just an R&D function,
it's an operational IT function as well.
So that's what it's all about.
>> David: Awesome, well let's go to George.
I saw you watching the key note.
I think still watching it again this morning,
so taking notes feverishly.
What were some of the things that stuck out to you
from the key note speaker this morning?
>> There are some things that are sort of going to
bleed over from yesterday
where we can explore some more.
We're going to have on the show,
the chief architect, Renald Chin,
and the CEO, Ali Goatsee,
and some of the things that we want to understand is
how the scope of applications
that are appropriate for Spark are expanding.
We've got sort of unofficial guidance yesterday
that, you know, just because Spark doesn't handle
key value stores or databases all that tightly right now,
that doesn't mean it won't in the future
on the Apache Spark side through better APIs
and on the databricks side, perhaps custom integration
and the significance of that is that
you can open up a whole class of operational apps,
apps that run your business
and that now incorporate, you know, rich analytics as well.
Another thing that we'll want to be asking about is,
keying off what Jim was saying,
now that this becomes not a managed service
where you just take the labor that
the end customer was applying to get the thing running
but it's now automated
and you don't even know the infrastructure.
We'll want to know what does that mean for the edge,
you know, where we're doing analytics
close to internet of things and people
and sort of if there has to be
a new configuration of Spark to work with that.
And then of course what do we do
about the whole data science process
and the dev-ops for data science
when you have machine learning distributed
across the cloud and edge and On-Prem.
>> Jim: In fact, I know we have Pepperdata coming on
right after this,
who might be able to talk about
that exact dev-ops in terms of performance optimization
into distributed Spark environment, yeah.
>> George, I want to follow up with that.
We had Matt Fryer from Hotels.com,
he's going to be on our show later
but he was on the key note stage this morning.
He talked about going all cloud, all Spark,
and how data science is
even competitive advantage for Hotels.com.
What do you want to dig into when we get him on the show?
>> That's a really good question
because if you look at business strategy,
you don't really build a sustainable advantage
just by doing one thing better than everyone else.
That's easier to pick off.
The sustainable strategic advantages come from
not just doing one thing better than everyone else
but many things and then orchestrating
their improvement over time
and I'd like to dig into how they're going to do that.
'Cause remember Hotels.com
it's the internet equivalent descendant of
the original travel reservation systems,
which did confer competitive advantage
on the early architects and deployers of that technology.
>> Great and then Pepperdata wanted to come back
and we're going to have them on the show here in just a moment.
What would you like to learn from them?
What do you think will benefit the community the most?
>> Jim: Actually, keying off something George said,
I'd like to get a sense for
how you optimize Spark deployments
in a radically distributed IOT edge environment.
Whether they've got any plans,
or what their thoughts are in terms of the challenges there.
As more the intelligence gets pushed to the edge
much of that will be on machine learning
and deep learning models built into Spark.
What are the challenges there?
I mean, if you've got thousands to millions of end points
that are all autonomious and intelligent
and they're all running Spark,
just what are the orchestration requirements,
what are the resource management requirements,
how do you monitor end-to-end
in and environment like that
and optimize the passing of data
and the transfer of the control flow or orchestration
across all those dispersed points.
>> Okay, so 30 seconds now,
why should the audience tune into our show today?
What are they going to get?
>> I think what they're going to get is a really good sense for
how the emerging best practices for optimizing Spark
in a distributed fog environment out to the edge
where not just the edge devices
but everything, all nodes, will incorporate
machine learning and deep learning.
They'll get a sense for what's been done today,
what's the tooling is to enable dev-ops
in that kind of environment.
As well as, sort of the emerging best practices
for compressing more of these algorithms
and the data itself
as well as doing training in
a theoretically federated environment.
I'm hoping to hear from some of the vendors
who are on the show today.
>> David: Fantastic and George,
closing thoughts on the opening segment?
30 seconds.
>> Closing thoughts on the opening segment.
Like Jim is, we want to think about Spark holistically
and it has traditionally been best position
that's sort of this-- as Tay acknowledged yesterday
sort of this offline branch of analytics
that you apply to data like sort of repository
that you accumulated
and now we want to see it put into production
but to do that you need more than just what Spark is today.
You need basically a database or key value kind of option
so that your storing your work as it goes along
so you can go back and analyze it
either simple analysis or complex analysis.
So I want to hear about that.
I want to hear about their plans for IOT.
Spark is kind of a heavy weight environment,
so you're probably not going to put it in the boot of your car
or at least not likely anytime soon.
>> Jim: Intelligent edge.
I mean, Microsoft build a few weeks ago
was really deep on intelligent edge.
HP, who we're doing their show
actually I think it's in Vegas, right?
They're also big on intelligent edge.
In fact, we had somebody on the show yesterday
from HP going into some depth on that.
I want to hear what databricks has to say on that theme.
>> Yeah, and which part of the edge,
is it the gateway, the edge gateway,
which is really a slim down server,
or the edge device,
which could be a 32 bit meg RAM network card.
>> Yeah.
>> All right, well gentlemen appreciate
the little insight here before we get started today
and we're just getting started.
Thank you both for being on the show
and thank you for watching the Cube.
We'll be back in a little while
with our CEO from databricks.
Thanks for watching.
(upbeat music)