>> SQL Server 2019 introduces big data clusters.
It has Spark integrated.
Shiv is here to tell us all about that today on Data Exposed.
[MUSIC].
>> Hi and welcome to another episode of Data Exposed.
I'm your host, Jeroen and today we have Shiv here with
us to talk about Spark on big data clusters.
So welcome to the show, Shiv.
>> Thank you, Jeroen.
>> So spark, let's start at the basics. What is Spark?
>> Spark is a unified big data processing engine
that can work across your analytic workloads
>> That doesn't sound very simple.
>> So let's break it down.
>> Okay.
>> So first all let's talk about
big data processing, distributed big data.
So last few years,
what we've been seeing is a trend of
enterprises collecting lots and lots of data.
>> Sure.
>> From going from GBs of data,
today we see an enterprise dealing
with terabytes and petabytes of data.
Now there's a problem there.
The problem is that when you have such a large scale of data,
how do you really store that data, first of all, right?
>> Okay.
>> So at the start,
we started with a single machine and scaling it
vertically and having terabytes of hard disk per data group.
That vertical scaling was not really
the answer for storing distributed data.
A feasible, a more better solution,
more resilience solution was distributed data,
where we don't keep scaling
a single machine to take on more and more data.
What we do is we distribute the data across
n number of smaller machines and that's how we store big data.
>> So basically divide and conquer,
right? We divide the work.
>> Exactly.
>> Okay.
>> So now, we have taken care of
the problem of storing the data but that's not all.
The main problem is not storage of data,
the main problem is that I need to gain insights of this data.
>> When you process it,
that's where the value is, right?
>> Exactly. So processing of
this distributed data requires different engines.
Spark is a big data compute engine which can work across
distributed data and compute and do your workloads on that.
It's just not a distributed big data compute engine,
it's also something that abstracts about
the details of distribution from you.
As a user of Spark,
you won't have to bother about all the details of distribution,
that's the whole beauty around Spark.
>> So you just give it an assignment and it'll figure out
how to distribute the work and
be done as quickly as possible, hopefully.
>> Right. When you are dealing with such kind of data,
you don't want to be tied down to a particular language.
You're like this is a big data compute engine and now
you use the flashy language that I've invented to process that.
So Spark does something very beautiful there.
Spark gives you a choice of language.
If you're a Python programmer,
you can program in Python, Scala, Java,
R. R is very popular about our data scientists and
Spark gives you the option to use R for your workloads.
So that what is Spark as a distributed Compute Engine.
>> So basically you said
restore it differently but that's something we did for
big data because of the problem of scaling
upwards all the time with scaling
up and now we're scaling out as well. Right?
>> Right.
>> Then Spark works on the distributed layer
and gives you the flexibility of
choosing the language of choice to.
>> Yes. There was still a bit [inaudible] into my sentence.
I told you something about a unified Compute Engine.
So let's slice and dice that a bit.
As we saw, enterprises started getting more and more data.
The traditional workloads also move from
transaction workloads to workloads which are analytic in nature.
What do we mean by analytic workload?
Workload where I'm analyzing a lot of data to get
insights out of it and
then maybe doing machine learning or deep learning.
So traditionally, the phase from
the transaction workloads also moved to analytic workloads.
Analytic workloads had a variety of workloads from deep learning,
machine learning, analytics and streaming workloads.
Now, each of these workloads you don't want
a separate compute engine to really build your skills on.
>> Ideally you would learn one.
>> Exactly. That's what Spark does.
Spark is a unified compute engine that allows you to work
across all these workloads with the same set of principles.
That's what Spark is about.
A distributed compute engine that extracts out
the details of work distribution from you.
It doesn't have you bother about distribution details.
Second, a unified compute engine and above all,
which I find it as a very powerful developer feature,
it offers you the choice of language where you could use Python,
Scala, Java or R, whatever you choose from.
So that what Spark is.
>> Cool. That's very repressive.
I mean, Spark that's fine.
So what are we doing with Spark on SQL Server?
Do we have anything?
>> So Spark is basically ApacheSpark.
It's open source compute engine.
What we have done in big data clusters,
we have brought you together, this with SQL,
as a single unified offering with SQL and
end-to-end solution where you not just get the compute engine,
you get a complete end-to-end experience
on using the compute engine.
>> So that would mean that since Spark
has been integrated in big data cluster,
I can query the data in
the big data cluster
using old benefits you just describe from Spark.
>> Exactly. Using the client
tooling that you are all familiar with.
>> Wow, that's impressive.
Wow. So how do I learn more?
I mean, this sounds very new to me, where do I find something?
>> Please go to the big data cluster documentation,
SQL Server data cluster documentation and you will find a lot
of comms and articles about Spark.
You will find big data clusters,
Spark examples and the SQL Server samples repository.
That's where you get started.
>> Cool. So we'll make sure to include
those links in the description,
so you will find them there.
Thanks Shiv for coming to the show.
Thanks for explaining this.
I learned, finally, what Spark is.
Thanks for watching.
Please like and subscribe and I hope to see you next time.
[MUSIC]