>> Big data clusters provide
a way of keeping the cluster reliable by enabling
high availability for critical components and Mihaela is
here to tell us all about it today on Data Exposed.
[MUSIC]
>> Hi, and welcome to another episode of Data Exposed.
I'm your host Jeroen and today we have Mihaela with us to talk
about big data clusters and then
specifically the high availability for them.
So welcome back. This is must be the forth time I think.
>> Yeah. Thank you. Thank you for having me here.
>> Yeah. You're becoming a pro.
So most of the topics you talk
about is big data clusters and today is no different.
But then high availability, right?
>> Yeah. So there are a lot of things to
talk about when it comes to high availability.
>> Okay.
>> We're going to go through some of those aspects in this video.
>> Okay. Cool. Now let's get started.
>> So when we talk about data especially and databases,
we want to make sure that data is persistence.
So I just want to start with
this high availability talk with the storage recap.
>> Okay. >> So different layers in
the big data cluster have different options for storage.
Either you can do a local storage or
remote and we made it as granular as
you can opt in for local or remote
depending if you want to store data, or the logs.
So logs you don't want necessarily to make it redundant
because you might need it for
troubleshooting but then you don't want to keep them forever.
>> [inaudible].
>> Exactly. So when we talk about logs is
mostly you want to keep them on a local storage
especially because we were talking in the last video that we have
components in the clusters that are
collecting those logs and are starting them in an elastic search.
So you already have some dependency on that aspect.
When it comes to data, various components
have different requirements depending on how
a mission critical are and if there's
any user data that is stored in for data for example,
SQL Server master or storage pool like HDFS data is kept.
You do want to maintain redundancy for that.
But Compute pool or Spark,
there is no state.
It is just computes. So there's no point
to add additional redundancy to the storage.
>> Exactly. So you can choose local.
>> So we're talking here about
different options that you have to ensure
the reliability of those services
when it comes to data persistence.
>> Okay.
>> That's where we continue with the HA options, right?
So for SQL Server master if your story in your data locally,
you must ensure that you're adding
some additional redundancy to that
with Availability Groups and we are going to
see shortly how's that enabled.
When it comes to Data pool,
you use PVs in combinators to ensure that data is persistent.
>> So it's just PVs, right?
There's lot of acronyms here.
>> Yeah.
>> Eg PV, HA, everything PV is?
>> Suggests PVs it's a Kubernetes concept
that abstracts the storage layer of
Kubernetes and ensures if you're using persistent volumes.
So the notion is data persistence.
So if you're using persistent volume is it
means that Kubernetes ensures that
data is persisted on that storage.
>> Okay. Got it.
>> Again, that is no need to ensure
high availability for compute because it's stateless.
It have critical components
in the Hadoop Stack right when it comes to
HDFS NameNode and some Spark shared
services that you need to enable high availability for,
and very important I want to highlight here
the Control Service that you must have not only persistent volume,
you need to add some redundancy to that story.
So it has to be some remote redundant storage.
Don't keep your Control [inaudible]
locally because if that node is last here,
pretty much entire cluster is not in a very constrained.
>> Okay. So Control do have PVs on a remote storage?
>> Remote and redundant.
So you have to make sure that they add
some redundancy to that layer.
>> Okay. Noted.
>> So now let's see what that means for
SQL Server master and enabling AG's for that.
So this is a schema or
how the layout of various services that form the SQL Server,
high availability layer for SQL Server master.
Again, we have a primary which
is at least two secondaries right synchronous,
and we built components that
are ensuring that there is automatic monitoring,
automatic fail-over and orchestration.
If something happens with a primary,
it happens automatically, there is no need to do anything.
One thing that I want to highlight here is
that for big data cluster only at this time,
we also enable what we call a Contained Availability Group,
which means that now objects that
you store in master for example like
logins are also replicated to the secondaries, right?
>> Okay.
>> So until now this is along are sending us from
our customers to make sure that logins
are also replicated otherwise,
there are a lot of accusation and
manual replication they had to do.
Right now automatically everything is taken care of.
So from deployment, from adding
databases to the availability groups,
to adding this master replicated database availability groups.
So there is little if none
amongst operational management of
the availability group. That's pretty awesome.
>> Yeah. That's really awesome. I was going to say.
So but you mentioned availability groups now, right?
>> Yeah. >> Is that the regular?
>> Yeah. It is exactly the same feature that we
all know from SQL Server 2012, right?
>> Yeah.
>> One thing that it's very important.
There is no other cluster technology
that you're going to have to deploy or integrate with.
Its everything taken care of,
the services that are deploying with the HA supervisor,
the operator and of
course tightly integrating with Kubernetes in writing this case.
So we are taking advantage of these platforms.
>> So no more cluster technology.
So this is great for the mastering.
So now I trust the master instances is fine.
But there's more into BDC, right?
We're not only doing a SQL Server,
we're doing [inaudible] related stuff.
So tell me.
>> So let's look at what we're doing for Hadoop, for HDFS.
So HDFS NameNode must also be in
a highly available configuration because that's critical
for the Hadoop Stack,
and what we're seeing that the customer is telling us, ''Oh,
I want replication for NameNode'',
would also deploying Zookeeper which
is an open source cluster technology.
That's the component that is going to take care of coordinating
the monitoring and the fail-over if
needed of the NameNode to a standby secondary.
>> Okay. >> So deploying an additional replica
and Zookeeper is taking care of the orchestration aspect.
>> Okay.
>> In the same time it's also involved in
maintaining high availability for
some Spark share components like Yarn Resource Manager,
and in that sense for Spark we also deploy
multiple replicas for services like Spark History, Job History.
So to make sure that if something is
going on in OneNote that these services are hosted,
The [inaudible] would be picked up or additional replicas.
>> Cool.
>> So let's see how easy it is to
configure the high availability for the various components.
>> Tell me it's easy.
>> It is super easy.
>> Cool. I like easy.
>> We talked last time about how to configure your deployments.
>> Yes. I remember that.
>> There is the cluster configuration files
or deployment templates that you have,
and remember we're talking earlier about
the Spark share components.
>> Yeah.
>> I just say I just want two replicas of them and that's it.
We take care of picking up from there.
>> Is that all?
>> The Zookeeper. So again,
we have to go through all the components that we went through.
Zookeeper we're going to need three replicas to ensure quorum.
Then we also mentioned master,
SQL Server master instance and what do I do here?
I'd just say that I want three replicas,
and because SQL Server availability groups
also enables readable secondaries,
will give you the option to
deploy a service that is exposing an endpoint
to duty remotely workload
from from the secondary and you just have to
specify the port here in this case.
>> Right. So you do a high availability and as part of that,
you could also do the read-only, [inaudible]
>> Exactly. Yeah.
>> Cool. Is that how you read this just as one line [inaudible]?
>> Yeah. You just specify how many replicas you
don't worry about orchestrating,
deploying additional components like when you tell
us that I want three replicas for SQL Server master,
we deploy the operator,
we deployed the supervisor that is
doing the monitoring and everything else.
So everything is behind the scenes and that
is minimal orchestration for setting things up.
For folks that are very familiar with how
to configure an availability groups I think it's
at least four or five T-SQL statements
plus prepping endpoints and things like that.
So that's prickly asked.
It's taking that load from YouTube to
focus on actually running what's on the big data.
>> Right. It doesn't get more simple than this, right?
>> It is.
>> One line and then of course if the master instance if you want
more lines for read only but yeah that's really impressive.
Cool. So where can I find more about this?
How do I get started?
>> So definitely I'll show you
exactly some links that you can leverage
to for the deployment, for the configuration.
So you can find hear more about it in our documentation platform
but we also have a lot of samples out there
on how to configure things.
How to run workloads,
and everything you can go ahead to use
this links and leverage them for what do whatever you want to do.
You'll be our clusters.
>> Cool. Well, thanks again for sharing and talking though this.
This is a very impressive.
I like the easiness of creating this.
>> Yeah. >> This is clearly a lot of work.
>> Pretty awesome. Yes. Thank you.
>> Well, thanks. Thank you for watching.
Please like, subscribe, leave a comment,
and hope to see you next time. Thanks.
[MUSIC]