>> Within SQL Server Big Data Clusters,
there are multiple options to configure
the services running inside of the cluster.
Mihaela is here to tell us all about it today on Data Exposed.
[MUSIC].
>> Hi and welcome to another episode of Data Exposed.
I'm your host Jeroen, and today we have Mihaela with us to
talk about SQL Server Big Data
Clusters Configuration. Hey, how are you?
>> I'm good. Thank you for having me here today.
>> Welcome to the show.
So SQL Server Big Data Clusters Configuration,
tell me how does that work?
>> So yeah. Thank you for having me today.
We're going to cover a few aspects of how you can deploy
Big Data Cluster in
an optimized way to accommodate your type of workloads.
So in order to do that,
before deployment you need to plan how to scale your services,
how to configure different networking and things like that.
So we're going to cover a few of those things and
how especially on the experience side,
like how exactly you're going to do that,
we're going to build that cluster.
So what exactly can it configure when you apply to that cluster?
You can obtain to have
certain services that are mission critical for high availability.
You can also have security enabled,
like you can integrate
certain external services that are deployed within
that Big Data Cluster with Active Directory.
You can configure the scale,
you can deploy multiple Data pool Instances,
multiple Storage Pool Instances and so on.
Definitely storage, storage is important.
You want to scale in that aspect as well.
Networking, right, and you can configure the codes that you want
to external endpoints to be exposed through port placement.
It is important to pin different services depending
on resources that are required to different node types to
accommodate different types of hardware that you want
to use as well
as resources that the catalog allocated for various components,
and I'm going to show you how exactly you're going to do that.
>> Right. So basically
we have all of these configuration settings.
I can take Big Data Clusters and roll my own, right?
To make sure that it
fits exactly to what I have in my environment,
what I've acquired, all those things.
>> If you want to accommodate
your environment like infrastructure,
you want to accommodate your type of workloads
and if you're running more Spark or more SQL type workloads,
definitely those things are flexible and you can do that.
>> So hopefully you'll help me do that, right?
I wouldn't know how to start so.
>> Definitely. So there are tools
that we put out there so you can do that.
First of all, how you deployed your cluster using
ASCII data and that's also
the tool that you're going to use to configure before
deployment and later when
we're going to enable post-deployment configurations,
you're going to do the same thing and Azure Data Studio.
Like now, soon enough we're going to
have deployment wizard that will guide you through
the experience of deploying and then you're going to be
able to specify it through
that wizard experience what are
different tweaks that you want to have for various services.
Well I'm a big O of Microsoft Teams.
So I like my widgets. So is it going to be metric for me?
>> Yeah to the next, but definitely you need to pay attention to
what you want to configure and customize for again,
you don't want to deploy and then realize, "Oh,
I didn't put the port that I want to," or things like that.
>> Okay. So no next finish, pay attention right?
>> Yes.
>> Noted.
>> So let's see how that will work for example with ASCII data.
We're going to show you in
another session how you do that with the wizard.
I know it's your favorite.
>> I would like to see that. Yes, absolutely.
>> But let's see how that is going because you know to be fair,
the wizard did actually using
ASCII data and then turning the covers, right?
>> Of course.
>> So what are we going to do to help you?
We already have some built-in templates
that I showed you actually last time in the previous session about
deployment that we put there for you to accommodate
various platforms that you're deploying like if you're
deploying On-Prem or on Azure using Azure Kubernetes Service,
we have different configurations that
are particular for that type of environment.
So you're starting from one of those templates and
then you pretty much clone it and then you use that
copy to customize and then you can customize
it using something like VSO or again,
your favorite ASCII data tool.
It can actually do patching and you
can issue statements to replace
values within the configuration files,
and then you just deploy and the input to the deployment command
will be custom configuration topic data.
>> So I don't have to create the configuration from scratch?
If I wanted to, I could probably.
>> You could. I'm going to show them how they
look like and you're going to see that well,
yeah definitely it's possible.
But you want to start from something.
So we already put out there like for example the one for AKS,
you don't really have to modify anything in
order to have minimum configuration like
a test kind of cluster
that you want to ramp up with on the Data Clusters.
So nothing there is needed to modify, everything is set up.
I'm going to show you a more complicated line though.
>> So on using ASCII data of
course compared to the wizard would have
the added benefit I assume
to that I would be able to script this and automate all of this?
>> Yes. Again, everything that you have here are commands
that you can use in a script to automate it
either for your deployment pipeline, test CIC configurations.
So if you have multiple repeats of deployments,
then definitely scripting is your tool.
>> Cool. So can you show us how it works?
>> Yes, let me show you
the deployment configuration files and
how they will look like when.
So when you create that clone that I told you,
when you start from the template that
we built in within Data Cluster,
and I already created the copy and I use a template
for a cube ADM cluster like a Kubernetes Clusters is not AKS,
because I want to point out a few differences that you
need to pay attention when you deploy in a Kubernetes Cluster.
It creates two configuration file,
one for the controller service which is a set of
services that help you manage the cluster.
So what you have here,
the actual controller service then in this file
you're going to specify what are the docker settings,
where these older images that are required for Big Data Cluster,
what are their pool from.
Storage, settings, and I want
to highlight few things around storage.
If you specify a storage setting in the controller,
all the service into Big Data Cluster are going to
inherit these settings unless you
overwrite it in the file that is corresponding
to the Big Data Cluster services
and I'm going to show you how to do that.
Something to pay attention to because whatever you put
here is going to be used by SQL Server Master Instance,
by GFS and everything.
So you want to make sure that you either
overwrite or whatever values you put here,
they are according again to the type of workloads that you want to
run and the data size
that you want to load into the Big Data Cluster.
>> Can I see this as the master blueprint component?
>> Yes. So this is the brain of your Big Data Cluster.
Then some endpoints that again,
networking settings that are exposed for the controller.
So to actually connect to talk to the cluster,
to Management Service and then to the service proxy which is
an endpoint that is used for monitoring components.
I told you about security.
You can enable security integration with Active Directory,
that's another thing that you
would configure through the controller.
So everything from DNS settings and so on,
you will input here.
Then once you're done with this one,
you move to configure the actual services that
are running as what we call Big Data Clusters,
SQL Server Master, HDFS,
Spark and so on.
So if you look at this file how it's structured,
it has all the resources that are
deployed and we call them resources.
They are Kubernetes resources.
Within this resources, you have services that are running.
To give you an example,
in Master for example. What is Master?
Is SQL Server Master Instance and
what is the service that is running there? It's SQL.
So SQL Server we know in Big Data cluster it's running in master,
it's running in the Compute Pool
and Data Pool and in Storage Pool,
and with each one of them you can configure things.
So let me show you for example that we are here with the Master.
In this case, I obtained to enable high availability for
my and the way I do that, very easy.
I just say I just want three replicas and everything
happens behind the scenes magically. It's too easy.
>> This is almost too easy but this is perfect.
>> That you need to pay attention
to all these details. That's the only thing.
You just need to understand what are the services that are being
deployed and what can and cannot configure there.
Because I'm going to deploy with Active Directory Integration,
I have to specify the DNS names for various endpoints,
the external endpoint, and again,
I have Availability Groups configure for Master,
that means that I have readable secondary endpoints.
So I have to do that to configure that as well.
Again, my storage configuration.
So I want to highlight this because
I want more storage for example for
my Master Instance than what I
specified initially in the controller.
So I will have to override those settings.
Another important thing for a Kubernetes cluster,
we don't know the class name that you configured in your cluster.
It's not like we have
some default storage classes that for example,
Azure Kubernetes Service has.
So that's something that you will have to provide and it's
not built-in in the template that they provide for you.
So it's required when you deploy.
So that's about it. Similarly, to give you
an example for the storage pool what that have.
You can specify the scaling for various services.
So that's how you by specifying the number of replicas,
you can overwrite storage and so on.
So that's about it like there
are something that I did not include in here but
you can definitely find in
our documentation and I can point you to
where that is in our doc page,
is something how to do
pod placement like all these services to pin them to the type of
hardware that you want to deploy various services on.
So that's something that I saw
that customers are asking about and it's very
important when they deploy in
production to make sure that they leverage the hardware and
the infrastructure that they have for various type of workloads
either on the Hadoop or SQL Server site.
>> Wow, this was a lot of information.
>> It was, hopefully useful.
>> Yeah. No I'm very happy to see all this
configuration files I had a lot of trouble with it myself,
and I'm happy to see that we have templates that help me
get started and then that law of course,
the easiness of saying,
"Oh, I want this amount of replicas."
I mean, that is something that
I'm really looking forward to using.
So thanks a lot for sharing,
thanks a lot for being here,
and thank you for watching.
Please, like, subscribe, comment on
the video and I hope to see you next time. Thanks.
[MUSIC]