>> SQL Server big data clusters provide
security mechanisms to make sure
your data access is always secure.
Nellie is here to tell us everything about
it today on Data Exposed.
[MUSIC]
>> Hi and welcome to another episode of Data Exposed.
I'm Jeroen your host and we have Nellie today with
us to talk about security for big data clusters. So hi, Nellie.
>> Hi, Jeroen.
>> How are you?
>> I'm good. Thanks.
>> Okay. So security for big data clusters,
that must sound important.
So can you tell us how it works?
>> Sure. So if you're familiar with SQL Server,
you probably know that security is always
a top priority for us
and it's the same thing with big data clusters.
We prioritize this work
and note how important it is to our customers.
So I'm going to highlight some things around the security today.
We're not going to cover all the details,
but we're basically going to cover
on high level so that you know
the security capabilities of the big data cluster.
>> Okay.
>> As you know, a lot of
our SQL Server customers are enterprise customers,
big enterprises who run Active Directory.
We need to make sure that all your applications including
SQL Server and big data clusters and all of
these solutions can integrate well with AD.
>> Yeah.
>> That's the key capability that we're
enabling in big data clusters in 2019
is to actually for you to be able to do
this in a seamless and easy way.
Now I'll tell you why I emphasized easy and
seamless because typically when you're dealing with applications,
let's say any type of application,
joining an application to AD is not a big deal, right?
We just assume that any application supports AD integration.
>> Sure.
>> Now for big data clusters, it's the same thing.
It should just work.
It's just that we have to
highlight that we're running in a fully sort of
completely containerized solution here where
all the services are running in
containers and all the supporting services
are running in containers,
and this is running on top of Kubernetes.
>> Right.
>> So we've spent a lot of work on making sure that you get
a fully automated and seamless experience of
this completely containerized big data solution
on top of Kubernetes.
>> So basically, this all runs in container so that's
why the integration is interesting.
So what does it mean to have
a fully automated integration? What do I get?
>> Yes. So for that,
you might need a little bit of background.
>> Okay.
>> So when you're joining sort of a service to
Active Directory or Kerberos
that's running inside a Linux container,
for example, it's absolutely possible.
It's just that you have to apply
some manual steps to get that to work.
Now imagine you have a solution with hundreds
of these containers with hundreds of services running in there.
To do this manually obviously would not be
realistic and we don't want to ask that from our users.
So by fully automated,
I basically mean that we are
taking care of the complexity for you.
We have a service that's called the security support service.
So as part of deployment,
that service is going to take
some information from you as a user who is deploying
the cluster and then the service is going to basically perform
all of these steps for every single service in the cluster
to make sure that everything is AD joined.
>> Wow, that's impressive. That's very cool.
So basically, I can just let
the cluster configure it and there we go, right?
>> Yes, exactly.
>> Cool.
>> Now in addition to that,
we also spend a lot of time to
make sure that we get an integrated security experience.
By that, I mean that, for example,
when you're passing a query from Spark to HDFS,
because in the big data cluster we have Spark,
you can query data in HDFS.
These components already play pretty well together.
So these components are part of the same stack,
you can say, part of the Apache stack.
So there's a lot of functionality we can already leverage there.
But when it comes to SQL Server and SQL Server
natively talking to a component like HDFS,
that's actually a new functionality
that we're introducing where we
have the capability for SQL Server query scenarios,
I have to highlight, that if I
pass a query to the SQL Server master instance,
it's just a SQL query that is querying
over an external table sitting in HDFS, right?
So when I do that,
we make sure that my identity as a person who's
issuing this query flows all the way
through all of these different components
on the way down to HDFS where the data
is so that we can do an authorization check on the actual data,
and that's what I mean with integrated security.
It's integrated through all of these components.
So no matter where I issue the query,
from Spark or SQL Server,
the user's identity is always going
to flow through all the way to the source.
>> Okay. That's very impressive and very important to
any enterprise working with this kind
of stuff because you want to know your data is secure.
>> Exactly. I mean, you have auditing,
you have your audit logs to make sure that they are up-to-date,
and you don't want some service account to show up in those.
>> Exactly.
>> So it's one of the key requirements from our customers as well.
In addition to this,
we also have the integrated experience in forms of how you,
for example, interact with the cluster.
We have Azure Data Studio that you can
use to connect to big data cluster, right?
We want to give the experience of a single
sign-on for as many scenarios as possible.
So when I connect to a big data cluster,
I actually connect to the master instance,
but our tools will make sure that I also get access to Spark,
in HDFS, and all these other
components that are interesting in the big data cluster.
>> So that's all handled transparently to me?
>> Yes, exactly.
>> Okay. Cool.
>> Yes. Last but not least,
we have encryption and
encryption is very important for our users, right?
>> Of course.
>> We have made sure that all the communication,
even internal and external communication,
with the big data cluster is SSL or TLS encrypted.
In addition to that,
you can of course leverage
the many different encryption features of
SQL Server that we have and all of them that are
supported on SQL Server on Linux
because we're running on Linux containers here.
We're also working on expanding these capabilities and add
HDFS encryption soon so that we also have
those capabilities for data that's encrypted at risk.
>> Okay. Cool. So can you
explain to us a little bit more about how
this works under Kerberos?
>> Absolutely. So let's focus on authentication
first because it's important for you to
know what difference or
entry points there are to the cluster, right?
Here you see the five different endpoints
or entry points to the cluster.
Now we have this Kubernetes cluster,
so we have to basically specifically
expose certain endpoints that users
or client tools or any applications can
interact with in the cluster.
So if we start with controller,
as you might be familiar with,
controller is the brain of the cluster.
Controller is the one that keeps track of everything,
that deploys the cluster,
and all of these things.
Now to reach the controller endpoints,
you can see here that the main method you would interact
with it would be through the azdata CLI
but also through our tools.
It's mainly the endpoint that an administrator would use,
for example, to interact with the cluster.
But controller also has magic powers.
You can say controller can sort of reach out to other endpoints.
So for example, you can log into
controller through azdata and from
there issue commands that will
take you to SQL Server master instance and you just start running
T-SQL or you can run HDFS commands
directly in an HDFS shell in these type of things.
So that's what that endpoint allows you to do among other things.
>> Okay. Very cool.
>> Yeah. Next, an endpoint that you
might have heard about if you've
used big data clusters is the gateway.
Now, the gateway is actually the same thing.
The implementation detail behind this
is that it's the Apache Knox Gateway.
This is a gateway that typically protects, you can say,
the Apache components such as basically on the Hadoop side.
>> Right.
>> So we have Spark, Livy,
YARN if you want to connect to HDFS through web HDFS,
this is the endpoint that you use,
as well as from Azure Data Studio.
So that's good to know when
we're talking about the gateway what we mean there.
Then we have the management proxy which
is the gateway to metrics and logging tools,
and then we have obviously SQL Server master instance.
It's just SQL. It's just a TDS endpoint where you
connect to from whatever tools that you're familiar with.
The app proxy, last but not the least,
which is the way that you can
access the applications that
you deployed inside the big data cluster.
Now, all these different endpoints when you're running,
for example, in a secure mode,
when the cluster is running in secure mode,
I mean AD integration mode,
all these endpoints are allowing AD authentication.
So that's what I wanted to highlight here.
So when we're talking about AD authentication,
it's a full integration with AD for all the endpoints.
>> Right. In front of one of the five endpoints across the board.
>> Exactly.
>> Wow. Okay.
>> Yeah. So that's authentication basically for you.
Moving on, we also have authorization.
It's very important to protect
your data once it's inside the cluster,
once you actually manage to log in and authenticate.
Yes. So the key parts I wanted to highlight here,
and this is still high level.
I'm sure that we're going to have
additional talks about the details of these things.
But on a high level,
there are two levels of
authorization checks that you do in the big data clusters.
If we start with SQL Server, for example,
if I'm issuing a SQL Server query,
first of all, there are authorization checks
on SQL Server objects.
I need to have access to the tables that I want to query,
for example, so that we can make sure that I can access the data.
>> But that's the same authorization checks
as we had in previous releases with SQL?
>> Yes, it's SQL.
>> That hasn't changed?
>> This is just SQL.
>> Okay.
>> So basically, the permission model of
SQL Server is the same
whether it's running in big data cluster or anywhere else.
So you can still grant permissions on
specific tables and specific objects in SQL Server, right?
>> Okay.
>> But in addition to that,
now let's take the scenario when I'm
issuing a query against data that's
sitting in HDFS and I'm querying
an external table over HDFS in this case.
Not only do I need to have access to this table,
to the database where this table sits,
I also need to have access to
the actual files and data that's sitting in HDFS.
>> Sure.
>> That's what I meant with the two-level authorization checks.
So in this case, there's a check in
SQL Server and there's an additional check in HDFS.
>> Okay.
>> On the Spark side,
basically the Spark queries will flow and
the HDFS authorization check will
make sure that the permissions are honored.
>> Okay. So with all of this,
I can be sure that
the original user's identity is passed through
SQL Server to wherever the data is, right?
HDFS or however I access it, correct?
>> Exactly. Yes. Then as we sort of touched on before,
we'll have this pass-through identity
which means that the original user's identity will flow
all the way down to the data so we can actually
verify all the way
that this was the user that wanted to access the data.
>> Okay.
>> Yeah. So that basically is a summary on
high level on the security capabilities around
the AD integrations specifically for big data clusters.
>> So where can I find out more if I want to dive deeper?
>> Yeah. So if
you want to learn more about big data clusters in general
and we have security docs
covering details of what I just explained today,
you should go to this short link: aka.ms/sqlbdc.
>> Okay.
>> If you go there, you can learn tons about big data clusters.
>> All that there is to learn, right?
>> Yeah.
>> Awesome. So okay.
So basically, I need to go there
and start learning and start downloading.
Can I export it to a big PDF and
then read it in the nighttime to learn?
>> Yeah. Actually, I think you can do that. Yeah.
>> Don't print it, though. Just PDF, right?
>> Yes.
>> Okay. Cool. So thanks a lot.
This was very useful. Thanks a lot for sharing.
>> Thank you.
>> Thank you for watching.
Please like, subscribe, and comment on
the video and I hope to see you next time. Thanks.
[MUSIC]