>> Hey, friends. You know that data is one of
the most precious things that an organization can own,
but it's also one of the most important things to manage.
Now, you can bring your own keys to Apache Kafka on
Azure HDInsight to encrypt your data
at rest for greater control and security.
Dhruv Goel is here to show us how
it works today on Azure Friday.
Hey, folks. I'm Scott Hanselman
and it's another episode of Azure Friday.
In fact, this is our 500th episode. Congratulations.
>> Thank you.
>> So, I'm here with Dhruv and we're going to talk about
Apache Kafka. What do you have for me today?
>> Hey, Scott. Thanks for having me here, first of all.
I want to talk about
some new security features we have on
Apache Kafka and Azure HDInsight.
Just wanted to introduce what Azure HDInsight is first.
It's a managed cluster platform for a lot of
open source technologies that
are out there in the Apache framework,
stuff like Hive, Kafka, Spark,
HBase and Storm, all of those,
you can create clusters for an Azure HDInsight.
With an Azure HDInsight,
Kafka is a technology that people prefer for ingestion,
for streaming, and
all those different kinds of scenarios.
>> I could go and, I assume,
fire up a bunch of virtual machines myself
on the metal and install all of this,
but I assume that would take a very long time.
>> Exactly.
>> Then, I would have to maintain it and keep it alive.
>> Exactly. So, customers
who have migrated onto the Cloud and onto
HDInsight often remarked that it
used to take them months to get those things set up,
working correctly in their enterprise environment.
But, with HDInsight, you can get
that cluster up and running in 10 minutes.
You can get those services running within
10 minutes and that just blows them away.
>> Seriously, months, like, that's amazing.
>> Yeah. Yeah. It does.
So, within that, I
wanted to chat about specifically some of
the security features which we have in
HDInsight which customers really care about.
One thing you get with every HDInsight cluster
is security outside of your cluster in terms of VNET.
You can put your cluster inside of a VNET,
you can control exactly what kind of traffic flows in and
out of it by setting up networks security groups.
>> Okay. So, a VNET is a virtual network,
this is that software-based
networking that we've heard of.
It's as if you set up private subnets,
and plugged physical wires into-
>> Exactly.
>> -a network, and put your Kafka instance in there,
except you're doing it all in software.
>> Exactly. It gives you
very fine-grained control over
all the traffic that's flowing in,
and it gives you as a cluster comes up.
So, it's very easy to set up.
All you have to decide is,
do I want this kind of traffic to flow in or not?
All of this can be done easily
by Azure portal or on CLI's.
>> It sounds like there's a lot of
really fine-grained control around security,
around topography of this.
Do people who put together Kafka instances in
large systems like this maybe not do that very often,
they're not really thinking about
that as much as they should.
>> In a more complex environment,
if they tried to do that,
it takes them a long time to do it.
>> It's very difficult.
>> You need very specialist roles
to be able to manage security.
In an enterprise, you might have someone
who's a security expert,
but not a data expert.
That kind of communication might not
happen between the organizations.
So, it's very hard for those people who are
managing the cluster to
understand how the security is setup.
Now, in a big organization with
the security tools that are
available to you with HDInsight,
you can have that easily separated.
It's very simple for people to set it up,
so you don't need
everyone to be a security expert to set it up.
>> I see. So, I'm understanding that
like when you start putting that much data,
when ingesting that much data into one place,
you don't want accidentally have someone bump
into data that they shouldn't have access to.
>> Exactly. That's why
we have different layers of security with an HDInsight.
One such perimeter we talked about with the VNET,
the authentication and the authorization happens
with Azure Active Directory integration,
and with Apache Ranger,
and the stuff that we're going to talk
about in this episode is
about encrypting the data at rest with your own keys.
So, we call this BYOK, Bring Your Own Keys,
and that's something that's very critical for a lot of
our customers today because in a post-GDPR world,
they want that control,
that they have control
of the keys that are used to encrypt the data.
>> Interesting. Why is that so important?
I think about this multiple layers of
this onion as we're trying to secure things,
but you're also making this a very software,
as a service kind of an experience,
why wouldn't I just let you handle it?
>> Well, we have customers all over the world, right?
For a lot of customers,
whatever the local laws might be,
local requirements might be,
they want that control for their data.
Customers have the right to be paranoid about their data.
Data is the most precious thing for many organizations,
almost every organization out there.
So, having that control over data becomes very critical.
Microsoft is one of the most
trusted public clouds out there,
so many customers do trust us with their data.
Any data in Microsoft Azure is
encrypted with Microsoft keys.
Now, they want the next level of control which is,
"Hey, I want to bring my own keys."
>> Interesting. Now, does this
allow them to also isolate their topics?
Can they have different keys at different topic levels?
>> So, the key encryption
happens at a managed disk level.
So, it's completely transparent.
>> Wow.
>> So, that is not across topics.
For topic level security,
we'll talk about something else which is Apache Ranger,
which allows you to control which roles can have
access or which users
or groups can have access to certain topics.
>> So, we're going to have multiple videos and
this one we'll talk about bring your own key-
>> Yes.
>> -and how you do that, and then the next one,
make sure that you watch for that,
we'll talk about even fine-grained control.
>> Perfect.
>> Very cool.
>> All right. So, like I mentioned before,
we have by default,
any data that's coming into a Kafka Cluster,
goes into Managed Disks.
Azure Managed Disks by default are
encrypted via Azure SSE,
Secure Service Encryption,
which is Microsoft-managed keys.
Now, with BYOK, what you can do is,
you can bring a user
assigned identity, a managed identity,
say that your cluster, your HDInsight cluster,
has that identity and it
can access the key vault in which your key exists.
So, you use Azure Key Vault,
you can bring your own print keys using
an HSM or something like that which brings the keys
into the Azure Key Vault or you can create your own keys
within that and grant access to the cluster,
the HDInsight Kafka cluster, to those keys.
Those keys will be used to encrypt
the data that's sitting in the Managed Disks.
>> So, you said, HSM, that's a Hardware Security Module,
and then Azure Key Vault would be HSM, as a service.
>> Yes. Exactly.
>> Cool.
>> So, that's enough for the demo, for the slideware. [inaudible]
>> Yeah. Let's see some demos.
>> Okay. So, it's
super transparent for customers to set it up.
>> Okay.
>> I'm just going to quickly show.
It's a one-click setup.
You can see that in my Azure resource group right now.
I already have a key vault
and managed identity configured,
and those are pretty easy to configure as well.
So, I'm going to create a new cluster here.
>> Okay.
>> Search for HDInsight.
>> So, your here in
the portal and you're looking at kind of like
the store of all the different things,
there's lots of HDInsight things to pick from.
>> Yeah. Those were some of
the monitoring dashboards that
you saw below the first option.
This is the main HDInsight cluster option.
>> All right.
>> You click on "Create".
You give it a name,
let's say BYOK Kafka for Azure Friday,
and I'm going to choose option here.
So, this is where I choose the cluster type and,
as I mentioned, for Kafka you would just use Kafka.
We'll go with the default which is Kafka 1.1 right now.
>> All right.
>> I'm going to choose the
cluster admin name and password.
>> I suppose when you show this to people,
they have trouble getting their heads around how easy it
is to spin these things up because, like you said,
this isn't something that is brought up lightly,
it's difficult to put these things together.
This has been completely automated as a service.
>> Exactly. Like, it's not
just the setup of the infrastructure aspect of it,
getting all these services running,
getting them configured,
and once you configure them, you have to fine-tune them,
so that they're running at the best performance,
and then getting the security configured for that.
It takes a long time to set it up.
We're going to go through a few steps
in the portal and that's going to spin up the cluster.
>> Wow. I like to think
of it as concentrated best practices.
Like, maybe I could figure out how to
get something like this setup myself,
but is it using best practices
for performance for security.
>> Exactly. I mean,
we have so many customers and we get
the experience of helping
those customers achieve more
that we know the best practices very well by now.
>> Excellent.
>> So, the step comes here.
Basically, within security networking blade of it,
I say, I'm going to use Kafka Disk Encryption.
Now, what it asks me is the key URL,
so the key that resides in the Azure Key Vault,
I just need to provide the key URL of that.
So, I go into my key vault and I looked through the keys.
I already have a key configured.
>> While it says URL,
this isn't a public URL,
this isn't an address URL,
we're not putting a private key at a Dropbox somewhere.
>> No.
>> It's literally sitting inside of
this hardware security module.
>> In fact, with Azure Key Vault right now.
>> Right.
>> So, with an Azure Key Vault,
you get the key identifier.
>> Okay.
>> Right. You paste
this here and that's all you need to do.
Who can access this key? Not everyone can, right?
Like you said, the other thing
that is configured within the key vault is,
who has access to these keys.
You can see that there is
this managed identity that has
been created for this which is called,
which is this one right here, DHGoel Azure Friday managed
identity and that's the one
that has certain permissions to this key vault.
>> So, that's an application identity
rather than a personal identity.
>> Exactly.
>> Okay.
>> So, that's what I'm going to select when I
create my cluster and I'm going to
say that I want
this managed identity to be able to access the key vault,
so that's going to be associated with that cluster.
You can create a new storage account to
associate with that and I just give it a name here.
>> Okay.
>> You just go to the default settings.
We're not going to add any applications.
We'll select the default cluster size for this.
>> Any optional scripting actions,
any post-creation type things that you want to do.
>> Yes. That's not critical for this BYOK.
What I wanted to showcase was the only change for
the BYOK was that you needed to provide
the key URL and you needed to provide
the manage identity that you
wanted to associate with the cluster.
That's it. Super-transparent. You click on "Create".
This will take about 10 minutes to create.
I just wanted to show you,
what a BYOK cluster looks like after the fact,
and I have one here.
>> What if someone has an existing
cluster, they're watching this,
and they want to go, and encrypt,
but it already exist.
>> Yeah. So, today if your cluster is not already BYOK,
you cannot convert it to
a BYOK because this is handled by
an algorithm that does it very
transparently during the cluster set up part of it.
So, the disks are already encrypted,
but with Microsoft managed keys,
so you can't interfere with that,
so you would have to start a new cluster.
>> Okay.
>> So, any new cluster that you start with,
you can set this up and you can see that once the cluster
gets created it has encryption,
the key URL, which you can rotate,
you can delete down the line if you want
to change the keys
or something and you want to add a new key instead.
You can do that with the rotated APIs.
>> So, you have a great deal of flexibility.
Like, if someone potentially quit
the company or some organizational changes,
you now have control about rotating your keys-
>> Exactly.
>> -and moving their life cycle.
>> Exactly. So, you have that control.
You don't have to change any of your producer or
consumer client applications that interact with Kafka.
>> Really.
>> So, they work all as before,
so they keep on sending data to
Kafka and reading data from Kafka.
It's only the Manage Disks aspect of it,
which the OS handles in the background and uses its key.
>> Fantastic and I don't have to
change anything and all my clients
just remain ignorant of this security change.
>> Yeah.
>> Very cool. Is this available today?
>> It's in public preview today.
>> All right. Fantastic. I am learning
all about how to bring my own key and enhance
security in Apache Kafka
with HDInsight today on Azure Friday.