>> Hey, friends. Azure Data Factory enables you to do
hybrid data movement using
a self-hosted integration runtime,
but now you can share
one self-hosting integration runtime
across all your data factories,
which improves manageability and it reuses
that same highly available and scalable infrastructure.
Gaurav Malholra is here to show us how it works today on
Azure Friday. Hey, folks.
I'm Scott Hanselman.
It's another episode of Azure Friday.
I'm here with Gaurav, and we're going to talk a
little bit more about Data Factory.
But you're going to teach me about
something I had no idea about,
this idea of the integration runtime.
>> Yes. So, we have this piece of software called
self-hosted integration runtime that you
can use to do hybrid data movement.
So, if you have data,
let say non-prem SQL Server, Teradata, Sybase, DB2,
you can use this piece of software
that comes with Data Factory and needs to
be installed on a machine
that has direct line of sight to
your on-prem environment and
Azure and allows you to do hybrid data movement.
What was there before
the release that we are talking about today is that
if you had let's say 10 data factories in
your organization and all those data factories
were doing some sort of hybrid data movement,
meaning from on-prem to Azure,
then they need to have
one self-hosted integration runtime for Data Factory.
>> And you can maintain that, be conscious of it,
keep it updated, make sure it's healthy.
>> So, you can imagine that for
an IT admin it's kind of an overhead
because now they need to make sure
that it is installed in a machine,
the machines are up and running
because the actual data movement
happens via the machine
where this piece of software is installed.
With this release, we are saying that now you can share
the self-hosted integration runtime
across data factories.
Again, it will be, like you asked the question already,
it's quite beneficial for the IT admin from
a manageability standpoint and
the infrastructure works according to this.
So, if I have a visual,
so let's say you are in
an Azure Active Directory tenant and you are
Contoso AAD and you have
two data factories in your tenant now,
the Contoso_IT_DataFactory has a self-hosted IR
that you actually want to share across data factories.
So, what we have introduce is what we
call as a linked self-hosted IR.
So, in the second data factory,
which is the BI_Project_DataFactory,
you would create a linked self-hosted IR,
which would refer to the IR that was
created in this Contoso_IT_DataFactory.
Not only that, the IT admin has full control on who can
refer this particular self-hosted integration runtime
in other data factories in their tenant.
So, manually has to grant access to
the other data factory so they can
refer to this self-hosted integration runtime,
and I'm going to show that with your demo right now.
>> Please.
>> So, here, what I have is a contoso-it.
This Azure portal, I have a Contoso_IT_DataFactory.
You can see over here
that the visual tools that data factory provides,
I've clicked the "Connections" tab over here
and gone to the tab for "Integration Runtime".
You can see that I already have
self-hosted integration runtime that we
use for hybrid data movement created over here.
If I click "Edit" over here,
you will see that now we have a new tab,
which is called "Sharing".
In this "Sharing" tab,
you can see that all the data factories with
which the self-hosted integration runtime is shared,
and now you can grant permission
to any other data factory as well.
So, let's go ahead and
search for the data factory
that we want to grant access to,
in this case, it's called contoso-bi-project3.
So, I'm going to say yes,
I want that factory to be able to
refer to the self-hosted integration runtime,
so we have given that access.
Now for that factory to use this,
they need to use
this resource ID that we have there in the "Sharing" tab.
So, I have the corresponding data factory opened up here,
and now if I go in the integration runtime
again and click "New" and I want to do data movement,
so I'll select that.
So, as I mentioned,
you can create a linked self-hosted IR.
So, I'll click this "Linked Self-Hosted" IR over here,
and I can just say ContosoLinkedIR,
and here I'll paste the resource ID that I just
grabbed from
the original self-hosted integration runtime,
and I will say "Finish".
So, once I create this,
now you can see the connection has already been
created and it was lightning fast.
You can see over here in the "Edit" tab that I
have this integration runtime
referring to the original
self-hosted integration runtime with
all the nodes information
and the concurrent jobs that you can run.
>> These runtimes, it's not
a single service on a single machine.
It can be as large as it needs to
be to move as much data as it needs to be, right?
>> Yeah. So, that's a great point.
>> It's almost an integration runtime farm?
>> Yes. So, you could create a network of
integration runtime up till four nodes,
and we distribute the jobs in
an active-active topology in a round robin fashion.
So, not a single point of failure,
so if one node in your farm were to go down,
the other node would pick up the load.
>> Interesting.
>> So now, if I go in the particular pipeline over here,
I've created a pipeline called Hybrid data pipeline,
and if I go in this pipeline and I'm
trying to move data from a FileShare to a Blob,
so if I click "Edit" and I go to
the "Connection" tab and I click "Edit" over here,
so instead of using the default integration runtime,
I can come here and say that use
the linked IR that I've created to do the data movement.
So, if I do that, I need to
provide the password for that.
So, let's provide the password.
I'm going to do a test connection just
to make sure that it works.
>> Do I have to think about scale might I push too
much through an integration runtime
and it would tell me that "Hey, slow down."
or how do I know?
Because it sounds like a really powerful thing
to be able to reuse this,
but I wonder also if it could become a problem.
>> Yes. So, we have guidance published on.
So, as you can imagine that this is
a particular server on which you are
installing the self-hosted integration runtime.
So, if you're trying to let say move
a terabyte worth of data in that server on which are this
installed is just a single core or a
dual core machine with two gigabyte memory,
of course the throughput and
performance characteristics that you
would get would be very less.
So, we have guidance published in
our documentation on how much data you're
trying to move and what kind of
skew you should be using to do the data movement.
>> Okay.
>> All right.
>> Do you think that the thing that's
most interesting about this, though,
is that people can now reuse these nodes and they
may have slack or extra capacity?
>> Yes.
>> Because there's no reason to build out
a four-node system if you can't reuse it.
>> Yes. So, earlier the problem that we
heard from our customers and we got a lot of
feedback on this is even though you
have one integration runtime for Data Factory,
most other times our customers are doing nightly loads,
which means that server is up and
running for the remainder of their day and doing nothing.
Now they can the use of
single self-hosted integration runtime
and scatter that load.
In that way, they don't have to manage
multiple machines and they can
use the full power of that machine.
>> Fantastic.
>> So, you can see now, the connection
is successful over here.
I'm going to say "Update" this to use the linked IR.
Once this updates, we can go trigger a copy from
an on-prem FileShare to Azure Blob using this linked IR.
So, it still updating
the connection to make sure that everything.
>> Do these runtimes update
themselves or is it something
that I would update regularly?
Do the people who are already using
runtimes need to do anything to get this new capability?
They just need to upgrade to the latest version?
>> So, we have this "Auto update" capability
that you can set up.
Once you do this, we automatically
update your integration runtime
as we release new capabilities or add updates.
It's up to you, you can turn it
off and manually update it,
so total flexibility to our users.
So now, it has updated.
If I go to my pipeline,
I can actually first publish
these changes because I wanted to
use the new linked integration runtime.
Once these change is published,
I can just go ahead and
click a particular run to show you
that now how the data movement can
happen using this self-hosted IR.
>> And it's as fast as I can throw data,
as I can throw bandwidth at it,
it's as fast as I can throw
integration runtime CPU capacity at it?
>> Yes, exactly. Because the data movement
is actually happening on
the node where
the self-hosted integration runtime in installed.
>> Okay.
>> So, now I have triggered a data movement.
>> So, I've seen you move terabytes
around very, very, very quickly.
>> Yes. We're actually moving terabytes,
petabytes worth of data every month
for a bunch of our customers.
So, you can see that this is in progress,
I already had a run,
already happened for this.
So, you can see that in this particular case,
there was around about 659 megabyte worth of data that
was copied from a file system on-prem to
Azure Blob at 23 MB per second.
>> That's pretty fast.
>> Yes.
>> Fantastic. Then of
course the notification just pops up right there.
>> Yes. So, you can look at that,
you can essentially create
alerts on top of these pipelines,
as you can see over here,
and get notified on when that completion happens.
>> Well, it's such a much better use of resources.
>> Yes.
>> This means that people can then run everything through
one IR and then
repurpose the other machines for other stuff.
>> So, reusability is the key here,
and that's the whole point on the feedback that
we got from our customers and we enable this feature.
>> Okay. So, tell me again how I can get
this if I'm already thinking
about IR or already doing this.
>> So we have a blog post that's
out that you can use to understand
step-by-step on how you can share
the self-hosted integration runtime
across data factories,
and this blog post will provide you information and
also link you to their documentation,
which is there in our Azure documentation where
we mentioned step-by-step on how you can
enable this in your data factories.
>> Fantastic. Thank you very much for sharing.
>> Thank you.
>> I'm learning all about how to more
effectively share the integration runtime
and make hybrid data movement possible with
the Azure Data Factory today on Azure Friday.