>> Hey, friends.
Real-world data integration pipelines are complex.
They can require days of effort to put together.
Typically, multiple data engineer's
collaborate to build these integration workflows.
Gaurav Malhotra is here to show us how Git integration in
Azure Data Factory simplifies
this collaboration experience today on Azure Friday.
[MUSIC]
>> Hey, friends. It's Azure Friday.
We're digging into the Azure
Data Factory and this one is exciting to
me because you're have Git integration into Azure Data Factory.
>> Yes. So we have
Git integration for customers to do collaboration,
source control, versioning, and we've
made it even better with Data Factory right.
So today with Data Factory you can essentially come and say
that I want to back my entire experience with a code repo,
so that I can get all of these benefits.
It's available for you to do that in few simple clicks.
So you can come to the Data Factory visual tools
and click "Setup code
repo" and now we have options
for both DevOps Git as well as GitHub.
With GitHub we support
both public GitHub as well as the enterprise
GitHub which is the on-prem version of GitHub with that as well.
So for this particular demo,
I'm going to use Azure DevOps Git.
So if I select Azure DevOps Git,
we're again doing all the heavy lifting to make
it super simple for customers to set up this.
So we automatically pull up
all the accounts that you have access to.
So you can just select one of the accounts that you can do,
and we automatically pull the projects.
Here you can create a new repo or use an existing repo.
>> Okay.
>> You can select the collaboration branch.
So generally, this collaboration branch is master but if you
want to use any other branch as collaboration,
you could do that as well by clicking "Create New" over here.
>> Okay.
>> This check box says that I want to import
all my current stuff which is in
the Data Factory because I could have build pipelines.
>> Right.
>> Without any blinkered integration into the Git repo as well.
>> Okay.
>> So if I go press "Save",
at this point we're creating all the connections
to make sure that you're Git integration is enabled.
If I go to the Git repo now and refresh this,
you'll be able to see that all the code representation
of your Data Factory as we saw is here now.
>> What is the format, is it JSON or?
>> Yes. So if I were to click this,
you can see that it's all JSON over here.
>> Look at that.
>> Yes.
>> You serialized, effectively,
the entire Data Factory and all resources associated with it,
in a way that can be exported and imported.
But I assume other people can merge
these and collaborate on them in an approved Git fashion.
>> Yes. So now, if I go over here your Git repo is enabled.
If I go in my altering canvas,
you can see that the Git is there,
and I'm working on the master branch.
But you as a customer could be developing different features.
Different Developers in your organization
could be developing different features,
so they could be enabling new branches
which could be different feature branches and working off of it.
Once you're done with building features,
you're going to merge that back to the master.
From Data Factory we only
allow you to publish from the master branch.
Now, one thing that I want to specify is that within Data Factory,
we're not building Git inside Data Factory.
So we are, essentially,
for all the features, for example,
to do code reviews and merging it back to the master,
we have links where we take you to the Git repo.
So what I did over here was.
>> I see you jumped over into the either GitHub or Azure DevOps,
that way you understand that Azure Data Factory is using Git,
but you're not reproducing any repo.
You use the same flow that you're used to using?
>> Exactly. So I just click "Create pull request"
over here and it took you to DevOps Git,
and if you had anything over here you can merge it back to master,
open code reviews and all of that
and work with Data Factory like that.
>> So this is yet another example where
the Azure Data Factory goes to where the users are,
works in the way that they want to use.
Your being helpful, but you're not being
overly prescriptive and saying you must do it like this.
>> Yes. The other thing that this
grants you is that now you could do partial saves.
So we have two modes.
The live mode what we call as,
if you look over here that you would, essentially,
go into Data Factory mode on the live mode,
or it could be in the Git mode.
We generally don't recommend customers to
keep switching between those two because.
>> Pick one.
>> Yeah pick one. It could create complications or in terms
of where your Data Factory assets are.
So if you are in Git mode over here,
you can see that, now, there's a save button enabled.
So if I have anything over here,
I could, essentially, do partial saves.
Now, with the Data Factory mode,
this is not there because there you need to have
a valid pipeline for you do published with a Data Factory.
Generally, in the data integration world
your pipelines are complex,
you take multiple hours to build.
So you want do intermediate saves,
because you don't want to lose your work.
So with Git integration that gets enabled for you.
In addition to this your performance is faster,
because now we are loading from
the local Git repo instead of the direct Data Factory service.
So you'll see a faster performance for your workflows.
Also, it benefits for you
to do continuous integration and deployment,
because a lot of times customers have different environments.
They have a Dev environment,
Test environment, a Prod environment,
and they want to take whatever is deployed and
the Dev to the Test and then to Prod.
So with this integration,
you can enable that CICD.
So here, I have a graph which, essentially,
shows a visual which,
essentially, shows you how you can do that.
So here, you have your different working branches.
So this is, essentially, your Git and you
have your different working branches.
You create your pull request and you merge it to the master.
From there you click "Publish" in your Dev Data Factory.
The moment you click "Publish" we
automatically generate an adf_published branch in
your Git repo and you have
Azure DevOps Git release hooked up to that.
>>Right.
>> The first thing in that Git release is
a trigger and which gets automatically fired once a
check-in happens in the adf_publish branch and then it takes
everything that's in your Data Factory to a Test
and then to a Prod.
Now, because Azure DevOps release is a workflow,
you could have intermediate steps would do
best before you push from one environment to another.
So you could be caught using
our PowerShell cmdlets to do certain test.
>> You need quality Gits.
>> Exactly. So this entire continuous integration
and delivery pattern and gets
enable once you enable
Azure DevOps Git or GitHub integration with Data Factory.
>> Okay so it may not be initially familiar,
but it is a flow.
It's just like we use Git Flow for
our websites and Dev Test and staging.
If you're familiar with GIt this
will feel very comfortable I should
work nicely to integrate with
someone's existing Data Factory process.
>> Exactly. We continue to make
this more performant for the users,
because you can imagine you could have thousands of pipelines.
>> In deed.
>> Which you would be pushing from one environment to another.
So we have done all the hard work to split the ARM templates,
because behind the scenes we, essentially, use ARM templates.
So you've done the work to split
those ARM templates to make sure that you get the
best and class performance and throughput with Data Factory.
>> Very cool. This is available now,
and people can make this happen
with their existing data factories?
>> Yes. This is available now.
We have documentation available.
So if you go to our documentation I'm actually
looking at continuous integration and delivery.
>> Lovely.
>> There's visually authored
data factories as well in which you can see how it can
enable the Azure DevOps Git integration
or the GitHub integration with Data Factory.
>> very cool. Well this is fantastic.
I'm learning all about how to build data integration pipelines,
collaboratively, using Git integration in
Azure Data Factory today, on Azure Friday.
[MUSIC]