>> You may know that Azure Data Factory supports
integration with Git repos and VSTS.
But did you know that Data Factory
now integrates with GitHub repos too?
Gaurav Malhotra is here to show us how it works.
Today on Azure Friday.
Hello. This is Lara Rubbelke with Azure Friday.
Today with me I've got Gaurav Malhotra
who is a senior program manager with Azure Data Factory,
and he's going to show some great new updates that we
have to integrate Azure Data Factory with GitHub.
So, what would you like to show us today.
>> So, last time when I was here,
we said that the entire UX
that we have with Data Factory,
the visual tools that we have which allow you to build
your data integration work flows in a good free manner,
can be backed with a VSTS create account.
>> Yes.
>> So essentially, you can do source control,
collaboration, and versioning using that Git integration.
So today, what we're seeing is that not only you can
use VSTS Git as a call repository,
but you can also use GitHub.
So, you can just select this,
and as you can see there what I
did was we have this dropdown.
So, right now I'm in
the Live Data Factory mode where you do changes,
and when you say Publish,
it goes directly to their Data Factory Service.
There's no report backing it.
So, if you want to look at audit trail,
as to who changed what, you cannot do that.
So, if I say "Setup Code Repository
from here," it opens up this drawer on the right.
Here, you can now select GitHub.
Once you select GitHub,
you can provide your GitHub account.
So, I'm just going to type my GitHub account here.
As you type the GitHub account,
restart pulling your repositories
in the GitHub account automatically.
So, you can see that I have
one repo in my GitHub account,
and once I select my repo,
I can now provide what is
the collaboration browser that I want to use so that
different variables who are developing in
the feature branch will use
this collaboration branch as the source of truth.
So, in this case, I'm going to select
"master" as my collaboration branch as well,
and keep the root folder as is.
With this checkbox, what we're seeing is that
as you can see in my current factory,
I have a bunch of pipelines
and data sets, links services,
connections that I've created,
and I will import all of that
in GitHub at this moment as well.
So, I can keep this check box as is, and here,
with this radio button we are saying that we
want to import all of
the code representation of
this Data Factory and do the collaboration branch,
which I just selected over here as master.
So, once I clicked "Save" at this point in time,
we're creating that connection to the GitHub account.
We are taking everything that is there in
your Data Factory and putting
the code representation into your repo.
So, this is my GitHub.
If I go in this particular repo now,
you'll see that all these folders have been created,
and inside each of these folder,
there is this code representation
of your Data Factory assets.
You saw that we had folders for data sets,
integration runtime,
link services, pipelines and triggers.
So, now if I do any change,
it would be reflected in this particular GitHub account.
>> Fantastic. So, can
see how this works as you make some changes?
>> Yes. So, let's do a very simple one where I
have this pipeline where
I'm getting data from a bunch of sources.
I'm prepping data in Notebook,
doing some transformation again in data lakes notebooks,
and loading my data warehouse.
So, here I'm just going to add
some description to my pipeline saying that,
"Pipeline to improve
online traffic sales using machine learning."
Let's just say that. So, once I said this,
you can now publish.
You can just say that I want to save this,
or save this to my master branch.
At this point, if I go to
my Data Factory and look at this Pipeline.json.
So, if I go to Pipelines and look
at the Improve Online Traffic Pipeline.json.
As you can see there was a 12 second update,
and you can see the description is right here.
So, as I'm saving,
those updates are going to
your GitHub account right there.
The other thing that you can do is that,
right now I was in the master branch,
but you could create new branches.
So, the typical way to help us work is that they
create feature branches to do their development.
Then they merge those changes
back to the master branches per request.
So, let's say I want to create a feature branch,
and I want to use my alias to create that feature branch.
So, at this point
that branch is are automatically created in your GitHub.
So, if you go to GitHub and you look at the branches,
if I go back here and look at my Git repo,
and if I look at the Branches,
I can see that there's
another branch that just got created.
Now, I'm in my feature branch.
So now, what I can do is that I can
essentially go to any other pipeline.
Let's go to this particular pipeline,
and I just had some description that I
dived in earlier that I'm just going to copy over here.
So, I'm just going to say it's a database pipeline to
clean data and classify records.
So now, let's just save this right.
Now, this particular change is going in
my feature branch, which is common.
Now, let's say if I want to dig
these changes and deploy to the Data Factory.
So, what we'll do in Data Factory is that we're
not trying to build Git inside Data Factory.
So, you're only allowed to
publish from the master branch,
which is the source of growth.
For you to take these changes through the master branch,
you need to create a pull request.
So, if you click this link,
we actually take you to GitHub.
So, this not the one.
If I say "OK" over here,
and if I say "Create Pull Request", so,
it takes you right to
that location and you can see that it
specifies that this is
the description that we have changed.
I could say "Create Pull Request",
and I created a pull request at this point
and I'm trying to merge it back automatically to master,
it didn't detect any conflicts,
and at this point you could open code reviews and have
other developers code review your code as well.
I'm going to say Merge the pull request.
So, at this point,
this change is now in the master.
So, if I go here and
change the branch to the master branch,
and look at this particular pipeline,
I would see the description of data over there as well.
>> Great.
>> So, you can see now the description is in for
this particular pipeline is there in
the master branch as well.
Now, I can publish these changes to my Data Factory.
>> That's great. So, you can really set up
some good collaboration across
developers working with Azure Data Factory,
creating their own branches, merging,
doing pulls and merges into the master.
>> Yes.
>> Fantastic.
>> Yes. Right now we are on public GitHub,
the one that I'm showcasing.
We're also working to
release enterprise GitHub integration.
So, if you as a customer have an enterprise GitHub,
and you want to use every Data Factory,
you will be able to use that as well.
>> Okay.
>> Also for authentication,
you can use the auto authentication
to ratify your GitHub credentials,
and use the GitHub account of your choice.
>> Great. Now, where would we go to learn more?
How can we start playing with this on our own?
>> So, you will see that we have documentation
on our Azure Data Factory documentation where
you can go and look at all of
these things that I just talked
about in terms of data integration
with Data Factory and get started with that.
>> Thank you so much. This is great to see.
I think the solid integration
that we have with GitHub is going to be
a great help for the developers that are
building out Cloud Data Applications.
>> Yes.
>> So, thank you very much.
>> Thank you.
>> Today on Azure Friday we've been learning all about
Azure Data Factory and the GitHub integration.