>> Hey friends it's another episode of Azure Friday.
I'm here with Rob, and he's talking to
me about Azure Data Factory and
some of the news stuff around
Continous Integration and Deployment.
>> Yes. So with Data Factory now,
we have added support for customers
to do Continuous Integration and Deployment
and do the industry leading practices to essentially
achieve their goal of moving their pipelines,
data sets, link services from one environment to another.
So generally, how it works for our customers
is that they have a Dev Data Factory,
they have a Test Data Factory,
they have a Prod Data Factory.
And developers are essentially
developing in their Dev Data Factory and
once they are done with development they want to automate
the process of moving from one environment to the other.
So with this integration that we
have done with Data Factory, with ARM templates,
and integration with VSTS Git you
can automate this entire process and do it.
>> Has this traditionally
been done in a fairly manual way?
>> Yes. This has always been manual.
Our customers would write
their custom scripts to achieve this process.
>> And no disrespect to data scientists or the community,
but is there the same process rigor
that we have in
the software engineering community in
the sense of checking my code,
I run my unit tests,
I do it in dev, I do it in test, I do it in staging.
There is a standard pipeline
for Continuous Integration and Deployment.
Does that exist in this community?
>> Actually no, and this is what
we are trying to do and we're trying to
automate that process into
end so that once they essentially
check-in in their dev environment
and they are comfortable in their test pass,
they can define their releases to
dig their code from one environment to another.
>> Sounds like a lot of what I call the administrivia.
>> Yes.
>> It is removed from you and put into this pipeline.
>> Yes.
>> So you're already creating
a data pipeline and we saw
how to do that with data bricks.
You're also now creating an operations pipeline
for the larger environment.
>> Yes, exactly. So we're allowing them to operationalize
their pipelines across environments
and how they can automate that process into it.
>> Cool. You have a demo.
>> Absolutely. So Data Factory Visual Tools are live
now and you can essentially
create your Data Factory
and click "Author and Monitor" tab.
Once you click your "Author and Monitor" tab and go to
your "Author" tab in your Visual Experience,
you can see your pipelines and we have
this new ability for you to
export your Data Factory as ARM template.
So you can click this button and
export your entire factory,
and this includes all the assets for your pipelines,
all assets for your data factory,
the pipelines and data sets,
the link services, their integration runtime,
the triggers, all exported as ARM template.
So once you export that as an ARM template,
we generate two files for you.
We generate the actual template
and we generate a parameters file.
So this template would be seen
across all environments and
you will have one parameter file for each environment.
So your dev environment,
your test environment and all these parameters
file contains is your connection strings,
your secrets, and all of that information.
So you can essentially now pick
this template and deploy it to
your UAT Data Factory and to
your Prod Data Factory and only thing
that you have to change is the configuration file.
So, you can again create processes to do it.
But what we've also done is that with
our Visual VSTS Git integration
we have defined a process on
how you can automate this end to end.
So right now I am in what we call S3 Data Factory Mode.
So if I do anything,
it goes and publishes
directly to my data factory servers.
But I can hook up a VSTS Git repo with my data factory,
so I can select the VSTS Git.
It automatically loads the tenants that I have access to,
I can select my Microsoft Tenant,
I can select the accounts that I have access to,
so I'm going to just pick out my account and it
automatically loads the projects
that are existing in your VSTS Git account.
So I'm going to select my Data Factory project and I can
create a new Git repo or I can use an existing one.
So in this case, I'm going to use
an existing one that exists and,
again, it is going to ask you that
which branch do you want to use for collaboration.
Most of the customers use Master as
the collaboration branch but it is totally
up to you if you want to use another branch.
And then you'll define your root folder and you check
this box to say that whatever
data exists in my data factory,
I want to import it to my VSTS Git account as well.
And all of these assets will be imported
as JSONs in your VSTS Git account.
>> That was my next question.
So, what we're putting into Git,
just to be clear if there's
any confusion for people who are watching,
it's the textual representation of the factory.
It's not certainly terabytes of data,
it's the description of the factory,
not the contents of the data.
>> Yes, it's the metadata. As we call it.
>> The metadata. Exactly. Just for people who aren't
familiar, the ARM templates,
or Azure Resource Manager,
is that ability to basically now I can say,
"File", "New My Data Factory".
>> Yes, exactly.
>> And dev and test and staging prod.
>> Yes, and we actually link you from this UX as well.
So you saw that I exported as an ARM template,
and you can click this import
ARM template link and all we are
doing is we are redirecting you to Azure Portal,
where you can import your template.
So you can select that exported template,
provide your configurations,
and just import that data factory
into whichever environment you want to do that. Right so.
>> When changes happen to the template in source control,
does that kick off a process and
then validate and rerun the factory?
>> Yes. So, right now when I
hooked up the VSTS Git account you can see that
a branch was created just now and in this branch all
your data sets integration runtime are defined as JSON.
So it's the text representation
of your entire Data Factory.
Now, what you can do while here is that you can define
or release and I've actually a
good visual for you to show this.
So you have your working branches in Git and
this generally we call that as feature branches.
You create your pull request
and push the stuff to master.
And once you're there in the master you
publish to your Dev Data Factory and,
once you publish to your Dev Data Factory,
we actually create ADF
publish branch in your VSTS Git report.
And once that happens,
you have defined a VSTS release
which has a trigger defined in it,
and the moment the ADF publish branch gets generated,
the trigger files and automatically takes
your entire text representation,
which is the JSONs for your pipeline triggers,
and take it from one environment to another.
So you can take it from dev to UAT,
from UAT to prod.
So that entire process can be automated by
defining VSTS release for it.
>> So you haven't just simply turned on Git integration,
you're actually coming with a
formalized process here almost
to Git flow specific to Azure Data Factory.
>> Yes and we have actually defined
this process end to end in our documentation.
We even wrote an Azure Blog for it.
So it's all published
today for our customers to take a look at.
We have the documentation,
we have the blog published,
which is listed step by step on how you can define
these releases and automate
this process end to end as well.
>> And would you recommend that anyone who is
an Azure Data Factory customer
today should definitely explore doing this?
>> Yes. Absolutely right.
Because Continuous Integration and
Deployment is part of
any enterprise scenario that you would look.
So, every customer has
different environments and they want to
do it for the purpose of source control,
for the purpose of access grant,
making sure that only certain people have
access to certain environments and it's
a very common practice to do
continuous integration and deployment
across their data pipelines.
>> So then people who are watching right
now who have a data factory, they can do this today?
>> Yes. They can absolutely do this today.
As I mentioned, the documentation is live.
The blog post is there in Azure Blog.
You can search for continuous integration and
deployment using Data Factory,
it will take you to this blog
and you can read all about it.
>> All right, fantastic.
I am learning all about
how to make continuous integration and
deployment a reality with
my Azure Data Factory here on Azure Friday.