>> Hey folks, you know Azure Functions
is a server-less computing service that
enables you to run code on-demand
without having to deal with infrastructure.
Well, Azure Functions is now integrated with Azure Data Factory,
which enables you to run an Azure Function as
a step in your Data Factory pipelines.
Gaurav is here to show us how that works today on Azure Friday.
[MUSIC]
>> Hey, I'm Scott and it's another episode of Azure Friday.
I'm here with Gaurav Malhotra and we're
talking about Azure Data Factory.
There've been a bunch and great videos with you.
We've got a whole host of information about
Azure Data Factory and now you're
integrating it with Azure Functions.
>> Absolutely. So, we got a lot of
feedback from our customers where they wanted to use
Azure Function as a step in
their data integration workflows because with
Azure Functions you're able to execute
those small pieces of code in a server-less fashion.
A lot of times customers had requirements where they, let's say,
wanted to archive some data after they copied
the data using the "Copy Activity' in Data Factory.
Now, we will support those actions in due course of time,
but we wanted them to get unblocked
and use Azure Functions to write that piece of code.
So with this integration,
now you can use Azure Function as
a step in your data integration workflows.
>> That's a really nice elegant solution.
I could see where the simplistic way to
do it would be to allow them to do a small script.
>> Yes.
>> But it's not going to scale the way it needs to scale
for large data factories?
>> Yes. With Azure Function, it's very simple.
All it is is that on success,
on failure as you know that we did
a factory support presence constraints.
So you could have a copy activity.
So, let's say if I were to show you on my screen you could have
a copy activity and after
copy you want to say that I want to archive the data.
You could be using an Azure Function to do that, right?
It is pretty straight forward for
you to set up this Azure function.
It's live in our general section.
So it's just a simple drag and drop.
So after you drag your activity onto the canvas,
all you have to provide are two things.
You have to provide the function app URL.
So you have to set the Data Factory-
"Azure Function linked service"
in our Data Factory, which is nothing,
but the "Function app URL as well
as the "Function Key" and once you set that up,
all you need to do is give us your "Function Name, right?
We will call that function name
as a step in your data factory workflow.
So what I have right now is a very simple
one where I have Azure Function.
So let me actually show you.
I have an Azure Function which is just a HelloWorld example.
So this is in "My Subscription" and you can
go to your" Manage" portion to look at
the keys for this particular function
app so you can add a new "Function Key" or you
could go look at the function URL over here.
So, you can grab the key and the function URL,
paste it into your linked Service as I've already done over here.
You can see that I'm calling that
"Function app URL" and the "Function
Key" and then I'm just providing
the name of that Azure Function that I'm using.
So, it's called HelloWorldSample and you can see
that I'm calling that as "HelloWorldSample" over here.
And all this function does is that it just I
pass it a string and it
returns that string, it prints that string.
>> Sure.
>> So, you can see in the "Body" section, I'm saying,
"Azure Function Call Success" and I call that function.
So if I were to say just let's run it right now.
It's right now initiating
a Debug run for this particular function call.
You can see that the run is in progress right
now and it has already succeeded.
So it just returned back the string that we just published.
But this is a very simple example,
but you can imagine now if you have
complex functions that you have
written in Azure Function you could just
call that as a step and in your workflows.
>> Let me see if I can get my head around some of
this as it lives in Azure.
Azure Data Factory is kind of
server-less in that I'm not thinking about virtual machines,
but it does have a location, right?
It lives in a location.
What kinds of considerations do I have to have?
Should I make sure that my function and my factories are in
the same co-location and also how much data
am I flowing outside of
my- am I sending a gigabyte into a function,
how do I make that it's scalable?
>> Yes. So right,
to answer your first question,
it's just like anything,
you could use any computer,
any storage in any location with Data Factory.
It's same with Azure Functions so
that your Azure function could be in
any location or any subscription.
It could be in East, West,
all the regions where it is supported and you can call it
inside your Data Factory which
could mean any other region as well.
So there's no limitation on it.
>> But to save money on Ingress I would kind of
avoid sending large amounts of data outside one location, right?
>> Yes. So there's definitely
certain best practices on how you should do it,
but Data Factory as
a platform does not limit you from doing anything [inaudible]
>> Okay. So, that's great. So if I have
existing functions I want to call, that's great.
A best practice sounds like staying in the same location.
It seems like a good idea.
>> Yes.
>> Then am I sending just small pieces of data multiple times
so a lot of calls with a little bit
of data or one big call with a lot of data?
>> So with Data Factory it
essentially depends on what
exactly you're trying to accomplish, right?
It could be small chunks of data or you could be using
storage as your- like
Azure storage as your underneath storage layer,
you store your data and your function could directly, therefore,
do that as your storage because we don't
recommend sending too much data over the wire.
So, generally, the patent suggest to the customer is that you
persist your data to some sort of a storage
which could be Azure Data Lake Store or Azure Blob Storage,
it's totally up to you and your function in reference to that.
>> So, put it in storage and then send
the key and then it'll go and pick it out wherever.
So general data movement best practices should apply?
>> Yes.
>> It also, I assume, depends on how hard your function's working.
If it's a simple thumbs up,
thumbs down or if it's actually processing and doing
some AI or something smart and
calling other functions it could take a while.
>> Yes. Exactly. With Data Factory
the only requirement for you is
that when you call an Azure function,
and this is already published in our documentation there,
it has- the return of
that Azure function has to be a valid JObject.
>> I see.
>> So it's a requirement from our Data Factory standpoint
and not only we support just a direct function up call,
but if you are using routing or you're
using queries in your Azure Function we support that as well.
So, in this particular case if you look
in the documentation you are essentially saying I
want to call this HttpTrigger Azure Function
with name is equal to hello
as one of the queries that I want to parse.
So you could do that as well.
So all of those things are
supported with this Azure function integration as well.
>> Very cool. Is this live now?
I mean, I have existing data factories,
can I start using this today?
>> Yes. So this is live now.
This is available in your toolbox,in
the Data Factory toolbox as I was showcasing when I was demoing
and there's already a blog live for this which talks about this
in detail and it links to
our documentation which has more details on what can be done,
what is the exact payload that it
expects for you to send and all of that information as well.
>> Can you run this live? I assume that if you do a demo
will the output appear in
monitoring like any other piece of output?
>> Yes. So the output would appear in monitoring.
So if I were to go back and say "Trigger Now" and,
so it's going to run this particular function.
If I go to my monitoring we'll be
able to see this run in progress.
So you can see there, it's already succeeded.
>> Me happened immediately.
>> Yes.
>> That's fantastic.
>> Because this was a very simple function that we were calling.
>> Very cool. So people can integrate this now.
This can improve their existing data factories.
It's just another piece in the puzzle that they can add in.
>> Yes.
>> Fantastic. I'm learning all about how to integrate
Azure Functions with Azure Data Factory pipelines
today on Azure Friday.
[MUSIC]