>> Hey, friends. You know it's very common to kick
off an Azure Data Factory pipeline on a schedule,
but what if life doesn't happen on a schedule?
Well, Azure Data Factory now work with
Event Grid to kick off a pipeline when
a file is added to Blob storage and Gaurav Malhotra
is here to show me how it works today on Azure Friday.
Hey, friends I'm Scott Hansen and it's
another episode of Azure Friday,
and today, I'm here with Gaurav Maholtra.
We're talking about Azure Data Factory
and some new stuff.
There's support for event-based
triggers in Azure Data Factory.
>> Absolutely. So, event-driven architecture is
a common data integration packing that allows production,
consumption, and reaction to events.
What we are saying is that with Data Factory,
event-based data integration is fully supported,
and a typical use case to think
of this scenario is that today,
customers have their data integration pipelines
which are running on a schedule.
But some other times,
you do not know whether
your upstream processes are going to produce
your data at a particular schedule
and you want that once the data has produced,
then only your pipelines are kicked off.
So, in that particular case, from now on,
customers can use an event triggering Data Factory,
if you will, to kick off the pipelines.
The first event that we're supporting
is a fine landing in Azure Blob.
>> That makes sense because that
seems like a very reasonable thing.
You might have some other process,
you don't know where it is running upstream,
it could be from IOT devices,
from an event string that's coming from elsewhere,
and does that event get triggered when the Blob
lands or when the Blob finishes its last byte?
>> So, we support different events
where they're Blob-created, Blob-deleted,
and we have a bunch of patterns that we support as well
in terms of if you want to specify your container name,
if you want to specify a particular file
inside that container,
or a particular path,
or have file calculators,
so all of those patterns are
supported and I'm going to show
that to you in an interesting demo.
>> Let's do it. I love interesting demos.
>> All right. So, what you're seeing on your screen is
a Data Factory that I've created and from this,
I've launched the visual tools that we have.
In this, I have a pipeline,
a really simple pipeline that I'm trying to copy
from Azure Blob to SQL DW.
This particular pipeline has
a copy activity which has
a source and in this particular case,
the source is an Azure Blob,
as we can see over here,
it's a Blob storage.
We can see that I've defined the connection,
so I know what storage you're going to
look at and we can look at
these connections or we can test it and all of that.
You can see that in this particular file part,
I have parameterized it and I'm passing
those parameters as values
when this particular pipeline drowns an.
These values are specified by the trigger,
the event-based trigger that we have defined,
and I'll show that to you in a moment.
Then for the sync side,
as I mentioned that we are moving
into SQL Data Warehouse,
so I just have a SQL Date Warehouse data set
and I am coping
some customer information from
Azure Blob to SQL DW, all right?
So, if I go back to the pipeline here,
now you can see that if I clicked "Trigger" and say
"New," I have a Blob to
SQL DW trigger that I've already defined,
and this particular trigger
is an event-based trigger that I have
and you can see that I've specified
that my Blob path begins with source,
which means that in
my storage account which is called as copysource,
if I have a container named source in
any file that drops in that particular container,
I want this pipeline to be kicked off.
As I mentioned that we support
different expressions you can
specify for "Begins with" and "Ends with," and we
have a whole list of that in our documentation.
So, as you can see in this particular case,
I'm just specifying the container name but I could have
specified a particular folder inside that container,
I could have specified
a file inside a particular container,
and we can do it for both
the Blob path begins with or ends with,
and this is all
supported and you just have to specify through here.
As I mentioned in the beginning,
you can kick off a pipeline based
on a Blob creation or on a Blob deletion,
and in this particular case I want to kick off
a pipeline whenever a Blob is created.
So, I specified this trigger.
If I say "Next," as you saw in my data set,
I had defined two variables for the path and filename.
The event-based trigger supports these two variables.
So, whenever it runs,
it can pass your folder path and filename
for the Blob location to your variables,
as we are doing over here.
As you can see that, I am assigning
that from the trigger,
and in this particular case,
this trigger is activated already.
So, if I go in my storage account and I
have this people.csv file,
so if I copy it in my source container,
you will see you were there once we go back,
and if we go to the monitoring tab,
that once the monitoring tab opens up,
you can see that there is a pipeline run in progress.
So, as soon as the file dropped,
this pipeline kicked off and now,
you can imagine how customers can do
that for their event-driven pipelines.
>> Now, that file was only one K. If the file was maybe
a gigabyte or this huge file as sometimes files are,
I assume that it is only when that file is
finished uploading that it then triggers the event.
>> Yes, absolutely.
So, you can do that and it will
only trigger off once that has
finished loading there. Yes.
>> This can handle files of any size.
This is one of the great things about Data Factory
is that, again,
one K or one gig or terabytes,
like this thing is legit
and deals with huge amounts of data.
>> Yes and this is based on
our integration with Event Grid in the background,
so what is happening is that
when you create this event-based trigger,
we create an Event Grid
subscription as your subscription.
We have tried to make it seamless
for our customers, right?
So, in Data Factory,
they just call me and say,
''This is my storage account and this is the Blob.
This is the Blob location.
This is the container or filename that I want to
use to trigger event-based
pipeline,'' and behind the scenes,
we automatically have
a full spot integration with Event Grid.
We create that Event Grid subscription.
We subscribe to those events,
and do it seamlessly for our customers.
>> How does it deal with concurrency?
I talk about a one K file or a one gig file,
what if I drop 10 one gig files
all within a short period of time.
Do they all go in
line and it's synchronous or is it done asynchronously?
>> So, it all depends on your configuration.
So, if you specify a particular container
and you drop 10 files in it and once they complete,
it's going to kick off.
The pipeline is going to kick off.
If you specify a particular file
inside that particular container,
then it's going to just kick off
when that particular fire drops in.
So, it totally depends on the configuration that you have
specified for your Data Factory pipeline trigger.
>> So, it really is up to your configuration
and also what your needs are,
what your preferences are and how you want it to behave.
>> Yes, absolutely. So, as you can see,
that run is in progress,
but I already had around that finish and you can see
that this run completed successfully.
It was a very small file that I copied
and you can see all the stats over here,
and we can see that it succeeded completely
and even if you go to your "Trigger Runs" plane,
so in our monitoring,
we support trigger runs,
you can see that this trigger was
fired just now and I did the run yesterday as well,
and you can see that the link to the pipeline that
ran for it and you can click it
to see that execution as well.
>> So, people are already using this now.
You've got customers that are already
excited about this and making this happen.
This is in GA or this is in preview?
>> So, Data Factory as a service is a GS service now
and this particular feature is also a GA feature,
and we have a block post that
went out that has full information on how
this integration works and we also have
full documentation on
our Azure Data Factory documentation page.
If you go to "How-to
Guides" and then there is "Create Triggers."
There is an event-based trigger documentation that
specifies all the information that I talked about today.
>> This might be a little bit of an obscure question,
but I'm thinking about
all the different ways that data can
ingress and all the different things
that can look at that.
Can I have other things in Azure,
looking at the same storage account?
So, I've got something going in Data Factory,
maybe I have an Azure function or someone else that's
also watching a trigger on the same object?
Maybe I drop in a CSV and some
JPEGS and they're all in the same storage account,
I can kick off a whole series of events?
>> Absolutely. So, you can essentially
build an Azure function and call
it using our web activity that we
have in our control flow panel.
So, you can call that Azure function
and it can do whatever you just described.
You could kick off another process,
a third party process on top of that,
so it is totally possible.
Right now, we are supporting Azure Blob as
a source with the file dropping only in Azure Blob.
As Event Grid keeps on supporting more sources,
all those events will also be
enabled for this event-based trigger in Data Factory.
>> Fantastic. So, you would recommend that
people go to that blog post to learn more?
>> Yes, absolutely.
So, they can read all about this,
they can try just right now if they want,
and just put that in production for their pipelines.
>> Fantastic. Everyone can go
and get an event-based data,
event data architecture,
and use that data integration plan pattern.
So, I'm learning all about
event-based data integration with
Azure Data Factory today on Azure Friday.