>> Hey friends, you know data integration
can get pretty complex with lots of
moving parts and it may include
upstream and downstream dependencies.
This gets really interesting when
you're setting up job scheduling.
Well, now you can create
dependent pipelines in your Azure Data Factories.
Gaurav is here to show us how that will make your life easier,
today on Azure Friday.
[MUSIC].
>> Hey, I'm Scott and it's another episode of Azure Friday.
I'm here with Gaurav and we're going to talk about creating
dependent pipelines in your Azure Data Factory.
This is a new thing.
>> Yes. So, as I've always said
that the data integration projects can get overly complex
and a big part of data integration is scheduling
your data integration projects to run on a Cadance,
on an event, and a lot of times
customers have dependencies upstream and downstream.
So, that makes dependency an important
part of scheduling in your Data Factory pipelines.
So, what we are saying now with Data Factory is that you can
create triggers and make
those triggers dependent on other triggers.
So, for example, you could have
a daily trigger that's essentially processing your logs,
and then you have
some insights that you are getting on to the logs.
Doing some transformation which you only want to do weekly.
So, you can make the weekly trigger
dependent on their daily trigger so
that only when the daily jobs
for let's say seven days are complete,
then only the weekly trigger would run to accomplish its job.
>> I see. So, I'm going to guess that there
were some customers out there that would have
something run at eight o'clock and then something else
run at midnight and they're dependent,
but there's nothing in the system
that knows that they're dependent.
They're just hoping that the eight o'clock job will be done.
>> Exactly.
>> The 12 o'clock job will pick it up.
>> Yes. So, they will right now just space it
out and assume that it would complete so
that the next job when it runs would essentially the data formula
previous job is there for it to pick it up.
But now we are making that dependency available in
the system so that they can create that hard dependency
that only when it is complete then it runs.
Because you can imagine that they could be
a bunch of failures or things
that happen upstream that you are unaware
of that could cause delays,
that could cause failures,
and you don't want your downstream processes to run.
>> Well, if I didn't have this as
a native construct in Azure Data Factories I would have to write
a custom statement at the beginning of
my downstream Data Factory
that checks to see if it succeeded or not.
>> Yes.
>> This should be built-in.
>> Yes. So, lot of times customers are
actually using constructs that
are available in Data Factory like if,
else, and all of that to continuously-.
>> To simulate.
>> To simulate that feature and we are now providing
that as a first-class integration in Data Factory.
So, just to give you a couple of examples of what it
means so that it is clear to the customer.
So, you can see with this I have this
nice visuals in their documentation,
where you can see there you have
a trigger A that is dependent on trigger B.
So, the 10 to 11 time frame
of that trigger A run depends on that trigger B run.
So, only when it is complete then only the trigger A would fire.
Similarly, over here.
So, you could have straight dependencies to say that
10 to 11 dependencies offer trigger
depends on the 10 to 11 dependency of another trigger,
but you could also have an offset.
Offset is nothing but a timestamp which can
shift your Window start and end.
So, here in the second example you can see that I want
the trigger A which runs at 10 to 11 dependent on trigger B,
but the nine to 10 Window of that trigger B.
So, you can use this offset parameter,
set it to minus one,
and make that dependency in your Data Factory pipelines as well.
>> This is allowing you to have full control over when
things happen in a deterministic fashion.
>> Exactly, and not only that in some cases
customers come back and say that I want the Window size as well.
So, not only I want to make it
dependent at 10 to 11 or nine to 10,
but I want that Window size to be two hours.
So, I want the 10 to 11 to be depending
on nine to 11 of the trigger B.
So, you can see in this particular example
that I'm highlighting right now,
my offset is minus one.
So, I want the previous hour,
but my Window size needs to be two hours.
So, lot of customers have
these complex dependencies that
they want to create in the pipelines and as
you pointed out earlier that they were writing a lot of code in
order to accomplish it and now we are giving
that as a first-class integration in Data Factory.
>> That's so powerful because when you say a lot of code,
they could be creating more code to manage
dependencies than we're actually doing the Data Factory work.
>> Exactly.
>> Trying to use the Azure Data Factory as
a fully blown workflow
management system with dependency management.
It's just not their job.
It should be Hilton and now it is.
>> Yes. That's why we're doing the heavy lifting
for them now and it is so easy.
So, if you go to your Data Factory
and you go to your triggers section,
all you need to do is say "New."
Let's say I want to call this as lets
say "Daily log processing trigger."
>> What's great about this is that
it's refactoring via subtraction.
That means that someone's going to be able to go and take
their Data Factory and suddenly make it simpler.
>> This is only available for
tumbling Window type triggers because these are
all when doing dependencies that we are creating.
So, in this case I'm just creating
a trigger that runs let's say every 15 minutes.
We can do it every 60 minutes to say every hour as your want.
>> Sure.
>> So, let's say every 60 minutes that's
an hour and then we can create another trigger.
Let's call it daily processing trigger.
This is going to run our processing pipelines.
Again, let's make this 60 Minutes as
well and go in the advance section and
say that I want to link this dependent
on the daily log processing trigger.
Here I can specify
the offset and the Window size as well if I want.
>> So, let me ask what might be a dumb question,
but that's what I'm good at.
What if a log has an anomaly,
something goes wrong, it goes longer,
and my tumbling logs get to
the point where they start to overlap where it's
taken so long that it's messed up the schedule for everyone.
Do we skip one?
>> No. So, it will be queued up.
So, only when that will finish then the next one would
start and you could define concurrency as well to say that,
I want concurrency as one which means only when the previous one
has run and succeeded or failed,
whatever that status is,
then run the next Window.
You could set concurrency too as well,
which means that the first one
runs and when the second one time hits,
then it is actually supposed to be
run because we have said the concurrency could too.
We will start running as well.
>> So, there's no fear that I
could potentially get into a situation where I
have quite a backlog of
things that needs to happen and in that case,
I would want to scale and maybe pay
a little more money to clear out my backlog.
>> Exactly. So, it totally depends on
the scenario that you have and how you
have set up that particular scenario
with different parameters that we provide with Data Factory.
>> So, is this available now?
Is this something that I'm just going to notice?
The next time I log in it's already been in the live system.
>> Yes. So, we have a blog that's published for
this that talks about how you can
create these dependencies between
different triggers that you can create and
all the different techniques that I
talked about on making a Window dependent on the previous Window.
Not only the article to self dependencies as well,
you can make a trigger dependent on itself.
So, that's supported in the system as well.
Then we have documentation explaining about each of
these particular scenarios as well that you can go and use.
Not only you can create these dependencies,
but then you can also monitor these dependencies.
So, you can go in our monitoring tab and we have
this trigger run tab are
here and you can look at these dependencies.
So, you can say "View this" and you can see that this particular
trigger run depends on
their daily trigger and if you click the success status,
you can look at what exactly that trigger
run and what were the pipeline
runs that happened for that trigger.
So, all of that monitoring information
is presented to you as well.
>> Fantastic. The Azure Data Factory
continues to add great new features every month.
>> Yes.
>> All right. I'm learning all about how to
create dependent pipelines in
my Azure Data Factory today on Azure Friday.
[MUSIC]