>> Most data integration solutions have
a lot of moving pieces and can get very complex,
which also means it can fail for any number of reasons.
As such, being able to rerun failed activities or
reprocess updated data is a necessity.
Gaurav Malhotra is here to show us how you can rerun
an entire pipeline or start from an activity within the pipeline,
today on Azure Friday.
[MUSIC]
Hello. This is Lara Rubbelke,
and today on Azure Friday,
I have Gaurav Malhotra who is going to show
us some great new updates an Azure Data Factory,
to help you rerun activities within your pipeline.
>> Yes.
>> Talk to us about what you've built?
>> So, essentially, with data integration solutions,
failure is an important part of
any data integration process where your activities
inside your data integration work through or could failed
because an upstream system is not responding,
or data might arrive late,
or in certain situations,
they have run to completion but you find out that there's
a need to reprocess the data
because there's some data integrating issues.
So, you want to be able to
read on your activities inside the pipeline.
So, until today with Data Factory,
you could rerun the entire pipeline,
and a pipeline could be as
complex as if you see on your screen like
it could be a logical grouping of a number of activities.
Here, what we are saying now that instead of not only
supporting just the entire rerun of the pipeline,
you could also rerun an individual activity
inside the pipeline as well now.
>> That's really important.
As you call out, these pipelines can
take hours in some cases, be very complex,
and if you have to rerun the whole pipeline,
you've probably lost your SLA for that pipeline.
>> This is one of the most important feedback
that if we look at our feedback forums,
our customers are giving to the point
that for them to meet their SLAs,
they need to be able to run
individual activities in the pipeline to support such scenarios.
>> Great. Well, do you have something to show us?
>> Yes, absolutely. So, if I go to the Azure Data Factory portal,
and we are on the overview page.
So, if you go on to the "Monitoring" tab,
you'll see all your pipeline runs as we
have been showcasing already.
Now, you can see that as you dive into
your "Pipeline Run" and look
at the activity runs inside the pipeline,
not only you can see the activity runs but we have
also brought the diagram view to the monitoring interface.
>> So, you have good visibility and can see this?
>> Yes. So, you can see for that particular run what was
the pipeline definition you run with and what were
the activities inside that pipeline
then you actually did this run?
So, you could now
select the particular activity inside the pipeline,
and you could essentially rerun from here,
or you could rerun from the tab up above.
So, you could select this and say,
yes I want to rerun,
and at that point, what it is going to do is it
is going to start the rerun of the pipeline.
It is going to skip all of
the earlier activities because we
said that I want to run from here to downstream.
Once this starts, you'll be able to see that
the first activities was skipped as you see the state is skipped,
and it's also on the nodes under diagram view.
The activity that we started with
Dataprep is in progress right now.
>> Okay, great.
>> So, another thing that our customers came back and said
that we want to be able to rerun the pipeline runs,
and you can go back to your all
your pipeline ends again using the Breadcrumb View.
They mentioned that they want the ability to
see whatever pipelines they have rerun.
So, now, we have enabled "Rerun History" for you.
So, you can come in and say that I want to look at
the rerun history for all my pipeline grants that I have done.
In which case, once you enable this toggle,
you can see that I can see now
three pipeline runs for this particular pipeline or you
could give this uncheck density under the
"Action column" that for this particular pipeline run,
I want to see the rerun history,
and you click that "Action" and you'll be
able to see the entire rerun history.
So, you can see that the last quarter attempts failed, and again,
you can go inside and look at the actual failure,
and the entire all the logs and everything is available for you.
So, this is just a link to
your Databricks Workspace where you can look at the entire logs,
and you can just come back and look
at the entire rerun history over here as well.
>> Great. Now, you showed a manual rerun,
what if I wanted to automate that?
>> So, whatever we showcase in the UX today,
it's a mandate like with every Azure Service that it
is available in the programmatic interface as well.
So, if you wanted to do this through SDK,
Python SDK, Darknet SDK, ARM Template,
REST APIs, PowerShell cmdlets with
all those interfaces this capability is available as well.
>> Great. So, I can sense that if some failure happens,
I can automate that the response will be rerun that activity.
>> Yes. So, you could automate the same point.
Because you don't want to be touching
your production workflows and most of the cases,
people have these automation buried.
So, you could essentially use
those SDKs to essentially codify this end-to-end.
>> Great. Is there anything more you want to show us?
>> I think for this particular rerun,
this is the first feature that we have launched,
is available for you to use right now.
We have complete documentation to tell you how you can use that.
We're essentially launching a blog post
as well for this for you to know.
>> Great. Well, thank you again for joining us today and sharing
this great new important feature to help with our data pipelines.
>> Yes, thank you.
>> Okay. We've been learning all about how to rerun an activity
within an Azure Data Factory pipeline today on Azure Friday.
[MUSIC]