>> Hey, friends. When data integration activities fail,
that ends up impacting everything downstream.
Constant manual monitoring is inefficient,
is time-consuming, and it's just plain boring.
Gaurav is here to show us how you can
improve your operational productivity by using
alerts on data integration events to
monitor your Azure Data Factories proactively,
today, on Azure Friday.
[MUSIC]
>> Hey, folks. I'm Scott,
and it's Azure Friday.
I'm here with Gaurav and we're going to talk more
about Azure Data Factory.
You've got a way to proactively monitor my pipelines with alerts.
>> Yes. So, the volume variety and
velocity of data has been increasing, right?
Which means that the organizations
or enterprises have less time to
react to events and
maybe respond to the failures that happened to the system.
So, it's very important for them to
proactively monitor their data integration
workflows because the manual approaches don't
work and they are generally inefficient and time-consuming.
So, what we are saying now is that we did a factory.
You can create alerts and
these alerts are integrated first-class in Data Factory.
Behind the scenes, we are using
the Azure learning infrastructure like every other Azure service,
but there are available first-class in
Data Factory user experience for
them to configure so that they can
proactively catch those failures
in their pipelines and essentially erase
those alerts before any failures
impact their system downstream or upstream.
>> That's really nice because Azure Data Factory is known for
its really excellent monitoring and how fast
and how clean it is to monitor stuff.
This seems like a natural extension to that.
>> Exactly. So, what we have is that if you go to
our "Monitor" section now and if
you go to the "Alerts and Metrics" section,
we have this ability for you to create a new alert rule.
So, you can go create a new alert and here,
just provide the name of the alert.
Let's say you want to say, fail all pipelines.
This is important why I say fail
all pipelines is because you can think of
this as a template for raising alerts on any failures,
on any pipelines in your data factory, right?
So, you can say,
let's say, we wanted to be a set of an alert.
>> Okay.
>> If we add a criteria or here we want to on field activity runs.
Here, I am able to select different dimensions as well.
So, I can select an activity name,
pipeline name, or failure type.
As I mentioned that I want this to be
a failure across any pipeline in my Data Factory, right?
Because I'm creating a template for raising alerts on any failure.
So, I won't select any value over
here and I would just say "Add criteria."
Then, I can select whether I want on
this particular criteria to
be let's say I'm using an existing group,
and in this particular group,
if I want to raise an email,
an SMS, or an Azure app notification
or a voice call to be raised when this alert fails.
>> Those groups are not specific to Azure Data Factory.
Also, Azure alert groups.
>> Exactly right. So, like I said,
this is all integrated with Azure alerting.
So, those are Azure Alert groups that you create.
So, if you already have one existing for any other Azure Service,
you'll be able to see that over here as well.
>> Nice.
>> Yeah. So, let's say I'm going to say I
want the email to be saying to me if it fails.
Not only that, I want an SMS as well on particular number.
I'm not going to give my number so let's say
some Vietnam whether I type over here.
So, I say "Add notification" and at that point,
the action group is added as well.
I say "Create Alert Rule."
At that point, what we are doing in
the system is we are going in creating this particular alert
and it's going to show up in the portal once it's created.
So, now, if any failure
happens across any pipelines in your Data Factory,
you will be notified by a phone call.
They then SMS in your phone and because I just configured SMS,
if I configured voice as well,
I could have got a phone call as well and an email notification.
That email notification looks something like this where it
says that what kind of a rule triggered for what resource.
It has a link to the resource as well.
So, if you click this, it takes you
directly to your resource where you can look at
your particular failure and fix it and essentially,
this is us saying that we'll be using this.
Organizations can improve their operational productivity because
all the DevOps guys in their organization can just click this,
identify the failure, work with
the data integrators to get
the pipeline fixed if there's any issue,
and then have the things rolling
again for their data integration workflows.
>> Is it primarily a binary thing
where it's failures versus success or
might there be like an anomaly that isn't
a failure that I would like to know about,
but I don't necessarily want to fail and call that a failure?
>> So, right now,
it is based on the metrics that we
publish to Azure Monitor, right?
So, right now, publish as I was showcasing when I was
creating the loads if I were to go back and show you.
So, we have these particular metrics
on which you can raise alerts.
>> So, there's a lot of things there. It could be
a memory thing. I just want to know about it.
Maybe I'm actively debugging something and I just want to
be notified about CPU issue or a memory issue.
Even though things are successful,
you might be bumping up against a metric that I'm concerned about.
>> Yes. As you're aware like we in
Data Factory team are constantly looking for customer feedback.
So, if we find any particular metric
that customers are interested in,
we will definitely go add it over here so
that they can raise alerts on that metric as well.
>> That's a really great point. One of the things that we've
learned over a number of videos that we've
done with the Data Factory team and yourself is that you
are extremely responsive to feedback.
So, if you've got an idea for
a metric that could then potentially feed
into an interesting alert that would
help you proactively improve your productivity,
definitely, let them know, it's important.
>> Yes, exactly. So, we'll be able to add that and you'll be able
to raise alerts on that particular metric as well.
>> Very cool. So, this is available today, and I've got this.
This is just already rolled out.
>> Yes. So, this is all available.
Again, we have a blog that's published that talks
about this alerting capability.
Then, we have documentation as well that explains
step-by-step on how you can configure
these alerts and you can get cooling with that.
>> Fantastic. You continue to improve all the time.
Be sure to subscribe to their blog.
They're always adding new features and lot of great stuff.
>> Yes.
>> I am learning all about how to proactively monitor
my Azure Data Factory pipelines with alerts,
today, on Azure Friday.
[MUSIC]