>> Hey friends, now you can monitor the health of
your Data Factory pipelines using
the Azure Data Factory Analysis OMS pack
from the Azure Marketplace.
Rob Malhotra is here to show me what that is,
and how it works, today on Azure Friday.
Hey friends, I'm Scott Hanselmen
and it's another episode of Azure Friday,
I'm here with Rob Malhotra talking
about the Azure Data Factory,
and when you've got a lot of
data factories you've got one that you care about,
you want to monitor it, and there's probably
a better way than me just hitting refresh a lot.
>> Absolutely. So, data integration solutions
are complex with many moving parts.
>> Indeed.
>> One of the major thing
that our customers want to do is to make sure they
are able to monitor
their data integration workflows, aka pipelines.
They don't want to monitor it like a hummingbird.
In some cases, they want to make sure
there are alerts on top of that,
and they want to customize
those alerts to make sure that they
can raise those alerts on
particular events that they care about.
So, what we're saying with
the integration with Operations and Management Suite,
is that now Data Factory has
a service pack that is available in Azure Marketplace.
You can enable this service pack
which will essentially give
you pre-can views to monitor your data factories.
This is essentially based on our integration with
Operations and Management Suite, OMS.
I will show you how easily you can do this,
and enable this for your Data Factory
and visualize this for your Data Factory pipeline.
>> Is this for anybody?
Or there are people out there who are using OMS,
the Operations Management Suite already,
are there people who are monitoring other things,
and are they going to bring their data factories in?
Or is this an opportunity for
people who maybe aren't familiar with
OMS to be a first time they've ever seen it?
>> So, the people who are already
using OMS and now want to make
sure that their Data Factory matrix also
flow to the same OMS account or a different one,
can do either of those.
They can essentially enable
the diagnostic settings for their Data Factory
which would allow them to route
their data to Log Analytics or OMS.
From there, they can monitor
the health of their Data Factory Pipelines.
>> Very cool, is it easy to set up?
>>Yes, absolutely. So, as you can see on
my screen that this is Azure Marketplace,
and this is the service pack
that we have been able to with there,
and all you have to do is just click "get it now".
When you click "get it now," it takes you
to your Data Factory,
you select your particularly Data Factory which
I have already done in my particular case,
its called ADF customer demos,
and you enable diagnostics for your Data Factory.
So, you can essentially turn on
diagnostic settings for your Data Factory,
and once you click the "Turn
on diagnostic settings," it takes
you to do this blade in the Azure portal.
You click "Send to Log Analytics," and you
specify you Log Analytics account that you want to use.
You can create a new work space or use an existing one.
Let's say, I'm going to use my line,
and you want to send all the matrix that we have enabled,
so you can say all the ActivityRuns,
PipelineRuns and TriggerRuns,
and you can just select all matrix as well,
so in future if you enable other matrix,
they will just flow right the way
to go to OMS account as well.
So, once we enable
these diagnostic settings, I've only done,
so that's why I didn't save,
I have this work space that's provisioned.
This particular work space,
as you can see that now it is
showcasing that for my particular Data Factory,
which is this UxDemo Factory that I have,
the data is already flowing to this OMS account.
When I click this preview,
you can see now all the particular pre-can views
that we have enabled for your Data Factory pipelines.
You're able to see pipeline-.
>> Look at that!
>> Yeah.
>> It's just lights up a whole dashboard for you.
>> Yes, so it's a pre-built dashboard
that you can enable just by a few clicks,
for your Data Factory.
It has all the Pipeline Runs by Data Factory,
your Activity Runs, your Trigger Runs,
top 10 errors that you have faced in your pipeline,
top 10 activities errors
that are there inside your pipeline.
If there were any trigger failures,
which I don't have, it would
have been enabled there as well.
>> It makes me wonder if there's any reason
that everyone shouldn't turn this on,
this is a great bit of
information that you have trapped in
your Data Factory and you could simply turn it on
by plugging in all of last your Data Factory.
>> Not only that, this also
enables cross Data Factory monitoring.
Now, what you can do is,
now if you look at our monitoring portal that we
have in the Azure Data Factory UX,
this particular UX for monitoring is called Data Factory.
What if you have multiple data factories
and you want to monitor cross data factories,
so what you can do as a customer,
is that you can essentially hook up
all those data factories to a single OMS account,
and that way, all those matrix
would flow on this single dashboard,
and you'll be able to see them.
>> So, one dashboard for
as many data factories as you want?
>> Yes.
>> All powered by OMS?
>> Yes.
>> Very cool.
>> You can essentially dig deep into each of these tiles,
you can look at a particular one,
like in this case, I'm looking at the PipelineRuns.
So, you'll be able to see
the corresponding Log Analytics
query that we are running.
You can even enter that query based on your requirement.
So, let's say you want to do only for
a particular pipeline, you want to monitor that,
you can add that particular
predicate in this particular query,
and you can create an alert rule on top of it as well.
So, this is how you can create customized alerts for
your Data Factory and
monitor your data factory pipelines.
>> Of course, because you are integrating with
existing features of Azure,
you can do whatever you want with
that alert rather to hear one based
on grouping and the medium by which you want to alert.
>> Yes, and you could have this OMS account
hooked up with other services like Databricks.
So, that means all your data from
different services are flowing into a single OMS account,
you can look at not only data pipeline matrix,
like pipeline,those activity runs.
You can also look at your
Databricks executions if you will.
>> Very cool. Is this
something that's in preview, or is this ready to go?
I have this feature,
one of the many features I already
know I have but I've never turned on.
>> Yes, so the good thing is that
we all have documentation for this.
We recently announced, posted
an Azure blog announcing this particular feature as well.
Customers can enable this
today for their Data Factory pipelines.
>> Fantastic, I love learning about new features of
Azure that you snuck in when I wasn't looking.
Here on Azure Friday, we'll get to learn about that,
and it's just going to make
my experience that much better.
>> Absolutely.
>> Fantastic, I am learning all about how to monitor
my data factory pipeline using OMS in Azure Data Factory,
today on Azure Friday.