Hey friends now you can improve operational efficiency
and developer productivity by visualizing your Azure Data
Factory pipeline runs using gantt views. In just a glance,
you can tell which run took the longest you can filter by
status and tags to view the runs that you desire using
these gantt charts. Gaurav is back once again to show me
how it works. Today on Azure Friday.
Hey it's Azure Friday. I'm here with Gaurav and we're going to
talk about some of the new stuff happening in data factory
pipelines, you all have been known for your visualizations.
You've got great visualizations. The pipeline editor is amazing.
You like really pushing what's possible in the browser. You
like visualizing your data and what's interesting about this
from my perspective, is, you're looking at petabytes of data but
you want to look at it simply want to 20,000 foot view. So you
got a new way to monitor your pipelines to show me so one of
the most common feedback that we received from our customers is
that they can see their run
information about their. Pipelines running on a cadence
or on an event but they are essentially want to improve
productivity, where simple glance of it can tell that what
run took the longest or what run took the shortest time because
you can imagine if we were to go that and do that in a list. View
you will note down the Times and then remember it and just buy a
quick glance. You cannot tell right So what we are doing what
we have done in data factory now is that we've enabled what we
call as Gant views and with these can't views so you can see
that I have a grid of information as I was mentioning
that if I were to see.
Here, which run took the longest which run to the shortest or is
there an anomaly here. It's very difficult for me to tell
visually but the moment I do this right. I say Gant then it
immediately becomes of you and now I can tell that what ran so
I can see that these are all running fine be cause. It's the
same size of the bar organized tempo exactly exactly but now
imagine for bar would be bigger or smaller would immediately
know that for that particular day something happened.
Either I got more data or my some abstract process is
anomalous exactly and this is one of the things that the human
mind. In this case can absorb a lot more information visually
with that list. I was actually looking at the list looking for
patterns. Yes, this is a much more appropriate for you and
we've already getting feedback from our customers where they
love it. Because we were running initially previews for this and
there are customers who are telling us that they are saving
huge amounts of time by just visually looking at this rather
than glancing the list. View 1
by 1. And not only that right if you look at this particular
information for example, in this case. This failed right so you
can just click it and will present you the information
about the run ID the start the end, so all the data is there
from the list. Just presented better exactly and here also we
will tell you what is the duration deviation right so
which means that on average every run takes let's say 5
minutes but if this were to take 30 minutes. Then there's a
deviation right so we will tell you that deviation. We will take
tell you what is the longest duration for this run this
happen? For the filter that you've applied here, and what is
the shortest duration all your error information that happen in
the link to your pipeline runs if I were to click. This it will
take you right into your pipeline run so that you can get
more details about this and this kind of stuff you can even do
like a side by side comparison. If I wanted to compare 2, yeah,
so here what we are enabling 1st is this view. We eventually have
plans to enable those side by side comparison. If you want to
compare those 2 runs, but this is the starting point. What you
can do with this now is that let's a lot of our customers.
Tag their pipelines and they come and say this is my ingest
pipeline and I want to monitor all my ingest pipelines. For
example, So what you can do is it. You can essentially group by
annotation so in this case. I have tagged some of my
pipelines's in jail so if I were to click this and say apply. It
is going to now. Give me a view for only the pipeline that is
tagged in chest now. This is a great view for customers. Becaus
imagine that if you have bunch of pipelines different pipelines
are tagged in chest, you can just use a single tag.
We look at all those executions and this is again I've asked
you. This before, but I love asking you this because I know
the answer I've got this already right. I'm just going to load
up. My data factory pipeline. One day and I'm going to put it
up in the corner. The choice to switch from list to get and then
as with all things data factory another day I'll show up and it
will just be even better and even better. You keep rolling.
These things out on a weekly cadence. Yes, so we do release
weekly and we keep on making sure that we're listening to
that customer feedback because that's most important thing.
To grow a product to grow a service and that's what we're
doing this is purely from customer feedback that we have
heard from people that they want this feature. So we have
delivered this to them. It's really great point. Now let me
ask you this, if I'm in this dialogue dialogue right now. I'm
looking at this chart? How do I get customer feedback to a real
human being? Yes, so all you need to do over here is click.
This feedback icon and this email is optional. If you just
want to provide some feedback you can just give it over here
and is the first line of feedback to the product key.
We monitor this very closely no way this is replacing the
support channel. But this is essentially if you want to get
in touch with the product team with some feature request or
initially of facing we watch this very closely. So to be
clear. This is not support but if you want to improve this, if
you have a suggestion how to make this better. This doesn't
just go to Azure General Mailbox and get thrown away someone on
your team is going to see this information. We triage it every
day, we make sure that we respond to those customers every
week. Fantastic so many great things happening. When you look
at how to monitor your data factory pipelines with Azure
Data Factory. Check out Gant views you've got it
now, I'm learning about it today on Azure Friday.