>> Hey friends. You know like everywhere in Azure,
we are rapidly implementing changes in
the Azure Data Factory visual tools
and response to your feedback.
Some recent updates include
new functionality, better performance,
and security enhancements and Gaurav Malhotra is here
to show me what's new today on Azure Friday.
Hey folks, I'm Scott Hanselman and
it's another episode of Azure Friday.
I'm here with Gaurav Malhotra to
talk about Azure Data Factory,
some of the visual tool.
Now the first time you showed me this,
I was blown away because I
honestly didn't understand that it was in the browser,
like it's just a great visual way
to understand how data moves through the system,
and you're here to show me
some even better improvements in
the ADF in the Azure Data Factory.
>> Absolutely. So we did
some great improvements as you guys are
aware that we have these visionary tools
where you can essentially drag and drop
activities from the panels to your canvas,
disconnect them and build
your denigration workflows if
you will in a code free manner.
What we have done is that we have been very closely
monitoring the customer feedback
that we have been getting through various channels.
One of the biggest challenge
for feedback has been this a button where people have
been very ferociously using
this if you will and sending us feedback and we
have been very passionate about responding to them
and making sure that those concerns are addressed.
>> if I may, to be clear,
a lot of people see these happy face,
smiley face things and they think maybe it
just goes into a bitbucket somewhere.
>> Yes.
>> You and your team,
real humans, look at every bit of feedback, every piece?
>> Absolutely and we do reach
out to them and our customers are
just loving this way of reaching out to them.
So what has happened is that we have
been getting feedback on three fronts.
We've been getting feedback on how we can improve
the functionality and improve
productivity for our customers.
We've been getting feedback on how we can make
their data integration workflows more secure.
Finally, how we can improve the performance
so they can essentially reduce their time
that they spend on building
these complex workflows for
their data integration scenarios.
So, in terms of functionality,
we have done improvements in
altering as well as monitoring.
So in altering, what we have done is that we have
released new activities in the toolbox.
So now we have 20 activities in this,
20 plus activities in this toolbox,
and some of the new ones is the notebook activities.
So now you can essentially have
a notebook as part of your workflow.
So, you're essentially let's say you're gathering
data from 70 plus sources that we support,
and then you want to do the data prep or
that you want to transform
your data in Azure data bricks,
so you have a notebook over
there where you have written your Python Scalar or
Java code and you want
that particular notebook to be
operationalized as a step
in your data integration workflow.
So you can just use
this notebook activity as you see I've used to
work here and use that as a stepping your pipeline.
The other activity that we have added over
here is what we call as a filter activity.
So, you can essentially have
this filter activity right after
a copy activity if you want to.
The scenario here is
that you copy all the data to Azure but
you want your downstream operations to
run only on a subset of that particular data.
Let's say you copy 10 files and you only
want it to run on just five of those files.
So you can use this filter activity to filter your data
and what goes to your downstream processes.
Then we have what we call as
an Execute SSIS Package activity if you will,
and this Execute SSIS package activity
is for our customers
who have the SSIS packages
and they are lifting and shifting them to Azure,
and now they want to schedule that
on a particular cadence and
run that as part of the process.
So they can do that firsthand by just dragging
this activity and you can see
over here that they can just specify the package,
the IR on which it runs,
the package part and it's
just got to go and they can put a trigger on top
and it can run on a particular event or
a particular schedule and work for them.
>> Just to be clear that SSIS,
that SQL Server Integration Services.
A lot of people who did a lot of
that work on their local machines,
in their local data centers have now moved
their SSIS experience up into the cloud.
>> Yes, and if you remember last time when I was here I
mentioned that with Azure Data Factory now,
we are giving a part to those users
who are spending great worth of investment in
building those SSIS packages and
they want to utilize the power for Azure,
they want to use pay as you go model.
So, with data factor you can do that and
this integration now having an SSIS package activity,
you can just point to
the back as you want to run in Azure
and you can pull the trigger and it can
draw on a particular schedule or anyway.
>> Very cool. Right.
>> Then we have other activities that we have
added or here in the canvas.
For the lookup activity that we have.
So, let's search for the lookup.
>> Yeah, under general.
>> Sorry. So lookup.
With look up activity,
now we support all the 70 plus data sources that we have.
So, earlier, lookup was limited
to the particular three or four datasets but now we
support for all the 70 plus data sources that we have
in the system from which you can ingest data into Azure.
>> That's amazing. Like a huge scroll really
fast through it but it's basically everything.
>> It's every sort.
>> Anything you can think of in
postgrads to even from Amazon.
Anywhere there's data,
you can bring it into Azure Data Factory.
>> Absolutely. You want to bring it from BigQuery,
you want to of bring from Amazon S3,
Redshift, you want to bring it from Dynamics, SAP.
All of those sources that supported for Lookup activity.
>> Very cool.
>> So the other set of
improvements that we have done in altering
is that we have enabled quick actions for our customers.
So, as you can see that I dropped in activity.
You can now see that
this publish has highlighted and it is telling you
that there is one change that has
happened since the last time we have published.
So again, improvement in productivity for our customers.
You can use expand all,
collapse all to essentially
collapse your entire explorer,
toolbox you can essentially shift your resource Explorer.
So, all of these quick actions.
I know it sounds more but these are
very common feedback that
we got from our users that they wanted custom layouts.
So we enabled this for them.
Then what you can to do is now we can essentially look at
the entire code for a particular activity as well.
So you can essentially click this code button
and you can look at the entire Json code for this.
Not only you can do that,
but you can essentially edit this as well.
So, I could just type in right
now something that I want and it has
full IntelliSense as you are typing
a particular section and it will tell you what it is
supported and it is not supported and
you can essentially add that and
deploy directly as a Json to your data factory as well.
>> So you have full control from
the visual designer that of course we have
a low resolution screen here but
if you have a large screen and you zoom out,
you have quite a sophisticated notebook there?
>> Yes.
>> Then I can go if I speak this language better,
I can move faster in this experience.
I can go and do that as well?
>> Yes. So, it's totally up to
you whatever you are comfortable in.
You want to drag and drop but if
after that you want to edit the Json,
you can do that as a customer.
>> That's fantastic.
>> Then what we are saying is that
another feedback that we
received from our customers is that,
I have a particular pipeline built in and
I want to essentially use that as a pattern.
So, I want to clone it because I'm going to
use that as a baseline and then edit it.
So now you can essentially right click and clone
a particular pipeline in your data factory as well.
So once you clone, you can see that
a new pipeline got created pipeline b1,
and it's just a clone of the previous pipeline that we
cloned and all the settings are copied as well.
So you can start from this.
Not only that, if you want to clone
just a particular activity
on your canvas, you can do that as well.
So, total control to you as to what you want
to do for your pipeline and where you want to start.
>> Are you finding as a user of this
not only as a developer but as a user of this,
you're spending most of your time
in the visual developer now?
>> Yes. So, we essentially look
at our telemetry very closely and we see
that customers are loving
the visual experience and I can tell you that
more than 80 percent of our customers
today are trying this visual tool,
are using it day in day
out for their data integration scenarios.
>> Wow. So, people are living in this now?
>> Absolutely. That's why we're making
this improvements and listening
to customer feedback very closely.
So, in this particular case,
let's say you want to search for
your entire data factory
and search that is that in Amazon,
something that I created.
So you can just search it and we will
show you everything that you've specified
for in Amazon dataset
or a pipeline where it is used and you can see
that here all those results were
returned and if you wanted to edit each one of those,
you can do it directly from here.
>> It's interesting to see how fast
Azure Data Factory has become an IDE for Big Data.
>> Yes. So we are very focused on making sure that
with all these new improvements
that we're doing in functionality,
the performance is not affected.
So, we are investing a lot of time in making
sure that the navigation performance,
as well as the published performance
because when you click publish,
we generate an arm template in the background and publish
it to your data factory subscription,
and we want to make sure that that is
not affected as we keep on adding new features.
So we're paying a very close watch on it and
making sure that the proof
is high enough for our customers.
>> Customers are seeing this update on a regular basis.
They go to the Azure Data Factory on any given day,
new features, new improvements,
new performance, functionality, security, performance.
It's a continuous improvement?
>> Yes, it's a continuous improvement and we're making
sure that customers know about through different mediums.
So we are writing blogs as you see
that almost every other week or
every week we release a new blog
on data factory and not only that,
we have this Updates section on top and every week,
you can see that there is new stuff.
>> A little blog in there.
>> Yes. So, this essentially
takes you to the Azure blog and it tells you
about new features that are lighting up in
data factory almost every other week or every week.
>> Very cool. So you've got the blog,
mini blog right there,
and when you click on the smiley face,
it goes to an actual human.
Lots of big stuff happening to enhance
your productivity with Azure
Data Factory and in the visual tools.
>> Yes. I've talked about the altering features.
We also did improvements in monitoring.
So now as your data factory pipelines are in progress,
you can essentially look in
real-time to see how much data is being moved.
So if this process would be running right now,
then you can essentially
look in real-time. So, actually is-
>> Look at that. That's 107 gigs.
>> Yes. Let's kick off
this pipeline and we'll be able to see.
So I'm just going to read on this right now
and you can see that as this is running,
if I go inside this particular pipeline and look at this,
this is real-time progress that we're seeing.
So as data would move,
we'll be able to see how much data has moved and
how much progress we have made for this particular copy.
>> So, that's kicking that up and it tells you is a DIU,
is that Data Ingress Units?
>> So, that is Data Integration Units.
>> Integration units.
>> Yeah. So, it's a unit of performance memory CPU
and think of it as
the processing power that you
get for your copy operation.
The more the units, the better performance you would
get for your cloud to cloud data movement.
>> So in this case you're going to be copying from
Amazon S3 blog into data and you've already done,
they go 3,724 gigs.
>> Yes. So, you can see that it's-.
>> Almost a gig a second.
>> Yes. So, it's really fast.
We're seeing the throughput in real-time,
we're seeing how much data is written.
>> Is it done already? Yes look at that 32 gigs
just in the moment we were talking there.
>> Yes.
>> Fantastic.
>> Yes. So, this
is the performance improvements that we have done.
We've also made sure that
one of the common things is people want to
be alerted in case of failures.
Right. So, we have integration
with Azure Alerting infrastructure now.
So you can see that there is
this "Alerts" button on the top,
and when you click this "Alert" button,
it takes you do azure portal
where you can essentially specify
your data factory and you can
create Alerts on any of these events.
Failed pipeline runs, activity runs,
trigger runs and you'll be notified.
You can use SMS,
phone or email-based alerts
all the mediums that
Azure Alerting supports today
to raise events and on that.
Not only that, you can essentially also
look at the metrics for your data factory.
So again, we have integration with Azure metrics.
So, if you click this link,
it goes to Azure portal and you can look at
your different events for
your data factories in this case
I'm looking at the successful pipeline runs but you can
look at other metrics as well and see how
your pipelines have been progressing for
the last day or
different time period that you specify over here.
So, this was
the functionality improvements that we have done.
Now what we've also done for our customers is
is made security improvements.
So one of the common asked from our customers was that
as you are creating these datasets,
VR essentially they are
essentially providing these connections strings or here.
So, what they wanted is integration with Azure Key Vault.
So you can see that for every key now they can
specify over here or they can
store that in Key Vault as well.
>> All right. Some people don't want to copy
paste, there're Azure keys.
>> Yes, so they can maintain those in keyboard
at a single place for all their credentials.
>> Very nice. So, improved functionality in every level,
security features like Azure Key Vault and
I've noticed as you've been clicking around performance,
performance, performance, always a
focus with Azure Data Factory
>> Absolutely.
>> Fantastic. I am learning all about
the great new productivity features in
Azure Data Factory and
the visual tools today on Azure Friday.