>> There are always a lot of updates coming to Azure Data Factory,
get updated, see some demos and learn how to
get more involved this week on data exposed.
[MUSIC]
>> Hi, I'm Anna Hoffman,
and welcome to this episode of data exposed,
today we have a very special episode planned.
We're going to bring the Azure Data Factory team on to
tell us all about what's new in Azure Data Factory.
Without further, I want to bring on Noelle,
a program manager from the Azure Data Factory team.
Noelle, thanks so much for joining us on the show today.
>> Thanks for having us,
we're really glad to be here.
>> We're really excited to have you on the show.
It's been a while since we've gotten
our fix of Azure Data Factory updates.
I'd love to just ask you to learn more.
>> Yeah. Every month we release a lot of new capabilities,
and if you take a look at this page,
we've got a little blog that says,
what's new in our Data Factory?
This is something that we update every
month so that the users are updated
and the community in
general is updated with what we've been working on.
Our product group and engineering
groups have been working really hard
to make all these great updates,
so definitely check on those.
As you can see, in January,
we made a lot of improvements to data flow
and a lot of data movement capabilities,
and in February, we've got
some really interesting features as well.
Today, I've also brought in two of my team members,
Mark Kromer and Abhishek Norine,
and they will be demoing
some really cool capabilities
that will really improve your productivity.
>> Awesome. Thanks so much Noelle,
and exciting that we have more guests,
it's a special thing to have so many people on.
First, we're going to bring on Mark and Alicia, and I think Mark,
you're going to get us started and tell us
a little bit about something
specific that's new in Azure Data Factory.
>> That's right. Thanks,
Noelle. Glad to be back.
What I want to do is talk to you about
a specific feature that we just recently
launched in Data Factory as part
of mapping data flows and flowlets.
What flowlets do for you is allow you to
build composable units or reusable units of
logic within your data so that they can
be reused in other parts of your Data Factory.
It makes for a very nice composable set of units,
and if you have logic that you
reuse many times within your data flows and factories,
now you can make
those as oracle flowlets and then you can reuse them.
They're self-contained, they can be added as
another transformation as part of your pipeline in your data flow.
That one being said, let me spend
another minute or two just to show you what that looks like.
On my screen, I have a fairly medium complexity,
I would say, in terms of a data flow.
But I shrunk that down to size so that
you can see how this would fit in a single screen.
There's a lot of different things going on in here.
I have sources,
I have actually one source in here,
I have a couple of syncs and a bunch of
transformations in the middle.
This is actually a deduping data flow within Data Factory.
It's very common, I do this a lot within my work.
What I want to do is I want to take this
without the source then sync.
I just let the logic in the middle that's
performing all of the deduping logic,
and I want to make this something that's
reusable. How would I do that?
I'm in my Data Factory U1 in the browser,
I'm going to turn this to a flowlet.
I can grab this entire section of
the data flow that is not
the sources sync, everything is highlighted.
With the Right-Click action, I can see
Create a new flowlet, as simple as that.
What I'm going to do is going to land into
a new screen that looks just like a data flow,
but it is a special screen.
You see the icon indicating and this is a flowlet,
then give us a name that is going
to indicate that this
is something that's reusable or this is a flowlet.
Now you'll see that there are new sets
of transformations automatically for me,
called input and then output,
and this is going to allow me to describe essentially contract.
Think of this as a function within a code paradigm,
and I have to describe the parameters or
the signature for this flowlet of what's
going to come in and then what is going to come out,
so I'm just doing auto mapping,
everything that comes in and gets
created is going to also come out.
What this means is, I can map this into
another data flow and I can reuse this.
Let me show you what that looks like.
This is my flowlet, then come over into another data flow,
and I can use that exact same duplicate,
the one that was a huge hard to read graph on my previous screen,
and I can add that into this data flow.
This data flow that you see on the screen,
that is the exact same thing as that large graph,
because I'm using this flowlet right here,
and the flowlet points to the flow that I've created,
and it's all the logic for me.
When you add it as you click "Plus" anywhere within
your data flow graph and you choose the flowlet transformation.
Then you can pick from a list of flowlets,
and all of your flowlets will show up here on
the left side under data flows,
how the icon indicates that is reusable as a flowlet.
Yes, that's a quick flyover
of what the new flowlet feature within it.
>> Awesome. Thanks,
Mark, that's really cool.
I think a lot of
developers are taking this micro-services approach,
and this reminds me of
a micro-services approach for auto Data Factory,
which is awesome because you can reuse those things,
especially the complicated things
over and over again. Thank you so much.
>> Exactly, you can reuse it,
and I would say too is it minimize this work.
This is pretty hard to see
because there's so much going on in here.
Particular detail scrunched into a flowlet and then reuse that,
and it has this nice orientation
now where it's only this size on your graph,
so it also makes it so that you can shrink
down all that logic into a single unit answer.
>> Awesome. Thanks so much, Mark.
Cool. That was a great demo.
I think we have one more demo planned for today.
For that, I'm going to bring on Abhishek.
Abhishek, why don't you take us through what's new?
>> Sure. Thanks, Anna.
So today I'm going to talk about
a new activity which we're introducing in Data Factory,
and that's called Script Activity,
it's similar to the execute secret task in SSIS.
It basically lets you execute
common operations like DML, DDL statements,
DDL stands for Data Definition Language in case if you
are dropping a table or creating a new table.
These are common operations as part of
your DDL journey you face while you use
data flows for transforming the data times you
too need to drop tables,
add tables, and run
certain execution commands on top of the tables.
With this script activity,
you will be able to do that,
and until today we have other work
around using store procedures lookups.
But now you have a native script activity using which you can
execute these commands directly against Azure SQL database,
data warehouse, synapse analytics,
including oracle and snowflakes.
Here we go. Here's is a quick demo of it.
You can see I'm checking for certain tables and dropping if
it already exists and then going ahead and creating just a sample.
This is just a sample demo.
In this case, we are just going to add
a few rows into the tables and you
can see there's different cars,
creating a cars table.
This is done, convertible [inaudible] run,
and it has prices along with it.
What we will do is, we want to quickly find out,
and so the first execution
basically creates the table into the system.
The second one runs slightly more complex query in SQL,
of course, and everything is in line,
which means you get the realness of
CICD as part of your Data Factory pipelines,
all the CICD,
all the code goes into
the metadata that you use in this Data Factory.
Everything gets stored in there and it goes across environments,
and that's one of the key things,
many customers use this for populating
their desk data across environments while doing CICD and the OPS.
You can do that pretty much now.
In this example, we are just
writing this query, getting some data.
Then the second part of the demo that I wanted to show here
is how we can easily consume this data
coming out of the script into
a downstream activity using
the new expression builder enhancements that we have done.
What I'm doing is I'm going to send these information,
the output of this query,
which basically returns only two rows actually,
so we are fine with that.
These are top two best steels in terms of car,
and I'm going to send that on the Teams Channel.
This is just as a demonstration in this particular case.
While doing that, I set up a new team channel pipeline and this is
where you have expressions
and expressions could be complex for any data engineer.
With IntelliSense, we actually make it super
simple to detect problems in your expression,
not just while authoring it,
so IntelliSense, of course, helps you with authoring.
The moment you start typing things like expressions at pipeline,
you start to see the whole grammar in it so you can
easily just go ahead and type things,
press tab and then start to see all the methods.
Of course, it makes it super simple.
But beyond that, it also helps to detect problems.
In this case, you can see it shows
a squiggle says, there's some problem,
and let me quickly fix that,
so I'm just going to fix this and let's see,
it says rename to pipeline.
Seems like there was a variable with the name pipeline name,
and I had probably spelled it wrong and it just fixed that for me.
This is the IntelliSense support that we added
on which makes productivity much more better.
Now coming back to the pipeline
and quickly hitting on the run button,
and that should basically send us the teams notifications out,
and it should take a quick few seconds.
We can see the script activity executed in just a few seconds,
the quick execution that happens,
there's the third activity which is basically sending
the top two cars that was found in
the deal as part of the team's notification.
It's going to take a few seconds,
and I have the notifications on my side.
Quickly let me share that one as well.
Bringing up my team's notification,
you can see it basically popped out
a couple of values Top 1 and Top 2 over here.
That was probably the quick demo
that I wanted to show. Back to you, Anna.
>> Awesome. That was so cool.
I can think of so many ways that I
can use this new scripting capability,
and of course, IntelliSense is really going to help me
because I always knew that.
It's great that you guys are thinking about that
right from the beginning as you're building this out.
I also like the example,
I know this wasn't a new thing,
but I didn't know you can hook up
actually the data-factory to team,
so I'm always learning new things,
so thanks so much for sharing that with us.
Wow, this has been a packed episode, I've learned a lot.
Just thank you all for coming on the show.
Noelle, as we wrap,
are there any final tips and
tricks or call to action for our viewers?
>> Yeah, if you'd like to see
some more demos about some of our latest capabilities,
definitely check out our YouTube channel,
it'll be in the description below.
You can check out all the pre-recorded videos that we have,
Mark and I will also be doing a live-stream on March 7th,
so that would be interesting for you to join as well.
On Twitter, we're often retweeting
different blog posts for you to check out and read,
and we also do
some updates letting you
know when there are new product releases as well.
>> Awesome. Cool. Thanks so much, Noelle.
To our viewers, thanks for joining us today.
Seems like there are a lot of action items.
If you guys want to get more involved,
stay up to date on the Azure Data Factory or just check
up the latest updates or some of the demos that you saw today.
Be sure to check for the links in the description.
If you like this episode, go ahead give it a light.
Leave us a comment and let us know what you're doing with these
new Azure Data factory updates,
and we hope to see you next time on Data Exposed.
[MUSIC]