Hi, I'm Donovan with another episode of Azure Friday I'm here with Gaurav and we're going to
learn about data factory V2 welcome to show thanks for having me down so why is V2
what makes V2 special so essentially we have the energy service for two
years now okay least it as your factory we won back in October 2015 as a
gs servers and what we learned over a period of time is that there's a bunch of data integration packed
instead we don't support in eighty if we want so what we have done is that we have released a
brand new version of data factory with?
Great flexibility in the app model to
support additional control flow features like iteration loops branching
pattern meter processing so every pipeline is a parameterized endpoint now
also you can essentially call it from data factory constructs like triggers or you
can call it from the other as your services like
casual functions so it gives our customers flexibility do
mimic those data integration patterns that were not of your supported in data factory bone
right so this sounds to me I'm old-school guys sounds like biz talk to me is so is this
base talk in the cloud is something different so this is something different because this dog is targeted
towards enterprise application integration say think of all can degrading with
wal-mart and this is more of data integration scenarios so data
factories a manage data integration service and to out that allows you to base your
dl ap pipelines end-to-end and asher are so this is taking
data from one polishing and putting it into another data source for me to be able to use absolutely so
on and we actually have support for sixty five data sources
now they're allowed when we allow you to ingest data from those sources into asher
okay and there are certain sources that we support for addressing data as well perfect so show me what
how it works so you're going to click class and data factories there in
data and analytics so you go here you click data factory and
you'd be presented with the blade where you can provide the name of the data factory select and
add your subscription and provide other details so once you create
your data factory and if you bring to are dashboard you would have something
like this and it from here what we have done is that we have
enabled alter and monitor
the tile which you can click and all it is going to do is that
it is going to launch this brand new ux that we have base ground-up
okay and with this ux what we are doing is that we
are exposing certain quick actions that if you wanted to get started with data factory we
too and you wanted to comment radio force pipeline and you can do
that you want to copy data from a bunch of sources you can launch that if you're
in exercise user the video if we do we announced capability where
you can lift in shift your exercise packages to measure so you can actually
configure your exercise integration runtime over here so if you were to click this
image show you exactly how you can do that okay so you can just provision be engine as
you're and we would essentially run your packages on these reactions and,
we'll take that infrastructure portion of e from
you okay great so going back essentially you have
a quick overview use about data factor you reduce feature tutorials so if you wanted to
use the app to copy data to transform data how you can do that these are all
feature tutorials that show you step-by-step how you can use this
new visual experience to do that okay so what I'm going to do is that I'm going to quickly go
click create pipeline and essentially this going to take us to
our author dab which we call and it is presenting us with
a blank pipeline canvas and now I'm ready to create
my activities in the pipeline so for this demo I'm just going
to copy data from amazon sthree to blob so I'm just
going to drag search for a copy activity and drag-and-drop a copy activity on
the canvas be able to name this as three to blob copy.
And, we are essentially going to provide the source so in this case the
sources going to be amazon is three so I'm going to
select amazon sthree from this so, a bunch of sources that we support like image
and research have support for sixty five data sources right now so I select amazon
sthree and at this point I'm just going to provide amazon
sthree connections I've already created bag with this ux we're also enabling for you to
test your connections and if you want to edit your password you can do right from here as you can see
that connection is a valid connection okay and,
you can now browse your amazon is three storage so when I
keep browse all I'm doing is that it is opening up all your folders that
idea in amazon is three account ok so
I want to move some customer information from to blob so I'm going
to select this customer information file and I can look at quick previews of the
data as well so when I click preview data we're able to go in fetch the forest and
rows all of these are configure able layout so you can full-screen them you
can drop them to any resolution you aren't ok so at this point and,
my amazon is three data set this set I specified from where I want to
move their data and what exactly I want to move I just named amazon
sthree ds and I go back so my sources configure now I want to move
it to blob so I'm going to create a blob connection now okay so I select as
your blob and now I'm going to say and
let's named amazon as your blob datasets ok again I've created a blocked
connections I'm just going to select act and I'm going to provide the folder where I warn
their customer information five-two we move ok so I'm just going
to select data folder and say finish and at this point
have specified manager block data set as well let's rename this pipeline to say
as three do the job copy pipeline
okay and one of the other things that we have done with this is that we redid super simple
for you to validate your pipelines before you publish so you can see over here that we have check
marks should tell you actually everything that you needed to be specified is already deer
let's say for some reason you did not specify the source over here you can
actually take validate and at that point but eggs exactly tell you what is missing okay
in these are applicable links so you can click this and it will take you
exactly the location what is missing I can see this being really helpful the larger this becomes right it
would be like a needle in a haystack figure out what you forgot to configure absolutely so that is why we
made sure that validation is front-and-center off this particular
experience and another great-great thing that we have done is that a lot of times if
hold from customers is that they have production
work flows running and before they do the publish to their production work
floor to any edit should add workflow they want to guess runs to make sure that whatever new
additional ended that they are doing is working or not got so at this point I
haven't still published this to my data factory service but I can do a desk going
to just validate that everything actually it runs in edge or not but the test
run won't actually executing so my blob stores so, for example,
that data folder inside a blobs george
could be empty right now fire run test run it would still be empty at the end, you know, so,
it will actually do work test run for you guys so it will actually do
a disappearing to tell you that the data was actually copied from the
fact of amazon is three account to this as your
block so once you do that you array to publish and you can now publish
our changes to the data factory service ok so now how do I schedule this to
run I mean I probably would want this to run on a basis like
a continuous basis or on a on a trigger some sort how
do I do that so once this publishes then what we can do is that we can actually
have a attached triggered to this particular pipeline
ok so we have support for what we call this can-you triggers and we have support for
doubling window triggers so let's click this and you can see that it
is actually publishing to the data factory service and it's successfully publish and,
when I kick trigger and presented with this interface where I can either select
an existing ones if I had some
triggers already dear I correct fashion existing trigger to a pipeline or I could create a
new one so you can see that let's call this as good you trigger?
And, I on to schedule it every minute every hour every day
every week every month and we have options within this as well so if you
want to vicki anymore one a particular day of the week at a
particular time on that day in the week you could do that as well perfect, yeah,
this started me think I have app that actually downloads the last lottery numbers but you do that
like every monday wednesday friday in a certain time and might
can I think I could use this to go and harvest those cs v falls for me but what you did
years a using the copied it from one to the other can I do something more sophisticated like I have a cs
v now I want to parse it actually put inside of a database or something that's a little bit more of a data model
that I can use later yes so if you wanted to take that cs we in week to as
you're let's say as your sequel that reason so in the sink section
v selected and as your blob so you could have actually created in as
your sequel database connection gotcha
and pointed to a table and the cs we would have moved to that particular table as well
okay and one of the interesting things that we support is that we call
is mapping right so you can map the columns on the left side
to the columns on the right side so by default we're going to do a one-to-one mapping as you
can see over here diag we go to use microsoft research technology to do
need a mapping by name okay and if the name don't match
the map match than we do it by location by
position okay and you have the option to change it if you want
refuting that property ten maps two properties zero on the right-hand side you can change and I can
see looks like you can decide if you don't even want excluded or included you have complete control over that as
well absolutely so what I'm what I'm interested in now is that so I have the cs refile
and when I downloaded it's every entry what happens when I do it for the second time do I
get duplicate entries or does it only move the one new entry because every night it's going to use update that
same-size refile so we have support for what we call this parameterization
ok so imagine that you're moving from amazon sthree account and
every day on sthree there's a for this structure that is generated by the time step
okay so you can actually in your amazon sthree when you
edited you code if you go to connection you can parameterized your
file but you can see add the rate folder dart and, use
day-time
functions we have support for expressions functions system variables and you can use that
so every day when it runs is going to evaluate that function gold
would that particular folder with bag date time stamp big that data and
call ptwo as your sequence and they're also you can say to
add it all to wipe or reload so it's sort of value to you gotcha code that's the part that
was a c one other thing as caught my attention is we've been poking around is there's a code button
over there I love the fact that you allow me to come in here and graphically manipulate and build
it but sometimes I kind of what that as the starting point and now I want to take complete control is that with that
but does for me absolutely so one of the things that we are targeting is that we're targeting
board enterprises as well as eyes sweets okay now if you think about highest
mis david start from this interface but ultimately they want to stamp it out for
their different customer shot so they can essentially come here build their
forest pipeline click discord withholding tags into that I see they can take
that jason and they can use it to bury the pipelines further so,
another thing that I want to point out for eyes wes is that like I showed you the
basic scenario where we came in we build the pipelines and we essentially
publish directly to the data factory service right but let's
zoom and I'm just going to discard the unnecessary
the changes that we did ok so, their view back we've already
nauseated yes we have it yes so we can essentially attach a
core repository to in this data factory
okay and we have support for v ts gate and get have
this coming in future okay so when you say we st is gate we automatically
go and grab your gate accounts?
That you have access to all the projects inside that
gate account okay and you can essentially create a new report or you can
create an existing repair so let's assume that we you want to use an existing report
and at this point when I click save all it's going to do
is going to attach this repository with my data factory now from
there on every time I do any
attitudes I will get source control collaboration I'll get a
versioning with my data factory got it okay perfect so that code that's
being generated in the background is now also going to be version for you so you can use it like a normal
perfect yes so as I'm sure casing this to you once this saves I'm
we're actually navigate to migrate account and you can see that the mode or
here change too we st is gate earlier it was data factory also at
it you can remember okay it so it's we st is gate and when I go to my
reborn now if I look at the branches I
have a master branch and all my juices are right here my goodness
so you now any time you add any new pipeline there jason will automatically
be over here in your data com you can I can I modified from here as well and see the change is reflected in
the graphical one yes, and, the other thing that we are doing over here is that let's see
now you have it in gate and let's say you want to
publish your data factory service right you'd get some changes you publish to the data
factory service so what we dual where there is v essentially
generate publish branch which also has the template I see so
now I swedes or even enterprises for that matter can
pick this template and stamped out across different environments but it's a day of
p prague or a product right because you can actually leverage arm templates inside of
vhs even further exact release management and build actually automate the deployment of
that's fantastic, yeah, I'm a huge fan of the sts if anyone knows me so it's really nice to see
that integration is already built in yes and the other thing is that once you have triggered
these then you can also monitor these pipelines
end-to-end so we have this monitor tab where if you go it's a simple simple
list view interface ok so you can see over here that are pipelines have been running
and the first one succeeds and we have I want to show the failure so you can see that how
failure looks like initial case you the errors right there as well in this
particular case ok so that's one of the difference between vone
and V2 instead of slices we have what we call this pipeline runs
okay each pipelined run has activity runs inside it so,
I have a ctwo blob and in this particular case I had an earlier pipe and which
had a spark activity attached to it as well so that's the second activity
inside that particular pipeline right you can
rerun it or cancel and already run existing run that's happening okay
value to you we have layouts over here which can snap to your time zones if you
wanted to look all these d times are being displayed here in a particular time zone
today you could do that as well and
then we have these quick options to essentially look at what graduate and things
got the last thirty days last week past twenty four hours and of course you can do
filtering to say I want to look for a particular pipeline I could do that as and this is a huge release
there's so much in there and, you have my like my brain is racing
I'll like let's finish this shows I'll play with some of this stuff because I have apps that
removal on a data and I had to write all that code myself and I can leverage this
service here to go and do all that low-level moving of the data
and I can focus on adding the value not a plumbing of getting my data from the lottery system into
bicycles server database you cannot do that I can schedule
is not even something after worry about anymore so this is really a need exist
a neat feature that I think I can a lot of us are going to find cool clever ways of easier and one of the
other things that we're making it very simple with this interface is moving data to asher
it so you can see that we have the copy activity which you can use for a party but
our per file copy and you can even parameterized to say that every day run it for different file
in a folder but let's see if you just want to quickly grab data from a bunch of like, you know,
that I have data in these bunch of tables in
sequel organizer blob or amazon st or amazon red shift or
an oracle you can actually use what we call as so,
we have to change the mode and you get essentially use what we call as
a copy due to have wizard okay and all this does is that it
just quickly ask you a bunch of questions and select your store give
us your settings and we will automatically create these pipelines in the
back-end night us a user if you want to be abstracted from these
pipeline concepts and just want to provide is a bunch of connections and just want to provide
us these other tables and I want to move so think of it as the detain
pulled functionality that you have in sequel server got it okay this is stack
guarantee in but we can now we can schedule it and we can nationalism pipeline fantastic
yes is very exciting stuff and then what we are making sure with this
really releases that we are providing
your guided codes ok so if you click these guided tours that we can do exactly what you can do on
a screen so these guided tools is actually infinite ties to every screen so it
wound based on screen to screen and we tell you what you can
go on that particular screen and we are documentation under there too that take us to docks
at microsoft-dot-com yes so once you click this it actually takes you to that
your documentation and we have quick links on the overview page once you
come down we have links for pricing documentation the state
is if you want to give us any feedback or raise a question all the update said
we do we release them over here so all of that would be over here as well and test plus we are
open for your feedback as well anytime awesome you want to give us a smiley or frowned
you can send it to us this is by no means replacing the css channels are raising
support request but let's say you want to raise a feature requests and you want parking
benoit immediately very closely tracked this and we're going to essentially
keep a close eye on this and make sure that our customers are happy fantastic
we're learning all about data factory V2 here on asher friday?