hey everyone we are back here and we're talking about Azure data factory all you
data engineers and analytics professionals please join us
we're gonna talk about the data factory new capabilities we're gonna talk about
debugging we've gonna talk about monitoring we're gonna talk about all
the various data sources and entire pipeline with an agitator factory and
Gaurav who's a PM with Azure data factory is gonna guide us through that
hi everyone we're back here with another episode of data exposed and this time
we're talking about Azure data factory we've got a guest in the room as well
and we let Gaurav introduce himself here sure Thank You Nishant, hi everyone I'm
Gaurav I'm a program manager in the azure data factory team and today I'm
here to talk about some of the exciting new stuff that we have done in data
factory and we will walk through all the details step by step and explain to you
how you can do various data integration workflows using data Factory
awesome thanks Gaurav and of course I'm Nishant Thacker, I'm your host. I'm a
technical product manager for the analytics and AI services here so let's
get diving into this absolutely so Gaurav tell me a little bit about Azure
data factory first what is it why is it important and why should a data engineer
care for it thank you so data factory is a managed data integration service in
Azure that allows you to build your ETL or ELT workflow so extract transform
load or extract load or transform workflows in Azure and using data
factory you can build your workflows acha data pipelines and to end and
ingest data from a bunch of sources do transformation over various computes
like data bricks and ash or and then egress your data and load
data warehouse which can be hooked up to your part bi dashboards where you are
monitoring and gaining insights into your data that's awesome this is really
important for data engineers who are creating these big data pipelines on a
daily basis you really don't need to go ahead and now create some scripts or go
ahead and create all of those exhaustive coding environments around these data
engineering pipelines you can use easy to maneuver drag and drop tools to now
create your pipelines and bring them to life and orchestrate them very easily
along with collaboration capabilities within your team so Gaurav let's see
something in action what does agitator Factory look like so basically this week
we are announcing the general availability of new capabilities in
Azure data factory data Factory is a ga service in Azure portal so you can go to
plus new analytics and you will see data factory right here and you can create a
data factory so what I have done is that I've created a data factory on in my
Azure portal I have pinned it to dashboard and when I click it you'll get
a blade like this so on this particular blade you can see information about your
data factory about authoring and monitoring on authoring and monitoring
which is essentially the visual drag-and-drop experience that we have
created for data factory for building your pipelines you can see some
monitoring information about your pipeline runs your activity runs so
let's quickly get started and see how you can navigate through this experience
so this is awesome right as I see over here you're actually in the azure portal
so everything is natively integrated into Azure and I can tap into Azure data
sources with all the data that I possess and create those pipelines within Azure
data factory yes easy to get started here yes
let's see something here so once you click author and monitor you land on to
this new experience that we have built from ground up it's been in preview for
six months or so right now so a bunch of customers are
already using it and on this overview page you can see that there are some
quick actions for you to get started so you can create your pipeline you can
copy data from a bunch of sources you can lift and shift your exercise
packages or you can even back this experience you with a repository for
source control and versioning so we'll go through each of those one by one but
let's get started with creating a very basic pipeline and before I go there
this page also has some quick videos for you to understand more about data
Factory and step-by-step tutorials where we have invested a lot of manpower in
developing these tutorials to help guide our customers so you click create
pipeline and once you do that what it does it it gives you a blank canvas and
you can see that on your left you have your resource Explorer which is listing
all your existing pipelines and data sources and you have all your activities
in the toolbox so we have 20-plus activities today and you can just simply
drag and drop these activities on the canvas so let's say you want to copy
data from us particular source to another so in this case let's copy data
from Amazon s3 to Azure blob so I'll select the s3 data set and for fun you
can look at what this data set is you can essentially configure the connection
you can look at the file path you can look at quick previews of the data to
see how your data set is and here we just pull up the top 10 rows these are
completely configurable layouts that you can maximize and see every piece of it
your file format settings and everything for your data set so I've configured the
source and you can see that we are guiding you as a user to fork with this
experience so you can see there's a red box that tell you that there's one
required property that you have to fill and validation is front and center of
this experience so you can always click validate and we give you this pane on
the right which essentially tells you exactly what is required and these are
all clickable links which takes you exactly to the location of what is
required for this particular activity to be self fulfilling if you will now what
really interests me over here god of is that I'm seeing a red circle above that
particular activity and that kind of tells me that there is a debugging
functionality built in somewhere yes tell me a little more about that
absolutely so just I'm going to jump to another pipeline which I have already
built and so that we can actually explain and understand this in great
detail so in this particular pipeline I'm actually grabbing data from a bunch
of sources then I'm essentially prepping the data in Azure data break so I'm
running a node break in agitator breaks and then I have another notebook which
is essentially improving the product visibility in the catalog that I'm using
on my retail website and then I'm copying my data to DW and if any of this
operation fails I want to send me a failure email so let's say if I just
wanted to run the first step I just want to check that whether copying data from
dynamics works or not so I can put a breakpoint here and just run that
portion of the workflow so this is very useful for our developers now because
you can imagine that in the data integration world you can have complex
workflows and sometimes you just want to see that a portion of the workflow runs
successfully or not so in this case they can just put a breakpoint and they can
just debug until that location this is very interesting curve I can't tell you
how much I am pleased to see this functionality because being a data
engineer and analytics professional myself I have always missed this
capability while designing my pipelines and now it makes life so much more
easier for data engineers what's even more interesting to see is that the
compute that is happening the heavy processing that is happening is actually
leveraging the best-in-class environments for compute like SPARK in
Azure Databricks in this case and now I'm able to leverage all the power of
Apache Spark in an optimized environment and bring it to life in a seamless
pipeline with an azure data factory and I'm sure I can orchestrate these
executions as well yes absolutely so data factory allows you to use any
compute whether it is as your data breaks so you
can operationalize your notebooks you want to use hive Pig MapReduce or HD
inside you can use that as well and once you so one thing that we have done and
made sure is that we want to increase the productivity of the data developers
so we have made iterative development and debugging front and center of this
particular experience so once you are ready with your pipeline and you're
ready to operationalize you can just put a trigger on the pipeline to say that I
want to run this on a schedule I want to or I want to run it on an event so you
could say that I want to run it every minute and here also again like I
mentioned usability is the key so let's say you want to generate monthly reports
and you select the monthly options you can select the last and we will
automatically calculate whether it's a leap year or not whether it is 30-day
month or a 31-day month so we are just trying to make things easier for our
developers or you can use an event based trigger which has basically we are
integrating here with as your event grid so all the if your file drops in as your
blog you could trigger a particular pipeline and as event grid supports more
and more sources they will automatically light up in this event base trigger into
the factory as well this is really interesting so now I don't need to know
a particular time I can actually wait for an event to start off my pipeline as
new data comes in I can start my pipeline process it and get the latest
data into my reports or dashboards or data warehouses or even spin up machine
learning pipelines with the latest data and retrain those pipelines as they come
yes and this was a very important capability that our customers were
asking because as you can imagine with the schedule you don't know with data
integration getting more and more complex that whether you're upstream
operations are complete or not but with the bank base now you can signal to say
yes I am ready go kick off my pipeline and then your entire data integration
process will run very interesting stuff now curve tell me a little bit about
monitoring with Azure data factory can i see previous runs
and I see what went right what went wrong can I even dig into individual
tasks with those previous runs absolutely so right now where we were on
this experience is what we call is the author tab but we also have is a monitor
tab so in this case I'm just going to discard the changes that we were trying
to do and once you jump on to the monitoring pane you can see that we have
a simple ListView based interface today that allows you to see all your pipeline
runs and again here we are trying to make things easy for you so you can see
that there's a time zone so we automatically snap these runs to your
browser time zone so if you have a team working in India and they are trying to
look at the runs they don't have to do the calculation math calculation to say
when the run started it will automatically snap to their browser time
zone interesting can we dig into one of these runs absolutely so in this
particular case as you can see that I've run a data breaks demo pipeline so if I
go inside this particular pipeline I can see that I've run a notebook I can see
all the inputs and outputs for that particular notebook I can see how much
time it took for it to run or if I had defined any properties on it I'll be
able to see all of those this is really interesting I'm assuming this was a very
simple pipeline but in general these pipelines have multiple tasks in them
and and you can do that for each and every task within that pipeline yes so
you can have a simple pipeline you could have multiple activities within a single
pipeline as you are seeing on the screen right now or let's see this one in this
particular case as you will see nothing ran so just in a pipeline run ran and
there was no activities that were there inside that particular pipeline so it's
just showcasing you in this case we are just doing a copy from from one source
to another so you are just seeing that particular operation and not only that
you can see all stats of your copy as well so you can see that the source was
this and I copied 100 gigabyte of data from Amazon s3 to my Data Lake store at a
throughput of 500 MB per second and we are showcasing all the statistics as
well to show you how much time the data was
the queue how much time it actually took for the data to land to the store so all
of those statistics are available for you to look at this is really novel over
here Gaurav and I can tell you I'm really excited by the fact that my
previous experiences with secret server integration services find a lot of
similarity in this new environment with Azure data Factory and the new
capabilities that we've introduced into Azure data factory as it comes across
now I know we saw that there was a SSIS option as well can you tell me a little
bit more about what that SSIS packages looks like with an agitator Factory yes
absolutely so one of the things that we are trying to do with the new
capabilities inside a factory is that we are giving a path forward to the
SSIS community that have invested decade's worth of time and effort in
building those SSIS so we are saying that now you can lift and shift your
exercise packages to Azure and how you can do that is using this SSIS
integration runtime so we are taking that infrastructure piece away from our
customers so you can just come in and say that I want to create four VMs in
Azureand you have all the skew so if you have an on-prem infrastructure and
you're using a particular skew on frame that entire exact VM would be available
in Azure as well and you can run exercise on it and we actually joke
around and say that we have bug by bug compatibility which means that if you
hit a bug on on-prem SQL server we can guarantee you that it you will hit
the same bug in Azure as well because it's the same exercise engine that we
are running in Azure well that's just for joking here but the idea is that you
have pretty much point-to-point compatibility with functionality on SSIS
that you are so familiar with and you're so used to on in your on-prem
environments that you can bring them all up to the cloud and execute seamlessly
this was really an interesting thing to see over here Gaurav and then thank you
so much for your time and and guiding us through some of the new
capabilities in Azure data Factory I'm sure all the data and engineers and
analytics professionals watching this video are really excited about this
capability and can't wait to try it out so please go to Asha comm slash ADF or
Azure data factory and you can go ahead and try out Azure data factory with all
of its new capabilities with all of the enterprise saas because now all of these
are generally available absolutely thank you so much for joining us today and
looking forward to having you for our next video thank you