Hey, friends, when it comes to integrating hybrid data at enterprise scale,
Azure Data Factory is what you need.
We're going to learn about
some exciting new capabilities, Change Data Capture and Managed Airflow.
Mark and Abhishek are here to show me what's new today on
Azure Friday.
Hey friends, I'm Scott Hanselman, and today I've got Mark Kromer
from the Azure Data Factory team and he's got his friend Abhishek.
And to share with me some new capabilities.
How are you, sir?
I'm good, Scott. Thanks for having us on.
So you brought me some demos of two different features.
You're going to show me one first, and I think that would be Change
Data Capture, what we call CDC.
That's right.
So, within Data Factory, as I know that you're well aware already,
we've been, you know,
a leader in the data integration and retail space for a long time
and a really exciting thing that we've invested in the past.
I would say, you know, 12 months in a recent announcement
that we had to really take this to the next level is allowing our users,
our customers, to be able to bring in data from many different data sources
in a really, really fast and efficient way.
And we do that through what's known as Change Data Capture, CDC,
the announcement that we're making.
And that's you know, we're talking about
today is you can see what I want to do is I want to show you on my screen.
Yeah, thanks for that. Thanks for that.
If you look at my screen, you'll see now Change Data
Capture is actually a top level resource within Data Factory.
So, you know,
we've had a lot of love and care and feeding around pipelines
and data flows and and power query within ADF over the years,
what we've done is we've taken the change data capture features
that have been present in the product to be able to get just
the latest rows and data that have changed from your sources.
Taking that to the next level and made it a top level feature now
within Data Factory.
So it's really super simple to get started.
You don't need to build a pipeline, you don't need to graph out a data flow.
You just you can see on the screen
what I'm doing with a change you to capture here.
I'm just picking a source in a target.
After you've configured everything, you're going to click a start button.
In fact, what I'll do is since we're here, let me just click stop on this.
Now, I actually stopped this one from running
and you have a button, so you don't need to make it trigger.
You don't need to set a schedule.
You just say, I want my data from here to go to here.
In this case, I'm looking at a SQL Server table, which is called sales
product, and I am putting that data into a Delta format.
So I'm creating a Delta leak out of that.
And once
I click the start button data fact, you will just execute this continuously.
It's as simple as that.
Interesting.
And by bringing it up to the the front, you are elevating it.
Like change.
Data capture is as important as data flows,
as important as templates, as important in data sets,
because you're going to be using it all the time
as you work your way through Azure Data factory.
Yeah, precisely. Exactly.
So let me give you an example of what that means.
So if you look at what we've had for CDC
previously within Data Factory, I'll show you an example of this.
So here's an example of a CDC enabled dataflow.
Now, this is a very simple one.
I just have some source data and then some target data. And of course,
I can apply my transformations in between those two things.
But on my source, if I were to run this a pipeline
without using change to capture this would read from my movies, file
or folder every hour, 15 minutes, whatever I set the schedule for.
But all the data that's in there will be read every time.
What we did probably around this time last year was we enable
the feature called Changer to capture within data flows.
So you see this is just a checkbox.
We we always in
the refactor aimed to make your data engineering job super simple right
so we enable this checkbox and now these different options
you can have you can say give me just the incremental changes
or give me everything that's already there and then subsequently incrementals.
What's real nice about this, Scott, is that data factory natively
takes care of all that for you.
We maintain the checkpoints, the watermarks.
You don't need to build any metadata driven pipelines.
You don't need to managing that yourself.
Just check the box and rest assured that we've got you covered for this.
That was easy, right? Customers love that.
But what happens is you have to know how to build a dataflow.
You have to graph a a pipeline, right?
So the pipelines look like these at these artifacts here.
So these are going to be really simple ones.
But in this case, I have a dataflow.
Maybe I'm also copying some data.
You know, you got to build a workflow for that.
If all you want to do is get your latest changes and put them into a data store.
Now we have this capability.
This is what we're talking about and this is what we just announced.
All you have to do is say, I want a new change due to capture.
So you say new. It's called a mapping. A new mapping.
You will give a name,
pick your source.
A real common change you to capture.
Source is going to be your SQL database, right.
And what connection do you want to use?
After that, you're going to get a list of all the tables that have either CDC
enabled or a column that you can use
to use to get the latest changes
so you don't have to have the native SQL CDC enabled.
These are the sequencing able tables here, but you can pick a table
that has a date time column or numeric column in it, and we can use that for you.
So it's smart enough to give me a CDC
like experience, even if I haven't explicitly turned it on its equal.
That's right.
And a lot of this has been a long time thing with SQL Server.
DBAs don't want to turn on CDC natively on their tables, and so that's fine.
We will allow you to use columns to do a watermark style,
and you're doing this in such a way that you're going to take pressure
off my database.
Certainly we don't want to be beating on it
every 5 minutes you're doing or whatever the appropriate and right thing to do
is to enable both that database to do its regular day job,
but also support these kind of data factory chains.
That's exactly. So it's it's a great set.
So I'm going to do is let me finish this right here.
This is all you have to do for a configuration, not call this demo for Scarlet's.
And what we do next is
when you're done with all this, what's really cool is
you don't need to set any kind of triggers or pipelines.
You just set your latency.
So in terms of your comment about, you know, creating saturation
on that database source, you say, how often do you want to check those changes?
And we were only get the things that changed during that time.
Two things I want to finish up this demo with is we are releasing over
the next couple of weeks this capability here called Less than one minute latency.
This is super exciting and super new for Data Factory, since
everything within data factory over the past, what is it now?
Eight years that we've been in market
is that it's always batch oriented.
With this CDC feature, it is continuous,
so you can set a latency to say, wake up every 15
minutes and run, but you could also say now less than one minute.
So it's essentially continuous.
And then a little bit after that is when we're going to enable this
feature up top, which you will see grayed out for now, which is real time.
And that's we'll come back with you, Scott,
and talk to you about that when that lands.
But that's Kafka in a bit of sources.
That's cool.
I love that you actually are showing us those icons.
You've grayed them out, but you say that they're coming soon.
Now, if people want to see change data capture in their existing data factory
pages, they just click preview experience at the top there
and they can start playing with it right away.
Yeah, it's actually already enabled, but it does have the preview tag.
Like you said, we're just calling that out.
Just so you're aware that it's public preview, it's fully supported,
but there are plenty of organizations that we work with that say that, well,
I want to wait for it to become generally available for use in a production mode.
That's fine.
It's fully supported.
We will support you and make you successful in its end.
You're really excited for customers to begin using that first.
And also that you brought a friend who wants to share managed airflow
and see how that works with Apache airflow.
Yeah, so I vishakha
once you give some insights into what we're doing with open source and airflow.
Awesome.
Thank you guys. Thanks, Mark.
Thanks, Scott.
So let's talk about Apache airflow and what it is and we are what we are.
Coming up, the new offering with so Apache airflow is an open source
orchestration tool and it kind of aligns very well with Data Factory,
which is an orchestration tool for data processes.
The value add that airflow adds is it's pure python.
So python developers love Apache airflow, and then it has robust integrations.
It integrates with a lot of different sources, a lot of different connectors
across cloud, multi-cloud, all of those, and then it's open source.
So there's a huge amount of community contributions into it.
On the right hand side, you see a bunch of providers
which are already available, so you can quickly get started writing your code
and leveraging all those community providers
to kind of connect to those different sources.
Now with that, I'll quickly run through where we are right now.
So what we are doing is we are offering now a managed airflow
as a service, Apache, F2 as a service, and we are terming that as managed overflow.
And it's super simple to get started with.
So earlier as customers were setting up airflow on Azure,
they would kind of start with Azure VM, setting up Azure VM
and then
going ahead installing the airflow runtime, the software on it, Python
runtime, all the different runtimes is going to take a long time
to kind of set that up and then you have to manage that.
Now what we did is we simplified that process and you can simply click on this
UI in the ADF in the Manage Hub under the Airflow tab,
you can click on the plus sign and quickly create the airflow instance.
Look, a few important things here.
Since we made it cloud native, we made sure we integrate it with Active
Directory as well so that with the same open source software you get the promise,
the security promises on Azure like integrated with the the audit
logs, the metrics and Azure monitoring metrics and stuff, all those things.
You get that for free on top of it. And so
you can choose the size of the cluster
that you want for this airflow and then you can add extra nodes.
These extra nodes are for the workers,
the additional workers that you would use for processing the data.
And and you simply hit on this create button.
And that's how simple it is.
It's going to take a few minutes, maybe about 10 to 20 minutes,
and then you will have your environment up and ready to get started with.
And then the next step, which I will run you through,
is I have an environment which I have already created.
So I probably use that
and I'll use the one with air flow with SSL, and I'm going to quickly
launch it and we get the whole airflow promise together.
This is the same the UI that you would get when you're using it on prem
and you see there's no tag.
Snags is basically equal into the data pipelines that you would be building here
and it's empty.
There's nothing.
So let's go ahead and get some tags up and running.
And so what I'm going to do is I'm going to import a few tags
and I'm going to say import files, choose a connection to a blob storage.
So you go to put your files until blob storage
and we will connect to that.
And what we expect here is you would have your tags
in a folder called tags, and that's the typical pattern.
And airflow is you would put all your code files in a Dak's folder
and we would import all these files from back.
So this is how simple it is to get started with creating the new environment
and then uploading your existing
files and clicking on this import button.
It's going to take a few seconds and it's going to import these files
into the environment and it says it successfully import it.
Now we're going to move back to our F the UI, hit the refresh button
right here, and you would see that you have both these two tags.
And just to kind of give you a quick overview, these two tags,
you know, one of the tags is basically calling an existing data
factory pipeline for moving high performance data
and then other pipelines basically calling you even processing pipeline.
Again, this is using data flows and it kind of brings together,
you know, into a single pipeline which orchestrates that
and you would be able to see these air flow pipelines and status of that.
It's right now running is going to take a few seconds and that's going to be done.
This is how hopefully, hopefully, hopefully our audience is familiar
with Apache Airflow.
But if they're not,
it is worth pointing out, just as an interesting
point of note for folks that may not be deeply familiar with this,
that you're the word that you're saying is not tagged with a T, it's DAG
with a D directed acyclic graph. Right.
And these are those those pieces, those atomic pieces
that build up the molecules of which the the workflow is expressed.
They're the tasks that get built up into relationships.
And this is just really like you can see you just switched into the actual airflow
dashboard, right?
This is like a custom fork, is it?
It's the real open source Apache airflow pattern, right?
That's very cool.
And I noticed also, I just saw the version number go by 2.43.
I assume you can pick versions.
I know 2.5 just came out a couple of months ago.
I don't have to worry about
that anymore like I would if I had to make a virtual machine.
Absolutely.
So that's one of the key offerings that you will always be updated.
So we will keep you updated on the latest versions.
Of course, not the major versions, but the my new versions
will come out, roll out as a transparent update to users.
Fantastic. That's so cool.
Yeah. Being able to do and this is a thing, right?
This is the whole point of the cloud is I don't want to have to worry about
the care and feeding of the airflow instance.
I just want to use it.
And airflow is a very dynamic python driven world and I really want
to focus on that part of it and not the keeping it running part.
Absolutely.
Very cool.
So is this in preview or is this available right now?
Can people use this immediately?
Yeah, they can use it immediately.
This is in public preview right now.
We just announced it
and then it's ready for users to kind of use it in under public preview.
Very cool. Let's bring Mark back in.
This is a very exciting time for Azure Data factory.
You all just boom, boom, boom, bring in other features.
Yeah I think there's and there's a wide range of things
that we cover
two different very different features within due to factory
but they expand the scope of the data factory where we have
just so many use cases that our customers just want to be able to do with Azure.
And we're just excited to bring that to them through these ways.
And you know, from the open source adoption of airflow to the ability
to get your data in through a pipeline using change you to capture
or through changes to capture resource,
you know, we feel like we've got you covered with Data Factory.
That's really a great point.
Yes, it is expanding the Azure Data Factory Cinematic universe,
whether it be on the front end
when we're capturing the data or how we see that data move through,
the tools are available and folks can start playing with these immediately.
Thank you both for hanging out with us today.
Thanks, Scott. It was a pleasure. Thank you.
I am learning all about change, data
capture and Azure data factory, as well as managed airflow in Azure Data Factory.
You can check them out today on Azure Friday.
Hey, thanks for watching this episode of Azure Friday.
Now I need you to like it.
Comment on it, tell your friends retweeted.
Watch more Azure Friday.