Have massive amounts of data you need to work with?
Unsure how you can ingest, how you can transform,
and how you can load them all inside of Azure?
No matter where the data comes from,
no matter what form it is in,
Azure Data Factory is the solution.
I am going to tell you all about it, coming up now.
Hello, everybody. My name is Adam Gordon.
Welcome back to another exciting episode of 'What is Azure?'
In this episode, we are going to tell you about...
what is the Azure Data Factory service.
Of course, as always,
we are going to answer our famous three questions.
What is the service or feature or functionality?
Why would your company want to use it?
What is the business proposition
and value that it provides to you potentially?
Of course, equally importantly,
how do we actually get this thing to play,
standing up and put it to work for you?
Let's begin by jumping in and talk a little bit about
what Azure Data Factory is,
and of course, as a result of that,
what it is you can do with this particular service.
Now, you can see on thelightboard in front of me,
got an interesting diagram.
When we look at this end to end,
what we are starting with other here
is the database we are going to use
that is linked to ultimately,
a Cloud based instance of our data.
But, what happens in the middle right here between the two,
is actually the magic that the Azure Data Factory provides.
Now, our on-premises database,
this is a Microsoft SQL database, of for that matter,
could be any kind of database actually.
It is going to provide the beginning information,
the ingredients that allow us to actually use the data factory
to do very important things.
Specifically, in the language of data and data management,
it lets us do one of two things.
We can either do what is called ETL,
Extract Transform and Load.
Or we can do a derivative of that,
what is called ELT, Extract Load and Transform.
What that means is ultimate,
we are going to put ETL OR ELT right over here,
just so that we can kind of keep track of that logic,
and see these as basically things
that we are going to engage in or activities we are going to do,
targeted at our data.
What we're gonna do is take the data sitting in this database,
so whatever format it is, allow it to be moved through
what you see in the middle here, this pipeline.
This pipeline represents one of the key important elements
of what the data factory does for us.
It creates connections
and allows us to bundle together activities,
transforming of activities represented by the gears,
that work on our data in various forms,
represented by our grid and data here on the left.
Though it is a little hard to see,
transformed by a different color on the right
as it moves through the pipeline
and ultimately gets re-positioned in our database,
that is going to live in the Cloud,
where we can then consume it, work with it,
and do a variety of things on it.
The extract, the E in Extract Transform and Load,
is about pulling something out.
Pulling our data out of the database and making it into...
available into a form that we can operate on,
then transform the T in the Transform,
or rather the T in ETL, Extract Transform and Load,
is going to let us somehow manipulate
and change that data from green to orange,
and through the processes represented by the gears,
make it into something.
Typically by moving, manipulating, modifying format,
form and function, that we could then use in other places.
Our L for Load is going to represent our ability
to make that data available in a consumable format
at the end of that process, or at the end of that pipe,
that allows us then to make it available,
expose it and of course make it consumable.
So ETL, Extract, Transform and Load,
or ELT, Extract Load and Transform,
whatever order we do them in,
are really going to be the key activities
that the data factory brings to bear for us.
So it is going to be a process-driven through capability,
or a solution, a function in Azure,
that allows us to bundle together in one place
all of these activities,
and allows us to very easily lay them out,
set them up, configure them,
and of course, consume them once we're done.
Now, interestingly, how we consume them
is not going to be the focus of this episode.
But, we have done another episode
on the 'What is Azure' series
on the Azure DevOps platform and the capabilities.
Believe it or not, the way we link Azure Data Factory
and tend to consume the output,
in other words, what happens when we get over here,
is that this pipe, this pipeline, and other pipelines like it,
are associated with an Azure DevOp instance.
So if you want to see
what the Azure DevOps piece is all about,
take a look at that episode in the series.
You will see me explain the Azure DevOps capabilities,
and show you the different platforms
as well as the dedicated portal that we can manage it through.
One of the aspects I touch on briefly there
when we set it up when we answer the question,
how does it work?
Is the use of pipelines.
This will be a great example
of one of the pipelines we could create,
tying the data factory into the Azure DevOps capability.
So just a little extra added bonus there for you.
But when we think about the 'what',
we are thinking about a solution
that allows us to take data that is on-premises,
in a variety of formats,
and through a transformative series of events,
manipulate, modify it and ultimately change it
so it could be loaded and made available
in a standardized solution
that allows our services to expose and then analyze...
ultimately visualize and consume that data
in a variety of ways
that customers both internal and external can benefit from.
So that's really what the 'what’ is.
Let's talk about the 'why'.
Why would a business want to use this?
Let's think about a common business case
that a large organization,
perhaps your organization may actually have,
and a problem you have to solve.
Let's say this data that's been sitting here
that we haven't really given a name to or thought about,
it's not just any kind of data,
but it is data from all of our customers
and their interactions with us,
as they buy and consume goods and services on our website.
Or, if you don't like that particular scenario,
maybe this is data from patients in a medical practice.
Got to apply a bit more security, concern about privacy to it,
but still, we can easily use this process
to do the same thing that I'm about to describe.
Maybe we are in the gaming industry,
maybe this is all the data from all of the player sessions
and all the logs of all the games
that have been spun up by our customers on our platform.
We want to be able to take that
and run it through some sort of pipeline,
standardize and transform it,
load it into an Azure Sequel Database,
that is going to be a data warehouse
known as the DW down here.
Data warehouse,
that is going to allow us to expose that data,
and make it consumable through complex analytics,
business intelligence, machine learning,
and provide insights into our customer profiles,
and maybe find up-sell opportunities,
maybe find additional issues and concerns
that we want to address for those customers
by adding value, and creating new features.
All of these things can be done
as a result of using a data factory and pipelines
to consume that data,
standardize it, and expose it for analysis.
So any of these scenarios whether they are medical patients,
whether they are customers in a retail space,
whether they are gamers on a platform;
they are going to produce huge amounts of raw data,
but it is going to be in unstructured different formats.
We need to standardize it,
we need to move it from where it is into the pipeline,
load it as we extract it,
so the extract of E will take place over here,
the load will take place here,
and then we are going to start transforming it.
So this will be the T,
the transform is going to be these gears doing stuff for us.
Then over here, we are going to be able to take that data
as it is transformed and move it into one or more structures
that will allow us to work with it online
in a variety of formats.
Loading it, standardizing it, and making it available for us.
This is the value proposition
that really helps us to frame the answer to the question,
why would you want to use it?
Any of these large scale data solutions are going to require
this kind of capability to make sense of.
Ultimately if your business falls into this category,
data factory may be the answer to that question,
how do we work with our data?
More importantly, how do we standardize and understand it?
In order to take a look at how this takes place
and what we do in order to stand it up,
we are going to take one of those magic field trips
we always talk about doing,
we are going to transport ourselves over to the studio.
I am going to show you how to set up a data factory instance,
and we are going to explain the logic
of how this all comes together.
Alright everybody, so now it's answer the 'how question'.
How do we actually get this data factory thing set up?
I am in the Azure Portal,
and we are going to begin our journey here.
We are going to be on the home blade,
we can actually go and search in the Marketplace
under all services,
or if you have already used the data factory...
so I'm going to zoom in and show you,
and you searched for it recently,
may very well be in your shortcut list.
But if not, go to the search bar right up top,
start typing data factories, you will see that icon pop up,
we're going to click there
to be able to load the data factory area.
When we do that, I am going to have a data factory
that has already been created as you can see,
the AZ09 data factory,
it is a version 2 data factory, a newer version.
But, for you, this is the first time you're doing this,
it will be blank and you will be invited to create one
using the create button right there,
or the create button in blue
that shows up in the middle of the screen.
Now for purposes of time, I have created a data factory,
I am going to quickly show you what the wizard looks like,
but we are not going to bother going all the way through it.
I then want to click into the existing data factory, show you
there is really not that much we can do directly
in this part of the portal.
But, we have a link to a very important tool,
which is the design studio for the data factory,
where we actually build all of those pipes,
build all of those extract transform capabilities
that we were talking about at thelightboard.
So we can begin to really automate
and extend our capabilities using the data factory tool-set.
So let's get started, we are going to go ahead and create.
When I click on that, we get our create data factory fly out.
We have some little questions
we are going to be asked by section here.
Basics, get configuration,
which we can do now or opt to do after we set it up.
Networking, advanced, then we review and create.
This is a pretty standard wizard in this instance.
Most of the things were probably used to having provided
are probably going to be here.
Let's just center that.
You will see that we have to provide
our subscription or resource group.
We don't have one, we are invited to create a new one.
We have to provide instance details,
where in the Azure Fabric globally
we want to host our data factory instance.
The name and or version,
that's the V1, V2 that I was talking about.
V2 provides some additional capabilities.
V1 is just an older version.
If you're not sure, go with V2,
it will make your life a lot easier.
We can click through on the bottom to navigate,
or we can click through the tabs as we go.
On the get configuration area,
we have to go ahead and configure
either a Git repository
or the Azure DevOps capability that I spoke about
when we were talking about the 'what' and the 'why'.
I said you know?
We can link this to an Azure DevOps instance.
I am going to show you the DevOps portal
to show you where we get the information we need
if we do the DevOps instance.
Our DevOps account, project name, Repo name,
branch and root folder if we are going to be doing this
and or the equivalent information for our Github instance.
You will see that we changed that just a little bit.
This has to be legitimate Github
or Azure DevOps instance information.
So you need that account information.
If you don't have it, or if you want to set it up later,
just check the box and we will provide it
through the data studio once we get it set up,
we will be able to provide that configuration after the fact.
Now, if I want to use the Azure DevOps instance,
I can go to my DevOps portal.
I can figure out based on that, what my account is.
The highlighted account is the one I would use.
If I have a project, in this case, the AZDevOps1 project.
I would need the name of that to be able to configure against.
So I would get this information from my Azure DevOps...
either administrator or somebody in the DevOps team
that has control of and access to this project in the portal.
Get that information, I can transfer it right back over here.
That was the DevOps studio
by the way where we do our design.
I can simply just populate this,
or I can opt-out and do it later.
Networking, I would have to decide
how I want to manage my virtual networks,
and or whether I want to set up
a public or private end-point for connectivity.
Public end-point would be available to anybody
that can see it, resolve it using a public IP address.
The private end-point would be internal,
only available to consumers of the service
that know how to find the Vnet,
the virtual network that it is attached to inside Azure.
Just have to be aware of that and think about that,
and obviously make choices accordingly.
Under advanced, I have the ability to specify encryption
and how I want encryption to be configured for my data.
Microsoft encrypts the data by default.
Microsoft does that is called Microsoft managed keying,
or provider-managed key, MPK.
But, I can enable encryption using what is called
a CMK solution, customer-managed keying.
Where myself let's say hypothetically
is the customer or the tenant that owns the data,
and owns the data factory instance,
will take on the responsibility of setting up
and managing those encryption keys.
So if I check that box off,
I have to provide some additional information,
specifically, I need the key vault URL,
this could be a key vault in Azure by the way,
and a user assigned identity for the encryption
so I can specify,
and will need to specify both of those,
and then I can go ahead... and when I'm ready to review.
Clearly, I haven't provided any information,
so this is not going to validate.
But that's okay, you have seen the logic of what we would do.
I have done all that to configure my existing instance,
let's just click through there,
and we can see that this has been set up,
it is successful, toasted in the West US,
it's assigned to my subscription, a version 2 data factory.
All the basics are there, and there is not a lot I can do here.
I said this is really like a shell in a placeholder object
inside of Azure directly.
I do have activity logs, I have access control capabilities
to control who gets in here.
I can modify some of the settings I would have configured
with the wizard, the networking, the managed identities,
look at the basic properties.
I can lock the resource if I wanted,
preventing people from inadvertently removing it or modifying it.
I can monitor it, and I have an automation capability
to control and set up some very basic tasks.
Really it is just notification on billing.
You can see I built out a task and scheduled it to do that.
But that's it.
The real big important thing that I do here,
is not set up all the activities that I talked about
as we answered the 'what' and the 'why' here,
but rather I can connect to the thing that I mentioned,
which is going to be the Azure Data Factory Studio,
and I have a link for it right here.
When I click there,
what I am doing is opening a separate portal,
and we can see the portal right here,
it is the Azure Data Factory Portal.
Notice the URL is different,
ADF, AzureDataFactory.Azure.com.
I can then set up my code repository
if I opted out of the get repository,
or Azure DevOps capabilities and the set up in the wizard.
I'm prompted to set it up and authenticate it here,
so I can do it right from here.
Then you will see right down below here,
I can go in and follow the prompts,
and the easily clickable objects I can ingest,
I can bring a copy in,
I can copy data and bring it in, that's scale.
Or automate that on schedule.
I can orchestrate to set up pipelines but without a lot of code.
I can transform, there's the T right in our ETL.
I can configure sequel server services,
like sequel server services that allow us to do analytics,
and or to do data management, a variety of things.
I can do integration, SSI to be able to run that integration
and ultimately load that data
into my target Azure data instance,
in this case, an Azure sequel database.
So I could do that all from here, I can monitor,
although I don't have anything
currently going on with pipelines,
but once those pipelines are set up,
connected to Git, and or the DevOps instance,
I can monitor them from here, I can interact with them.
You will see that I can set them up and see when they run,
I can look at the triggering events, I can look at integration,
I can look at data flow and debug it if necessary.
So I can get a lot of metrics and actionable data from here,
and I can manage all of the different aspects
of all the capabilities;
the linked services, the integration run times.
I can look at the ability to work with
either Git as the Git configuration we would set up,
automate with our own templates,
and I can look at security down here.
All the things we set up in the wizard
are all available right here that I can work with.
Once I have set all this up,
I am targeting my Azure DevOps Instance,
I can actually then bring these pipelines in
and I can create the other connection,
the other side of them which is to come in over here.
I can actually go into my project,
and I can have my pipelines set up
and point that other end of the pipeline
from that actual data factory right here,
bring them in and tie them into Azure DevOps,
and use the DevOps fabric
to be able to create those automations,
and create an end to end solution,
that allows me not only to monitor,
but to extract transform and load,
as well as then work with that data,
and create projects surrounding it.
I can bring two services together
by using these capabilities if a chose to.
This is a high-level overview
of what we can do with Azure data fabric.
But of course, it is a lot more complicated than that.
Although this particular solution
may involve the integration of a lot of moving parts,
we talk about how to do all this kind of stuff
and a whole bunch more over at ITProTV.
You can always join me over there
if you want to follow up on this little tidbit of information,
this bite size nugget of knowledge around
how you can better use and utilize Azure,
and all the services and capabilities associated with it
to drive value in your organization.
But also to understand just what the heck is Azure
and what you can do with it.
Until next time I am going to wish you happy Azuring,
good luck integrating,
and of course make sure you check us out,
and let me have the opportunity to spend more time with you,
and teach you about all the cool things you can do beyond
on what we do here.
I will see you soon everybody, take care.