>> Thank you. Thanks, Marvin. Thanks, Aspen
and Petine for that excellent overview of
domain driven design and how
organizations can adopt data mesh thinking.
I'm going to wear my customer hat and assume I'm
a technical architect at Tailwind the
company that Petine introduced in the previous session,
our travel company which is interested in all of this.
But as you can imagine, I have a lot of questions.
To help with that, I have Marvin with me from Microsoft,
who will help us understand how to get
started with the implementation aspect of this.
Welcome, Marvin.
>> Thanks so much. Thanks for having me.
Let's get started looking into how
the Cloud-scale analytics can
help you set up a data mesh architecture.
>> I have watched the previous presentations
and I really enjoyed learning about the content.
But where can I go to learn more about
this Cloud-scale analytics, can we start with that?
>> Sure. That's a really good question.
Today, we have actually, multiple artifacts
that I would like to share with you
today and that help you on your Cloud-scale analytics journey.
The first one I would like to call out is
the guidance that is publicly available today in
the Cloud adoption framework on Microsoft Talks under the short
link akm.ms/adopt/cloudscaleanalytics that you can also find here.
The documentation includes best practices around security,
governance, policies, network setup,
access control, and a lot more.
Hereby, as mentioned previously,
we consider 80 percent of it to be prescriptive and 20 percent
of it being customizable based on the customer requirements,
based on industry and so on and so forth.
In summary, this asset really provides you with
a prescriptive architecture designed by SMEs.
It's documented end to end,
and it helps you get up and running much faster
and reduce the time to value inside your data platform.
>> That's great. It's always useful to have
a manual in your hands when you're building something.
So let me ask you this.
What else is available to get
organizations like ours started at this?
>> The second artifact that I would like to share with
you today is our reference implementation.
The reference implementation that we have built provides you with
the core infrastructure artifacts to set
up the data platform such as data mesh.
Overall, these artifacts reduce the time
as every customer has to start from scratch.
You can use these to get started on
your data mesh journey much more quickly.
The repositories come actually with various deployment options,
including the option to use a deploy to
Azure button with a customized Azure portal deployment experience.
Of course, there are also CICD pipeline options,
so that you can deploy the solution using
Azure DevOps or GitHub Actions
depending on what you would like to use.
What I would like to mention also is that
all the services and all the infrastructure that is
preconfigured according to best practices
based on the experience that we had with other customers,
based on the learnings in the energy industry,
for example, pharma industry, but also office.
Lastly, I just want to emphasize
that the reference architecture as well
as the reference implementation has to be
tested and validated with customers.
But that's not also the code artifacts.
They are open and publicly available today on GitHub at no cost.
So the various architectural solutions are split across a set of
repositories and they can be found
within the Azure GitHub organization.
But that's not all. On top of that,
we have also been working on templates for data applications,
which can then be used by
customers to be able to much more quickly,
also realize value and insights inside their data landing zone.
These data application templates can then be used to
quickly spin up a use case environment
inside your data landing zone,
which can then be handed over to a project team to start
evaluating data and generate insights.
These teams can then much more quickly develop
data products which can then be shared within an organizations.
So there is a template today available for batch use cases.
There's one for streaming use cases and there's one for
analytical purposes such as Data Science and Machine Learning.
Hamood, what do you think about those artifacts?
>> That's terrific.
Thanks for sharing all these resources.
I love the wealth of information, and the infrastructures,
code in the form of templates that comes
with all of this so we can get started quickly.
Let's say that via Tailwinds are excited to get started.
How do we start deploying the data platform?
Where do we begin? Can you talk about that?
>> Sure. We should actually
first make sure that the foundation has been set up.
We are always recommending our customers to first adopt
the Cloud adoption framework and
the core as a landing zone framework.
As this will help you to set up the core infrastructure for
network, identity, monitoring, and a lot more.
That is actually required for
any landing zone and not just our data environment.
Once that has been done, we can start building on top of that.
We can start deploying the data management landing zone.
This is the first Cloud skill analytics artifact that
really is here to enable anything related
to data management, and governance, and
acts as a core instance for these purposes.
On top of that, now we can start
setting up our data landing zones,
which can be separated based on a number of
factors that were mentioned previously, like data domains,
cost allocation, location,
data classification, or other legal implications.
>> I really like what I'm seeing here.
I see Microsoft's Cloud adoption framework at the base.
I see a data management landing zone
which gives me the ability to govern my data platform.
This is perfect. Let's talk about the next steps for our company.
Let's say we now need some help
identifying how many landing zones we should deploy.
So we've identified a couple of data domains,
such as travel management,
customer management, and corporate functions such as finance,
HR, and other supporting services.
We, at Tailwind, are concerned about
sharing costs between these three functions, i.e.,
the travel management, the customer
management, and the corporate functions.
How does this pattern help us split the cost across all of these?
In addition to this, our business units will like to create
a self-service data platform so teams can develop independently so
that domains can own their data and give access
to whoever requests that data across the organization.
So my question for you is, Marvin,
how would this look like in a data mesh pattern?
How can we structure our data landing zones?
What do the beginnings of data mesh look like for Tailwind,
keeping in mind our broad range of functions that I just
described earlier and our strict data residency requirements,
given that we're based in Europe?
>> Thanks so much for providing
these super valuable insights of your business.
Lots of questions, and based on the on the provided details,
I would propose to split the data estate across three data lines,
one for the customer domain, one for the travel domain,
and one for the corporate functions.
Each landing zone will have to land into
a separate subscription for scale and separation of duties,
and it will also allow you to allocate costs
easily as it was one of your requirements.
Even though your data platform is now
split across multiple subscriptions,
you can still rely on in-place data
sharing between these data landing zones,
without having to maintain multiple copies of
the same data set across multiple storage accounts.
Each of these landing zones would then
be deployed into West Europe to be complied with
the data residence restrictions that you mentioned.
Inside each of these landing zones,
we can now start realizing value through the data applications,
and throughout automation, we can then realize
a self-service platform that can be
used by any use case team within your organization.
>> Great. On that point about realizing value,
I'm super excited to realize the value of data mesh.
As was rightly pointed out earlier by Petine,
the most important thing for Tailwinds is
the ability to sell airline tickets and travel packages.
For this reason, we would like to focus on our customers first.
We'd like to start small and scale as
needed in the future by onboarding new domains.
For now, we'd like to start with focusing on
our customer to make sure we're delivering the best value to them.
We want to start by deploying
a single landing zone where we can start integrating
our data and start building
a data product that allows us to run reporting,
to look for insights on customer behavior, for example,
and spending patterns as they purchase travel products from us.
We're also thinking of utilizing
Azure's AI offerings to build a travel recommendation engine,
so we can offer our customers deals based on
their travel preferences or even do
things like recommending local attractions to them.
I'm wondering how Cloud analytics can help us
with achieving these goals and
whether all of this sounds realistic.
>> It sounds very reasonable.
The great thing about Cloud scale analytics is that it
allows you to start small and grow your data estate over time.
Based on your feedback, I would propose to get started with
a single data landing zone for the customer domain first,
and then later on, you can extend the data platform with
additional data landing zones for
the travel domain and the corporate data domain.
Inside the first landing zone,
we can then stop focusing on the customers and
start developing the recommendations use case you just mentioned.
Should we get started with the deployment? What do you think?
>> Absolutely. Let's dive into it.
I'm really excited to see this in action.
>> Sure. Let me first walk you through the deployment of
the data management zone and the data landing zone.
What we will look at now is we will start at
the data management zone repository and we will use
the deploy to Azure Button to initiate the first deployment of
your data management zone and your data landing zone.
What we can see here is actually,
as mentioned, the data management zone repository.
If you scroll down on the weekly file in any of the repositories,
we will find those blue buttons which you can use to
initiate the deploy to Azure Button deployment experience.
This will bring us to the Azure portal and will bring
us to a customized portal environment of front end.
Here you have to provide a couple of details.
You have to say in
which subscription we should deploy the data management zone,
in which region, of course.
In your case, that's West Europe and we'll
use that as a one subscription.
Then you have to provide a prefix that
will be used for all the resources that
are being deployed inside the subscription as a prefix.
The same then applies for your data landing zone,
so here we are just deploying one, we're using ESA -2,
and we are using West Europe as a region.
Again, we need to provide a prefix and then also a password
as some of the services that we are deploying require a password.
For example, the virtual machine scale set on which
we are hosting the self-hosted integration runtime.
Lastly, you can also deploy in Azure Bastion,
which we will do here, then you can apply sub-tasks, and lastly,
there is a review and create process where we
want some final validation to
check over all the inputs are looking good,
and then you can hit the "Create" button.
You will end up with
the first data landing zone and data management zone.
These are briefly the steps to deploy those two landing zones.
Do you have any question, Hamood?
>> Absolutely. Now that I have
the data management landing zone and a data landing zone set up,
what I'm wondering is, what actually
gets deployed inside my subscriptions?
Can you talk a bit about that?
>> That's a great question. Let's actually now
look into our subscriptions to understand the setup.
That's actually what we will see now is the two subscriptions
that we have been using before for the initial deployment quite.
ESA-1 was our data management zone,
ESA-2 was our data landing zone.
We will now look at what has been deployed.
What we will see here is, as mentioned, ESA-1 and ESA2,
and there are a number of resource groups that have been
deployed for the deployment on crypto.
First we will look at the governance resource group.
In here we will find our central
purview instance which will be used for
all your governance tasks across all data landing zone.
If for example, look into the networking section,
you can see everything is
preconfigured according to best practices,
so we are disallowing public network access and also
all the services get deployed with
private endpoints across the board,
and if supported, of course.
Besides that, in the data management zone,
we also have a container registry, for example,
for Docker images that are used across landing zone,
and we also have, for example,
a couple of consumption resources here.
For example, the Synapse private link up
to also privately access the Synapse
studio across all data landing zones
and across all data applications.
There's more, but let's look into the data landing zone.
In the data landing zone, also,
a number of resource groups have been created.
Let's first look into the storage research group.
Here you will find free data lakes.
One is actually the raw data lake,
one is enrich and curated layer,
and then we also have a workspace layer which can
be used as a scratch space for the application Teams.
In addition to that, we also have a resource group for
external storage accounts which are used mostly for push patterns.
If, for example, a search in scenario requires you to push data
from source system to Azure
and requires an account key, this is what use.
Then we have a couple of shared
integration resources where we have
a data factory and a data bricks workspace
which are used for your metadata ingestion framework,
or many customers use that for that.
The same applies for products.
We have a few product resources,
and a data bricks workspace as well as a Synapse workspace
that can be shared among data application Teams.
Then we also have a few metadata resources that I used once for.
My SQL Server is used as the hive middle store,
and we have a SQL Server and SQL database
which is used for the metadata ingestion framework.
Lastly, we have a one times resource group that is used for
actually hosting the self hosted
integration one time on a virtual machine skill set,
and from here it can be shared among other data factories.
And then we also end up with a couple of
resource groups that are empty actually,
but you can start using those
for your data applications to start building
your data products and then share
these data products later on among the organization.
This is what the data landing zones
and the data management landing zone contain.
Do you have any questions about that, Hamood?
>> Well, first of all, thank you for
that walk-through of the data landing zone.
As I hinted earlier, we're looking to quickly
realize value and also we're
looking to be able to show how
this architecture provides value to my organization.
Can you show me how to build
a data application which allows me to ingest, let's say,
customer data from my Oracle database,
which is on-prem into the customer management data landing zone?
I would also like to make this data set available within
our environment for consumption by other data applications.
In addition to that,
there is already a data science team waiting to get
started on building the recommendation engine
that I mentioned earlier.
They want to train machine learning models and create
new curated datasets for other teams to consume.
My question to you is how would such a use case be
implemented within the data landing zone that you just showed us?
>> Thanks, first of all, for showing these next steps for me.
Let's walk through how this use case can
be realized inside your data landing zone.
According to our principles,
we would first split the data integration and
the data application data product development
into two architectural concepts.
The data integration would be
executed by our metadata ingestion framework,
so we would use our metadata ingestion framework to
first land the data initially
onto the wall and curated individually.
Once that has been done,
our data product team can then get started and
we can then use the use case team,
that will be able to deploy their own resources
within their own dedicated data application resource group,
and here they can then start building
this use case that you just mentioned, the recommendations engine.
But let's look again into the portal and see how this can be done.
>> Sure.
>> What we can see here, again
inside the one time research group,
we have the self integration one times and those are shared with
the data factory inside the Shared Integration Resource Group.
We will use that to actually load the data from
your Oracle database into your instance, into your data lakes.
The data lakes are already connected here to the manage phenotype.
We have private endpoints,
we have also linked services and here is
Oracle database as linked service.
Then we just have to execute and set up
some pipelines that we can then use to land the data
first initially onto the raw data lake
and then we can move it onto the enriched and curated layer.
In the portal, this looks like the following.
First we will land it here on
the wall layer and then ultimately on the end,
which then curated layer.
From here, the data application team can then pick up the data,
and they can then, inside their application Resource Group,
they can spin up their own resources,
and as they are working on a machine learning use case,
they can actually use our analytics data product template
to quickly spin up an environment.
What they will get is also machine learning
workspace and they can use that machine learning
workspace to actually get started with the model training,
with the model development and with that template,
they will also get computes automatically configured on
the VNet and everything is set up for them automatically.
>> Great. Thank you so much, Marvin.
This was a lot of great learning, I really appreciate the demos.
I'm going to take off my customer hat now and just say that,
we've only scratched the surface here and
shared with you how you can quickly get started at deployments.
We're hoping that you can use the wealth of resources that were
referenced here today to help
your customers on their data mesh journey.
Thank you, again, to everyone who joined and listened in.
I'll hand this over now to our panel of
experts who are waiting to take your questions.