Hey friends, Microsoft Dataverse,
formerly known as Common Data Services,
a SaaS data service in the
Microsoft Power Platform.
But did you know that you can explore
your Dataverse data in Azure?
Synapse Analytics?
Josh, Sabin, and Priyanka are here
to cover Azure Synapse Links for
Dataverse - Today on Azure Friday.
Hey friends,
I'm Scott Hanselman and it's Azure
Friday. I'm here with Josh Kaplan
to talk about Microsoft Dataverse.
How are you? Good,
Scott, how you doing?
I'm doing very well - excited to learn
about new things so this is formerly
known as the Common Data Service.
Yeah, so Dataverse has been
around for a little while.
Uh, formerly known as a Common Data Service,
and I'm from the Azure Synapse team
and we wanted to make the two kind of
come together and talk to each other.
Very cool,
very cool and and how is Azure
Synapse being used typically?
What's a person out there
using Azure Synapse for so?
Azure Synapse is analytical platform so
you can think of it as your cloud scale.
Data warehouse also brings with
it all the everything you need to
kind of build out that analytics
solution including all the all
the data transformation pieces,
the data prep pieces brings with you,
distributed SQL pieces and also
you can actually run Spark
Analytics across it as well too.
And this is one of the tools.
In the toolbox of someone who is going to be,
you know,
having a data pipeline of like huge
amounts of data moving through a
system and then analyzing it in
that system rather than giving
you individual tools they have
to stitch together yourself.
We try to give you one service that
has everything in it and they all
kind of work together and monitor
together closer together so it's more.
toolbox as a service
than tool as a service,
the service solution.
This service solution that's very cool.
So with Sam's link for dataverse,
diverse actually is another
service inside Microsoft.
It actually I have a little illustration
to show it's actually the common store for
all Microsoft business application data,
so you can think everything
coming out of Dynamics 365.
Everything coming out of Power Apps and
you can imagine that's pretty useful data.
But you know,
like,
just like I said,
with Synapse you want to have
one solution that has everything
in it all stitched together,
and so you don't,
so you don't have to get it yourself.
We're finding that to get the
data out of dataverse and in the
Synapse you were then going to have
to stitch it together yourself,
and we saw Dataverse actually gives
you a nice easy way to get all your
your business data out into a data lake.
Within we see people building pipelines,
extracting that data and immediately
putting in a SQL database just
so they could run some queries or
getting these custom connectors
to go and try to use spark against
that data. And a lot of manual steps.
So with sampling for dataverse.
We tried to remove all those steps,
so basically the idea was we could
have a business admin inside dynamics
could within a few clicks and able to
get engineers and data scientists to
collaborate and work over that exact
same data directly with instant apps.
It's amazing how low the barrier
to entry has gotten.
The things that we used to do
manually just a few years ago are
now really is available to everyone.
Yep, try to get it down to just a
couple of clicks and I'm sure next
will try to get down to no clicks.
Very cool. Bring Sabin in
here from the Dataverse team.
He can show you how it's done.
Let's do it.
Come on in Sabin.
OK, hey so to further elaborate
on what Josh was explaining.
Like you know,
I'm sure the question is like tell
me more about data words like
where does the data come from?
Like that you're pushing it to Synapse,
right?
So if you if you look around
like if you all the customers,
most if not all are in the
digital transformation phase,
so they're trying to digitize data
and that's where Dataverse comes in.
Dataverse is part of this Power Platform
strategy, like Power Apps,
Power Automate, Power BI.
And Dataverse is central to it,
so Dataverse let's you very
easily structure business data.
So shaping and modeling the data,
putting it in tables and our
entities as you call it,
so that is at a very natural,
more Dataverse.
And it lets you define business logic.
So all these business applications sitting
on top of Dataverse as a bunch of logic.
So it still lets you define that logic,
apply that logic.
So that's where Dataverse comes in
and signs and some good examples of
this is if you think about Dynamics
365 for Sales,
which primary job is to help
guide the sellers?
Make your next win convert more wins
marketing app field service project service.
So these are some of the popular
Dynamics apps and also some modern
applications like the donor apps which is
part of the Microsoft for Nonprofits Cloud.
All these particular customers
who are using these apps are
storing that data in data words.
So you can think of a rich
set of relational data.
A bunch of.
Files, which is basically attachments
logs which is stored in Cosmos DB.
So to that end Dataverse
set of heterogeneous stores.
So you have Azure SQL,
you have Cosmos DB,
you have Azure Blob and of course
we're leveraging Azure Search
for doing a bunch of search.
So think of data words as a low
code managed platform managed by
Microsoft which is structuring
the business data and supporting
interconnect applications.
So as I explained you know.
Enterprises using these applications
are automatically storing their
data in data verse.
So as Josh explained,
there's always been a need for
wanting to slice and dice data.
Kind of runs, run a bunch
of AI or ML on the data.
So what we do with Azure Synapse Link for
Dataverse is let you democratize this data,
push that to Synapse so we take that data,
push it to Synapse and the super
advantage here is you know the
customers don't have to look for hey,
what tools do I need?
To process the data,
analyze the data Synapse provides your
the box a rich set of tools and we'll
see some of that in the demo here,
wherein we link the dataverse
environment with the Synapse workspace
and start analyzing rapidly.
So that's what, in a nutshell,
Synapse link for Azure is for data versus.
Very cool and I hope we're gonna be
able to see some demos and some exciting
senses of what this looks like absolutely.
So let me take you quickly
to the power apps portal,
and as I was explaining this is what
dataverse looks like behind the
scenes like so you have all account
table like all these standard table
and for today's demo I'm going to
use the account table which has a
bunch of data for customers and
we're also going to use what I call a
custom entity called Product Reviews.
And this is where we are storing.
Customer sentiments,
so to start with,
as explained in the power apps portal,
there is something called Azure Synapse Link.
On the left hand side of tab the
experience starts with clicking
on new link to Data Lake.
So as I said we are linking the database
environment to Synapse workspace
so this is in previous select that
and based on the user who is logged
in I automatically pull the Azure
subscription for the user so I can
pick and choose or I can type up.
In this case it's Synapse bug.
Bashes my subscription.
And the resource group in this
case is automatically picked
because there's only one.
Otherwise you can pick and choose.
Similarly,
I have multiple Synapse workspaces
that have already pre created,
so I'll pick one from here and it
automatically picks the storage
account associated with Synapse.
That's it. Then you hit next.
In this screen I'll pick the dataverse
table that I want to push to Synapse.
So account is one of them and
the other one is product review.
So this is a custom table,
so we have a standard table
account and a custom table,
and I can hit save now for demo purposes.
Have already pre created this profile
so I can just go back to that.
And as you can tell,
you have account and product
reviews around 9200 thousand rows.
So now here we do some nice magic here.
So instead of having the customer go
to a different portal portal dot Azure,
putting your prayers look for Synapse,
we make it very easy and seamless
for customers.
We basically stitched the clouds so
we can just the customers can go
click on the shortcut to just take
them to Azure Synapse workspace and
there I'll hand it off to Priyanka.
I will show you how there are
different Synapse analytics tools,
starting from Serverless SQL, to ADF,
to spark that the customers can use to
analyze their data versus data after you.
Thanks a bit.
Ah, so now when the data is are
made available into Synapse using
Synapse link for dataverse,
we can see here the tables and
tables which were selected or during
the export activity I can click
on the new SQL script and select
Top 100 rose to start my analyzing
to start analyzing the data.
I have connected here to the Serverless
SQL pool, and I'm just running,
just hitting the run button here
and I can get the results right
here without doing anything,
just without even writing the code.
It's a,
it's a good head start to start analyzing,
similar to new SQL script.
We have also option available for
Spark users to load data to data frame,
and they can start analyzing
data right from Synapse.
The code is populated in the
Pyspark language, but users can.
You just can't select any language
from the available options and they
just need to attach spark full to
start executing or get the results.
It's the same time I have already
executed this and here is the
result so we can see here.
This results of accountable or the rows
in the account table are available here.
To start analyzing,
users can add additional transformation
logic to get their data into
the desired shape they want.
Now when we saw this code,
first experience or we also want to
go ahead and see what about the code
free experience that user would like to have.
Let's go and see about this.
Here in the mapping data flow we are
providing the native workspace DB
connector which is which allows to
connect to the database directly.
I can see the list of tables
populated here which are available
in the database I have selected
here accountable and if I click
on the data preview I will be able
to get the results here just like
to see what the data is look like.
I I think I think additional
transformation logic on top of it.
I can get the data to the desired
shape and then export it into the
either data lake or data was database
or also the syndicator SQL pool.
The choice destination depends on users,
what they want.
We are also enabling data scientists
here so when other machine learning
workspace is connected to the Synapse
Workspace user get the machine
learning menu here and they can
use the data to train a new model
or predict with a model.
So let's see what are the product reviews
and what are the products sent in.
What are the sentiments behind those reviews?
So if we see here. I am selecting
here the sentiment analysis.
By default, all the subscriptions which
I have access to will be listed here.
I'm selecting here the Azure
Cognitive services account.
The link service Archabald
link service and the secret.
Click on continue.
We just need to provide here the text
column which contains the review comments
and click on the open net notebook.
Wow, we have here the notebook
which is pre populated with
entire code and I did not write a
single line of code for this one.
I just need to attach a spark
pool and get run it.
So I have already executed this notebook
and we can see the results here.
All the sentiments are all those
sentiments have been have been predicted
for each individual review comment.
We can also write back this data to the
database and then build power BI reports.
On top of it,
business analysts can consume this
data within power be I report start
building reports from Synapse and
also they can edit it and publish
it to the power BI workspace.
So we just saw how three personas
data engineers,
data scientists and business
analysts and work in a single
unified workspace providing end
to end solution to the customers.
So bacteria, Josh.
Thanks Branca, so Priyanka showed
what the server the data engineer now
working directly over that data versus
data using the full power of that T
SQL engine and actually notice there
shouldn't actually even have to create
any compute or provision in any capacity.
We actually were service capacity there
by just turning on sampling for dataverse.
She's actually use that full T SQL
language using her service capacity,
scale out the queries across multiple
different CPUs and get results
back pretty instantly.
Data scientists also able to
come in there use this.
Start capabilities of Synapse
directly on that same data,
so it's just really tried.
Everything down to just a few clicks to
turn on all these different possibilities.
Very cool,
well thanks so much for bringing your
friends to share this with me today.
Thanks for having us.
Alright,
well I'm learning how to accelerate time
to insights with Azure Synapse links
and dataverse today on Azure Friday.
Hey, thanks for watching this
episode of Azure Friday.
Now I need you to like it.
Comment on it, tell your friends,
retweet it.
Watch more Azure Friday.