>> Azure Synapse Data Explorer
recently went public preview at Ignite.
Learn all about it and see a cool demo
this week on Data Exposed.
[MUSIC]
>> Hi, I'm Anna Hoffman and
welcome to this episode of Data Exposed.
Today, I'm joined by Henning from the Azure Data Explorer team.
Henning thanks so much for joining us today,
before we get started, can you tell us
a little bit about what you do?
>> Yes, of course. Thanks for having me here.
Hi, this Henning,
I'm a Program Manager in the Azure Data Explorer product group.
In my product group,
I usually take care of features around
security and geo-spatial queries.
>> Awesome. Cool. We're all happy to have you here,
and today we're going to be talking about something new,
at least something new to me called Azure Synapse Data Explorer,
and this is something I really don't know much about.
So I'd love if you could tell myself and the
viewers a little bit about the scenarios
and what this is used for.
>> Of course, we do so.
Today we're going to talk about Azure Synapse Data Explorer.
That's a new capability of Azure Synapse bringing in
the Azure Data Explorer capabilities out there into Synapse.
It enables use cases for log and telemetry management.
Let's talk a little bit about
the key pillars of Azure Synapse Data Explorer.
It's actually the same ones
as you get for Azure Data Explorer itself,
so it provides near real-time insights on big data,
high volume, high-velocity, high-variance.
It provides you with an optimized engine
for interactive query exploration.
One of the really key pillars,
here is our intuitive query language called Kusto Query Language.
You're able to store telemetry data from logs,
custom Logs, car telemetry data,
industrial IoT data in it.
The good thing about it, it can scale from
gigabytes to petabytes with these.
Couple of the scenarios that you can implement with it
is near-real-time big data
analytics on custom events and log data,
you can build a centralized near-real-time observability solutions
from telemetric sources.
You can analyze security logs and last but not least,
also IoT analytics solution.
This is the one that I would like to
create a quick demo for you and tell you a story.
Let us think about manufacturing scenario where Alice,
she's a production manager at Vandelay Industries,
receives a complaint regarding the quality
of a batch of widgets shipped to Contoso.
The information that she needs to use in order to investigate is
scattered across multiple systems with
no easy way to combine it or analyze it.
We have data from an ERP systems such as purchase order,
we have data from a manufacturing execution system,
such as the production schedule,
and we have a ton of telemetry data coming from
all different machines on the production line.
Now, the solution looks a bit like you
put some data into Synapse and then you use,
for example, Azure Synapse SQL to ingest,
transform and store data from
this ERP systems such as SAP and the MES system,
and use Azure Data Explorer to ingest
the telemetry data from the manufacturing process and
orchestrate joint queries to reason across
multiple databases and perform times series analytics.
Last but not least,
means to leverage Azure Digital Twins to
model the entire manufacturing process,
and contextualize manufacturing telemetry.
What you see on this slide is actually two things.
First of all, you see the data integration flow,
the data integration using
Synapse pipelines for the production order and scheduled data,
and the data integration of the telemetry data at
the bottom for the production line by an IoT Hub,
Azure Digital Twins, and last but not least
into Azure Data Explorer.
Now that we know how the data gets ingested,
we can have a look at how syntax KQL query
is used to combine data from SQL,
Azure Digital Twins, and times
series data in Synapse Data Explorer.
>> Awesome, cool, let's take a look.
>> Let's take a look, right.
What you see here is a dashboard that you can use in
order to find out who did what
and what happened at the production line.
You see I selected here
the purchase order that is interesting for me, a resolution,
and then know that this has been created on the 9th of
April with production order number at a certain plant in Munich.
See a couple of statistics for it,
so something it's interesting in order
to investigate what's happening,
you see that overall there have been
a couple of purchase orders of 30,000,
some data points for the manufacturing process.
You get some telemetry data,
and what you get here as well is the data of the actual machines,
in this case, it's CNC machines and conveyor belts.
What you see here is their telemetry data
and the data that they're sending and
the red dots to indicate that there has
been definitely some anomaly.
As well, you see that on
this CNC machine and you also see it on dismissed CNC machine.
But what is really interesting is on the conveyor belts,
you see a strong indication that there has been an anomaly which
might have caused the purchase order
to be broken somehow or malfunctioning.
Having that indication,
you can use this data in order to tell the customer,
"Listen, something went wrong,
we know what it actually was.
It was some of our conveyor belt.
Sorry about this, we're going to help you."
>> Awesome. Wow, that's really cool.
I've never seen this before,
it looks really awesome and makes it really simple.
It seems like to find these anomalies
or anomaly hunt, which I'm a big fan of.
>> We can also go little bit deeper into that by
showing the actual KQL query, which is doing that.
It looks quite simple, it's just you're
using a stored function to do that.
But the stored function is actually more than that,
so in the stored functions there are a couple of
things which is actually
implemented to see here the KQL scripts has a new capability
of Synapse Data Explorer in the Synapse Studio.
This is way of executing this.
This is the data that the UI will get in the end.
But the function which is used for that is very interesting
because it's combining data from the Digital Twins,
from the ERP data and so on and so forth.
This is the function that I'm using for this analysis.
It's getting to production data from an SAP system via SQL pool.
It's extracting the interesting data.
Once it has interesting data about the purchase order,
it's clearing the Twin for the plants and
the line ID to get the data of all
of the conveyors and CNC machines that are necessary.
Now that I have the identities of the CNC machines,
I get their actual telemetry data and present it to the UI.
What you get with Azure Data Explorer in KQL,
in Synapse Data Explorer is
not only a query federation engine to SQL,
to Azure Digital Twins to time series data.
But you can also execute on top of
that time series operations like anomaly detection,
forecasting, regression analysis,
and all of that. That's really good.
>> Awesome. Wow,
this is really cool.
I'm definitely going to have to go take a look at this.
From my understanding when we're recording right now,
this is currently in public preview.
If you are going to highlight
maybe your favorite thing about the capabilities
in the public preview, what would you highlight?
>> Well, it's definitely hard to highlight
only one of my favorites.
If I want the say yeah, okay definitely,
it's the capability of creating KQL scripts in the Synapse Studio,
this is really awesome.
I like it a lot, and the other one is not so obvious,
but it's implementing all of the enterprise promises with
Azure Synapse Data Explorer that
synapse provides to our customers like Managed VNet,
data exfiltration protection, firewall capabilities.
This is something that is little bit under the covers,
but it's very important for our customers.
>> Awesome. Well, thanks so much
Henning, this has been really useful,
really interesting, we're going to put some links in
the description for our viewers to learn more.
Additionally for our viewers,
if you like this episode,
go ahead and give it a like,
comment and let us know what you're
doing with Azure Synapse Data Explorer,
and we hope to see you next time on Data Exposed.
[MUSIC]