>> Azure Data Explorer and
Azure Synapse Analytics recently announced
a new integration which makes it easier to
use Azure Data Explorer to analyze your data.
Learn more in this episode of Data Exposed. [MUSIC]
Hi, I'm Anna Hoffman, and welcome to this episode of Data Exposed.
Today, I'm joined by Manoj,
a Program Manager on the Azure Data Explorer team,
Manoj, thanks for joining us today.
>> Thank you, Anna, for having me here.
>> Yeah, of course. Before I get started,
can you tell us a little bit about what you do?
>> Yeah, I'm a Product Manager of Azure Data Explorer team,
focusing on some of
the Synapse integration that we have been working on.
I wanted to show you
a quick demo for
the new thing that we added in Synapse this semester.
>> Awesome. Thanks, Manoj,
and it seems like you're
the perfect person to be talking to us today because we're going
to be talking about Azure Data Explorer
and some of the integrations with Synapse.
Now, before we get into
those new integrations and
the really cool demo that I know you have prepared for us,
I love if you could just give our viewers a brief introduction
of what Azure Data Explorer is and then
get into some of those new stuff and integrations.
>> Sure. Azure Data Explorer is a fast,
fully managed service on Azure that helps
customers to do analytics on a really large volume of data,
especially data streaming from IoT devices,
infrastructure applications, web blocks,
fixing data, log and telemetry data, and time-series data.
It's specific for these scenarios.
Customers can get alerts,
reports, dashboards, and on top of that.
It covers a wide variety of data including text data, log data,
some time series data,
semi-structured data like JSON,
it can handle all these three type
of variety of data at a really large scale.
>> Okay. Cool. It seems to be
a lot of different things you can do with it,
it's a specific language you use.
I mean, it's Kusto query language,
and you also have some new integration to talk about.
I'd love to hear more about them.
>> Sure. On this slide,
I'm showing a small picture here.
Think of a scenario where you
have customers who are sending their telemetry data,
sensor data, or any kind of other log data and that's
coming into Azure and that they wanted to do alerts,
reports, and query that data.
These are very typical basic scenario on the data.
The way Azure Data Explorer would help in this use case
is customers can stream
the data into any of these sources on Azure.
Then ingest that into Azure Data Explorer
where the data is cached,
indexed, and readily available for querying at really low latency.
Now, this is good for doing reporting,
alerting, dashboarding, ad-hoc queries.
But we have been hearing from customers that there are
scenarios where they want to enable batch analytics on the data.
They want to retain data for a longer duration on
our Data Lake and
try machine learning training on top of the data,
do a long-range analysis on their archived data.
For that, we've enabled capability where we can constantly
export data out into
a storage where the data is stored in Apache format,
partitioned, and readily available for running Big Data Analytics.
On top of that, customers can use
Azure Synapse to access their data and the Data Lake.
Specifically, we're going to talk about
Apache Spark within Synapse.
What we have introduced is a Spark connector,
which is built-in into Apache Spark
in Azure Synapse that tells you to connect data.
That helps you to connect to the raw data or the live data
that's coming into Azure Data Explorer from Synapse so you
can access both like you can go access
the cold data sitting in the Data Lake as well as
the fresh data from the Azure Data Explorer hard cache.
>> Cool, that sounds pretty interesting.
Now, from what I understand though,
I can use any Apache Spark service
to connect to Azure Data Explorer.
Why might someone choose to use this integration with Synapse?
>> Yeah, that's correct.
The connector that we had is
open source and can be used in any other Spark distribution.
The special thing that we have done in Azure Synapse is deeply
integrated so your credentials are
managed within the Synapse workspace,
using machine MSI authentication,
using your AD Pass-through Authentication.
It's all native experience and
you will see it once I show you the demo.
We're giving more native experience within
Synapse so that all personas,
especially data engineers and data scientists,
can actually interact with the data in their choice of the engine.
>> Awesome, cool. Well, I love
to see an example of this thing in action,
if you have something.
>> Sure. I'll switch to the demo.
So we are in Synapse workspace.
I've already created a workspace and I'm directly jumping into
a starting place where
how you can connect to your Azure Data Explorer cluster.
We assume that you have already
provisioned Azure Data Explorer cluster,
data is flowing into your databases,
and then you want to access that data
from Apache Spark in Azure Synapse.
The way to get started is go to the Linked,
push in under the data hub,
click "+" sign, connect to the external data,
and this is the new external data connector that you've added.
Continue, and then basically fill in the details.
It can sense it from the existing subscription or you
can enter manually and specify
the cluster that you want to connect,
specify the service principal ID and the password and the secrets.
Then you also can share
those secrets into a keyboard and make it more secret.
Once the service is created,
you can access that within
the Linked section here under Azure Data Explorer.
I've already created a couple of
linked services here as you can see,
and I can browse from here
and directly interact with
my data which has been streaming in here.
In this scenario, I'm showing you I'm connected
to a database called Occupancy where
I'm getting some data in near
real-time around thermostat readings and so on.
I want to access that through Spark and use some of
the Spark capabilities that data
scientists or somebody want to use.
As I mentioned earlier, it's a deeply integrated experience.
You saw it created a link service and data and then
you can reuse it in another places in Synapse,
and right from here, you can jump to create a notebook,
Spark notebook, which gives you the sample code
and it's pre-populated and
you're not specifying any credentials here.
It's coming from the linked service.
Then you can just start from here.
This is really a quick way to get started.
Let me show you a notebook that I've already created here.
This is a very simple way to connect
to Azure Data Explorer clusters,
connect to the database,
specify the query KQL,
and then just run it,
especially you have to select a Spark pool here.
You can see within few seconds,
you have your data and then you can analyze it through Spark.
It is a very simple scenario,
but assuming the data comes into
DataFrame and you can do some rich stuff here.
The key scenarios where customers is using this is accessing
the archive data from Data Lake and unsync that data,
pushing it into Azure Data Explorer
for their operational reporting,
or dashboarding and so on, and various scenarios,
or if they're doing some analysis where
they want to merge the data from cold which is
there in the Data Lake and also get the data
from Azure Data Explorer and do that analytics.
This is a very quick way to do it.
>> Awesome, cool. This seems like really easy to use.
It's using notebooks, which a lot of people are
getting more and more familiar with.
In the example, you showed,
very quickly, you were able to return just a few rows.
But what if we had, say,
a lot of data we wanted to return,
like millions of rows,
is that something Azure Data Explorer can handle?
>> Yes, definitely. Assuming the data
is in the hot cache on Azure Data Explorer,
this connector is optimized for making
large data movements and millions and terabytes of data,
there are additional options that we expose that are
available in our GitHub sample repo,
where you can put it in a forced distribution mode through which
it uses an export mechanism under the hood that
can run in parallel and it doesn't
have the restrictions that generally you will have
running a single query in an ad hoc way.
It does support distributed mode which
can scale to large amount of data.
>> Awesome, thanks so much, Manoj.
Manoj, as we close out this video,
you mentioned a GitHub repository,
is that where you think users should go to get started?
>> Yeah, definitely. We will include that as part of
the video reference materials.
So we also have a Get Started link
and the documentation that we can share.
Before you go, I just want to show you one more thing here.
A common scenario is we looked at the query.
We also have a way to push that data so you can read data from
Azure Data Explorer as well as push it
into Azure Data Explorer or stream data.
All these examples are available here,
where you can just create a sample one.
For example here, I'm reading data from Azure Data Explorer,
think of like you mentioned about reading last data.
Let's say I'm reading last data,
and then after that,
I want to go and do some data processing in Spark.
This is a very simple one.
It's just grouping by, but it could have a complex scenario.
Then after that, you just do a DataFrame dot write,
and right from here,
you can push data into
Azure Data Explorer as simple as
writing four or five lines of code.
Similarly, you can read from storage and then push
that data into Azure Data Explorer very easily through Synapse.
Then take these notebooks if you want to automate it.
There's many different things that you can
do once you have this connectivity.
>> Awesome. Thanks so much for showing us this,
Manoj. It was really useful.
Seems like it's really easy to get started.
We also put some notes in the description for you
to learn more and dive into this.
If you like this video,
please give this video a like.
Let us know what you're doing with
Azure Data Explorer and the Synapse integration.
We hope to see you next time on Data Exposed.
[MUSIC]