>> Today on the IoT Show, we have
Manish from the Azure Data Explorer team,
who is telling us about how to use and
why using ADX in a context of an IoT solution.
[MUSIC]
>> You're watching the IoT Show,
I'm Olivier your host.
Today with Manish, we will talk about Azure Data Explorer.
Manish, thanks for joining us on the show today.
>> Thank you for having me here.
>> Awesome. So Azure Data Explorer,
you will tell me a bit more about what it is,
but first before you jump in there,
who are you, and what are you
doing in the Azure Data Explorer team?
>> Sure. So I'm a PM in Azure Data Explorer team,
and I'm here today to give you
an overview of Azure Data Explorer more in IoT context,
and how customers can take advantage of the technology
to build rich IoT analytic solutions around it.
>> Love it. So we have
plenty of solutions and applied to the analytics,
and they are specialized for certain types of analytics.
In that case, we'll dive into how we can do
this rich analytics on the IoT data from devices.
>> Sure.
>> Tell me more about the principles of Azure Data Explorer,
and what this PaaS service offers.
>> Sure. So it's
an interactive analytical service for fast-flowing data,
so it helps customers to cut down
the time-to-insights by integrating
with those streaming sources like IoT Hub, Event Hub, Kafka,
and big data pretty quickly in,
in just really large data in a really short time,
as well as provide
a rich interactive query where
customers can query billions of records just in seconds,
and so on, and build rich analytics on top of it.
>> Okay.
>> The service is also fully managed,
so customers don't have to worry about
index maintenance and auto-scale.
It supports auto-scaling, so
[inaudible] it can auto-scale customer scale.
It's a very flexible tool.
Like adopt according to your workload and customers
can switch between computes when they need it.
We have pretty rich interactive language that helps
them to do advanced time series analysis,
and do other kind of analytics on top of our platform.
>> Okay. Tell me about
the types of patterns then that Data Explorer is used into.
So you're talking to us about IoT Hub,
Event Hub, like streams of data.
Like in that diagram that you have here,
you're showing us these kind of
patterns, tells us a bit more about that.
>> Yeah. Sure. So it fits in
an aura of real-time analytics pattern
where you have data coming in,
and you'd use Azure Stream Analysis
for doing continuous streaming analysis,
enrich your data as the data is flowing in,
and use Azure Data Explorer for doing
more Ad hoc and on-demand analysis on a really large-scale data,
and you purchase this based on your business needs and so on.
Then we have a rich set of integration with
other big data services that helps you to read the data,
integrate with other big data systems,
and build rich analytical solutions.
>> Okay. Awesome. As an IoT guy
I'm going to ask you: what are the use cases?
When do I need to use,
or when should I use Azure Data Explorer?
>> That's our next slide. So Azure Data Explorer is
optimized for querying over structured data like Time series,
unstructured data like Text data,
and semi-structured data like JSON and XML.
So today we're just going to focus on IoT use cases.
Mostly in IoT use cases we come across like Time series data,
and a lot of customers have JSON data that they want to bring in.
In some scenarios where you have the sensor data,
plus they also bring in some telemetry data
from controllers and so on,
and merge them together and do rich analytics.
So typically, use cases would be customers
building their own and IoT analytical solutions,
where they need full control on their data sources,
data schema, data management,
and analyze Time Series
and telemetry data using our built-in capabilities,
and I'll show you some of these in other demos,
as well we have built-in machine learning capabilities and
integration with the big data systems like Spark and Data Lake,
and so on, that makes it really powerful.
>> Yeah.
>> Also where our business model is
more based on the compute that customers is using,
and they can also take advantage of reserve compute,
so it's very suitable for
really large workloads where they can
get pretty good discounts when they go and get the [inaudible].
>> Yeah. Basically the cost is adapted to the amount of
compute they're going to use basically thanks to Data Explorer?
>> Yeah.
>> Got it.
>> It's amount of compute,
it's amount of data, and so it's combination of couple of things.
>> Yeah. Which an IoT scenario can grow really fast?
>> Sure. Yeah.
>> Totally.
>> Yeah. Customers building
Multi-Tenant SaaS solutions where
they have their own ingestion pipelines,
they have their own visualization layer,
and they're looking for a data platform where they can
host data and then query in a really low latency.
So those are the really common scenarios where we have
customers going across using us for other scenarios as well.
But as an IoT context.
>> Yeah. In these kind of use cases,
we need to pound on the notion of platform.
Not turkey, but platform solution.
Because we are basically implementing
all the plumbing for them to focus on their value-add.
Which is these algorithms to extract the insights,
and the insights that they're going to do
and take action with, right?
>> Yeah. So pretty much we have
connectors to all the relevant technologies,
which helps customers to build their solution pretty quickly,
rapidly, as well as with very low maintenance.
>> Okay.
>> So let me show you some demos.
>> Yes please. We're here for that.
>> Yeah. Okay. So in order to get started,
customers will have to provision a cluster in portal,
create a database, and ingest their data.
So they can use our SDKs and APIs to backfill.
Many customers have historical data
sitting in Blob Storage or Data Lake they want to bring
in and connect the real-time sources like IoT Hub, Event Hub.
So we provide those deep built-in integration.
Like in here, this is
my demo cluster where I have bunch of databases,
and customers can easily configure the ingestion pipeline here,
and connect to their choice
of system and then bring the data in easily.
So assuming my cluster is created,
my database is created,
and I've started ingesting data,
I'll show how customers can do rich analytics.
For this demo I'm using our web UI,
but this is one of the experience
that customers can use for doing Ad hoc analysis.
But there are other experiences like integration with
Grafana, Power BI, Notebook,
if you have SDKs, APIs,
REST APIs and so on that customers can use to get data out.
>> Nice.
>> So this particular dataset is NYC Taxi dataset with
about 1.5 billion rows, very small dataset,
this contains trip information for customers
taking yellow and green taxi in New York City,
and it has information about pick up and drop location,
and how much fare was there,
and how much tip they paid, and so on.
So we'll do some analytics on this.
Again, I'm just showing you a subset of the capabilities.
Its capabilities are pretty quite large.
So the first one is when you have this data,
you want to do a quick analysis on the data.
So in the first query, by the way,
I'm using our web UI,
and the way you start querying is type the table name,
and then start writing your queries.
The Intel RealSense in our tools
guides the user with the right column to select,
right syntax to select and so on.
So very rich powerful query language called KQL.
>> Nice.
>> So in this case, I'm querying my table called trips,
which has about 1.5 billion rows,
and then filtering for a specific timeframe,
and creating a Time Series dataset,
and then rendering as a time-chart.
So these render commands are very handy to do
a quick analysis right in the Explorer itself.
So here, we can see there is a certain trend,
there are peaks, and spikes,
and depths, and so on, then there's a trend.
So let's say I'm running this business,
I want to understand what's going in my business,
it's very rich thing here.
Now, as a human eye,
we can see that there are peaks and dips here,
but let's say I want to do
an automation and detect the true automation.
Let's say it could be a sensor reading or
something which is going anomaly and so on.
So detecting anomaly is pretty easy.
So I'm using a similar query that I'll just ran,
but just extending that query by calling
this additional operator called series_decompose_anomalies,
and we'll just send that same data through that function,
and here we detected those anomalies.
I can tweak those anomalies based on certain thresholds,
and then build alerting and
automation around it to go take action.
So all this is built-in.
Customers don't have to take data out of the system,
and also it supports single time series,
as well as multiple time series.
For example, let's say you have
certain device in a certain location,
room, building, and so on.
So you could just extend this query to say anomaly by room,
by location, and so on.
So you can do multi-dimension queries. Yeah.
The next one is your customers
can do regression analysis as part of the system.
Again, we're just using the built-in capabilities here.
Now here, we saw there was a trend,
and then it spiked and certainly started declining.
So this jump, you can detect this with
this operator where there's
suddenly something changed and the values change.
So this was a regression.
This is a two-fit line.
We also have one-fit line which just shows you a linear relation.
>> Each time is one linear in your query.
>> Yeah.
>> That's actually super [inaudible].
>> It's just extending less likely.
Now, understand there is a decline in the taxi ride,
and this decline started around 2014.
Any guess on what could happen that time?
>> Yes. No.
>> No? What happened?
>> So this private app-based rides
started coming up during that time.
>> Do we say a name now?
>> Yeah. So we want to see
what those private app-based rides are causing,
what's the impact of this,
and do an overlay of that.
So for that, I have another dataset called FHV_Trips,
which is about 500 million rows.
So now, I'm running a query here
which will union the data from these two tables,
run these, aggregates on this 1.5 billion rows,
400 million rows, and
then we'll see a trend on what's happening here.
So we're filtering data for a time frame then creating
aggregations and then creating
those aggregations in a seven-day buckets of window.
So now we see that it started declining.
>> That was pretty fast. We didn't cut that. It's all live.
>> Just six seconds.
>> Okay. So now you are aggregating the
two and showing them on the same chart.
We see the growth on one side and the decline on the other side.
>> So this is pretty powerful.
In an IoT use cases,
you may also see here there's one sensors started declining.
What happened to the other sensors? What happened to it?
Can you get some data from telemetry and so on?
So you can do the comparison here.
>> That was pretty solid.
>> So all the all these queries are just running within seconds.
Lastly, now we found that they're growing pretty fast.
You can do time series forecasting.
Again, that's built-in.
We're going to go and forecast for
next 28 days and see what's happening there.
>> How much trouble?
>> It's going to cause?
>> The company, yeah.
>> So that's everything within 2.6 seconds.
Again run through, aggregate that other data,
created a forecast model for us.
Now, this is my original data and then this is my forecasted data.
>> You see the trend continuing.
>> The trend continues month or month and so on.
So I just showed you
some of the built-in capabilities to do time series analysis.
There are also other ML algorithms that help
customers to auto-cluster within the platform.
If these capabilities are not enough,
we understand that machine learning and these area is pretty huge.
We also have a Python plugin,
where customers can create their custom UDF.
So in this case, there's another dataset,
but I've created a custom UDF and embedded my Python code.
I can just call that custom UDF in my query.
So in this case, this is trying to detect
occupancy based on certain values but this running Python code.
So this is my custom UDF which I have defined here,
which has Python code in it. I'll just run that.
This is running on the same nodes where the data is very fast,
and it can also take advantage
of the distributed nature of the platform.
So this actually, we got the data,
send it to the Python engine,
and got the results out.
So this is very useful in extending
the capabilities if the existing capabilities are not enough.
The final thing I want to show you is there are scenarios where
customers have to cache or bring data into Azure Data Explorer.
So it is cached on SSDs that's why the queries run really fast.
We saw that within a billions of records just few seconds.
But there are scenarios where customers have
some historical data sitting in data lake,
and they would like to go and query that.
So we have the concept called
external tables through which
customers can define an external table,
point it to a Blob storage or data lake,
and then just query with the same language.
So as a user,
you do not have to search tools,
and you're just changing the table name.
So in this case, we're going to go and query external table,
TaxiRides and just let's run this query.
This is actually going and seeing how many count off
cab types for the specific time frame.
This particular query actually went to the data lake,
got the data, and then you can call this aggregates.
So those are the ones.
So we looked at all the rich powerful capabilities that we have,
and I definitely encourage
customers to go through the documentation.
We have lot of content here as well as
one final thing is very attractive
to especially IoT customers is the integration with Grafana.
So we have a built-in connector for Grafana that helps you to
write queries and go and connect to your database,
and do build some rich visualization on top of it.
So the one that I'm showing you right now is actually
pulling in data from one of our demo data set called GitHub.
We have been populating this data live by calling
public GitHub APIs and getting
public repo information and then
just just trying to get this as fast as we can.
So you can see now the data has been refreshed like
two seconds, 10 seconds latency.
So latency pretty low.
We've seen this in real time.
These are all built in. So let's go and look at some query.
So I showed you some IntelliSense capabilities.
All those are available in Grafana itself.
So when you type queries here,
you'll see the same IntelliSense capability built into Grafana.
>> Pretty straightforward.
>> So yeah, in the short time,
that's what I wanted to show you today.
We have more to cover, but
we'll have more sessions around that interface.
>> That's great, Manish. So do
you want to bring back the slide where you have the
resources links so we can actually
invite our audience to go learn more?
So we have links here for the product page, for the documentation.
We got to add them in the description.
If you want to get faster onto the documentation,
you can go to aka.ms/IoTShow/ADXdocs.
So that's going to be your short link for getting into the docs.
Then you have these tech community as well
where you can engage in conversation,
get in touch with the product team.
There's going to be answering down there if you have questions.
Manish, great insight into
Azure Data Explorer that I didn't know much about.
>> Thank you for having me here.
>> Thanks guys for watching the IoT show.
Hope to see you soon. Don't forget to subscribe.