[MUSIC]
>> Hi, this is Steve Michelotti
of the Azure Government Engineering team.
I'm joined today by Jean-Sébastien,
Principal Program Manager on the Stream Analytics team,
and we're here to talk about Stream Analytics
in Azure Government. So welcome.
>> Thanks.
>> So why don't we start out by telling us a little bit,
what is Stream Analytics?
>> Sure, Steve. So let me give you a few scenarios.
So for example, if you have some data,
you want to take some real-time decision like food detection,
like when people are paying in store with a credit card,
you really want that decision to be made
very fast, in just milliseconds.
You don't want people to walk out of the store
and five minutes later say, "This was a fraud.
We should cancel the transaction."
People are already out with the goods they just purchased.
>> We need a real-time detection.
We don't want to know
five minutes later, we want to know right now.
>> Yeah. >> Okay.
>> Because five minutes later it's too late.
There's lot of other scenario like
when you are loading a page on the web,
for example, there's a lot of things that need to happen.
Like if you are looking for cross-sell,
up-sell product supported by people who bought that,
you want to load that at the same time you load the rest.
So you only have about less than 20 milliseconds to do that.
Another scenario like if you do IoT scenarios,
Internet of Things, if you detect some engine that get too hot,
you have to stop them immediately.
So you have only few seconds to react.
We also think about warm scenario.
So scenario I mentioned before were hot part scenario,
but you can also think about scenario about a minute or more,
and after you go more to Batch Analytics.
>> Okay. So we have the IoT scenarios,
certainly relevant for government scenarios,
and you mentioned warm path,
hot path, those as well.
>> Yeah.
>> Okay. All right, cool.
So let's keep going. Yeah.
>> So Stream Analytics is part of
a big Azure family set of
products where you can connect to IoT Hub and Event Hub.
So that basically how you get the data flowing from devices,
or other programs, other business applications.
>> Okay. So should we think of Stream Analytics like
the traffic cop when we're dealing with
IoT or is that not the right way to think about?
>> Traffic cop or even more.
You can do several set of analytics.
You can just do simple things like just filtering aggregation,
but you can do very complex analytics including machine learning.
>> Okay.
>> You can bring your own model and as the data flows,
you can actually take decision based on historical data,
and score in real-time the data that is flowing.
>> Okay.
>> So we are talking about millisecond latency,
and then you can decide if you do action in the route.
So basically, you can just trigger some symptom,
some texts or something like this,
or feed the rest of the Azure ecosystem
like you can feed to a Data Lake for the Analytics later,
to Cold Path Analytics,
SQL Database, Cosmos DB and much more.
>> So real-time analytics, sophisticated,
machine learning, if you need that.
So lots of possibilities?
>> Yes.
>> Okay.
>> I can see this is how it can fit into
a sample architecture, for example?
>> Yeah. >> Okay.
>> That's really the simple diagram,
but you can take several feeds and join the data together.
So it's not only one data fit.
You can actually take something from IoT Hub.
You complete it with something from
Event Hub so the data is flowing,
and you augment it with data addressed.
So for example, historical price and things like this,
and you can then take decision based on that.
>> Okay. So I can have multiple data
feeds flowing in to Stream Analytics,
and I can configure to mash it up the data
however I want analyze in real-time and make decisions?
>> Yeah. >> Okay. Great.
>> So that's a simple diagram.
The important thing about
Stream Analytics is it just run everywhere.
First, in the Cloud in Azure.
So we have 27 datacenter including Azure Government,
but also now we run an Azure Stack.
So basically, you can bring your own hardware
and you have the full Azure running there,
Stream Analytics runs there,
so it's very good for privacy
also if you want to run closer to the data,
and we go even smaller on IoT Edge.
So think about running on the machine,
on the car, everywhere.
So you can pre-process the data.
You can remove all the noise or all the data that
doesn't need to be sent to
the Cloud so you can save in bandwidths,
or you can also make sure you remove
any private information so you can be 100 percent compliant.
Then in the Cloud, you can do
some multi-device analytics and you use the power of big data.
>> Okay. So we have the compliance,
we have the power of the Cloud,
but we also run at the Edge,
Azure Stack and its scenarios like that?
>> Yeah.
>> All right. Cool.
>> So a lot of customers are asking,
so why do you need
this product compared to other product open source?
So here, I just have all advantage summarized in one slide.
The first thing is it's very easy to use.
So it's a simple job service,
so you don't need to provision from cluster
by the machines or grade, we do everything for you.
It's very productive because we have a very simple SQL language,
and we have all the tool so you can do
the authoring of the porthole in Visual Studio,
VS Code so you can choose your own environment.
We have zero code integration.
I will show that in the demo in just a sec.
>> Okay. >> There's about 15 of their products
in the diagram before I show you on tab,
SQL, but we have about 15 of them.
>> So if I'm someone who's comfortable SQL,
I can be relatively productive with Stream Analytics here?
>> Yeah. In a few minutes, you can have your first job running.
>> Awesome. All right.
>> Its enterprise-grade, so we have [inaudible] ,
and also we guarantee repeatability.
Whatever happen, if you stop and restart,
we have all the same results.
So it's completely available,
or if you're on mission-critical pipeline,
you can have two pipeline in parallel with the exact same results.
We guarantee no data loss.
Whatever happened, we will guarantee that.
At least once processing, because we
guarantee that we never lose anything.
As I've mentioned before,
Cloud Edge Stack, so we run near the data,
wanting it to run as a product with exactly the same language,
exactly the same NET Form factor but exactly same query.
>> Okay.
>> You don't need to learn anything else when you deploy the Edge,
you just can take the same Query and say,
now, I have too much data, I want to
move that from the Edge to the Cloud.
Let's just take that and then add
it to run on that at scale.
>> Write once, run everywhere.
>> Yeah.
>> Okay. All right.
>> Recently we were feature in Forrester as a leader.
So when you look at Streaming Analytics,
there's a lot of competitor.
But I just wanted to highlight that this product
is coming from years of research from Microsoft research,
implemented in streaming schematics
and now we are clearly there in the space.
>> Okay. Cool. All right.
So with that context in mind,
why don't give us a little introduction
to the demo you about to show us?
>> Sure. So let me show you the diagram.
So first, the context is I have some data flowing.
It's a data from New York taxi.
So we have the trips coming.
They'll tell you where the trip starts
with the latitude, longitude,
and where the trip stops,
how many passengers are there,
how much is a trip and such data?
>> So real-time analytics on the taxi data,
where the trips are going,
how much they're costing, that thing?
>> Yeah. Then we send this data to Event Hub and Stream Analytics.
Among the data is neighborhood data.
So imagine the shape of every neighborhood so we
know where the trips start and stop.
Instead of having just a number from latitude, longitude,
we know this starts from SoHo
and finishing in Chinatown or something like this.
>> Okay. So it's the way I should think about
this as the IoT devices are sending
the messages to Event Hub and
Event Hub is sending them under Stream Analytics?
>> Yeah. Actually, Stream Analytics is taking to the Event Hub.
>> Okay.
>> We are a full-model. So basically,
depending on what you need to do
with the data, we take the latest data,
or you put the two hours windows,
we will put two hours of data
to meet the analytic version.
>> Like a subscription model?
>> It's not really a portal, it's really like
a full-model where we will read as much data as we need.
>> Okay.
>> Because sometimes a code can be very simple,
you just want to analyze one point in time.
>> Yeah.
>> But sometime you will realize, I want to compare this data
to a very long windows of data.
So we will take more data for this case.
>> All right. >> We automatically do that.
When you start the job, we look at the query and say,
I need two hours of data or two days of data,
and now I start to put the data run on.
>> Okay. Makes sense.
>> But that's the first step, and after,
I send the data to SQL database and to
Power BI and to a Data Lake.
So we can have multiple outputs,
and depending of the need, I will choose one of these.
For real-time analytics and dashboarding,
I will use a Power BI.
For long-term archival, I will use
my Blob Storage to Azure Data Lake.
>> Okay. So Stream Analytics is doing the analysis and
then it's making the decisions about where to send the data on to?
>> Yes. In my query, I will show you the way to do it.
I have several steps and say,
this one maximum calculation or
this one, we just want to keep it for later.
>> Great. All right. We'll see it.
>> Last step I will show is basically we are
producing all this historical data
and we want to make sense of that.
For example, at one point of the day where my taxi should be.
So we can take few years of data and run a big batch query to
calculate what should be the normal behavior
or normal pattern for a given day,
given hour, given neighborhood.
>> Yeah.
>> So you can detect what is abnormal compared to normal,
and send some alerts and more driver to
take advantage of additional passengers.
>> Okay. So we can see anomalies in
the data and things that we didn't expect?
>> Yeah.
>> Okay.
>> So that basically is a setup
and I can show you they are in the Azure portal.
So here I'm in Azure Government portal.
I've got all my resource already here.
So it's very easy to access.
>> So this is a dashboard you setup
on the Azure Government Portal,
so for your specific project,
you created this portal so you can
click on other resources easily.
>> Yeah. I've already created everything before to save some time,
I will just show you what I did in this metric job.
So first, this is the overview but I
just want to look at the input.
As I show you right before in the slide
that we have 15 input and output,
and it's zero code. What's a zero code?
Basically, you just have to click here and
choose one of the resource you already have.
You don't need to write anything, you just say,
"I'm in this subscription,
and I want to choose the 70 while
the data is flowing," and you click, and you save.
>> So this the event hub that we mentioned?
This is the input data coming in?
>> Yeah. Here everything is great because the job is running,
so we made sure that you cannot break something that's already
running because some people are
running very important mission critical things.
So when it is running, we make sure you cannot make a mistake,
and modify things, and break your customers.
Then you can do that for several input,
here I have also the historical traffic data,
and also the reference data for
the region I mentioned so the shape of the neighborhood.
>> So some of the data coming in might
be real-time data coming in,
and the other data sources are
reference data that you might use to help make decisions?
>> Yeah. Worlds and neighborhoods will not change very often,
the historical data can be recomputed with Spark,
and you can update it every day, every month.
So this one is slow-moving data,
and then everybody is kind of almost fixed data.
Then I can show you the output for this one,
I'm sending a few things to SQL database,
and every data lake for long term archival,
and then probably I will take the data from here.
>> Okay, so that was already mentioned before
the real-time, and what we see in Power BI,
and the long-term archival data lake or SQL whatever we want.
>> All the magic happening this query,
so again it's great because it's running but we have
a rich authoring experience here,
you can see the color, we have some advice here to tell you,
"This query are you sure?"
To make sure that people know what happened here,
and we give you some warnings on hinting,
if you are doing that the most efficient ways to do that.
So it's very rich offering experience right in the portal,
and we've Visual Studio AVS quite an addition to that.
>> But I can see it just looks pretty much just like SQL
so it can be pretty productive relatively quickly.
>> If you look at the query I can give you
more and more information but what we do for example,
here I'm taking my data from Event Hub,
and I'm basically cleaning the data,
only choosing the data I need
making sure there's no error in the data.
Sometimes if some device have some issues they
will give you some couple of random latitude,
longitude that cannot be possible or
instead of a number they give you some text or something.
>> So it just cleans it out,
so it flows nicely.
Later, you should do some analytics,
you don't want to ever bend the data.
>> After what I do,
actually I also added the location of the Microsoft Store,
and so you can see how many pick up
are happening next to Microsoft Store.
So maybe or drop off,
if a lot of people are coming here I get my stuff
ready to welcome more people.
So it's literally this few lines here,
I can do as Geospatial Analytics here
by calculating the distance between the arrival point,
and the other location here.
So we have unbounded Geospatial Analytics,
here I have a keyword for special analytics here,
so it's easier than asking
the people to do all the math by themselves,
and the same for the neighborhood.
So if I look at the neighborhood data,
the within, basically I'm
checking if my point is in one of
the neighborhood I have from the list of neighborhood in New York.
Basically I can also do more simpler things
or just a tumbling Windows to
aggregate over the minute level because we just have so much data,
if you want to find the dashboard that could be too
much or even sometimes if you want to save it,
at some point two years from now you may
not need to have detailed data.
So aggregate is the minute level,
and you can see it's also very
simple again just like these three lines,
where just aggregate, and grouping by my window.
>> Okay, so real-time analytics, decision-making,
aggregation, you kind of have it all in this query language.
>> If you really don't want to do anything with the data,
this is even simpler,
I just wrote it with this code here.
I just take everything from one location to the other location.
So everything is there all my input,
all my output on my query there,
and the job is already running,
and accumulating some data here.
>> Got it. All right.
>> So I can test it because I
have a SQL Server Management Studio here,
so I can just test the device coming here,
you see here I have 500 rows just loaded from my trips.
So you can see the shape of the data,
I can see the number of trips,
the average passenger because the aggregate the minute level.
So average passenger for this minute in all New York,
the average time for each trip,
and then every minute I can generate a new window here.
So everything looks to be good,
I can also do a Power BI is mentioned here.
So basically,
I build my dashboard here based on that data in SQL Server.
>> This is just PowerBI Desktop,
and pointing to the Azure Government
Location Stream Analytics you were just pointing at,
and this enables us to just
easily build the dashboard to this location.
>> So I can take all the data that I created here,
and mashup with all the data you already have in
the data center or in SQL server rules.
I can see my data is flowing here,
I can explain you what I'm doing here,
to exist the data in
real-time here in black and for example in this graph,
and then my historical pattern here.
So I see we were lower than the historical pattern here,
so maybe the taxi company can say at
this point maybe we have too many people on the road,
we can have more break or something,
and finally we reach that pattern here
where actually that's close to
the normal pattern so people may need
to go back to the current situation.
Here we have the result by neighborhood.
So I broke down with my geospatial query,
I broke down all the data I have by neighborhood,
and to know whether actually it should stop.
So I can see some neighborhood are much more
active at this time as in other neighborhood.
So you can combine the data in many ways,
and you can also publish real-time stream datasets,
I can show you when it's live actually.
Power BI kind of did that in the real time,
so you can see here the data flowing.
>> So here we're actually seeing movement.
We're actually seeing the real-time data coming in,
and we're, because you publish this to PowerBI online?
>> Yeah.
>> Okay.
>> So this one is the streaming dataset,
and I just put the data here on
the x-axis just for five minutes so we can see real movement here.
>> This really brings home the power of
the real-time analytics because we
see the data coming in in real time.
>> Absolutely.
>> All right cool. So this has been
a great introduction to how to get
started with Stream Analytics on Azure Government,
we saw the input sources,
the stream analytics doing the analysis, sending the data,
and then seeing how we can output this
with real-time reporting as well with Power BI.
>> Okay, great.
>> Thank you very much. This has been
Steve Michelotti with Jean-Sébastien from
the Stream Analytics team talking about
stream analytics on Azure Government. Thanks for watching.
[MUSIC].