>> Hello, I'm Lara Rubbelki and today on Azure Friday,
I have Jean-Sebastien Brunner who is
a PM on the Azure Stream Analytics team.
He's going to walk through some of
the great new updates that we
are releasing and delivering with Azure Stream Analytics.
So, Jean-Sebastien, what would you like to show us today?
>> Hi, I have a lot of feature to show today.
We recently announced that
we have a few feature from time management.
For example, we have a new feature called substream that
enable to show different timeline
for different IoT device.
>> Okay.
>> We also have a feature called
Session Windows, completely new,
and few new things coming on
IoT Edge with Stream Analytics, now running custom code.
>> Oh, that's fantastic. So, before
we get into the different features,
give me a quick summary.
What is Stream Analytics?
>> So, Stream Analytics is basically a product
that allows you to run
large-scale analytics for your real-time data
and in a very simple way.
So, we first developed it for productivity so you run SQL,
everything is managed for you, and you
can basically start a pipeline in less than five minutes.
>> Great. So, you're basically able to just
start running SQL queries over data that's in motion?
>> Exactly. After you click on
"Run", and they run 24/7 for you.
>> Okay. Well, let's see what you've been
announcing at the recent Build conference and since then.
>> Okay. First, let's talk
about the advanced time management
and the substream feature.
But I just want to show a little context
and because we are talking about
time management and time distortion
I need some slides for that.
So, I want to show you, for example, in the world of IoT,
you can see the ideal case is
all my device are synchronized,
exactly same time on the clock at the milliseconds.
Basically, the device, data
arrive in the cloud at the same time for the device.
Looks wonderful but, of course, that never happened.
>> That's not real world.
>> What happened actually instead of having
this nice aligned clock is you have
this time skew and basically all the device send the
data at their own pace. Basically,
if you want to make any decision here,
you have to reorder these events. What it
means is you have to wait for the events.
What you will see is some delay
because you have to wait for
the latest clock basically, and
then you can take decision and send some alerts.
>> Typically, when you're using ASA or
Azure Stream Analytics to issue some kind of an alert,
you want to be able to issue
the alert as soon as possible.
>> Exactly.
>> What you're saying is we've had
to wait for all of that data
to be synced up to ASA to make that alert happen.
>> There's some case, you have to because if
you want to combine the data of all these device
together and make a query that
is cross-device, you would have to.
But if you just want to narrow it on one device,
why not just looking at this specific clock?
That is new feature and the beauty of it,
it's just one additional keyword here.
You add the "OVER" keyword.
You said, "Oh, you partition your
device," and you will be
able to look at them independently.
So, another cover, we basically run as many queries.
It's more efficient than that your number of device.
It's just look at one beautiful simple query.
You can see there's no alerts.
Basically, whenever it's arriving in the cloud,
we run the alerts based on
the time clock of this particular device.
>> That's great. Very useful.
Very useful. Okay, and what else do you have?
>> So, we also introduced a new window.
So, before, we only had fixed time windows.
So, we have like the tumbling windows,
sliding windows, and hopping windows.
Now, what we have is
a new windows called Session Windows.
It's super useful when
you actually don't know how long will be your windows.
I'm thinking about if you want to do
click Stream Analytics or log analytics.
Or in this example, we are looking
at Twitter streams just to
see how many tweets we have on particular topics
and we want just to group them by identity.
So, when there's a gap,
we just close the session
and we wait for the next one to come.
In term of retail and click analytics,
you can imagine all this and now you can do.
Again, it's as simple as the older windows.
We just have to group by a keyword here and your keyword
called "SessionWindow" and nothing else new.
It's just the same kind
of group by and windows you know from before.
>> Okay, great.
Great. All right,
what else do you want to show us?
>> We have a new feature on the Edge.
>> Yes, yes. So, talk to us
first before we start going into the Edge.
We've been hearing quite a bit around IoT Edge.
What is IoT Edge?
>> So, IoT Edge is the ability to run
some analytics on devices and when I say devices,
could be my laptop but it's more like IoT device like
a connected car, on a manufacturing floor,
you want to run on some robots or you want to
run an offshore rigs on some boats.
Basically, whenever you have
some limited connectivity or
you have so much data that
you cannot send it to the Cloud.
>> Right. Right. So, you want to maybe do
some of the analytics closer to where
that data is being generated and that gives you
that ability to lower the latency and doing that,
query and analytics but also sustain some of
the occasional connectivity concerns that you might have.
>> Exactly.
>> Okay.
>> Then you use the cloud to reuse the power of
the cloud to process
a large amount of data coming
from thousand or million of device.
>> Okay. With Azure Stream Analytics,
we've had the IoT Edge
available for some time in preview and
Azure Stream Analytics has been
available virtually from the beginning.
>> Exactly.
>> Okay. So, talk to me,
how do you deploy and manage
Azure Stream Analytics on the Edge?
>> So, it's very simple. I'm going to show
you the main steps.
So, basically, as you know,
you can create a new job in
the portal of Stream Analytics and we
just have an option to choose between cloud and Edge.
>> Okay.
>> So, you can choose and then the workflow is exactly the same
creates your input, your output, your query,
and it's exactly the same language.
So, you write the SQL
queries and if you want to run it on
the Cloud because you
know there's not too much latency issue,
you can run them when you realize it may
need to be closer to the device,
you can take this query and run
it on the Edge on the container.
>> So, you virtually had the same Azure Stream Analytics
that you can run either on the Edge or in the Cloud?
>> Yeah.
>> That's great.
>> We continuously feature
here because this is a feature in preview.
We also built, for example,
we had a custom code feature.
That's very interesting because when you think
about the world of manufacturing,
there's a lot of complex messaging coming
from the robots and the machine and we want to
be sure that people can use some library or
can use some complex code to
understand what happens there,
process it, also use
all the power of the SQL from Stream Analytics
and send the data to the cloud for
further analytics or training machine learning model.
>> Okay. Okay. What else would you like to show us then?
>> Actually, can I actually show you how to do that?
So, actually, I'll show you we can create a job in
the Cloud, but actually, we have
a plug-in for Visual Studio.
When we talk about custom code,
I think it's probably the best way to show it to do that.
So, I just have a project here in Visual Studio.
It's a Stream Analytics project.
Basically, what you can do is to test
the projects and deploy it when it's ready.
I just have some data
from some robots and I just
want to run and select start just to start.
I can run locally, I can be
completely disconnected for deployment I don't even
need to be connected to Azure or I can
start even without connecting to my account.
It just runs the same kind
of algorithm we have in the cloud.
What you can see, I just run the query here,
is basically the output's here of the sample file.
So, I just did a select star on some data coming from
some robots, and you can see it's very hard to understand.
We have some data like X,
Y, and some data seems to be encoded.
Actually, what happened here is
the manufacturer of the robot have a library to
understand that and help you
to take some decision based on this data.
What we did in this project is
actually importing the library from
the robot manufacturer and you can see I have here
my C# class project
that have a library coming from my manufacturer.
I can also use that library in my own code.
So, very simple code here but,
opening the right one, this print is a little small here,
but just using that library, understand everything.
Basically, what you can do here is to understand when we
need an adjustment on there because
we start to have some problem.
We also can check the health of the robot,
check when the battery is off,
it need to be changed.
Everything is based on that library
with little customization
from the particular factory
because different robot they need different code.
So, you can adjust every one every for the whole word.
You can set break points,
test all your query before actually
running it to the Edge or running it to the cloud.
Maybe let me show you what it looks
after, just the full query.
So, I already type it here because it
should be a little long but here,
I just call my custom quick function.
As we have in the cloud for just quick function,
it's just one called UDF,
that the name of your function,
and then you can extend the SQL with
the function you wrote in C#.
Then you can run it locally,
it runs the kind of miniature Stream Analytics engine,
and then you can see the results.
We decoded all the information,
added our customer logic and we
can see that the robot needs
battery service for
a consumption check or arm adjustment.
So, using custom code and
customization from this particular factory
to understand the robot.
But the beauty of that is now, it's tested.
I can actually deploy to several device in a few clicks.
>> Okay, great.
>> So, I can deploy to the Cloud here. But I
want to show you the experience in IoT Edge.
Basically, if you go to the IoT help,
you have this new IoT Edge device blade here.
What you can do, I have few device here but just to show
you how easy it is to deploy Stream Analytics job.
I don't have thousands of device so
that I would just do it on one device.
So, I just need to click here.
But it should be a very similar
for thousand of device. It's exactly the same.
You just target your device and it's as simple as that.
So, I already deployed this one here but I
want to show you how easy it is to add this job.
For example, I click here and
I just click on "Add Stream Analytics Module."
So, it's fully integrated in the portal and
then what you can do is just to select the job,
select the job I just created.
You click here, it's creating an API,
creating a docker container for this job,
and sending all the information
to this deploy mode for IoT Edge.
>> Okay. So, it's sending it out to the Edge.
>> Not exactly. No, I need to click on "Next,
next ", but basically, that's what it's doing.
So, here, I can click on "Next",
specify some routes for my device,
here just have some simulator,
and after I send the route to
my ASA module, I click on "Next."
I can always see exactly what the deployment look like.
Actually, when I click on "Submit",
that's when it sends
the deploy mode and my module to all my devices.
Here it was one but imagine if it's like a full factory,
it could be hundred of thousand of robots.
>> Thousands and thousands.
Exactly. Okay. Great, is
there anything else you wanted to show us?
>> So, there's some feature we have
some quite a while now and I just want to remind
people because we keep improving
it and I want to talk about Geospatial.
So, basically, in Stream Analytics,
in addition to time windows,
we also have native Geospatial support.
So, we can have queries that are based on
distance or you can
trigger some events when
something goes outside the boundaries,
geofencing and everything.
Again, it's completely integrated
within the SQL language.
So, it's always few lines
of code to design this kind of powerful query.
>> In that similar syntax of
SQL that we have with all of the other ASA queries?
>> Exactly.
>> Great, great. Well, thank you. Thank you.
It was really great learning all about
Azure Stream Analytics with you today
and some of the new features and
deployment options that we have with IoT Edge.
It's very exciting to see that stuff emerging.
Where can we go to learn more?
>> So, we have our website.
It's Stream Analytics on Azure as well as
a documentation and if you listen to our Twitter feed,
we always post in new data over there.
>> Yeah, with lots of great tutorials and
samples and people can go learn more,
just pull down those samples and start playing.
>> Yes.
>> Great, great. Thank you very much.
>> Thanks a lot.
>> We've been learning all about Azure Stream Analytics
today on Azure Friday.