>> Most of the time, data coming from IoT devices needs to be
enriched with metadata from other Data sources or some context.
Ashita, is here to tell us everything about
the new feature in IoT Hub called the message enrichment,
today on the IoT Show.
[MUSIC]
Hi everyone, thanks for watching the IoT Show.
I'm Olivier, your host,
and we have Ashita with us to talk about
Azure IoT Hub message enrichments.
Ashita, thanks for joining the show again.
>> Thank you Olivier, how are you?
>> Fine, awesome.
So for those who have not seen the previous episode,
shame on you, you should go and watch it.
Can you introduce yourself and what your team is
doing within Microsoft in the Azure group.
>> Sure. I'm Ashita Rastogi,
I'm a PM in the Azure IoT Hub team.
I focus on data integration features,
so making it easy for our customers to connect their devices
and make the data flow into Azure
or their own business applications.
>> So Azure IoT Hub is our gateway for devices.
When we published it first,
it was doing basic things,
allowing to securely connect devices and
ingests telemetry messages from these devices as you would expect.
>> Yeah.
>> But as our customers are using Azure IoT Hub,
they're asking for more features.
>> Yeah.
>> This new feature you're here to talk to us
about is about adding or stamping.
These messages are coming in, for what purpose?
>> Yeah. So we have a lot of customers who are ingesting data into
a single endpoint from many devices and many different IoT Hubs,
and they want to know what IoT Hub
actually send this information or they want to know
what device when tag was used on this so that it's
easy for them to post processes using that information.
>> So insta enrichment name, right?
You enrich the telemetry data and the metadata content into
the message envelope or body with,
and we'll see in details exactly what that means,
with that metadata coming from
other sources or from another reference,.
>> Right, exactly. So it's a very simple configuration to do.
All you need is basically
what is the key value for the enrichment,
sorry, what's the key, and then
what's the value for the enrichment,
and what endpoint it needs to be applied to?
All the data going to that end point
will be stamped with the same information.
>> Awesome. Well, let's look a little
bit and into your diagram here.
What is the various differences,
and what's applied where,
you were saying like the metadata will
be stamped depending on the end point your data is going to.
>> Yeah.
>> So tell us a bit more about that.
>> Sure. So just to revise a little bit about IoT Hub,
whenever you deploy an IoT Hub,
the customers get a free built-in Event Hub Endpoint.
In addition to that, you can also configure
some custom endpoints like Event Hubs Service,
Bus topics Service, Bus Queue,
or a Storage Blob,
and we have message routing as a feature that allows you to
configure routes to send data to
these custom endpoints. Enrichment.
>> Quick question on that one.
Actually I got the question very often,
I think it's great to clarify each time we talk about routing,
is it an either or,
or can you actually decide to the same message from
specific device goes to
different endpoints depending on
what you want to do with that data.
>> You can send the same message
to the same end points or different points.
So it's not a unique message going to unique endpoints.
If you want that same event message to go to different end points,
you can configure routes.
>> You can configure routes to do an or and not just an if.
>> That's right. Yes. It all depends on
what routing query you're
using in the route that you create to that endpoint.
>> Got it. Makes sense.
>> So message enrichments can be applied to
not just the events you're routing to custom endpoints,
but also do the blends in Event Hub,
and we support adding enrichments to
not just the telemetry events that are routed,
but you can also add it to device to in change notifications and
lifecycle events that are also data sources route supports.
>> Yes. Love. It seems pretty useful actually.
I can think of tons of scenarios to use that.
You have a little demo to show us how that works, right?
>> Before I jump in the demo,
let's talk about what an IoT Hub message looks like.
>> Okay.
>> So here we have a sample IoT Hub message,
where on top you can see there are some system properties.
These are properties that are standard on all idea of messages.
It's an envelope that we add to the message body,
and these are mostly populated by
IoT Hub except these two important ones.
One being content type and content encoding.
We require users to add this on their device SDK,
so that IoT Hub can understand that whether this is JSON or not.
In order for users to do a query on the body on the route,
this must be set to JSON so that we can
understand how we can parse it.
The UTF-8 should be the encoding for that.
>> So real quick on that one.
So you mentioned the fact that in our SDKs,
we're actually providing the only elements
for people to have these properties added in the messages,
if someone is using their own MQTT client
or piece of code to send messages because people can,
they have to have these properties in the head of the message.
>> That is correct. Yes. Also, as you mentioned,
in the SDK, users can define
application properties which is this group here.
These are extremely customizable,
people can say, add whatever key-value they want,
but what happens sometimes,
why we made enrichments is sometimes you don't
know the application property that you want to apply in the SDK.
It may come from the device twin or
something outside of the device.
Then the body is just body of the device data.
Here we have a sample where it's weather, data, and temperature.
So what happens in enrichment
actually is going into the application property back.
>> Okay.
>> Yeah. So in order to demo this,
I'm going to do a couple of things.
First is I'm going to route data to a Blob Store,
and then I'm going to configure enrichments,
and then I will use
a simulated device to flow
the data and see that enrichments are applied.
>> Which is straightforward.
>> So let's go ahead and add a custom endpoint,
which is of tied blob in my IoT Hub.
>> These operations you're doing while you're doing them,
they can be done using the Azure CLI,
and a Service SDK as well, right?
>> Absolutely. So you can add enrichments using
the IoT Hub Update and create
REST API using the routing properties and also using the portal.
>> Okay. That portal is great for development and demos,
but also in production sometimes it's not.
>> Absolutely. It makes it so easy.
Notice that our data is going to be in JSON.
So I'm going to click "JSON" as my encoding format here
>> Things are working the same in Avro and JSON, right?
>> That is correct. It's just up to,
JSON integrates well with many other Azure services.
So people want to format it that way.
While this running, we're going to go and create
a route to this custom endpoint.
All we have to do is pick
our existing endpoint that we just added.
Now, routing supports different data sources.
We have picked the Telemetry.
>> Okay.
>> We can opt to choose a query but for simplicity,
we are going to choose [inaudible]
>> So we are going to basically route everything.
>> Route everything and we can always add queries.
So routing actually supports querying on
not just the body of the message but
also properties and also device trend.
Now that we have that,
let's look at what enrichment looks like.
So we actually have this capability in
preview today and we're walking on a GA soon,
but you can actually apply 10 enrichments for IoT Hub tier.
So for basic and standard IoT Hub you can have 10,
for free, you can have two enrichments.
Know that for basic IoT hub,
there is no device being supported,
so you can't add device in.
If you do, it will come out as a string path.
So what I'll have to do here.
First, let me show you the device twin that I have.
So what I've done is added a new device
and I used the connection string for this in my simulation.
So here is my device twin.
All I've done here is that I added a new property called Humidity.
>> Okay.
>> I set it to 68.
So what I want is I
want the value out of this device twin in my message.
>> In the telemetry.
>> Yes.
>> Without having the device to have to send it.
>> Exactly and without having to do
the device twin calls for every device that I have.
>> Okay.
>> This value can change and I don't want to have to
change and make new calls every time a twin changes.
So this will automatically sum the latest value.
>> Nice.
>> So I'm going to go in my routes again and add some enrichment.
So let's say, I want to call this humidity
and then all I have to do is say.
>> You are missing an i in humidity.
>> Thank you. Then I'm going to add the device twin paths.
So my path here is added it in the properties.desired.humidity.
Then, I have to select what Endpoint I wanted to apply it to.
So I'm going to pick my blob.
>> So where we see it's not either or it's actually
your choice of which one you
want that specific enrichment to be sent.
>> Applied to. Exactly. Know that
any enrichment added to this endpoint will,
irrespect of how many routes you have,
every message going to that
will be stamped by the same information.
So similarly, just as I explained in the multi-tenant scenarios,
users want to know which tenant gave what message.
So some of our customers needed IoT Hub name.
So let's say, my IoT Hub name is or I can say, tenantID.
Then, all you have to do is $iothubname.
>> Okay.
>> Apply it. Users can also add static.
>> Like their own data, right?
>> Their own static strings.
>> First name or something.
>> Right. So I'm just going to say, welcome message.
>> Yeah.
>> I want to add it and I will say,
Welcome to the IoT Show and I would just get that string.
>> Okay.
>> Hit "Apply". So while that is being applied,
I'm going to show you my blob store.
So here's a very easy way to look
at the data and it's really facile.
My blob-store, I actually have my IoT Hub's folder.
This is basically a hierarchy of the date
and the file name that we gave for the blob and today being 30th,
I'm going to look into my data folder.
Here's some data eroded before applying enrichments.
So you can see that this
is actually the application properties.
So it's really empty, status is the only thing we have,
and then these are our system properties until this point.
These were standard that we have.
I want to make sure that my content type and encoding are
correct and then the body had weather and temperature.
>> Okay.
>> Location, Seattle. So what we're going to
now is run this simulation.
So here's a simulated device.
Again, I have my system properties set up.
I have my JSON body
here with weather and temperature, also have location.
>> Okay.
>> I'll go ahead one.
>> So we don't see any humidity,
commanding there and we don't see the message.
It's just the temperature sensor.
>> That is correct.
>> Data is correct.
>> Yes and humidity is going to come from Temperature.
So I'm just running this once every five seconds.
>> Okay.
>> I'm expecting this to show me new data file in two minutes.
So another thing about enrichments is that you can also
apply enrichments to data going to Event Grid.
>> Okay.
>> So we just launched device telemetry going to Event Grid.
>> Yeah.
>> You can stamp information.
This is very useful because you
don't have device ruin information,
once data is out of IoT Hub.
>> Got it.
>> Prevents doing those clans.
>> Hence the stamping the notion they have now.
>> Absolutely.
>> It's just like stamp was whatever notion,
the meta data you need to rise.
>> Yeah. Let's refresh this.
So we are looking for new information in our blob,
and here there's a new folder now.
Let's look at what data is in there.
So here, you will notice that my properties now has humidity,
a tenant ID, which is my hub name,
and then a welcome message.
>> It should be any Telemetry messages
from an IoT device. Welcome to the IoT Show.
>> That is correct.
>> Love it.
>> Everybody will know. Then, the other did the same.
So now, for a user,
they can parse this really easily and identify
what they want to do with this data.stream.
>> I love it. So if people want to learn more,
show us back to the slide that we had this aka link.
>> Yeah.
>> So aka.ms/iotshow/iotmsgenrich.
>> Yes.
>> IoT message enrichment.
>> Yes.
>> Ashita, it was a great demo of this new feature of IoT Hub.
Hope to see you soon on the show again.
>> Pleasure. Thank you.
>> I want to thank for you watching,
don't forget to subscribe and go onto
that aka link to learn more about IoT Hub message enrichment.
[MUSIC]