(upbeat music)
- Welcome back to the "Azure Cosmos DB Essentials" series!
My name is Mustafa,
I am the Portfolio PMM for Azure Cosmos DB,
and this season we're joined by Mark Brown,
a PM on the Azure Cosmos DB product team.
In previous seasons,
we've covered tips and tricks for getting started,
fundamental NoSQL concepts, as well as how and why
leveraging a distributed database in your application
can improve performance, availability, and flexibility.
This season we wanna take use cases a step further
and dive into some of the architecture's customers are using
to build lightweight applications.
Today we're going to start with one of the major use cases
for NoSQL, Internet of Things, or IoT,
and real-time telemetry.
So Mark, thanks for joining us this season.
- Thanks, happy to be here.
- Before we jump into the actual architecture,
can you share a little bit about why Azure Cosmos DB
is so well suited for the IoT/real-time telemetry use case?
- Sure, so let's start with a scenario.
Let's say I've decided to build a sensor device
that can do things like monitor temperature, humidity,
carbon monoxide, and even has a motion sensor on it.
I'm going to sell this device to commercial buildings
and offer a service they can use to monitor their buildings
and do things like integrate with environmental systems
to turn AC and furnaces on or off.
Or turn lights on if someone walks into a room.
Or basic security,
if say a motion sensor goes off during after-hours.
- Right, so this sounds like a great device and service
that, say a building management company would love to use
so they can easily gather all sorts of data and insights
that let them know how their buildings are performing.
For instance, they could use the data
from the motion sensors and time of day
to know whether they can adjust the environmental controls
for after-hours, turn off lights, et cetera.
- Precisely.
We've got the device
and we need a way to capture this data and then process it
such that we can send signals back to the device
or to other systems.
We also need a way to present this data
to building management personnel.
This may be one or more dashboards
to help them visualize data
starting at high level aggregating data
from across the devices so they can get a good picture
of the health of their buildings.
Now, this is obviously very specific to this use case,
but the example can be easily extrapolated
to retail inventory monitoring,
or healthcare device monitoring,
smart home device monitoring, tons of different use cases.
Azure Cosmos DB is great for ingesting
large amounts of IoT data for a few reasons.
First, Azure Cosmos DB is horizontally scalable.
This means that the database
can continue to grow in size and throughput seamlessly
as we ingest data.
Second, Azure Cosmos DB
has an amazing change data feature called Change Feed
that allows data written to Cosmos DB
to drive event-driven applications and services
providing a way to easily integrate
data written to Cosmos DB into other microservices.
Third, Azure Cosmos DB is schema agnostic.
Schema is managed at the application level.
In IoT scenarios,
it's not uncommon for successive generations of devices
to emit slightly different data.
This schema agnostic capability
means that changes to the data being ingested
does not require changes to the database's schema
in the back end.
Lastly, Cosmos DB has a unique capability
to build analytics on operational data with Synapse Link
that lets developers build applications
with near real-time analytics on their operational data.
- Okay, so with these characteristics in mind,
lets talk through an architecture we would likely use
when building out this IoT application.
- Sure.
So every IoT workload starts with the device.
IoT devices are constantly reporting data.
They are telemetry collectors
and the frequency with which they report data
largely depends on the workload itself.
For instance, a device that measures moisture in the soil
may only send data once an hour.
A device monitoring something with real-time implications
may report multiple times a second.
A device like ours that's used for environmental data
may report pretty much every second.
Regardless of the interval in which the data is sent,
as the number of devices grows,
the amount of data grows with it.
In scenarios combining high frequency data
and large numbers of devices,
the amount of data transmitted and ingested
can grow quite large.
In a typical IoT scenario,
you would have devices that are managed and communicate
with some device management service or gateway.
Devices will be initially be provisioned and configured,
and then deployed into the field.
Those devices then emit their data back to, say IoT Hub,
which acts as a gateway,
collecting the data emitted from each device.
Once ingested, IoT Hub
may do some basic transformations to the data
and may even forward the data from the raw devices
to something like blob storage as a cold store.
On the hot path,
data will typically then go to a streaming layer
such as Spark.
Here, the data is picked up and processed in real-time
doing things like creating moving averages
and other sorts of aggregates on the device data.
From there, the data is then sent to a storage layer.
When using a database like Cosmos DB,
this data can be quickly indexed
and used to serve queries on the hot store.
Here, you may typically build things
like mobile or web applications
that can be used by field personnel
to get real-time data from specific devices
or maybe look at a dashboard
with an aggregate data over several devices
in a specific area.
In our scenario, maybe a field engineer
is seeing that a floor is empty
and opts to turn down the AC.
This is also where things like Change Feed can come in handy
if the data needs to be forwarded to other microservices
after any initial processing.
For instance, say one of our devices
has detected both increased temperature and CO2.
It's possible there may be a fire.
The microservice could then send a message
back to another service to alert personnel
or send a message to IoT Hub to turn on the device's camera
to confirm there's actually a fire there.
The fact that you can respond to data in near real-time
is what's important here
and things like Change Feed give you that capability.
- That makes sense.
So it seems in this scenario with the building sensors,
there's quite a bit data that comes into Cosmos DB.
How do we deal with all this data, right?
I just imagine that even just a small building
with a few hundred sensors,
over a small amount of time like a month,
there's gonna be a ridiculous amount of data
just sitting in Cosmos DB, especially as a hot store.
- Yeah, that's true.
So this is something we need to think about.
Cosmos DB is horizontally scalable,
which means Cosmos DB will happily add storage
as the amount of data grows.
And this is fine.
Users shouldn't have to think about
whether they have enough storage for their device readings
for that day, or week, or even that month.
But Cosmos DB stores data on SSDs
and it's often inefficient economically
to store large volumes of old data, especially old data,
in this type of data store.
You could store petabytes of data on Cosmos DB,
but if you're not actively using it, why would you?
So when planning, you should ask the question
of where should this data live over time and for how long.
Typically for a hot store,
device telemetry like this is TTL'd, or Time to live'd
over some period of time.
Now, that amount of time can vary.
In some cases, say in a healthcare workload,
the raw device telemetry
may be sent to a cold store in blob storage
and even processed data stored in Cosmos DB
might also be sent to a cold store,
maybe for, say auditing purposes.
It all depends on the workload.
Now, there are a couple of ways that users can move data
from a hot store like Cosmos DB to a cold store.
They can use Change Feed that I talked about earlier,
such that whenever new data is written to Cosmos DB,
it gets picked up and written to blob storage.
But there's actually another option
that's kind of new and pretty cool,
and this is a new capability
that we released not to long ago.
And that feature is called Synapse Link.
- So can you explain how that works?
How would you use Synapse Link
to provide a cold storage option
for data in Azure Cosmos DB?
- Right, so Synapse Link
is a way to build near real-time analytics
on operational databases like Cosmos DB.
How it works is as data is written into Cosmos DB,
the service takes that data and writes it
into a completely isolated and separate column store
fully managed by Cosmos DB.
From there, you can create workspaces in Azure Synapse
and write scripts for, say Spark or SQL
to do your analytics.
The cool part is that the underlying storage
for this column store
is the same blob storage I mentioned before,
so the cost is basically the same.
Not only that, but there's no impact
to the performance of Cosmos DB
when this data is copied from Cosmos to the column store.
Because the storage is inexpensive,
you can TTL your data off of Cosmos
and just simply leave it in Synapse Link.
Best of all, you only pay for storage to keep it there.
Unlike storing data in blob storage,
you can easily query the data.
If you stored the data in blob storage as JSON files,
you would have to ingest the data back into Cosmos DB
or some other database,
and then run queries to access the data.
- So, from an architectural standpoint,
I feel like that really simplifies things now
because instead of having to move that data
into a cold store and facilitate that process
with something like an Azure Function,
we can just point directly from IoT Hub,
straight into Cosmos DB, turn on Synapse Link,
and from there leverage something like Microsoft Power BI,
we can visualize the whole thing.
- Yep.
- So that really leaves us with what, like six parts?
Our devices, IoT Hub, Spark streaming,
Cosmos DB as a hot store,
Azure Synapse as both analytics and cold storage,
and then Power BI to visualize the whole thing?
- Yep, once the data is in Azure Synapse,
I can take advantage of everything within Azure Synapse
as a service.
I can do near-real-time analytics.
I can stream that back to Cosmos DB
as like a real-time serving layer
for things like dashboards and Power BI.
I can take that data and train models
that can then be used to do things
like anomaly detection for my devices.
I could build advanced analytics
and use that to offer additional services to my customers
to make even more money.
Best of all,
the overall architecture is simple and efficient.
- So before we wrap up this first video,
let's do a quick summary
of why this is one of our leading use cases.
IoT and real-time telemetry applications
have high data volume demand
and often need to respond to events in real-time.
Cosmos DB being a scale out database
with rich change data capture via Change Feed
is a 1:1 solution to the needs of this use case.
Additionally, all of this telemetry is collected
so that we can provide real-time insights,
do deep analytics, and identify trends across the data.
That HTAP capability of Cosmos DB
means we can do this in near real-time with Synapse Link,
and the hot/cold store behavior of Cosmos DB
means we don't need to manage an ETL pipeline
to make that possible.
So with that,
we'll wrap up the first episode of season three.
Join us in the next episode
where we're going to do the same thing
for a product catalog solution
that we see customers building with Azure Cosmos DB.
Thanks to mark for joining us,
we hope you learned something, and we'll see you next time.
(upbeat music)