[MUSIC]
>> Hello, everyone. My name is Sudhanshu Vishodia.
I'm a Software Engineer at
Azure Cosmos Database Team at Microsoft.
My role involves assisting customers implement
highly scalable and performant applications in Postgres.
In today's demo, I'm going to show you how you can
build highly scalable IoT application with millions of
devices on Azure and why you should
choose Azure Cosmos Database for Postgres to build it.
Before we move further, let me quickly brief you on what IoT is.
IoT basically is a huge network of interconnected devices which
are capable of transmitting data
over the Internet or private networks.
Sensors, gadgets,
appliances are some of the common examples.
Let us first understand the characteristics of IoT devices.
IoT devices typically require scalability,
high throughput, high concurrency, and low latency.
Since IoT devices are growing at a really rapid pace,
so while designing an IoT app,
it becomes crucial to choose the right technology only,
which can keep up the pace and allows
the system to grow without
affecting its performance over the time.
The term heterogeneous basically means the ability to
support a variety of devices from different manufacturers.
Since the data generated
from different devices would be different,
and that is where semi-structured datasets comes into
play and we'll talk more about it in the coming slides.
So why Azure Cosmos DB for Postgres?
It's a great platform to build
time-series IoT applications on Azure.
To start with you get a great performance in terms of injection,
throughput, concurrency, and the
millisecond response time for your real transaction.
It offers you native support on sharding,
which gives you the part to paralyze
queries across the worker nodes.
The smaller shots, you get better index and
gadgetry or time series workload like IoT,
the shards can be further broken into
smaller subjects using range partitions on time.
This would allow you to further prune the data for performance.
Cosmos database for PostgreSQL also meets
your scalability requirement of IoT workloads.
Since there is no upper limit on
the number of worker nodes that you can provision,
which basically means you can scale it to
millions of devices without any problem.
Also, it supports online rebalancing.
No more downtimes whenever you add new notes to your cluster.
It also allows you to store data for a variety of devices
having different schemas together as JSON, JSONB type.
This gives you a flexibility to store and query
both structured and semi-structured data together without losing
any of the SQL functionalities that comes with
the relational databases like
giant look-up indexers, foreign keys, etc.
Cosmos database for Postgres also supports IoT applications,
which need geospatial data.
It is a reference architecture,
which explains the data flow of the events,
right from the point of data instruction,
builder time they are consumed.
The complete lifecycle can be divided into five phases.
First, the devices or
the sensors will collect the information from
their physical environments and
the series of events are transmitted over the network.
Azure IoT Hub will then intercept the incoming events
and route it to Azure Stream Analytics before the processing.
IoT Hub do not process any data on its own and just
acts as a Hub for
bidirectional communication between
your IoT application and the devices.
Azure Stream Analytics pretty will help
us in extracting the required data fields and implement filters.
Basic aggregations and sample the events for
business requirement before loading
this data into Azure Cosmos database for Postgres.
You can also use Azure Databricks or
Kafka as an alternative to Azure Stream Analytics.
Once the data is ingested in Cosmos database,
all the complex data processing,
the business logics can be implemented at this stage.
The processed data will then be used to build
meaningful metrics and is exposed to the end-users for action.
What does demo a data flow has
been set up as explained on the previous slide.
Currently, we have already registered one million devices,
including thermostats, coffee machines,
and wind mills on Azure IoT Hub.
We have to use IoT Hub device provisioning service
to provision the devices at this scale.
These devices are being powered
by Simulator Program at the backend
which generates a throughput of
roughly 45 million events. I'll repeat this one.
These devices are being powered by
assimilative programs at the backend which
generates a throughput of roughly 45 million events per minute.
These events will then follow the shortest stream analytic slope,
and we ingest it into the Azure Cosmos database for Postgres.
Similarly, read through these simulated using
Postgres VG bench extension and it turns
the desk with a variety of 3D patterns like single select,
Cosmos slips, updates, and delete transactions.
Azure Cosmos Database for Postgres has been
used as a target with one coordinator and 16 worker nodes.
We have also developed a couple of Barbie dashboards,
the first one tries to capture
the meaningful metrics around IoT devices.
While the other provides a snapshot of
the database performance in terms of right throughput,
scale and the query response things.
Let us quickly jump to the demo and show you
how the data is modeled in Azure Cosmos database for Postgres.
Lets us first have a look at the cluster configuration.
For cluster configuration,
I'm going to run a query on digital snowed here.
As you can see, I have a good loss in 70 nodes here,
but of which 16 are worker nodes and one is the coordinator node.
Like I said, we are using
a 16-worker node configuration for this test.
In this demo, we are ingesting data into the events table.
The table definition shows connection device id, partition id,
id as structured data, and payload,
which is a JSONB type field for semi-structured data.
The events with different data formats are being loaded as JSONB.
Event processed UTC time is for tracking the time lengths.
Additionally, we have also implemented partitioning on
even processed UTC time
before they pun the data for improved performance.
Cosmos database for Postgres provides
APIs to automate the time-based partitions and here is
one example wherein am using create
time partition functions to create the partitions in future.
If you look at the index here,
this index was created to improve the performance of the queries
which build on even processed UTC time and the device type.
If you have a look at the index carefully,
you'll see the beauty of Postgres.
Wherein it allows to create composite indexes on
a combination of structured and semi-structured data sets
from the JSONB fields.
Even stable here is distributed on device id,
which is the natural dimension for IoT workloads.
At last, let's have a look at how events table
is sharded into multiple smaller chunks at the backend.
For that, you can query either underscore tables.
As you can see at the backend,
the main events table has
been distributed into smaller chunks called shards.
If you look at the size of
the first four partitions that we have already loaded,
you see that we have already loaded around
four terabytes of data into this table.
Here is a dashboard which brought
a few metrics around the events for
different IoT devices that we have.
The first solution here is lots of variation in average, minimum,
and maximum temperature for
different thermostats with respect
to time in form of a line chart.
We have a similar graph here of the machine,
which represents similar parameters.
Here, we have a graph for pop windmills.
This basically represents the total number of rotations
with respect to time and these graphs,
the good thing is that these graphs are
being refreshed in a real-time basis.
You can reference the dashboard to get the latest feeds.
You'll see the dashboard has refreshed.
In the second dashboard here,
you can see we are ingesting at the rate of around
750,000 images per second into our table,
which has 15 billion events already loaded.
The size of the table is
roughly four terabytes as I've shown you before.
And with the composite primary keys,
the indexes that it has and with all of that,
we are still able to achieve
a very good throughput on Azure Cosmos database of Postgres.
On the other hand, if you'll see that
each load is being run at the concurrency of 200 users.
We are still able to achieve
less than 10 milliseconds response times for
all of the different transition queries that we are running.
Whether it is update,
delete, the different variance of select.
The important point to note here it is
all the read transactions are
running in parallel with the ingestion jobs.
They're just ingesting data into a table,
but just having 15 billion records in it.
So you see that skill?
That is just incredible.
This was what Azure Cosmos database Postgres promise to
deliver for building
scalable IoT and time series applications on Azure.
To summarize the talk,
we saw Cosmos database for
Postgres capability to handle
high intuition and low latency workloads.
How native data distribution helps in achieving
the skill required for IoT Time Series applications.
We also saw how JSONB could be used to store data with
different schemas from variety of IoT devices and at last,
the SQL expressiveness which allows us to query
both structured and semi-structured data together.
With that, I'll wrap up this session
and I hope you would find it useful. Thank you.
[MUSIC]