[MUSIC].
>> Hello. Welcome to this session.
We'll be talking about building the future of
time-series data using Cosmos database.
I am Mandanna, I'm a Principal in
the modernization practice of Infosys.
The modernization practice in Infosys structure is
a horizontal working vertical line of businesses.
This gives us a unique vantage point to work with
various enterprise customer problems across the verticals.
Today we'll talk about one such customer channel.
The agenda for today is to talk about
the introduction to the use case.
How Cosmos database has supported us for this particular use-case.
We'll get into the details of the data model and the design.
We'll also talk about the challenges and
optimizations with their design or usage.
We'll talk about the outcomes and
the key considerations that we had in this implementation.
Introduction to use-case.
We have used Cosmos database with Cassandra API as
a time series to every IoT,
that is IoT product implementations.
In a typical IoT implementation,
you would have what we call as an edge area which,
largely deals with the IoT gateways, the sensors.
It's all connected and largely deployed on the field.
And then you have the Cloud area where the data from
these devices are consumed and processed and stored.
Then you have the app section where the data which is stored,
is further taken and processed to make certain decisions.
Now, we'll talk about the scale,
the query patterns, and the key requirements
that we had for this project.
From a scale perspective,
we had over 5000 IoT edge devices
that are going to be deployed across the globe.
Each of these edge devices were further connected
to say three bunch of sensors.
The idea was to each IoT device would further
transmit the data and overall scale was expected to
be having around two million observations
per minute at any point of time during its peak performance.
That is the scale that we were looking
at from a very patronage perspective,
we were largely a light-heavy product.
We had most of our use case was
about ingesting a lot of data and the reads were very minimal.
For each of the queries, you would always have the tenant ID,
the sensor, and observation details handy.
Most of the use cases were also around,
quering the recent set of data
and not really too much of a history.
From a storage perspective,
the key requirement was to have
a managed distributed database which was globally available.
Considering the design of the data,
we wanted it to be a time series date.
It is something which supports time series data which is
scalable and potentially useful
and support cross-platform as well.
We had use cases where you could
potentially scale the product into
multiple Cloud providers and
also you could take it to compromise if need be.
A database that we wanted
should consider all these scenarios
and to support these scenarios.
Now we chose Cosmos database.
We'll talk about why we chose it
and what made it very special for us.
Cosmos database has various APIs.
It has Gremlins, SQL,
Cassandra, and others.
Cosmos being the Cloud-native database,
it is a fully managed database.
Microsoft takes care of the
underlying infrastructure and maintenance
ensuring that the consumer has zero operations overhead.
Cosmos database is also one of
the key services which is available across most of the regions,
wherever it is deployed,
ensuring that you have a global presence.
Building an application and consuming Cosmos database
with Cassandra API ensures that the underlying storage
and the associated customer application is portable.
Meaning if you go ahead and set up a new Cassandra cluster
in another Cloud provider or in any location
the product would work
seamlessly because the underlying implementation
with the storage is exactly the same,
except for the connection source, of course.
Cosmos databases can also provide you with
the unique option of multi-tenancy,
and the concept of
request units makes it an ideal solution for us.
I'll discuss more in detail about
the multi-tenancy and the ideas in further studies.
In this level,
we'll talk about the overall data model and the design.
But like I said in an IoT solution,
you have what we call Azure edge area and the Cloud area.
If you look at the top section,
there are sensors and
these sensors are connected by your edge devices,
which further pushes the data into IoT Hub.
From the IoT Hub,
we have a bunch of micro-services or any computer which
can process the data and finally push the data into
this storage in our case, Cosmos DB.
What you see in the middle section is basically the design
of the Cosmos Database.
So within a customer's account,
we had the keyspace and within a keyspace,
what we have is a customer's table is what I have done as tenant.
Within a tenant, we were capturing the sensor ID,
the observation name, and the timestamp.
A combination of these three would be the primary key.
The timestamp itself was the cluster key.
The idea was when you put the timestamp as the cluster key,
you will cluster all the data from a particular timestamp together
and also ensuring that the cluster key
is ordered in a descending order.
Ensuring that the latest data is always on the top.
The same table structure was repeated for
each tenant ensuring that you could then
you could segregate your customer's data into various tables and
the Cosmos legacy gives you
that unique opportunity to scale at each tenant level.
What you could do is you could
scale and allocate IDs for each tenant
independently and that's something that
I have represented at the bottom to see if you consider.
>> Now this design had one drawback though.
The primary key that I mentioned about sensor ID,
the observation, and time stamp.
This meant that at any point of time to query this data,
it was a must to have the sensor ID and observation name.
Now you could have scenarios where a particular query
requires up to get all the latest data from a particular sensor
without necessarily passing the observation names.
We don't have an optimal solution for such scenarios.
To handle such scenarios one of the ideas was to
put these or store these observation names in a persisted storage,
something like an Azure table.
In case you need to query
all the latest data from a particular sensor,
when you don't have the observation names,
you'll query the Azure table to get hold of
other observation names and then take that
further to go back and query
the database to get all the required data.
The other challenge that we noticed was with each tenant,
there was a set of RUs that were allocated and
whether there was queries fired
against the Internet or not in spite of that,
there will be at least 10 percent of the RUs which are always
allocated no matter whether there was consumption.
It was getting charged straight.
It was unused capacity.
To handle this,
the idea was, can we optimize this?
How better can we optimize it?
One of the implementations that we rendered was to move
the RU allocation from the tenant to the key space itself.
Meaning within a key space,
you allocate a certain set of RUs and
whatever tenants are available under the key space,
everybody would share the same set of RUs.
This will ensure that if one particular tenant is
not really utilizing the complete capacity,
the other tenant could consume it.
That makes it mostly a well-optimized solution per se.
Now that we've discussed about the problem, the scale,
I want to talk to you about
the key outcomes and what are the considerations that we have,
and how you could utilize customers database.
Like I said, the idea was really to test for around
1.5-2 million records pushing
into the database on a continual basis every minute.
These messages varied from around 8-10 KBs,
below the custom register application
that which could constantly pump data into the application,
which will process the data and further push it into the database.
Each charged message had around 10-50 observations,
which were constantly getting pushed.
Now, when we tested for the scale
of over an hour and a couple of hours,
what we noticed was always
P95 messages were getting
stored within the database in less than 10 milliseconds,
and we were getting the response in less than 10 milliseconds,
which is what Cosmos promises.
That's something that we noticed as well.
Then there were around P99 messages
were in the scale of around less than 20 milliseconds,
which is usually expected.
The RUs were allocated at the key space level,
and like I said, we moved into manual allocation as well.
We saw that there was around 140K RUs were actually getting used.
Even though we had allocated a little bit more higher,
140K RUs is what we noticed as actually getting utilized.
What that means is moving the RU allocation
from tenant to key space is a key thing that we need
to do just to make sure that it is optimally utilized.
Otherwise, the RU allocation was a bit more higher.
Also scaling from automated
to manual is something that we need to
consider based on the use cases.
Meaning, if you are expecting in your use case,
if there is going to be an unpredictable flow of data,
then you will want to make sure that you are in automated.
But in that case, we knew
the number of tenants that are coming in.
We knew the number of devices that we are adding.
We had an idea of,
what is the scale going to be?
We could either on a gradual fashion scale as required.
But if your business has an unpredictability of load,
then you would probably want to do automated scaling.
Cosmos also allocates the RUs
based on not just on the consumption,
it also considers based on the underlying storage as well.
What it means is,
if your database has
stored more and more data than you would want,
there are more of RUs that you want
to allocate it'll be a bit more higher.
To handle that you will want
to have an archival strategy as well,
probably to study data.
If it's not the required,
you would want to keep it in some other storage.
Like in our case, we went and installed the historic data into
a blob storage and use [inaudible].
Cosmos really works very well for a low time-series database.
To really reap all the benefits of Cosmos database,
one should continuously revisit
the RU allocation and find your speech part on what works for you.
With that, I think I've wrapped up my session. Thank you.
[MUSIC]