- Hello, everyone.
My name is Prashant Agrawal, and I'm working
as Senior Analytics Specialist SA in AWS,
where I primarily focus on Amazon OpenSearch service,
and today, I will be talking about how you can retain
more data for less using storage tiering
with Amazon OpenSearch service.
So before getting into the storage tiering,
let me take a step back and talk about what is
Amazon OpenSearch service.
So OpenSearch service is your fully managed service
that makes it easy to deploy, operate, and scale OpenSearch
along with legacy Elasticsearch cluster in AWS Cloud.
Let's take a quick detour and talk about
what exactly is the OpenSearch.
So OpenSearch is a community-driven, open-source search
and analytics suite, which is derived from Apache 2.0
licensed Elasticsearch 7.10.2 and Kibana 7.10.2.
So talking about the data, here is some numbers
where worldwide data is expected to hit
around 175 zettabytes by the end of 2025,
and out of those 175, 50%, around 90 zettabyte,
is expected to be generated only from the IoT devices,
and out of those, 80% of these dataset is unstructured,
so where 71% of enterprise reported
that unstructured data is growing way faster
than the business data.
So let's see what are some common datasets,
and how does that solve the business problem?
So when we talk about the application,
you can send application-specific data to OpenSearch service
to know if your applications are up and running,
and serving that request with low latency,
and if there are any errors which you want to find out,
or if you want to tell us of the issues
by analyzing the logs.
Then on the security side, you could use OpenSearch
to restore security logs, which could help you to identify
if there are any suspicious activity or any suspicious login
to the application, or if there are any fraudulent activity
happening on your application.
Lastly, observability helps you to get inside
into your infrastructure and troubleshoot the problem
in the realtime, which will directly influence
the overall customer experience by building
these resilient apps.
So as data grows, the cost of restoring this data
grows significantly as well.
So when we talk about log data, you might not need
all the data to be accessed all the time with low latency,
and probably, you can retain those infrequently accessed
or older data at a scale where restoring that older data
could be cost-prohibitive, and also, you cannot perform
the same analysis on the data what you have
into the hot tier, like into the recent data,
versus what you can do into the older data
because you might be running some query intensive data
you might have because older or infrequent data
is more query intensive than the recent data,
and you would want to reduce the impact of such query
so that it does not impact your production listers
or products and data, but what if you just want to restore
those data for longer duties, and maybe you might have
some degree tree or the compliance medium,
so in that case, you will still need to archive
that data for longer periods.
So let's see what is the data storage options
with Amazon OpenSearch service and how it can help you
to reduce the cost.
So basically, there are three storage tiers: hot tier.
Basically, hot tier is nothing but which enables you
to keep recent data readily accessible
and indexed in high-performance storage
to enable querying and conducting the realtime analysis
on those data.
Then, UltraWarm and cold storage helps you to keep
older data still available for analysis
in lower-cost log storage on the available storage
like Amazon S3, so where
maybe you can run reporting
on the older data which you can certainly
brought about in a JSON, or maybe if you haven't
restored data into the cold tier for longer duty since,
then you can refer the or you can leverage
the colder storage.
So talking about UltraWarm, this feature was launched
in May 2020 and really got a lot of traction
from various customers, and one of the reason is the cost
because in some cases, you can see up to 90%
of the cost when you store the data with UltraWarm,
and UltraWarm nodes optimize for caching
and then the query for indexes is stored in the S3,
and these queries are fast enough to learn interactive
log analytics as well as plotting
the visualization in dashboard.
In one single domain, you can store maximum
to three petabytes of data, but the only trade-off
between the hot tier and the UltraWarm
is that data stored into the UltraWarm is immutable,
so you cannot modify any data when it is
into the UltraWarm storage, but if needed,
I guess you can bring back those data to the hot tier,
perform the updates, and then send back those data
again to the warm tier.
Now, talking about the cold storage,
so it was launched somewhere around mid-2021,
and it decoupled the data stored into Amazon S3
to the computer link on the UltraWarm data nodes.
With cold storage, you can virtually store
any amount of data.
So as you're just out carting all those data
to Amazon S3 and paying near about S3 pricing for the same,
then you can use the policy which we call indexing statement
admin policy that can help you to move data automatically
from the UltraWarm to the cold tier,
and then there is the OpenSearch task for UI as well
where you can easily see all the indices
into different tier like hot, UltraWarm, cold storage,
and at the same time, you can use the UI
to move the indices from one tier to other.
Now, the only trade-off with the cold storage
is that in this tier, data is not searchable,
so you will have to get S3 data to UltraWarm nodes
if you want to run any query on those data.
This is really a good feature for archiving purpose,
where you would like to store data into durable storage
for longer duties, and if you have any degree tree
or the compliance data to store you need
for longer duration.
So in a nutshell, you can use the hot tier
for faster access, which requires the low latency
for your data, then you can use the UltraWarm tier
to store large amount of read-only data
with lower storage cost, and then you can use
the cold storage to extend the lifecycle of data,
which might not be queried too often
and just needed for some complex purpose.
So this concludes a quick overview of the storage tiering.
If you would like to know more about UltraWarm
and cold storage or looking for a demo,
reach it out our getting started page,
and a link of them, which talks in depth about the feature.
So thank you for listening,
and feel free to reach out to us
if you have any further questions.