- Okay, so hello everyone.
My name is Prashant Agrawal and I am working
as an Analytics Specialist Solution Architect in AWS,
where I primarily focus on Amazon OpenSearch Service.
And today I will be talking about managing time-series data
with data-streams using Amazon OpenSearch Service.
So, before getting into the data-streams,
let me explain about Amazon OpenSearch Service.
So, Amazon OpenSearch Service
is a fully managed service that makes it easy
to deploy, operate and scale OpenSearch
along with legacy Elasticsearch cluster in AWS Cloud.
It also has tight integration with other AWS services
such as Amazon Kinesis Data Firehose,
AWS Lambda, CloudWatch, and so on.
So let's take a quick detour
and talk about what is OpenSearch.
So OpenSearch is a community driven,
open source search and analytics suite
which is derived from Apache 2.0
licensed version of Elasticsearch 7.10.2
and Kibana 7.10.2 projects.
It consists of a search engine, which is OpenSearch
that provides you a way to perform search on your data.
As well as, we do analyze your logs
using visualization interface called OpenSearch Dashboards.
So using OpenSearch,
you can easily index the data of any type
such as logs, metrics and traces,
and then you can use visualization interface
using the OpenSearch Dashboard or even the API
to perform aggregation on top of those data.
Next, we also have set up plugins that OpenSearch provides
which adds the comprehensive functionality
such as generating alerts on the log data
or look for anomalies using machine learning algorithm
or maybe use the trace analytics
and the observability feature
to capture logs, metrics and traces,
and then make the code addition across
those logs, metrics and traces.
So let's jump on next and see why do we need data-stream.
So one of the common use case with OpenSearch
is to index continuous generating time-series data
such as logs, metrics and traces.
Previously, automating index rollover
means you had to first create a write index,
configure a rollover alias
and verify that indexes are being rolled over as expected.
That makes your bootstrapping process for an index
to be more cumbersome than it needs to be.
So with the data-stream it is optimized for time-series data
that is primarily append only in nature
and simplifies all the initial setup process for the data.
So let's see how the flow works
with and without data-stream.
So prior to data-stream,
this is how you user set up their indexes
for time-series data.
So they start with setting up an index
by specifying the index mapping and settings.
And, as their data grows,
they encounter the scaling issue with their logs and data.
And at that time they start thinking
about creating daily rotating indices
on the basis of dates
or maybe you use the Rollover API
to create an index for each date.
At this point, user has got more than one indices
to run the search, so they can either use the wildcard
to specify the index such as logs hyphen nginx hyphen star
or maybe too, they can use the create an alias
by creating the alias for those time-series indices.
So here in the next one you can see,
they can define the index alias for all relevant index
to bring it under one umbrella.
And here, searches can be performed
across all the underlying indices on the alias.
But how about performing the write updates
and not indexing the data,
as in how do we determine that which index
would be getting the write request, right?
So in this case, they can solve it
by marking the latest index as the write index
and this can be solved by bringing the common setting
such as the mapping index settings,
index patterns into an index template
which can be again configured on all new indices
getting created as a part of the rollover process.
So now, user needs to call the rollover
and calling it manually on daily basis
is not a practical solution
and they have to manage it by themselves.
So in that case,
they could use the Index State Management Policy
to define the index pattern and rollover policy
to rollover the indices.
It could be done either on the basis of time or the size.
With this legacy method,
user has to be aware of so many concept
to manage and maintain those indices
and they need to set up them explicitly.
So if you consolidate it, so we can see that,
what are the typical challenges
with the time-series indices.
So, OpenSearch has basic support of time-series data
where one can name indices on the basis of timestamp
by prefixing it with some name and then suffix as time,
where you can have some indices
on the basis of day, week, month, etcetera.
If everything is working fine
then why do we really need data-stream?
So managing your time-series indices,
you need to perform index rotation
then your log ingestion tools
such as Beats, logs test
can help you to rotate those indices on dates.
But, what if you hit onto some event
such as Black Friday or Cyber Monday
where your data is significantly larger than the usual.
So it creates the data's cube for those days
where you are getting a large amount of traffic.
So once index is rotated you need to make sure
that the write index is pointed to the current date index
and then index alias is rollover as well.
So here alias is nothing but a way to query
all the latest data which points to the current
or latest underlying index for the specific log.
This is why we have the data-stream
come into the existence.
So, let's see how data-stream works
at very high level and solve some of these problems
being a first class citizen to store the time-series data.
So data-stream has one or more
auto generated backing indices
and prefixed by dot ds, then the data-stream name
and the generation id.
So if you look here,
here the data-stream name is dot ds logs hyphen nginx.
Then the backing indices are like dot ds
hyphen logs hyphen nginx hyphen 000001
then 02 and 03 and so on.
Now coming back to the write operation,
so for any write operation,
you send the request to data-stream
then it will send it to the latest index.
Once an index is rolled, it will become read only
and you cannot add any new document
to the rotated index.
So typically in the log analytics scenario
there is rarely a modification.
And if needed, you can run update by query
or delete by query in the older backing indices.
Now let's talk about the search
like how do we perform the read request from those indices.
While learning this search query, data-stream routes
request to all the backing indices
and of course you can filter out the data
by timestamp to perform the intelligent search
on those data sets.
You can also use the query such as the aggregation query
or maybe use the filters like the Boolean filter,
merge clause, suit clause, etcetera to filter out the data.
So this is how read and write works with the data-stream.
So, let's talk about
some of the data-stream caveats what we have.
So, because as I mentioned data-stream
is primarily designed for the append only data.
So in order to create a data-stream,
you first need to create an index template
which configures a set of indices as a data-stream.
Then you need to ensure that each document
has a timestamp field
otherwise it will be added by default.
Then, recent data is more relevant
which makes it suitable for even the hot warm architecture
with respect to the Amazon OpenSearch service.
Then there are certain operations which you cannot perform
that will, that are termed as the blocked operation
like clone, delete, close, freeze.
So those kind of things could not be performed
because these operation could hinder the indexing
and that's why these are blocked.
Talking about the Ingestion API,
so you can run the simple API or you can do the bulk API,
but the only thing is if you're running the bulk API
you now have to explicitly specify the op type as create,
that is the pre requisite for sending any bulk request
using the data-stream.
So let's jump into the workings of data-stream
and see it in action with a short demo.
Okay, so let's see how it works in real world.
So, I have logged into the OpenSearch Dashboard
and I'm going to create a data-stream
but as of now I do not have any index template
and if you create a data-stream
without creating an index template, it will error out.
So if you recall my data-stream caveat slide
so in order to create it,
you need to create an index template
which configures a set of indices as a data-stream,
and data-stream object indicates that it's a data-stream
and not the regular index template.
And then the index pattern matches
with the name of the data-stream over here.
So in this case, if I'm going to create an data-stream,
it is giving me an error
like no matching index template found
for this particular data-stream,
so, now I'm going to create the index template
by specifying the index pattern as logs hyphen nginx.
And then I have the number of shards as one
and number of replica as one.
Then I have the timestamp field as @timestamp.
Now I have created an index template.
Let's verify
whether I have the current index template or not.
So, on running the GET index template logs nginx,
you can see the index template details
like the index patterns, name,
number of shards, replica of what we have configured.
So let's move on and then create the data-stream now.
So in this, basically when you create the data-stream,
so you can use the data-stream API to explicitly create it
or even like data-stream will initialize
as soon as you send the first document to the index
which is matching the index pattern name.
So here I'm creating the data-stream explicitly
by running this command and it gives me the acknowledge too
that means data-stream has been created.
You can verify the data-stream and associated index
by running the GET data-stream logs hyphen nginx.
In this case, it returns me the name of the data-stream,
the timestamp field and what is the backing indices
for this particular data-stream,
and then what is the generation, what is the status
and the template behind this particular data-stream.
Now let's start into sending
or writing the data into the data-stream.
So here,
in order to ingest data into data-stream
you can use the regular indexing API,
for example over here,
I'm writing the POST logs hyphen nginx
which is my data-stream name,
then I have a body where I have a couple of like message
and @timestamp.
So as I mentioned earlier
you need to have the timestamp field
otherwise it will give me an error.
So, after you run this particular command
your data is being indexed
and your document has been created
onto the data-stream with the backing indices
as ds logs nginx 00001.
Apart from sending the individual request
you can also send the bulk API request as well
in order to ingest data into the data-stream.
The only thing is you need to have the operation type
as create when you are sending any bulk request.
Now if I run this command, it inserts two document
where I have these two documents into my bulk API as well,
one with the message 'login success'
and another one as the message 'login failed'.
Let's see how the search works
and how you can perform searching the data-stream
and get all the documents from that stream.
So you can do a simple search request
like GET your data-stream name
followed by the underscore search API.
It will return you all the data
for that particular data-stream.
So I have inserted three documents on it
so you can see one is with 'login attempt',
one is the 'login success',
and another one is the 'login failed'.
So this way you canceled the data from the data-stream
and you can further use the Query DSL as well
to run advance query such as the bool query with filter
for running aggregations and what not using the data-stream.
Next talk about how you can perform the rollover
onto the data-stream.
So you can use the ISM Policy
in order to define the rollover policy
on the basis of size or maybe on the basis of the date-time.
Apart from that you can also run the manual command
to rollover the data-stream
by running the command as POST API
with your data-stream name followed by underscore rollover.
Once you run this command, it will give you the details
about your old index and the new index.
So now your old index has gone into the read only mode,
you cannot perform any write operation on this,
and all of the data what you are going to send,
it will be sent to the new index
which is the ds logs nginx 000002.
You can verify the rollover operation as well
by running the GET data-stream logs hyphen nginx
and here we have the generation two
means it has been rolled over,
this the second generation of the data-stream.
And then it has the details of your timestamp field
and then all the indices.
Next, we can talk about
whether we can delete the write index or not.
So, if you run the command to delete the write index
which is the 02 as of now,
it will error out because you cannot perform
the delete operation on the write index.
So if you have to delete the write index
then you have to delete the index template
and then you have to delete the data-stream.
So before showing you the delete command
let me quickly go to the index management page
and there we can search for all the indices what we have.
So go to the indices tab, show the data-stream indices
then you can select the data-stream.
So here I have two data, two indices on the data-stream
logs hyphen nginx 000001 and 02
and you can see I have three documents on the 01
and there is no documents on 02
because we haven't run any Ingestion API
to ingest the data after the rollover.
Now let's move to the command again
and how you can perform the delete.
So now if you go and try to delete the data-stream
it will delete everything and if you go to,
fetch the data-stream over here
like perform any search,
so it will say like no index was found
because we have deleted the data-stream.
So once you delete the data-stream
it will delete all the indices or all the backing indices
as well the write indices from that particular data-stream.
So this is all about a quick demo about the data-stream
and let's get back onto the deck
and then we can talk about what we discussed
as a part of the demo.
So I will show all the script over there in the form of deck
and followed by couple of documents
like how you can get started with the data-stream
and then what should be the next step
or the next action on that.
Thank you.
So in the demo we have seen
how you could create a data-stream,
without creating an index template, it will error out.
So if you recall the data-stream caveats,
in order to create a data-stream
you first need to create an index template
and this shows like how you can create the index template,
you can refer this script for creating the index template.
Once we have created the index template,
you can verify those by running the GET API
to make sure your index template is created.
Then after you have created an index template,
you can create a data-stream
where you can use the data-stream API
to explicitly create a data-stream.
If you're not creating it explicitly
then as soon as you send the first document,
it will create the data-stream.
Talking about the ingestion,
you can send the individual request as this one
or you can refer the demo which I showed just now
where you can send the bulk request as well
with the op type as create.
Then last but not the least,
you can run the search request
using the GET data-stream name slash underscore search.
It will give you the top twenty documents
for that particular data-stream
and then you can further use the Query DSL
to run advanced queries
such as using the bool query with filters,
running aggregates and send what not.
Lastly we saw in the demo how you can manage those indices
and data-stream from OpenSearch Dashboard,
where you can go to the index management,
filter the data for the data-streams
and then it will list down all the backing indices
for a particular data-stream.
So that's what we covered as part of demo.
So coming back to the conclusion,
so in a nutshell, data-stream helps you
to intelligently manage the time-series data
where developer or operation team
can perform focus on the development
and growing their business
rather than spending time on management tasks
such as managing time-series indices,
performing the rollover and so on.
So this concludes a quick overview
of data-stream along with a demo.
If you would like to know more about data-stream,
please check out our documentation
which you can get from the QR code that's on here.
So thank you for listening
and feel free to reach out to us
if you have any further questions.
Thank you.