In this video, you'll learn what index
lifecycle management is and how to set this up for your Elasticsearch deployment.
When you are ingesting time series data
such as logs, the amount of data will keep growing over time.
Yet there's a limit that a single Elasticsearch index can handle.
Creating indices too big or too small will affect the performance of your cluster.
Index Lifecycle management, also known as ILM, will help you solve this issue.
ILM ensures high query performance
and reduced costs by periodically rolling over the data to new indices.
As data gets old and infrequently queried,
it can be migrated to less expensive infrastructure.
You can set up policies that will define when Elasticsearch will create a new index
and when the indices will move to other nodes.
It helps you optimize your performance and cost.
To start ingesting your time series data
to Elasticsearch, send your data to a data stream.
The data stream, along with the ILM policy,
will be responsible for managing underlying indices.
When you start ingesting data to your cluster, you need high performance
infrastructure to bear the pressure coming from the indexing operations.
The data will be sent to hot nodes.
The hot nodes are dedicated to ingesting and processing frequently queried data.
Once an index reaches the desired size or
age, the index will roll over a new one that will be used to ingest new data.
After the rollover, the old index will be transferred to a warm node.
Other operations can also be applied,
such as reducing the number of shards and segments of your indices.
The warm nodes are used to store less frequently queried data.
When data gets old, it is less likely to be searched.
When this is the case, the data will be moved to the cold tier.
As data transitions into the cold tier, it can be compressed and shrunk.
The cold nodes can be allocated
to an infrastructure with lower specs than warm and hot nodes.
The cold tier is still a responsive query tier, but the data becomes read only.
It is cheaper to store data on cold nodes, but the queries will be a bit slower
once data is no longer being queried or being queried
rarely, it may move from the cold tier to the frozen tier.
Data stored in a frozen tier will be stored in a snapshot.
This reduces local storage and operating
costs while still letting you search frozen data.
Because Elasticsearch must sometimes fetch frozen data from the snapshot,
repository searches on the frozen tear are typically slower than on the cold tear.
Eventually, you can set up your ILM policy to delete the data.
You can do so if you are no longer
required to store some data and you know that you will not be searching for it.
After learning the concepts of ILM,
let's take a look at how you can effectively set up ILM for your cluster.
The first step is to create a cluster.
If you want to use the different data tiers,
you need to make sure to have nodes
with roles assigned to the tiers you are using.
Let's create a deployment that will have hot, warm, cold and frozen nodes.
Connect to your cloud console and click Create deployment.
Give it a name, choose the cloud provider, region,
and the stack version that will be used for your deployment.
Next, click Advance Settings where we can
manually select the nodes that will be part of your cluster.
As you can see, we already have two hot nodes in the cluster.
If we click Info, we will get more information about these nodes.
Notice the roles associated with these nodes.
This is how ILM will know where to allocate the indices.
An index in the hot phase will be allocated to a node that has the role hot.
Let's add a warm data tier.
Click Add Capacity and add two more nodes to the cluster.
It's important to have at least two warm
nodes so you can ensure data resiliency with replicas.
When clicking on Info, you will see that these nodes are using different hardware.
With a higher disk to memory ratio, these nodes will have the data warm attribute.
Next, add a cold data tier.
For this tier, one node will be enough.
Unlike the warm tier where you are using replicas to ensure resiliency,
the cold phase will leverage snapshots to save space.
Finally, let's add a frozen tier.
Notice that this node has a much higher disk to memory ratio.
It will be used primarily for storing snapshots and searches must be rare.
After setting up your nodes, click Create Deployment.
Download the password to a safe place as
we will be using it later to send data to your cluster.
After a few minutes, your deployment will be ready.
Click Continue to open Kibana.
Next, define a policy that we want to apply to our logs. Under management,
select the Manage Index Lifecycles tile.
In this view, you can create a new policy or update an existing one.
By default, Elasticsearch will create
a policy for the logs that you are sending to the cluster.
Let's take a look at it.
Click on the Log policy.
You can see that this policy is already attached to multiple index templates.
When you will be ingesting log data
with the Elastic agent, this policy will be automatically attached.
Next, let's update this policy to fit our needs.
Start with the Hot phase.
The first step is to define when the rollover should happen.
Recommended settings usually work best for your use case,
but for the sake of simplicity, we will shorten the hot phase.
We are going to set it to a maximum of 100 documents.
When one of the three conditions is met, a new index will be created.
In that scenario, the number of documents will be reached first.
After rolling over the index, we should move the index to the warm phase.
You need to define when the data will be moved to this phase.
If we set the age to zero minutes, it means that the index will move
to the warm phase as soon as all its actions from the hot phase are done.
In this situation, it will be moved just after the rollover.
We can also shrink the index by reducing its number of primary shards.
Having multiple primary shards is great
to scale for indexing operations, but once you are in the warm phase,
you don't necessarily need multiple primary shards.
Next, let's activate the cold phase.
The index will be moved to this tier phase once it reaches the age of five minutes.
We can also convert the indices to fully mounted indices.
This will use snapshots to backup data and we don't need replicas.
Then activate the frozen phase.
The indices will reach this phase after ten minutes.
Finally, you can decide when to remove the data.
In this scenario, we will delete all the logs older than 1 hour.
Save the policy so we can start sending logs to your cluster.
We will use the Elastic agent to send logs to your Elasticsearch cluster.
If you come back to the Cloud console,
you will notice that the disk allocation of your hot nodes has slightly changed.
There is some data indexed in these nodes.
Now let's see if the ILM policy we defined is behaving correctly.
Open Kibana and choose the Dev Tools.
By default, ILM checks for indices
that meet the policy criteria every ten minutes.
We are going to reduce this interval for an easier demonstration.
Next, open Stack Management and Select Index Management.
In the Data Streams tab, you will find the logs-generic-default stream.
This is where the Elastic agent we configured is sending the logs.
If you open this data stream,
you will see that this stream is attached to the logs policy we just updated.
Also notice that the stream is linked to the logs index template.
Let's open it to see how this is configured.
On this view you can see that any indices
matching the following pattern will inherit this policy.
If you click on preview, you will see the settings and mappings
applied to these indices, including the lifecycle policy.
If you are using a different method
to send your logs, you will need to make sure that you have an index template
that links your logs to the policy you want to apply.
You can create one from this interface.
Let's move back to our data streams.
When clicking on its number of indices,
you'll be able to visualize the indices attached to this data stream.
So far, we have only one index which has less than 10,000 documents.
Let's wait a few seconds to get more than 10,000 documents.
After a few seconds, you now have two indices.
The first index has a total of 13,588
documents and a second one has been created.
If you click on the name of the first index, you can see that the index is
correctly attached to the logs policy we updated previously.
You can also see that this index is currently in the warm phase.
The second index is in the hot phase.
Let's open Stack Monitoring to visualize the shard's allocation
open the indices section and filter the indices by the name of your stream.
If you open the first index you will see
that this index has a primary and a replica shard.
The shards for the first index are
allocated on nodes two and three which are warm nodes.
The shards for the second index are
allocated on nodes zero and one which are hot nodes.
After five minutes let's go back to Index Management.
You still have two indices but the name of the first one changed.
If you click on the new name you will find
that the index has been moved to the cold phase.
Notice also the storage size of the first
index changed. For the same amount of documents
we are using half the disk space.
Reopen Stack Monitoring to analyze the shard's distribution.
You can see that the first index only has one shard.
The replica has been removed.
You can also see that the shard has been moved to node four which is a cold node.
If you wait long enough you will see that the frozen phase will kick
in and the logs will start being deleted after 1 hour.
Thank you for watching this video on how
to set up ILM for your Elasticsearch cluster.