- Hello, everyone.
My name is Arun Lakshmanan.
I'm a OpenSearch-focused analytical specialist
at Amazon Web Services.
Today, we are going to talk about the tools
that are available for us to transform the data
and analyze in OpenSearch.
Amazon OpenSearch service
is a fully managed service that makes it easy
for you to deploy, operate, scale OpenSearch cluster.
It also supports 19 different versions
of legacy Elasticsearch clusters in the AWS cloud.
OpenSearch is a fully open source community-driven
search and analytical suit derived
from the Elasticsearch 7.10.2 and Kibana 7.10.2.
The OpenSearch project consists of a search engine,
that's OpenSearch.
A visualization engine, that's OpenSearch Dashboards.
As well as a set of tools, plugins and client libraries
that helps easy for you to interact with OpenSearch cluster.
The basic interaction flow works in this way.
You have your source data that comes from various sources
that gets converted to JSON documents by these tools
and it gets ingested into OpenSearch cluster.
Once it is indexed, they become searchable instantly.
With the provider API clients and OpenSearch dashboard,
you can access the index data in the OpenSearch cluster.
So the data sources can be like very deep of range
in different size and shapes.
Like the static resources like your application data
from your web application microservices
or your business application databases, etc.
Or it can be coming from a streaming like your automotive
or home devices or manufacturing pipeline,
or it can be a simple application data
that sits in the databases.
You need transformation when you are sending
this data to OpenSearch cluster.
What you need transformation is basically the sources
can be verbose or noisy in nature.
You want to filter out the noises
and you want to ingest and analyze the data point
which matters to you the most.
Along the way, you might want to enrich your data
by adding metadata like the instance name
from where it gets generated
or you can use tags that tag your environment name
or application name, or you might want to do
some kind of computation or simply,
you want to enrich your data
with a third-party system, like adding a geo information
to the IP address for your access logs, etc.
So one option is to Data Prepper.
Data Prepper comes from OpenSearch project.
It helps you to simplify the collection
and analysis of logs, traces, and metrics.
It is a fully open source
with the Apache license version 2.0
Fluentd is one of the popular option in this tool.
Fluentd is a open source data collector.
It is a high performance and it has built-in resilience.
It can handle buffer in the memory or in the file system.
Since it is written in a plugin ecosystem,
the community has like over 1,000 plugins
that are available for us to transform our data.
It also supports streaming at the edge.
It uses common SQL language process
to access the stream of logs.
If you have a strict memory requirements,
you can consider Fluent Bit.
Fluent Bit and Fluentd share the same concepts
except the fact it's written in C.
It is very suitable to deploy and collect the logs
and metrics from the Kubernetes cluster.
You can forward the logs from Fluent Bit
to Fluentd to process in a scale way.
It also has SQL streaming process that helps you
do streaming at the edge.
The other popular option is Logstash
which is popular for a very long time.
You can construct a complex transformation
like you can filter, you can drop, you can add metadata.
All of them you can do with Logstash.
It also supports out-of-the-box
but most common log types like Apache common
or error logs or combined logs or NGINX logs
and it handles the batch as well.
The one common con of this architecture
is you might need to run this in the own hardware.
You can also use a Lambda, which is a serverless
native AWS solution.
That's a big benefit over Logstash.
You don't really have to manage.
It is serverless.
The biggest downside is that you have to write everything
that Logstash is doing,
and you have to do everything in the code.
So that it could be significant challenge
especially for a complex log lines
and given that Logstash is a complex piece of technology
that is there a lot in Logstash.
And the other thing that I have already mentioned
is the concurrency.
It drive too much concurrency to the OpenSearch cluster.
And there is no built-in error handling mechanism.
All you need to do is in the code.
You have to handle the errors.
The other option is to write a custom code from the scratch.
If none of these tools are working for you,
probably you can write a fully custom code.
Or if you have a data collected in the streaming,
like MSK or Kinesis (indistinct) system.
You can use Amazon Glue
to process your data and send it to OpenSearch.
Or you can use a Flink Connector for OpenSearch
or Kafka Connector for OpenSearch.
For some of these operations, you can use a managed services
like Amazon Glue or MSK Connect to send data
from Kafka to OpenSearch.
The main con is you may need to write
a lot of complex code and manage on your own.
In this demo, I'm going to start
with a simple event processing
with Apache access log events.
Then enriching the log events with geo information
the third, filtering out the events and send the events
to two different destinations in OpenSearch.
Finally, with streaming at the edge,
I will be aggregating the events
based on custom enriched attributes of the event into that.
Before we get into the demo, let's take a look
at the structure of the Fluent Bit data pipeline.
It has three major components.
One is input that are your sources.
Filter is basically for processing.
And then you can emit your processed records
to multiple destinations.
Optionally, once it hits the storage tier
you can have an independent stream processes subsystem
which is a very powerful feature,
which we will see in the demo.
Where you can perform streaming operation
at the edge that will greatly reduces the stress
on the backend system.
Which is open source in our case.
For this demo, I'm using Amazon Linux on EC2 instance.
I'm going to run the single line install command
on my EC2 to install full deck.
Now it gets installed.
Let's go to the demo directory.
Let's take a look at the sample file.
So this is the typical Apache access log
which we will be processing.
It has about 3,000 lines in it.
Fluent Bit comes with prebuilt parsers
for most common log patents
like Apache logs, header logs, and NGINX,
Kubernetes, MongoDB.
And you can write your own parts according to your data.
Let's take a look at the Fluent Bit data pipeline.
So the first section is a service section
that's a service level configuration
where our parser files is.
And I have my input.
Input I'm tailing the sample log file from the head.
And I'm filtering it with the Apache to log parser.
And I'm outputting it to my OpenSearch cluster.
I'm leaving the log stage format on
which helps to rotate the index daily
based on the event timestamp.
And the log test prefix is my index name.
I'm disabling the output here
because the output can be enabled
during the development of this pipeline.
Once it is stable, you can actually disable.
Otherwise, it'll be very verbose in production.
Before I run this, I want to watch the index
in my OpenSearch cluster.
So I'm going to watch.
Secondly, I have three in indexes.
Those are all like ecosystem and that's in OpenSearch.
Let's run this minus C where my configuration file is.
So the moment I run this,
you can see a new index that's been created
based on the event timestamp.
Let's close this.
So the next demo is to enrich the log lines
with the geo information.
For this, you need to download.
For this, I'm using GeoLite.
I can download this from MaxMind.
Here is my geo database.
Let's take a look at the configuration
that uses this file.
Here, I have the additional filter section
for geoip2 that matches the tag Apache log
where my geo database is.
And which key I should look for the host IP address?
In this case, it is a host.
Once it is resolved, I'm creating a new attribute
in the data called country with a country name in English
and the country code as ISO code.
And I'm emitting this to a new index.
Let's run this.
Now you have this index created.
Let's take a look at how this has been created.
I'm going to the dev tools.
Run this command.
Now I have this new index created.
You can see a new attribute has been added to the index
with a country and ISO code based on this host.
Let's look at the third one.
The third demo, we are going to filter events
based on the status quo.
For this, I'm going to use a stream.
So here is my streaming configuration.
Since in this configuration, I'm creating a stream task
with a name http_code_401,
I'm creating a new stream with a name http401
and tag the stream as http401.
And selecting everything from Apache log,
which is our sample log where the code is 401.
Let's take a look at the data pipeline.
I'm using the stream file as a service configuration
the stream creates that tag, http401.
We are matching it in our output
and sending it to a new index.
So now, we are collecting all the events that are collected
in a stream to a separate index in OpenSearch.
I'm just outputting it for our understanding in this demo.
Let's run this configuration with jq.
Now all the events with the status code 401 are collected
in a separate index here.
Let's take a look at the fourth one.
Here, I'm creating a new stream that is country stream
with the tag country data.
I'm selecting country as a country name
and counting all the events
as a total number from the tag Apache log
in every five seconds window, group by country.
Let's take a look at the data pipeline for this.
In this pipeline,
everything, the data on the standard out,
if it matches country data.
Let's run this.
Let's run this command.
After five seconds,
you can see the events have aggregated by country
and the count.
This concludes the demo.
Now you have a fair understanding of how Fluent Bit works
and how you can filter process
and enrich your events with its plugin ecosystem
and powerful stream processing at the edge.
You can also write custom processes, parsers and plugins.
So since there are multiple tools are available,
you can choose wisely.
Thank you.