- [Instructor] In this video,
you'll see how to integrate Fluent Bit
into CloudWatch Container Insights
for Amazon Elastic Kubernetes Service, or Amazon EKS.
With this lightweight solution,
you can efficiently stream Amazon EKS logs
into CloudWatch Logs,
reduce your log delivery resource footprint,
and track your log stream
to prevent bottlenecks and delivery errors.
To get started,
let's review some information about Fluent Bit,
which we'll be using to send logs
from an Amazon EKS cluster to CloudWatch Logs.
Fluent Bit is an open source
and multi-platform log processor and forwarder
that allows you to collect data and logs
from different sources
and unify and send them to different designations
for logs storage and analytics.
These statistics demonstrate the performance advantage
Fluent Bit has over Fluentd in CPU and memory usages.
Let's navigate to Amazon EKS
and take a look at the cluster we've set up
for the purposes of this example.
Let's drill down into the cluster
and review its configuration.
This cluster has the latest Kubernetes version installed.
It also has three worker nodes.
Next, let's take a quick look at the AWS Cloud9 environment
we've set up for the per purposes of this example.
Before we start setting up Fluent Bit,
let's review the documentation.
This documentation provides details
on how to deploy Fluent Bit to send logs
from containers to CloudWatch Logs.
The first step is to create a namespace.
Let's copy the command to do so
and paste it into the Cloud9 console.
Next, let's return to the setup documentation
and copy another command.
We'll use this command to create a config map
with the cluster name and region to send the logs to.
Next, we'll copy the command to download
and deploy the Fluent Bit DaemonSet to the cluster.
For our purposes, we'll use the optimized configuration
rather than the one that is more similar to Fluentd.
Finally, let's copy and paste the command
to validate the Fluent Bit deployment.
As you can see, we have three Fluent Bit pods running.
Let's make sure
that events are being sent to CloudWatch Logs.
We've confirmed that the events are being sent.
Let's navigate to CloudWatch.
We'll search for our EKS cluster among the log groups.
Here are the three log groups associated with our cluster.
Let's drill down into one of them
and check the application logs.
Let's expand our view of the Log stream column.
Now let's search for logs related to the billing app.
We can drill down into the individual log streams.
Here we can see all the logs running on this billing app.
We can drill down further to see even more specifics.
Next, let's go over some best practices
for creating a successful log delivery pipeline.
First, when defining your Fluent Bit image,
it's recommended that you use the latest official version
of Fluent Bit.
Second, always define memory and CPU limits
for each Fluent Bit pod.
Third, it is recommended to define multiple input streams
based on smart logical groupings.
Another best practice is to use the Kubernetes filter
to enrich the log entries.
Let's take a look at the optimized YAML file
we implemented earlier.
As you can see, we're running Kubernetes filter,
which is providing metadata for our logs.
It's also recommended to run AWS filter
alongside Kubernetes filter, as we are here.
Another recommendation is to increase
the memory buffer limit of Fluent Bit to 50 megabytes
for application logs.
Finally, AWS recommends enabling a built-in HTTP server,
which makes it easier to monitor metrics.
After enabling the HTTP server,
you can create dashboards resembling these graphs
in CloudWatch Logs or Grafana.
You've just seen how to integrate Fluent Bit
into CloudWatch Container Insights for Amazon EKS.
You can learn more about this topic
in the description and links for this video.
Thanks for watching. Now it's your turn to try.