In this video, you’ll see how to monitor
Amazon Kinesis Data Analytics for Apache Flink applications.
With this solution, you can monitor Flink applications in real time using Amazon
CloudWatch metrics and the Apache Flink dashboard, identify errors
and exceptions in log streams, and proactively troubleshoot application issues.
Amazon Kinesis Data Analytics for Apache Flink is a managed Apache
Flink solution for building streaming applications.
It supports sub-second latencies and auto-scales to meet demand.
You pay only for the resources used.
With Kinesis Data Analytics for Apache Flink, you can rapidly create, test, deploy,
and scale streaming data applications.
It drives best practices through design, defaults, and automations,
and allows developers to focus more on application development.
It also enables monitoring of streaming Flink pipelines.
We’ll start in Amazon Kinesis, where we already have some Apache Flink applications.
Let’s look at one.
This application was created for monitoring lag.
For demo purposes, we’ll use the Amazon Kinesis Data Generator to send a high
volume of test data to our Flink application.
The Monitoring tab of our application displays some Amazon CloudWatch metrics.
We can see more by viewing the CloudWatch dashboard.
Let’s look at how well our application is keeping up with throughput.
The millisBehindLatest metric measures the time difference between the latest
received data and the latest received event by the stream.
The metric has been growing in the last 15 minutes due to the high volume of
data we sent, indicating that the application could fall behind.
Pay close attention to this metric if it exceeds your business latency
requirements or continues to grow.
Next, we’ll look at the containerCPUUtilization metric, which gives us a complete picture of
all processes in terms of CPU exhaustion at the container, and failures resulting from that.
We’ll view the metric for an application called cpu_spiking.
Let’s remove the first metric we viewed from the graph.
The metric now displayed represents the total utilization, on average, across
all the TaskManagers in the Apache Flink application.
If this metric were spiking or growing, we might have an issue with a slow or
busy operator that is taking up resources in the task managers.
Let’s look at another metric called containerMemoryUtilization.
This metric represents the overall percentage of memory utilized by
TaskManager containers in the Flink Application cluster.
This metric’s continued growth would be indicative of an application in need of scaling.
Let’s look for another metric called fullRestarts.
We’ll select the full_restarts application,
which was created to demonstrate an application that is restarting continuously.
Temporary increases in the fullRestarts metric can be expected due to patching
or recoverable errors.
A continuous increase, however, is indicative of an issue in the application.
If your application is experiencing an increase in fullRestarts, it would be good
to investigate application logs and the Flink Web Dashboard to learn more
about what is causing these restarts.
The lastCheckpointSize and lastCheckpointDuration metrics monitor
the data stored in an application state and the length of time for each checkpoint to process.
If checkpoint size and duration continue to grow, job failures can occur.
Next, let’s look at a metric called numberOfFailedCheckpoints.
We’ll select a previously created application that reads from and
writes to Kinesis Data Streams.
This application has an increasing number of failed checkpoints.
If this issue is not addressed, further issues with the application could ensue.
It is a good idea to investigate issues with the application such as
backpressure or application code issues that could lead to this.
Now, that we’ve seen some of the available metrics, let’s see how we
can pull them all together in a custom dashboard.
To do that, we can use an AWS CloudFormation template.
Let’s create a new stack.
We’ll upload a template file that can be found in the resources section of this video’s description.
We’ll give our stack a name.
Next, we’ll specify the name of the Kinesis application we are monitoring.
We'll also name our dashboard.
Next, we’ll provide the region where the Kinesis Analytics application is created.
We’re not reading from Kafka, so we’ll skip that parameter.
Our application is reading from a stream called tumbling-window-stream.
We’ll keep all the default settings for configuring stack options.
Let’s create our stack.
The dashboard has been created.
We can navigate to it from the Outputs tab.
The dashboard has all of the metrics that we referenced earlier in this video,
as well as some other metrics such as uptime and downtime.
Many of the metrics include an alarm threshold.
For example, the CPU utilization threshold is set to 80 percent.
If you proactively enable the Autoscaling feature on Kinesis Data Analytics for
Apache Flink, your job will automatically scale the number of
KPUs when this threshold is exceeded.
The dashboard also includes metrics showing the Flink application progress,
such as incoming and outgoing records per second.
Next, let’s go to Amazon Kinesis and take a look at how to identify
exceptions in application code.
Let’s select one of our existing applications.
Next, we’ll open the Apache Flink dashboard for our application.
The dashboard shows a running job called Word Count.
This job reads from a Kinesis Data Stream, does some aggregation,
and then writes to another Kinesis Data Stream.
Each of the blue boxes in the job workflow represents a set of operators,
which are known as “tasks” in Apache Flink.
A quick check of the Exceptions tab indicates there’s no root exception in our application.
Let’s look at a Flink dashboard for a different application.
This time we see a checkpoint exception in our Exceptions tab.
Let’s return to the Word Count job for the first application we viewed.
Here, we have the data flow for our application.
We can interact with the blue boxes.
Let’s select the box in the middle of the flow to see its sub tasks.
Subtasks are the parallel units of a task that comprise the distributed application.
If a task has 4 subtasks, that means 4 threads on the task are performing
the same set of operations.
Sub task 0 has received roughly 87,000 records and sent more than 240 records,
whereas Sub task 2 hasn’t sent or received any records.
This is an example of data skew, which can lead to longer or shorter
than expected checkpoints and a degraded application.
Under the TaskManagers tab we can view the logs of each individual TaskManager.
The Watermarks tab is relevant when doing EventTime processing.
Under the BackPressure tab, we have the ability to see the
backpressure of the application.
Backpressure occurs when data is ingested into an operator faster than it can be emitted.
Backpressure is induced by the downstream operator, which is
represented by the blue box to the right of the one experiencing backpressure.
When troubleshooting backpressure, look for underprovisioned tasks or sinks,
slow operators, or longer checkpoints.
Here, we can see the percentage that an operator contributes to the overall
backpressure, as well as which operators are busy.
Starting in Apache Flink 1.13, we can learn more about backpressure on the
job Overview tab of the Flink Web Dashboard.
Occasionally, you'll find no jobs running on the Apache Flink dashboard.
This means the job never started.
Let’s return to our application in Amazon Kinesis to diagnose why
an application wouldn’t start.
We can look at this Logs tab to see if there any errors.
The page is empty.
Let’s go to the Configuration tab and find the log group for this application.
We’ll select the Log group for this application.
Next, we’ll select Log insights.
Let’s find our log so that we can query it.
Here, we’ll search for any throwable exception information from the job.
The throwable exceptions have started to populate.
We can use this information to find out why the application didn't start.
You’ve just seen how to monitor Amazon Kinesis Data Analytics
for Apache Flink applications.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it is your turn to try.