In this video, you’ll learn about the Amazon Kinesis Client Library (KCL).
KCL is a Java library that helps you consume and process data from a Kinesis data stream
by taking care of many of the complex tasks associated with distributed computing.
Amazon Kinesis Data Streams ingests a large amount of data in real time,
durably stores the data, and makes the data available for consumption.
You have multiple options for consuming data from the stream.
You can use third-party tools such as Apache Spark.
You can leverage an AWS SDK.
You can use built-in integration with AWS services such as Kinesis Data Firehose,
Kinesis Data Analytics, and AWS Lambda.
Or, you can build a custom consumer application using Kinesis Client Library
along with the Kinesis Connector Library.
Our focus is KCL, which is a Java library.
Support for languages other than Java is provided using a multi-language
interface called the MultiLangDaemon.
This daemon is Java-based and runs in the background when you are using
a KCL language other than Java.
KCL comes in two versions, 1.x and 2.x.
For information about the differences and on how to migrate from KCL 1.x to KCL 2.x,
please refer to the link provided in the description for this video.
Note that if you want to use enhanced fan out consumers with dedicated read
throughput must use KCL2.x
KCL acts as an intermediary between your record processing logic and Kinesis Data Streams.
It handles complex issues out of the box, such as adapting to changes in the stream
volume, load balancing streaming data when resharding occurs, coordinating distributed
services and processing data with fault tolerance, and checkpointing processed records.
Let’s look at some key KCL concepts.
A KCL consumer application is a custom built application designed to read and process records from data streams.
KCL consumer applications are typically distributed across multiple
instances running simultaneously.
A worker is a high level class that a KCL consumer application instance uses to start processing data.
The worker initializes and oversees various tasks, including syncing shard and lease
information, tracking shard assignments, and processing data from the shards.
Note that in KCL 2.x, workers are called schedulers.
A lease is the data that defines the binding between a worker and a shard.
Distributed KCL consumer applications use leases to partition data record
processing across a fleet of workers.
A lease table is a unique Amazon DynamoDB table used to keep track of the shards being
leased and processed by the workers of the KCL consumer application.
A record processor is the logic that defines how a KCL consumer
application processes the data it gets from the data streams.
At runtime, a KCL consumer application instance instantiates a worker, and this
worker instantiates a record processor for each shard to which it holds a lease.
The Amazon Kinesis Data Stream Developer Guide provides extensive
information about developing custom consumer applications using the Kinesis Client Library.
First, let’s review the documentation on developing custom consumers with shared throughput.
Topic links are provided for developing consumers with the AWS SDK for Java or with KCL.
We’ll select KCL.
You can use either KCL 1.x or KCL 2.x to build consumer applications that use shared throughput.
Let’s view the information for KCL 1.x.
We’ll choose the link for developing in Java.
The page provides various development resources, including a GitHub link.
Here you have Kinesis-client library application code.
Code is provided for building a lease table, a checkpoint, a worker or scheduler, and more.
You can explore this GitHub link for ready to use code samples.
Let's go back to the previous page and scan the step-by-step task instructions
and sample code available there.
You can copy the sample code and modify it to implement your consumer application.
Similarly, you can access guidance and code for developing consumers with dedicated throughput.
When developing consumer applications that use dedicated throughput (also
known as Enhanced Fan-Out consumers), you must use KCL 2.x.
This code shows an example implementation
that can be copied and modified to build consumers that use Enhanced Fan-Out.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it's your turn to try.