In this video, you’ll learn about Amazon Kinesis Data Analytics, an Amazon
managed service for Apache Flink.
Amazon Kinesis Data Analytics helps you gain actionable insights from streaming data.
To get started, let’s look at how data is streamed.
At a very basic level, all stream processing applications include five stages.
The process begins with data generated by multiple sources, or producers.
These sources may include mobile SDKs, IoT devices, click stream
products, or application logs that produce real-time data at high velocity.
Second is stream ingestion, when data from tens of thousands of data sources
is collected and ingested in real time.
This stage may include value-add processes such as authentication or throttling.
This layer can also filter out data from poor sources.
The third stage is stream storage.
Data is stored in the order received for a set duration, and can be replayed
indefinitely during that time.
The fourth stage, stream processing, is when the stored data is analyzed to gain insights.
Records are always read in the order in which they are received,
which makes stream processing different from other types of processing.
The final stage is to send the processed data to a destination, such as a data
lake, data warehouse, or database,
where it can be held longer term for analysis or auditing purposes.
Real-time streaming use cases fall into three broad categories.
Analytics applications produce metrics that help with decision making in real time.
An example would be a dashboard that constantly refreshes its information.
Streaming ETL applications transform, aggregate, and filter streaming data for
analysis, and then send it to a destination.
Stateful event processing triggers an external action when a set of conditions is met.
Anomaly detection is an example of this use case.
AWS offers various streaming data services designed to handle these different use cases.
Our focus is Amazon Kinesis Data Analytics.
Amazon Kinesis Data Analytics is a fully managed service that enables building
simple or sophisticated streaming applications using Apache Flink.
It supports millisecond latencies, is completely serverless, and provides
automatic scaling so you pay only for the resources used per application.
Amazon Kinesis Data Analytics works with Kinesis Data Streams,
Amazon Managed Streaming for Apache Kafka (Amazon MSK),
and Apache Kafka as sources.
Before discussing the service, let’s answer “why Apache Flink?”
Apache Flink supports a diverse set of use cases.
Flink uses a layered API approach.
Each API offers a different balance between control and ease of use.
Flink processing guarantees exactly- once state consistency, event-time
processing, and late data handling.
It is also easily scalable and is supported by a vibrant open source community.
Thanks to that community, there are more than 200 integrations from AWS
services or to third-party SaaS providers.
All Apache Flink connectors are supported on Kinesis Data Analytics as a managed service.
With Kinesis Data Analytics for Apache Flink, you can use Flink’s native APIs to
build any type of application you want using SQL, Java, Python, or Scala.
The flexible APIs allow you to focus on your business logic rather than worrying
about the infrastructure management that goes into building stream
processing applications.
You also get high performance.
Built on AWS services, Kinesis Data Analytics offers in-memory computing
with extremely low latency and gigabytes-per-second of throughput.
It also provides durable application state saves and strong data integrity.
Kinesis Data Analytics Studio provides a simple way to get started developing,
debugging, and running stream processing applications.
Its interactive notebook interface supports built-in visualizations, allowing you to
perform ad-hoc queries to inspect your data stream and view results in seconds.
Here’s what Kinesis Data Analytics does for you:
It makes it possible to rapidly create, test, deploy, and scale real-time data
streaming applications.
It drives best practices through design, defaults, and automation, allowing you
to focus more on application development and less on infrastructure management.
Finally, it enables monitoring of streaming Flink pipelines and powerful integrations
with other AWS services, including Amazon CloudWatch, so you can keep
an eye on applications without having to deal with any other external services.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it's your turn to try.