This video will help you get started with Amazon Kinesis Data Streams.
Let’s begin by talking about the importance of having fresh data.
Data loses value over time.
Making decisions based on outdated data can negatively affect the customer
experience and inhibit organizational growth.
Traditional batch processing of data results in business intelligence that is
often hours, days, or even months old by the time it is available.
However, many time-critical decisions require data that is only seconds or minutes old.
For instance, time is critical for real-time anomaly and fraud detection.
Likewise, businesses want to provide safe, real-time customer
experiences and personalization.
Event-driven architecture permits a greater degree of development agility
than would otherwise be possible.
It also enables businesses to decouple their microservices to increase resiliency,
and to analyze Change Data Capture signals across a variety of applications.
All this is achievable by processing streaming data in real time.
Streaming data is high-volume, continuous, and generated by thousands
of data sources, often simultaneously.
Processing it must account for record order and latency.
Let’s look at how data is streamed.
At a very basic level, all stream processing applications include five stages.
The process begins with data generated by multiple sources, or producers.
Second is stream ingestion, when data
from tens of thousands of data sources is collected and ingested in real time.
The third stage is stream storage.
Data is stored in the order received for a set duration, and can be replayed
indefinitely during that time.
The fourth stage, stream processing, is when the stored data is analyzed to gain insights.
Records are always read in the order in which they are received.
The final stage is to send the processed data to a destination, such as a data
lake, data warehouse, or database.
While the advantages to real-time data streaming are clear, building and
operating a custom streaming data pipeline is resource intensive.
The system can be difficult to set up and tricky to scale.
It can also be hard to achieve high availability, as this requires fine-tuning storage,
data batching, and compute resources for maximum throughput and low latency.
Maintaining real-time data streaming is error prone and complex to manage.
And all of this takes valuable time and money.
Amazon Kinesis Data Streams addresses these issues for you.
Kinesis Data Streams is a cloud-native, highly available, serverless streaming
data service built to handle billions of events.
Kinesis Data Streams handles ingestion and stream storage
stages of data-streaming use cases.
It provides one simple solution that allows any organization to ingest large
data volumes per second from thousands of sources, in the right order, and have
consistent, scalable performance.
The data is stored in shards within data streams, and customers can replay
them and consume as much as needed during the retention period.
Data stored in the stream can be processed by real-time applications
to generate real-time analytics, execute real-time ETL, and then deliver the continuous
data to an end destination such as a data lake, warehouse, or database.
To ensure high analytics performance for certain stream data, customers can
set up enhanced fan-out, or EFO, consumers to avoid data-read congestion.
Each EFO consumer is given its own dedicated bandwidth within a shard
to ensure consistent low latency.
This feature is supported by Amazon Lambda, Apache Flink, Kinesis Client Library, and more.
In this example, Shard 1 has been set up with three EFO consumer applications.
As a result, these three will not have to fight for bandwidth with other applications.
To further help customers with managing growing data volume,
Kinesis Data Streams provides hands-free scaling.
This is now the default option when you create a data stream, ensuring
that you only pay for what you use.
Streaming data is stored in the order received for a set duration of time,
and can be replayed indefinitely during that time.
The default retention period for stream data is 24 hours, but customers can
extend it to as long as one year, according to their data replay and retrieval needs.
On the data ingestion side, Kinesis Data Streams can be connected with tens of
thousands of data sources from Amazon and other data producers.
On the data consumer side as well, Kinesis Data Streams can be connected
with many applications from Amazon and third-party providers.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it's your turn to try.