In this video, you’ll learn about scaling Kinesis Data Streams for high throughput and low latency.
Real-time stream processing with high throughput and low latency has a wide variety of use cases.
One important use case is anomaly and fraud detection in real time.
Within financial services, organizations need to identify security breaches,
network outages, or machine failures within milliseconds.
They must also respond to potentially fraudulent credit card transactions
or sign-in attempts in near-real time by checking live activity streams against
historic account usage data.
Another use case is tailoring customer experiences in real time.
Media and entertainment companies, for instance, use website visitors’
clickstream data to gain immediate insights and personalize users’ online
experiences with curated content.
Real-time stream processing with high throughput and low latency is also used
to empower IoT analytics, facilitate predictive maintenance, nourish
marketing campaigns, and support healthcare and emergency services.
Amazon Kinesis Data Streams is a serverless data streaming service that
allows any organization to ingest, store, and process large data volumes at scale.
A Kinesis data stream is a set of shards, each containing a sequence of data records.
Producers put data records into shards,
and then consumers get data records from shards and process them.
Kinesis Data Streams allows you to choose between two capacity modes
that determine how capacity is managed and how you are charged for data stream usage.
Provisioned mode is best suited for consistent traffic with predictable capacity requirements.
You specify the number of shards for the data stream to match the expected data
traffic and pay for charges at an hourly rate.
Each shard ingests up to 1 megabyte and 1,000 records per second,
and emits up to 2 megabytes per second.
If capacity is exceeded, the application will be throttled.
If you require an increase in the shard limit for a stream, you can contact
AWS Support Center to request this.
On-demand mode is best suited when the data stream’s throughput
requirements are unpredictable and variable.
In on-demand mode, data streams automatically scale to handle gigabytes
of write and read throughput per minute.
You pay only for the actual throughput use.
Streams with an on-demand capacity mode can scale up to 200 megabytes of
write and 400 megabytes of read throughput.
If you require an increase in capacity, you can contact AWS Support Center to request this.
Overall, if your data volume is predictable, use provisioned streams.
If your data volume can spike unpredictably
or if you don't want to manage the number of shards, use On-demand streams.
An Amazon Kinesis Data Streams producer is an application that puts user
data records into a Kinesis data stream.
Data from tens of thousands of data sources can be collected and ingested in real time.
There are three ways for customers to build data producers.
First, Amazon service integrations enable you to build streaming-data use
cases across a variety of data source types.
Second, some third-party applications have native integrations for producing
data into Kinesis Data Streams.
Third, AWS toolkits and libraries allow you to build custom data producers.
More details can be found in our documentation.
A consumer is an application that processes all data from a Kinesis data stream.
Records are read in ingested order to enable real-time analytics or streaming ETL.
You can use AWS services, third party applications,
or build your own data consumer with the Kinesis Client Library or the AWS SDK.
The type of consumer you’ll build will depend on whether you need dedicated throughput.
Standard consumers share throughput when reading from a shard.
Every shard in a stream provides up to 2 megabytes per second of read throughput
and is limited to five transactions per second.
If multiple applications consume messages from a shard, then 2
megabytes per second of read throughput are shared among them.
Consider using standard consumers if you have fewer than three consuming applications,
you can tolerate around 200 milliseconds of latency, and you want to minimize cost.
Enhanced Fan-Out consumers get dedicated throughput of 2 megabytes per second.
This allows you to scale the number of consumers reading from a data stream
in parallel while maintaining high throughput and low latency.
You can register up to 20 consumers per stream to use Enhanced Fan-Out.
If you need to increase this limit, you can contact AWS Support Center with a service request.
Consider using Enhanced Fan-Out when you have multiple consumer applications
reading from the same stream, or you require latency of around
70 milliseconds or better for message delivery to consumers.
Additional charges will apply.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it's your turn to try.