In this video, you’ll learn about Amazon Kinesis Data Firehose, a fully managed
service that reliably captures, transforms, and delivers streaming data to data lakes,
data stores, and analytics services.
To get started, let’s review how data is streamed.
At a very basic level, all stream processing applications include five stages.
The process begins with data generated by multiple sources, or producers.
These sources may include mobile SDKs, IoT devices, click stream
products, or application logs that produce real-time data at high velocity.
Second is stream ingestion, when data from tens of thousands of data sources
is collected and ingested in real time.
This stage may include value-add processes such as authentication or throttling.
This layer can also filter out data from poor sources.
The third stage is stream storage.
Data is stored in the order received for a set duration.
The fourth stage, stream processing, is when the stored data is analyzed to gain insights.
Records are always read in the order in which they are received,
which makes stream processing different from other types of processing.
The final stage is to send the processed data to a destination, such as a data lake,
data warehouse, or database, where it can be held longer term
for analysis or auditing purposes.
Within this overall flow, Kinesis Data Firehose covers stream ingestion,
storage, processing, and destination delivery.
It provides near real-time streaming extract, transform, and load (ETL)
processing for any data, at any scale, and at low cost.
Let’s take a closer look at the ETL process.
Sources of streaming data include AWS service logs, IoT device data,
clickstream data, and enterprise applications, among many others.
Stream ingestion options include AWS SDK, and Amazon Kinesis Data Streams.
For logs generated by distributed applications, you can also use Amazon
Kinesis Agent or open-source agents such as Fluent Bit.
Stream storage and processing are handled by Kinesis Data Firehose,
which can perform two types of transformations.
You can use a Lambda function to do custom transformations, or use built-in
transformation capabilities such as data format conversion or dynamic partitioning.
Kinesis Data Firehose also supports other capabilities such as buffering,
compression, and encryption of data.
Stream destination options supported by Kinesis Data Firehose include
Amazon S3, Amazon Redshift, Amazon OpenSearch, and Amazon API Gateway,
as well as many third-party destinations.
Kinesis Data Firehose enables a wide variety of use cases.
These include click-stream, log, and IoT analytics, along with offline event
analysis and network security monitoring.
Let’s go over some of the benefits of using Kinesis Data Firehose.
Creating a pipeline using Kinesis Data Firehose is easy.
With just a few clicks, you can create a delivery stream, select your destination,
and start streaming real-time data.
With Kinesis Data Firehose, your delivery streams automatically scale up to handle
gigabytes per second, maintaining data latency at levels you specify.
Kinesis Data Firehose synchronously replicates data across three facilities in
an AWS Region, providing high availability
and durability for the data as it is transported to the destinations.
Kinesis Data Firehose is seamlessly integrated with more than 30 AWS
services and streaming destinations, making it easy to ingest and deliver streaming data.
Kinesis Data Firehose handles data loss and data corruption scenarios through
automatic retries and provides data buffering and transformation capabilities with no code
development or maintenance required.
And finally, with Kinesis Data Firehose, you pay only for the amount of data you ingest.
There are no minimum fees or upfront commitments.
Amazon Kinesis Data Firehose buffers incoming streaming data for a specified
period of time or to a specified volume before delivering it to its destination.
The buffer is flushed to the destination upon whichever threshold is hit first.
Buffering helps to avoid small files in Amazon S3 and reduce high storage costs.
Kinesis Data Firehose can convert the format of your input data from JSON to
Apache Parquet or Apache ORC before storing the data in Amazon S3.
Parquet and ORC are columnar data formats that save space and enable
faster queries compared to row-oriented formats like JSON.
If you want to convert from an input format other than JSON, such as
comma-separated values (CSV) or structured text, you can use an inline AWS
Lambda function to transform it to JSON first.
Kinesis Data Firehose also supports dynamic partitioning.
This enables you to continuously partition streaming data using keys
within the data and then deliver the data grouped by these keys into
corresponding Amazon S3 prefixes.
In this example, the user has partitioned the data based on customer ID, year, month, and day.
The data is delivered into specified Amazon S3 prefixes
within the configured bucket.
Partitioning your data minimizes the amount of data scanned, optimizes
performance, and reduces costs of your analytics queries on Amazon S3.
Kinesis Data Firehose also allows you to configure your stream with a Lambda
function to transform incoming source data before delivering it to its destination.
You can use existing Lambda blueprints for this purpose, or write custom code to
transform your source data.
You can monitor Kinesis Data Firehose using several different Amazon features.
Kinesis Data Firehose sends Amazon CloudWatch logs and custom metrics with
detailed monitoring for each delivery stream.
Kinesis Agent publishes custom CloudWatch metrics to help assess
whether the agent is working as expected.
Kinesis Data Firehose uses AWS CloudTrail to log API calls and store the data in an Amazon
S3 bucket, and to maintain API call history.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.