In this video, you’ll see how to process data from Amazon Kinesis Data Streams
with Amazon Kinesis Data Analytics, a managed service for Apache Flink.
With this solution, you can read streaming data from a Kinesis data stream,
transform the data and output the results to another Kinesis data stream,
and run real-time queries on the transformed data from a Studio notebook.
Before we begin, let’s review the Amazon services we’ll be using.
Amazon Kinesis Data Streams is a fully managed service for ingesting high
velocity streaming data at any scale from a range of AWS services as well
as external devices like mobile apps and sensors.
Once data has been ingested, it can be consumed by a number of different
services and applications.
In this video, our consumer will be Amazon Kinesis Data Analytics,
which is a managed service for Apache Flink.
We’ll use Amazon Kinesis Data Analytics to read data from Amazon Kinesis Data
Streams, perform some transformations upon this data, and then output the
results to another Kinesis data stream.
To get started, let's walk through the steps for provisioning an
Amazon Kinesis data stream.
First, we’ll give our data stream a name.
Next, we’ll choose our data stream capacity.
On-demand mode provides a stream that adapts to variable input and output traffic.
As shown here, the maximum write capacity for ingesting data in on-
demand mode is 200 megabytes per second and 200,000 records per second.
Read capacity can scale up to 400 megabytes per second.
With provisioned mode, we need to specify the stream's capacity.
For Kinesis Data Streams, the unit of scale is a shard.
The write capacity per shard is one megabyte per second and one thousand records.
Read capacity is two megabytes per second.
For automatic scaling, we’ll select on-demand mode.
The default data stream settings can be edited after the data stream is created and active.
For demonstration purposes, we have already created two data streams,
so let’s cancel out of this creation process.
We have one input and one output data stream.
Let’s switch to our Python environment to explore how we’ll produce data into our input stream.
Kinesis Data Streams can capture data from many sources, including application
and service logs, clickstream data, sensor data, and in-app user events.
For this example, we’ll generate data by running an Apache Flink Table API locally.
We have defined our table as the destination sink.
We also specified a table schema for sales orders that will be generated,
a Kinesis connector, the name of our input stream, our AWS Region, and the output format.
We’ll be using a built-in Apache Flink DataGen source object to generate
mock data for our sink table.
Let’s run the program.
The program has started producing data.
Let’s return to Amazon Kinesis to view our input data stream.
We’ll open the Data viewer tab.
Next, we'll select one of the shards from the stream and get its records.
Now that we’ve confirmed we’re generating data into our Kinesis Data Stream,
let’s return to our Python environment.
Here, we have another Apache Flink application.
In this case, the application reads the data from our input stream, transforms
it by performing some aggregations, and then writes the transformed data
to our destination stream.
This destination sink schema is formatted for aggregated data.
We’ll use a tumbling window to aggregate the total count and sum amount of each
product in the various orders we're getting in 10 second intervals.
The transformed aggregated data is then written to our destination data stream.
Now that we’ve viewed the code, let’s see how to create a Kinesis
Data Analytics streaming application.
The Apache Flink configuration defaults to the most recent version.
We’ll give the application a name.
For access permissions, we can either create or choose an AWS Identity and
Access Management (IAM) role.
Then we can review the settings and create our streaming application.
For our purposes, we’ve already created an application that is actively processing data.
Let’s switch to that one.
First, we’ll review the application’s configuration.
Here, we specified the application code location.
The path includes the Amazon Simple Storage Service (Amazon S3) bucket
where the code is stored and the name of the compressed code file.
Additional configuration options include Snapshots, Checkpoint configuration,
Scaling, Logging and monitoring, and Networking.
Now let’s open the Apache Flink dashboard to learn about the running application.
This graph provides details about the running job, which is useful for
troubleshooting and fine-tuning application performance.
Now let’s go to Amazon Kinesis Data Analytics Studio, where we can create
a Studio notebook that will allow us to interact with the streaming data.
We can use a Quick create method or create a notebook with custom settings.
We then give the notebook a name, and specify an IAM service role and an AWS
Glue database to define the metadata for our source and destination.
We’ll be working with a previously created Studio notebook,
so we’ll switch over to that now.
Let's open this notebook in Apache Zeppelin.
In Apache Zeppelin we'll open a code file that contains our order viewer.
This will allow us to view the transformed orders that are being written to our output stream.
First, we define a table that specifies our schema and the details about where we
are reading our order data from.
Again, we’ll specify that we are using the Kinesis connector.
We'll be reading from our order output data stream.
Let’s run the code to create the table.
With our table created, we’ll run a real- time query to view incoming orders
where the number of products in the order is greater than one.
Behind the scenes an Apache Flink job is started, the output of which is
presented to us in this Zeppelin notebook.
Our real-time query refreshes the results every 10 seconds
We can click the FLINK JOB link to view the job in the Apache Flink dashboard.
In the dashboard, we can see our job is running and explore its job graph.
You've just seen how to process data from Amazon Kinesis Data Streams
with Amazon Kinesis Data Analytics.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.