In this video, you’ll see how to process streaming data with
Amazon Kinesis Data Analytics.
With this solution, you can create a streaming source for an Amazon
Managed Streaming for Apache Kafka (Amazon MSK) cluster, build a stream
processing application using a Studio notebook, and leverage Apache Flink
for notebook data operations.
Amazon MSK is a fully managed service for running Apache Kafka clusters on AWS.
To get started, let’s create a new cluster.
We’ll use the Custom create method.
We’ll give our cluster a name and leave the Serverless cluster type option selected.
Let’s continue to the next page.
Here we’ll define the virtual networking environment for our cluster.
We’ll choose a VPC that we previously created in our account.
To achieve high availability, we’ll specify three availability zones and the subnet for each.
Next, we’ll need to choose a security group.
We’ll select the Custom option and choose a group that we previously created.
This group has the required AWS Identity and Access Management (IAM) permissions.
Let’s continue to the next page.
Amazon MSK serverless clusters work with IAM role-based authentication.
We already have an IAM security group policy defined, so let’s continue to the next page.
We’ll retain the defaults for metrics, and we won’t add any cluster tags.
Now we’ll quickly review our cluster settings and create the cluster.
The creation process for this serverless cluster will take up to five minutes.
Now that our cluster is successfully created, let’s navigate to Amazon
Kinesis to create a Studio notebook.
We’ll do this from the Streaming applications page.
We’ll create this Studio notebook with custom settings.
Let’s give our notebook a name.
We'll leave the runtime as Apache Flink 1.13 and continue to the next page.
For Kinesis Data Analytics to access our data sources and destinations, we must
provide an IAM role with the required permissions.
We'll choose an IAM role that was previously created in our account.
Next, we must specify an AWS Glue database that defines the metadata
for our sources and destinations.
We’ll choose the default database.
Let’s continue to the next page.
We’ll keep the default configurations for Scaling and Logging and monitoring.
Under Networking, we’ll choose the VPC connectivity option that bases the VPC
configuration on our Amazon MSK cluster.
The Studio notebook automatically populates the subnets and security
groups with the cluster’s networking settings.
We have the option to deploy our Studio notebook application executable to an
Amazon Simple Storage Service (Amazon S3) bucket.
We’ll decline that option for now and leave the remaining configuration
sections with their default values.
Let’s continue to the next page.
Finally, we'll review our Studio notebook settings.
By default, Kinesis Data Analytics Studio includes connectors for Flink SQL,
Kafka, and Amazon MSK IAM authentication.
Let’s go ahead and create our Studio notebook.
Now that our Studio notebook is successfully created, let’s run it.
Starting the Studio notebook can take a couple of minutes.
Our Studio notebook has successfully started.
Let’s open it in Apache Zeppelin.
We’ll import two notes containing code samples that we'll be using for our demo.
The first is for ingesting data to create our data stream.
The second is to process the ingested data stream.
Now that we've imported both code files into our Zeppelin notebook, let's open
each note in a separate browser tab.
Here’s the code that ingests data into our Amazon MSK cluster.
And here’s the process code for the Flink application.
Let’s now retrieve the connectivity information from our cluster.
Back in Amazon MSK, we’ll select the serverless cluster we created earlier.
We’ll view the client information.
Let’s copy the bootstrap endpoint so that we can use it in our Zeppelin notebook.
Now, let’s return to Zeppelin.
Our newly created Amazon MSK Kafka cluster is currently empty.
To confirm that, we’ll run the code in this cell to see a list of topics currently in the cluster.
As expected, the cluster has no topics.
We can run the code in the next cell to create a topic for our cluster.
We now have a topic named AmznReviewsTopic.
Next, we’ll produce some data so that we can do real-time processing on it.
We have a sample dataset in an S3 bucket.
We’ll be reading the data files of the dataset to run a sample producer.
The producer will create a data stream which we will use to create
our processing application.
Let's run the code and start producing the data.
Now that our Amazon MSK cluster has a topic with data being produced to it,
let's switch over to the process code in our notebook.
We’ll use this code to create a streaming table that connects to the Kafka topic
we created and pulls data from it.
The code points to our serverless cluster, providing the properties of
the bootstrap endpoint and ensuring it uses SASL and IAM authentication.
Let’s run it.
Now we can start querying the table in real time to view the streaming data.
Let's run the select query.
While the select query runs, we’ll create a destination table for query output.
The table will output files in CSV format to an Amazon S3 bucket location.
Let’s run the code.
In the previous cell, we now see data coming into our source table,
which means Apache Flink is able to read real-time Amazon
reviews from our Kafka topic.
We’ll now stop the source table select query, as we have confirmed
that we're able to read the data.
Our next step is to create the checkpoint configuration for inserting streaming
data into our destination table.
Now we’ll run a SQL insert statement that inserts customer_id, product_id,
and star_rating data into the destination table from our source table.
While the code is running, let's switch back to the ingest notebook.
We can see the cell for the producer is still running because it has to read all the
data in the files and keep producing the data.
Let’s return to the process notebook.
While the streaming data is inserting into our destination table, let’s find the
location of the Amazon S3 bucket we created.
Let’s view the bucket.
The output folder contains product partition folders that each contain data files.
This notebook can be run continuously to create streaming SQL jobs on top of the cluster.
You’ve just seen how to process streaming data with Amazon Kinesis Data Analytics.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.