In this video, you’ll see how to analyze streaming data using Amazon Kinesis Data Analytics Studio.
With this solution, you can create a Kinesis Data Analytics Studio notebook,
connect to an Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster,
and interactively analyze events from the stream.
We’ll start from the Amazon MSK console, where a cluster has already been created.
We’ll connect to this cluster using a Kafka client that we’ve already created.
Let’s enter the Kafka console command to list our topics.
Notice that we have two topics: input stream and output stream.
The input stream is where we’ll be ingesting data into Amazon MSK,
and the output stream is where we’ll get our results from the Kinesis Data Analytics application.
To simulate the workload, we’ll use a simple Python script that generates
random sets of sensor data.
The script sends random records to the Amazon MSK input stream topic in JSON syntax.
Let’s start the producer.
Now that data is being produced into our input stream topic, let’s return to the
Amazon MSK console and create our Kinesis Data Analytics Studio application.
First, we’ll select our cluster and choose the option to process the data in real time.
For this example, let’s create an Apache Flink - Studio notebook.
Kinesis Data Analytics will attach the required AWS Identity and Access
Management (IAM) policies for reading from the Amazon MSK cluster.
It will also input the VPC configurations that we need.
Let’s give the Studio notebook a name.
Kinesis Data Analytics will create an IAM role with the permissions that we need.
Let’s create an empty AWS Glue database.
Let’s refresh the list and select the database we just created.
This database will define the metadata for our sources and destinations.
Let’s review the settings and create the Studio notebook.
The notebook has been successfully created.
Let’s review its configurations.
Before we can use the notebook, we’ll need to run it.
The notebook will take a few minutes to start.
Once the Studio notebook has been successfully started, we can open it in Apache Zeppelin.
Let’s create a new note and call it “Sensors.”
Next, we’ll create a Sensor data table describing the format of the data in our stream.
The first line in this command tells Apache Zeppelin to provide a stream
SQL environment for the Apache Flink interpreter.
The subsequent lines use familiar SQL statements to create a table that will
store the sensor data in the stream.
We’ve specified our input stream as a topic, and we’re using the bootstrap
server of our Amazon MSK cluster.
Let’s run this command to create a table.
We can find the table we just created in the AWS Glue Data Catalog database used by this notebook.
Here’s the sensor data table we created.
Let’s return to the notebook.
Now we can run SQL queries on the data table and use sliding or tumbling
windows to get a better understanding of what’s happening with our sensors.
First, let’s select all the content in the sensor data table.
To get a better understanding of this data, we’ll switch to the bar graph view.
Let’s modify the settings to group the results by status to see the average current temperature.
As expected, we have different average temperatures depending on the status.
The higher the temperature, the greater the probability that something is not
working correctly with our sensors.
Next, we’ll run an aggregated query using SQL.
This time, we want the results computed on a sliding window of one minute,
with the results updated every 10 seconds.
To do this, we’ll use a HOP function in the GROUP BY section of the SELECT statement.
Let’s keep these results in table format.
If desired, we can send the results of this query to the destination topic
output stream we created earlier.
To do this, let’s create a table called “sensor_state.”
Notice that we’re using the output stream from our demo cluster.
Now that our table is created, we can use an INSERT INTO statement to continuously
insert the results of the SELECT statement into our sensor_state table.
We’ll use the Kafka console to confirm whether Kinesis Data Analytics Studio is
actually ingesting data into our output stream.
We can confirm that data is being received into our output stream.
This data can now be used for other downstream applications, dashboarding, and further analytics.
You’ve just seen how to analyze streaming data using Amazon Kinesis Data Analytics Studio.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.