In this video, you’ll see an example of streaming ETL with Amazon Managed
Streaming for Apache Kafka (Amazon MSK) and Kinesis Data Analytics.
With this solution, you can run Apache Kafka clusters on AWS, build data
processing logic for streaming applications, and consume data in
near-real time to gain faster insights.
For our example, we’ll generate some sample forklift data and stream it to an
Apache Kafka cluster running in Amazon MSK.
We’ll then use Kinesis Data Analytics to perform real-time data processing
and send the processed data to Amazon OpenSearch Service, where we can view
and query a dashboard of results, even from a mobile app.
Amazon MSK is a fully managed service for running Apache Kafka clusters on AWS.
To get started, let’s see how to create a Kafka cluster.
We’ll use the Quick create option, which defaults to the recommended settings.
For the cluster type, we can choose Serverless or Provisioned.
The Serverless option automatically scales with application load.
The Provisioned option allows us to customize the broker settings for the cluster.
We can also customize these cluster settings.
For the purposes of this video, we already created a cluster,
so let’s cancel out here and go to the Clusters page.
Here’s our cluster.
Let’s look at it.
The cluster is active and ready to receive data.
We’ll now run two background processes to start sending continuous data to our cluster.
This dummy data stream represents real- time operations data for a fleet of forklifts.
The operations data we are sending includes jobs assigned to forklifts,
along with forklift performance and health metrics.
Now, we'll go to Amazon Kinesis to perform data analytics processing
on our streaming Kafka data.
On this tab, we can see our streaming applications.
Let’s select the Studio tab.
From here, we can create a Studio notebook to build our application.
Let’s create this notebook with custom settings.
We’ll give the notebook a name and continue to the next step.
We’ll leave the default AWS Identity and Access Management (IAM) permissions.
Next, we can specify or create the AWS Glue database that stores the metadata
for the source and destination referenced from our notebook.
In this case, we’ll choose an existing database.
Let’s proceed to the next screen.
Let’s retain the configurations for scaling and for logging and monitoring.
Let’s update the VPC connectivity setting.
As our Amazon MSK cluster is running within the same AWS Region and
account as our Amazon Kinesis application,
we can select VPC configuration based on Amazon MSK cluster.
When we specify our MSK cluster, the network settings are automatically populated.
Studio notebooks also add default connectors.
Connectors are required for connecting to and querying Amazon MSK data.
We’ll leave the defaults for the other sections and continue to the last step.
Now we can review and create the Studio notebook.
For the purposes of this example, we already have a notebook created,
so we’ll cancel this one.
Here’s the notebook, which has the same configurations we just saw.
Let’s open it in Apache Zeppelin.
With Apache Zeppelin we can write processing queries in SQL, Python,
or Scala, and test our analysis in real time.
Here’s the notebook we created for this demonstration.
We’re using Flink SQL to create two tables to connect with the two Kafka
topics that are streaming our forklift data.
We’ll run our SQL code to create our tables.
The tables have been created and are ready to be queried.
Behind the scenes, forklift data is still being continuously streamed to our Kafka topics.
Let’s run a query to select our forklift job table.
The incoming data on the Kafka topic is streaming to this table.
This table is showing all forklift job data: the forklift ID, job ID, when it has started,
when it is ending, who is driving it, and so on.
Next, let's run an aggregation query on our forkliftjob table to see in real-time
how many machines are running, faulty, or ideal.
Studio notebook allows many visualization formats for query results.
We’ve chosen the bar chart for our machine status results.
We can also change the settings to display different columns from our database table.
The chart changes in real time as the streaming data arrives.
Our forkliftstatus table provides deeper insight into forklift machine status.
This table includes details such as coolant, battery, and shock levels.
Next, we’ll run a query to see real-time aggregation about max shock level,
max battery level, max coolant level, and machine status at a given time.
Our query retrieves aggregate battery, coolant, and shock data every 10 seconds.
Now that we’re happy with our data processing logic, we will create an
OpenSearch based off our SQL code.
We’ll create another sink table that will be populated from our view data.
An Amazon OpenSearch Service cluster is connected to this table by an index.
Let’s quickly switch over to Amazon OpenSearch Service to see where
we retrieved this index from.
We have a previously created forklift cluster running here.
The index settings from this cluster are used in Apache Zeppelin in the sink table creation.
The sink table is now created.
Let’s switch to OpenSearch.
On the OpenSearch dashboard, we can see the data we’re consuming.
We can filter by specific fields to drill down further.
We can also look at metadata for a specific machine.
Studio notebooks are useful for creating ad-hoc queries and building data
processing logic, but aren’t designed to be run continuously.
Ultimately, data processing logic should be deployed as an Apache Flink application.
We can build the Flink application right from here and then deploy it as
a Kinesis Data Analytics application.
Once created, the streaming application can be accessed within Amazon Kinesis Analytics.
You’ve just seen how an example of streaming ETL with Amazon
MSK and Kinesis Data Analytics.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it's your turn to try.