In this video, you’ll see how to seamlessly integrate AWS Lambda with Amazon
Managed Streaming for Kafka (Amazon MSK).
With this solution, you can use Amazon MSK as an event source to trigger an
AWS Lambda function that processes events from Kafka topics.
We’ll begin in the Amazon MSK console, where we already launched a cluster
with AWS Identity and Access Management (IAM) authentication enabled.
Let’s view the client information we’ll need.
We’ll copy the bootstrap server private endpoint connector, which will allow us
to establish an initial connection to our cluster.
We’ll also copy the Amazon Resource Name (ARN) of the Kafka cluster,
which we’ll use to create our access policy.
Now let’s navigate to the AWS Cloud9 IDE, which we’ll use to create our Kafka client.
First, we’ll connect to an Amazon Elastic Compute Cloud (Amazon EC2) instance
that already has Apache Kafka installed.
We’ll create a topic called exampletopic and pass it the bootstrap server
connector that we copied earlier.
Now that the topic has been created,
let’s start our Kafka console producer using the same bootstrap server connector.
Our Kafka console producer has been created.
Next, let’s navigate to the IAM Management Console.
We’ll create a new IAM policy so we can give our Lambda function the necessary
permissions to access the Apache Kafka APIs and the Amazon MSK cluster APIs.
Let’s configure this policy to allow the Lambda function to access our Kafka
cluster operations, our cluster topic, and our consumer groups.
Here, we have passed the ARN, specified the cluster, and specified exampletopic as the topic.
We’ll also specify the consumer group.
Let’s review and name the policy.
Now, we’ll create the policy.
The policy has been created.
Next, we’ll create the role for the Lambda function.
We’ll use Lambda as our AWS service.
Next, we’ll add KafkaConsumerPolicy, which is the policy we just created.
We’ll also add an AWS managed policy called AWSLambdaMSKExecutionRole.
Now let’s name, review, and create the role.
The role has been created.
Next, we’ll create a Lambda function so we can use Amazon MSK as an event trigger for Lambda.
We’ll author the function from scratch.
Let’s name the function.
For the runtime, we’ll select Node.js 12.x.
For the permissions, we will use the existing LambdaMSKConsumer role that we just created.
Let’s create the function.
Now that our function has been successfully created, we can input our code.
We’ll replace the default code with a simple script that logs all the messages
that we receive from our Amazon MSK cluster.
Let’s deploy our changes.
Next, let’s go to the Configuration tab and create an event source mapping
for our Lambda function.
We’ll add a trigger.
We’ll select MSK as our source.
We’ll select the cluster that we saw earlier.
We’ll use the defaults for the batch size, starting position, and batch window.
For our topic name, we’ll specify exampletopic, which is the topic we created earlier.
Now, let’s add our trigger.
It will take a couple minutes for the trigger to finish creating.
Once our trigger is in the Enabled state, we can review its details.
Let’s return to our producer client and send some messages to our Amazon MSK cluster.
Now, let’s return to our Lambda function to see how it reads and
processes these messages in the logs.
We'll navigate to the Monitor tab.
We can view the CloudWatch logs to see our log streams.
Let’s open the first log stream.
Here, we can review our Lambda function’s event payload, which we
received from our event source mapping.
We can expand individual records to see details about the message,
including its partition identifier, timestamp, and encoded value.
Once the message has been decoded, it is logged as well.
Here we can see the message that was sent from MSK to Lambda.
You’ve just seen how to seamlessly integrate AWS Lambda with Amazon MSK.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.