In this video, you'll see how you can monitor Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters using Prometheus and Grafana.
With this open-monitoring solution, you can ingest Apache Kafka metrics into Amazon Managed Service for Prometheus (AMP),
and then add AMP as a data source in Amazon Managed Grafana so you can visually monitor and the health and performance of your Amazon MSK cluster.
The architecture we’ll set up includes an Amazon MSK cluster and an Amazon Elastic Compute Cloud (Amazon EC2)
instance for the Prometheus server in a Virtual Private Cloud (VPC).
We’ll also create a workspace and VPC endpoint for AMP and a workspace for Amazon Managed Grafana.
When Open Monitoring for Prometheus is enabled on the cluster,
AMP can ingest the Amazon MSK metrics and send them to Grafana for querying and visualizing.
To start, we'll create a new Amazon MSK cluster.
We'll use the “quick create” creation method and retain the default settings.
While the Amazon MSK cluster is being created,
let's configure an AMP workspace where we can ingest our cluster metrics.
Let’s open a new tab and go to AWS CloudShell.
We'll execute the command to create a workspace.
Let's visit the AMP console to confirm the workspace is active.
We can see the workspace.
Let's return to the CloudShell environment.
Now we'll create a secure VPC endpoint for the AMP workspace.
Public endpoints for AMP are available, so this step is optional.
Let's execute the command to identify the default VPC.
Let's also capture its security groups and subnet IDs.
Now we can use these factors to generate the VPC endpoint.
We'll use AWS Single-Sign On for authentication and create a user.
Let's supply our credentials.
We'll retain the service managed permission type and move on.
We'll select Amazon Managed Service for Prometheus as our data source
since that's where our Amazon MSK metrics will be collected and queried from.
Let's complete the workspace setup.
While the workspace is being created, let's grant access to our user account.
We'll also make our user an admin of the workspace.
Let's check on the status of the Amazon MSK cluster.
We can see that the cluster is fully built and has an active status.
Next, we'll build an Amazon EC2 instance that will scrape metrics from our Amazon MSK cluster
and deliver them to the AMP workspace.
Let's open a new tab and navigate to the AWS CloudFormation console.
We'll create a new stack and upload an existing template file.
Let's quickly name the stack and fill in the details.
We’ll need to supply the ID of our AMP workspace, which is where the stack will remote-write the scraped data.
Let’s retrieve that and paste it in.
We also need the Amazon Resource Name (ARN) of our Amazon MSK cluster where the stack will scrape the metric data from.
We'll leave the rest of the parameters as the default and select the subnets to apply to the EC2 instance.
The next step is to configure the stack options.
We’ll leave these as is.
Let's acknowledge the terms and create the stack.
Here we can see the creation of the MSK-Monitoring stack is complete.
Let's navigate to AWS Systems Manager to confirm that the stack document was created.
Let's also confirm that the document commands were run.
Finally, let's check the Amazon EC2 instance.
It's in a running state, so we can proceed with the next step.
From here, we'll connect to our Amazon Managed Grafana workspace and add AMP as a data source.
Let's sign in with our admin user credentials.
Let's access the workspace’s data sources and configure a connection with the AMP workspace we created earlier.
Let's add the data source.
Now we can adjust the settings and rename the data source.
As you can see, we have confirmation that the Prometheus data source is working.
Now we can add a dashboard to monitor the Amazon MSK cluster metrics.
We'll import a JSON file we've already created.
As you can see, the dashboard has no data to display yet.
To allow metrics to flow from Amazon MSK to AMP to Amazon Managed Grafana,
we'll have to enable open monitoring with Prometheus on our cluster.
Back in the Amazon MSK console, let's edit the monitoring settings.
Let's check the box to enable open monitoring with Prometheus and save our changes.
Now when we return to the Grafana dashboard,
we can see it has been populated with the Amazon MSK metrics.
We can use these metrics to monitor cluster traffic and identify performance bottlenecks in near real time.
You've just seen how you can monitor Amazon MSK clusters using Prometheus and Grafana.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.