In this video, you’ll see how to securely send cross-account Prometheus metrics from
Amazon Elastic Kubernetes Service (Amazon EKS) to Amazon Managed Service for Prometheus (AMP).
With this solution, you can view all your Prometheus metrics from a centralized account,
configure fine-grained permissions for cross-account access with Amazon EKS,
and achieve private data transfer with virtual private cloud (VPC) endpoints.
The architecture we’ll use includes two workload accounts,
each running Amazon EKS clusters and a Prometheus server,
and a central monitoring account with an AMP workspace connected to Amazon Managed Grafana.
We’ll work in the Cloud9 integrated development environment.
For demonstration purposes, we’ve already set up some AWS accounts.
This is workload account 1.
This is workload account 2.
This is the central account.
Let’s return to workload account 1.
This script and others we'll use in this video rely on dependencies such as kubectl, eksctl, and helm.
Let’s begin by creating an AWS Identity and Access Management (IAM) role for the Kubernetes service account with permissions to assume a cross-account role.
Next, we’ll prepare the environment by setting the region, Availability Zones, and CIDR blocks.
Next, we’ll get the workload account ID and configure AWS authentication.
Now, we’ll create a new VPC.
Let’s name the VPC and enable DNS.
We’ll create an internet gateway and attach it to the VPC.
Next we'll create the subnets and the associated route tables.
Let’s enable public IPs on the subnets and associate route tables with them.
Before we create the EKS cluster, let’s review our VPC from the VPC Management Console.
Here’s the VPC we created.
Let’s take a closer look at its associated subnets and internet gateway.
Now let’s go back to workload account 1 and create the EKS cluster.
We’ll now repeat this entire process to create an EKS cluster for workload account 2.
Let’s fast forward to when the second cluster has been created.
To help manage user access, let’s associate the OpenID Connect (OIDC) provider with both clusters.
Now let’s go to the EKS console in workload account 1 to find the cluster we created.
Let’s review the configuration and compute details.
We can see that some instances have been provisioned.
We can’t see the nodes because some authentication is needed to visualize them.
Let’s navigate to Amazon Elastic Compute Cloud (Amazon EC2) to view our instances.
Two EC2 instances are running EKS workloads using the IP addresses on the VPC we set up.
Let’s view the EKS cluster on the second workload account.
Next, let’s check for the EC2 instances.
Here they are, with their IP addresses.
Let’s continue our setup and create a cluster OIDC provider in workload account 1.
Let’s view the trust policy we created.
The policy includes the current account ID.
We can now create the role.
Let’s find the role we just created in the IAM console.
Here it is, with the correct identity provider.
We’ll now repeat the process to create a cluster OIDC provider in workload account 2.
Let’s skip ahead to when this is finished.
As you can see, this role now has permissions to assume a role across accounts.
Next, we’ll go back to the central account and create a Prometheus workspace.
Now we need to create an IAM role that corresponds to the role permissions we set up in the two workload accounts.
Let’s get the two workload account IDs and pass them down to the role.
Let’s go ahead and create the IAM policy.
Let’s take a look at the policy.
Notice that we’re only granting AssumeRole permissions to the EKS role on these specific workload accounts.
Now we can create the role.
We’ll provide remoteWrite access to Prometheus.
Let’s check for this role in IAM from our central account.
Here’s the role we set up.
As you can see, the role has Prometheus remoteWrite access.
This access is essential.
The EKS cluster service account is associated with this role, and this role needs permission to write into AMP in the central account.
Let's review the architecture of this setup.
Notice that the EKS cluster will use the service account, which is connected to the role we created.
Let’s return to workload account 1 and deploy a Prometheus server inside the Amazon EKS cluster
so we can collect environmental metrics.
We’ll need the workspace ID of the central account, so let’s get that.
Let’s paste in the workplace ID and create a YAML file we’ll use to configure the Prometheus server.
Let’s take a look at the YAML file.
Notice that the service account is using the AWS Account ID of the local account.
You can also see the Prometheus server’s remoteWrite URL, queue configuration, and SigV4 specifications.
Let’s deploy Prometheus.
You can use kubectl to see if each pod is running correctly.
Not every pod has been created yet.
Let’s wait a moment and try again.
At this point, Prometheus should be running,
and the metrics should be visible from the central account.
Let’s verify this by connecting to Grafana on the central account.
For this example, we’ve set up a workspace with Amazon Managed Grafana,
which provides a native integration with AMP.
You can also run your own Grafana server.
Let’s configure AMP as a data source for this workspace.
In this case, the region is Ireland.
All the workspaces in the selected region will appear here.
Here’s our AMP workspace.
Let’s add the data source.
Now let’s go to the Explore mode and find the data.
We’ll look in the AMP workspace.
We can see all the metrics that Prometheus is sending right now to AMP.
Let’s query the origin of these commands.
As you can see, the metrics are starting to flow.
We can see the IP addresses the metrics are coming from.
Let’s see if they match the CIDR range of our instances in EC2.
One of the IP addresses here matches an IP address displayed in Grafana.
As does one of these IP addresses.
The metrics are flowing through the public internet via the internet gateway.
Let’s verify that only public internet IP addresses are currently being used for the Prometheus endpoint.
We can see public IP addresses for the Prometheus endpoint.
Traffic is secure because we’re using HTTPS and SigV4,
but to also make traffic completely private, we can add a VPC endpoint.
Let’s do that.
Let’s switch to the VPC console to confirm that the endpoint is being created.
The VPC endpoint is currently pending.
Let’s wait a moment.
The VPC endpoint is now available.
Let’s check the traffic again.
Within a few minutes, traffic will begin flowing privately, without any additional configuration on our part.
As you can see, traffic has shifted from public addresses to private addresses for the AMP DNS state.
Let’s return to Grafana and refresh the query.
Metrics are now flowing from the private endpoint.
Let’s quickly create a private VPC endpoint for workload account 2.
As before, we’ll create the Kubernetes workspace, install Helm charts, and deploy our application.
The Prometheus pods are running.
If we go on Grafana, we should see two additional sets of metrics.
We now have four time-series values coming from four IP addresses,
confirming that metrics from both workload accounts are flowing through AMP in the central account.
You've just seen how to securely send cross-account Prometheus metrics from Amazon EKS to AMP.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.