In this video, you’ll see how to run queries across Amazon Managed
Service for Prometheus workspaces.
With this solution, you can deploy Promxy, an open-source Prometheus
proxy, in your container cluster to enable a single Grafana data source for
querying multiple Prometheus workspaces.
Let’s quickly review the solution architecture.
We have three Amazon Managed Service for Prometheus workspaces
receiving metrics from multiple Amazon Elastic Kubernetes Service
(Amazon EKS) clusters.
Promxy is deployed between Amazon Managed Service for Prometheus
and Amazon Managed Grafana.
Two ingress resources provide basic authentication and application load balancing.
Note that we’re using Amazon EKS, but other platforms can be used.
To get started, let’s go to Cloud9 to prepare our Amazon EKS cluster.
First, we’ll associate an AWS Identity and Access Management (IAM)
role with our service account.
To do this, we need an IAM OpenID Connect (OIDC) identity provider for our cluster.
Let’s run a command to determine whether we have one configured.
Our cluster does have an IAM OIDC identity provider.
If we didn’t have an IAM OIDC identity provider, we could create one by
following these instructions in the AWS documentation.
The first step in our deployment is installing the Application Load Balancer
Controller, which automatically provisions an application load balancer
when a Kubernetes Ingress resource is created.
This website provides instructions for creating the necessary IAM policy, role,
and Kubernetes service account.
For the purposes of this example, we’ve already gone through this process,
so we’re ready to deploy the application load balancer helm chart.
Let’s return to Cloud9 and do so.
Now that the controller is deployed, let’s verify the resources.
Promxy doesn’t provide an authentication mechanism, so we’ll use
the Ingress NGINX controller for authentication.
Let’s deploy the Ingress NGINX helm chart.
Notice that we’re setting the service type to NodePort.
This is to prevent the NGINX controller from automatically creating the classic
load balancer for Ingress resources.
Instead, we’ll be using the AWS Load Balancer Controller to create an
application load balancer.
Next, let’s verify the resources.
To complete the authentication setup, we’ll create a username and password,
then convert them into a Kubernetes secret.
Let’s examine the secret to confirm that it was properly created.
Let’s go to the AWS Management Console.
To prepare for the Promxy deployment,
we need to create an IAM role to give Promxy permissions to query metrics
from Amazon Managed Service for Prometheus workspaces.
We’ll create a custom trust policy.
The trust policy will specify the account ID, Region, and OIDC identity provider information.
Next, we’ll add the AWS managed policy that provides query access to Prometheus.
Let’s enter a name for the role and create it.
Before we deploy Promxy, we need to make some changes and additions to the files.
Let’s clone the Promxy GitHub repository to our local file system.
Next, we’ll create some supplementary files to modify the default Promxy
deployment and create additional files for the application load balancer
and Ingress NGINX resources.
The first file we’ll create will provide values to overwrite the default Promxy
configuration and define additional values that we’re adding to the Promxy deployment.
First, we define the service account to be used.
We then disable the default Ingress resource provided by Promxy.
Next, we create two new Ingress
resources, one for the application load balancer, and one for NGINX.
Next, we define a sidecar container that utilizes AWS Signature version 4 to add
authentication information to calls that Promxy makes to Amazon
Managed Service for Prometheus.
Finally, we define the Prometheus workspace connection details.
Note that the entire static_configs section is duplicated, and each
contains a separate workspace ID.
This section should be replicated for every Amazon Managed Service
for Prometheus workspace to which Promxy will proxy requests.
Next, we need to create the files that define our Ingress resources.
Let’s go to the templates directory.
Here, we’ll create two files.
The first one is for the application load balancer Ingress resource.
Second, we’ll create the file that defines the NGINX Ingress resource.
Next, in addition to adding the AWS Signature version 4 sidecar
container to our override file, we need to ensure that it gets added to the
default deployment.yaml file so that the container gets deployed.
Near the end of this file, we can see that the sidecar container has been defined.
We are now ready to deploy the Promxy helm chart.
Let’s go to the base directory where the Promxy GitHub repository was cloned
and initiate the deployment.
Let’s verify the resources.
Once the deployment is complete,
the application load balancer will be automatically deployed.
Let’s obtain the application load balancer URL.
We’ll use this URL when we define the data source in Grafana.
Now we’re ready to define our Grafana data source and create visualizations.
Let’s create a new Prometheus data source within our Amazon Managed Grafana workspace.
We’ll name our data source “Promxy.”
Let’s paste in the application load balancer URL we obtained.
We’ll enable basic authentication and specify the username and password we defined earlier.
Let’s save and test the data source.
Notice that it says the data source is working.
Let’s explore the data source.
As you can see, we have multiple EKS clusters, each writing to a
separate Amazon Managed Service for Prometheus workspace.
Let’s open a dashboard that displays the Prometheus metrics.
On the left side, the dashboard shows Prometheus metrics
aggregated by pod and EKS cluster.
On the right side, the same metrics are aggregated across EKS clusters
and across Amazon Managed Service for Prometheus workspaces.
All of this is done using a single data source, pointing to Promxy, which
proxies the queries to every Amazon Managed Service for Prometheus
workspace that has been defined.
You’ve just seen how to run queries across Amazon Managed Service
for Prometheus workspaces.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.