In this video, you’ll see how to set up multi-cluster cost analysis for Amazon Elastic Kubernetes
Service (Amazon EKS) using Amazon Managed Service for Prometheus and Kubecost.
With this solution, you can get granular visibility into your EKS cluster costs
without manually scaling the Prometheus instance, and get recommendations for
optimizing your Kubernetes workloads to reduce costs.
To get started, let’s navigate to the AWS Cloud9 console, where we’ll
create an Amazon Managed Service for Prometheus workspace.
The workspace is being created.
This is the workspace where we'll ingest our EKS cluster metrics.
Now let’s return to the AWS Cloud9 console.
Notice that we’re signed in to our central account, which is where we created the workspace.
Now let's navigate to the Amazon Managed Service for Prometheus
console to check the status of the workspace.
The workspace has been created and is active.
Now let’s switch to our workload account, where we’ll create the Kubecost deployments.
We’ll be deploying a federated setup,
so we’ll store our ETL data in an Amazon S3 bucket to improve query performance.
For our purposes, an S3 bucket has already been created.
Now let’s return to the Cloud9 terminal.
Our workload account contains two EKS clusters.
Let’s set up some environment variables in the primary cluster.
First we’ll define the Kubecost version.
We’ll use the most recent version available.
Next, we’ll define the AWS region.
Next, let's define a variable for our S3 bucket.
Next, let's specify the AWS Account ID.
Next, we’ll define a variable for the remote write URL of the workspace we created.
Finally, we’ll define the query URL of the workspace we created.
Now, let's log into our secondary cluster, which is running in the
same account, and create the same environment variables.
The remote write URL will be the same for each cluster because they will both
use the Amazon Managed Service for Prometheus workspace as a back end.
Now that we’ve set up the environment variables, let’s clear the terminal.
Let's return to the primary cluster and create a Kubernetes secret to allow
Kubecost to write ETL files to the S3 bucket.
First, we'll create a Kubecost namespace.
We’ll create the secret from the federated-store.yaml file.
Let’s view the contents of this file.
Notice that the file contains the information for our S3 bucket.
Next, let’s create the secret in the Kubecost namespace.
Next, let’s retrieve the information for the secret to make sure it was created successfully.
Everything looks good.
Next, we'll log into the second cluster and perform the same steps.
Now let’s log back in to our primary cluster.
Next, we need to grant cross-account permissions to our Kubecost service
accounts so they can successfully write the cost metrics into Amazon
Managed Service for Prometheus.
First, let’s look at the trust policy we’ll attach to the IAM role for the
kubecost-cost-analyzer-amp account.
The policy includes two principals:
one OpenID Connect (OIDC) provider for each EKS cluster running in the account.
Let’s create the IAM role and attach the policy to it.
Next, let’s look at the permissions we’ll assign to this role.
These permissions will enable users to do an Amazon Prometheus remote
write in the central account, and will grant Amazon S3 full access.
Next, we’ll attach this policy document to the IAM role we created earlier.
The IAM role has been created.
Now let’s navigate to the IAM console and ensure everything was set up correctly.
The S3 full access and cross-account permissions policies have been
attached to the EKS-Kubecost-cost- role, and we’ve enabled this role to
assume the EKS-AMP-Central-Role.
Here’s the trust relationship we established earlier.
Now let’s return to the terminal and create the kubecost-cost-analyzer-amp account.
We’ll create the service account in the first cluster within the Kubecost namespace.
Notice that the command specifies the role that we just created.
Let’s confirm that the service account has been created.
Next, let’s make sure the role was assigned successfully.
We can see that the EKS-Kubecost- cost-role has been attached to the
kubecost-cost-analyzer-amp account.
Now, let's log into the second EKS cluster and run the same command
to create a service account in it.
Let’s ensure the service account was created and the correct role was assigned.
Now, we need to create the kubecost-prometheus- server-amp service account in each cluster.
Before doing so, let’s log in to the first cluster to create the IAM role that will be used by this account.
Let’s view the trust policy that we’ll attach to the role.
The policy contains two principals: one OIDC provider for each cluster.
The policy will also enable the account to assume the role we’ll create.
Next, let’s create the IAM role for the second service account and attach the trust policy to it.
Next, we’ll attach the cross-account permissions policy to the role.
Now let’s return to the IAM console and quickly review the role we just created.
Here’s the trust policy we attached.
Here’s the permissions policy we attached, which gives the account
permission to access the Amazon Managed Service for Prometheus workspace.
Now that we’ve successfully created the EKS-Kubecost-prom-role,
let's create the kubecost- prometheus-server-amp account.
We’ll create the account in the first cluster.
The first cluster now contains the two service accounts we added.
Next, let's create the kubecost- prometheus-server-amp service
account in the second cluster.
We can see that the correct IAM role has been attached to this service account.
Now that we’ve created the service accounts, let’s return to our central
account to create one more IAM role.
Before we create the role, let's look at the trust policy we’ll attach.
The policy contains the two IAM roles we created earlier,
enabling both roles to assume the EKS-AMP-central-role we’re creating.
Now, we’ll create the role with Amazon Prometheus full access and attach the permissions policy to it.
Now let’s return to the IAM console and review the role we just created.
The role contains the permissions policy and trust policy that we attached.
We’ve now configured a two-way trust relationship between the IAM roles.
Now, let’s return to the terminal and create the Kubecost pods.
First, let’s take a look at the config values that will be used by the Kubecost helm chart.
Notice that the Kubecost helm chart contains the Prometheus endpoints and the sigv4 is
accurately pointing to the region where the Prometheus workspace was provisioned.
Additionally, the role that we created in the central account is being parsed with the sigV4 configuration.
Next, let's make sure that we are in the primary cluster.
Next, let's run the helm command to deploy the Kubecost pods.
Since this is the primary cluster, the command specifies the primary-federator.yaml file.
Next, let’s check the status of the Kubecost pods.
The Kubecost pods are all running.
The kubecost-prometheus-server-amp service account, which has the Amazon
Managed Service for Prometheus workspace, will be used by the kubecost-prometheus-server pod.
The kubecost-cost-analyzer-amp service account will be used by the kubecost-cost-analyzer pod.
Both service accounts have the necessary permissions to analyze the workspace.
Let’s also take a look at the logs to ensure that there are no errors.
Everything looks good.
Now let's log into the second cluster to install Kubecost and integrate it with
the Amazon Managed Service for Prometheus workspace.
We’ll run the same command as we did for the first cluster,
but this time we’ll use the agent-federated.yaml file.
Next, let’s check the status of the Kubecost pods.
The Kubecost pods are running successfully.
The federator pod only runs on the primary cluster, so it doesn’t appear here.
Now that we’ve successfully deployed Kubecost, let’s navigate to our S3 bucket
where a federated folder has been created.
There are three subfolders: one for the primary cluster, one for the secondary cluster,
and one that’s combined so we can monitor both EKS clusters at the same time.
Now that everything is set up, let’s return to the terminal to set up the Kubecost UI.
First, we’ll enable the service as a load balancer.
Once we save our changes, the load balancer will be created.
The load balancer is now being created.
Let’s navigate to the Amazon EC2 console to monitor its status.
Let’s look at the instances.
The status of each instance currently shows as "OutOfService."
Once the health check is complete, they will update to "InService.”
Now that both instances are in service, let’s access the Kubecost UI.
The Overview page shows information about our two clusters, including the
total cost incurred, possible monthly savings, and cluster efficiency.
Next, we’ll explore the pages under the Monitor section.
On the Allocations page, we can aggregate the data by multiple dimensions.
For example, we can view cost by namespace and cluster.
The Cloud Costs page provides information about other AWS services and their respective cost.
It will take some time for data to begin appearing here.
This page shows all of our Kubecost monitored clusters, as well as any
clusters captured in our cloud bill.
Next, let’s look at the Savings page.
This page shows how much we can save by implementing recommendations,
such as ensuring that we have right- size cluster nodes and containers.
You’ve just seen how to set up multi- cluster cost analysis for Amazon EKS
using Amazon Managed Service for Prometheus and Kubecost.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.