In this video, you'll see how to use AWS Distro for OpenTelemetry (ADOT)
to send cross-account metrics on Amazon Elastic Container Service(Amazon ECS).
With ADOT, you can collect and correlate datametrics, configure fined-grained permissions for cross-account access with Amazon ECS,
and leverage centralized metrics collection across multiple AWS services.
For this demonstration, we'll work in theAWS Cloud9 integrated development environment (IDE), but you can use your preferred IDE.
We have two accounts: a central account for monitoring, and a workload account for hosting our ECS cluster.
To start, we'll create a new file in the workload account and load it with a policy that grants assume-role permissions to our ECS task.
Now in the bash terminal window, we'll create an AWS Identity and Access Management (IAM) role from the policy.
It grants cross-account permissions to the Amazon ECS tasks we'll build later.
Let's visit our IAM console to view the role.
Now we'll go to the central account to create an Amazon Managed Service for Prometheus (AMP) workspace.
AMP is the monitoring solution we’ll use in this example.
The workspace is where we'll aggregate metricsdata.
Let's visit the AMP console to confirm the workspace has been created.
Next, we'll create a set of permissions on the central monitoring account
that allow cross-account access to AMP with the minimum amount of restrictions.
In a new file, we’ll add a policy to allow assume-role access.
Now we can create an IAM role with write permissions to the AMP workspace.
To do this, we’ll need to find and supply the workload account ID.
Now we'll create a policy that gives the workload account permissions to perform assume-role actions in the central monitoring account.
Here's the file with the ID from the monitoring account.
Next, we'll create an IAM role for the assume-role permission and attach a policy.
The central monitoring account needs only remote-write access to AMP, so we've accounted for that in the policy.
The IAM role assigns permissions for the workload account to push data into AMP.
Let's check that the policy appears in our IAM console.
Here you can see the role we just created.
As you can see, the policy gives permissions to push data into Prometheus.
Now we'll set up a role for the central monitoring account to access the workload account.
This cross-account access ensures both accounts can connect with the AMP workspace.
Let's get the central monitoring account ID, and then go to the workload account page.
Here, we’ll paste the account ID, and then add the policy that grants the central monitoring account assume-role access.
The policy file contains code that enables the assume-role action we established in the central monitoring account.
Back in the bash terminal, we’ll attach the policy to the role we created for remote-write access to AMP.
Let's check the IAM console again to see the ECS cross-account task role.
With the cross-account permissions complete,
we can now configure the ADOT collector and Amazon ECS infrastructure in the workload account.
First, we'll establish a directory for the ADOT collector.
The directory appears as a new folder in the workload.
Next, we’ll attach the cross-account role policy.
We'll create another directory for the main ADOT application.
Next, we'll install the Amazon ECS command line interface (CLI) tool to deploy applications using Docker Compose for Amazon ECS.
Now we can access the demo-app repository
and create a simple application that exposes a Prometheus metrics endpoint using code available on GitHub.
Next, let's build a Docker file to allow our application to run inside Docker.
To recap our infrastructure so far, we have the demo-app application set up to capture Prometheus metrics.
We also have a Docker file that will allow the application to run inside a Docker container.
Next, we'll build Docker images and host them on Amazon Elastic Container Registry (Amazon ECR) to use as part of an Amazon ECS task.
With this code, we'll build the Docker image locally.
Let's set a default region.
Now we can create the Amazon ECR repository to host the Docker image.
Let's check it out in the Amazon ECR console.
Back in the workload account, let's log in to the Amazon ECR repository.
It's empty, so let's push the Docker image into the repository.
Once the process is completed, let's check the latest version of the image in the Amazon ECR console.
The next step is to configure the ADOT collector using a custom Pipeline that collects, modifies, and exports its data.
We're building the pipeline in the workload account, so first we'll capture the workspace ID and central monitoring account ID.
Now let's move into the aws-otel-collector directory.
We'll create a basic YAML configuration file.
The configuration code defines how the data will flow: receivers connect to the AMP metrics,
processors initiate the ADOT collector to capture environmental metrics,
and exporters send the data to the endpoint, our AMP workspace.
Let’s edit the code to change the region.
Next, we'll build a Docker file within the ADOT collector to coordinate with this configuration.
We'll supply code that captures the latest files from the collector.
We also need an Amazon ECR repository for the collector.
Now we'll build a Docker image that pushes our data into the repository.
Now we'll define an assume-role policy and attach permissions so the collector-to-Amazon ECR task can run.
Now we can create our Amazon ECS cluster.
First, we'll use Docker Compose to create a link between the ADOT collector and application containers.
Let's supply the workload account ID and create the Docker Compose file.
Our code links both the ADOT collector and the AMP application to the Amazon ECR repository hosted by the workload account.
Let’s view our Amazon ECR infrastructure so far.
We have two repositories: the custom ADOT collector and main application.
Next, we'll create an Amazon ECS cluster hosted on an Amazon Virtual Private Cloud (Amazon VPC) in the workload account.
We'll use the Amazon ECS CLI to natively integrate with our Docker files and launch it with AWS Fargate,
a completely managed container environment.
Let's name the configuration and specify the region.
Using the "ecs-cli up" command, let's create the cluster.
Let's view the cluster in the Amazon ECS console.
Now we'll apply two security groups to the VPC.
Next, let's compose a new YAML file to establish the network configuration and task size for when we run the Docker images on Amazon ECS.
Now let's add the two subnets built when we created the VPC.
We’ll also add the security groups.
Now that the cluster and the configuration are ready, we’ll deploy the services.
Let's compose the ECS CLI configuration.
We'll also set up an Amazon CloudWatch Log group.
We can check on the progress in our Amazon ECS console.
Amazon ECS is currently running the task definition.
We can also see that the task has the two container definitions we established:
the ADOT collector and the AMP application.
At this point, the workload account is actively collecting and sending metrics to AMP.
We can monitor this activity in our CloudWatch Logs.
From these active log events, we can tell that the ADOT collector is sending metrics across the workload account
into the AMP workspace on the central monitoring account.
We can verify our configuration and visualize the metrics using Amazon Managed Grafana.
To do that, we'll first need to create a new workspace.
Let's go to the Amazon Managed Grafana console and create one.
We'll use AWS Single-Sign On (AWS SSO) for authentication and create a user.
Let's supply our credentials.
We'll retain the service managed permission type and move on.
Our data source will be Amazon Managed Service for Prometheus since that's where our metrics are stored.
Let's complete the workspace setup.
While the workspace is being created, let's go to AWS SSO, where we can manage users.
This user will be allowed to use SSO to access our Grafana workspace.
Let’s return to Amazon Managed Grafana.
Our workspace is now active.
Let’s make our user account an admin so they can add resources and create dashboards.
Now let's enter the workspace through its URL.
As a final step we'll configure AMP as a data source for Amazon Managed Grafana,
completing the metrics collection architecture.
We'll select the same region that hosts all our other services.
Now let's go to the Explore page and see if our metrics in the central account are flowing from the workload account.
From the options, we'll choose the AMP workspace.
Here we can see that several metrics labels are already available.
Let's find and use an “up” metric for our first query.
Our query results show that the application has been running since 5:34 p.m.
Now let’s run a rate query on the application itself.
Because it produced results and encountered no errors, we can see that our cross-account metrics collection architecture is working properly.
We can even query Amazon ECS metrics.
For example, let's measure the CPU utilized by the ECS tasks.
We can repeat this cross-account configuration for multiple workload accounts
and send that data into the same central monitoring account.
You've just seen how you can use ADOT to send cross-account metrics on Amazon ECS.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it's your turn to try.