YURI GRINSHTEYN: When you run applications in production,
you need to monitor the infrastructure they run on.
You also need to measure the golden signals of application
health like error rates and latency.
Today I'm going to show you how to instrument your service
to expose application-specific telemetry with Prometheus
and how to configure Google's Manage Prometheus service
to ingest those metrics.
I'm Yuri, an SRE at Google.
Welcome back to "Engineering for Reliability with Google Cloud."
Last time, we started using Google's managed service
for Prometheus by replacing the Prometheus image
with Google's forked image.
That enabled ingesting Prometheus metrics
into the managed back end and querying them using PromQL.
Those metrics were then also available in Cloud Monitoring.
And you can start using them in charts, dashboards, and alerts.
That was enough to start collecting infrastructure
metrics from our cluster.
Today, we're going to go two steps further.
I'm going to show you how to instrument your service
to expose telemetry and how to configure Prometheus
to ingest those metrics.
We'll do that by setting up Managed Prometheus Collection,
a zero-management way to run Prometheus and the way
to go if you're new to Prometheus
or setting up a new cluster.
All right, let's get into it.
The first step is to instrument our service itself
to expose a dedicated endpoint that Prometheus
can use to scrape for metrics.
We covered this in our previous episode on custom metrics.
But let's have a quick look at an example in Python.
We'll of course link to the full sample in the episode notes.
This is an example of Python Flask app that
just has a basic home page.
Let's walk through the code together.
First, import the necessary modules.
Then define the metrics you want to capture.
In this example, we're going to track
total requests, failed requests, and a latency distribution.
We're going to use a Decorator to time how long it
takes to load a specific route.
And we can increment the counters as needed.
Finally, we specify the endpoint where Prometheus metrics
should be exposed.
Now we can see the data on this /metrics endpoint.
We're now ready to move on to the second step
and configure Prometheus to collect and ingest this data.
Let's start by verifying that our application is ready to be
monitored by Prometheus.
We can use port forwarding to confirm that it's exposing
Prometheus metrics on the /metrics endpoint.
In order to get our application monitored by Managed Prometheus
collection, we'll need to do two things.
First, we'll use this gcloud command
to install Managed Prometheus in our cluster.
Note that you can do this using kubectl if your cluster is not
running in GKE.
At this point, we can confirm that infrastructure collection
is working by querying metrics that the Managed Prometheus
collector is ingesting about itself, for example.
Now, we're ready to get our application monitored
by creating a new pod monitoring resource.
The selector specifies which pods to monitor
based on label value.
And the endpoints specify where to scrape for metrics.
As we saw before, our application is exposing
Prometheus metrics on the /metrics endpoint on port 8080,
which is called metrics in a deployment configuration.
Once that's applied, let's run a query
to confirm that application metrics are being ingested.
And there we are.
We can also query this metrics in Metrics Explorer
simply by searching for it like this.
And that's how you can instrument your application
for Prometheus Monitoring and have its metrics be collected
and ingested by the Google managed service for Prometheus.
Thanks so much for watching today.
We learned how to instrument your application for Prometheus
monitoring and how to configure Google's managed
service for Prometheus to collect those metrics.
Next time, we'll have a look at how
to migrate an existing Prometheus
deployment to the managed service for Prometheus.
And if you have any other topics you'd
like me to cover, please comment below.
Don't forget to like and subscribe to never miss out
on more "Engineering for Reliability with Google Cloud."
See you soon.