SPEAKER: If you run your services on GKE or anywhere,
really, you need to have a way to get some basic signals
about service health.
At a minimum, you want to collect the golden signals
for your service.
Traffic is a measure of how much demand
the service is responding to, usually measured in requests
or queries per second.
Saturation is a measure of how much of the available capacity
the service is using.
Errors, or error rate, should tell you
the fraction of requests you're able to serve correctly
or incorrectly.
And latency is a measure of how quickly you
are able to serve your users.
You can read more about golden signals in the "Monitoring
Distributed Systems" chapter of the SRE book.
We'll link to that below.
Of course, you need a way to counter this information
for your services.
My name is [? Uri. ?] I'm an SRE at Google Cloud,
and today, I'm going to show you how to instrument your Python
services to emit metrics like these with OpenTelemetry,
an open source observability library.
Welcome back to Engineering for Liability.
We'll start with walking through the code
to instrument your application.
Then we'll have a look at how to query and use
those metrics in Cloud Monitoring.
Our example service is written in Python
using the Flask framework.
It receives web requests and does some processing
before responding.
So we'd like to track the request rate, error rate,
and the distribution of the processing latency.
As always, we'll link to the full code sample,
but here's how you can add metric instrumentation to it.
First, we set up the exporter to send metric data
to the OpenTelemetry collector.
It's a dedicated component we'll use
to collect telemetry and forward it onto Cloud Monitoring,
and we'll cover it in more detail in our next episode.
Next, we define the meter provider, including information
about our service we'll be able to use to query our metrics.
Next, we define our actual metrics.
We'll use incrementing counters to track
requests and errors and a histogram to track latency.
Finally, we use these metrics to record our data.
We increment the counters on request in the error and record
latency before returning.
And now, we have metrics to represent the major signals
of application health--
request an error account and a latency distribution.
Now that our application is instrumented,
let's deploy it and have a look at the metrics.
Our application is written in Python
and instrumented with OpenTelemetry.
We use this Docker file to build a container image.
We use this YAML file to deploy our image to a GKE cluster.
And here are the three replicas.
Next, we create this YAML file and use it
to deploy the OpenTelemetry collector.
And here it is running in the same name
space as our application.
Finally, we have some pods generating traffic
against our service.
Let's go to Metrics Explorer and first look
for our total request metric.
We'll group it by service ID.
This shows the request rate by [INAUDIBLE]..
We can do the same for the failed request metric.
We can even have Cloud Monitoring calculate
a ratio of the two for us to easily measure
error rate for our application.
Finally, let's also graph request latency.
Because this is a histogram metric, we can use a heat map.
This will automatically calculate the average and tail
latencies for our service.
So that's how you can deploy a service instrumented
with OpenTelemetry to GKE and view its health
metrics in Cloud Monitoring.
Thanks so much for watching today.
We learned how to use OpenTelemetry to instrument
our service to admit basic signals of application health,
like traffic and latency metrics,
and how to query them in Cloud Monitoring.
I hope this helps you to keep your applications reliable
and your users happy.
Don't forget to like and subscribe to never miss out
on more Engineering for Reliability with Google Cloud.
See you soon.