YURI GRINSHTEYN: Last time, we covered
how you can use Google Cloud platform metrics as Service
Level Indicators to measure the reliability of your services
and set reliability targets using SLOs.
But what do you do if you need custom telemetry for your SLIs?
Today, I'm going to show you how to use Prometheus
and OpenCensus to write custom metrics
and define SLIs and SLOs using them.
Welcome to Engineering for Reliability with Google Cloud.
[MUSIC PLAYING]
Imagine a basic web application architecture like this.
While we can use the metrics emitted by the load balancer
to track things like request counts and response latency,
what if our application running on GKE
needs to keep track of specific transaction details
and write a custom counter for them?
What if we want to use this kind of custom telemetry
as our SLI for the service?
This is where custom metrics come in.
Today, I'm going to show you how to use
two open-source approaches to custom metric telemetry.
The first one is Prometheus.
Prometheus is an open-source systems
monitoring alerting tool kit, originally built
with SoundCloud.
It has a very active developer and user community
and joined the Cloud Native Computing Foundation back
in 2016.
This is the architecture Prometheus
uses to collect metrics.
The general idea is that your service
needs to expose a dedicated metrics endpoint, which
Prometheus will then scrape to collect metric data.
That data is stored in the time series database
and can be queried through a web UI or an API.
We'll link to the full code sample in the show notes,
but let's have a look at how to instrument
an application written in NodeJS to emit counter and histogram
metrics.
Start by defining the metrics themselves.
For each, specify the metric type, its name,
and a help message that will describe
what the metric is capturing.
To track latency, use a histogram,
and call its observe method.
To increment counters, call their inc method.
Finally, create a dedicated endpoint in your application
that Prometheus will use to scrape for metrics.
The instrumentation is done.
Now, if you go to the /metrics endpoint in your app,
you'll see a page that looks like this.
The metrics and their values are written out here.
Now you need to configure Prometheus
itself to scrape this endpoint.
You do that by adding some annotations
in the deployments spec in your application,
enable scraping, and define the port
and endpoint where the metrics are available.
Next, you need to ingest these metrics into Cloud Monitoring.
We'll link the detailed instructions in the show notes,
but the basic idea is that you pass your Prometheus server
with a collector that will send these metrics to Cloud
monitoring.
Once your metrics are available there,
you're ready to use them for SLIs and SLOs.
Here's an example of how you can use a latency metric, written
by Prometheus, to create a distribution-cut SLI
and set an SLO target for it.
A distribution cut allows you to capture the percentage
of values that fall within a specified
threshold, as defined by range.
This SLO expects that 98% of responses
complete within 500 milliseconds over a rolling 28-day period.
Prometheus is a popular open-source monitoring
framework, and it's well suited for monitoring applications
running in GKE.
But what if you don't want to maintain
that additional infrastructure?
In that case, you can use OpenCensus
to write metrics directly using the Cloud Monitoring exporter.
OpenCensus is a single distribution
of libraries to collect metrics and distributed traces
from your services.
Let's have a look at how to instrument your service,
configure the Cloud Monitoring exporter,
and use OpenCensus metrics to define SLIs and SLOs.
Again, we'll link to the full code samples in the show notes,
but let's take a quick look at instrumenting
an application written in NodeJS to emit counter and histogram
metrics.
Start by configuring the Cloud Monitoring exporter.
This will send the metrics you captured directly
to Cloud Monitoring, where you will then use them
to create SLIs and define SLOs.
Next, define a measure for each metric.
Measures describe how the data points
themselves will be recorded.
Create a view for each measure.
Views are basically the [INAUDIBLE]
you'll be adding data points to.
They also include information about how
that data will be aggregated.
For our latency histogram, you'll
need to specify your bucket boundaries.
It's important to at least make sure
that you have buckets that capture
requests that are going to fall within and outside of your SLO.
However, you'll likely want more than just
those to allow you to, for example, measure PINI latency.
Finally, record values using the record method.
Unlike Prometheus, this is the same
for counters and histograms in OpenCensus.
You won't need to configure any scraping.
The exporter will write the values directly
to Cloud Monitoring.
You can then use them to define SLOs like this.
Here's an example of a distribution cut
SLI using the response latency metric, written by OpenCensus,
with an SLO of 98% of responses being completed
in under 1,000 milliseconds.
Thanks for joining me today.
In this episode, we looked at how to instrument your code
to write custom metrics with OpenCensus and Prometheus,
use them to define Service Level Indicators,
and set Service Level Objectives against those.
You're now ready to start defining SLIs
by using these open-source libraries in your own services.
In our next episode, we'll look at alerting on our SLOs
with error budget burn rates.
Don't forget to like and subscribe to never miss out
on more Engineering for Reliability with Google Cloud.
See you soon.
[MUSIC PLAYING]