SPEAKER: Last time, we covered how
you can use service level indicators to measure
the reliability of your services and how
to use service level objectives to set reliability targets.
But what metrics should you use to accurately measure
your reliability?
Today I'm going to show you how to use out-of-the-box platform
metrics in Google Cloud to do just that.
Welcome to Engineering for Reliability with Google Cloud.
[MUSICAL INTRO]
Let's start by remembering that you
want to be able to express your service level
indicators, or SLIs, as ratios or percentages.
This allows you to normalize various measures of reliability
and set reliability targets in a uniform way
across your services.
Generally, SLIs fall into two categories.
Request-based SLIs measure the fraction
of good units of work against the total.
For example, what percentage of requests to an HTTP endpoint
were successful in a given time window?
Window-based SLIs measure the fraction of time intervals
during which the service meets a threshold of reliability
over a given time window.
Now that you know what you're after,
which metric should you use?
Do you just start writing custom telemetry to create them?
Now there may be scenarios where that makes sense.
And we'll cover those later in the series.
But let's start with metrics that Google Cloud Services
create for you right out of the box.
Let's start with a specific example.
Here's a common architecture for a web application
that uses a global load balancer to distribute traffic
to multiple GKE clusters.
For this example, you can use these metrics
created by the global load balancer for both availability
and latency SLIs.
Let's have a look at how you can use
service monitoring in the Cloud operations suite
to define a service and set an availability SLO for it.
From the project dashboard, let's go to the monitoring menu
and select services.
There are a number of services already created
in this project.
Let's create a new one.
We'll use a GKE deployment as the basis for our service
definition.
Let's name our service and submit.
Now that the service has been created,
let's define an availability SLO for it.
The first step is to set the SLI, or service level
indicator.
Services defined by App Engine, STO, or Anthos
recognize availability and latency SLIs automatically.
But we've defined a custom service.
So we have to select other as the metric.
We then need to decide whether our SLI should be
request-based or windows-based.
We'll link to another video that describes
the difference between two in the episode notes.
But for now, we'll simply calculate a proportion of good
to total events using a request-based SLI.
The next step is to select the metric
that we want to use as our SLI.
Because the service is being provided by an external load
balancer, we can use the request count metric.
We'll count all requests in the total filter
and just those that return a 200 in the good filter.
Note that this approach does include
potentially erroneous requests, like those that return 404s,
in the total filter.
And you should carefully consider whether that's
the behavior you want.
Once we've made our selections here,
we can evaluate the resulting graph
to determine whether the SLI will be properly evaluated.
Next, we'll select the compliance period for the SLO.
We want the evaluation period to be
long enough that we can meaningfully
evaluate and allocate error budget toward engineering work,
but not so long that the error budget takes
too long to recover from an incident.
We also recommend using rolling windows
for operational concerns rather than
calendar to avoid issues with uneven numbers of weekends,
for example.
If you're just getting started, a 28-day rolling window
is great.
Note that this is not the alerting window.
We'll cover alerting in SLOs in a future episode.
Now let's set a performance goal for our SLO,
evaluate the SLI against the goal over the last hour,
and click Continue to finalize our setup.
Let's name our SLO in a way that will
allow us to easily determine what it's measuring
and create it.
Immediately, we can see that we're actually
in the negative for error budget for this particular service.
This is because we're using a rolling period,
so the service is evaluated over the last 28 days.
This is a new service.
And we simply haven't had enough good traffic
to the service to recover all of the error budget.
This will change as our service serves more and more traffic.
So that's how you can use service monitoring in Cloud
operations to define a custom service and set an availability
SLO for it using an out-of-the-box platform metric.
If you have a request response service that uses Cloud load
balancers, using metrics like request count
is a great way to easily define an availability SLI
and set a target using an SLO.
But what about a data processing pipeline?
In this example, you want to monitor
how long it takes for data flow to process
data elements in the pipeline.
This is sometimes called a data freshness SLI.
Data flow captures this information as two metrics.
System lag measures how long it takes the entire job
to process messages.
Per stage system lag measures how long
it takes for a given step in the pipeline to process messages.
Because our service isn't running in a GKE,
we'll have to create a custom service
definition using the API.
Make a request to the services endpoint in your project,
providing the service ID and display name.
Once the service is defined, you can create an SLO for it.
You can use the system lag metric
that measures how long it takes the job to process messages
as an indicator of freshness.
Let's have a look at how to do that in service monitoring.
This time, let's start directly on the services overview
screen.
Our data pipeline service doesn't have any SLOs.
Let's create one now.
Because this is a custom service which does not
support default availability and latency metrics,
we'll use other as the metric type.
Next, we'll select a windows-based evaluation type.
The system lag metric measures how long
it takes data elements to make it through our pipeline.
This metric isn't tracked per element.
So we don't have a way to count a ratio of good events
to total.
So we'll need to use a windows-based SLI, where we'll
track its average over smaller windows
and count the ratio of those windows that meet our threshold
over our evaluation period.
Let's select the metric and specify our window criteria.
Based on the historical data shown here,
we can expect the average system lag to be under two seconds.
A five-minute window is a great place to start.
Note that shorter windows will be more responsive to changes,
but could also result in a noisier signal.
Now that we've configured our SLI, let's define our SLO.
We'll again use a 28-day rolling period and set a target of 99%.
This means that we expect 99% of our five-minute windows
over a 28-day period to have an average system
lag of under two seconds.
Let's give our SLO a good name and see
how our service is performing.
We can now see how our SLI has been
performing against our target and how much
error budget remains.
And that's how you can set up a data freshness SLI
for a streaming data pipeline.
You've now seen how to use platform metrics
to define SLIs and SLOs for services
that use load balancers, and for data pipelines.
But what about cases where none out-of-the-box metrics work
for you?
One option that might is to use logs-based metrics.
Let's have a look at how to do that
using Cloud logging, Cloud monitoring, and service
monitoring.
Here's a service that emits a log entry for every request
that includes a status code.
Successful responses include a 200 status code,
server errors include a 500 error status, and so on.
From these log entries, we can create
a logs-based metric that will count the number of requests.
The status label will allow us to track successful requests
separately from failed ones.
We can view the metric in Metrics Explorer
to confirm that it's working as we expect.
We're now ready to configure the SLI for our service
from this metric and set an SLO for it.
Let's go to our services overview, select our service,
and create an SLO for it.
Because we're essentially using a custom metric,
let's select other as the metric type.
We'll create a request-based SLI,
because well want it to calculate
a ratio of successful requests to all requests.
Let's select our metric to be used as the SLI.
We'll count the total number of requests
by using the unfiltered metric.
We'll count the number of good requests
by only counting those that return a 200.
Let's have our SLO evaluated over a rolling 28-day period
with a target of 95%.
Now we can see how our SLI is performing
against our availability target and how much
error budget we have remaining in this evaluation period.
And that's how you create an SLI and SLO using
logs-based metrics.
Thanks for joining me today.
In this episode, we looked at how
to use platform metrics to define service level indicators
and set service level objectives against those.
We covered using load balancer metrics
for request-response services, data flow metrics for data
processing pipelines, and logs-based metrics for services
that may not emit good SLIs out of the box.
You're now ready to start defining
SLIs for your own services.
Don't forget to like and subscribe to never miss out
on more Engineering for Reliability with Google Cloud.
See you soon.
[MUSICAL OUTRO]