YURI GRINSHTEYN: We covered using Service-Level Indicators
to measure the reliability of our services
and using Service Level Objectives to set
reliability targets in our first episode.
But what happens when that service has a problem
and we start burning error budget?
How can we configure alerting to let
us know that our users are unhappy while managing
our operational load?
Today, I'm going to show you how to calculate and use
error budget burn rate thresholds
as an optimum approach to your alerting set up.
Welcome to "Engineering for Reliability" with Google Cloud.
To help us, let's review chapter 5 of the "SRE Workbook"
called "Alerting on SLOs."
If you haven't read it before, no worries.
We'll link to it in the episode notes below.
The main thing we need to take away from that chapter
is that the best way to have sufficiently sensitive alerts,
while at the same time reducing false positives,
is to use multi-window, multi burn-rate alerts.
But first, what's a burn rate?
Error budget burn rate is a measure
of how fast, relative to your SLO,
your service consumes error budget.
For example, a service that has exactly zero error budget
left at the end of the SLO evaluation period
has a burn rate of 1.
A service that consumes all of its error budget
halfway through the SLO evaluation period
has a burn rate of 2 and so on.
We calculate our burn rate by looking at three main inputs--
the SLO and its evaluation period,
the amount of error budget consumed or burned,
and the window over which that burn happens.
That's our alerting window.
Let's walk through a specific example.
Let's say our service has an availability
SLO of 95% over a rolling 28-day window.
We want to know if we're having a problem that's
impacting end users.
A great place to start is to create
an alert that lets us know that we've consumed 2% of our error
budget in the last hour.
We use this equation to calculate the burn rate
threshold for our alert, which is 13.44.
We also want to know if we're seeing
a slower burn rate, too, 5% of error budget in six hours.
This time, our burn rate threshold is 5.6.
We want our alert to notify us only
when we're still actively burning error budget to reduce
the number of false positives.
To do this, we'll create both of our alerts with two windows--
a long window and a short window.
A good starting place is to make the long window 12 times
as long as the short window.
By using multi window alerts, we keep our detection time low
and minimize our reset time so that the alert
doesn't keep firing once the incident has been mitigated.
OK, that was a lot.
The main thing to remember at this point
is that we can use error budget burn rate alerts to detect
issues with our service long before our SLO is actually
threatened.
Now, let's have a look at how to set them up
in Cloud Operations.
Start at the Dashboard page.
Select Monitoring from the Products menu,
and click on Services.
Here's a service with an availability SLO defined
but no alerts configured.
The SLO is evaluated over a rolling 28-day window.
In order to create an error budget burn alert for the SLO,
you need to specify two things.
The first one is the look-back duration
or the window over which the error
budget burn rate is evaluated.
The second entry is the burn-rate threshold
or the value of the burn rate that
must be true over the look-back back window for the alert
to fire.
To configure a fast error budget burn rate alert
that will fire if 2% of error budget
is burned over an hour, set the look-back duration to 60
minutes and the burn rate threshold to 13.44.
For a slow burn alert that will fire if 5% of error budget
is burned over six hours, set the look-back duration
to 360 minutes and the burn rate to 6.
Name your loading policy, specify your notification
settings for the alert, provide any documentation
or additional information that should
be delivered with the alert, and save the alert policy.
Finally, confirm that the alert has been created.
Now, if the availability of the service degrades,
the alert will fire, and an incident will be created.
We've configured an alert based on error budget
burn rate threshold, but what happens when it fires?
Let's have a look.
Let's see what happens when an error budget burn
rate threshold alert fires.
From the Services Overview page, we
can see that we have a service with an SOL alert firing.
We can show just those services with firing alerts.
We can go to Service Details to confirm
that we have an unhealthy SLO with a firing alert.
We can expand the SLO to confirm that we're
seeing a degradation in our service level indicator.
On the Error Budget tab, we can confirm
that we're starting to see additional error budget burn.
Further down, we can see the logs being generated
by the logs for our service.
In this case, we're seeing an increased number of 500 server
error logs confirming that our service is currently
having an issue.
Thanks for joining me today.
We covered a lot.
You learned about why alerting on error budget
burn is a great way to balance keeping
users happy against operational overload.
We then figured out how to calculate error budget burn
rates and so how to use them to set up
alerting for your services in Cloud Operations.
Thanks so much for watching.
Don't forget to like and subscribe to never miss out
on more "Engineering for Reliability" with Google Cloud.
See you soon.
[MUSIC PLAYING]