YURI GRINSHTEYN: We use SLOs in error budgets
to set appropriate reliability targets for our services.
But what happens when there's an issue?
When you set up a learning to let
us know that users are unhappy?
Today I want to show you how to use error budget burner
alerts in Service Monitoring.
This is the Stack Doctor.
[MUSIC PLAYING]
We've covered using threshold-based alerting
before, but learning on SLOs is a bit different.
The main thing to keep in mind is
that we want our alert to fire long before our services out
of SLO, essentially as soon as we start burning error budget.
To start, let's cover some math.
To help us, let's review chapter five of the SRE workbook,
called Alerting on SLOs.
If you haven't read it before, no worries.
We'll link to it in the episode notes below.
As covered in that chapter, your learning strategy
should include four considerations.
They are precision-- what fraction of your alerts
are actually triggered by a real or significant event?
Recall-- what fraction of your events results in an alert?
Detection time-- how long does it take for an alert
to fire once an event starts?
And reset time-- how long after the event is over
does it take for your alert to reset?
There's no right values for these considerations.
It all depends on what you want out of alerts.
However, you'll probably want to find the right balance for you
while minimizing your operational load--
that is, the number of alerts you actually
receive and have to respond to.
The best way to do this is by alerting on error budget
burn rate, which is where the math comes in.
Error budget burn rate is a measure
of how fast, relative to your SLO,
your service consumes error budget.
For example, a service that has exactly zero budget
left at the end of the SLO evaluation period
has a burn rate of one.
A service that consumes all of its error budget
halfway through the evaluation period
has a burn rate of two, and so on.
We calculate our burn rate by looking at three main inputs--
the SLO and its evaluation period,
the amount of error budget consumed or burned,
and the window over which that burn happens.
That's the alerting window.
This, then, is the equation we'll
use to calculate burn rate that we'll use when we configure
our error budget burn alerts.
Our burn rate is the amount of error budget consumed
by the service multiplied by the SLO evaluation period
and divided by the alerting window.
OK.
That was a lot.
If you're a bit lost at this point, that's totally OK.
All you really need to understand
is that we can calculate the error budget burn rate
if we know what our SLO is and how much of our error budget
we want to consume and over what time before firing our alert.
The next question you might have is, well,
how much error budget should my service
burn before firing an alert?
Here, we again return to the alerting on SLOs
chapter of the SRE workbook for two simple recommendations.
They are 2% of error budget in one hour
and 5% of error budget in six hours.
This way, you can both catch higher error rates quickly
and lower error rates over time without overwhelming
the on-callers with alerts.
Let's walk through a specific example.
Let's say our service has an availability
SLO of 95% over a rolling 28-day window,
and we want to create an alert that
lets us know if we've consumed 2% of our error budget
in the last hour.
We use our simple equation to calculate the burn rate
we need to hit, which is 13.44.
Let's use the same equation to calculate the burn rate
threshold for our other alert, 5% of our budget in six hours.
This time, our rate is 5.6.
OK.
So far, we've covered our budget burn rates
and how we calculate them.
We're now ready to set up our alerts.
Here's our service with our SLO configured against it.
We're going to create an alert that
lets us know when the service consumes 2% of its error budget
in an hour.
This is where our budget burn rate becomes important.
To configure the alert, we need to specify the lookback window.
In our case, it's 60 minutes.
We also need to specify the burn rate
we expect to hit during the window as our threshold.
Remember, we already calculated that to be 13.44.
Now, when our service has an issue,
we'll see the error budget decrease,
and we'll get an alert that looks like this, showing us
the error budget burn during the incident.
Thanks for joining me today.
We covered a lot.
We learned about the considerations that
should go into your learning strategies,
and why alerting on error budget burn
is a great way to balance them against operational overload.
We then figured out how to calculate their budget
burn rates, and saw how to use them
to set up alerting for your services in Service Monitoring.
Thanks so much for watching, come back soon for more.
This is the Stack Doctor.
Stay healthy out there.
[MUSIC PLAYING]