[MUSIC PLAYING]
LIZ FONG-JONES: Welcome to episode seven
of our series on SRE and DevOps.
I'm Liz Fong-Jones, a site reliability engineer at Google,
where I teach Google Cloud customers how to build
and operate reliable services.
SETH VARGO: [YAWNS]
LIZ FONG-JONES: Where have you been, Seth?
SETH VARGO: I'm Seth, I'm a developer advocate.
[YAWNS] I'm sorry I'm late.
I was up all night dealing with these alerts
from these systems.
It was just a really hectic night.
LIZ FONG-JONES: Ugh, that's the worst when that happens.
And what experience were your users having?
Were they happy, or were they impacted by the alerts?
SETH VARGO: [YAWNS] Honestly, that
was the most frustrating part.
I get paged about this single server that
had slow or low disk space or low network bandwidth,
but only a small percentage of queries were affected.
And by the time I got there, the load balancer had already
routed traffic away, so there was nothing for me to do.
LIZ FONG-JONES: Sounds like you have too many alerts,
and they're not doing you any good at actually catching
problems affecting real users.
And that leaves you really, really tired
when something does actually break.
So let's see how we can maybe reduce some of that noise.
SETH VARGO: So my partner and I would greatly
appreciate getting to sleep through the night
unless things are actually on fire.
But I'm worried, too, because I've heard rumors
that the rest of my team is actually
ignoring alerts and putting their phones in Do Not Disturb
mode because none of the alerts are actionable,
or very few of them are actionable.
And I'm worried that that's going
to put our business at risk in the event of a real outage.
LIZ FONG-JONES: Yeah, so you have service level indicators
and service level objectives.
Remember last time?
SETH VARGO: Yeah, we do.
We have an automated system that verifies that you can log in,
you can add things to your shopping cart,
and you can check out.
And our service level indicator is
defined based on whether that probe, that process,
succeeds without error and whether it completes quickly
enough in our time frame.
And then our service level objective, or SLO,
is that integrated over, say, the past 30 days.
And the SLI must be satisfied for 99.9%, or three nines,
of all of the probes.
LIZ FONG-JONES: Great, so you're measuring your performance
and what users are experiencing.
But are you using that to alert?
SETH VARGO: Wait, you want me to add more alerts?
I want to sleep more.
LIZ FONG-JONES: Well, yes, but we'll
eventually wind up turning off the redundant alerts that
are measuring potential causes of problems
rather than corresponding necessarily
to actual unhappy users.
So remember, what we're trying to do is make our users happy.
So by using the SLO measurements to decide whether our users are
experiencing actual symptoms of pain,
then we can use that to guide us,
and everything else is extraneous.
SETH VARGO: OK, that makes sense.
LIZ FONG-JONES: So let's take an example.
You said you wanted to have your SLI satisfied
99.9% over 30 days.
So that means that you have an error
budget of 43 minutes per 30 days of 100% outage,
or potentially longer if you had a partial brownout, right?
SETH VARGO: Right, that math sounds familiar.
LIZ FONG-JONES: OK, so you'd want
to find out and be able to intervene
to put the system back to correctness relatively
quickly if you are expending error budget far more
quickly than the error budget indicates.
For instance, you'd clearly have a problem
if you spent a quarter of your error budget
for the whole month within two hours,
either by being hard down for 10 minutes
or by having a partial brownout of 10% of your probes
for those two hours.
SETH VARGO: Well, I see.
And if that pattern continued for the next few hours, or even
minutes, I could potentially wind up
spending my entire error budget for the quarter or the year.
LIZ FONG-JONES: Exactly.
But we can't put it on a hair trigger
or else it will falsely alert us when
we have a blip for two minutes, rather than something
that winds up repeating itself within the next two hours.
So we can have some situations where
a partial brownout wouldn't necessarily
endanger our error budget.
SETH VARGO: Yeah, I've had enough noisy pages
to want to avoid that happening ever again.
But I think I can spot a small hole in your logic.
Like, what if I deployed some code that introduced,
say, a small performance regression?
Something maybe half of a percent of my probes
will constantly fail.
I'd never catch it by looking for too many failures
in a window of hours.
That would be indistinguishable from other noise in the system.
LIZ FONG-JONES: Yes, you're exactly correct.
So we also need a slow burn alert
to track down over the past week if we're exceeding our error
budget at an acceptable rate.
For instance, we could check whether more than 0.15%
of queries were failing when we're only allowed 0.1%.
And we could raise a lower priority ticket
that someone could look at during business hours
once we are sufficiently confident.
SETH VARGO: So as I understand it,
we need both fast burn and slow burn alerts
to provide a feedback loop on our SLO
and make sure that we're fixing real issues that
are affecting real users.
LIZ FONG-JONES: Exactly.
And Stackdriver SLO from Google Cloud
can help you by automatically determining and showing to you
what your error budget is, how much of it you spend,
and alerting you when it's in danger.
And once you've set up these SLO-based alerts,
you can turn off all of the noisy alerts
because you're only getting alerts
about the high importance problems that are actually
causing user impact.
SETH VARGO: OK, but how do I actually find
the problems in my system so that I
feel confident turning off the no longer needed alerts?
I'm worried that I'm going to lose data or be
unable to fix issues now.
LIZ FONG-JONES: That sounds like an excellent subject for us
to discuss on the next episode about observability
of distributed systems.
Thanks to everyone for watching, and we'll catch you next time.
Please remember to like and subscribe below.
[MUSIC PLAYING]