[MUSIC PLAYING]
SETH VARGO: Hi there, and welcome to the sixth episode
on SRE and DevOps.
My name is Seth, and I'm a Developer Advocate
at Google focused on infrastructure and operations.
LIZ FONG-JONES: And hi.
I'm Liz, a Site Reliability Engineer or SRE at Google
where I teach Google Cloud customers
how to build and operate reliable services.
So Seth, I have an error budget now.
And it's based on a level of reliability
that my customers expect, but I don't
want to have to wait months to find out whether or not I'm
going to meet my error budget.
SETH VARGO: Definitely.
Collecting all of that data can be time consuming.
That's why you need a Risk Analysis.
LIZ FONG-JONES: A risk what?
SETH VARGO: An analysis of what items or risks that might
cause you to not meet your SLO.
LIZ FONG-JONES: I see, yeah.
I know all the ways that my system has failed in the past.
And I can think of many different ways
that my system might fail in the future.
But how does that help me figure out
if my error budget is realistic?
SETH VARGO: So you need to figure out
how often each failure is likely to happen
and how severe it will be if and when it happens.
LIZ FONG-JONES: I see.
But that still doesn't help me compare the impact
on my error budget.
How do I know if a high severity but infrequent form of outage
is better or worse than a frequent but low level
degree of degradation?
SETH VARGO: Right.
So that's actually the secret.
You just have to multiply those metrics together.
Here, let me show you.
Tell me about one of your common incidents or failure modes
that you're worried about.
LIZ FONG-JONES: So for example, I
have a primary database that I have to backup every month.
And it takes the entire database down for two hours
every, single time.
SETH VARGO: So that means that your time between failures
is roughly 30 days every month and that your impact
duration is 120 minutes.
It also impacts your entire user base, since during that time,
the entire system is offline.
LIZ FONG-JONES: That makes sense.
SETH VARGO: So that means you'd have about 120 minutes times 12
months times 100% of your user base
which is about 1,440 bad minutes per year.
LIZ FONG-JONES: And my error budget
says that I'm supposed to achieve
2 and 1/2 nines or 99.5% reliability over the year,
meaning that I can have about 2,600 bad minutes per year.
SETH VARGO: Right.
So that means your monthly database downtime is consuming
half of your error budget.
But I'm sure that's not the only kind of downtime you're seeing.
LIZ FONG-JONES: No.
As an example, every couple of weeks people
complain that the system is slow on Friday afternoons,
because everyone is trying to build and test
their code before they go home for the weekend.
SETH VARGO: Yeah, that happens in, like,
every software development shop.
LIZ FONG-JONES: So it's so slow that it causes half our users
to be unhappy with us.
And what's worse is it takes us half an hour for the support
tickets to come in and then half an hour
to actually provision more capacity.
So that is probably not very good, right?
SETH VARGO: Yeah, so if I were to put this
into the same framework, I'd say that this
is a 60 minute outage, 30 minutes to detect
and 30 minutes to resolve, times 26 times per year,
affecting about half of your user base.
And that's 780 bad minutes per year.
LIZ FONG-JONES: Right.
So this outage could wind up being about half
as bad as the database backup downtime.
So that makes a lot of sense.
SETH VARGO: Great.
So we can fill out these calculations
together for any other sources of outages
that you'd like to catalog.
LIZ FONG-JONES: Great, I'll get right to it.
[MUSIC PLAYING]
LIZ FONG-JONES: OK, whew.
Now I'm done with that.
SETH VARGO: So now that you've compiled all of the risk data,
we can sort the outages by the number of bad minutes per year.
Indeed, as we suspected, the monthly database outage
is consuming the majority of your error budget.
Combined with these other small sources of outages,
it's clear that you're going to exceed your error budget.
LIZ FONG-JONES: That's not really good.
SETH VARGO: Well, but it's not all bad news.
Since we've gathered our estimates
and quantified this data, we can find out
what things will bring us back under the error budget.
For instance, anything that consumes about 25%
of the error budget or more is something
that we want to address.
LIZ FONG-JONES: That makes sense.
And if I address all of the large risks
and I still wind up exceeding my error
budget, what do I do then?
SETH VARGO: So then you need to pick a combination of risks
to address that will bring you within the budget, not
necessarily all of the largest ones.
LIZ FONG-JONES: Right, because some of the larger
risks I can't necessarily address without rewriting
the whole system.
That will take time.
So you're saying that if I accumulate enough
of these smaller things and fix them,
that will potentially bring me back within my error budget.
SETH VARGO: Right.
So now you have a plan for how to estimate risk and a way
to create plans to stay within your error budget.
Alternatively, if there's no way to mitigate all of that risk,
you now have numerical evidence to go back
to your key stakeholders to revisit
the possibility of a different SLO that would give you
a larger error budget.
LIZ FONG-JONES: Thanks, Seth.
And thanks to everyone for watching this episode.
Hopefully this is helpful for teams
trying to put DevOps into practice through SRE
techniques.
I'm really excited to figure out how
to set up effective alerts for our service,
so we can know whether we're in danger of violating
our SLOs on the next episode.
SETH VARGO: Hey Liz, what's that button down there?
LIZ FONG-JONES: Oh, that's the Like button.
You should press it, if you enjoyed this.
SETH VARGO: And this one over here?
LIZ FONG-JONES: That's the subscribe button,
so you can find out when we post our next episode.
[MUSIC PLAYING]