SETH VARGO: Hi there, and welcome
to the fifth video in our series on SRE and DevOps.
My name is Seth, and I'm a developer advocate
at Google focused on infrastructure and operations.
LIZ FONG-JONES: And I'm Liz, an SRE focused
on teaching Google Cloud customers how to build
and operate reliable services.
SETH VARGO: OK Liz, so I've defined my SLI and--
what are you wearing?
LIZ FONG-JONES: What do you mean?
The dress and this flying broom?
It's what we wear in the future.
SETH VARGO: You're from the future?
LIZ FONG-JONES: Yes.
Just keep going, it'll all make sense in the end.
SETH VARGO: OK, Liz.
As I was saying, I've defined my SLIs and my SLOs
and I calculated my error budgets.
And my teams are working together instead of fighting,
and I'm done!
I've successfully implemented DevOps and SRE practices.
LIZ FONG-JONES: But you have customers, right?
SETH VARGO: I mean, we tell our investors that.
LIZ FONG-JONES: So if you have customers,
they might want to connect their servers to your service.
You have an API, right?
SETH VARGO: [UNSURE MUMBLING NOISES]
Maybe?
LIZ FONG-JONES: Well, now you have
to teach your customers how to do DevOps and SRE.
SETH VARGO: Wait, so you're saying that all of the stuff
we've talked about in all of these videos,
I have to go teach that to my customers?
LIZ FONG-JONES: Yes, yes, you do.
SETH VARGO: But all of those things like toil and error
budgets and SLIs and SLOs-- all of that?
LIZ FONG-JONES: Well, you technically don't have to,
but have you ever had a user be angry at you
because your service is down?
Yeah.
Well, what happens if your service behaves
in a way that's in spec for your beautifully-crafted SLOs,
but that breaks the customer expectation?
SETH VARGO: Yeah, I can see that,
and no one likes unhappy users.
We certainly don't.
LIZ FONG-JONES: And you can imagine that at Google's scale,
that with that many unhappy users,
we'd want to do something about it.
So anytime you have a consumable service
that has an API, that's going to happen.
And the best way to make for your customers to be happy
is to communicate clearly about how your service is intended
to behave, which coincidentally means
exposing your SLOs to them.
SETH VARGO: Interesting.
So I have to teach them how to make
a good SLO for their systems, and that
will teach them how to interact correctly with my system?
LIZ FONG-JONES: Exactly.
SETH VARGO: OK, but I'm not sure I follow.
How is that going to help?
It seems like a lot of upfront work.
LIZ FONG-JONES: So once upon a time,
we didn't talk about SRE publicly.
And we thought of it as our secret sauce,
and we tried to keep it secret.
And then we learned a very hard lesson--
our customers had problems, and they had problems
because they had used our systems in ways
that we weren't expecting and that neither of us
were communicating about.
We didn't tell our customers how our systems were designed,
and they didn't tell us what they were expecting.
And then when everything broke, they got very upset,
and you can imagine that multiplied by millions
or billions of users.
SETH VARGO: Well, I mean, at Google's scale,
I can imagine how that would quickly add up over time.
LIZ FONG-JONES: Yep, it sure did, and we learned eventually
that we can move faster and ship features to our customers
faster if we're openly communicating
with our customers and helping them do SRE.
So the time that we spend teaching people
about how to do SRE--
why is it paying off a lot?
Because when users apply these things to their systems,
it makes them have much less friction with us.
SETH VARGO: I see.
So an initial, upfront investment
on education around SRE has long-term dividends?
LIZ FONG-JONES: Exactly.
And it was so successful that now we
have an entire team, which is the team that I'm
part of-- the Customer Reliability Engineering Team,
or CRE.
And CREs are regular Google SREs that are focused externally.
Instead of working on our own GCP applications,
we instead help customers who are implementing
their applications on GCP, and we
help them also implement SRE practices like SLIs and SLOs.
And that means that in the long-term,
we can share information.
We can share our SLOs with our customers
and have our customers share their SLOs with us.
So we'll know if we break one of our customers,
or our customers can figure out whether an issue they're seeing
is a glitch on their end or whether it's
something that's in GCP.
SETH VARGO: That's so cool!
Thanks, future Liz!
[SCI-FI TELEPORTATION SOUNDS]
That was weird.
[RUNNING FOOTSTEPS]
[PANTING]
LIZ FONG-JONES: I'm sorry I'm late, Seth.
Everything OK?
I hope I didn't throw off the schedule.
SETH VARGO: Not a problem.
We were just talking to a future version of yourself.
LIZ FONG-JONES: Wait, a future version of me?
That's so cool!
So you're saying future me invented a time machine
and then went back in time so as to make me not late to this?
SETH VARGO: Exactly.
LIZ FONG-JONES: That's really cool!
What did she say?
SETH VARGO: So we were talking about the Customer Reliability
Engineering program-- or the CRE program--
here at Google, but I'm still a little bit unclear
about how it plays into this bigger DevOps picture.
LIZ FONG-JONES: Ah, yes.
So CRE is Customer Reliability Engineering,
and we focus on breaking down the organizational barriers--
not between individual silos in a single company,
but instead between a Cloud customer and its platform
provider.
And we also think about, how do we make failure
a normal, accepted thing?
So for instance, we make sure that everyone
has service-level objectives and error
budgets, so we communicate about what's
an acceptable level of reliability
so that everyone doesn't panic when there's
a little bit of downtime.
And also, we focus a lot on measurement.
So we make sure that everyone has visibility into the system
so that customers can see how the platform is performing,
and the platform can see how the customer service objectives are
faring during an outage.
So in that way, we have a single pane of glass between everyone.
SETH VARGO: Great.
Thanks, Liz.
And thank you, everyone, for watching.
Don't forget to subscribe to the channel
and stay tuned for future videos--
and maybe a future Liz.
Check out the description below for more information
about Customer Reliability Engineering at Google.