SETH VARGO: Hi there.
And welcome to our 10th episode on SRE and DevOps.
My name is Seth and I'm a developer advocate
at Google focused on infrastructure and operations.
LIZ FONG-JONES: And I'm Liz, a site reliability engineer
at Google where I teach Google Cloud customers how to build
and operate reliable services.
So I've just wrapped up an incident.
And I definitely don't want to have the same set of things
happen in the exact same ways.
What should I do to learn?
SETH VARGO: So Liz, you want to conduct a blameless postmortem
or retrospective.
There are a few key things that go into every retrospective,
but there are a variety of ways of collecting that information.
First, you'll need some basic metadata,
like what systems were affected and what humans
were involved in that process.
LIZ FONG-JONES: Right.
It doesn't make sense to conduct a postmortem without involving
everyone who played a part.
SETH VARGO: Right.
And then you need to start assembling any records
that you have and start to begin understanding a timeline,
like, how did we find out about the event?
When did we start responding?
What mitigations did we deploy, either temporarily
or permanently?
When did the incident actually conclude?
And it's important to note that this
is a collaborative process.
Initially, the draft is written by the incident commanders
but all participants collaborate and add
their relevant information.
It's usually helpful to use a collaborative tool
like Google Docs for this step.
LIZ FONG-JONES: I'm starting to understand now
why you emphasized the role of the planning lead last time.
It's definitely a lot easier to [? coord ?] things
as you go rather than having to piece everything together
after the fact.
SETH VARGO: And you'll also need to understand what things you
did that you'll need to roll back or clean up
afterwards, things like temporarily increasing
the size of a VM or directing traffic away from a zone.
LIZ FONG-JONES: I can definitely see
how it could get expensive or risky if I didn't put those
back the way that I found them.
SETH VARGO: But most importantly in the process,
though, is making sure that you have a blameless culture,
ensuring that people feel like they won't be faulted or fired
for the mistakes that they may have made.
LIZ FONG-JONES: I definitely see how
I'd be really reluctant to say, I messed up,
if I were afraid of being fired or ridiculed.
SETH VARGO: Exactly.
And participating in a postmortem
is a learning opportunity for everyone involved,
including the company, not some kind of punishment.
And it works best if your team can
think about the possibility of a well-intentioned human
to cause a system failure as a bug in the system,
not as a bug in the human.
LIZ FONG-JONES: Right.
You can't fix humans.
SETH VARGO: It's often helpful to ask questions like, "what
did you know at the time" or "how did the system
behave" as opposed to "why did you restart the process"
in order to avoid putting people on the defensive.
LIZ FONG-JONES: Right.
So how do you actually organize this information
once you're collecting it though?
SETH VARGO: So every organization
has different ways for communicating.
As I said above, at Google, we tend
to use Google Docs and comments.
Other organizations may prefer something
like a live conference bridge with someone
taking notes in the background.
LIZ FONG-JONES: Right.
That makes sense.
And you reminded me by mentioning
taking notes during an audio only conference bridge
that it's not really a great idea to, during an incident,
having audio only conference bridges because you have
to have someone record very detailed
notes in order to reconstruct what happened after the fact.
But we haven't really gotten to anything
about how do we prevent the root cause from happening again.
SETH VARGO: Well, be careful there, Liz.
Remember that failures in distributed systems almost
never have a single root cause.
It's usually more than one contributing factor
that's causing an intersection or some kind
of cascading failure.
So you'll want to write down everything that contributed
to this abnormal behavior instead of just trying
to identify the root cause.
LIZ FONG-JONES: So what I should be focusing on
is how do I mitigate individual kinds of failures
and how do I prevent them from intersecting
in these same ways.
SETH VARGO: Right.
And then you'll want to keep track of that is explicitly
filing an issue in your issue tracker of choice
for each action item so that you'll
make sure to prioritize it for the future.
LIZ FONG-JONES: And I can definitely tag these things
with the postmortem tag in my bug tracker
so I can see how at risk I am of repeat outages based off
of how many high priority items that I've checked off
and how many remain outstanding.
SETH VARGO: Exactly.
And you'll also want to capture some overall themes like what
went well, what went poorly, and, honestly, what
did we just get lucky with.
LIZ FONG-JONES: I also imagine that recognizing
people that went above and beyond
and systems that worked well and worked as designed
is a good positive motivator as well.
SETH VARGO: Exactly.
And one last key thing to keep track of,
once you understand the incident in its entirety,
is to write down machine readable metadata
so that you can track improvements
to your overall incident response management process
over time, such as decreasing the total time to resolve
incidents, or knowing whether a release was involved,
or a cascading failure, or config changes, et cetera.
This can help you identify meta patterns in your outages
and build either process or technology
to help prevent or mitigate those incidents in the future.
LIZ FONG-JONES: And that's where Google statistics come
from that approximately 70% of our outages
are from binary configuration changes,
because we've been recording that metadata
over the past seven years.
SETH VARGO: Hopefully, these tips
help you prepare for your next incident.
And thanks to everyone for watching this series so far.
Please write to us on Twitter if you
have feedback or suggestions for future topics.
Thank you for watching.
Please be sure to like and subscribe below.
[MUSIC PLAYING]