[MUSIC PLAYING]
LIZ FONG-JONES: Welcome to episode 9
of our series on SRE and DevOps.
I'm Liz Fong-Jones, a site reliability engineer at Google,
where I teach Google Cloud customers how to build
and operate reliable services.
SETH: And I'm Seth, a developer advocate
at Google focused on infrastructure and operations.
LIZ FONG-JONES: So Seth, I was hoping
you would help me figure out why are
things so chaotic every time one of my services break?
SETH: So what kind of chaos are you experiencing?
Too much data?
Slow internet connections?
Corporate firewalls?
VPNs?
LIZ FONG-JONES: No, that's not actually what I mean.
I'm talking about human chaos.
There's often both too much communication and too little
communication at the same time.
Everyone wants to participate.
People are doing random things that don't actually help.
And people actually doing the work aren't talking.
SETH: Ah, yes.
Humans are definitely an important part of the incident
management process.
It sounds like you need to define very clear roles
and responsibilities for each of your humans,
so that they know exactly what they can
do without getting in the way.
LIZ FONG-JONES: I'd love to figure that out.
How do I get started, Seth?
SETH: First, you'll need a process
for declaring incidents, a dashboard
for viewing your current incidents,
and some kind of database for who
to contact when an incident is in progress
for different kinds of incidents.
You'll also need to set thresholds
for when to declare an incident and who
can declare an incident in your organization.
Depending on the scale of the incident,
you may also need to bring in specialists
from other parts of the company in order
to help resolve the incident.
LIZ FONG-JONES: Right.
I wouldn't necessarily want to declare an incident
every single time and get paged, but I also
wanted to feel supported in declaring an incident if I
need to bring in other people.
Then what?
SETH: So I mentioned having someone to contact.
The most critical role in the incident response process
is that of incident commander, or IC,
who is responsible for making strategic decisions
and delegating additional roles to other responders.
The other roles can also be played by the incident
commander but tend to be delegated as the incident grows
in size or complexity.
The first role that tends to be created
is that of the operations lead, who
is in charge of gaining a detailed understanding of
and actually changing the state of the system--
like running commands, grabbing log files, et cetera.
LIZ FONG-JONES: I gather that only knowing
having one person touching the system makes a lot of sense,
but I'm not sure how that's helpful to have the incident
commander isolated from the details.
Don't they need to know all those details?
SETH: So yes and no.
I mean, have you ever been driving a car in the dark
in some foreign country on the opposite side of the road,
and you're trying to get where you need to go, but you can't?
LIZ FONG-JONES: A few times, yes.
And it's super difficult. I usually
have to pull over because if there's no one else
in the car with me, I can't both look at the map
and figure out where am I going, where am I trying to go.
And then, I drive safely at the same time.
I can't do that all at once.
SETH: Right.
So I think you just answered your own question, right?
If we apply that same analogy to the operations world,
it's hard to both plan and perform
to keep track of all of the fine-grained details
at the same time.
So it's usually best to separate the two
roles to minimize the overhead of context switching.
LIZ FONG-JONES: I see.
That makes a lot of sense.
So the incident commander and the operations lead under them,
but you mentioned multiple roles.
What other roles might an incident commander
want to delegate?
SETH: Well, have you ever had external stakeholders,
like a support team or executives, who are bugging you
for updates during an outage?
LIZ FONG-JONES: All the time.
SETH: Or maybe if you're a public-facing company, perhaps
you have a presence on social media
where your users are constantly nagging you
about the service being down.
LIZ FONG-JONES: Ah, it's super distracting, especially
if I'm the operations lead.
But isn't that just the incident commander's job to do?
SETH: Sometimes-- up to the point
that the incident commander can no longer effectively
do both the planning and the communications work.
At that point, you'll want a communications lead
for communicating with other teams or external stakeholders.
LIZ FONG-JONES: I see.
And what if an incident grows so large
that the decision making responsibility,
and the planning, and the logistical work becomes
too much work for one person to handle?
Then what?
SETH: So in that scenario, you might
want to have a dedicated planning lead--
having someone be in charge of writing the plan,
maintaining the running notes document,
and starting to draft the post-mortem.
Another common role is a logistics lead,
who handles things like securing rooms,
making sure that people have food and water,
and other supplies.
Or you could get more creative, depending
on the needs of your organization and your business.
LIZ FONG-JONES: That makes a lot of sense.
What can't the incident commander delegate, though?
It sounds like the incident commander
can delegate anything.
SETH: So the incident commander can't
delegate overall decision-making responsibility.
The buck stops with them.
They have to help their subordinates make decisions,
tiebreak in the event that people can't agree,
and coordinate adding new people into the incident response
hierarchy as deemed appropriate.
LIZ FONG-JONES: And what if there
needs to be multiple people working hands on keyboard,
or multiple people need to be doing the communications?
How does that work?
I thought you said only one person's
in charge of operations.
SETH: So additional subject matter experts
can join under the operations lead.
Or the overall communications lead
might want to designate someone to be on the customer support
team, while another person is handling social media
for external communications.
LIZ FONG-JONES: And what about the role of managers
and technical leads?
Do I automatically become the incident commander
just because I'm the most senior person who turned up?
SETH: No, not unless you're specifically asked to.
The person who declared the incident
is always the incident commander until they
designate a successor.
It's up to them to decide what roles in which
to place people joining or even turn people away
if there are too many cooks in the kitchen.
For long running incidents, the incident commander
made delegate a successor in a more appropriate time zone
or geography.
As an incident evolves in size or complexity,
some incident commanders may want to give up the role of IC
and instead become the operations lead.
LIZ FONG-JONES: Great.
That should help clear up some of the confusion
and chaos for my next incident.
Thanks, everyone, for watching.
And we hope you'll tune in next time
to learn what to do after an incident.
To find out when we post our upcoming episodes,
please like and subscribe below.
[MUSIC PLAYING]