LIZ FONG-JONES: Hi there.
My name is Liz.
And I am an SRE teaching Google Cloud customers
how to build and operate reliable services.
SETH VARGO: And I'm Seth, a Developer Advocate focused
on infrastructure and operations.
In the previous episode, we discussed SLOs and error
budgets as a means of balancing availability and new feature
development.
In this episode, Liz and I discuss toil.
So Liz, from what I understand, toil
is just work that I don't like to do,
so emails, meetings, commuting to work.
So what can I do to reduce my toil?
LIZ FONG-JONES: Hold on a second, Seth.
Those things aren't specifically toil.
So we call those things "overhead,"
because they are things that are not directly tied
to a production service.
So they're just a requirement of any office, job really.
Most people at the company, not just SREs,
have to do expense reports and so forth.
And that's not really toil.
SETH VARGO: OK, so if toil isn't just work that I don't
like to do, what is it, then?
LIZ FONG-JONES: Toil's the kind of work
that's tied to running a production service that
tends to be manual, repetitive, automatable, tactical,
and devoid of long-term value.
Additionally, toil tends to scale linearly
as the service grows.
SETH VARGO: I see.
So each time an operator needs to touch a production system,
that represents some toil time.
But what if I codify all of the times that I need to touch it,
and I just run a script instead of manually
running the commands?
LIZ FONG-JONES: You've reduced the amount
of toil you have to do, but you haven't made it not toil.
Because you still have to run that script every single time
the system requires an intervention.
So that still is manual.
And it gets even worse the larger your service grows,
because you have to keep doing it
over and over and over again.
SETH VARGO: OK.
And you also said something-- the task
has to be repetitive, too.
So if I just do it maybe once or twice,
is that still toil or no?
LIZ FONG-JONES: That is probably not toil,
at least the first or second time, or at least
it's not toil that's worth automating.
You really want to automate toil when it gets to the point
that you're doing it three or four times,
or where you are taking hours and hours out of your day.
SETH VARGO: I see.
So if a task can be automated, it probably
should be automated.
What about things that are pretty common in operations,
like pager alerts?
LIZ FONG-JONES: So pager alerts are almost always toil,
because of the fact that they are interrupt driven
and they're reactive.
You don't know when it's going to happen.
And it just swallows a bunch of your time just
to get the system back to where it was before.
SETH VARGO: OK, so what if I get paged in the middle
of the night to provision additional capacity
for a service?
So I look through the system, and I realize that the only way
to do this is manually.
I have to click buttons.
I have to run commands.
So I do that.
I get the service back up and running.
And I go back to sleep.
And in the morning, I wake up, and I realize
that I need to automate this.
So I spent about six or seven hours building some automation
so that the service can scale automatically in the future.
That's a lot of toil.
How do we account for that?
LIZ FONG-JONES: Not quite, because of the fact
that the time that you spent during the middle of the night,
between when you got paged and when
the system returned to normal from your manual action,
that's toil.
However, the latter part of what you did during the day--
that's a worthwhile project that actually
reduces the toil over time, because the toil has
no enduring value.
But the automation that you do in order
to reduce the amount of toil you do, that's real useful work.
SETH VARGO: I see.
And I think it's clear now why you've coined a new term
to define this type of work.
There's always going to be tasks that you don't want
to do, like overhead, like email, and meetings,
and commuting, and expense reports.
And there's always going to be kind of operationally grungy
tasks, like restarting services, that
are devoid of long-term value.
So they aren't actually toil in certain cases.
In previous episodes, we talked about how everything in SRE
is measured, though.
So how do we measure toil?
LIZ FONG-JONES: So there are two ways that we measure toil.
First of all, we try to concentrate the toil
into blocks, for instance, during your on-call week,
so you don't get pages and tickets off
of your on-call week.
So it makes it a lot easier to measure
when it is toil, when it is not toil.
Additionally, we have people track their toil time,
sometimes by sampling.
And we send quarterly surveys to make sure
that people understand how much of their time is spent on toil.
Do they feel like they have more room?
And then toil is also everyone's responsibility.
So we'll often ask the product development software engineers
to measure their amount of toil as well,
to make sure that we're spending everyone's time well.
SETH VARGO: So you mentioned that the goal
should be to reduce toil.
Shouldn't the goal be to eliminate toil entirely?
LIZ FONG-JONES: Part of the problem
with eliminating toil entirely is that there are often tasks
that are not worth automating.
If you spend an hour fixing something by hand,
and it happens maybe three times a year,
but it would take you 10 hours to write the automation,
that's going to have a very long payoff threshold.
Whereas if you have something that's firing all the time,
you might want to go and get it out of your way.
So as you reduce the amount of toil
and leave alone the things that are not worth automating,
you'll find that there's some amount of toil
associated with each service.
You can keep the total amount of toil
you do somewhere between 30% and 50% of your time,
and spend the rest of your time doing worthwhile projects that
reduce the toil, so that over time, you
can accept more responsibility for more services,
or your service can just naturally grow in size
and have your amount of work really remain constant.
SETH VARGO: I see.
So SREs get to spend at least half,
if not two thirds of their time working on real engineering
work.
The SRE stands for Site Reliability Engineering.
So they get to focus on things like improving
reliability, performance, utilization, monitoring.
And all of those things can actually
reduce toil in the future as well.
LIZ FONG-JONES: Yes.
That's exactly correct.
And also, we share our best practices.
So when we do these items of research
into how we make our services better,
we can often apply them across multiple different SRE teams.
So we'll publish white papers within the company
or on our blog, describing to SREs how they might make sure
that their toil stays low and within budget,
so that we can spend more time making our services better
in the long term.
At the end of the day, the E in SRE stands for Engineering.
And engineering work is really what
lets our organization scale, and meet
the demands of all of the applications and services
and machines that we support.
SETH VARGO: Great.
This has been really helpful.
And I think I have a much better understanding of toil
and its negative consequences.
LIZ FONG-JONES: But toil isn't always entirely negative.
Sometimes, it's worthwhile to have
some amount of predictable and automatable work
for individual people to cut their teeth on.
For instance, let's suppose that you have someone new
to your team that wants to learn how
does the system actually work before you put them on call.
Well, you might have them poke around at the system
manually in order to fix something that you already
had lying around.
Also, it's really helpful to have things
that people feel satisfied by doing, where they
get an immediate reward, rather than working
at a product for the long term and not necessarily knowing
when it's going to finish.
So sometimes, it's helpful for people
to do those tasks in the short term.
But we don't think it's a good idea for people
to focus entirely on doing toil, because that doesn't really
advance the engineering part of their job,
and it can cause career stagnation and unhappiness
in the long term.
SETH VARGO: So toil and toil budgets
do not have a direct relation to the DevOps pillars
that we've discussed in previous episodes.
But they do closely influence the desire
to measure everything and reduce organizational silence.
By giving operators a quantitative measurement,
toil and toil budgets ensure a balance
between administering the system and improving it.
This has really helped a lot.
I think it's a lot clearer now why
we say that cost SRE implements DevOps.
LIZ FONG-JONES: Thank you everyone for watching.
Don't forget to subscribe to the channel.
And also stay tuned for our next video, where
we discuss how Google teaches SRE practices to customers
through the Customer Reliability Engineering Program.