YURI GRINSHTEYN: The most important feature of any system
is its reliability.
This is the driving principle behind cyber reliability
engineering as invented and practiced at Google.
This is because you can easily lose the trust
and respect of your users.
If your service becomes too unreliable,
or is simply perceived as being too unreliable,
those countless hours you've spent designing and building
the most amazing user experiences
are all for nothing if your users can't access them.
But it's not enough to just declare that your services must
be more reliable.
Reliability must be a shared goal for many different parts
of your organization.
Today, I'm going to start showing you
how you can consistently define reliability targets
for your services using SLOs.
Welcome to Engineering for Reliability with Google Cloud.
[MUSIC PLAYING]
The first thing we need to define
is what we mean when we say "service."
For our purposes, we'll mostly refer to one of these three.
The first kind of service may be an application with a user
interface that's accessed by actual humans
to perform various tasks.
We refer to those humans as "users,"
and we very much care about keeping them happy,
so that they continue to use our service.
We'll measure their happiness in terms
of things like availability and performance.
The second kind of service may be
an API, accessed by other systems or services
to send or retrieve information.
We may refer to the systems that use our service as "consumers."
Their happiness is also important,
since this interaction may involve
exchanging data that will ultimately
be consumed by human users.
Availability and performance will, again,
be our key measures of reliability.
The final kind of service may be a data processing pipeline.
Here, we'll care about things like the throughput
of the system, its ability to process data correctly,
and the time that it takes data elements to traverse
the entire pipeline as measures of reliability.
These three types of services can end up
having pretty complex architectures
and infrastructures.
But they all share the same key feature, reliability.
If that's true, then we should treat reliability
like other features of our service.
We'll need to plan and design for it, architect for it,
and measure it.
But how do we measure the reliability of our service?
One way to measure reliability would
be to use uptime, or the percentage of time
when our service is up.
However, "up" is hard to define and measure,
and for complex distributed systems with many failure
modes, this is not very useful.
Instead, we can use an event or interaction-based model.
We can count the number of times our users attempt
to interact with our service over a given time window
and divide the number of successful attempts
by that total to get a percentage.
But how do we define what counts as an interaction,
and what makes for a good or successful interaction?
That's going to depend on the kind of service
we're running and managing.
For example, a website or mobile app
can use HTTP requests, separated by status code,
as an interaction.
The thing we'll actually be counting and measuring
is called the Service Level Indicator, or SLI.
Next, we need a definition for "good" or "successful."
That will depend on what our users expect from our service.
In this example, we're defining success for the profile page
as receiving an HTTP 200 response code.
Now that we've defined what success looks like,
we need to define what we'll include
in the total or valid set of requests.
For example, we may want to exclude 400s from our count,
because those are likely caused by misconfigured clients,
and we don't want our reliability measure
to be penalized as a result. And now, our availability, as slide
definition, is complete.
Here are other potential SLI types
you can consider using for your service.
The SLI menu helps you choose an SLI for your service
based on who your users are, what
they expect from your service, and how they use it.
OK, we know what we're going to use to measure
the reliability of our service.
Next, we're going to set a reliability target
for our service.
The target we set is called the SLO, or Service Level
Objective.
It's a combination of our SLI, a threshold, and a window of time
over which it's evaluated.
The key thing we're after is a target or threshold
above which users are happy and below which they're not.
It might be tempting to set our target at 100%,
but that would be impossible to meet, and result
in not being able to make any changes to our service
and production.
We have a simple rule of thumb for where
SLO targets should be.
A typical user of the service should
be just about happy with the service
when it operates at those targets.
If the service was any less reliable,
you'd no longer be meeting their expectations,
and they would start to become unhappy.
We call this the happiness test.
When we talk about the happiness test,
we say "barely met," because there
are downsides to exceeding your reliability target, too.
Once your users are happy with the reliability
of your service, additional reliability has little value.
Maybe your users would be happier
if you built new features or made other service improvements
with the resources that you're dedicating to far exceeding
your reliability targets.
We've defined reliability as a key feature of our service
and agreed to use SLIs to measure reliability and SLOs
to set reliability targets.
But how can we actually do those things?
Let's have a look at how to use SLO monitoring and Cloud
Operations to do just that.
Start from the project dashboard in the Cloud Console.
Go to the Monitoring menu and select Services.
This will bring you to the list of services in your Workspace.
As you can see, in this example, there's
a long list of services.
Services may be created automatically
from App Engine services, GA entities, the Istio service
mesh, or Anthos services.
You can also define your own custom services.
Let's start by figuring out which of our services
are having issues right now.
To do that, sort the list by the SLOs with firing alert column.
You can see that the default service here is at the top.
Let's have a closer look at it.
We've defined three SLOs for this service.
There's a calendar month response latency
SLO, which is currently healthy, with plenty of error budget
still remaining.
There's a rolling window availability
SLO, which is also healthy.
Finally, we have a calendar week latency SLO.
There's plenty of error budget left,
but it's currently showing as unhealthy.
Let's dig in to see what's going on.
First, we can see how our SLI is behaving against our target.
In this example, we're seeing regular periods
where we're having trouble meeting that target.
The Error budget tab shows us our error budget
burn over the selected interval.
This rapid rise is a great example
of why using calendar-based rather than rolling SLOs
carries risk.
We recover all of our error budget
at once when the calendar period rolls over,
rather than by actually increasing reliability.
To further investigate our service,
we give you a variety of performance and health metrics
that are automatically gathered for it.
Because this service was defined based on a GKE namespace,
we have access to Container, Pod, and Node
metrics for that namespace.
As an example, we can switch to the Container Restarts
and Uptime view to see how stable our service is.
We can see that a number of our containers
restarted right around the same time.
Further down, we can see logs being
emitted by our service and the Kubernetes entity details
below that.
This is how SLO monitoring in Cloud Operations
brings together all the information
you need to measure the reliability of your services,
using SLOs and the diagnostic and troubleshooting information
that you might need to identify, triage, and mitigate issues.
Thanks for joining me today.
In this first episode, we covered some common types
of services, the importance of treating their reliability
as a key feature, and how to use an interaction-based model
to measure reliability.
Next, we learned about using service-level objectives
to set reliability targets.
Finally, we looked at how to use SLO monitoring in the Cloud
Operations Suite to measure the reliability of your services
using SLOs.
Thanks so much for watching.
Don't forget to like and subscribe to never miss out
on more Engineering for Reliability with Google Cloud.
See you soon.
[MUSIC PLAYING]