[MUSIC PLAYING]
SETH VARGO: Hi there, and welcome back
to our eighth episode on SRE and DevOps.
My name is Seth, and I'm a developer advocate
at Google focused on infrastructure and operations.
LIZ FONG-JONES: And I'm Liz Fong-Jones, a site reliability
engineer at Google, where I teach Google Cloud
customers how to build and operate reliable services.
So last time, we were talking about [INAUDIBLE] alerts.
And Seth, you were really, really skeptical
about letting go of your alerts about every single potential
root cause in the system.
SETH VARGO: Yeah, honestly, it's terrifying to let
go of all of these alerts that we painstakingly built
in the past that have happened as a result of an outage
or would have prevented an outage from happening.
LIZ FONG-JONES: I totally hear that.
But what we need are better ways of finding problems
in our systems that don't necessarily
rely on making assumptions about what's broken before,
rather than layering on technical debt
after technical debt after every outage.
SETH VARGO: So I've heard there was a word for that--
observability?
LIZ FONG-JONES: Yeah, that's it.
SETH VARGO: We need to make our systems observable,
rather than only being able to debug
what we've thought to log or monitor or alert on
in the past.
LIZ FONG-JONES: Exactly.
So observability breaks down into three key areas--
structured logs, metrics, and traces.
Structured logs are emitted by each process
and may contain a mixture of information,
ranging from records of individual user queries
to debugging information generically associated
with our service.
Metrics reflect aggregate-type data
about the performance of services,
such as the number of queries, which
is a counter, their latencies, which is a distribution,
and the CPU load, which might be a gauge.
And traces allow for individual execution flows
to be traced through the system, including
timing and dependency details.
SETH VARGO: So I see, and all of this makes sense.
I still don't understand.
How does this help me better understand and diagnose
my problems?
LIZ FONG-JONES: Let's take an example.
So we use metrics as part of our service level indicators,
right, in order to monitor outages
that are spending our error budget faster
than we can tolerate.
So once we have an indication that something's wrong,
we need to figure out where in the system it's broken.
And can we mitigate it by draining a zone or specific set
of replicas?
And we might need to drill down into the metrics
in order to find out what are the commonalities
between the slow and erroring queries?
For instance, do they have a shared zone, or a shared set
of dependencies, or maybe a different version
of the code that is new that we just rolled out?
So we might use traces after that
to find the interaction between the components that's causing
the slower failing queries.
And we might need to use the logs
to understand each failure in detail.
SETH VARGO: So if I'm understanding correctly,
once I get paged, I might look at these kind of dashboards
of all these different metrics with common starting points
for queries, such as break down by region, et cetera.
And even though I'm only getting paged on these kind of higher
level issues, I'll still collect information
about these lower level issues, which could
help me debug the root cause.
LIZ FONG-JONES: Exactly.
And then you'd need to edit your queries to further drill
down or get breakdowns intersected
with other properties, for instance,
the build version or the set of feature flags.
You don't necessarily need to alert on every metric
you collect, just on the error budget.
But having that information available
really, really helps when you do need to diagnose problems.
SETH VARGO: OK.
So I'm with you so far, but then where does tracing come in?
LIZ FONG-JONES: If you see that queries are still
at the top level, it's often helpful to be
able to skip down directly to where
you think the problem might be happening,
layers deep in the system.
So the best and easiest way of doing that
is having a visual representation
of where is the time spent in your queries?
Where is the new bottleneck that's increasing your latency?
SETH VARGO: OK.
So I've pulled up on the traces dashboard,
and I can see that most of my queries
are completing in about 50 milliseconds
in Cloud Big Table.
But there are a few queries that are orders of magnitude larger
taking longer to complete.
I also notice that many of the slow queries
happen to be on the same row in Cloud Big Table.
LIZ FONG-JONES: That sounds an awful lot
like a hot spot, Seth.
SETH VARGO: Definitely.
It smells that way, too.
I see now, and I can look at the relevant logs
on individual replicas that the tracing is suggesting
to help understand why that same row is being queried repeatedly
and why that query is so slow.
So I can see how this architecture helps me identify
and solve problems faster.
But what if I'm not already using these types
of observability tools?
What if I'm coming from a traditional monitoring
solution, or kind of embarrassing to admit,
but I still [INAUDIBLE] through log files manually.
How do I get started with something like this?
LIZ FONG-JONES: Don't be so scared, Seth.
That's a great question.
I'm glad that you asked.
Google Cloud offers a suite of tools called Stackdriver,
including Stackdriver Monitoring, Stackdriver APM
Trace, and Stackdriver Logs that let
you adopt all these techniques.
And there are really cool integrations
between all of these various observability methodologies.
But you should start by measuring your SLOs first
in Stackdriver Monitoring and Stackdriver SLO.
SETH VARGO: Great.
I'm glad to have learned so much today and hope all of you
did as well.
Thank you for watching, and we hope
to catch you next time, where we'll
discuss how to resolve incidents with as little chaos as
possible.
To find out when we post our upcoming episodes,
please like and subscribe below.
[MUSIC PLAYING]