SPEAKER: In our last two episodes,
we learned how to manage GK infrastructure
and the individual services running on it.
But as our applications become more complex,
more services are involved in serving user traffic
and completing transactions.
It then becomes more and more critical to understand
how requests traverse multiple services
and how each service contributes to overall latency.
This is what distributed tracing does.
It captures the latency of user requests
and how long it takes each service in the path
to return response.
Today, I'm going to show you how to capture latency
in distributed applications using OpenTelemetry
and analyze it using Cloud Trace.
Welcome to "Engineering for Reliability" with Google Cloud.
Let's start by reviewing the basics of distributed tracing.
These were originally described in Google's research paper
on the Dapper distributed tracing framework.
We'll link to it below if you'd like to read it.
When a user request comes in, we want
to create a trace that'll describe
how our system responds.
Traces are composed of spans where each span represents
a specific request and response pair involved in serving
the original user request.
The parent span describes a latency as observed by the end
user.
Each of the child spans describes
how a particular service in a distributed system
was called and responded with latency information captured
for each.
Now, let's take a look at a basic example.
We have a front-end service that responds to user requests
and makes calls to a back-end service.
It has to wait on that back-end call
to complete to return the response back to the user.
We want to trace this interaction
to understand both total latency and latency in each service.
We recommend that you use OpenTelemetry
for distributed tracing.
It's an open-source project that provides
a single set of APIs, libraries, agents, and collector services
to capture distributor traces and metrics
from your application.
Let's have a look at how to instrument
your code for tracing.
We'll also link to the full code sample in the episode notes.
We start by sending up the Cloud Trace exporter.
This will let us send the traces we
collect to Cloud Trace for inspection and analysis.
In the front-end code, we start tracing
and create the root span.
We then make a request to the back end
with the appropriate context.
This is data embedded in the HTTP headers
that our back end can use to append
subsequent spans to our traits.
In the back-end code, we extract the context
from the incoming request to create additional spans.
This allows us to explicitly measure
the latency of our back-end servers
and create custom spans if, for example, we
want to time how long specific back-end tests might take.
Now, let's run the code and have a look at how
we can use Cloud Trace to inspect request
latency for both incoming user requests and back-end calls.
From the project dashboard, let's go to the Products menu
and select Trace.
The trace list shows us all the traces
that have been captured in our project over the selected time
interval.
Let's switch that to the last day.
Then, let's filter for the traces
for just incoming requests for our front-end service
using the label that we set in our code.
Next, let's select a relevant trace from the resulting list.
This shows us the waterfall chart
with the parent span at the top and all
of the child spans below that.
We can see the latency for each span
in the chart to quickly determine
the source of the latency in our application overall.
To get more information, we can enable events.
These are annotations we've added in our code
to more easily understand exactly
what's happening at each span.
Finally, we can click on any of the spans to get details for it
over here on the right.
This is how we can use Cloud Trace to analyze
both the overall latency of user requests
and how they traverse our services
to find the primary contributors to latency issues.
Thanks for joining me today.
We covered the basics of distributor tracing
and saw how to use OpenTelemetry to collect traces.
We then saw how to use Cloud Trace to inspect
traces and analyze latency.
I hope this helps you keep your services fast
and your users happy.
Thanks so much for watching.
Don't forget to like and subscribe to never miss out
on more "Engineering for Reliability" with Google Cloud.
See you soon.
[MUSIC CHORDS]