YURI: Do you develop distributed applications, or systems made
up of lots of microservices?
If so, you've surely had to deal with issues
where users complain about the service being slow.
In a distributed system it's often
hard to figure out which dependency is causing it.
This is where distributed tracing comes in.
Recover using OpenTelemetry to capture and analyze latency
from distributed applications in the previous episode.
If you haven't watched it, we'll link to it
in the episode notes.
But in brief, we want to create traces for user requests,
and spans for the service, the service requests
whose latency we want to analyze to determine which services may
be contributing the most to the overall latency
experienced by our users.
OpenTelemetry is an open source library
that allows us to easily instrument our services,
and capture traces from.
My name is Yuri.
I'm an SRE at Google Cloud.
And today, I'm going to show you two ways
to instrument your services with OpenTelemetry.
I'll cover easily getting started
with automatic instrumentation, and using
manual instrumentation for maximum control.
Welcome back to Engineering for Reliability.
[MUSIC PLAYING]
Let's have a look at a simple application that
has a front end that serves user requests,
and makes its own request to a back end
to retrieve additional data.
We'll use this for both of our examples today.
So ideally, we want to capture the latency
of the overall request as perceived
by the user, the latency of the back end request
as seen by the front end, and how long the back end
processing takes.
The front end of our application is written in Flask,
and it uses a request library to make back end calls.
The back end is also a Flask app.
When it receives a request, it does some processing
before returning a response.
To use automatic instrumentation,
we can run both our front end and back end
using the OpenTelemetry instrument command.
The command will try to automatically detect
packages use bar code, and apply automatic tracing
instrumentation on them.
We get distributed tracing exported to Cloud Trace
without having to make any code changes.
Let's take a look.
Our front end is running locally using the OpenTelemetry
instrument command, instructing you
to send tracers to Cloud Trace.
And the back end is running locally as well
using the same command.
Let's send a request to our application
and confirm the response.
In the Cloud Trace UI, we can find the resulting trace.
It's comprised of multiple spans.
The root span is the overall request.
And we can see how long it took.
It was made to our service running a local host
on port 8080.
[MUSIC PLAYING]
The first child span is the back end request
that was also made to local host,
but on port 8081, which is where our back end
service is running.
So this is how you can use OpenTelemetry
automatic instrumentation to capture traces
across distributed services without the need
to modify application code at all.
That was really easy.
But there is a caveat.
We couldn't create a custom span to time back end processing
without adding instrumentation in the code.
So let's have a look at how you can manually
instrument your service to explicitly control
how spans are created.
We'll link to the full code sample in the episode notes.
But let's start by reviewing our front end code.
First, we need to import the necessary OpenTelemetry
libraries that will let us manage spans and context
propagation between our services.
We'll use the request context to send information
about our trace to the back end, so that it
can append its own spans to it.
Next, we configure our exporter to send traces to Cloud Trace.
Finally, we're ready to make our back end request.
We create our root span that will
serve as the parent for the subsequent spans.
Next, we create a span specifically
to time how long the back end request takes.
Lastly, we make a request with HTTP headers
that will carry the spam context to our back end.
In our back end service, we first
extract the spam context from the incoming request headers.
We then create a dedicated spam using that context to time
how long our back end processing task takes.
Now, let's run our services and inspect our trace
in Cloud Trace.
In this example, we also have our front end running locally.
But this time, we're using the basic Python command.
Our back end is also running locally using the same command.
We can confirm that our application
is responding by sending a request
and confirming the response.
Now, we can go to Cloud Trace and find the resulting
trace in the trace list.
[MUSIC PLAYING]
The trace has our root span, the back end request span, and now,
a dedicated span for back end processing.
For each of these, we can see exactly how long
they took, which allows us to figure out
exactly where the time is spent in serving the user request.
So this is how you can use manual instrumentation
with OpenTelemetry to send a request context over the wire,
and use it to a open child spans from downstream services.
Thanks so much for watching today.
We learned how to use both automatic and manual
instrumentation with OpenTelemetry for distributed
tracing of our Python services, and understand latency
in distributed application.
I hope this helps you to keep your services reliable,
and your users happy.
Don't forget to like and subscribe to never miss out
on more Engineering for Reliability with Google Cloud.
See you soon.
[MUSIC PLAYING]