[MUSIC PLAYING]
SPEAKER: I'm excited to have Priyanka Vergadia, a customer
engineer at Google Cloud, here with me
to show us how to set up service level
monitoring in Stackdriver.
Hi, Priyanka.
PRIYANKA VERGADIA: Hi, Stephanie.
Thank you for having me.
SPEAKER: Thank you.
Google has published our practices
for managing reliable services at scale and site
reliability engineering books and this SRE workbook.
One of the main principles of SRE
is using service level objectives.
Can you show us how developers can do the same?
PRIYANKA VERGADIA: Yeah.
Let's dive right in and let me show it through a demo.
SPEAKER: OK, great.
PRIYANKA VERGADIA: Perfect.
So I have this website here that is built to order some stuff.
And so I'm going to put something in the cart.
It's an e-commerce website, so it lets you order things,
and then it places an order.
So I have my credit card information in there,
and we just placed an order.
Now, behind the scenes, there is a lot going on
in this particular application.
So if you look at what's going on here,
this is the service architecture of the application.
And it has a front end.
It has various different services
talking to the front end-- the check out,
the catalog, the product catalog, the card service.
There's a lot that's going on here.
And if you see further, they're pretty much all
written in very different languages as well.
So there's a lot of complexity behind the scenes
that's going on.
And this is all done through microservices, as you can see.
So it's important to understand that a simple website can
be set up with different services in the back end
and have different languages that it can be
written in-- so complexities.
SPEAKER: Right.
And microservice architectures mean
the app itself is broken into many small pieces.
And doesn't that make it hard to understand
how all the pieces fit together and where
to start investigating when there is a problem?
PRIYANKA VERGADIA: It becomes very hard for the operations
team, especially in microservices architecture,
to figure out what's going on, which services is up,
down, talking to each other.
So there's a good tool in Stackdriver Monitoring called
Topology, where you can actually look
at the topology of your different microservices
talking to each other.
So I have it open here, and when you open the service topology,
I can look at a specific service that I'm interested in,
and then see which other services it is talking to.
So it gives me a view.
I can click on one, and it gives me a view of what's going on
and which services it is talking to,
which is pretty helpful when you're
working in an environment that has
so many different microservices that it's talking to.
SPEAKER: Wow, and this is great because,
with microservice-based apps, you can clearly
see the inter-service dependencies
and use that to double-check new code rollouts.
But let's say you're setting up alerts.
Ops staff can burn out a lot of the times
from getting notified of all the potential causes of a problem,
making it hard to align SREs with customer interests.
How can we identify which behaviors really actually
matter for that service during service degradation?
PRIYANKA VERGADIA: Yeah.
So there is also another tool within Stackdriver Monitoring
for services, where you can actually set up SLOs and define
your SLOs for the services so operations does not actually
get affected by just looking at all the different alerts that
can come by.
It's actually a practice within our SRE teams,
where we make sure that our SRE staff is only monitoring
alerts that are related to customer issues and front end
customer related issues and not every single issue that might
be a cause to those issues.
So let's see that in the demo as well.
SPEAKER: Great.
PRIYANKA VERGADIA: Yeah.
So if we click on Services here in the Stackdriver Monitoring,
let's look at front end.
And when we go do that, you're able to see the service level
optimization.
So let me click on SLOs.
And in SLOs, you obviously see nothing here.
So we're going to set one up and say,
we want to set up for availability,
and then we can do it rolling or calendar.
I want to say, in the month, if we reach 99%,
or low availability, then I want to get alerted.
That's a good way of saying that you're not
going to be alerted for everything
but you still want to be alerted for when we go less than 99%.
And then you can also set up a latency alert.
So in this case, if we say latency--
and I may want to set up differently
for different services.
For the front end, you could say,
my pages should load in less than 1,000 milliseconds,
otherwise I want to be notified.
And in this case, also, let's say it's 90% our goal,
and then we submit that.
So these are the two alerts that we just set up
for latency and availability.
And they are not just thresholds because we
are saying this percentile.
So it's not going to be chatty and noisy.
It's going to be, if this happens,
then we really want to get notified because the service is
not working fine.
So that's a good way of identifying, making sure
that you're not burning out your operations staff.
SPEAKER: Excellent.
How do you follow these best practices given the resources
that you have?
PRIYANKA VERGADIA: Yeah.
So you also have metrics and diagnostics in there.
So if you click on Metrics, you're
able to see the request counts and the response latencies,
which will guide you through the request and response.
And if you click on Diagnostics, that actually
takes you to the Trace.
So now, if I want to drill deeper
into the 95th percentile, which is probably
the one I want to look at because it's not performing
very well, then I can click on that
and see a specific request, why it was slow, for what reasons,
and which requests were actually slow.
So in this case, we can see that the checkout service was
a little bit slow in send and receive,
so we may want to delve deeper into that
and find out what's going on there.
So the Diagnostics tab and the Metrics tab
are really good friends for troubleshooting.
SPEAKER: Great.
And, at some point, you probably need
to dig a little deeper into the services signals.
Let's say, maybe you received an SLO alert
and there's no obvious upstream cause,
or you have a customer complaint outside of an SLO alert
that you need to investigate, or maybe you
need to see how the new rollout of your code version is going.
How do you get a coherent display of all the signals
for a specific service?
PRIYANKA VERGADIA: Yeah, so I think the best
thing to do at that point is--
you got an alert through SLO.
You want to look at metrics and diagnostics
to kind of figure out what's going on
because metrics and diagnostics are going
to be helpful to understand which specific service is not
performing well.
As you can see in diagnostics, when we drilled deeper
into a specific service, we were able to find out
that checkout is the culprit.
And so we can drill deeper into that.
So those metrics and diagnostics would be your great friends
in order to find out what's actually going on.
SPEAKER: So Stackdriver service monitoring
gives you a whole new way to view your application
architecture, reason about its customer-facing behaviors,
and get to the root of any problem that arises.
So thank you so much, Priyanka, for showing us
how Google is transforming the ops experience of cloud native
and microservice deployments.
PRIYANKA VERGADIA: Thank you for having me.
It was fun.
[MUSIC PLAYING]