SPEAKER: In our last episode, we covered the best practices
for deploying and using cloud operations in an enterprise
environment.
But we still left some questions unanswered.
How should you monitor your services?
How should you deal with alerts?
And what about managing costs?
Today, we'll look at best practices
that you can use to get the most value out of cloud monitoring.
Welcome to Engineering for Reliability with Google Cloud.
Just like last time, let's first figure out
why monitoring is important at all.
What are we trying to achieve when we use cloud monitoring?
We're after three main objectives in parallel.
Our first objective is to keep our users happy.
That means we need to know whether our service is
meeting their needs.
And the best way to do that is through well-defined SLOs
and error budgets, and alerts that
let us know when our service is burning error budget.
Our second objective is to be able to quickly and effectively
respond when those alerts do fire.
This means having enough data about the state of our service
to be able to triage incidents, identify effective mitigations,
and accurately confirm the effectiveness
of those mitigations.
Our final goal is to do the first two while managing
the associated costs.
Now that we've covered the whys, let's turn to the hows.
We'll start by talking about what to monitor.
From there, we'll get into how to best manage alerts.
And finally, we'll cover cost management
for cloud monitoring.
Let's get into it.
Let's start by talking about the obvious first question.
What should you be monitoring?
For the answer, we can turn to chapter 6
of the "Site Reliability Engineering" book on monitoring
distributed systems.
Fundamentally, the objective of monitoring
is to answer two questions.
The first one is all about symptoms.
What are our users experiencing, and is that experience
in any way less than ideal?
The second question is all about causes.
What's causing the symptoms we've identified?
That cause may be direct or intermediate,
but it's how we start to determine
how to restore user experience back to the desired state
if it's degraded in some way.
We answer the first question by clearly defining our service,
identifying good service level indicators, or SLIs
that tell us, as accurately as possible,
how well our service is able to meet our users' expectations,
and defining good service level objectives, or SLOs, that
set a target that our SLIs have to meet.
We've covered defining services, SLIs, and SLOs
in cloud monitoring extensively in the series,
but it's always worth revisiting these topics.
The first thing to answer to the question of what should we
monitor is, how well is your service
meeting the needs of your users as expressed by good SLOs?
The next thing to make sure you're monitoring covers
is what the "SRE" book refers to as the golden signals.
Traffic is essentially a measure of how much
user activity the service is responding to, usually
as a count of incoming requests for user-facing services
or data elements for data pipelines.
Error rate is a measure of what percentage of the time
the user activity is not successful,
which, of course, depends on the service and user expectations.
Latency is a measure of how long the service takes
to service a request.
And finally, saturation is a measure
of how much of the available capacity
the service is using at a given time.
A great way to ensure that you have
easy access to these signals is to create a dashboard
like this.
It shows the four golden signals for a given service at once.
It makes it easy to correlate, for example, increases
in inbound traffic to latency degradations or increased
capacity requirements.
You could automate the creation of dashboards
like this for your services.
We'll link to some examples in the episode notes.
Finally, you may need to augment your monitoring
with custom telemetry to capture service-specific information.
We covered doing this in a previous episode.
Review that for an introduction to instrumenting your services
with OpenCensus and Prometheus.
So that takes care of the first question,
what you should be monitoring.
We've just covered SLOs, golden signals, and custom telemetry,
which brings us to alerting.
How should you configure alerts?
What should you alert on?
And how should you respond to alerts?
Let's take a look.
Let's again return to the monitoring chapter of the "SRE"
book.
These four criteria should be applied
to alerts that go to pagers.
Every time the pager goes off, we
should be able to react with a sense of urgency.
But we can only react with a sense of urgency
a few times a day before we become fatigued.
Every page should be actionable.
Every page response should require intelligence.
If a page merits only a robotic response,
it shouldn't be a page.
And ideally, pages alert us about a novel problem
or an event that hasn't been seen before.
That means we need to take action
to prevent pages from reoccurring once we've
mitigated the triggering condition
and, hopefully, identified the root cause.
Alerts that don't meet these criteria should not be pages.
Alerts that aren't actionable are of no use at all.
Alerts that require human intervention
but that aren't urgent should be filed as tickets instead.
Cloud monitoring supports multiple notification channels.
And this allows you to set up alert delivery that
meets your needs.
For example, you could use email or PagerDuty delivery
to trigger pagers.
Then configure a webhook or Pub/Sub integration
to file tickets for non-urgent alerts.
You can also automate the management of your alerts
by using an automation tool like Terraform.
This helps to ensure consistency and avoids human errors
in alert configuration.
Finally, it's important to understand
the tools you have available to capture and optimize
your monitoring costs.
Let's have a look at those now.
First, it's important to understand
how you incur monitoring costs.
Platform metrics emitted by Google Cloud Services
are always free.
You are charged for other metrics,
such as custom metrics or those written by Istio,
based on storage volume.
Refer to the Pricing page for the latest
version of this information.
The Monitoring Settings page shows you exactly how much data
you are ingesting for each project in your metrics scope.
You can measure your usage by using billing reports
and filtering them to just the monitoring service
to see exactly how much monitoring charges you
are incurring in your project.
You can even figure out exactly how much
data is being ingested by each custom metric
by creating a chart and using the Metric Bytes Ingested
metric.
These tools do a great job of helping
you understand the sources of monitoring costs.
But what can you do to optimize those?
Let's have a look.
The way that you use labels for cloud components
might impact the volume of time series
that are generated for your metrics and monitoring.
For example, you can use labels to appropriately
report metrics to cost centers on your bill
and to signify whether specific environments are production
or development.
Adding these labels means that additional time series
are generated in monitoring.
You can calculate the total number of times series
by multiplying the cardinality of all labels.
For example, if there are 11 cost center
values and 5 environment values, that
means 55 time series are generated.
This is why the way that you add labels might
add significant metric volume, and, therefore,
increase the cost.
We recommend carefully limiting the number of labels
and avoiding label values with high cardinality.
Metrics sent from the Cloud Ops Agent
are another source of monitoring costs.
The agent collects app and system metrics
from common third-party apps and additional VM-level metrics.
If you don't need them for certain VMs,
you can reduce the volume by not sending these metrics.
You can also reduce the metric values
by reducing the number of VMs that use the agent.
Custom metrics are chargeable metrics
created by using the monitoring API or integrations
like OpenCensus to monitor any metrics that
are user instruments.
The more services that you instrument to send metrics,
the more custom monitoring metrics are generated.
If you want to reduce metric volumes,
you can reduce the number of custom monitoring metrics
that your services send.
Thanks for joining me today.
We've covered best practices for setting up and using
cloud monitoring, including what to monitor,
how to manage alerts, and how to optimize your monitoring costs.
I hope these recommendations help you keep your users happy.
In our next episode, we'll tackle best practices
for using cloud logging.
Thanks so much for watching today.
Don't forget to like and subscribe to never miss out
on more Engineering for Reliability with Google Cloud.
See you soon.
[MUSIC PLAYING]