YURI GRINSHTEYN: Stackdriver is a great platform
that lets developers and operators measure and improve
service reliability.
However, collecting a lot of data
will make it hard to find the information you need.
And you may be paying too much.
Today we'll get into understanding and managing
your logging and monitoring costs.
This is the Stack Doctor.
[MUSIC PLAYING]
Over the last two episodes, we reviewed
using OpenCensus and Prometheus to add
custom metric instrumentation to your services.
But what is that going to mean for your cloud spend?
Let's start by reviewing how Stackdriver monitoring is
priced.
This is the current pricing.
But make sure to check the pricing page
link below for up-to-date information.
The first thing to understand is that monitoring
is charged by the volume of data required
to store monitoring metrics that aren't generated
by GCP services.
This means that metrics from the monitoring agent or custom
metrics sent by instrumentation like OpenCensus or Prometheus
will result in charges.
The second thing to understand is how
to translate metrics to volume.
The key to that is to use a rough rule of thumb,
to treat a single time series data point as 8 bytes of data.
So if we sample a single time series every minute
for a month, that'll generate about one third
of a megabyte of data.
You will then be charged based on the total volume of data
collected for all of the metrics that are
being collected in the project.
Now that we understand how monitoring is priced,
we need to see how much data we're actually
ingesting and storing.
And we can do that in two places--
in the Billing Console and in Stackdriver itself.
First, let's start with the Billing Console.
You can access it from the Cloud Console
by going to the Products menu, selecting Billing, and then
going to Reports.
From there, you can search through just the costs
associated with the Stackdriver monitoring,
and narrow the report to the projects you
are responsible for managing.
This is a great way to understand
your historical usage and costs.
You can also see usage information in the Stackdriver
Workspace by going to the Workspace settings
and selecting Usage details.
Here, you can see exactly how much
metric data is being ingested from every project that
is being monitored by the Workspace.
Finally, metrics that show how much monitoring data is
being ingested are available in Stackdriver itself.
You can use them to do things like create dashboards
and alert on unexpected spikes in monitoring data.
Now you know where your monitoring costs
are coming from.
But what can you do to actually manage that volume
or even reduce it?
The first thing to look at may be your custom
metric instrumentation.
Frameworks like OpenCensus and Prometheus
generate a lot of metrics when you first start using them.
You may not need all of that information.
For example, the node exporter will
provide the same information you were getting
from GKE node monitoring.
Review the work you've done to add custom metrics
to your service, and determine whether all of them
are really necessary.
The second thing you should review is your use of labels.
The way that you use labels for GCP components
might impact the volume of time series
that are generated for your metrics and monitoring.
For example, you can use labels on your VMs
to appropriately report metrics to cost centers on your GCP
bill and to signify whether specific GCP environments are
production or development.
But adding these labels means that additional time series
are generated in monitoring.
If you label your virtual machines with cost center
and {} environment values, then you can calculate the total
number of time series by multiplying the cardinality
of both labels.
So if there are 11 cost center values and five environment
values, that means that 55 time series are generated.
This is why the way that you add labels
might add significant metric volume
and therefore increase the cost.
Of course, labels are helpful when
you're looking to organize your resources,
but you may be able to reduce your complexity and cost.
Finally, you can look at your monitoring agent configuration.
Metrics sent from the monitoring agent are chargeable metrics.
The monitoring agent streams application and system metrics
from common third-party applications
and additional GCP virtual machine-level metrics.
If you don't need the detailed system metrics or metrics
from the third-party apps for certain virtual machines,
you can reduce the volume by not sending these metrics.
You can also reduce the metric volumes
by reducing the number of VMs using the monitoring agent.
Now that we've covered how to understand and manage
our monitoring costs, let's move on to logging.
Let's review logging pricing.
The model here is very straightforward.
You are charged based on the amount
of volume data ingested in the project
over the free threshold.
You should know that unlike monitoring,
where some metrics do not incur any charges,
all logs ingested incur charges.
Next, let's move on to reviewing the logging charges.
As with monitoring, this information's
available on the Billing Console, the Cloud Console,
and in Stackdriver itself.
In the Billing Console, you can filter the data to just
report on Stackdriver logging to understand
the exact costs being generated by each project.
To better understand why a particular project might
be generating high logging charges,
you can see detailed information under Log Ingestion
in the Logging menu.
This view provides a service-level breakdown
of exactly where the logging charges in your project
are coming from.
To get even more detailed information,
you can select the View and Metrics explorer
for any row to get a historical view of how
these resources were generating log data over time.
You can then use this view to create charts and dashboards
to easily access this information in the future.
Now that you understand how to capture, measure, and analyze
your logging costs, let's dive into some options
for managing them.
Our primary mechanism to control log congestion
is log exclusions.
They allow you to either broadly or precisely specify
exactly which logs will or will not
make it into Stackdriver Logging and be
available in the Log Viewer.
However, you should remember that logs aren't just discarded
when they're excluded.
You still have the option of configuring log exports
to have them be sent to cloud storage for archival,
BigQuery for analysis, or Pub/Sub for delivery
to subscribers, even if they're subject to exclusions.
Exclusions and exports work together,
letting you choose which logs you
want to keep for real-time visibility
and which logs you want to archive without incurring
logging costs.
The more complex your applications,
the more monitoring and logging you'll need.
Knowing how these are priced and how they can be configured
can help you manage the resulting costs.
Thank you for joining me today.
And come back soon for more.
This is the Stack Doctor.
Stay healthy out there.
[MUSIC PLAYING]