SPEAKER: In our last episode, we started
looking at best practices that you
can use to get the most value out of Cloud Logging.
We covered how to ingest and store logs,
how to centralize them, and how to manage log access
and retention.
Today, we'll wrap up this topic by looking
at how you can use advanced log queries to find
the exact logs you're looking for
and how to manage logging costs.
Welcome to "Engineering for Reliability with Google Cloud."
Let's start with using logs for troubleshooting and debugging.
I'm going to show you how to use advanced queries to quickly
find the logs you're looking for,
and how to use error reporting to automatically
identify, classify, and track application crashes.
An advanced log query is the Boolean expression
that specifies a subset of all the log
entries in your project.
You can use them to choose log entries from specific logs
or services, specify time ranges,
match specific metadata or user-defined fields,
and even sample logs.
Let's have a look.
From the Project Dashboard, let's go to logging and select
Logs Explorer.
In previous episodes, you've seen
how to use the log's field explorer to, for example,
select air logs being emitted by our front end service running
on a GKU cluster.
Notice that our selections are actually
converted into a query.
We can modify the query directly without using field selections.
For example, we can negate one of our expressions
to look for all logs that aren't errors.
If we want to only get the logs for a particular time interval,
we can easily specify that using the Time Stamp field.
We can even retrieve just a sample of the logs
by using the Sample function.
If this is a query we'll need later,
we can save it, and then access it
whenever we need it in the future from the Saved Queries
tab.
So you can use advanced queries to quickly and easily get
the logs that you want.
Saving your queries allows you to get back to a set of logs
you need to review on a regular basis.
Advanced log queries are a great way
to quickly find the exact logs you're looking for.
But what if this was even easier?
You may remember that we covered error reporting back
in episode 9.
If you haven't seen it, no worries.
We'll link to it in the episode notes below.
Just remember that it's an easy way to track, analyze, and get
alerted on errors in your application.
Advanced log queries and error reporting
are really useful if you're trying to understand
the behavior of your services.
Being able to quickly find the logs you're looking for
and easily track application crashes
should definitely help you keep your services healthy.
Finally, let's get into managing logging costs
which is a key part of optimizing your Cloud spend.
I want to show you how to view your logging usage,
how to track your logging costs over time,
and how to control the logs that you ingest and store.
First, let's have a look at how to use information in the Cloud
Console to understand how much of your bill is due to logging
and how much logging is being ingested and stored
in your project.
From the Billing home page, let's go to Reports.
Here, we can select our project and the Cloud Logging service.
This shows us the logging charges
we've incurred for the current month
or for another time interval.
If we want to understand where these costs are coming from,
we can go to logging log storage and look at the usage
details for our default bucket.
This takes us to Metrics Explorer
and shows us the volume of logs ingested
by the various resources in our project.
So that's how you can use the billing report
to stay on top of your logging costs and the utilization
metrics to track where the logs are actually coming from.
Understanding your logging costs is
key to being able to manage them.
But actually controlling those costs,
is primarily accomplished by reducing the amount
of logs you ingest and store.
We covered log retention in our last episode.
Remember that you are not charged
for storing logs for the default retention period.
As such, you can reduce your log storage costs
by carefully considering whether you
need to retain logs in Cloud Logging for longer
than a default retention.
If you do need those logs or other use cases,
consider exporting them to Google Cloud
Storage for archival or BigQuery for analytics.
Your primary mechanism to reduce ingestion costs
are log exclusions.
Exclusions are configured when you set up log sinks.
If you're looking to reduce the volume of logs being ingested,
you can modify the default sink to add an exclusion filter.
Remember that excluded log entries
won't be available in the Logs Explorer or Cloud Debugger.
Log entries that aren't routed to at least one
log bucket are also excluded from error reporting.
But with user-defined, logs-based metrics,
our computers can log entries in both included and excluded
logs.
Of course, the other way to reduce log ingestion costs
is to produce less logs from your applications.
For example, you can carefully adjust the log levels
to control log volume.
This is especially useful for serverless services,
where there's no way to view excluded logs.
Consider both managing log volume in your application
and using exclusions to balance observability and costs.
Thanks for joining me today.
We covered best practices for querying logs in Cloud Logging
and managing your logging costs.
I hope these recommendations help
you keep your service reliable and your users happy.
Thanks so much for watching today.
Don't forget to like and subscribe to never miss out
on more "Engineering for Reliability with Google Cloud."
See you soon.
[MUSIC PLAYING]