in our last episode we covered using
distributed tracing with open telemetry
to identify the sources of application
latency but what do you do once you know
which microservice is causing the issue
or what if your service is serving an
elevated error rate what do you do then
this is where logging comes in today i'm
going to show you how you can use cloud
logging to ingest route store and view
logs from your services and use them to
fully understand what your service is
doing welcome to engineering for
reliability with google cloud
but first what do we mean when we say
logs we use logs to specifically
describe an event that takes place in
our system logs are written by our code
by the platform our code is running on
and the infrastructure we depend on
because logs and modern systems are the
descendant of and sometimes still are
text log files written to disk
a log entry analogous to a line in the
log file is the quantum unit of logging
an entry will generally consist of a
time stamp that indicates either when
the event took place
or when it was ingested into the logging
system and the payload either as
unstructured text data or structured
data most commonly in json
logs can also carry associated metadata
like the resource that's writing them
the log name and the severity for each
entry
in google cloud logs originate from
multiple sources
first google cloud services emit logs
for both auditing and troubleshooting
purposes these are enabled by default
and cannot be disabled directly
the infrastructure you provision and
apps you run in google cloud can also
emit logs
these are largely controlled by you as
the user
finally networking components and
services emit logs as well some of these
like bpc flow logs can be controlled by
the user while others are considered
part of the platform and cannot be
disabled
no matter where the logs come from cloud
logging receives log entries through the
cloud logging api from there they pass
through the log router the syncs in the
log router check each log entry against
existing inclusion and exclusion filters
that determine if the log entry should
be sent to storage destinations
including cloud logging buckets or
excluded entirely from ingestion
you can use syncs to route logs to
multiple destinations such as google
cloud storage for archiving bigquery for
analytics or pub sub to be processed
further by services such as cloud data
flow or consumed by third party systems
this is also where exclusions happen if
you have any configured each sync lets
you enter one or more exclusion filters
which let you exclude matching log
entries from being routed to the sync's
destination typically these are used as
a cost optimization approach
next logs are sent to log buckets which
act as containers in your google cloud
projects to store and organize your logs
data
the logs that you store in cloud logging
are indexed optimized and delivered to
let you analyze your logs in real time
note that these are different storage
entities and cloud storage buckets
you can create syncs to route all or
just the subset of your logs to any
locked bucket
this flexibility allows you to choose
the cloud project in which your logs are
stored and what other logs are stored
with them
once your logs are in buckets you can
use log views to control who has access
to the logs within your blog pockets you
can centralize or subdivide your log
storage based on your needs
custom log views provide you with a
granular way to control access to the
logs in your buckets
for example consider a scenario in which
you store all of your organization's
logs in the central project
because log buckets can contain logs
from multiple projects you might want to
control which projects different users
can view logs from
using custom log views you can give one
user access to logs only from a single
project and another user access the logs
from all the projects
so far you've seen how platform
application and networking logs are
ingested through the cloud logging api
and how the logs router sends them to
their destinations either in log buckets
or log syncs
next let's have a look at how you can
view and query logs in cloud operations
starting with the logs explorer
to access the logic explorer go to the
products menu
from logging select logs explorer
initially this shows you all logs in
your project
on the left side you can use log fields
to select just the logs that you care
about
for example if you have a service
running on gke and you want to figure
out what errors it's generating select
kubernetes container as the resource
the name of your container and error as
the severity
this will filter all the logs down to
just error logs being produced by your
service
the histogram shows you the volume of
logs that matches your query over the
selected time interval
if you want to zero in on a specific
point in time just drag the selectors to
the time you care about and rerun the
query
logs that match your query are shown
here in the query results section to get
the full details of a log entry select
it this will show you both the metadata
and the payload
to see other logs that have the same
payload or any other field click a field
and select show matching entries to
rerun the query for those
the query is shown up here
you can edit it directly and you can
save it to be used later
save query show up here
logs explorer even generates suggested
queries for you based on what's
happening in your project
this is how you can use the logs
explorer to find the exact log entries
you need to determine what's happening
with your service
the logs explorer is great at helping
you find the logs that you need when you
need them but what if you're debugging a
service and want to watch logs in real
time this is where log streaming comes
in let's have a look
in order to stream logs you first have
to find the logs that you care about you
can do that by writing a query or by
using the log field selectors so let's
say you want to troubleshoot a cloud
composer workflow select cloud composer
environment on the left hand side
once you've run the query you have the
option to stream logs click that button
and the query results section will
expand to make it easier to see and
select log entries
the logs are now streamed with the
newest entry showing up at the top of
the list to pause the stream so that you
can inspect the particular entry just
scroll down
then select an entry just as before
once you're done with that entry you can
restart the stream and you'll see the
most recent logs at the top of the list
again if you have the information you
need stop the streaming get back to the
query
so that's how you can use log streaming
to see the log entries as they appear
essentially in real time
streaming logs on the console is
fantastic but what if you prefer to use
the command line well there's now a g
cloud command you can use to do just
that let's have a look at that
to stream logs from the command line
simply use the gcloud alpha logging tail
command
use a filter to get just the logs you
care about you can use the query editor
to get the filter correct
when you enter the command the tail
session will be initiated and the logs
will start streaming into the console
as you can see this can be hard to read
so you may want to redirect this to an
output file to make it easier that's how
you can use the command line to stream
logs by using the gcloud alpha logging
tail command
thanks for joining me today we covered
the basics of cloud logging how logs are
ingested routed and stored how you can
query logs from log buckets using the
logs explorer and how you can stream
them in real time using the console and
the command line i hope this helps you
find issues faster and keep your
services reliable and your users happy
thanks so much for watching don't forget
to like and subscribe to never miss out
on more engineering for reliability with
google cloud see you soon