[Music]
welcome to our series on ollie in depth
regionalizing your logs with google
cloud
i'm mary a product manager with cloud
logging one of my favorite parts of
working in the logging space is that
everybody needs logs
gaming retail finserv telecom government
sectors everyone who runs software has
logs now the need for these logs can
vary a lot
but a common theme is that everyone
needs to bring logs from multiple
applications vms devices and possibly
locations in the world together to get
the whole picture
cloud logging can help with this
offering a fully managed solution to
centrally store analyze and get insights
from your logs at scale
today we're going to focus on how to
keep blogs in cloud logging in a
specific geographic region for
compliance
we have lots of regions to choose from
as part of google cloud
so getting started is just two simple
steps we need to create a regionalized
log bucket and point all the log syncs
to this new regionalized log bucket
it can take less than 60 seconds to walk
through setting up a new project to keep
logs in a specific region let's take a
look
for step one we're going to go to the
cloud console
we need to create a new regionalized log
bucket
go to logging
and specifically within logging we'll
choose log storage
i'm going to go ahead and create a log
bucket
i need to choose a name
call this regionalized logs
optionally i can add a description and i
need to choose a region this cannot be
changed later so it's important that i
get it right
i can choose from many regions around
the world let's say for right now that i
want to keep logs in the united states
to comply with fedramp
but i also want to choose u.s west one
because it's a low carbon region
go ahead and select next the retention
period can be changed later i'll go
ahead and leave it at 30 days for right
now but i could choose any retention
period from one day up to 10 years
great so now we have a regionalized
logbucket but there's nothing in it
so next we need to go ahead and redirect
a logsync
to point logs into our new regionalized
bucket
the easiest way to do this is simply to
redirect our default log sync
to point to our new regionalize bucket
so i'm going to go ahead and edit the
sync
and instead of selecting our default
bucket i'm going to go ahead and select
our regionalized log bucket
i'm going to leave everything else the
same including which logs will go into
the sink
and that's it
go ahead and save that
and go back to our log router so that we
can see that everything has been saved
and indeed logs will that arrive from
now on will go to our regionalized log
bucket
that's it
regionalization in just two steps and
all of this can be automated with
terraform in addition to being performed
through the cloud console as well
all right so just to recap we created a
regionalized logbucket and sent our
logsync to this new regionalized lock
bucket instead of the global bucket
pretty easy right
let's take a behind-the-scenes tour of
cloud logging and i'll tell you about
how the engineers who build cloud
logging work really hard to make
compliance as painless as possible for
you staying for the tour
awesome let's start with the life cycle
of the log entry there are four stages
to the lifecycle of a log and our
engineers examined each step to make
sure it meets your data residency
requirements we have log generation and
collection which is where the logs are
generated and sent to the logging api
log processing and routing log storage
and retention and log analysis and
access control before we examine each
stage let's look at the system
architecture diagram to get the big
picture
first a log is generated by an
application it could be an application
that you developed or a third-party
application like sap or nginx or a cloud
service like bigquery
that log entry will be sent to the
google cloud logging api at
logging.googleapis.com
it most likely will be sent using one of
several open source logging agents that
we support including fluentd fluentbit
or stanza
but it might also be sent using client
libraries or if you've got some really
special requirements maybe by calling
our api directly
from here it gets processed by the log
router where it's evaluated against the
log syncs like you saw earlier when we
redirected the logsync to point to our
regionalized log bucket
based on the log syncs you can send it
to storage within cloud logging or you
can send it to bigquery cloud storage or
pub sub where you can discard the log
entry if you don't want it
the log router also processes the logs
aggregating them into logs based metrics
or alerts
assuming that the log entry matches a
log sync with a destination in a log
storage bucket it will then be indexed
and stored in our storage system
optimized for log analysis
which brings us to the fourth and final
stage of the log lifecycle which is
querying or analyzing logs this usually
happens via the cloud console ui but can
also be done via gcloud or api logs that
are not written to a log bucket will not
be available for querying through cloud
logging even if they were sent to a
destination like bigquery or cloud
storage
great so now that we've got the big
picture let's go through how each
section works and what it means for data
residency for logs
as we said earlier logs can be generated
from applications running anywhere in
the world within google cloud or without
we wanted to make sure that sending logs
is easy rather than forcing you to
choose a region for logs for every
single application or deployment
so we offered a global endpoint for
writing logs
now usually a global endpoint on a cloud
api would mean that we'd send logs to
the best regional endpoint based on
factors such as latency availability
load etc but we also wanted to make
compliance the easy default so we did
something special with our global api
and we region locked it what that means
is that if you send logs to the global
endpoint from anywhere within google
cloud we'll keep those logs in the
region the requests came from
so logs from an app running in us west
one will stay in us west one
and uh logs that are
generated in europe west one will stay
in europe west one and so on
if you're sending logs from on-prem or
another cloud for example or you want to
override that default of processing the
logs in the region in which your app is
running you can explicitly call the
regionalized api endpoint when writing
the logs instead of the global api
endpoint so you have complete control
over where your logs are processed
so to summarize default compliant but
you have control if you want it pretty
cool right
let's go to the next stage and see what
happens after the logs are received by
the api
the logs now enter the log router where
they're processed and sent to whatever
destinations you choose
this is my second favorite part of the
process because it's where some google
magic happens
if you've run your own logging pipeline
at scale you may have needed to run
something like kafka in front of it to
handle buffering and processing the log
router though is built on top of google
pub sub which reliably receives and
processes the logs eliminating the need
for you to run your own buffering
pipeline
each log entry is matched against a set
of rules called log syncs which you
control
all of the processing is going to stay
in the region in which the logs were
received
but what if you want to centralize logs
from multiple regions together for
troubleshooting or maybe analysis and
bigquery
you control the destinations of the logs
so even though the logs were received
and processed in europe west 2 or even
maybe in a u.s location you can still
store them in a log bucket in europe
west one
this makes it really easy to reason
about compliance without worrying about
what region processed the logs
next we'll move on to storing logs in
cloud logging using log buckets
i sometimes get asked if a log bucket is
the same as a cloud storage bucket
they're completely different
technologies we use a special logging
storage system called t-cube which
stands for time tuple table it's an olap
system optimized for managing logs at
scale
fun fact we use tq both for managing
your logs as well as for managing logs
for debugging applications within google
like gmail and google search
so tq processes over two and a half
exabytes of logs per month which is a
pretty incredible scale
we've deployed t cube in every region so
when you create a log bucket we keep
your logs in t cube in that region
pretty simple
there's a small catch though can you
spot it
i'll give you a hint it's on this
diagram on the left hand side
you can see two log sinks on this
diagram of the log router the required
and the default log sync that go to
matching log buckets and if you look
closely you'll notice a lock next to the
required log bucket
if you were paying close attention
during the demo at the beginning you may
have noticed that the required bucket is
also global
which basically means that google
manages the region for you
wait what's up with that
don't worry we've got a good answer
the required log sync exists to capture
logs that are generated by google and
about your architecture not your
application
these logs are critical for security
purposes which is why we require them of
all projects
let's take a look at the logs that are
sent to the required bucket
there are three types of audit logs that
are included admin activity system event
and access transparency
admin activity answer the question of
who took what action on a google api
for example a specific service account
granted access to a cloud storage bucket
another example would be that alice
created a new gce vm or the bob deleted
a firewall role
they wouldn't include read or write to
access like carl reading a gcs file you
can see why these types of actions are
really important to capture for security
next we have system event logs
these are when an automated service
within google cloud takes an action like
migrating a vm during maintenance kind
of boring
finally we have access transparency
audit logs
these won't capture if anyone in your
org reads or writes or changes data but
they will tell you if someone at google
accesses your data as someone who works
in cloud and supports customers let me
assure you that we take data privacy
really really seriously but there are a
few legitimate reasons we might access
your data like if you call support and
ask us to troubleshoot an issue for you
in this case you'd see a log entry with
a support ticket so you'd know who from
google did what again important but not
particularly sensitive
since none of these are customer
generated logs we generally advocate
that this isn't subject to data
residency and that keeping it in a
global bucket is okay but it takes
enough time for us to explain it that
we're working on some extra org level
controls to let you choose what region
this required bucket should be located
in as well so check back soon for this
in the future
that reminds me that we do have org
policies to govern what regions new log
buckets can be created in to help manage
data residency across your organization
and finally we're at the end of the log
life cycle with log analysis and access
control i told you that the log router
was my second favorite part
this is my favorite part because we help
solve the problems that your users care
about without letting data residency get
in the way let's take a look at the
cloud console
let's say i'm now a developer rather
than a logging administrator i want to
focus my time on my application not my
ops tools i love the cloud logging makes
it easy to search my logs from in
context if i'm in the gke console or
troubleshooting something in cloud run
or from other places within the gcp
console
but my cso says we need to keep logs in
the united states
i hope everything just works as usual
i'm going to start by searching for an
arbitrary string across all the logs in
my project let's see what i get
right i can see a bunch of matching logs
that's good but how do i know whether
these are being kept in the us it just
worked which is great
suppose i want to dig more deeply to
understand what's going on behind the
scenes
right now i'm searching across all logs
that are coming into this project but
instead i can choose logs that are going
and are being stored into
my default or global bucket
there shouldn't be any logs there let's
check
great no matching logs
let me instead select logs that are
going into my regionalized log bucket
which i can see is in us west
one and i see the matching logs and can
filter down and use them just as usual
well i'm not sure that we succeeded in
making data residency fun but at least
it was easy and i can focus on my
application
that's it
i hope you enjoyed our detailed tour on
how regionalization for cloud logging
works we're continuing to work on
expanding controls while striving to
make compliance as painless as possible
if you have any questions requests or
feedback please visit our community page
where google product experts and folks
from the community are answering
questions
search for google cloud community and
cloud operations happy logging
you