[MUSIC PLAYING]
LOU DE LA TORRE: Hello, everyone.
Welcome to the Google Cloud Video Learning Series.
My name is Lou De La Torre.
I'm a customer engineer at Google Cloud.
And today I'll show you how to export
logs from Google Cloud Logging to BigQuery.
Here I am in the Google Cloud console.
You can quickly find the BigQuery service
by typing BigQuery in the Search Products and Resources field
and then selecting BigQuery.
In the Navigation pane in the Resources section,
select your project.
On the right side of the window in the Details panel,
click Create Dataset.
In the Create dataset window under Dataset ID,
enter a unique name.
In Data location, accept the default, which is US.
In Default Table Expiration, select Number of Days
After Table Creation, and I'll enter 7.
Under Encryption, accept the default Google-managed key
and select Create Dataset.
Next, we'll want to turn on audit logging for all services.
By default, data access audit logs
are disabled except for BigQuery.
In order to enable all audit logs,
we'll need to first download the current IAM policy as a file,
then add the audit log policy yaml object
to the current policy file, then update our project
with the updated policy.
I'll demonstrate how to configure your project's data
access audit logs using the gcloud command and Google Cloud
console.
First, open the cloud shell by clicking on Activate Cloud
Shell.
Then read the project's IAM policy and store it in a file
by running this command.
Click on Authorize Cloud Shell if prompted.
Then edit the policy file stored in temp/policy.yaml using VI,
and add the data access audit logs configuration.
Be sure to preserve the bindings and ETag
sections without changes.
Otherwise, you might cause your project to become unstable.
Save and exit the file, then write the new IAM policy
to the project by running this command.
Next, we'll need to refine the logging filters to export audit
logs, including admin activity and data access audit logs.
You could also enable logs for specific resource types
if you like.
We'll use the gcloud logging sinks create command
to create a sink called gcp_logging_sink_bq
for the project by running this command.
In the service account entry, return from the API call.
The identity is included in the response.
This identity represents a Google Cloud Service account
that has been created for the export.
Until you grant this identity write access
to the destination, log-entry exports from the sink
will fail.
So let's go ahead and copy the identity.
To add the permissions to the service account,
select the data set.
In my case, gcp_logging_export, and select Share dataset.
In the Add members field, paste the service
account that we just copied.
Then select a role.
Select BigQuery and BigQuery Data
Editor to provide the permissions necessary.
Click Add, and click Done.
After you create the logging export by using this filter,
log files begin to populate the BigQuery dataset
in the configured project.
You can now run a broad range of queries against the audit
logs to better understand who's accessing data on Google Cloud,
or you can use a tool like Google Data Studio
to help you visualize the data.
Well, that concludes the demo.
Please stay tuned for future videos
on how to use other components of the Google Cloud Operation
Suite as well as other services.
Have a great day.
[MUSIC PLAYING]