SPEAKER: Welcome back to another episode
of "BigQuery Spotlight."
Today, we're talking about BigQuery data governance
so that everyone in your organization
can easily find and leverage the data they
need to make effective decisions,
while minimizing overall risk and ensuring
regulatory compliance.
Let's jump in.
[MUSIC PLAYING]
The term data governance is thrown around quite a bit,
but what does it actually mean?
Data governance is everything you
do to ensure your data is secure, private, accurate,
available, and usable inside of BigQuery.
For us, governance starts during data onboarding.
Let's say your team has received a new request.
Someone from your e-commerce department
wants to add transactions data into a new BigQuery table.
The first step in ensuring governance
is understanding the data.
For example, figuring out where the data is coming from
and what information it contains.
In this case, the transactions will
be extracted from a database, and you already
know the schema.
So let's start by creating a new blank BigQuery table.
Next, you'll need to make sure that sensitive data is
protected.
To do so, you'll want to categorize the information
in your new table.
For example, marking specific columns
as having personally identifiable information.
You might remember that tables can have labels.
While labels are a great way to easily categorize
BigQuery objects, they are a bit limited,
and can't be used for specific columns.
Lucky for us, BigQuery has deep integrations
with Data Catalog, Google Cloud's data discovery
and metadata management service.
Data Catalog automatically tracks technical metadata
for BigQuery assets.
So we can already find our new BigQuery table,
and see things like column names, descriptions and when
it was created.
But the really cool part is that you
can create schematized tags that act as annotations
to capture metadata.
We've already created a tag template
that tracks data governance information, so
now let's fill it in and attach the tag to the Email column.
Great.
Now we're tracking that this entity contains customer email
addresses.
But you might also notice there's a Comments column
that stores freeform text customers have
added to their order.
There's a chance that some of these rows
contain more sensitive information.
This is where DLP comes in.
DLP, or Cloud Data Loss Prevention,
is Google Cloud's service designed
to help you discover and protect your most sensitive data.
With DLP, you can create an inspection job that scans
comments for email addresses.
DLP can scan data that lives in Data Store, Google Cloud
Storage, or BigQuery, and can send you
an email with the scan results, save the results of that
to a table inside of BigQuery, or even write the results
directly into Data Catalog.
Many BigQuery users choose to leverage an orchestration
tool, like Cloud Composer, to build
data onboarding pipelines.
For example, you might load your transactions data
in a locked down staging project inside of BigQuery
and kick off a DLP scan when new comments are added.
The results will be listed in the staging table's metadata
inside of Data Catalog.
If the scan shows that any sensitive information is
included in a comment, then you can also
kick off a D identification workflow,
where DLP replaces PII with something
like an asterisk or a hash symbol.
Then, the new data is finally appended to the production
BigQuery table.
Now that you've de-identified and classified your data,
you're ready to design access policies.
To understand data access in BigQuery,
you need to familiarize yourself with Identity Access
Management, or IAM.
If you haven't already watched our video on BigQuery IAM,
I recommend pausing and taking a look.
We've linked it in the comments below.
Great, so on to data sharing.
When it comes to granting access to BigQuery data,
many administrators choose to grant
Google Groups representing your company's different departments
access to specific data sets or projects so policies
are simple to manage.
Say you have a project that stores your organization's
e-commerce data.
You might give the e-commerce team the BigQuery Data Viewer
role in this project.
Now, if someone on the e-commerce team
needs data from a different team,
like the product development department,
you may be inclined to just grant them
access to that data set.
Instead, you could use an authorized view,
or an authorized UDF, so that your e-commerce analyst can
get limited access to product data
while still keeping more manageable team to project
permissions.
So this all sounds great, but what about those PII columns
that we flagged before in Data Catalog?
You definitely don't want the e-com team
to have access to those, unless they have special clearance
from security.
With BigQuery, you can control access
to specific columns using policy tags from Data Catalog.
Here, we've created a taxonomy, which
classifies the level of clearance needed to access
different types of data.
Anyone who needs access to our highly sensitive data
should be added as a fine-grained reader
to the high resource.
So in our case, we'll just add the e-com PII clearance group.
Next, we can apply these as policy tags
to columns in our BigQuery table,
marking the columns with PII as highly sensitive.
And to take it one step further, maybe you
don't want all e-commerce folks to get access
to the transactions data.
Instead, each analyst should only
be able to view transactions for the product
category they work on.
You can control access to specific rows
in BigQuery using a row-level access policy,
like the one we're showing here.
Between storing data in different containers
and leveraging authorized views, UDFs, policy tags,
and row-level access policies, you
have a lot of options for sharing data
in a secure manner.
Check out the documentation below for some more guidance
on pros and cons and general best
practices for policy access.
OK, so we've onboarded our data set and we've shared it.
The last step we'll talk about is ongoing monitoring,
including ensuring that data is accurate.
This is where data quality comes in.
There are lots of different tools
that allow you to declare quality validation
tests by writing custom assertions using SQL.
For example, you may want to check and make sure
that you don't have any duplicate entries
in your table.
We've actually created an open source framework
for data quality validation that's
included in the comments below.
And keep an eye out for Data Form,
our tool for developing SQL pipelines,
which will be open to new customers soon.
Just like we mentioned before, you
can use an orchestration tool like Cloud Composer
to programmatically run these tests
and alert you if something seems wrong.
You can also use the Data Catalog API to make sure
that quality metadata is added to tags
and is easily discoverable.
Aside from monitoring quality, you
might also want to monitor who's accessing the data
and how they're using it.
Well, you're in luck.
In a few weeks, we're diving into monitoring your BigQuery
deployment, so be sure to tune in.
And remember, stay curious.
[MUSIC PLAYING]