EVREN ERYUREK: Hello everyone.
Welcome to our session on data governance.
As some of you may remember, back in November in London
we had announced our partnership,
and today I'm delighted to be with Jim again
to talk about our progress to date.
Let's take a look at recent Goldman Sachs reports.
Security breaches, data loss, legal and compliance issues
are slowing down many in their move to public clouds.
Fundamentally they're worried about loss of control.
When you look at BigQuery it was designed as a cloud data data
warehouse.
It was designed to address the needs of data driven
organization in a cloud first world
with its reliability, security, and its governance.
So what is data governance and why it matters?
Data governance is a strategic part
of an organization that sets in place policies and procedures
to maximize utilization and value of data for decisions
and operations, while ensuring that equality
standards, privacy standards, security controls,
and regulatory compliance.
GCP provides robust tools and technology
to protect and govern throughout the lifecycle of the data.
We offer built-in protection at scale
by default. Data is automatically encrypted while
in transit and at rest.
We supply tools and technology to efficiently govern
all your data.
For example Cloud DLP helps you discover and classify
sensitive data assets.
Cloud IAM provides access control and visibility,
security policies and our newly G8 Cloud Data catalog
helps you discover and manage all your data.
We support compliance requirements
with all third-party certifications.
Finally, we believe that the trust
is created through transparency, and for that we provide tools
like Access Transparency which gives you the visibility
and accesses by either the support
teams or the engineering teams at all times.
Let's talk about some of the enhancements
that we've done around data classes
and fine-grained access.
Now with BigQuery's column level control
you can start thinking about data as a set of data classes,
some more sensitive than others.
Data catalog will control policies per data class.
Audit logs will refer to these data classes.
Let's take a look at a deeper level and the use case.
Use data catalog to define a taxonomy of policy attacks
and add access permissions for users,
groups apply these policy tags to BigQuery columns.
In this example, you may have first responders.
They may need to have access to some of the PII information
like location and phone and so forth.
On the other hand, auditors would have additional access
to maybe financial information.
Now you'll be able to provide all these controls very easily.
On GCP we provide a platform that's end-to-end governed.
You can imagine almost like two zones.
A free zone where not much of the sensitive data
is really present.
But a governed zone where a lot of your governed data needs
are placed.
Past couple of years, we have been
focusing on building the best in class capabilities for you
to achieve better governed zone through BigQuery for all
your enterprise needs, by establishing
fine-grained access like column, table,
role levels, and whatever.
By integrating DLP to drive all data classification.
By introducing Data Catalog to discover and tag your data.
However, data governance is a vast space and many of you
have complex hybrid environments.
As we announced last November in London
we have partnered with Collibra, the market
leading governance solutions provider
to help you with all your data governance
needs in this hybrid world.
And I'm delighted to inform everyone
that Collibra is now GA on GCP.
And we're working together on joint roadmaps
to further enhance our offerings to everyone.
As we saw before Data Catalog and BigQuery allows
you to govern data the finer granularity through policy tax.
But governance is much more than that.
We also need to apply, revoke, modify access
in response to events across your enterprise.
As I mentioned earlier, we've already
begun integrating Collibra into GCP services.
Our catalogs are designed to work together
to help system integrate achieve data
governance for hybrid systems, whether it's
your multi-client needs or cloud and on premise.
With Collibra on GCP you will have the best of the two worlds
to achieve all your data analytics
and data governance needs.
My partner Jim will now walk you through Collibra
on GCP journey.
JIM CUSHMAN: That's great.
Thank you, Evren.
I really appreciate the introduction.
And excited to share with you guys a use case.
Now there's basically countless ways
that you can identify value from this combined offering,
but we're going to settle in on a use case defined by George.
He's a business analyst working on the Google Analytics
platform and he has a goal.
His goal is customer satisfaction.
Now that's a year-long endeavor, but over the year
he's looking for different data sets
that answer different questions that
show how customer satisfaction is achieved, maintained,
it grows, and it reduces churn.
So in this situation, we're going
to focus on a moment in time where George
is looking for additional data beyond what George is working
with thus far to answer some questions about customer churn.
The process is going to follow something like this
and it's going to be a use case run-- what
we call data provisioning.
This is a situation where there's only a subset of data
that George currently has access to, but looking for what else
might be available to use.
So George will start with an example
of using Collibra to actually work on the Google Cloud
to search the catalog.
And once that's done our job is to help navigate George
through the shopping experience, go out and fetch the data
for George, and present it back to George for consumption.
So let's go ahead and jump in.
So let's suppose George starts-- the first thing
that George is going to do using this shopping metaphor is
try to find a door to open.
And so George being kind of a non-technical person
might start with more of a business term.
So George is going to type in "customer churn".
And customer churn for George is a definition that's
been defined in his business glossary
and is shared within the entire company.
Now when George first sees this he
can then navigate to different facts and facets
such as picking business terms, and these business terms then
provide various things within the glossary
that George can choose from.
If George chooses customer churn it
takes George to the next step.
Now this is like opening up a door
to our graph of information available to George.
And some of the things that George might see
is which models are actually available underneath
the definition of customer churn,
such as customer order or product.
Some financial metrics, tags, and other things.
And this is a means of George basically
getting oriented within the store
and finding out where to go to next.
And let's suppose George says customer
is a really important thing, but I'd
like to know a little bit more about it.
So one of the first things we do is
show George what is the content of customer as a logical model.
And here we're listing all of the different attributes
and fields that really make up the definition
of the canonical definition of a customer.
In any one of these George can click on one of the definitions
and learn more about them.
Again, as an asset within the graph that again relates up
to the customer demand model.
But again, back to customer churn as a business term.
Now, Cliff might say this is great.
I like the content here.
So let me learn more, so I'm going
to navigate down the aisle.
As Cliff navigates down the aisle
of this-- kind of customer defined aisle--
we're going to find various things that
are groups such as systems.
And within systems might have databases,
and with them that might have schemas.
And eventually you get down to tables.
These are catalog assets that tell
Cliff about the various things that the company is
aware of that have been classified and mapped
to the logical model of customer.
And so this kind of gives you an inventory of ultimately
what is in this file that the shopping known by customer.
Furthermore, Cliff might want to say OK, tell me more.
And this is where we show the integration and the inheritance
to a table all the way back to policies, and rules,
and regulations, and limitations,
and reference data that make up the body of governance that's
within the Collibra system.
So in this situation, Cliff can see
very quickly that I had a table within here called customer
product sales that had actually been classified through machine
learning to the customer domain, and by definition
then it inherits all the items that
are classifications and standards and policies
that you see in green illustrated
to the left of the screen.
Once Cliff understands this, Cliff might say, OK I like this
but I want to know where these products have actually
come from.
Where do these tables come from and where do they go,
and maybe what happens?
And that's what we call lineage.
This is the idea of how did the data actually arrive
at this point in destination?
Where else does it go after this destination?
Who uses it?
And did anything happen to it on its destination to me,
such as some type of transformation object?
So we can actually capture all the underlying transformation
that may have happened to this physical asset
that you're considering for shopping purposes.
So this gives Cliff like a where's this stuff made--
is it made in the US?
Is it made in Italy?
Somewhere else?
I would like to know, as well as what happened on its journey
along the way.
Now the next thing Cliff might say
is, OK I'm now down to five different items
to choose on the shelf.
How do I delineate between them?
How do I know which one is right for me?
Well that oftentimes is being done through tribal knowledge
or what did I do in the past, but it's generally not
objective or defendable and doesn't always lead
to trustworthy information.
So what we offer in the Collibra solution
are highly visible and easy visualization
around a profile score.
And the goal is to provide Cliff very quickly front of label
demarcation about what is the best for what they want.
So in this situation we're showing various attributes that
make a customer of interest to George,
and showing which system actually has the best.
So for example the customer Billings
seems to have the high score for email,
whereas customer product sales has
the best information for both address and for churn date.
Since those are important it seems
like customer product sales is something that would
be pretty important to George.
Now maybe that's not enough front of label.
Some of us shoppers like to turn the label around,
and if necessary we can get some detailed statistical
information about that column using stats
around no values, anonymous values,
distribution frequencies, et cetera,
that can really help Cliff choose which of these data
sets to use for his analysis.
Now Cliff has said OK, this is the one I want.
I'm adding customer products sales,
as you see illustrated in the center here.
And this is in Cliff's shopping basket.
Now you can see there are other data sets that were already
in there.
Let's just assume Cliff had already shopped for things
and maybe these were sitting in wait.
But the next big thing that Cliff wants to do is say,
you know, I'm not just buying product at the grocery store,
I'm actually asking for data and I need to ask for a request.
So Cliff starts the request process.
And each customer can set up its own need
for how they manage this process.
But ultimately the data owner itself
that would grant authorization to this data
would want to know why do you want it?
How long are you going to use it?
Where are you going to use it, et cetera.
And so George enters this level of information in the data set
request.
And one thing I'll draw your attention to
is that George has actually selected lease.
Lease is this notion of a short term duration
but I want a physical copy of the data,
whereas borrow would be an example of something I just
want to have virtualized for a very, very
short period of time.
And buy would be I actually want to take
a full copy going forward.
And that ultimately might also say,
where do I want to take it?
And here we're actually going to bring the data set
to Google BigQuery.
And of course there are dates for how long
George would like to use it.
Now after George has actually submitted his request
it needs to go over to the data owner.
So we're going to switch perspectives really
quickly and look at Joanna, and she receives an inbound request
from George.
And so you can see that Joanna is seeing the same information
that George entered in in the request in the center panel,
and off to the right is where Joanna would then
determine whether she wants to grant approval for this or not.
Now, I'm showing you a manual version for this now,
obviously for effect, but you can set up
rules and workflows that can automate this process based
on who's making the request, for what purpose, for how long,
and where is it going to go.
But you can either manually do it
or you can set up automated rules.
But since she actually approves this request then
it's going to go back to George.
And when it comes back to George there's one last thing
that he has to do.
He has to acknowledge all of the items
to the right that are part and parcel of actually
taking a lease of data for a period of time
to Google BigQuery.
And you'll remember those were established
in those green boxes earlier on when we looked at the customer
domain.
And they were all inherited through
to the data set that George is now asking for.
So once George has actually approved of this request
he can now check out.
And so this the really interesting part, because again
George is not a technician.
We don't want George to have to do
a bunch of assembly, et cetera.
We're now going to kick off a data provisioning process.
And so we really are finished up with step one here.
And then the next step is that we're actually
going to have an edge-based appliance.
It really is close to the data center,
wherever it happens to be.
Suppose it's on your own premise.
It could be in another cloud.
It could be a SaaS-based solution
that you're working with.
It is this virtual appliance that actually is reading back
to the Google Cloud server to talk to the bigger cloud
and say, do I have any instructions?
And the instructions would be this request
that George has asked for and received approval.
They would be downloaded and they
would execute a distributed query
against the underlying data set or data sets that
have been approved.
After that data is extracted it would then be composited,
which is basically the process of looking
for any duplication within that data set
and compositing it down to a golden record
choosing the best attributes from amongst the contributors.
Once that's been done then we also
want to acknowledge that not all Georges are alike.
Some Georges might have access to certain data elements
that others are not.
And that's where we deploy this policy enforcement
around access management.
We would actually eliminate data that's not actually
available for George to look at, either by redacting
it or masking it before it's passed along to be loaded
into BigQuery for the request.
Once that's done the data is then encrypted,
transformed up to the appropriate landing
spot for BigQuery, and then subsequently loaded
into BigQuery.
And once that's been done now George
is notified that the data set is ready for processing.
Now when the data is ready for processing George
can do a couple of things.
He can look at the data in the raw directly within BigQuery.
But more likely George is going to want
to look at this and VI reporting or analytic environment
of choice, and let's suppose it's Looker.
And so that's exactly how George is
going to start working with this new data set
and maybe either alone or in combination with other data
sets that were already available to George within BigQuery
as well as Google Catalog, or in combination or separately.
So now George has actually been--
almost really at the click of a button--
had data provision for his use without having
to write any code.
And it's all been done under a governed use case.
And so that's really what we call data use provisioning.
But to really round out the use case
we have to send notification and reminder to George that you're
just using this data for a short period of time.
In order to comply with the data use agreement
you actually have to get rid of it when you're done with it.
So we send a notice off to George
to let him know exactly that your use of the data
has expired and it's time to get rid of it.
And of course then we'll notify Joanna
if the same has occurred.
If in fact, it's been compliant and George did get rid of it.
Otherwise, it would get escalated
and she would be able to follow up accordingly to make sure
that we stay in compliance.
Now this is just one of many, many use cases to follow.
We thought it was a good illustration of how
the combined solution works together
in a highly relevant use case.
In terms of where we are today, Evren
was kind enough to announce that we're available on Google
Cloud as of July 6th.
Now we are also working on more and more tighter integration
and you'll see where we are now.
We're at storage with "principle of least privilege"
with data class level securities available within Google.
And through the integration with Collibra
we also offer this ability to shop and discover and make
that request for data.
And it could be data that's already in BigQuery or in cloud
storage that they're just asking for approval,
or it might actually need to be provisioned.
In situations where it actually needs to be provisioned
we'll be covering that within the next year's roadmap.
But I think I'd also like to share with you that later this
year-- maybe by the end of the year--
Collibra Data Intelligence for your convenience
will be available on the Google Marketplace.
And if you've enjoyed what you've listened to here
I really highly recommend that you follow your other peers
in the industry.
Even though we've just announced this a short period of time ago
we have a large number of customers
who've already expressed interest
and are starting that journey with us.
So if you would like to follow in their footsteps, please,
I would suggest you follow up either at the Google Cloud
website or the Collibra website.
On behalf of Evren and myself I would
like to thank you for joining us,
as well as wish you guys safety in these difficult times.
Thank you.