VIDYA SHANMUGAM: Hi, everyone.
Welcome to Next 2021.
We're very excited to have you join our session.
In this session, we'll talk about how
Google Cloud Platform and technologies like BigQuery,
Looker, and Apigee can help your organizations transform, unlock
innovation and flexibility by adopting a multicloud strategy.
I'm Vidya Shanmugam, Product Manager at Google Cloud.
I also have with me Joe Malone, Product Manager at Google
Cloud, and Nitin Doegar, Data Engineering and Enablement
Manager at Johnson & Johnson.
In this session, we'll talk about
the various multicloud challenges and GCP's approach
to solving them.
We'll share BigQuery Omni's general availability.
We'll also provide an intro to cross-cloud transfer service
and see a demo of how BigQuery Omni works.
You'll hear from our customer Johnson & Johnson
about their multicloud journey with BigQuery.
Let's jump in.
Multicloud is not going anywhere.
In fact, it's continuing to grow at a rapid pace.
Even from the time we launched BigQuery Omni,
our multicloud service, in preview over a year ago,
the number of organizations who are relying on multicloud
for their infrastructure needs has jumped from 80%
to over 90%.
Enterprises are not consolidating
on a single platform or a solution.
Instead, they're expanding and they're proliferating
the data stack across clouds.
We have heard from our customers about the challenges multicloud
analytics and management creates,
some of them being data silos.
Analysts and IT leaders want easy access and single view
to the data across clouds.
They want secure management of this data.
There are too many copies of data,
making it inconsistent and not fresh.
Analysts and data scientists have advanced analytics needs.
They want to analyze and visualize this data
to help their organizations with intelligent and data-driven
insights.
Customers want simple and flexible pricing.
Our goal with GCP's multicloud approach
is to not only provide our customers
tools to manage this decentralized data but to also
solve their business problems.
Let's see how.
We have a portfolio of tools to offer
to address the various challenges we've talked about.
For example, BigQuery helps you simplify integrate and analyze
your data across clouds at scale.
With modern BI tools such as Looker,
you get aggregated dashboarding and reporting.
Looker's in-database architecture
allows you to directly connect with underlying Cloud
or hybrid database, enabling cross-database queries
or joins.
Looker's cloud-native and agnostic architecture
makes it compatible for any multicloud strategy.
Developers can build applications and access them
either through the BigQuery API or Apigee,
irrespective of where the data resides in.
One of the key areas we're investing in
in our multicloud portfolio is in BigQuery Omni.
We are super-thrilled to announce
the general availability of BigQuery Omni for AWS
and Azure.
We will be launching in two regions, AWS US East
1, and Azure US East 2, and continuing to roll out
in more regions in 2022.
BigQuery Omni is core to delivering
on our multicloud promise.
Many of our previous customers have told us
that a truly differentiated solution needs
to be secure, needs to provide cross-cloud analytics at scale
through a single pane of glass.
I'm happy to share that BigQuery Omni delivers on all of these.
Omni allows you to cooperate your data silos
through a single unified experience.
You don't need to move or copy your data.
Omni also helps you integrate your data
across clouds through a net-new capability called
cross-cloud transfer.
Cross-cloud transfer allows you to bring your results
back to GCP if you choose to.
You can leverage capabilities like BigQ ML and Looker
on this combined data to get the benefits of GCP and BigQuery.
We're very excited to be the first cloud provider to offer
Cross-Cloud analytics at scale.
Omni also helps you securely analyze your data
across another public cloud using
a fully managed infrastructure.
You can query the data without worrying
about the underlying hardware.
Compute resources run local to the storage,
providing you a seamless data analysis experience.
Our vision is to provide a cohesive multicloud platform
for analytics.
Let's see how these pieces fit together.
As you saw, BigQuery helps solve your challenges
with data silos.
On top of that, you can get consistent, fine-grained
security and governance through a feature
called authorize external table that's coming soon.
In the future, you can also leverage materialized views
for cross-cloud scenarios and get fresh data and insights.
With cross-cloud transfer, you can do cross-cloud analytics.
This data now becomes universally accessible to tools
like Looker and ML models and all your analytics needs.
As you can see, our unified platform
will solve many of your multicloud challenges.
Let's take a deeper look at cross-cloud data
transfer before we dive into the demo.
We have heard from our customers that it's simply not
enough to provide access to underlying
data in different clouds.
Customers want someone to combine the relevant data,
for example, marketing and inventory data, ads and buyer
profile data, to drive intelligent, data-driven
decisions in the simplest and most cost-effective way.
We are excited to announce the first
of the cross-cloud features, cross-cloud transfer.
Cross-cloud transfer enables customers
to bring just the data they need when they need it.
The preview of this feature will be coming soon.
With that, I'd like to pass it on to Joe,
who will be showing us a demo of these awesome capabilities.
JOE MALONE: Thank you, Vidya.
Hello, everybody.
My name is Joe Malone.
I'm a Product Manager for BigQuery Omni.
I'm here to present a demo.
So let's hop in.
Before I hop directly into the demo,
I want to provide an outline of what I'll be covering.
First, I'll show you how provides a single pane of glass
to provide access to data across clouds.
Two, I'll show you how we drive a consistent governance model.
And then three, I'll show you how
we provide simple data movement when necessary
to move data between clouds.
And just as a heads up, I'll be referring
to a fictitious scenario in which I assume
that I am an analyst for a mobile gaming company that
has multiple titles that has data residing across all
of the different clouds.
So this will demonstrate how Omni
helps to solve the challenges of an analyst who is dealing
with a multicloud reality.
So now here we are in BigQuery.
And as an analyst, what I want to take a look at
is all of my player data for my titles across all of the clouds
that I have.
So if I expand my project, you'll
see a list of all of our different data sets on here.
I first want to look at this player data.
So I'll do a search.
So we'll see player data for AWS as a data set, player data
for Azure, player data for BigQuery,
and then player data for GCS, or Google Cloud Storage.
Now, what these represent is data
that are residing in all different types of storage
environments across our different clouds.
So we see our player data for gBall,
our basketball title, being stored on AWS,
gPuck, our hockey title, on Azure,
for golf, BigQuery, and then for gHoop,
our separate basketball title, on Google Cloud Storage.
Now, historically, what we would have asked a customer
to do in order to analyze data in BigQuery
is move a copy of the data from AWS to BigQuery itself.
But because we have computing locally on AWS,
we can now query underlying data.
I want to give you a sense of what this gBall data looks
like in AWS.
Here we are in S3, is that file being stored on AWS
through a connection, a "managed by a service" account,
we can now analyze that data on S3, query the underlying file,
and return results back to BigQuery
without moving the data at all.
So I'll do just that.
I'll just query all the different user IDs
across that particular file, run this query,
it executes locally, and then returns
just the results of that query.
Now, the great thing about Omni being a single pane of glass
is the same way in which we run that query against AWS data
is the same way in which we would
run that query against Azure data, against BigQuery data,
and against GCS data.
So all of this access methodology
does not incur any more skills or copies
of data on the analyst.
Next, let's talk about an awesome new feature
that allows us to centralize governance
across all of our data sources.
We are launching a feature called
authorize external tables here in preview very soon.
All of these data tables are actually
set up with one of these authorized external tables.
And what these tables allow us to do
is delegate access and control fine-grained security policies
at the table, the column, or the row level.
So for example, here, I've added a policy tag
on top of our PII data fields.
So first_name and last_name have these sensitive data policy
tags.
So for our AWS data source, let's
go ahead and query this data.
And what you'll see here if I do a select star,
because I'm trying to query a column that I do not
have access to that security tag against,
then you will get an error around not
having that level of access.
Now, this is great.
Because as a security team and as an admin team,
I can consistently apply these tags
across all of the data sources, whether it's on AWS, Azure,
BigQuery, or Google Cloud Storage, in a very uniform way
so that it's more consistent and more secure.
Similarly, on this same data set,
if I were to avoid accessing the sensitive columns
and just do the columns I have access to,
you'll see at the top that I also
have row-level security layered on as well.
So in a very simple fashion, we can
layer on column and row-level security
so that the users only have access to the data they need,
regardless of the cloud that they're working against.
The final feature that I'll demonstrate for you today
allows the analysts to combine data across clouds.
We are going to launch a product feature called
cross-cloud transfer in preview soon that allows an analyst
to move data from AWS or Azure into BigQuery so that you
can support more advanced analytics like joins.
So in order to demonstrate this feature, what I've first done
is created an aggregated table, which is currently
just a copy of the gGolf data on BigQuery,
but filtered down to only the fields that we need.
So just running a quick aggregate on top of that,
you'll see that this is simply just aggregated data for gGolf.
So grouping by the titles, you'll
see that there are currently 45,000 weekly interactions
on gGolf.
Now what I want to do now is combine that gGolf data
with the gBall data as it exists on AWS.
So what I'll do first is filter this, similarly,
just the fields that we need to transfer
using an export data query.
So we are going to match the schema.
So we combine the two tables.
See just the fields are the same as we
have in that aggregate table.
This will filter down the file, export the data file
into AWS in S3 as a new file.
So let me refresh in S3.
You'll see that filter file in AWS.
Now the real feature that we are demonstrating here
is this opportunity to load that filtered file
into the aggregated table that you see right here.
Now, this is our cross-cloud transfer feature
that enables, with just simple SQL,
the ability to load a file from AWS or Azure
or GCS into BigQuery as a new table
so that you can do advanced analytics.
So I will run this now.
And there you go.
We've loaded this file into GCP.
Now, just to show you that data is now
all in that aggregated table, I ran an aggregation.
And now you'll see that the two titles are now combined
into the same aggregated table.
So with cross-cloud transfer, this load statement
allows an analyst with native SQL
to be self-sufficient in combining data across clouds.
They no longer need to tap on the shoulder of a data engineer
or security member to do get the data transferred
across these two environments.
So in summary, Omni will provide a single pane of glass
with consistent governance and simple data movement
so that analysts can drive democratization
of data, secure analytics, and differentiated value
across all clouds.
Thank you.
And Nitin, over to you.
NITIN DOEGAR: Thank you, Joe.
And I'd like to thank Google team today
for giving me this opportunity to present at Google Next.
I'm Nitin Doegar, and I'm part of Data Engineering
and Enablement group at Janssen Pharmaceuticals of Johnson
& Johnson companies.
My group manages data and analytics application
for Janssen Commercial in North America.
We collect, organize, and make available data
from various internal applications
and third-party data providers to help our analysts
and data scientists provide insights to the business.
We have petabytes of data stored across commercial clouds,
and it's increasing every day.
We use multiple Cloud environments
such as Google, Azure, AWS, and internal company cloud
for data collection.
And this is because of various reasons,
for example, where the data is generated, how historically
technologies were approved within the company,
cloud costs, et cetera.
As I previously mentioned, we have data
stored across multiple clouds.
Many times, users get into the issue
where data has to be brought together to a single platform
before scientists or analysts can work on it.
This becomes a repetitive activity
where this has to be done often and sometimes many times
in the day.
Bringing data together also leads
to duplicating of the data across clouds,
which increases both our storage and compute costs.
Working with multiple clouds, users first
have to spend time learning the cloud-specific tools or data
warehouse databases rather than focusing
on their core work, which is providing insights
to the business.
What users are looking for is a faster time
to market, which means, for them,
faster insights into the data and faster
processing of the data.
So how does Omni solve all these challenges?
What we envision is that, since Omni can work across clouds,
it can provide more efficiencies to our analysts and data
scientists in many ways.
They will be able to spend more time analyzing the data
rather than trying to bring it together.
They will be able to use a single platform to access data
stored anywhere in the cloud.
Once the connections are configured,
users can go to one place, write SQL queries,
and access data stored anywhere.
It will help the users to update their reports and models
any time by just including the missing elements or newly added
data sets.
There's no dependency on engineering teams
to first update the data pipelines to access the data.
And for the organization, Omni will
provide a fully managed platform that
is both flexible and secure.
We will not have to manage compute shortages capacities
at any time, we'll not have to train new members
joining the team on all the different technologies
that we use across clouds.
We can manage data access and data securities
from one place using BQ Omni.
Also, since the BQ Omni computes are
based on [INAUDIBLE] technologies
and are managed by Google, we won't
have to worry about scaling the Omni
system for any slowness in performance or things
like that.
In our POC, we saw pretty good performance results with Omni,
more of which I will share in my next slide.
A few months back, we did a POC on BQ Omni
where it was still in pre-GA phase
to understand the capabilities of the platform.
For the POC, we use industry standard 1 terabyte TPCDS
data set stored in Parquet file format
on the cloud-native storages.
We executed the same queries across
multiple different warehouses that
support reading data directly from the cloud storages.
And here are some of the aggregated numbers that we saw,
where, overall, Omni performance was much better.
In our first test, we connected Omni to J&J cloud
to make sure that it can connect using J&J data access policies
that are already restrictive.
And to our surprise, we saw that we can connect to Omni easily,
without issues.
We found that Omni was significantly better and faster
than other similar applications that
are available on the clouds.
We could write back query results to other cloud storages
easily.
And multi-user and parallel queries
had no performance issues with BQ Omni.
So what this all means to us.
How we see Omni is that it can be a single pane of glass,
using which we can connect to various cloud storages
and access data using SQLite queries.
It can provide faster time to market
where the data users can get access to data quickly
and are able to process data at a much faster rate.
It's a fully managed platform, which reduces
our administration overheads.
And our team can free up doing more work.
Omni is easy to set up with other clouds.
And most importantly, it enables self-service for end users
where they have much more flexibility to read and write
data across clouds.
With that, I'd like to pass back to Vidya.
VIDYA SHANMUGAM: Thanks, Joe, for the great demo.
Thank you, Nitin and J&J, for betting
on Google Cloud and BigQuery.
With that, we're very excited for you to try BigQuery Omni
and share with us any stories of how your organization embraces
multicloud.
Thank you for watching this session
and for being a part of this event.