EMILY RAPP: Hey, everyone.
Welcome to "Analytics in a Multi-Cloud World."
I'm Emily Rapp, product manager at BigQuery.
We are facing a data explosion.
175 zettabytes-- that's 21 zeros--
21 zeros of data in the world by 2025.
Where's it coming from?
We have cat videos, server logs, monitoring.
There's retail data.
When you're thinking about e-commerce,
it's a lot more than just that final transaction.
It's every point on the sales cycle.
IoT, manufacturing, even my washing machine
is connected to the internet, and therefore generating data.
We have all of this data at our fingertips.
But we just can't quite get to it
because we're living in this world of data silos.
There's organizational separation.
IT departments are often centralized.
And they are separate from the central data department,
if there is one, which are even then
further separated into the business units.
And those business units also have
analysts, data scientists working and trying
to understand problems.
It's fragmented.
You have data originating from a bunch of different sources.
It's all in different formats with various levels
of processing.
This means that our analysts, our data scientists,
our users--
our users are using too many tools,
have to get data from too many places,
and they're making too many copies.
Why is this happening?
A key component is shown in one of the Gartner cloud adoption
surveys.
81% of companies who are using public clouds
are actually using more than one cloud provider.
And so that means that multi-cloud management is
becoming-- actually, it is a true challenge.
IT leaders, they want to be agile, flexible, secure.
And analysts and data scientists,
the ones that are actually doing the day-to-day work
with that data, they're just trying
to help their organization make the right decisions.
And even if organizations decide to have a centralized data
approach, like we talked about in the earlier slide,
there's still always going to be separate business
units, subsidiaries, partners.
Multi-cloud is and will be the world we're working in.
We need to solve for these macro challenges.
So today, we're introducing BigQuery Omni,
a flexible, multi-cloud solution that lets you analyze data
across clouds.
You don't have to leave BigQuery.
It's your familiar interface, same experience.
But now, you can actually reach out--
reach out across those silos.
BigQuery Omni is now in alpha.
And what's really cool is that it's powered by Anthos.
We don't have to do that moving and copying of data.
SPEAKER: As a data analyst in the marketing department,
BigQuery Omni enables me to use historical sales
data in my upcoming marketing campaigns.
Using our orders data, we're going
to validate the hypothesis that we should
be focusing our marketing efforts beyond just
food and beverage.
Our first query uses the HyperLogLog approximation
function to sort the departments by number of orders.
We are using our read-only external connection,
Retail Read, to get the data from the Orders External table.
The query returns the departments,
sorted by number of orders.
And as we can see, all top 10 are either food or beverage.
This validates our hypothesis that we
should be focusing our marketing efforts on this customer
segment, household, or personal care product purchasers.
Now we need to build a list of IDs
using our customer data set.
The customer data has personal information,
including name and zip code.
And we need that data to stay within the AWS region.
However, we do have a de-identified user ID field
that we can use in our campaigns.
Our next query will build our audience segment,
buyers of household or personal care products.
Since our campaigns are regionalized,
we only want customers from Oregon or Washington.
Now, we have a list of IDs that can be used in any marketing
activation channel.
We want this data to be accessible to our marketers,
not just the analysts.
So I use our right connection to export the query result
to the marketing folder that they have access to.
And that is how you can use BigQuery
Omni for data-driven marketing activation.
EMILY RAPP: That was pretty cool, wasn't it?
Let's talk about bringing multi-cloud analytics
to your data.
First, we have to break down those silos,
get you to getting insights faster.
Tap into the power of BigQuery.
It's cost efficient, no moving or copying.
You just instantly query, instantly
get your results with that consistent experience.
It doesn't matter if your data is in Google Cloud, AWS,
or, coming soon, Azure.
You can just use the same standard SQL,
the same BigQuery familiar interface,
write queries, build dashboards.
Because the reality is data, it's not just numbers.
It's not just bytes stored in some data center somewhere.
It's the way that we can actually communicate.
We can tell stories.
We can answer questions.
We want you to be able to have the ability to actually work
across those data silos.
That data that was at your fingertips,
let's get closer to it.
And we're able to do that because under the hood,
we use Anthos.
We think about BigQuery.
BigQuery is the separation of storage and compute.
What does that mean?
It means you can scale up and say, hey, sometimes when
things are going crazy, you may need a little bit more compute.
In other cases, maybe now you're gathering a bunch of data
and so you need storage.
We were thinking about how do we solve this data silo question?
But the challenge was we have to figure out
how to get compute to the data, even if that data is
in another cloud.
So BigQuery Omni, we run the compute clusters,
known as Dremel, on Anthos clusters in AWS.
We have a secure connection, because it's
really critical-- we think of that single pane,
that the control plane and the metadata
can remain on Google Cloud, but only the query result
actually passes through the BigQuery routers
and then our secure connection to the next region, AWS
or Azure.
That connection can also be used when users choose
to bring the results back.
But the key piece is it's choice for you.
You can bring them back or you can do everything within AWS.
BigQuery Omni is a fully managed service.
A question I hear is, oh, so I have
to use my Anthos or my Compute?
Nope, we got that covered.
We do need your storage.
And it's really critical that you maintain the access control
to that storage.
But with the compute resources running locally,
all that's done behind the scenes.
And you can just answer those questions.
So how do we actually solve some of those business scenarios
with BigQuery Omni?
This one's funny.
This is my old world.
Up until recently, in September, when I joined BigQuery,
I was the infamous marketing analytics adds measurement.
Specifically, how many people saw my ad?
It's a surprising hard question.
And actually, it's even harder to actually think about OK,
now I figured out what's going on in my marketing campaigns.
What about my audience behavior in my apps on my site?
What impact did those two have on my sales?
So let's think about how could we solve that problem?
How could we see the impact?
We could visualize.
So use BigQuery.
Query Google Analytics 360 ads data in Google Cloud.
And then tap into your e-commerce data and app data
that's on AWS S3.
Now with Looker, you can build a dashboard,
so you or those executives that we all have to worry about
can actually look at what the audience behavior is
and purchases alongside spend.
It's pretty amazing what seeing something visually can do.
It's like oh wait, there's a spike.
And then you also talk to people.
It's like no, really?
Building a dashboard when data comes from all of those
different places, and more importantly,
making sure that if you share that dashboard with someone,
they're not going to see something that they shouldn't--
it's surprisingly challenging.
Now let's get a little bit deeper.
This is something I'm sure probably many of you
are familiar with, the OK, my business team
said you need to use the app data that we
have to do smart things.
And we need that data.
And we need that data here.
And it needs to be formatted and easy to get to and fast.
So as a data engineer, you end up running thousands
of daily jobs to process logs and build cross-cloud ETL
pipelines.
And what happens if those pipelines go down?
So I've actually seen it where instead of fixing the pipeline,
companies will sometimes just hire
consultants to respond to tickets to answer
the questions.
It's that complicated.
With BigQuery Omni, it's not always
just about getting new things with dashboards.
It's looking at your existing workloads
and how you can optimize them.
You have those petabytes of click stream
logs that we talked about.
And for some reason, every app always
ends up going into different S3 buckets with parquet files.
Today, you have those thousands of jobs
that we talked about running.
And all you're trying to do is build audience lists and CSV
files of your active users.
Replace that with a single SQL query that any analyst can run.
So now that any analyst or data scientists can just
run a SQL query, let's think about how that data
scientist can actually use the data from sales to not just
understand their existing audience,
but reach new audiences more effectively.
SPEAKER: Let's get started with BigQuery Omni.
In the BigQuery UI on Google Cloud,
choose the public cloud region where your data is located
and run your query, the same query
you would use to work with data that's stored on GCP.
The query is passed to the other public cloud.
And all of the compute is done within that public cloud
region.
The results are returned to the BigQuery UI.
Or you can choose to export the results directly back
to your data storage on that public cloud, which
means there is no cross-cloud movement of results or data.
You can start using BigQuery Omni right away.
There's no need to do any formatting or transformation
of your data.
BigQuery Omni supports formats such as AVRO, CSV, JSON,
ORC, and parquet.
You also don't need to move or copy your raw data out
of the other public cloud, do any cluster management,
or provision any resources.
All computation occurs within BigQuery's
multi-tenant service, which is running on the region
where the data is currently located.
Behind the scenes, BigQuery's query engine
is running on our Anthos clusters
within the BigQuery managed service.
We get the data from data storage within your account
only once you've authorized permissions
via your other public clouds' IAM roles.
Those roles are tied to a BigQuery Omni connection
object.
You can change, grant, or revoke access at anytime.
And the connection object can be shared to multiple users
within that same project.
We maintain a secure connection to bring your query results
back to Google Cloud.
Welcome to BigQuery Omni.
EMILY RAPP: OK, so we did that query.
We exported it back to AWS.
And now it's time to activate your cross-cloud data
story, which means think about that Google Analytics
data that's coming in.
Think about the S3 data storage.
And take those audience lists and do things
like look-like targeting.
Ever had that experience where you buy something
and you still see the ad over and over and over again?
And you're like come on, I bought that thing already.
Stop wasting your ads on me.
Now, we can actually solve that, because we
can tie that commerce data to the ad platform
in a safe and secure way.
So let's recap.
Embrace a flexible multi-cloud analytics solution
with BigQuery Omni.
We're going to break down silos, get straight
to the getting insights part of data analytics.
It's the fun part, anyways.
Having a consistent experience across clouds--
it doesn't matter where your data sets are.
You should be able to use the standard SQL BigQuery
interface, write your queries, build your dashboards.
It's all about taking that consistency and familiarity
to accelerate the time to insights.
You want to be able to wake up in the morning,
check your phone, be like, OK, we're good,
and go back to sleep, not have to worry about writing
a whole ton of different queries in different tools
before you know the state of your business.
And all this is possible because of Anthos.
Dremel was built by Google.
Dremel is what powers BigQuery.
Anthos-- it's that next stage.
Thinking about that Gartner quote,
81% of companies-- it's probably, actually, more
than that, I would say, maybe.
Think about everyone has some part of their business
that's on some other cloud.
And so by bringing the data to you,
we bring the compute to where your data is.
We worry about the infrastructure.
Don't worry about spinning up clusters, managing them.
We got that.
You focus on driving the business,
doing some really cool data science,
making the decisions you need with data,
not worrying about how do I set this up
and what pipeline should I run, and oh, wait, they broke.
Now I have to hire a whole bunch of people
to just answer tickets.
Let's get started.
Get started with BigQuery Omni.
Let's take it to the next level.
Take your multi-cloud analytic strategy
that you've built from a platform and IT perspective
and bring it to your data.
BigQuery Omni is in alpha.
So if you're interested, please go to the link
and fill out the form.
Let's break down some silos.