FELIPE HOFFA: With over 500 sessions at Next '19,
it's understandable that you might have not seen them all.
So if you missed Modern Data Warehouse with BigQuery,
where the BigQuery lead Jordan Tigani shares how
we are transforming BigQuery into the data
warehouse of the future, don't worry,
because this is a rewind.
You might already know that BigQuery is our enterprise data
warehouse built for the cloud.
It can easily store exabyte of data,
as well as run frequent queries over petabytes of it.
Everything is encrypted, durable, and highly available.
And these are some of the features that
make BigQuery really unique.
It's fully managed and serverless.
It handles streaming data to provide real time analytics.
It has built in support for machine
learning and G analysis.
And high speed in-memory BI engine
was just added for instant interactive dashboards.
And of course, during the talk we
had quotes from industry analysts and customers
that can attest to all of this, as well as petabyte scale
interactive demos.
But let's focus on the main message of this talk.
How is Google Cloud putting all of this
together to build our vision of what
a data warehouse of the future should look like?
Let's start with the first two elements.
A data warehouse should handle storage and compute.
And by that, I mean the ability to collect
all the data you need to have together
and the ability to analyze it and combine it at any scale.
In fact, one of our customers is storing more than 250 petabytes
of data in BigQuery.
And others have run a single query
over more than 5 petabytes.
To pull this off, you really want your data warehouse
to be truly serverless.
You get to decide on your data and the queries
you want to run on it, while we take care of running everything
else behind the scenes.
And what's the magic behind the scenes?
BigQuery's architecture that separates storage
from compute and a petabyte network connecting both sides.
Cool!
But that's not enough.
A modern data warehouse should be
able to work with real time data.
To prove this during the talk, Jordan ran a dataflow pipeline,
generating data out of more than 600 CPUs.
As you can see here, these nodes were
generating more than 2 gigabytes of data per second.
And he showed how BigQuery was able to instantly run
queries and find patterns over this incoming data.
What else do you expect from a modern data warehouse?
Centralized storage.
And this is something new with BigQuery.
We already know that BigQuery can store all of your data
and analyze it.
But what if we want to read it from other systems?
BI tools, Batch and Stream processors, Hadoop, Spark,
machine learning platforms, they should all
be able to read data storage BigQuery.
And we just announced that BigQuery stores
API to enable fast parallel rates without the need
of exporting data first.
What else?
Security the trust.
To make sure we can put BigQuery into the hands of our most
sensitive customers, Google has developed features with them,
like customer managed encryption keys and access transparency.
If anyone managing your cloud data warehouse
has to look at your data, you should know
when and why this happened.
And you get that with BigQuery.
Next, data sharing.
You want to make BigQuery available to anyone
that needs to analyze data in your organization.
On one hand, we have a strong permission model.
But we also want to make the power
of data accessible to those who need it.
So with BigQuery, you can save and share
queries, as well as data sets.
And your analyst can create beautiful reports
with Data Studio and third party tools.
And to make all of these reports fast and scalable,
BI Engine is a new block that acts as a transparent catch in
between, an in-memory database that knows when to bring data
from the back end, where you only
need to decide how much memory you want it to have.
And we're making BigQuery increasingly accessible,
even to users that would rather do everything in a spreadsheet.
And the last bit of a modern data warehouse, predictive.
With infinite storage, you can know what happened in the past.
With real time streaming data, you
can tell what's happening now.
And with machine learning capabilities,
you can predict what will happen next.
Now BigQuery saved room to create and run machine
learning models natively, inside.
Yes, you can create a model and then
use it to predict, within BigQuery and only
using a couple SQL commands.
To the initial linear and logistic regression models,
Google has added [INAUDIBLE] clustering, and matrix
factorization, and even the ability
to import any TensorFlow model you created outside BigQuery,
and even trained TensorFlow models
without leaving your favorite data warehouse.
This is our vision of a modern data warehouse.
But it doesn't stop here.
Take the full talk to see how one of our customers
puts everything together to power
the future, driverless cars.
That's it for now.
If you'd like to see the full talk,
check out the link in the description below.
Now it's your turn to try BigQuery.
I'm Felipe Hoffa, and remember to subscribe.