[MUSIC PLAYING]
LEIGHA JARETT: Hi, everyone.
I'm Leigha.
GABE WEISS: And I'm Gabe.
Welcome to the Data Cloud Summit to all
of our data scientists, engineers, developers,
and professionals.
We hope you've enjoyed the sessions in the summit so far.
LEIGHA JARETT: In this session, we're
going to take complex, fragmented data systems,
simplify them, and bring them together on a massive scale.
In our demo, we'll show you how Google Cloud can
help a company build a new application that
delivers great consumer experiences,
while keeping it simple, open, and integrated for developers.
GABE WEISS: Meet our fictitious company, Cymbal Solar.
They own a solar company that sells renewable energy products
and allows consumers to track their savings
and their ability to resell excess power.
As a part of their product revolution,
Cymbal partnered with Google Cloud.
Cymbal wanted to build a modern data cloud
that takes information from various sources to power
a unified insights platform.
LEIGHA JARETT: And Cymbal relied on an old database architecture
and struggled to manage all types of data
across multiple clouds.
But now, Cymbal wants to deliver a platform that
makes new data easily accessible to their consumers
so that they can better understand their own energy
usage.
GABE WEISS: The new architecture will enable Cymbal's developers
to go faster by providing a serverless,
scaled experience for application development, data
science, and data insights.
And finally, Cymbal wants to introduce modern B2C
interactions, including gamification,
to help change consumer behavior in real time
and make the whole consumer experience a little more fun.
LEIGHA JARETT: That's right, Gabe.
And to summarize Cymbal's Data Cloud transformation,
we'll focus on four objectives.
First, empowering developer experiences
by using serverless wherever possible.
Second, enabling petabyte-scale queries that
can be performed in real time.
Next, unifying insights across multiple clouds.
And finally, simplifying business intelligence
with a unified semantic model.
GABE WEISS: So to achieve this, we're
going to show you some of the latest innovations from Google
Cloud as of today at the Data Summit.
We're going to leverage five key products--
Cloud Spanner, the relational database
that scales with our mission critical data; BigQuery,
the industry's leading data warehouse solution;
Serverless Spark so we can build new machine learning models
without managing infrastructure; Vertex AI to deploy and monitor
those models; and finally, Looker,
to serve all of Cymbal's data for its business users.
LEIGHA JARETT: And by combining the differentiated power
of each of these products, we'll create a modern architecture
to serve both internal and external data use cases.
GABE WEISS: And by the way, any of the architecture
diagrams that you see was built from an awesome new tool built
by our one and only Priyanka.
LEIGHA JARETT: So cool.
Thanks, Priyanka.
All right, so let's explore how this architecture comes to life
in our app Cymbal Solar.
Here we're starting at the home page,
viewing our consumption data and details on our house.
Now energy consumption is being surfaced from BigQuery
using Looker's API and underlying semantic
model, while our personal information is
being powered by Spanner.
GABE WEISS: It looks like we're actually seeing
some consumption in the app.
Can you dig in some details a little bit?
LEIGHA JARETT: Yeah, let's drill into those details.
So, as we look deeper into this detailed usage,
we can even see recommendations being
surfaced for how to change our energy consumption.
GABE WEISS: We'll cover this in a bit, how easy it
is for data scientists to build production-grade models,
both ML and semantic, in our architecture.
LEIGHA JARETT: And check this out--
I can take direct action from this app,
so setting up new tools, like power monitors or other IoT
devices.
But with millions of devices for our app to manage,
we need a highly performant and globally consistent database.
So thanks Spanner.
GABE WEISS: For the developer, we've
simplified how easy it is to build, maintain, and enhance
our consumer products.
LEIGHA JARETT: Yeah, that's right.
So now let's go a little bit under the hood
to show just how we've enabled this consumer journey
using our Google Cloud architecture.
GABE WEISS: Our consumer journey all
starts with our ability to collect and transact
on data at scale.
With our sensor data and user transactions in Spanner,
we know our application will seamlessly
scale for millions of users across the globe
and with five nines availability.
Spanner's fully-managed capabilities
also ensure that our team can spend less time
on database tuning and more time on feature development that
benefits our consumers.
LEIGHA JARETT: And let me just gush a little bit more
about Spanner.
So for our developers, the new Postgres interface
means they get the familiarity of working
in Postgres but with the scalability
and reliability of Spanner.
Plus, with granular instance sizing,
we can leverage Spanner in more places
during our development workflow and make
sure we have the right sized environments
for our development needs.
GABE WEISS: Once we've generated data from our application
and from our devices, we ingest and transform data
across these sources and many others using standard SQL
with BigQuery.
LEIGHA JARETT: So just to add some more color here,
we have our data being assembled across Spanner
using federated queries from AWS using BigQuery Omni's
cross cloud analytics, and also from our new Google Cloud data
lake using BigLake capabilities in BigQuery.
So for each of these data points, all we have to do
is create a connection to the data from BigQuery,
and then we can just use SQL to query data from any source.
Plus, we can control our data using the same BigQuery access
policies across both external and BigQuery native tables.
Meaning, just like Gabe said, even though our data sources
are spread across different infrastructure,
we can prepare this data for analytics using standard SQL.
GABE WEISS: Next, using BigQuery and Serverless Spark,
we can build and submit our notebook
using a single pane of glass.
This enables us to run massive preprocessing and computation
for ML using the best tools for the job.
LEIGHA JARETT: Now this is huge because we
don't need a Gabe to understand the underlying mechanics
and cluster setups and configurations.
GABE WEISS: Right.
So using serverless lets developers
focus on development and Google to focus
on giving us the best and right size
infrastructure for the job.
LEIGHA JARETT: So we've talked about our architecture
at a high level.
But now let's focus on how this allows us to scale
with our consumer growth.
Like we said before, we used to rely
on old database-centric architectures.
So data science at scale was really a struggle.
GABE WEISS: While we had data science ambitions, in reality,
our data science team spent most of their time
trying to get data or being production support, trying
to deploy and troubleshoot models.
Our goal with the new architecture,
in addition to modernization, is to let engineers in each area
do what they love.
LEIGHA JARETT: And with this new architecture,
our developers can focus on their specialties,
meaning we're better equipped to build high quality data
products.
GABE WEISS: It's also going to mention
that in the prior architecture, we
would have had to stitch this together
using many products and many codebases.
Using Google Cloud, we can be intentional about using
products that all work seamlessly together.
LEIGHA JARETT: OK, let's jump ahead
and talk about how we provide our transformed data
and models to the application.
Gabe, can you talk about how we did this at Cymbal Solar?
GABE WEISS: Sure.
So let's start with our data science
flows for our data scientists.
We built our models using Serverless Spark
that we were able to execute from BigQuery's UI.
Once we have our models built, we
can leverage the MLOps capabilities in Vertex AI
to provide end-to-end management of both
our models and our processes.
LEIGHA JARETT: Right.
And once our models are created, they're
checked into the Vertex AI model registry.
Now, this lets us see changes that happen in our model,
giving us complete version control.
GABE WEISS: And then next, we create a live endpoint
with Vertex AI endpoints.
This scale to zero service allows
us to not worry about how much infrastructure we
have for our models and focus on using Vertex AI endpoint's
model management capabilities to track our model effectiveness.
LEIGHA JARETT: Not only that, but we
can build faster because our data scientists can
use programming languages that they're already familiar with.
And with frameworks for model deployment and monitoring,
MLOps processes can all be performed
in a single pane of glass.
So far, this new architecture has really
simplified the amount of non-core work
that our teams have to do.
Now we're crunching petabyte data sets across our clouds,
unlocking more use cases, and further transforming
the business.
GABE WEISS: OK, so let's walk through that last component,
how we democratize access to this data,
serving it up to our internal BI users
and our external consumers.
Leigha, you're a product manager.
Can you walk us through some of the capabilities
for this application?
LEIGHA JARETT: Of course.
So with Looker, we create a modern, governed, universal
semantic layer that queries data across our data clouds.
The semantic layer contains our business logic,
written as code, that can be defined and shared
across our entire organization.
GABE WEISS: Looker queries databases
to get the freshest available data
and then serves insights for the model to users.
Insights can be served in dashboards, alerts,
embedded experiences, or custom data applications.
LEIGHA JARETT: And not only can our team
consume data in the Looker UI, but they can also
access governed data in Data Studio, a connected Google
Sheet, like I'm showing over here, or even other BI tools.
Now, with these new integrations,
our stakeholders can work in the tools
they're most comfortable with while still giving
our analytics team the confidence
that these metrics are accurate, since they're all
coming from our universal semantic model.
GABE WEISS: We can also embed insights
to their direct workflows, removing BI swivel chair
operations where users need to look at a dashboard
and then take action on another screen.
Similarly, for our application engineers,
we're going to leverage that same universal semantic layer.
And from there, we can share our insights
from our data using feature-rich APIs, extension frameworks,
or embedded analytics.
LEIGHA JARETT: And not only does this model give us
confidence in our data, but it also
ensures that all our information is securely managed,
including row-level security.
This means that when our Cymbal Solar app users query data,
they only see what they're supposed
to see-- so their own data.
GABE WEISS: And last, Looker enables
us to serve our customer across clouds and databases using
the universal semantic layer.
We can access, govern, serve, and take action
on any data that's available using a SQL connection
and provide insights from across the data cloud.
For Cymbal, this allows us to create a differentiated data
product.
We can increase app stickiness and provide premium product
offerings for customers to learn from and act
on all the valuable insights that we've created.
LEIGHA JARETT: Exactly.
OK, so to wrap things up, let's quickly
review Cymbal's data cloud architecture.
It starts with data that we collect from our products
and within our applications, which
then land in our data lake transactional databases
and other clouds.
That data is then unified inside of BigQuery.
GABE WEISS: Next, our analysts and data scientists
can work with the data using serverless platforms
like Serverless Spark and Vertex AI
so they can focus on the right development tasks.
And last, we leverage Looker to democratize insights and allow
both internal and external stakeholders to take action,
closing the data feedback loop.
All of this is made possible and simple with Google Cloud.
LEIGHA JARETT: So that's a wrap on our exploration of the Data
Cloud made simple.
Thanks to all of our data engineers, scientists,
analysts, and professionals who are able to join us
for the Data Cloud Summit.
And data on.
[MUSIC PLAYING]