[MUSIC PLAYING]
GURMEET GOINDI: Welcome to a session
about building awesome data-driven applications using
the latest database services that Google Cloud has to offer.
My name is Gurmeet Goindi.
I go by GG and have the honor of having my colleague join me
in this journey, Yoav.
YOAV EILAT: Good to see you, GG.
GURMEET GOINDI: Likewise.
YOAV EILAT: So let's explain what
we mean by data-driven apps because it's
a pretty good assumption that every app reads and writes
data of some kind.
But I suspect many of us could do more to analyze that data
and make it useful.
We want to look at three representative examples.
Example one is fraud detection for credit card transactions.
Let's say there's a purchase of a gym membership on my credit
card, and the credit card company
wants to check if it's legit.
GURMEET GOINDI: When did you go to gym last, Yoav?
YOAV EILAT: Well, you know, I'm planning to do more exercise.
GURMEET GOINDI: OK.
YOAV EILAT: The point here is to collect all the data that
can be useful, what the credit card
company knows about gym purchases
and what it knows about me.
The second example is a reporting example.
Let's say a DVD store wants to see which DVD
is selling better, "Avatar--
The Way of Water" or "Top Gun--
Maverick."
GURMEET GOINDI: Did you say DVD?
Have you heard of this thing called streaming before?
YOAV EILAT: Well, our team likes old fashioned examples.
So we want to put all the sales data in one place,
and then we can run any report that we want.
Example 3 is pricing analysis.
Let's say that I'm planning a coupon to increase sales.
I want it to increase profits not decrease profit, right?
So what's better, 1% off or 2% off?
In this case, the example is about running
analysis on my data based on some statistical model.
GURMEET GOINDI: If you look at where the data is coming
from today, traditionally, developers
had to work with two separate types of stores, one
for handling transactional applications that
run the business, and the other is a data warehouse
to provide insights on how well the business is doing.
YOAV EILAT: And what about data lakes and lake houses
and object storage and all of those things?
GURMEET GOINDI: Yes, there are many places
where data could be.
But let's start by focusing on the very common scenario
that we have here.
It actually works pretty well, but it also
comes with the overhead of managing, monitoring,
and keeping these stores in sync.
And let's not forget about security.
I heard it's important.
But with the advancement in Cloud technologies,
the boundaries between these stores are blurring.
And that provides us the opportunity
to make things easier for our customers.
YOAV EILAT: So speaking of that, we actually
made some predictions about that.
I'm sure all of you read every Google Cloud blog,
so you could not have possibly missed these predictions
we made in December.
But just to recap, our technology experts
made 10 predictions for the next few years of IT.
You can see the link in the description below.
There were two predictions that we
want to call out because they're very relevant to what we're
discussing here, making decisions
based on your application data.
The one on the left was from Andy Goodman's
the VP and GM for database products at Google Cloud.
He predicted that the barriers between transactional and
analytical workloads will mostly disappear,
the two locations that GG just mentioned.
The one on the right was from Richard Seroter
of our developer relations and outbound PM team.
He predicted that multicloud technologies
will make it possible to freely switch your primary Cloud
provider.
So if you're not able or not willing to put everything
on one Cloud, you'll have the means to move it anywhere
you want.
Obviously, this also applies to your data.
It can be generated by multiple apps on multiple clouds
or outside the Cloud.
And you should be able to analyze it wherever you want.
So let's start with the first prediction.
Let's say that I have a database, probably
many databases, and a data warehouse.
How do I analyze data that's coming from both sides?
GURMEET GOINDI: Let's go knock down some barriers.
The three examples that we started with just
happen to match the three best ways to integrate
databases and data warehouses.
YOAV EILAT: Wow, what are the odds?
It's a perfect match.
GURMEET GOINDI: No match fixing here at all.
So in the first example, where we fly credit card
transactions, we will use federated queries,
meaning that we will query across database and the data
warehouse.
In the second example, where we run report from our DVD sales,
we will be streaming replication,
meaning that you will stream data from our databases
to your data warehouse.
YOAV EILAT: I do know what streaming is, GG.
GURMEET GOINDI: I'm glad.
I'm glad to hear that.
And in the third example, where we do a pricing analysis,
we will use integrated analytics,
which means we can skip the data warehouse
because your database is really good at running
analytical queries.
With that, let's dive in.
From a database nerd like me, federated queries
are really exciting.
We are bringing to fruition of how
you can gather meaningful insights by combining
historical data that's been sitting in your data warehouse
systems, such as BigQuery, for data that is still
being created within your transactional system,
such as Cloud Spanner or Cloud SQL.
YOAV EILAT: So I see that this query combines customer data
from BigQuery with sales order data from Cloud SQL.
It seems to be the canonical example of a federated query.
GURMEET GOINDI: I'm sure it is.
Even if I ask Bart about federated queries,
it gave exactly the same examples.
If you want to correlate customer buying behavior,
say, with their most recently placed orders,
you can do that by joining the customer buying
behavior data stored in BigQuery with the custom
order tables stored in Cloud SQL for Postgres
by exuding a simple external query in BigQuery
to call into Cloud SQL.
And voila, you have your answer, no pipelines, no ETL jobs,
no data movement.
And we even push down SQL processing
to Cloud SQL from BigQuery.
The cool thing is that the data doesn't need to be moved.
It's queried all in place.
Also, check out federated queries
between BigTable and BigQuery, which
uses a very similar mechanism called external tables.
YOAV EILAT: OK, let's move on to the second method.
This one is based on replicating the data, or at least some
of it, from our operational database to our data warehouse.
We do that by streaming every change as it happens.
On Google Cloud, you can use data stream, a Cloud service
that, you know, streams data.
It's super easy to set up.
You define which data sources to stream from
and where to stream to.
And it starts to securely replicate your data.
It's completely serverless and even tracks schema changes
in your database.
You'll notice in the datastream screenshot
that we're replicating from an AlloyDB database to BigQuery.
We're replicating all the information
about our DVD sales, actors, films, customer data, inventory
information, and all of that.
GURMEET GOINDI: Amazing!
It makes me wonder, when is streaming replication better
than federated queries?
YOAV EILAT: I would say that streaming is a good solution
if we don't want to decide ahead of time which
reports we're going to run.
Maybe I want to compare two movies today,
like "Avatar" versus "Top Gun," but tomorrow I'll
want to run a report about inventory levels or something
else.
So I just want all the data in BigQuery,
and then I can run any report that I want.
And the data will be fresh because it's streamed
continuously into BigQuery.
GURMEET GOINDI: Well, I know who not
to take the movie theaters with me given your movie choices.
Let's take this one step further.
What about a world where you can run meaningful analytics
or, one might say, operational analytics right
against a transaction database, no streaming
changes, no setting up pipelines,
no federation of anything?
With that in mind, we developed the columnar engine
for our AlloyDB database.
AlloyDB's columnar engine makes analytical queries
run up to 100x faster by keeping frequently queried data
in an in-memory columnar format for faster scans joins
and aggregation.
It's right there in the database for you to use.
YOAV EILAT: OK, let's put the integrated analytics aside
for a moment.
Can you tell us more about what AlloyDB is?
GURMEET GOINDI: Would love to.
AlloyDB is our fully managed, Postgres compatible database
that we built to help you cost effectively
modernize from legacy databases or scale your existing Postgres
apps.
It's Google's newest Cloud database
with all the goodness of Postgres
and all the goodness of Google technology.
And in our performance test, AlloyDB
is four times faster than standard Postgres
and two times faster than Amazon's comparable Postgres,
compatible service for transactional workloads.
YOAV EILAT: I think I know which service you mean, GG.
GURMEET GOINDI: I'm sure you do.
Yes, that one.
And AlloyDB is up to a hundred times
faster for analytical queries and scales
to thousands of [INAUDIBLE] for read workloads.
It also provides a four nines, no-nonsense SLA, which
is inclusive of maintenance.
And we took a hard look at how we can remove or reduce
toil for Postgres developers and brought
in a ton of machine-learning algorithms
for vacuum management, memory management,
and how you move data across caching tiers and storage
tiers.
Lastly, we integrated a Vertex AI service
so developers can invoke Vertex AI models straight from the SQL
query.
YOAV EILAT: Nice.
So let's go back to the other prediction about multicloud.
I think we presented some good options today
for running data-driven apps on Google Cloud.
But it breaks my heart to admit that some databases are not
on Google Cloud.
And some analytics don't run there either.
Maybe they never will.
You've seen several scenarios, GG,
where they need to run in other places, right?
GURMEET GOINDI: Correct.
Sometimes people can't move to the Cloud as fast as they want.
Their workloads are restricted to on-premises data
centers because of the complexity
of their applications, the complexity of their network
topologies, or the need to run multiple clouds,
or need to run a disconnected fashion
due to regulatory or data sovereignty requirements.
Or they may be running their applications at the edge,
like a retail store or a telco facility.
Customers tell us that they want to eliminate
unfriendly licensing relationships, expensive fees,
vendor lock-in faster than they're able to deplatform
everything to Cloud.
And they asked us to find a path to support them
in their in-place modernization journey with AlloyDB.
They also want us to support a tier one provider like Google
when they are on this modernization
journey with AlloyDB.
YOAV EILAT: So with that, we're really
excited to tell you about AlloyDB
Omni, a downloadable version of AlloyDB
that you can run anywhere.
We introduced it a few weeks ago but wanted
to give some more information today
that will be useful for application developers.
AlloyDB Omni uses the same engine
as the Cloud version of AlloyDB, and it's still
100% compatible with Postgres.
It's based on the Postgres kernel
and uses the same wire protocol.
And it supports Postgres extensions and flags.
But it's more than two times faster than Postgres
for transactional workloads, and it's up to a hundred times
faster for analytical queries because it includes the AlloyDB
columnar engine.
GURMEET GOINDI: Wait, Yoav, did you see a downloadable version?
Aren't we like a Cloud company?
YOAV EILAT: We are, but we want to meet people where they are.
Sometimes you need to modernize an older application
without moving it to the Cloud.
And it's not your imagination.
We actually have AWS and Azure logos here.
AlloyDB Omni will run great on those clouds.
GURMEET GOINDI: Yeah, I was wondering about those logos.
I also noticed that the slide says more than 2x
faster than Postgres.
But AlloyDB in the Cloud is 4x faster than Postgres.
YOAV EILAT: Thanks for paying attention.
It's because of the difference between the high-performance
storage that we have in Google Cloud
and the storage available on premises.
The Cloud scale is so much better.
Still, it's a huge performance gain in both cases.
And in terms of price, AlloyDB Omni
is an enterprise-level database at a fraction of the cost
of legacy databases.
So it's a great way to modernize applications to Postgres.
GURMEET GOINDI: You might be wondering
what the installation is like.
Nobody wants to go back to the old-school installation
scripts.
We tried to make AlloyDB Omni as easy
to install and use as possible.
It's packaged as a container.
All one needs to do is install Docker,
pull the container repository, download and run the installer.
And there you have a commercial database
up and running to handle some of your most
demanding applications under five minutes.
To patch a database, just repeat steps 2 to 4,
and you're good to go.
YOAV EILAT: And now that you have a downloadable database
that runs analytical queries up to a hundred times faster,
your decision making is much closer to real time.
You can get business insights from your operational data
by running queries directly on the database.
And by the way, I was looking for a way
to illustrate 100x difference in speed,
so I looked up the speed of a Boeing 787 on Wikipedia.
And it has a maximum speed of 690 miles per hour.
A good runner can do about 6.9 miles
per hour, which is equivalent to finishing a marathon in just
under four hours.
GURMEET GOINDI: Yeah, we don't want our customer queries
to feel like a real marathon.
YOAV EILAT: Exactly.
GURMEET GOINDI: Now, Yoav, you may
wonder how we do this, how we make the analytics go 100x
faster.
So let's take a deeper look there.
We implemented the engine as a Postgres extension,
so it's transparent to your apps.
And they don't need any code changes to use it.
Secondly, the machine-learning algorithms
moves the data between DRAM, the ultrafast cache,
and the storage, depending on your usage pattern,
giving you the performance of memory and the scale
and economics of persistent storage.
Processes for a long time have been
shipping with SIMD instruction set, which
makes vectorized operations go really fast.
Now, if these operations are executed
against data stored in in-memory columns,
then we can speed up those operations
by a couple of orders of magnitude.
AlloyDB builds on these hardware enhancements
and brings this technology to Postgres
and extends to AlloyDB Omni, where
you can extract a lot more value out of a CPU
no matter where you run your workload.
We also took this opportunity to bring
in some of the unique techniques like storing min/max values
and learn metadata alongside the data stored in the columnar
structure to further give you an intelligent response
from the columnar engine.
These techniques for a long time were only available
in expensive commercial databases.
And all of the other benefits with the added advantage
of Google machine learning makes the user experience
simple and transparent.
YOAV EILAT: So all of that is happening behind the scenes,
and I don't need to do anything?
GURMEET GOINDI: Exactly.
Let me give you an example of a machine learning
algorithm in AlloyDB.
The AlloyDB memory manager uses machine learning
to predict your memory usage and automatically adjusts memory
as required for your OS or system task
or through the database, almost completely
preventing the dreaded out-of-memory events, or ooms.
Or you might also say, no more ooms and no more oops.
As with all things ML, these algorithms
will keep getting better over time.
But don't take my word for it.
Let's do a quick demo, courtesy of our colleague Randall.
And remember your example you have of analyzing discounts,
1% is better or 2%?
Let's run that one on AlloyDB Omni's columnar engine
and see what we get.
[MUSIC PLAYING]
RANDALL: Well, thank you, and welcome to this demonstration
of how fast analytics are on AlloyDB Omni.
AlloyDB Omni is Postgres compatible.
You can see here I'm running PG admin, which
is an advanced Postgres application for managing
Postgres.
I'm managing several Postgres instances,
including an AlloyDB instance on our Cloud service.
And this instance, which is actually
running on my Windows laptop with a Debian Linux
virtual machine, this is the platform
where I'm going to be doing this demo.
You don't need a big hardware platform to run AlloyDB Omni.
It'll run anywhere you want, including on the Edge.
So for this demonstration, I loaded a TPCH database
with the free Hammer DB tool.
I loaded over a billion rows into the database.
You could run this very same benchmark using the same tools.
The scenario is that I'm running an online store that's very,
very busy on Black Friday.
Thousands of orders are being entered.
I want to find out right now if I give a larger discount,
will I actually get more revenue?
All right, so let's run this query
and find out how long it takes to run without the columnar
engine.
And as you can see, it is running.
It's going to take a long time.
I'm going to use the miracle of video editing
to move forward into the future.
So you don't have to wait for the query to finish.
OK, we're back.
The query ran in quite a long period of time.
It ran in 9 minutes and 56 seconds.
That's pretty long.
And the result, though, was encouraging.
When I offered a 2% discount, I did indeed get more revenue
than with a 1% discount.
Well, something else happened while we were waiting
for the query to finish.
There was an outage, and priority one trouble tickets
were filed.
Users could no longer enter new orders,
and I caused it by running these analytical queries.
Let's take a look at the top command in Linux.
Very little of the CPU is available, maybe 12%.
And there's massive waiting for I/O.
So now, let's see how fast this query
runs with the columnar engine.
I don't need to make any code changes.
It automatically uses the columnar engine
if it's appropriate.
OK, how fast is it going to run?
The other query without the columnar engine
ran in over nine minutes.
Let's see.
Click, boom!
Less than a second.
There's even more good news.
I don't have to have indexes on these columns.
So if I do have them for analytics,
I can drop those indexes.
And my transactions get faster in the system.
So not only is the columnar engine really fast,
it can make the entire database faster.
And with AlloyDB Omni, it actually
gets faster the more you use it due
to the autopilot machine-learning algorithms
that are managing the database.
Well, AlloyDB Omni sure is fast.
Give it a try.
You can download it right now.
Thanks so much for watching.
[MUSIC PLAYING]
GURMEET GOINDI: You can always take it for a spin
because AlloyDB Omni is free for developer use.
And as we said earlier, it's really easy to get set up
and get started.
YOAV EILAT: So, if you're eager to give this product a try,
this is how.
We put a link in the description below.
Go there, sign up, and give us your feedback.
Now you can make both of the predictions
we talked about come true.
Remove the barriers between transactional and analytical
workloads, and run the same database, AlloyDB, everywhere,
on and off Google Cloud.
And before we go, we also want to invite
you to check out our entire database product line.
We mentioned several services today, Cloud SQL, AlloyDB,
Spanner, BigTable, Datastream.
Those are all relational or key value.
You can explore additional options here.
There are in-memory and document database options and a database
migration service to help you migrate from other databases
to Google Cloud.
GURMEET GOINDI: Thanks, everyone,
for joining our session.
It's always fun for us to talk about databases
and the innovation that is happening in this field.
We hope these spark some ideas and concepts
that you can leverage in your application development
journey.
Have a great Google I/O, and see you at Google Next.
YOAV EILAT: Bye-bye.
[MUSIC PLAYING]