[MUSIC PLAYING]
GABE WEISS: Hi, In this video, I'll
introduce you to Datastream.
Then I'll walk through setting up a stream from Oracle
to Cloud Storage.
I'm Gabe Weiss, a developer advocate for Google Cloud.
Datastream is a serverless Change Data Capture, or CDC,
and replication service.
Datastream makes it easy to synchronize
data across heterogeneous databases and applications
reliably and with minimal latency.
Datastream provides a simple, integrated experience
to guide you through every step of the process.
It's serverless, so you don't need
to worry about managing or monitoring instances.
And continuous data replication means
you can unify data across multiple sources
and applications and make it accessible in real time.
It helps power a wide range of use cases,
including data replication for analytics, database migrations,
event driven architectures, AI and machine learning, data
lakes, and many more.
Whether your source database is on premises,
self-hosted in Google Cloud, or running on another Cloud,
Datastream helps you quickly unlock
the benefits of real time CDC.
Now let's walk through the Datastream experience.
You can easily find Datastream by searching for it
in the Search box at the top of the Cloud platform console.
You can also find it from the Google Cloud console navigation
under Big Data.
To get started, you'll create a stream.
This represents the end-to-end process
of streaming changes from your source database
to your destination.
First, let's define some basic information about the stream.
I want to ingest data from an Oracle database
I have running on GCE to a Cloud Storage destination.
Once I've defined my source and destination types,
Datastream shows me the configuration required
for successful streaming.
Datastream is unique in that it reflects all of the source
and destination configuration requirements
directly within the creation process
to take out the guesswork involved.
For example, since I chose Oracle as my source,
I can see the requirements for setting up
the necessary login configurations
and permissions on Oracle.
After confirming the necessary prerequisites,
I'm ready to define my source.
I can do this by creating a connection profile, a resource
that represents the information needed
to connect to a database.
These profiles aren't locked to an individual stream.
This means I can reuse profiles if I want to,
and also allows me to hand off creating the source
configuration to someone else in my organization
who's in charge of connecting to the database.
I've already created a connection profile
for an Oracle database running on Google Cloud's VM,
so I'll select it.
Whether I create my source connection profile now
or use the one I create in advance,
I can test connectivity to make sure the data
stream is able to reach it.
Datastream supports both public and private connectivity
methods to easily and securely connect to sources,
whether they're hosted, be it on Google Cloud,
on premise, or on other Cloud providers.
Once the connectivity test passes,
I can continue to configure the stream for my source.
Next, I want to select the objects that this stream should
pull from my source.
Using the selection UI, can define
which schemas and tables to stream from my Oracle source.
I will select the entire HR schema and a couple of tables
from the movie schema.
For the objects I selected, I also
need to bring over the historical data.
So I leave the backfill historical data enabled.
Now I'm ready to define the destination for my stream.
Here again, I can create a new connection profile
for my destination or use an existing one.
I'll select the GCS bucket I have ready to go.
Now, Datastream allows me to specify a path within my Cloud
Storage bucket where the streamed data will be stored.
For this demo, I want files to be written to the root
folder of the bucket.
Datastream allows me to find the right balance between file size
and data freshness that's right for my use case.
It's important for me to have my data available quickly,
so I'll reduce the writing delay to destination to 15 seconds.
This means the files written to GCS will be smaller,
but the data will be available faster.
Well, that was easy.
I've connected and configured my stream.
All that's left is to validate the end-to-end stream
configuration and start moving data.
Validation passed successfully.
I can trust that my stream will run smoothly.
I'll create and start the stream to run it immediately.
Once my stream is created, I can monitor its progress
and see if it encounters any issues.
Datastream will perform the historical backfill
and continuously replicate changes as they happen.
I can monitor different metrics of the stream,
such as throughput, latencies, and more.
That's it.
I've quickly and easily shown you
how to continuously ingest changes from an Oracle database
into Cloud Storage using Datastream.
And since Datastream is serverless,
it'll automatically scale up and down to handle my data volume
and maintain minimal latency.
With change data now continuously streaming
into Cloud Storage, I can easily leverage products
like Dataflow to process the data as needed.
For example, I could use a purpose
built template to replicate the data into BigQuery or Cloud
SQL for Postgres.
Datastream is the fast, easy, and reliable way
to seamlessly deliver change data
streams from your database to Google Cloud
to support Analytics, database replication, and event driven
architectures.
It's serverless, high performance, and easy to use.
Now, it's your turn to give it a try.
You can get started from the Google Cloud console or learn
more at cloud.google.com/datastream.
Thanks for watching.