DEREK DOWNEY: Hi, I'm Derek Downey, a Developer Advocate
for Google Cloud.
I am excited to share some great news if you are using Cloud
Spanner and have downstream systems that
need to be aware of data changes in near real time.
Getting data changes from one system into another,
especially in real time, can be challenging.
To address this, Cloud Spanner just
released Change Streams to make it
easy to set up pipelines to process changes made
to your Spanner data.
In this video, I will explain what change streams are
and what is required to set them up.
Then I will show you how easy it is to set up a change stream
and a pipeline to process changes
made to your Spanner data into a downstream BigQuery analytics
environment.
Finally, I'll go over some typical use cases for change
streams and how Spanner makes it easy to enable those use cases.
[MUSIC PLAYING]
So, what are change streams?
If you have ever heard of change data capture, or CDC, that's
a good starting point for understanding change
streams in Cloud Spanner.
Change streams keep track of changes
made to data through DML or Spanner mutations that can then
be processed by pipelines to publish in a downstream system.
There are many reasons you might want
to get data changes from one system into another.
You might have a data warehouse and you
need real time information to improve business decisions.
You might want to trigger downstream application
workflows, for instance, restocking an item
if the inventory level drops beneath a threshold.
You might have a need to keep an audit trail of changes
in a separate system for compliance purposes,
or perhaps you want to keep track of changes
for disaster recovery.
Whatever the reason, the process to set up a change stream
in Spanner is straightforward.
First, create the change stream in your database.
A change stream is another type of object in Spanner,
like a table or a secondary index.
So you can add, modify, or delete change stream objects
using DDL.
You can create a single change stream
for all tables in a database, or you
can create a change stream for specific tables or even
specific columns within those tables.
Second, you need a destination to receive the changes.
This can be BigQuery, Google Cloud Storage, Google Pub/Sub,
or any other system to receive the stream of changes.
And finally, you set up a pipeline
to process change events from the change stream and Spanner
and publish to your downstream system.
You can easily build data flow pipelines
to process the change stream events,
leveraging Spanner I/O from the Apache Beam repository,
or you can connect directly to the Spanner API
and write your own processing logic.
This might be best to explain with an example.
I currently have an inventory service for an e-commerce store
that is backed by Spanner.
I also have BigQueries set up to track inventory behavior
over time.
Having this analytics environment
has enabled me to make good decisions
on what items to keep in stock.
Currently, to get data from Spanner into BigQuery,
I had to set up and maintain additional pipelines
to extract the data in batches before loading into BigQuery.
These batches cause some delays that
have caused headaches for my analysts.
With change streams, I can move to almost real time streaming
of data changes into BigQuery and simplify
the logic for processing.
To set up my change stream, I go into the Cloud Console
page for my Spanner instance.
Then I add my change stream via the DDL page.
The syntax is fairly straightforward.
You need a change stream name.
Change stream names are unique to the Spanner database.
Next, you specify the entities you want
to track in this change stream.
You can track the entire database, a set of tables,
or specific columns for tables.
Last, you specify the retention period for change stream data.
The records that are tracked are stored
for this retention period before automatically cleaning up.
By default, the retention period is 24 hours,
but can range up to 7 days.
And you'll pay only for the additional storage required
to keep the change events longer.
Now, I also have my BigQuery environment set up
that is used for inventory analytics.
All I have to do is set up the Dataflow pipeline for BigQuery.
So, I go into Dataflow in the Cloud Console,
then create a new pipeline.
I provide the pipeline and name.
The region I select is "us-central1"
to be near my Spanner instance.
And then I select the Cloud Spanner to BigQuery template.
This template leverages the Apache Beam Spanner I/O
connector.
Here I need to provide some more information.
First is the Spanner instance name and database name.
This is the location of the change stream
object I created earlier.
Next, I need to provide a metadata instance and database
name.
This is needed for the Spanner I/O connector
to keep track of the changes it has processed.
The metadata instance and database
can be the same as where the change stream is located,
but I'd recommend at least storing the metadata
in its own database.
Once we've provided the metadata location,
then we provide the change stream name.
And finally, I specify a BigQuery data
set to store the data changes.
There are some optional parameters,
but this is enough to get started
processing the change streams from when
the pipeline is created.
That's it.
After a few minutes for the pipeline to get set up,
I will start receiving data changes into BigQuery.
As part of releasing change streams,
the Spanner team has also released Spanner I/O
to the Apache Beam repository.
Spanner I/O can be used to process data change events
from Spanner change streams.
There are also three main Dataflow pipeline templates
available that leverage Spanner I/O to quickly
get started processing Spanner change streams.
The first is the BigQuery template
that I just walked through.
This allows you to set up BigQuery
to receive changes from Spanner in near real time.
There is also a Google Cloud Storage template.
One example of sending data changes to Cloud Storage
would be to keep an audit archive of detail change events
on important data for regulatory purposes.
And finally, there is a template to publish data changes
to Pub/Sub.
This is useful to trigger workflows
in external applications that are
listening on a downstream Pub/Sub related
to those changes.
If you need to publish data changes to another destination
or if you need additional processing of the data changes,
then you can create a custom pipeline
that leverages the Apache Beam Spanner I/O connector.
It is also possible to connect directly to the change stream
API to create a customized data pipeline without using
the Spanner I/O connector.
It's possible to have multiple change
streams in the same database and even multiple change
streams tracking the same tables or columns.
Here are some things to keep in mind with multiple change
streams.
You can have up to 10 change streams per database
and up to 3 change streams can track the same non-key column.
You should maintain no more than 5 concurrent readers
for a change stream partition or performance may degrade.
Change streams is a great new feature in Spanner
that enables real time data pipelines
based on data changes.
I have shown how to set up a basic pipeline
to get Spanner data changes into BigQuery
using the available Dataflow template.
There are many ways change streams can be leveraged,
from triggering downstream application workflows using
Pub/Sub, storing data within Cloud Storage
to enable data recovery, or for setting up an audit
trail for compliance purposes.
Let us know in the comments how you're using change streams,
and if there are other templates you'd like
us to provide in the future.
[MUSIC PLAYING]