KRISTEN O'LEARY: All right, everyone.
Thank you for joining.
And welcome to Migrating Cassandra Workloads
to Cloud Bigtable.
Brief introductions.
My name is Kristen O'Leary.
I'm a developer relations engineer.
And I focus on Bigtable client libraries.
BILLY JACOBSON: And I'm Billy Jacobson.
I'm also a developer relations engineer
focusing on Cloud Bigtable, sample code, and tutorials.
So Kristen's going to kick things off in this presentation
by talking about why you might want to migrate
to Bigtable from Cassandra.
And different application structures you might have that
are fairly common in Cassandra.
Then with all those applications and potential applications
there, she's going to walk you through how you would migrate
each of those to Bigtable.
Then once that's done, I'm going to walk you
through all the code and sample resources
we have available to make that process as easy as possible.
So Kristen, why don't you kick things off?
KRISTEN O'LEARY: Thanks, Billy.
So we're going to start with reasons
why you may want to migrate from Cassandra to Bigtable.
So we're going to focus on just two different examples.
The first is maybe you're currently
struggling to maintain availability, durability,
and performance of your current existing distributed database.
So some more examples include maybe you're
struggling with stability at scale.
Or maybe there's just too much operational load on your SREs.
Or you're just generally dealing with overhead
of managing replication, repair, and things
like garbage collection.
A second point could be you may want to move up the tech stack
and spend less time and money on things
like managing the different complexities of a distributed
database.
So examples here include maybe you
want to reduce the total cost of operation
of your infrastructure and do things
that were different than before and things
that weren't possible before.
So now we're going to walk into some technical steps.
So of course step 0 here is you need
to plan for your migration.
So before you dive into a migration from Cassandra
to Bigtable, it's really important to deep dive
into Cassandra use cases and gain a better understanding
of how Bigtable and how they're the same
and how they're different.
You should do things like map concepts, schemas, and queries
from Cassandra to Cloud Bigtable.
A quick tip--
Cassandra primary key, you can look
at as the same as equal as a Bigtable row key.
That's a great starting assumption.
Billy is going to talk about that later.
But that's a great place to start
when you're looking and thinking about this type of migration.
Next, we're going to walk through an example
of a migration flow.
So step 1, you can look into doing dual writes.
So you're currently obviously writing
into your Cassandra database.
Now you want to try and write both
into Cassandra and Bigtable.
So any new data will go into both databases.
Step 2, now that you have new data in Cloud Bigtable,
you want to copy over the old data from Cassandra into it.
So you want to do copying of existing data
from Cassandra into Cloud Bigtable.
Step 3, option, you can compare between the two.
So make sure that your data is consistent between Cassandra
and Bigtable.
Step 4, redirect you reads.
So we're writing into Bigtable.
And now that the data should be equivalent to what
we had in the Cassandra database,
we can also start reading into Bigtable.
So redirect your reads from Cassandra to Cloud Bigtable.
Step 5, stop your dual writes.
So now we're basically using Bigtable to read and to write.
No need to dual write anymore.
So you can turn off the writing into your Cassandra database
and only focus on Bigtable.
And Step 6, now you're done.
And you can decommission Cassandra.
So next up is migration paths.
We're going to talk through four different migration paths
that you may encounter and how to proceed with them.
So here are some examples of common Cassandra migration
paths.
And they each have different migration strategies.
Again, we'll look at them more in detail on each side.
We're going to start with batch load, global serve.
So you may have an application where
you load data into one region.
And then you may want to replicate the data out to other
serving regions.
So in this example, the first thing we would want to do
is rewrite our batch jobs to support Bigtable.
So this job that does all of the writing,
you now want to support Bigtable as well as Cassandra.
Secondly, you want to rewrite your API to support Bigtable.
So instead of just writing to Cassandra, now
we're writing to Bigtable.
And now we want to start changing API to do things
like read as well.
Step 3, you want to run your batch jobs in parallel
to load the data, both into Cassandra and Bigtable.
And then finally, once you're done,
and once you feel that data is equivalent,
you can release to switch your API layer from Cassandra
to Bigtable.
So you'll see here in the diagram,
first we had just the Cassandra instance.
And now we're writing to Bigtable as well
to serve the end users.
So next step is when downtime is acceptable.
This is another common application instance
where your application you may be able to accept some downtime
and have a maintenance window.
So this is an example of how you can do
the migration during that time.
So first up, you want to stop writing to Cassandra
during your migration window.
And then you just want to do a big batch copy from Cassandra
to Bigtable.
And so once you've done that, the data should be equivalent.
And you can optionally compare to ensure
that your data is correct.
And then switch over your serving layer entirely
to Bigtable.
So now your API will talk to Bigtable instead.
And then you want to decommission Cassandra.
So you can see here in the diagram.
We've now switched over the API layer
to talk to Bigtable as well.
Next is online with time to live.
So this is another common workload
where let's say you have something like a cache or user
sessions where you have data that eventually will expire.
So we'll use that to our benefit here.
So that, first up, of course, we want
to rewrite our code again so that we're writing data
into both Cassandra and Bigtable.
So now we're getting all of our new data
into the table as well.
And then you want to wait until your time to live expires.
So once your data that was only in Cassandra has expired,
the two databases should be equivalent.
Again, you can do things like do asynchronous
callbacks to verify that the data is
consistent between the two.
And then finally you can decommission Cassandra.
So again, it's the same diagram, we've
written to both Cassandra and Bigtable,
and then eventually do the time to live.
Both sets will become equivalent.
And last but not least, we have online with persistent data.
This is a workload that's user-facing,
semi-transactional, can't accept any type of downtime.
So how do we work in that case?
So you want to, again, rewrite your API layer
to read and write to both systems.
And then ensure there's managed consistency in the API layer
just in case something is down.
You want to batch copy and back fill your data into Bigtable
because, again, you can't accept any downtime.
And then you want to do asynchronous callbacks
between the two--
again, and then we keep saying this--
to make sure that your data is consistent and equivalent.
And then once you're satisfied with the consistency
of your data, you can shut down Cassandra.
So we've talked about a lot of different situations
and examples.
Now Billy is going to take us through some of the actual code
that you will use if you want to do a migration from Cassandra
to Bigtable.
BILLY JACOBSON: Thank you so much, Kristen.
We've got a lot of starting points from there.
So from those paths you were going to take,
there are two major aspects to your migration--
migrating your queries and doing your batch data migration.
So let's first look at migrating queries.
And if you're using Cassandra or any kind of database,
you have a lot of queries you're going to want to migrate.
So you're managing your tables with queries,
any kinds of garbage collection with queries.
You're writing your data.
You're reading data.
And you might also have analysts that
need to perform table-wide analytics
and are more comfortable with SQL
than some of the programming languages
that we have client libraries for.
So I'm going to walk you through a bunch of examples.
And they are all available later in this codelab,
Cloud Bigtable for Cassandra users.
You can learn the basics of migrating
a lot of these queries.
And all the code that's in these slides will be available there.
So don't feel that you need to pause the video
to figure out what the code is.
You can find the link later.
So let's look at an example database
just to make things a little bit more
concrete through these examples.
So we're storing data about mobile devices in a time
series.
So we have information about their cellular plan,
how much data each phone has, if it's connected to the Wi-Fi,
connected to a cell tower, and which operating system it's on.
And you might have a lot more data in here.
We also are storing the device type, a device ID, and the date
that that data is stored on.
And we could store that in the row.
But also we've upgraded it into our row key
because this is where we are doing our queries from.
This is very similar to our Cassandra primary key.
So all your queries for Bigtable are going
to be based on your row key.
Kristin mentioned earlier your primary key
can be pretty much translated into your row key
by just concatenating the important parts of it.
So here we have device type, device ID, and date.
And Bigtable queries are based on row gets,
based on the row key and row range scans,
based on the row keys as well.
So these features are really going
to determine which queries are going
to be the most performant.
So to get started with Bigtable, you're
probably going to use our command line utility, which is
similar to the Cassandra shell.
And this is called the CBT tool.
Here I'm showing you how to create a table with the CBT
tool.
And you'll notice with Cassandra,
you're used to defining all the columns in advance,
defining your primary key.
And we don't do that in Bigtable.
We just defined the table name and any column families.
And you can define your garbage collection rules
at the column family level.
So the CBT tool is great for creating your tables,
getting everything set up.
But you can also use it for reading and writing
data, deleting data, managing your app profiles,
managing your garbage collection.
And if you have any scripts for database automation
that you have set up, maybe using Cassandra shell
or anything like that, the CBT tool
is a great way to just swap those out
if you're switching over to Bigtable.
But for your application, you're probably
going to want to use one of our client libraries.
We have our client libraries available in eight programming
languages.
Go, Java, Python, Node.js, just to name to few.
And we have documented all the different ways
you're probably going to want to use Bigtable there.
So we're going to just look at the Java client.
But again, all those other client libraries
will be well-documented with examples similar to these.
So here if we're writing data, in Cassandra
you're used to doing an insert.
You're going to specify the columns and their values.
Where, Bigtable, we do writes as a mutation.
So we'll take our row key and create a mutation
from that where we set a cell's value so the cells are
in intersection of the row key, the column family, and column.
And then we provide a value for it.
And you'll also note here, we're providing a timestamp.
Bigtable will let you store multiple values
at different versions in a cell.
And you can use that as part of your time series storage
strategy.
And with garbage collection, you can
determine how many versions are kept
or how old those versions are.
So one of the parts of your migration
is going to be dual writes potentially.
So to do that, you're going to pick
your client library of choice.
And then alongside any of your insertions,
you're going to create a mutation
to match that alongside of it.
So now let's look at two select statements.
Here, we're just selecting one row,
because we're selecting on all the parts
of the primary keyword.
So we've got the device type, the device ID, and a date.
And in Bigtable, we're just going
to translate that into a row key by concatenating
all that information.
And here we're using the pound symbol
to separate each piece of information.
But you can really use any symbol you want.
So we take our row key and then we just perform a get.
And you can do whatever you want with the data from there.
So if you wanted to verify that your data had been migrated
correctly, you could query Cassandra and Bigtable
at the same time, wrap those in an asynchronous callback block,
and then log any deviations that you've
got until everything is matching or you've sort
out any of those errors.
So you can also do this for multiple rows.
Here, we're scanning for all the dates or all the dates
after 2019, or just 2020 at this point, for a specific phone.
So with Bigtable, we can do a prefix scan for that.
We're going to, instead of doing the full row,
we just take part of the row key.
So we're taking the device type phone and the device ID,
which is 4C41, or whatever it is.
And then we're just taking the year
because all the data for the device that's in 2020
will have that as its prefix for their scan.
So we pass that prefix to the scan.
And then we can get all the rows for that.
And you can use this in multiple ways.
So let's say you want to get all the data for 2020 and 2019,
you can create a scan which is two prefixes at once,
and get all that data back at once.
And you perform queries and create ways to do that.
But if you're going to have a little bit more nuance
in your queries than what the row key can provide,
maybe just for one-off queries or you're
trying to learn new things about your data,
you're going to use filters.
And this is very similar to allow filtering in Cassandra.
So we have timestamp filtering, value filtering.
You could do row key regexes if part of the row key
is going to help you out.
But again, like Cassandra with allow filtering,
you don't want to let it just do full table scans.
So we recommend that you have a range as part of that query
or do a prefix and just limit how much data
you're going to be scanning with that filter.
And then a final option for migrating your queries
to Bigtable is to use Bigtable as an external data
source on a BigQuery.
And this is just easy.
It is really easy to do.
You set up configuration file to translate some of your data
into BigQuery.
And then you can do table-wide analytics on it
with a SQL-like language.
So this is a lot more performant for your table-wide analytics.
You can save some of the data after you
query it to Bigtable to hand that to an analyst
if they just want to draw on a lot of queries
on the same set of data.
Or we actually see a lot of customers
write data to Bigtable and BigQuery.
And they use the data warehousing
features of BigQuery and the speed and QPS
that Bigtable provides as a database.
So that's all about how to migrate your queries.
But most of the scenarios that Kristin went through
include a batch data migration.
And we know this can be intimidating.
You have a lot of data.
You don't want to lose that.
You want to make sure it all stays in sync.
So we've made it super easy with our bulk load tool.
This is going to run on Cloud Dataflow,
our managed and hosted data Apache Beam runner.
It does an automatic schema conversion for you.
It will copy the data.
And there's no coding needed.
You'll just pass in your configuration file
with the tables, key spaces, and server information.
And then you can run it with the command line.
And what's great is this is all open source.
So if you want to take our bulk load tool
and make it work from some more of your specific requirements,
feel free to do And on Apache Beam,
we have the Cassandra I/O Bigtable I/O,
and even BigQuery I/O connectors.
So you can write any pipelines that
connect these three to make whatever dataflow
is going to work best for you.
So now you understand why you might want to migrate,
how to determine the path to get from Cassandra to Bigtable,
you can do your dual writes, stated verifications,
and your batch data migration.
So what are you going to do next?
If you're an existing customer, we
recommend that you connect with one of your account teams
to get started.
They'll help connect you to a lot of the resources we have.
But if you're not, you can check out
some of these links we've got.
Here's a link to the codelab I mentioned earlier.
So you can get started on seeing how your queries are
going to migrate.
And you can even check out a talk from Cloud Next 2019
where Spotify discusses how they migrated from Cassandra
to Bigtable using a lot of the techniques and tools
from this talk.
Thank you so much for attending.
And happy migrations.