SHITANSHU VERMA: Hello.
I'm Shitanshu Verma.
I work for Cloud Bigtable at Google Cloud.
Many of our customers want fully managed services,
especially databases.
They don't want to manage the infrastructure.
They don't want to manage the databases and keeping it up.
Cloud Bigtable is a fully managed NoSQL database
that scales linearly and with potentially infinite scale.
It supports HBase API.
So naturally, a lot of customers who are running on-prem HBase
want to move to Cloud Bigtable and experience
a fully managed solution.
We at Cloud Bigtable have worked and provided
many solutions and many tools to migrate
from HBase to Cloud Bigtable.
And the migration journey from HBase to Cloud Bigtable
is now very smooth with Google-provided, well-tested
tooling.
In the following video, I talk about one such migration
journey and the tools that are available to you to migrate
from HBase to Cloud Bigtable.
So here it is.
Enjoy.
[VIDEO PLAYBACK]
- Hi.
Welcome to Cloud Data Summit.
And today, we are going to talk about database migrations
from HBase to Cloud Bigtable.
I am Shitanshu Verma.
I'm a software engineer.
And I work at Google.
So we are going to talk about, what is Cloud Bigtable
and why it should be your preferred destination for HBase
workloads?
We'll talk about migrations approaches
with and without downtime.
And then, how can you get started on your migration
journeys?
So what is Cloud Bigtable?
Google Cloud Bigtable is a fully managed, scalable, NoSQL
database service.
It can offer you low-latency, high-throughput
reads and writes.
We want you to move your HBase workloads for Cloud Bigtable,
because, as your scale increases,
you may face a bunch of issues with HBase.
You can have stability issues at scale.
You can have high-operational load on your SRE and DevOps
teams.
And you can have the overhead of managing replication, repair,
and garbage collection.
Whenever you start facing these challenges,
you should start considering managed solutions.
And we offer you Cloud Bigtable, which is Google Cloud's managed
solution for HBase.
So why would you choose Cloud Bigtable for HBase?
Well, first of all, the most important part
is Bigtable is HBase API compatible.
You don't have to change your workloads.
You don't have to change your applications.
You don't have to change any code.
Cloud Bigtable will work with your HBase applications.
It's easier and faster to migrate from HBase
to Cloud Bigtable.
Google provides well-tested high-quality migration
tooling so that you won't have to code, plan, and test
your migration tooling.
You can just take your data from HBase to Bigtable
and then start your applications here.
It is used by multiple big enterprise customers, as well
as Google.
A lot of big Google products are on Cloud-- on Bigtable.
It's a fully managed service, which
means you don't have to worry about batching.
You don't have to worry about provisioning, configuring, et
cetera, et cetera.
It's a low-touch, fully managed service.
So each Cloud Bigtable can give you
low-latency, high throughput system with infinite scale.
It is highly available by multi-region,
multichannel replication, and automatic failovers.
So you don't have to worry about failing over.
You can automatically failover to a cluster
if a cluster were to go down.
It provides workload isolation.
And it's built for internet scale applications.
So let's talk about migration approaches.
Primarily, there are two migration approaches.
One, which is offline migrations, which
require you to take downtime.
Here, you will take the database down for writes.
So you will not have any writes on the database
while the migration is in progress.
It is relatively less complex to do offline migrations
and usually preferred.
But in the cases where you can't take live migrations,
when you are running customer serving workloads
on top of your database and you can't really
take long downtime, you use live migrations, wherein you stream
writes to both old and new databases,
in our case, HBase and Cloud Bigtable,
and then do your historic data copy from HBase
to Cloud Bigtable.
Once everything is done, you can switch over your application
to Cloud Bigtable.
So we'll talk about both the online migration approaches
and the online migration approaches for HBase
to Cloud Bigtable migrations.
And I'm going to introduce a new live migration
tooling that we're launching in that summit.
So let's talk about offline migration steps.
Step one is you assess and plan your migration.
You do performance benchmarking, figure out
what workloads you need to migrate, so on and so forth.
You install the relative tooling.
Then you take the snapshot of your HBase tables.
And then, you export your HBase snapshots to GCS.
Once you have the HBase snapshots on GCS,
your data is on Google Cloud.
Then before you start importing them,
you will need to create your tables in an HBase instance.
So Google Cloud provides you a tool
that connects to your HBase cluster, creates--
takes information about all the tables,
all the splits, all the column families
that are there, and creates the same tables on Google Cloud
Bigtable.
So you don't have to worry about copying each table's column
families, garbage collection policies, split points,
and creating them manually.
It will automatically create all that for you.
The next step is, you run a Dataflow pipeline, which
Google provides, to import your HBase
snapshot onto Cloud Bigtable.
Once you do that, you run a hash table job
on HBase, which is provided by Hadoop,
to compute hashes for each of your tables and all ranges.
And then you run a sync table job,
which is kind of a mirror job, which computes the same hashes
and compares them to make sure that all the data that
was copied is imported properly on the Cloud Bigtable.
Once you have matched the hashes,
now you're 100% sure that all the data
has arrived successfully without corruption to Cloud Bigtable.
Once that's done, you can bring your application up
and have it routed to Cloud Bigtable
and then, later on, deprecate your HBase clusters.
Now I need to stress--
I want to stress again that all of these migration steps,
every tool and everything that you need, is provided to you.
You, the customer, don't have to write any migration tooling.
Cloud Bigtable will provide you benchmarking
solutions, which can benchmark very high-QPS workloads.
And you can tune your row-to-row sizes, number of versions,
what your new key looks like.
And then, you can--
HBase provides you the capability to take the snapshot
and export the snapshot.
Bigtable-- Google will provide you a schema translator tool
to create your tables.
It will provide your import snapshot data flow pipeline.
It will provide you a sync table pipeline
that goes along with the hash table
that Hadoop HBase provides.
So you don't have to write any code
to do migrations, which is great.
Now let's look at online migrations.
So in online migrations, apart from everything
that you do in offline migration,
the extra step is streaming the rights
to both Cloud Bigtable and HBase.
Now, there are various ways you can do these tool writes.
You can have your application write them synchronously,
as part of the write, write to both databases
and succeed only when both writes exceed.
You can have write to the primary database
and then have a queue and asynchronously
write to a second database.
This saves you latency.
You can have a message worker, which your applications write
to message worker, and then some worker
writes to both HBase and Bigtable
from the message worker.
Or the way we prefer, the preferred approach for us,
is enable HBase to Bigtable replication.
I'm going to talk about this replication later, which
is a preferred way.
It solves most of the problems without having
any impact on the HBase applications that you run.
Your business applications are not impacted at all.
So let's look at the steps.
The steps look very similar to what
you see in offline migrations.
The new step here is addition of enable HBase
to Bigtable replication.
So this could be any of the tool write steps.
So you plan the migration.
You install the stuff.
Then you enable the dual writes.
Then you take snapshot, import snapshot, validate your stuff.
And once all the validations are complete,
then you can just turn off the dual writes
and restart your applications, writing and reading directly
from Cloud Bigtable.
Now this dual write step is hard.
And I'll tell you why.
And I'll tell you what we suggest to support it.
So let's look at the new library, the new tooling
that we are launching today for HBase
to Bigtable live migrations.
It's called Google Cloud's HBase to Cloud Bigtable Replication
Library.
So this replication library is essentially an extension
to HBase application service.
It is installed on HBase region servers and master.
It basically reads-- it listens to the HBase vol log events
from the HBase application service,
translates them to Cloud Bigtable mutations,
and writes them.
This application is eventually consistent, but guarantees
you the ordering of mutations on both HBase and Bigtable
So if you are doing a PUT followed by a DELETE on HBase,
it is guaranteed that the PUT is going to be followed
by DELETE on Bigtable, as well.
And so the data will not diverge.
Many methods cannot guarantee this ordering.
And hence, you have all kinds of data deviation
between HBase and Bigtable.
This replication library guarantees your ordering.
It is eventually consistent.
It can handle all, kinds of, failure and unavailability
of the destination HBase cluster or Bigtable cluster
for all durations.
So you don't have to worry about if a write succeeds on Cloud
Bigtable and fails on HBase or if the write succeeds on HBase
and fails on Bigtable.
That will never happen, because you'll always write to HBase.
And then Cloud Bigtable Replication Library
will follow that in order.
So how do you operate this HBase Replication Library?
To get the replication working from HBase to Bigtable,
you first install a replication library jar, this jar
that we provided it to you.
You install the jar on HBase, all the HBase nodes.
Then you set up authentication, which basically
means getting a service account JSON file
and installing it on the HBase nodes.
You set up HBase site.xml to talk about destination project
and destination Cloud Bigtable instance.
And then you reboot the HBase server.
Once you reboot the HBase cluster,
now the HBase master and HBase region server nodes
know about the replication library.
Then you add a Cloud Bigtable peer using the add_peer command
from HBase shell, which is how you
add any HBase application peer.
And once you enable replication, by add_peer,
now the replication will be active.
All the writes that you are writing on HBase
will go to Cloud Bigtable.
Now this is very important to note
that this is an asynchronous replication that
does not impact your writes or reads from HBase.
So any application reading or writing from HBase
will not even know that the replication is enabled.
They will have no impact.
Your business can go as is without an impact.
Another benefit, here, is that if you
have multiple applications reading or writing
from each HBase, none of them need to change.
You don't need to do n-number of modifications
for each application.
The applications need no change at all.
Just the HBase region servers will start shipping the changes
to Bigtable.
And if something goes wrong, you can always
remove the replication to Cloud Bigtable.
And your HBase cluster will recover.
So there's one takeaway that I want
you to take from this talk.
It is that migration from HBase is to Cloud Bigtable is easy.
And it's fast.
And it's done by high-quality, well-tested tools
that Google provides you.
You, as a customer, don't need to write anything.
You just need to plan your migration and execute on it.
And we are here to support you on your migration journey
for anything that you need.
With that said, I would encourage
you to explore the documentation,
learning what Bigtable is, reading some code labs on Cloud
Bigtable, and reading our end-to-end migration
guides for both online and offline migrations.
With that, thank you so much.
[END PLAYBACK]