DAN MCGRATH: Hi, I'm Dan McGrath.
I'm the product lead for Google's NoSQL databases.
And I'm here in this session to give you some insight
into modernizing your HBase workloads with Clout Bigtable.
But before we get into that, I want
to give you a quick introduction into Cloud Bigtable,
and if you're already familiar with it, talk about some
of the recent additions to the service.
And to kick that off, I'm going to give you
a little bit of trivia that you might not have heard before.
Because this year, 2020, marks 15 years
of Cloud Bigtable being in production here at Google.
We've spent a lot of time in that 15 years optimizing
Bigtable from the experience of running everything
from small zonal workloads to our largest and most demanding
workloads with deployments spanning the globe.
Here is a diagram from a design doc from one
of those very first deployments of Cloud Bigtable showing
a replicated system across three regions.
It was the following year, in 2006,
that Google released the Bigtable white paper, leading
to the development of HBase.
After beating production for 10 years,
Cloud Bigtable was released so companies outside of Google
could benefit from the same technology used here at Google.
And to make it easier for developers,
HBase-compatible client libraries backed by Bigtable
were also released.
Now Cloud Bigtable is a fully managed NoSQL document service
for use cases where low latency, random access data,
scalability, and reliability are very critical.
It seamlessly scales from your first gigabyte
to data sets that are at petabyte scales.
It can handle millions of queries per second.
And as a fully managed service, it
allows your team to be freed up from database
and infrastructure management tasks,
allowing them to dedicate their time to where the business can
add value.
It's completely compatible with the existing Apache HBase
setup, enabling seamless migrations for existing HBase
workloads.
Now Bigtable offers you a database that provides speed,
scalability, a fully managed foundation,
and integration in the Google Cloud and Apache ecosystem
as its core base.
On the speed front, it does give you
consistent 99th percentile low latency for random access
reads and writes.
It's the same technology that powers
a range of billion-user Google services.
And it's proven in applications with millions of database
accesses per second.
It easily scales to match user needs with no downtime.
It's a perfect fit for real-time storage and analysis
of huge amounts of data.
One of the really nice things about Cloud Bigtable
is it gives you consistent performance
as your workload grows.
It can be easily scaled by simply adding nodes or even
removing nodes within the UI or command line interface.
It offers seamless cluster scaling.
So dynamically adding and removing these nodes
can be done without restarts.
And the storage system automatically scales as well.
So you only pay for the storage that you are actively using.
Through the separation of processing and storage,
Cloud Bigtable is able to automatically configure
throughput by adjusting the association of nodes and data.
In the rebalancing example in the middle,
if node A is experiencing heavy load,
the routing layer can move some of the traffic
to a less heavily loaded node, improving overall performance.
Resizing, in the third example, comes into play
when a node is added, again ensuring balanced load
across your nodes, ensuring best overall throughput
of the system.
It also offers zero-touch failover for high availability
and a range of other use cases.
Bigtable enables you to write once to a single replicate
and have it automatically replicated everywhere
with eventual consistency.
There are no manual steps to ensure consistency, repair
data, or synchronize writes and deletes.
It gives you the ability to have a configurable topology
and policies on how to route data, both at the read
and write side, to different replicates.
Common use case for the replication system
are high availability for your live serving apps,
or perhaps workload isolation, where
you have both user serving and batch workloads running
against the same database.
It's also great for a use case where
you want latency optimization to write locally
to a replicate closer to you, but then
be able to read that data everywhere,
perhaps in some centralized replica where all your batch
processing happens.
So let's quickly talk about some of the recent highlights.
Now on the fully managed side, Cloud Bigtable
integrates into Cloud IAM for user access and authentication.
At the end of 2019, we announced table-level IAM,
allowing you to have more granular rules on who can
access data within clusters.
We also announced multi-region replication
to enable you to replicate data across the globe
as well as a host of new regions that rolled out
to support Cloud Bigtable.
Earlier this year, in 2020, we also
announced single-node production instances.
So historically with Cloud Bigtable,
you were required to have three nodes
to be able to be supported by the service level agreement.
Now you can have a single node supported by the same SLA that
gives you 99.9% availability for single-cluster routing
or 99.99% availability for multi-cluster routing.
Something that I'm really excited about that we're just
announcing now, in July 2020, is the addition
of managed backups.
So manage backups gives you a fully automated method
to back up the tables within your cluster
and then very quickly restore them,
either into the same instance, or perhaps you
want to restore them into a development or QA
cluster for ease of testing.
Now let's have a look at how you can think about modernizing
both your existing and new HBase workloads with Cloud Bigtable.
I like to think about it in terms of two different paths.
One is the backward-looking path of why you want to modernize.
And this is where you're currently
struggling to maintain the availability of your system.
Perhaps you've had durability issues.
Or perhaps performance, when scaling the system,
has become toilsome for your team.
And so an example of this is where
you're struggling with stability at scale or during peak load
events.
Or you consider the operational load on your SRE and DevOps
teams too distracting from the overall core work
that you're trying to do.
Perhaps the overhead of managing the replication system,
repairs, and the data garbage collection system
has also become to toilsome for your team.
Cloud Bigtable gives you a great way
to automate away a lot of those troubles.
The other path where people typically look at modernization
is forward-looking.
So they want to move up the stack
and have their development teams working on something that's
more focused on the differentiators
that the company offers.
They want to spend less time and money
managing the complexities of a distributed database
and have more time to focus on what was not previously
possible to prioritize.
The really good thing about looking
at Cloud Bigtable with HBase is you'll
find most of the concepts map very seamlessly between the two
systems.
And that's obviously because of the shared history
both Bigtable and HBase have had for a long period of time now.
So all the fundamentals are going to be the same for you.
If you know how to design schemas on HBase,
you're going to be pretty good at designing them
on Cloud Bigtable.
Row keys, column families, column qualifiers,
cells, they're all going to be the same on Cloud Bigtable.
Even the cases where there are some differences in the feature
sets that are available, such as namespaces,
you will find ways to simulate them
within Cloud Bigtable using things like row key prefixes.
Now some of the terminology you're used to,
such as regions, are going to be slightly different.
We call them tablets in Cloud Bigtable--
very similar though.
The really good thing though is all those concerns
are automated and fully managed for you.
So you don't really need to worry about them anymore.
Similarly, things like snapshots, that's now
available as managed backups.
And tasks like compaction--
so you can't configure that, but it is automatically
managed for you.
So there's no maintenance work you
need to do on the compaction and garbage collection sides.
All in all, Cloud Bigtable supports HBase client libraries
from 1.0 through to 1.3.
So you should find the development environment
very familiar for you.
Now if you look at the architecture of Cloud Bigtable,
you'll see a lot of similarities here as well.
For replicating data, replication to other clusters
can send different types of traffic to different clusters.
So we have a way to set up policies
through a thing called app profiles that
talk about how failover happens and how you want to read--
how you want to route your reads and writes on the system.
Scaling is very simple in Cloud Bigtable.
It's as simple as adding nodes or even
removing nodes within the UI or command line interface.
A table is automatically sharded into blocks
of rows called tablets.
And this is all managed behind the scenes.
We use a bunch of heuristics around total data
size and traffic to do this for you.
One of the really interesting things is on the storage side.
This is it automatically scaling storage system as well,
where they're stored encrypted, at rest on disk.
And even if you're only running a single node of Cloud
Bigtable, rest assured that your data
is durable across multiple machines.
Now when you're looking at migrating your existing
workloads into Cloud Bigtable, the first step
is always around preparation and planning for the migration.
So typically, you want to-- going
to do a deep dive into the different use cases
that you have running on HBase.
And perhaps you'll go down a different path
for different types of use cases.
It's really good to build up an understanding of Cloud Bigtable
and where some of the concepts and capabilities
might differ than what the alternatives are.
So you'll want to map concepts, schema design, queries,
and how you're thinking about data consistency from HBase
to Cloud Bigtable.
Now there are some slight differences in
supported functionality between HBase and Cloud Bigtable.
We have documentation on this.
So I encourage you to read through the documentation
and have an understanding of where those differences may
lay early in the process.
Now depending on the use case, you
could end up going down a variety of different paths.
Some of them you will find incredibly easy.
Some of them might require some more planning upfront.
So we're going to look at the four different types
of workloads and each of the different migration strategies.
And we'll walk through them all.
So let's start with batch load and fast serve.
In this specific migration path, we only
have a single writing component.
And this simplifies things a lot.
So this is a batch job that might be refreshing data
periodically, such as models that you're serving up.
But then you're using HBase to have that fast serve
capacity for that data that's being written into the system.
Now the obvious first step here is
to update the batch job to write into both HBase and Bigtable
at the same time.
Once this is done, now you can spend some time verifying
the batch job to make sure that it's
writing the same information into both systems.
After this is done, we can roll out a change to the API layer
that is reading from HBase to also support
reading from Bigtable.
Once you're happy with the performance of the system,
you can cut over to reading purely from Bigtable
and then update your batch jobs to stop writing into Bigtable.
At that point in time, once you're
happy with the performance, you can
start decommissioning your old system.
Now, when downtime is acceptable,
this really simplifies things.
And this is by far the most common migration path we see.
It works for all types of HBase workloads
as long as you can accept a short downtime.
And the downtime will depend upon the size of the data
that you have in the system.
So obviously, at this point, you stop writing into HBase
during the migration window.
A slight difference depending on your type of workload,
you could leave the system in a read-only state.
Or perhaps you bring down the entire system for maintenance.
You'll then batch copy the data from HBase into Bigtable.
And we have things like Dataflow jobs
that can read sequence files and do fast loading into Bigtable
for you.
Then you'll want to switch over your serving path from HBase
to Bigtable, and perhaps do some calls to check between the two
systems to make sure they're in a consistent state.
At that time, you can exit your maintenance window
and resume writing, but now into the Bigtable system.
A more complex migration is where
you're migrating with the system online,
but your data has a time to live.
And so this is a common workload pattern
where you're using Bigtable for something
like more persistent caching.
Or perhaps you have a data set that has a natural cycle.
Perhaps it's updated every day and recalculated.
So what this allows you to do is to have
the code that writes updated to write into both HBase
and Bigtable with the same time to live information.
You then simply wait until the time
to live on all previous data in the system has expired.
At this point, HBase and Cloud Bigtable should be in sync.
You can run verification to make sure everything's performing
as expected on the write path.
Once you're happy with that, you can update the read side
to start reading from Cloud Bigtable,
wait for the information to finish expiring from HBase
once you start writing into it.
And then you should be safe to decommission your existing
system.
And you've migrated, without downtime, onto Bigtable.
The last one-- and this one is more complicated.
This is where you're migrating a workload that
has persistent data without downtime.
And so once again, the first step
here is updating the write layer of the system.
But you're also going to want to look at the read
layer of the system.
Once you've got the updates writing to both HBase
and Bigtable, you'll want to run a verification that that's
happening correctly and start batch copying all the existing
information that's in Bigtable.
Once you've migrated all the information,
you're two systems should be in sync.
You can then migrate the read workload,
verify performance of the system,
and start decommissioning the old system
at that point in time.
Now this one can take a little bit longer,
because you have to have both the dual writing
upfront and then the backfill operation.
So it's sort of a combination of some
of these previous approaches.
Now perhaps HBase isn't the only type of workload you've got.
So if you've got Cassandra as well,
which is also very similar to Cloud Bigtable,
I highly encourage you to tune into our session
on migrating Cassandra workloads into Cloud Bigtable.
And if you've got DynamoDB, we have another session
that talks about migrating that into Bigtable and Cloud
Spanner.
You can also watch Cloud Bigtable in action
through our showcase demo done in partnership with Spotify.
I would really encourage you to look at that demo.
And if you want to get hands-on and try it yourself,
we have the quick labs for Bigtable
that you can go through, get some hands-on experience.
And there is also the HBase to Bigtable
migration guide, which walks you through some
of the technical steps here, commands to run,
and tooling that's available.
So I want to thank you for tuning into this session
on modernizing HBase for workloads.
I hope you got something out of it.
And I'm looking forward to seeing you
move to a more managed future.