RADHA MATHUR: Hello, and welcome.
I'm Radha Mathur.
I'm a product manager at Google Cloud.
And in this session, I will be presenting an overview of HA
and data protection on GCP.
We'll go over the catalog of HA and data protection
capabilities for both IaaS and PaaS workloads running on GCP.
And I'm happy to share that I will be joined by Jeremy Tinley
from Etsy, who has graciously agreed
to describe how Etsy uses and benefits from some
of these capabilities.
Let's get started.
HA and data protection solutions are
required to maintain business continuity.
Key business continuity objectives
are to guard against accidental data loss
due to user errors, data corruption, or ransomware,
and to achieve resilience to infrastructure failures,
such as zone failures, competence failures,
or even failures of entire regions
that could be caused by natural disasters.
Common HA and data protection solutions
that enable business continuity include
backups, cross-zone high availability,
and cross-region replication and DR orchestration.
These solutions form a spectrum with respect
to the level of resiliency that they provide and their cost.
And the choice of any one solution for a workload
depends on the business criticality of the workload
as well as the tolerance of that workload to downtime.
For example, workloads that can tolerate a higher recovery time
could employ backups for protection
against both user errors and infrastructure failures.
Whereas mission critical workloads
that have stringent uptime requirements
would typically use replication-based solutions
for protection from infrastructure failures,
and in addition to that use backups for protection
from user errors.
Now, irrespective of where your workload is in this spectrum,
our goal at GCP is to provide a rich catalog of built-in data
protection primitives as well as ISP partner products
to help design a business continuity
solution that is aligned with the needs of your business.
GCP Compute, Storage, and many PaaS services
offer a wide range of replication and backup
capabilities.
These primitives serve as building blocks
that can help you form a business continuity solution.
In addition to that, we have a rich ecosystem of ISP partner
products that augment the primitives available on GCP
and support a wide variety of use cases,
from protecting GCE VMs, to enterprise applications
such as SAP, to replicating entire VMware
environments from on-premises to GCP for disaster recovery.
Various protection mechanisms can
be employed for protecting application workloads running
on GCP-managed infrastructure.
Let's take a deeper look at some of these HA data
protection primitives.
To achieve high availability in the event of infrastructure
unavailability such as zone failure,
VMs and Containers can be distributed
across zones in a region.
Similarly, both persistent disk and GCS
offer regional instances that replicate data
across zones in a region.
Capabilities such as VM machine images and persistent disk
snapshots enabling backing up of both VM configuration as well
as application data.
And similar backup capabilities are
in the pipeline for Anthos GKE Clusters as well
as Cloud Filestore.
Georedundant Cloud Storage skews help replicate data
across regions for disaster recovery.
And DR orchestration is enabled by Deployment Manager.
Now, in addition to compute and storage primitives,
we're adding primitives in the near future
to aid in application-consistent backups such as persistent disk
app-consistent snapshot hooks and SAP-certified backint
agents for Google Cloud Storage.
GCP PaaS databases and analytics services also
pack in a rich set of replication and backup
capabilities.
Managed databases such as Cloud Spanner and Cloud Bigtable
offer modes for cross-zone and cross-region replication.
Cloud SQL HA instances and Google BigQuery
regional instances provide high availability
against zone failures.
For data backups, PaaS databases on GCP offer primitives
such as managed backups as well as the ability to export data
in industry standard formats.
Cloud SQL also enables granular recovery from user errors
or application corruption via a point-in-time recovery.
Having looked at primitives suited for IaaS and PaaS
workloads running on GCP, let's look
at how one of our customers is using GCP's business continuity
primitives as building blocks in their data protection strategy.
I'm pleased to welcome Jeremy Tinley, senior staff
engineer on Etsy's database infrastructure team,
to share insights from their production
strategy for etsy.com.
Welcome, Jeremy.
JEREMY TINLEY: Thanks, Radha.
I've been at Etsy for six years, and over that time,
the infrastructure has considerably evolved.
We recently migrated to Google Cloud,
and this gave us a chance to rethink our general DR
strategy.
Google Cloud offers a wide range of services for DR. Selecting
and, more importantly, adopting them was very important to us.
But first, a little bit about Etsy.
Etsy is a global online marketplace
where people come together to make, sell, buy, and collect
unique items.
So who sells on Etsy?
Etsy sellers are people from all over the world.
They use their incredible talent to make things.
This can be blankets made in Morocco, jewelry from Ecuador,
or even soap made in Seattle.
They can also be people who have vintage items
that they're finally willing to part with.
On the flip side of the marketplace, you have buyers.
Etsy buyers browse the marketplace
looking for items based on personal style or need.
My current favorite is buying handmade chocolate truffles.
I usually buy them locally made from somebody
who's in my community, who's my neighbor.
And that also helps because the shipping is really quick.
My favorite thing about Etsy, though,
is that they enable a human-to-human connection.
You're not buying something from a warehouse.
You're buying it from a human being.
Working at Etsy is a bit similar.
Humans are crafting infrastructure and code.
Hence our phrase, code is craft.
With that, let's review our architecture.
Etsy is a traditional LAMP stack consisting of Linux, Apache,
MySQL, and PHP.
Since Etsy is an e-commerce marketplace,
you can probably imagine a typical interaction
with the website.
A seller lists new items by entering
in the name, the pricing, and description.
Buyers browsing Etsy get to read the seller's content.
They can initiate a conversation with the seller,
asking them questions about their products.
They add an item to their cart, and hopefully, they purchase.
After they've made a purchase, they
can track the shipping status.
And when they receive their item,
they can leave ratings and feedback for the store.
Both the front-end and back-end services
run on Google Compute Engine VMs, not on a hosted platform.
We do this because we want more hands
on control over the systems.
On the front-end, we use managed instance groups
to maintain availability of web and API servers.
This makes it trivial to add or remove capacity,
and failing servers are depooled automatically.
Our front-end workloads are stateless,
and this makes it straightforward to manage
disaster recovery.
Moving to the back-end, however, is
where things get complicated.
From the description above, you can
surmise that we run a traditional OLTP workload.
All of the data that buyers and sellers generate and interact
with is stored in MySQL.
From a DR point of view, we primarily
rely on MySQL native replication for multiple copies of data.
We execute periodic snapshots of our data, and for each one,
we run a restore test process to validate it.
For transaction logs, we sync those to Google Cloud Storage
buckets on a scheduled basis.
However, our world wasn't always so wonderful.
Etsy is 15 years old, and for most of that,
it was hosted in a traditional datacenter with bare metal
servers.
The good news about running your own data center--
you can manage everything.
The bad news-- you have to manage everything.
So let's look at one specific case dealing
with hardware failure.
I think we can all agree that computers are both amazing
and also terrible.
Hardware failures, especially disks, happen.
This meant that we needed to be able to rebuild servers
as the need arose.
This was especially painful when dealing with large amounts
of data in MySQL.
The first step was just to provision a bare metal server.
On a good day, an hour was optimistic.
On a bad day, we would fight with UEFI settings, boot disk
issues, and Anaconda was not a good friend.
Let's assume that all that goes correctly, though.
After we've built our bare mental server,
we have to restore MySQL data.
We copy our data over the network,
decrypt and decompress it, bring a database online, and wait
for replication to catch up.
Although it was all automated, it still
took a long time to perform--
many hours.
This same slow down translated to our backup process.
We ran into a longer than desired RTO, or recovery
time objective.
Our MySQL data backups were being performed daily.
Backups took about three hours to complete,
which for us was painfully slow.
These backups were full backups sent over a network
to specific backup servers that had
lots of disk loaded in them.
However, the process doesn't end there.
As the old adage goes, a backup isn't a backup
until it's been tested.
Once our backup was transferred, a restore test process
was kicked off to verify that the data could be recovered.
This involved once again copying the full backup
over the network to restore testing server,
decrypting and decompressing, bringing the database online,
and waiting for replication to catch up.
As I said before, that whole process
took multiple hours to achieve.
When these problems are magnified at scale,
both for the amount of data and copies of data,
it begins to stress the infrastructure.
Etsy ran many, many copies of data,
each with a different purpose.
We have our main production workload that has redundancy.
We have geographic redundancy.
And others were artificially delayed
to allow faster online recovery in the event
of an inadvertent data change.
Thankfully, late in 2018, we migrated Etsy to Google Cloud.
In our relatively new cloud world,
we found easier solutions to these problems.
As I said earlier, the front-end takes
advantage of managed instance groups
to improve the speed at which failed nodes are replaced.
The host is failed.
The host is destroyed.
And it's rebuilt. The host is added back to production when
it's registered as healthy.
Before this magic, disks would have to be replaced.
Hosts would have to be rebuilt by hand.
No one even has to pay attention to these issues anymore.
This frees them up to focus on other areas of improvement.
Our new MySQL environment was designed
to take advantage of native Google Cloud features.
The big shift was to use persistent disks instead
of local disks.
We built our databases with one boot disk
for logs and state data, like PID files and lock files.
The second disk is only for MySQL data.
Performing a live disk snapshot in MySQL is--
no pun intended-- a snap.
There's no need to quiesce a database either.
When MySQL starts up, it's treated like a crash,
and it recovers itself.
By using persistent disks, we got very easy differential
backups with PD snapshots.
And coming from the world where we did full backups
every single day, this improved our backup times from two
to three hours down to three minutes.
We can now take many snapshots per day, giving us the ability
to roll back data to previously known good point.
This eliminated the need for an online delayed copy.
Also, our snapshots are multi-region,
so we no longer have to manage shipping full backups
across the entire US.
Or restore test process is similar
as it was before, with some wins.
After we obtain a snapshot, we again do a restore test.
However, this time in GCP we're able to provision a new VM
dynamically.
We create and attach a new disk from a snapshot.
We bring MySQL fully online and have it replicating.
By taking more frequent snapshots,
replication catch up time was significantly reduced.
This process used to run for hours.
We can now test a copy of data in just 15 minutes.
Another big win was to use Google Cloud Storage.
We consistently send our transaction
logs to a bucket with Lifecycle Management enabled.
This helps us maintain a retention policy that
makes sense for our use case, including automatic purging
of files when they age off.
And let me tell you, legal is really happy with that one.
Our buckets are also multi-regional,
giving us geographic redundancy with our backups,
just like our snapshots.
All of this translates to stress-free disaster recovery.
Recapping our results, we saw a big benefit with our migration
to Google Cloud.
Front-end server builds are much faster, and automatic.
On the back-end, we see a tremendous improvement in RTO,
both the speed at which we can provision,
down to how quickly the data can be brought online.
I think we've just scratched the surface with what's possible.
We plan on looking at additional DR
services that can be leveraged to make our life easier.
These results might have been possible
in a traditional datacenter, but it
would require additional staffing, infrastructure,
and capacity management.
As someone who participates in an on-call rotation,
I'm quite happy to offload those challenges to Google Cloud.
RADHA MATHUR: Thanks, Jeremy, for the insightful description
of the impact you've seen by use of HA and data protection
primitives as part of your GCP stack.
From backup of data disks, to replication of transaction
logs, to use of highly-available managed instance groups for web
servers, the solution shared by Jeremy
is representative of how primitives across the spectrum
can be used for different components of your workload
based on the needs of your business.
This concludes our overview of HA and data protection
capabilities on GCP.
Thank you for joining us.