SEAN-MICHAEL LEWIS: Hello, and welcome to my talk Do it live!
Fitbit's Zero-Downtime Migration to GCP.
I'm Sean-Michael Lewis.
I'm Principal Software Engineer at Fitbit.
And I was in charge of moving our monolithic application
from our legacy managed hosting to Google Cloud platform.
In today's talk, I'm going to talk about what was unique
or what we felt was unique about Fitbit's migration challenge.
We are going to talk about how we
thought about users in making decisions
to move for our migration.
Then we'll talk about the technology and processes
that we added and affected to make this happen.
We'll talk a little bit about how it went,
which was quite well.
And then we'll have some takeaways
that we think could be useful for you,
should you be moving from another cloud provider
or your own managed hosting into Google Cloud Platform.
So, like many companies, Fitbit started
with a monolithic application.
We had a single binary that did all the things.
It is quite large.
It handles all of our traffic, or it did for a long time.
And then, we started to break it up into what
we call microservices.
It's pretty standard when a company matures.
But while moving microservices might be quite easy,
each service can do its own thing, move its own data store,
figure things out and move into the cloud,
a monolith is much more challenging.
It's more challenging, it's bigger, meatier,
it takes up about 70% of our traffic.
So moving it was quite hard.
So our migration wasn't going to happen until the monolith move.
So let's talk about our monolith.
Like I said, it's a single Java binary,
about 1,000 instances backed by around 200 MySQL instances.
Those MySQL sequences are mostly sharded by user data.
So if a request comes in, I can get all of my data
from one shard.
We have a sharding library that manages the shards.
So it figures out where users go when new users sign up.
It allows you to expand and contract
the number of MySQL sequences we have.
And it also handles caching, for which we had
around 400 nodes of mem cache.
We also have Kafka, where we do asynchronous processing.
And those messages are largely processed by other instances
within the monolith.
So we'll talk about--
yeah, we'll talk about our users now.
Now that we know what the application looks like,
we thought about what it would look like for us to move
and who we should be thinking about as we're moving.
The most important stakeholder in the move is your users.
Sometimes, doing the best thing for your users
is not the most easy thing technically.
But that should be the thing you do first,
or the person you think about first.
So, how do we get to GCP with our users in mind?
We thought about two different ways of doing this.
One was to do it progressively, so move user by user,
or batch of user by batch of user.
Or we could do it all at once.
There are different pros and cons to each.
For the progressive, we thought that anything bad that
happened in the new environment as we were moving,
any new challenge that we experienced in our new home
would be limited to only the users that have moved.
So it should be a smaller impact overall to our user base
as we figured out things in the new environment.
Secondly, outside of normal availability problems,
just how our application tended to load in the new environment
would be able to be observed in a more gradual fashion.
We could see how the environment dealt with new load
in a more measured way.
We knew that network calls-- we wouldn't
be able to make every network call be in the new data
center in GCP.
So we would have to go back to the old data center sometimes,
and that would incur a lot of latency.
That's not a great user experience,
but it's one of the downsides of this approach.
And then lastly, we'd have to operate two full application
stacks simultaneously, be able to operate them
at the same time and manage them at the same time in two data
centers and two different hosting [INAUDIBLE] for us.
On the contrary, the all-at-once solution--
any challenges you have in the new environment, when
you cut over to the new environment,
it affects everyone.
So if you have a big outage, everyone's down.
It's basically a total system shock.
There's a lot of risk in that.
You can do a lot of testing beforehand,
but you don't know what's really going to happen
until you flip the switch.
The positive is that once you're in the new data center,
all of your network calls are local,
so your latency situation is as good as it's going to be.
And while you don't have to maintain and operate two data
centers at the same time, you need
the old one on standby in case something
goes wrong in the move.
So you're not really getting rid of the old one
when you cut over, at least until you're fully in.
So, which one did we choose?
We felt that overall, thinking of the users in mind,
that the progressive migration was going
to be much better for them.
On average, most users wouldn't experience any downtime
or any noticeable downtime.
That took some work, but we thought
that if we could slowly roll into GCP,
we would give our users the best experience,
and maybe even one they don't notice.
So, as we were thinking about how
we were going to do this progressive migration,
we had to think about, what are the things we want out of it?
How are we going to do it?
What were going to be our guiding principles?
So, the first one was to make our best effort to route users
to the environment where their data is resident.
As I said, we shard our data out by user.
So, if I move my data from our old hosting to GCP,
I should start routing my requests to GCP.
That was the goal of this migration, is to move
the user with the data.
We wanted to make sure that-- let's say
we didn't route correctly in my data is in GCP,
but I got routed to the old data center.
Well, that should still work.
We still want that to be an experience.
We don't want to be throwing errors when we have a misroute.
So we want an experience no matter what.
We want to be able to move users to GCP,
but if things aren't going well, we
need to be able to move them back.
And we also need to be able to move at variable speed.
If things are going great, we should
be able to move users more quickly.
If they're not going so well, or we're not sure,
we should trickle them in and get our sea legs, as it were.
Lastly, as we're making these decisions about fast,
slow, forward, backwards, we don't
want to be making a lot of gut-check decisions.
We want to be able to look at a chart
and say we're meeting in our SLOs.
We're going to have to move forward now.
And so, we actually needed to develop more SLOs
and have a better handle on them before we could do this
appropriately.
Let's talk about the timeline for this migration.
So as I said, we wanted to move users slow
and quickly, at different varying times.
But before that, we needed to actually set everything up
in GCP and make sure it was working.
And you can always run tests, but until you're really
routing production users there, it's not real.
So, the first users that we routed were employees.
We routed them through GCP.
All their data was still in our old data center.
They had slow experiences, but at the beginning,
a lot of things were broken, but we fixed bugs
until we had a complete experience.
And all the meaningful experiences
that happened in Fitbit were serviceable by going
through GCP, and even if data was in another data center.
Once we felt confident in that, we
decided it was time to start moving paying customers.
So we started moving users slowly--
in a slow trickle.
When we got to watches, our systems
took on that new traffic and made sure
that we were able to operate and monitor
and really made sure that things were working in GCP.
Then, once we got a lot of confidence,
we started moving users in big batches.
We moved them quickly.
And as you'll see, quicker than we even anticipated.
So, how did we change d architecture and technology
to make this kind of move possible?
As you'll see on the left, I have the same diagram
for our monolithic architecture, for our legacy data center.
We replicated that in Google Cloud Platform,
set the whole thing up.
We had fewer applications at the start and fewer MySQL
instances.
And those MySQL instances were empty
when we started as well because they had no data in them.
There was no data in GCP yet.
And then, we had connections from all the application
instances to all the MySQLs, in both the old data center
and in GCP.
So it fully connected between those two data
centers between application instances.
Those are the blue lines you'll see.
The application instances go to--
those are the lines that we had to add.
We had to add connections back to the old data center
from GCP, and to GCP from the old data center.
Then we added edge routing as well.
This was the logic that would allow us to route users
based on where they lived.
So we had to be able to push that logic somewhere.
It turns out we used Cloudflare, which
is a CDN and DNS provider.
They had a product called Cloudflare Workers
that we were able to push logic up
into and make these decisions.
And I'll talk more about that later.
But that made it so that we didn't
have to go into one data center to decide that we really
wanted to service it out of the other one,
and kick it back, and redirect.
It made everything much cleaner and easier to observe as well.
And then, lastly, we have connectivity between our mem
cache instances.
You'll notice that the app, in this diagram,
the applications only talk to their data center local mem
cached, but if one application-- if one data center makes
changes to its local mem cache, we
want that change to be replicated to the other one.
And so, we introduced a proxy that
would allow us to echo in validations to the other data
center.
I'll talk about that as well.
First, we'll talk about moving our users.
I said, we had, initially, empty databases
in GCP and all our full databases
in our legacy data center.
In our sharding scheme, we have this notion of buckets.
Buckets have users in them.
Mostly, there's only really two ways to do sharding.
There's buckets and there's ID ranges.
We used buckets as kind of like a hashing scheme.
But it allows us to move an individual bucket
one at a time.
So we can move one bucket to any other database server.
So when we were moving into GCP, we
could move one bucket from a database server that
lived in our old data center, and then
move it to GCP now that users have been moved.
This is the slow method of movement.
It takes about 20 minutes because there's
a lot of locking involved.
Users can still read their data, but they can't
write for that 20 minutes.
It's slow and gradual.
We can add parallelism to get a little bit more speed so we
can do many buckets at a time.
But we can't really move tons of users at once.
To move fast, move the whole shard.
And this is a typical replication strategy,
which is that our leader database would
be in our old data center.
We'd have a replica of it in GCP.
At some point, we would flip who the leader was and make
the leader the one in GCP.
Now, all the users on that shard, all those buckets
have shifted all at once.
And that is a very fast thing.
It takes about five minutes.
It can go back and forward.
Both of them are bi-directional, which
is important for our use case.
I'll talk a little bit about routing.
As I said, Cloudflare Workers is the technology
we used to provide some logic as to where our users'
traffic should be routed.
So, we have this idea of buckets, as I told you about.
There's a finite number of buckets.
In fact, the number was finite enough
that we could make a big map of it
and push that map in a JavaScript binary
up to this Cloudflare Worker.
We could also put a whitelist in there, which allowed us--
it was how we were able to route employees
before paying customers.
So regardless of where your data is resident,
we could say, oh, this user should always
go to GCP no matter what, no matter where their data is--
otherwise, see where their bucket lives.
So this is the flow diagram.
We have timed deployments of the logic.
The code that executes the logic, but also
the data that feeds that logic gets time
deployed into Cloudflare.
Then the logic is simply, is the user in the whitelist?
If so, send them to GCP.
If not, check where their bucket lives.
Is the bucket in GCP?
If yes, send them to GCP.
If not, or we have some kind of error,
or there is no user information, send them
to the legacy data center.
Eventually, once we had more than 50% of movement into GCP,
we made the default GCP.
The last technical aspect was keeping the caches coherent.
As I said, applications only talk to their local caches.
And if the other data center changes
the cache on their side, you want it
to be reflected in both places.
So we introduced a new piece of technology
[INAUDIBLE] router and open source mem cache proxy that
has the ability to echo deletes and other operations
to other data centers, or other clusters of mem cache nodes.
Our second cluster just happened to be at the other data center.
So, this was pretty challenging for us.
Although it's pretty straightforward
and it's a good tool, we don't think a lot about it now.
It was a stark difference from implementing the routing
from before.
The routing, being a managed service,
ended up being really easy for us to do.
It gave us very few headaches, we had very few incidents
around it.
Whereas, introducing a new piece of technology
just before you make a big move was much more challenging
and ended up working out for us.
But it started to show us the differences between what
might be possible when we have more managed services
at our disposal.
So, how did this all go?
It actually went quite well, better than, I think, any of us
expected.
The routing of employees before everyone else
uncovered many critical bugs.
So it uncovered around 60 bugs.
We closed 58 of them before we moved to a paying customer.
And I think those other two got closed as a won't fix.
They didn't end up being that critical.
So really, everything that was really
blocking an experience in GCP was closed
before we were able to do it.
And we did that live.
We did it in production.
I mean, we tested it in production
without affecting our paying customers,
which was super awesome.
And then, we also finished early.
We had a pretty hard deadline to be out at a certain time.
And we started this process--
if you look at the left of this graph,
you'll see that there's a very slow,
a very low trickle of throughput going to GCP,
which is the green line.
And you'll see that it slowly steps up over time.
And then around the end of August, or 8-30 on this chart,
you'll see it takes a big step up.
And then it takes some other bigger steps up and stays high.
Based on our SLOs, we were able to see that we were doing well,
everything was kind of under control,
and then we could really push this migration faster, and get
done sooner than we expected, which was
super exciting for everyone.
So, what are the takeaways from this process?
Progressive migration is based on user data locality.
It can provide the best user experience for your users
most of the time.
If you have a sharded monolith like we do,
you probably have some sort of user sharding or something
like that.
And so, if you can do it little by little,
that's going to be the best bang for your buck.
Your movement should have options, both speed
and direction.
It gives you flexibility for when things go wrong
or things go really well.
Decide what is really good or really well based on your SLOs.
And then also, we found that this was the first opportunity
for us having been in our own colo to use managed services
and really see the benefit of them.
Now that we're in Google Cloud Platform,
we think there's a lot more opportunities
for us to do that.
So, if there are opportunities to use them in your migration,
I would say you should.
That's all.
Thank you very much for your time.
And I hope your migration goes well as well.