[MUSIC PLAYING]
MARTIN OMANDER: It's scary to deploy new code to the Cloud.
There's so many things that can go wrong.
STEVE MCGHEE: Let's see how we can
make your deployment safer by bringing some SRE
thinking to them.
[MUSIC PLAYING]
MARTIN OMANDER: Steve, you told me the other day
that you have experience in reliability engineering?
STEVE MCGHEE: Yeah, I was a site reliability engineer, or SRE,
inside of Google for over a decade.
I worked on Search, Android, YouTube, and Cloud.
Now my job is helping developers understand
how to build reliable systems on the Cloud.
MARTIN OMANDER: I see.
And you said we could make deployments safer by applying
SRE thinking to them.
Tell me more about that.
STEVE MCGHEE: Sure.
Well, there are four deployment practices
that I have seen work really well when I worked in SRE.
We use them at Google, but anyone can adopt them and make
their deployments safer.
They are CI/CD, fast rollbacks, Canary releases,
and hidden feature rollouts using experiments.
MARTIN OMANDER: All right.
Let's talk about each one.
CI/CD is the first one in your list.
STEVE MCGHEE: That's right.
First, you should set up CI/CD.
If you're currently deploying your code manually,
automate that process.
It will make your deployments more reliable and repeatable.
MARTIN OMANDER: And CI/CD stands for continuous integration
and continuous deployment?
STEVE MCGHEE: That's right.
Continuous integration means that newly submitted code
is automatically built or compiled and then tested.
That helps uncover bugs sooner.
MARTIN OMANDER: Right.
I've noticed sometimes that people
forget about that testing aspect when they talk about CI/CD.
And I might have been guilty of that myself.
[LAUGHS] Sometimes we only talk about
the automatic deployments.
STEVE MCGHEE: Yeah, the testing part is important.
Your tests should cover most of your application,
and they should run automatically whenever
developers submit new code.
And they should let you know when they break.
MARTIN OMANDER: Yeah, and it takes real effort
to build a test suite that covers your application well.
STEVE MCGHEE: It does, but it's a good investment.
So that was CI, continuous integration.
CD stands for continuous delivery.
That means that newly submitted code
which passes CI can be deployed automatically.
That way, we can get it off of your laptop
and into the Cloud sooner.
MARTIN OMANDER: Ah.
But what if I don't want to automatically deploy new code?
It sounds kind of risky.
STEVE MCGHEE: Well, you can deploy it automatically
to a test environment.
That way, the latest version of the system
will always be available for testing.
When it comes to deploying your production environment,
there are many options.
Your system might be built automatically,
but an operator would still make the choice
whether or not to deploy.
If so, the deployment would then proceed
through an automated pipeline, for example, pushing the code
to each region or zone in some predefined order,
including production tests along the way.
If you want to go a step further and automatically deploy
into production every time, that is
called continuous deployment.
It's another CD.
This might sound scary.
But, actually, more frequent deployments
make each one safer.
Each one is smaller and can be identified and fixed
with more precision.
MARTIN OMANDER: So what if you take that
to its logical conclusion?
Should we deploy every hour, every minute?
STEVE MCGHEE: I like where you're going, but be careful.
For example, it's not a good way to launch a new product.
Sometimes you want control on exactly when something
is revealed to customers.
Also, if you have many system components changing
at different frequencies, this can be very confusing,
making troubleshooting even harder if something goes wrong.
So you may want to simplify your deployments,
like make them at 1:00 PM every day.
Over time, you can become better at this.
And you can increase that frequency
in a controlled manner.
MARTIN OMANDER: And what if there are some
steps that are really hard to automate?
STEVE MCGHEE: Even if deployment to production
is not fully automated, you can still
make it semi-automated or scripted.
That reduces the scope for mistakes.
Your deployments shouldn't require
people to do complex things perfectly every time.
Automate away the complexity.
MARTIN OMANDER: And the serverless
products within Google Cloud support the CI/CD?
STEVE MCGHEE: Yes, they do.
Many customers use Cloud Build and Cloud Deploy or GitHub
actions to automatically test and deploy new code.
If you're using Cloud Run, you can
take advantage of its point-and-click integration
with GitHub.
MARTIN OMANDER: All right.
If I want more reliable deployments,
first I need to add CI/CD.
So let's say, I have done that, I have that all set up.
What comes next?
STEVE MCGHEE: So once you have CI/CD,
you should enable fast rollbacks.
MARTIN OMANDER: And rollbacks mean
reverting to the previous version of your system?
STEVE MCGHEE: That's right.
MARTIN OMANDER: But, Steve, if I deploy
a new version of my system, and there is a bug,
isn't it better to actually fix that bug to keep rolling
forward, as it were?
STEVE MCGHEE: That sounds like it should be true but, usually,
it isn't.
It can be quite stressful to investigate a bug
and come up with a fix.
Even worse is trying to do that while the system is broken.
Imagine that customers are complaining
or there's a chance that data is getting corrupted.
So if you create a bug fix while you're stressed out like that,
can you be sure you won't also introduce yet another bug?
MARTIN OMANDER: [LAUGHS] Yeah, that's a good point.
So it's better to roll back instead?
STEVE MCGHEE: Yeah, because it buys you time.
If you roll back to a version that you know worked,
you can now investigate the bug and fix it at your leisure.
No one is yelling at anyone, nobody's hair is on fire.
MARTIN OMANDER: And your list specifically says
fast rollbacks.
STEVE MCGHEE: Yeah.
Well, we want it to be fast every time
because we know that the system is broken
and we want to minimize the time that it is in a broken state.
Sometimes you hear about terms like MTTR,
mean time to recover, which relates to this.
You also want to make sure that your rollback is reliable.
That is, that you know it will succeed in one shot.
That way, you're not afraid of rolling back.
When you feel confident in rolling back,
you will do it more often.
In other words, you'll buy time for the development team
more often.
You'll reduce the number of times they have
to fix bugs under pressure.
It does mean that you need to put in the extra work
to make sure that rollbacks are fast and reliable.
MARTIN OMANDER: All right, very good.
The next item on your list was Canary releases.
How do they work?
STEVE MCGHEE: That's right.
The idea is that you expose your new changes
to a subset of your users first instead of changing everything
all at once.
This is generically called gradual change,
and it is a core principle of SRE.
If the new changes work well, you can roll out the change
further.
If your changes have any bugs, you
can roll them back before the entire system experiences
problems.
MARTIN OMANDER: And how would we do that?
STEVE MCGHEE: There are several ways of doing this.
Essentially, we need to decide how
to split our users across current and new versions
of our app.
Sometimes we call these assignments cohorts.
And you can make this as complex as you'd like.
You can simply randomly assign users to versions
based on load balancer assignment.
This is very simple, but it has some drawbacks.
MARTIN OMANDER: But that means that a user might
get the regular interface one time and then
the Canary release if they click or reload.
For example, the toolbar might change
as the user is clicking through the application.
That would be distracting, right?
STEVE MCGHEE: That's right, good catch.
So, to combat that, you could keep users
on the version they're currently on
and only send new visitors to the new version.
That way, users get a consistent experience.
Either way, we start by assigning a small set of users,
say 1%, to a cohort that sees the new version.
And we grow that set slowly and in a controlled manner.
If there is a problem, we use the fast rollbacks
we already discussed.
MARTIN OMANDER: That sounds like additional work.
Why would you do all this?
STEVE MCGHEE: Well, the beauty of this
is that it makes your system appear
more stable than it really is.
Let's say that the new version of your system has a bug.
If you release the Canary version to 1% of your users,
any given user will only run a 1% chance
of experiencing that bug.
If you didn't use Canary releases, 100% of users
would see the bug.
In other words, with Canary releases,
your system will appear 100 times more stable to users
than it really is.
Who wouldn't want that?
MARTIN OMANDER: Oh, I would want that.
[LAUGHS] But it still sounds kind of complicated.
How do we keep these releases straight?
STEVE MCGHEE: Well, we really only
need our CI/CD automation to keep track of three versions
at any time.
You should start with three simple tags, previous, current,
and next.
We keep previous in case we need to roll back.
Current is the version that serves most of our users.
And next is our Canary version.
When there's a new release, current
becomes previous, next becomes current,
and the new release becomes next.
If we're using Artifact Registry in Google Cloud,
we can add these tags on versions of our containers.
MARTIN OMANDER: And how would I set this up in Cloud Run?
STEVE MCGHEE: Ah.
So, if you're using Cloud Run and you just
want to do the simple randomized Canary method,
you just use traffic splitting feature to implement this.
To start out, you set the traffic percent
to 99% for current and 1% for next.
If you want the consistent user interface,
like in the toolbar example, you'd
enable session affinity to stick users
on a single version at a time.
MARTIN OMANDER: And what if my QA team
wants to see the new version?
Do we have to wait for them to be randomly assigned
to this 1% Canary release?
STEVE MCGHEE: Yeah, no way.
In Cloud Run, new versions have their own URL,
which you can directly share with whomever
you want to see the new version without having
to hope to be chosen to see the Canary version.
MARTIN OMANDER: So how long do I have to run my 1% Canary for?
STEVE MCGHEE: It depends on your application, of course.
But the simple answer is to let the Canary bake
at 1% for a couple hours, then progress to something like 10%.
Let it bake longer, then 50%, then 100%,
or whatever makes sense to you.
Basically, you want time to detect and react to any failure
as you grow this exposure.
If you're being very careful, you
might want to consider a full day
to take in any sort of daily traffic cycles.
For example, peak usage in the afternoon, or running
reports at midnight.
You want to catch those bugs too.
MARTIN OMANDER: So this might mean that a release takes
many hours or maybe even days.
What if I want to release faster than that?
STEVE MCGHEE: Correct.
In systems that really need stability,
SREs are known to keep canarying for up to a week.
In these cases, we don't want to sacrifice velocity.
So we do something called pipelined canarying
where the next version can start canarying
while the previous one hasn't finished yet.
This gets a bit more confusing and you
need more than three tags.
But we can leave this as an exercise for the viewers.
MARTIN OMANDER: I see.
So I can release several different features quickly
while still having the ability to roll back.
But it seems like there are still--
they are deployed in order based on how they were committed
to source control, right?
What if I have multiple teams working on new features
and I don't want to have them wait on each other?
STEVE MCGHEE: Right, that's where experiments come in.
MARTIN OMANDER: Ah, right.
That was next on your list.
What are experiments?
STEVE MCGHEE: They are similar to Canary releases,
but they give you more speed and precision.
Let's say your team is working on the new image
format, the new login page, and the new toolbar
that we mentioned earlier.
If you release all three using a Canary release
and there's a problem, it may be hard to figure out
which of these three features caused it.
MARTIN OMANDER: Should we put the three features
in three Canary releases released one after another?
STEVE MCGHEE: We could do that.
But, as we said, it would take time
if we serialized all these new features like that.
So if you deliver a few large changes, it may feel safer.
But, over the last decade or two,
we've learned that it's really not.
Again, more frequent and smaller changes are safer.
MARTIN OMANDER: But it wouldn't be
very safe to just push every code commit to the production
environment.
STEVE MCGHEE: That's right.
We need speed and precision.
Let's say your team is working on these three new features.
You can separate the deployment of your code
from the exposure of that code to your users.
That is, you can have hidden code that your users never
really see.
This is helpful because you can introduce
partially finished code and know that it
won't confuse customers.
But it allows you to ensure there aren't any subtle failure
modes, say, introducing a null pointer exception
or other crashing behavior.
We call this an invisible change.
MARTIN OMANDER: So we deploy these three new features
we talked about.
But they're not active, they're not visible yet.
But, at some point, we want to expose them to users.
How do we do that?
STEVE MCGHEE: That's right.
Now we switch modes from an invisible deployment
to an expected change type of deployment.
For example, we may decide to show the new toolbar
because we've now validated it works better than the old one,
it's bug-free in production.
Even though we pushed the code many releases ago,
we can choose now to activate or expose this feature.
MARTIN OMANDER: And then, would all users see the new toolbar?
STEVE MCGHEE: Well, we can have these predetermined user
cohorts, just like in Canary.
These would be based on a strong identity,
like your Google login or some other mechanism,
maybe using cookies.
We would define what we call an experiment, which
would result in exposing this change to this cohort.
So, initially, we can expose this feature
to a small subset of users without having to wait
for our release to go out.
Just like with canarying, we are then
able to ramp up enrollment over time.
And, as always, we would measure success or failure
and roll back if needed as early as possible.
MARTIN OMANDER: And what would that give us?
STEVE MCGHEE: Well, this method can
help you find things like emergent cross-feature bugs.
For example, what if the new toolbar we're launching
happens to overlap with the purchase button in our app?
We might not have thought of this to test.
And we only really know about it when it hits production
and we notice sales dropping.
By using an experiment, we can catch
this very early with a cohort of possibly even a single test
user.
This cohort could be QA testing or a business analyst.
In order to mitigate this problem,
we would just have to disable this experiment.
We don't have to roll back any code.
It's super fast.
MARTIN OMANDER: So these experiments
are the next level of Canary deployments?
STEVE MCGHEE: Yeah, that's a good way of putting it.
We measure errors and successes for each cohort.
If there are errors, we fix them.
If there aren't, we gradually enroll more and more cohorts
into this experiment.
Eventually, everyone would be in a cohort
where they are seeing this new feature.
Then we'd make that feature permanent in our product.
MARTIN OMANDER: How would you implement these experiments
and turning them on and off?
STEVE MCGHEE: Well, this is actually pretty
cutting edge stuff.
One option would be to build your own.
Or there are a few vendors which provide mechanisms for this.
Essentially, you need some administrative tools
like an experiment management dashboard, experiment
enablement mechanisms, as well as
a library for wrapping your code to enable and disable these.
MARTIN OMANDER: Right, it feels a lot safer
to flip a bit in a database than to deploy
brand new code to the Cloud.
STEVE MCGHEE: Totally.
It can also be used to be far more precise in launching
features or products at a certain time,
like at a keynote event or during a live broadcast.
Be careful, though, of the thundering herd effect.
Make sure you have capacity in place.
MARTIN OMANDER: Ah, very good.
Great discussion, Steve.
Let's recap how we can make deployments safer
using SRE thinking.
STEVE MCGHEE: Sure.
So CI/CD is the first way to automate your previously manual
release process.
Get those tests running and push on a schedule.
Fast rollbacks are a great capability
or a generic mitigation for teams
to react to almost any incident involving bad code pushes
or configurations.
Canary releases makes your code look 100 times better
than it really is.
Who wouldn't want that?
And, finally, experiments are an advanced capability,
giving teams more speed and precision in their release
process.
MARTIN OMANDER: I like that you ordered these items
for us, Steve.
Now, I myself, I can't do all of them right now,
but I know where to start.
STEVE MCGHEE: That's right.
They're in priority order for a reason.
That means, you can choose not to implement all the items.
Maybe you feel that an experiment framework
won't bring a lot of value to your application,
at least not now.
So start at the top of your list and work your way down
until you feel good about your deployments
and you're meeting your SLOs.
MARTIN OMANDER: [LAUGHS] That's a great point.
Thanks for joining me today, Steve.
STEVE MCGHEE: Thanks for having me, Martin.
MARTIN OMANDER: And thank you, everyone, for watching.
If you have any questions for Steve or me, please let us
know in the comments.
Also, do let me know if there are any other serverless
topics you'd like to see in future episodes.
I read every single comment.
Until next time.
[MUSIC PLAYING]