Hey, there.
My name Is Brett Slatkin, and I'm a software engineer on the
Google Surveys Team.
I'm here today to tell you about the secret to safe,
continuous deployment.
You can find me online at onebigfluke.com.
So here's our agenda for today.
We're going to talk about the why and how of continuous
deployment.
I'm going to talk to you about making mistakes while doing
continuous deployment.
Then we're going to talk about how to be safe while
you're doing it.
And then finally, I'm going to show you some examples of our
team being safe with continuous deployment.
So the why and how.
This is your typical software release process on a team.
You write your code.
You commit it to your source code repository.
And at that point, you're really happy, and you have a
lot of motivation, because you've gotten
this feature to work.
You branch the code.
You maybe push it to a staging server.
And then you get to see it kind of live in a mostly
production deployment.
But then after that, nothing happens for weeks.
There's a release schedule, and it doesn't actually get
launched until everybody's ready, and the bus is leaving
the station.
So what I found on the App Engine team is that we would
have this release process that, to me, it was really
hard to stay motivated, because the time from when I
wrote the code to when it hit production was so long.
I would be happiest when I wrote the code, and
when I saw it live.
Now while I was working on App Engine, I was also working on
PubSubHubbub.
This was an open source project, so we would push this
to production every two days.
So in the span of time that it would take one release for App
Engine, we could do 10 for PubSubHubbub.
And that made me really happy.
I realized that seeing my code live is what made me happy.
And then that reminds of this quote from Ian Bicking, who
works in Mozilla.
And he said, "When it comes to deployment, App Engine is the
first thing that has really felt good to me." And that's
what really kind of brought it all together.
I was working on App Engine infrastructure.
I was also building an app using App Engine.
App Engine is great for deployment.
It makes it very easy to deploy your app.
And that's what I found was what made me the most happy
about PubSubHubbub.
I just loved writing the code and deploying
it as fast as possible.
So I was browsing the web one day and I came across this
article by Timothy Fitz, called "Continuous Deployment
at IMVU," which is a start-up, "Doing the Impossible 50 Times
A Day." And the process he describes reminds me a lot of
the tool that we had, appcfg update.
They made it ridiculously easy to deploy their software to
production.
They would do it all the time.
And I thought, I could use the same tool to do the same kind
of thing very easily.
But I never had.
So then, we were starting this new project called Google
Consumer Surveys.
And like any new software team, we were deciding on the
stack that we wanted to use to build the software.
So we wanted to use Django and Python.
We wanted to use Closure Library.
We wanted to use App Engine, and JavaScript, and HTML5, and
everything that it has to offer.
But we thought, if we're going to be deliberate about our
software stack, maybe we should be deliberate about our
processes beyond that, such as deployment or testing.
And so we did that.
So we adopted this as our process for continuous
deployment.
It's a kind of serial test and push.
It's kind of your textbook example of continuous
deployment.
So someone writes the code.
You do code review with a peer.
You check it into revision control.
Once it's in revision control, you run automated tests to
make sure that your code's passing tests.
Then you automate deploying to a staging server.
Then later on, some other person looks at the staging
version, makes sure that it looks good.
And if it does, they set that version to live, or make it
the default version.
So now we've been doing this for over a year.
It's been working really well.
And so here's a kind of a plot of the number of deployments
that we do over the year.
And this is the number of deployments per day.
So you can see we've peaked at 40 deployments in one day.
You can see that where the graph goes
to zero is the weekends.
And you can see our average number of deployments per day
is actually very good over time.
Here's another graph, which shows how long a version stays
live in production after we've pushed it.
And you can see on average during the week, it's less
than 24 hours, which means that we're constantly pushing
new versions during business hours.
And then on the weekends, we don't push software.
And you can see our time to deployment goes over 24 hours
to three days.
So that was working great, until we
started making some mistakes.
And here it is.
So I pushed a pony to production for all of our
paying customers.
This is one of the stupidest things I've
done in a long time.
The whole idea was that it was supposed to indicate
experimental features.
A lot of engineers call features that you want, the
pony features, something like that.
That's what we were trying to do here.
I didn't realize that it would be enabled for all users.
I thought it would only be for administrative users.
But it turns out, I pushed it for everybody.
And that was a big mistake.
When you talk to people who work in dev ops and do any
kind of system administration at all, this is their
nightmare scenario, this idea of having a pony in
production.
It's just embarrassing to the product for your users.
And I did this.
And that was a big mistake.
And it's not like we didn't have tests.
Like I said before, we had automated tests to make sure
that everything was good.
But nothing actually tested the site to say like, is there
a pony dancing on the screen on that page?
It's like, no, didn't test that.
Didn't think about that one.
We also had all kinds of monitoring and graphs.
We knew our error rates for data store operations, and
JavaScript client errors, and so on.
But none of that would surface a dancing pony on the screen.
And so here are the things we tried and failed.
We had monitoring.
We did test and automation.
We had continuous integration of our source code.
We did peer code review.
We deployed this to staging even, and we
saw the pony on staging.
But the pony was expected behavior on staging.
We couldn't tell the difference between the staging
behavior and the production behavior.
So that didn't work.
And then having a QA checklist wouldn't have caught the pony,
because it's not like, make sure there's no pony on this
screen for this thing.
It just would've failed.
So all the ideas we had didn't work.
But then we came up with this other idea.
And this is the secret I'm talking about.
How to be safe.
So what you're looking at here is something you may not have
seen before.
It's called the perceptual diff.
It's a perceptual difference.
It's kind of like a diff on files that you've changed when
you've written code.
But it's for the pixels on the screen as part of the website.
So on the left is an old version of this page.
On the right is a new version of this page.
And what you'll notice is that it's almost humanly impossible
to tell the difference between the two.
It's kind of like "Where's Waldo?" It's going to take you
a long time to find the actual pick it out, the actual
difference.
It's next to the globe.
It was something to do with the way that the pricing was
indicated on our page here.
And I have it highlighted in red below.
Because what we did is we hooked up a system for doing
this with a computer.
So computers are really good at detecting
changes like this.
And so we let the computer do the work for us.
And it was able to do a screenshot of the page on the
left, a screenshot of the page on the right, and then diff
the two perceptually, and show us what's changed.
And that's what's in red at the bottom.
Here's another example of the same thing.
This is our reporting interface on the left.
And on that page, there's a mistake.
There's a problem with the build.
It's down on the bottom left.
There's a histogram of response times.
It's pulled out here on the right.
So you can see what we expected on the top was 0 to
60 seconds response time.
What we found was the x-axis was totally garbled.
And what's awesome is that perceptual diffs are able to
automatically pick out this change and show that to you,
which is highlighted in red below.
So we're using a tool now to do this that automates this
process for many, many different parts of our website
to figure out, how is it changed, and have we regressed
in any way that's hard for a human to see?
So again, we screenshot the old release that's live in
production.
We screenshot the new release that's on the staging version,
hopefully using the same data, if possible.
We pair the URL paths together so that we're comparing--
even though the host names are different, the
paths are the same.
And then we visually diff the pixels.
So we tried this a few times for different
parts of our site.
We were actually just doing this big refactoring of some
of our user-facing code.
And we wanted to make sure that it was visually exactly
what it had been before.
But we thought, why don't we just use this for everything?
Why don't we use visual diffs for every single part of our
application?
So we have.
So here is our old process before.
And now we've modified it to add this parallel step.
Once the code hits revision control, we deploy it to
staging at the same time that we run automated tests.
And then we also do perceptual diff tests at the same time.
So the release pusher who's deciding, is this release
ready to go or not, they look at the test results of the
unit tests and the integration tests.
But they also look at the perceptual diff tests, and
they say, has my site visually changed in a way that I did
not expect?
And if it hasn't, then you set the version live, and you're
good to go.
So let me give you some examples of pdiffs.
I think it's really interesting to see the types
of problems you can find doing this.
And so I'm going to highlight a few of them.
OK.
So the first of these are things you'd expect to see.
So here, we added a button, this little building button up
at the top.
And here's the pdiff.
So on the left, there's the building button on the right.
That's before there was no building button.
So we added it.
You can see the diff.
Here's one when the Giants won the World Series.
And it was Halloween, and we thought, hey, let's do
something silly.
We're based in the San Francisco office, so we ended
up putting some Giants colors on here.
Did that for a day.
That was an expected change.
Here's another one, where we totally revamped the
Confirmation page for our billing flow.
And you can see on the left is how it used to be with this
gray kind of box.
And on the right is the new way we have it, where it's
kind of laid out in a grid.
At the bottom, you can see the diff.
All of the different pieces totally changed.
Very obvious diff.
Here's another one, where we started doing surveys along
with the Harris Poll.
And we had a promotion that it totally
changed on our home page.
Pdiff totally picked this up.
And that was pretty easy.
Where it starts getting interesting is cases like
this, where it's less obvious.
When you first glance at these two, it's really hard to tell
what's changed.
What's changed is at the top, you'll see that now there's
three steps instead of four steps.
So pdiff is able to pick that out really easily.
That's another feature launch.
Pdiff can also pick out this color changing on the map.
We had all blues before.
It was really hard to see, so we changed the color scheme.
So it picked out that all of that had changed, even thought
it was just by a little bit.
And then we also have another feature launched here, where
we added frequency to our surveys.
So that's kind of the bigger, chunkier changes.
What's interesting is the ones you can't
really see as a human.
So on the left, the padding on top of this div
changed by 40 pixels.
The diff showed that it had shifted.
And if you look at it visually,
now you can see that.
But you really couldn't pick that up unless you were
expecting it to be there.
And this is something that pdiff is very good at.
It's regressions.
It's very good at.
Here's another one.
These are two different versions of the same chart.
They look still to me identical.
But we subtly changed the way that the bar width calculation
was made, and the spacing between the
labels and the bars.
And pdiff totally picked this up, and showed actually it's a
huge change.
But it's almost impossible to see as a human.
Here's another one.
We had a table of information.
People glaze over when they're looking at tables of data.
And we lost the Midwest and regional bias table.
And pdiff was able to pick this up and show us like, hey,
this thing totally moved, and you lost this.
And it turns out it was a bug, and we had to go
back and fix it.
And then here's one of my favorites.
We changed it so there's a single pixel line above the
charts that moved.
That's it.
So pdiff saw this single pixel line and made us realize we
had a regression.
The diff below is actually very large.
But to a human, it's almost impossible to see that this
has changed.
So that was like, a really helpful one.
So the subtle differences is what it gets really well.
Here's another example, where the sort order changed on this
from lexical to numerical, or numerical to lexical.
And it's another example of, a human wouldn't see this.
We just glaze past it.
And pdiff can see it really well.
And so finally, we're using diff for everything now.
We even diff the emails that we send to our customers as
part of our onboarding flow and the purchase flows.
So this is an actual rendering of an email that we would send
to a customer, and we diff it on every release to make sure
that those emails look good and don't change over time.
Because once you send an email, you can't take it back.
So anyway, the conclusion of all this is that pdiffs are
the best thing.
If you have anyone you've ever worked with who says, well, we
can't do continuous deployment because it's not safe, this is
the thing for you.
Perceptual diffs are what make it possible to do it safely.
And that's what you should be looking at.
There's a lot of great tools now if you're doing this.
PhantomJS is the big one.
So PhantomJS is a WebKit build that does this-- it's a
headless WebKit that does screenshots.
Slimer's the same thing for Firefox and the Gecko engine.
And then PhantomCSS is a kind of nice wrapper around this to
make it easy for CSS refactoring changes.
You can do all kinds of integration with Jenkins, and
Travis-CI, and CircleCI, and there's many others, to do the
actual build and release process.
And then for deployment, there's a lot of companies
that are doing deployment right now.
A few open source projects.
But generally, we need more continuous deployment tools,
in my opinion.
There's also an open source project that I'm working on.
It's called dpxdt.
You can find it on GitHub.
And that project's still just starting out, but we're making
some progress to make this tool really easy
for anyone to use.
And then, you're more than welcome to check out my day
job on g.co/consumersurveys of where we're using continuous
deployment and perceptual diffs, building our product
and releasing all the time.
So thanks.
That's all I have to say today.
And thanks for listening.