>>Matt Lowrie: Keeping on schedule with lightning talks, next, we have Emanuil.
>>Emanuil Slavov: Yep.
The clicker?
This one.
Hi.
So I'm the last one standing between you and the lunch.
[ Laughter ] So as you all know in Google, they value food
a lot.
So I'm going to try to be as quick as possible.
So three hours to three minutes -- I usually give this presentation for about 40 minutes.
And now I have to eat my own dogfood and do it for about ten, 12.
So let's start.
So what is this all about?
We had a bunch of API tests, and we used to run them every night.
It took three hours to run them.
And now they run for three minutes.
So this is the actual scale, 60 [sic] times faster.
This white dot, for you guys in the back, this is 3 again.
So that's the actual scale.
So how did we do that?
So now comes the before and after slide, like those infomercials.
So this is the actual screenshot of the Jenkins.
As you can see, three hours, three hours, the last one here was four hours and 40 minutes.
And we used to run them every night.
And what would happen in the morning, we usually -- we have to play whack a mole, find the
developers responsible for the breakage.
And it was hard, because you go to the first developer, and, what did you do yesterday?
And he would say, I don't remember.
I got drunk last night.
Don't ask me.
Ask the other guy.
Go to the other guy, he'd say, I fell in love with this girl.
I don't know.
I don't remember.
[ Laughter ] And it was really tough.
And so we said, okay, this -- enough of this bullshit.
So we set a goal: Let's run all of that for three minutes.
So why three minutes?
Basically, the idea was to give comprehensive feedback after every code change to the developers.
And after every code change, you run all the tests that you have no more smoke test, no
more nightly runs and so forth.
So this is the situation now.
So two minutes, two minutes, this is a little bit more than three minutes.
And, yeah.
You want to see some numbers today.
And the numbers are not going to be meaningful to you.
They mean to us, but not to you.
And you're going to see some techniques that are not brand-new.
There is no magic fairy dust.
But it's all about continuous improvement.
This is what we all have in common, all of us are unique snowflakes, unique challenges
in our companies.
But if you set up a goal and take small, incremental steps to achieve this goal, put the time and
the effort, you will achieve some amazing results.
So this is our journey.
So the first thing that we did is dedicated environment.
All the tests were running on environment that was shared with the developers that we
used to put their code together before pushing it on production.
We didn't have a dedicated environment.
So the first thing that we did, run everything in dedicated environment only for automated
tests.
So this is how we start, 180 minutes.
By having a dedicated environment, we went down to -- about an hour of the execution
time.
So the next thing that we had to do is databases.
We were still using the shared database, because it contained a lot of data useful for those
tests.
But this database was heavily used by bad jobs and some other stuff that happened during
the night.
So we figured out if we wanted to really fastest, we need to have a separated database.
But this was a problem, because we had to rework most of our tests to insert the data
that they need.
So we had to reengineer the framework.
It might sound a little bit crazy, but this is what happens right now before every test
case.
So we call a bunch of APIs, and we call -- and we manually -- if we don't have access to
the APIs, we manually change something in the databases, so add some rows just to set
up the playing field, just to do the setup.
And then the real test case starts.
Before that, it's only setup.
But having able -- being able to do that means that now we can run the tests on empty database,
on full, on dirty, on integration, on stage.
We don't run those in production, because it's a little risky.
But you get the idea.
Basically, we are free to use every database.
So we saved about 30 minutes with that.
So next thing, external dependencies.
We -- what we do is -- software as a solution, we collect a lot of data from the biggest
social networks -- sorry, even Google+ is also there.
[ Laughter ] Sorry, someone from Google+ team.
Okay.
So -- so the idea was that all the test cases had in one way or another connecting to external
service.
And if this service was down, say Twitter was down and the test case were failing.
And this didn't have anything to do with the developer pushing the new code.
So we wanted to basically stop fake simulate -- you know, pick whatever you like -- those
with something that we control so that -- now I believe Google code is just hermetic servers.
Like, you don't have access to the outside world.
Everything is wrapped up and doesn't leave your network.
So this is the -- I don't know if you see the font.
This is the landscape as of spring of this year, a bunch of open source tools -- Wiremock,
soapUI, Mock Server, stubby4j -- that we evaluated.
But we had some specific requirements, like being able to operate as a transparent proxy,
basically.
We wanted to monitor all of the connections and figure out which we want to fake and which
want to pass to the real service.
I wanted to be able to return some real binary data, have some local storage, (indiscernible)
responses, et cetera, and none of those would fit the bill.
So we didn't want to do it, but we had to develop our own tool to basically scratch
our own itch.
It's called Project Nagual.
It's going to be open source really soon.
We're waiting for some external stuff to happen.
It's really cool.
And you'll be able to check it out, I think, in a month or so.
So by having -- by stripping all the external dependencies, faking HTTP traffic, we didn't
gain much in terms of speed, but we gained a lot in terms of stable tests, for -- non-flaky
tests.
For us, the biggest reason for flaky tests was the external services.
And sometimes the Internet is down, very slow.
But now we control everything.
So, moving to containers, we didn't want to do that initially, but we had this big, monolithic
application.
And developers decided that it's a wise idea to start splitting this whole big monolith
to smaller services, and we basically had to follow suit.
But we didn't have enough servers to mimic what the production environment looks like,
so we had to use containers.
And we used Dockers.
So what we did was create one service, one container.
And when we run all the tests, we were actually in a big surprise, the execution time grew.
[ Laughter ] Yeah.
And we -- we started scratching our heads.
And you see I did a lot of scratching.
[ Laughter ] And -- until we figured out what was -- the
problem was.
We use Docker, so as you know, it uses layered file system.
And it's really bad if you have high disk I/O kind of operations.
So we measured that for our case, it was about three times as slow, the I/O performance,
than using real file system.
So our biggest user of file system I/O was the databases.
And so we figured out, okay, let's run the databases in memory, so -- to see how did
they go, because we don't need the data.
It's only fake data.
We run everything in memory.
Now so we go back to where we started, basically.
But at least it didn't get worse.
So next, don't remove test data.
What we used to have, logic in the framework.
Back when we were using the shared testing environment, each test case would create the
data, and it didn't -- at the end, it would delete the test data so that we don't pollute
the whole environment when we do manual tests.
But now we did not need that anymore, because we kept a dedicated environment.
And the thing is, when you stop the container, just the data evaporates, so there is no need
for this heavy logic to do that kind of search and deletion.
So we were down to about 46 minutes.
And it's not much, but every little bit helps when you are up in that point.
So the next thing that we did was the obvious thing that usually everyone starts with, so
run stuff in parallel.
But according to our experience, this should be one of the last things that you do, run
stuff in parallel.
When you run them in parallel and you get variation in execution time, so you don't
know if it's normal variation or it's something caused by, you know, you trying to improve
the system.
So try to expend all your other options, and then try to run in parallel.
That's an actual graph of, you know, the threads that we run test in and the execution time.
Each one took -- as you can see, it's easy to see the optimum value is 10 threads for
this.
This is per hardware specific.
So if we add more memory and CPU, it may shift to the right.
But we run those in 10 threads.
And so the execution time was about five minutes.
Still a little bit off our initial goal of three minutes.
So we had to do something else.
And it turned out -- equalizing workload.
It turned out that the software that we used to run all the tests, the driver, it did not
do a good job to separate the test cases equally into each thread.
So, as you can see, on thread number 3, had 130 test cases in it.
So we -- everyone had to wait for thread number 3 while the others were idle.
So we had to manually adjust the workload.
So this is the picture after.
And it's not perfect, but it did the job.
So we were down to -- yeah, so we were down to about three minutes.
So this is how the whole picture looks like.
The actual results were even a little bit better.
So that's the screenshot.
And they took two minutes and 15 seconds.
But this doesn't make for a good presentation, I know.
It's a mouthful, three hours to two minutes and 15 seconds.
So stick to three hours to three minutes.
And at this time, we have exhausted our options to optimize (indiscernible) software, so our
only option now was to go after the hardware.
So a couple of months ago, we upgraded basically most of the RAM and the CPU, the machine that
we run.
And now the tests run for one minute and 38 seconds, without doing anything else.
This runs on single machine, on single server.
If we want to go below the one-minute mark, and this is what -- right now what we are
doing, it's super easy right now to scale horizontally.
So this is our current option.
This is what we are going to do.
We are aiming for sub one-minute execution time.
Just for comparison, our unit tests run for about 45 seconds, about 3,000 of them.
But they don't touch the database, the network, and the file system.
Those guys do.
So we -- So this -- we have high-level tests.
These are the most of the complaints that you get.
You know, compared to unit test, everything else is slow.
So we accidentally address all the issues that we had with high API and UI level tests.
So they are -- they aren't slow now.
They run for three minutes, and we're aiming for less than one minute.
They're not unreliable.
Like I said, we control all the dependencies and no one else uses those machines so they're
dedicated for fast tests.
The entropy is lower.
This is a little bit harder.
Cannot pinpoint the problem exactly.
Sometimes when your tests fail, high-level test, you get internal server error and you
don't know -- you're lucky if you get a stack trace.
Most of the times you just get internal error.
Something happened somewhere.
So it's really hard to figure out exactly what.
But having so much faster tests actually changed the behavior of the developers.
Now they start getting used to this fast feedback.
And before they used to push their code once a week.
From us, went two or three days a week, to once a day, to every hour.
And when they push three lines of code and the tests fail, then it's pretty obvious which
of the three lines is the problematic one.
So by having super cheap and fast, reliable test run, so we actually run everything on
the commit.
Like we run static code analysis, we run linters, we run custom tools, unit tests, and those
tests.
So there is nothing else to run after each commit.
We are able to push in production or continue to manually test, if we like.
So -- And those are no longer problems for us.
Yeah, so what we think is that in the future, this is what we should aim for.
To be able to run all of your tests, not only the unit test but everything for no more than
three minutes.
We think that this is going to become a standard development practice the way right now we
put all our code in version control system and we use CI.
So couple of years ago this was considered a crazy practice.
But what you want to do is keep developers in the flow.
What happens if you have a pipeline that it takes 15, 20 minutes to run all your tests,
so developer pushes their code, and what happens is they go -- while they're waiting, they
go to check their Google+, set no on ever, Facebook page.
They go to make a coffee, make a sandwich.
They check their, you know, Tinder app, Grinder, whatever the thing is.
And when they come back 15, 20 minutes later they have forgotten all of that stuff.
Build failed?
What did I do.
The idea is to keep everyone in the flow; okay?
So keep them in the flow.
That's my last slide.
Two books to help you with your journey.
The first is "The Goal" by Eliyahu Goldratt.
Introduces a tier of constraint.
So basically you have one big bottleneck.
You work on this bottleneck.
Then you have additional bottlenecks, but they are not visible until you fix the bigger
bottleneck.
Basically reads as a business model.
The next one is Toyota Kata by Mike Rother.
Introduces continuous improvement kata.
So you set up a goal and you take the small PDCA cycles by Dr. (saying name).
And every day you take small steps and small steps, and when you hit a problem you take
a step back, the way we did with all the containers.
So that's all I had to say.
[ Applause ] Thank you.
>>Matt Lowrie: Excellent.
Thanks, Emanuil.
And I thought my comments on JavaScript were harsh.
Google+ burn!
Ouch!
>>Emanuil Slavov: I hope they didn't do a parking (indiscernible) therapy at Google+.
>>Matt Lowrie: I think you're okay.
We probably have time for one question.
How do you ensure your API stubs stay current if and when the stubbed API changes?
>>Emanuil Slavov: Yeah, that's a very common question.
So there are three things.
The first thing is that we deal, say Facebook or Twitter APIs, and they announce a new version,
and we initially don't switch first to this version.
We switch only internally to using the, say, version 2.8.
And if we detect some problems with the stubs, with just the stubs, we have time to do that.
And when we are all set up and we think we are ready in testing, then we push to production.
Then we use two other things.
We also do extensive monitoring.
So if we monitor some problems, we -- with just the stubs.
We also -- third thing that we do is we do a lot of contract tests.
We basically ping Facebook or Twitter or Google+ for do we -- is this a JSON that we expect
to be?
So every ten minutes or less we do those kind of tests.
And we did not used to do those kind of tests, but what happens is that Facebook at one point
changed a little bit in their JSON responses.
They added additional field, just for one day, and it caused us thousands of dollars
of losses for us because we did not handle this case.
So we learned our lesson, and we use contract tests to check if what they return is really
what they said they would return.
>>Matt Lowrie: All right.
Very good.
Another round of applause.
[Applause] >>Emanuil Slavov: Thank you.