>>Tony Voellm: Okay.
So up next, we have this very action-packed set of lightning talks.
Again, each one is 15 minutes, including time for questions.
So it's really up to the speakers to determine how many questions they'd like to get.
First speaker up here is going to be Eduardo Bravo from Google, talking about building
scalable mobile test infrastructure for Google+ mobile.
So you're starting to see a bit more on the application side.
With that, Eduardo.
>>Eduardo Bravo: Thank you.
Hi, everyone.
I'm Eduardo.
I work at Google.
And as you guys probably figured out already, I happen to work for the Google+ mobile app.
And today I want to talk to you about what we've been doing for the last year and a half.
I want to tell you about the test infrastructure we have set up in place for the Google+ mobile
apps.
And I want to share with you the experience and the problems we have faced when we were
doing this.
I know mobile testing is still very painful.
I mean, we saw the matrix -- the test matrix; right?
We know how painful this is.
But I think after a year and a half, we're at a point where we're comfortable with what
we have.
Don't get me wrong, there are still a lot of things to do.
There are still things we haven't figured out.
But we're in a good state compared to one year ago.
So let's -- first things first.
What was our motivation to heavily invest in mobile test infrastructure?
Well, you know G+, we were late into the game.
We needed to catch up.
Right?
We needed a very aggressive development schedule.
We know manual testing won't scale to the demands of the fast-paced development we have
and of our release process.
So it was a no-brainer.
We need to do this.
We strongly believe in preventing bugs rather than finding them.
We are obsessive about precommit.
We try to pack as much tests as possible as we can.
We cannot do all of our tests, because we have some very slow tests, but as much as
possible, we try to clean -- to keep our build clean, make sure we don't introduce unwanted
bugs.
So we really take precommit seriously.
And, finally, which is my favorite point, which is flaky tests -- the team feels very
strongly about this -- flaky tests are worse than having no tests at all.
Now, let me explain why.
You see, we have Joe.
It's 5:00 p.m.
He's a little bit bored.
He already finished the work he had to do that day.
And he's like, well, I don't have enough time to write a new feature, but maybe I will write
a test.
Why not; right?
Let's give it a shot.
So he goes there, jumps into Android, writes his test.
Submits it, passes precommit.
He is very happy; right?
It's a good day of work.
You write a killer feature, and you write a killer test.
Ten minutes later, we get the email, hey, the test is failing.
And he looks at his steps, right.
He goes, like, oh, look at that, I just find the first bug with my test.
I should be very good at this.
As he is trying to figure out what's going on and where is the failure, he realizes that
the build turns green.
So the test is flaky.
This happens three or four times.
And after a while, nobody cares anymore.
The build host is polluted.
Whenever there is a failure, most likely people will think it's one of the flaky tests.
So that's why at Google+ mobile, we strongly believe we'd rather have no tests than flaky
tests.
And to make this, we believe in having a hermetic environment.
If we cannot provide an hermetic environment for our tests, we would rather not write them.
So what do we need to test?
We need to test Android.
We need to test iOS.
And, of course, we need to test the Google backend server.
The Android app and the iOS app have a similar release cycle.
So in that sense, it's not too bad.
The Google backend, it's a different story.
It's more aggressive, very fast.
But still we need to make everything play nicely.
So let's take a look at how our test infrastructure looks like.
For Android, we care about faster jUnit tests.
We use Robolectric.
You guys heard about it yesterday.
And just to give you a quick refresher, it allows us to decouple the dependency that
the jUnit tests have on the Android emulator.
This for the Google+ app is critical.
The Google+ is huge, so compiling itself is an expensive operation.
Then we need to ProGuard it to make sure we don't pass it Dalvikly that some app levels
have.
And, finally, we need to install it, which takes around 20 seconds.
So all this adds up.
Luckily, Robolectric comes to the rescue.
With all of that, we can compile fast and run the test even faster.
Faster development, happier devs.
Happier devs, more likely to write tests.
Then our second strategy for testing in Android is our UI hermetic tests.
We use Robotium, or let me rephrase that, Goobotium, which you guys can figure out where
the name came from, which is basically our implementation based on Robotium, for the
UI test framework.
We're considering moving to Espresso, which you guys are going to hear later about it.
Let me show you how this UI stack looks.
So if you can see the normal flow, that's how it looks -- that's how a normal test will
look.
We have -- we run all our UI tests on emulators.
We install APK, we trigger our tests, we connect to the backend, and there is the login server;
right?
What's the problem in there is all those dependencies that we have on external services.
The more dependencies we have, the flakier our tests are going to be.
And we don't want that.
It's very hard to work with flaky tests.
Hence, what we do is we start in memory Google+ fake backend server and set Android emulator.
We point the app against it.
In fact, the app never knows it's talking to a fake.
And this fake server is able to provide precanned data.
We can control this data.
Therefore, our tests are deterministic, and we have cut all the dependencies we have on
external services, which don't really matter for the UI part of the testing, as well as
we don't have to deal with the authentication problems that we will have with the whole
flow.
Finally, our third strategy is use Monkey tests.
If you guys are not familiar with Monkey tests, they come with Android SDK.
They are very easy to set up.
They are very effective.
These are the only tests we run on real devices.
They catch a lot of bugs.
And if you guys are not running these tests, you should really consider doing it.
They are the best bang for the BUCK.
Now, how do we do it in iOS?
Well, for our unit tests, we use a unit test case suite based on GTMSentestcase, this comes
bundled with Google Toolbox for Mac, so any one of you guys can use it.
It has worked very well for us.
And for our hermetic UI test, we use KIF.
We used to use UI Automator, but it didn't -- it was flaky.
It felt unnatural for our devs.
So we decided to move to KIF, which is developed by Square.
So kudos to the Square team for doing this.
It has proved way more stable than the UI Automator test.
And also, the way it works in process makes our test less flaky.
And on top of that, as I said, we care about hermeticism.
The way we do it here, which is a little bit different from the Android world, is instead
of having to start a fake server, what we do is we use (indiscernible), a feature.
And what it lets us do is, at run time, we can be able to feed pre-canned responses to
the iOS app.
That way, we can control the set of data we are fitting to each test, to each run, and
everything is hermetic.
There are no external dependencies outside of the iPhone simulator.
This has proved very effective.
The devs actually really like this approach.
And it has been much more stable than all the UI Automator setup.
Now, my favorite part.
We have tested all the clients; right?
But we also need to test the backend.
They all have different release cycles.
And we want to make sure that, yes, mobile apps don't break.
But also that the backend doesn't break mobile apps.
So we have two test strategies here.
First, we have end-to-end testing on Android where we start an emulator.
We start the Google+ app.
And we start the whole stack of the Google+ backend server.
We provide fake data, because we want the data to be deterministic.
Each test gets the same data.
And our test exercises the whole server stack.
The tests we have here is -- there are just a few of them, a handful of them.
As much as we would like to have more, these tests tend to take longer, so they are harder
to enforce on a precommit.
But still, they are good to have.
You really want to have some true end-to-end tests just to make sure nothing goes terribly
wrong when the backend servers pushes.
The second ones, which are my favorite ones, which are the replay tests, which are very,
very simple, but they have proved to be very, very effective.
How they work is, they simulate mobile clients' requests to the server.
And when the clients expect a certain -- (indiscernible) response back.
So let me show you with the diagram.
So if you remember from the previous slides, that's the normal flow.
And with the replay test, what we do is we chop away the Android player.
We have a set of golden requests that we know our clients generate, our mobile clients generate.
We start our backend server, again, providing fake data.
And as we're sending a request, we're asserting -- we assert that the respond corresponds
to the call and response.
If it corresponds to it, this means the server won't break the mobile clients, because we
know when we tested that, that response makes the mobile clients work correctly.
We have some smart (indiscernible) because there are things that change with requests,
like time stamps and things like that.
So we make sure that doesn't cause flakiness.
But this setup has been very, very good to the Google+ app.
In fact, this is the one that has caught much bugs.
And, yes, we have all these tests running at every commit.
I'm sorry for the image.
I was -- I hoped I was -- I could find a better one.
I am not really good at it.
Obviously, the Barack Obama one was much better than mine.
[ Laughter ] >>Eduardo Bravo: But I'm not very good at
this type of thing.
So what I'm trying to say is, we have three test strategies on Android, we have two for
iOS, and we have two for our backend.
And we guarantee that those tests are run at every commit.
Most of them are also run on precommits, which is what we really care about, too.
And, finally, I want to share about some of the lessons we have learned during this process.
First, since the beginning, care about making the mobile infrastructure scalable and easy
to use.
Don't gives devs a reason not to write tests.
The easier it is to write tests, where you provide the right environment, they will be
more likely to write more tests.
Second, mobile testing needs to prevent bugs, not affect productivity.
If you are spending too much time fixing a test, that affects productivity.
There is something wrong somewhere, either on the tests, either in the environment.
You don't want to stop development.
Reuse frameworks whenever possible.
Yes, we're Google, and we like to do things our own way sometimes.
But there are times also that it's also good to reuse.
And some of the frameworks I mentioned, like Robolectric, KIF, Google Toolbox for Mac,
are all available.
Everyone can use them.
They work very well for us.
And whenever you can, reuse.
And, finally, let me reiterate on this, keep tests hermetic.
Make sure you provide the right environment so people don't have to be worried about it.
And if you cannot provide it, don't write the test.
It's not worth it.
It's going to be more painful later.
Thank you.
[ Applause ] >>Tony Voellm: Yeah.
Nice job, Eduardo.
Very nice.
Thank you.
That always reminds me of how we build software at Google.
We have what is working, what is deprecated, and what's new but not yet working.
He found a nice, happy medium by finding something that works.
With that, we have time for one quick question.
We'll go off to the right here.
>>> Are the end-to-end tests flaky?
And if not, how do you keep them from being flaky?
>>Eduardo Bravo: They are not flaky.
We think Espresso is going to be the solution for it.
For the end-to-end tests that we show you, the flakiness is 0.2, which is really, really
good for the type of test that it's exercising almost full Google+ stack.
>>Tony Voellm: 99%-plus pass rate is pretty phenomenal.
With that, thank you, Eduardo.
[ Applause ]