So hi, my name is Ed.
I'm on the Mobile QA group at Google.
I'm working on a small search space project that you
probably don't interact with.
But this is a duh talk.
So this is trying to get across an insight that I have
that is probably perfectly obvious to everybody else and
it is perfectly obvious to me once you realize it.
So sufficiently advanced monitoring is
indistinguishable from testing.
So here's some basic facts, or basic observations I guess you
should say, that we run across a lot at Google.
Complex systems are unpredictable.
We've got big data centers running on lots of different
code and lots of different layers that changes all the
time and some other group's push can break your code and
there's just no way to predict that or really
prevent that sometimes.
Users, similarly, unpredictable.
They do some really crazy things sometimes, either
trying to break the systems on purpose or just accidentally
or just act in ways that you didn't anticipate in any way,
shape or form.
And the other thing to bear in mind is that a lot of the
problems that we really care about only happen in
production.
If something breaks when it's still in build cycle, you
don't get yelled at, you don't get paged.
You manager doesn't dock you promotion points,
that sort of thing.
But if something breaks in production, it's very visible,
you want to fix it very, very fast and you want to know
about very, very fast, so that means the you inevitably have
to end up doing monitoring on your production systems. And
of course, page things when people break and then they go
and fix them and hopefully nobody really notices, because
there are lots of failures that are not complete breaks.
But, if we have to monitor our production systems anyway, we
end up in this kind of silly cycle where we've got all
these really thorough regression tests and
functional tests that get run at build time and we ship the
code off to the data center and then we monitor it with
these tiny little queries.
You log in at one particular user, you see if the home page
is there and that's it.
But we've got all these wonderful tests, why aren't we
using all these tests in the production
system all the time?
Instead of logging in as one particular user and just
seeing whether the system is up and running at all, why not
go ahead and run your tests one per minute, all the time?
See if anything fails in production and just unify
these two concepts of monitoring tests and
functional tests.
So the other thing that you can do is use your users as
your monitors.
These production systems are running all the
time with real data.
Every event that's happening on your server is a test case
that you didn't have to think of.
It's a real test case, it's real data.
And it's doing something that you maybe wouldn't have been
able to think of yourself because users do crazy things
all the time.
And if your monitors that are looking at your servers doing
these crazy things to data that you don't necessarily
understand are actually trying to validate the state, well
hey, then you can take these interesting monitors and turn
them back into a test suite by just
simulating user behaviors.
So you can run these same tests prior to deploying
production servers just by throwing a lot of traffic at
them and seeing what your monitors do
on the staging server.
Brief, very nice.
So one technique that we found to be very valuable for this
is to use the monitors to kind of capture the initial state,
what query does the user type into the search box and then
follow that throughout the system, maybe only one in a
thousand queries because you don't want to overload the
servers, but specifically probe and verify the followup
states that you expect to happen during
this sequence of events.
So the thing that I really like about this is that doing
tests in this way forces you to be very general.
You don't write test cases with one specific input data
and one particular expected outcome.
That's useful too, but it's much better if you have the
ability of tests that can adapt to any sort of weird
user input and still be able to validate what a correct
result would look like.
The other kind of side point to this is that logs are real
untapped resources, at least around here, unfortunately.
The local code is spewing messages all the time that we
may not really be looking at and global announces of logs
can follow these events through the system.
And just one anecdote.
Whenever I have deployed a new monitor, I've found a nasty
new bug and one thing that really helps is that going to
you developers and saying, look, this happened on 10% of
user queries beats this happened to me.
Thank you.