>>Tony Voellm: So the next speaker up is Michael Klepikov, who was also one of the organizers
of all of this. So he deserves our thanks for that.
I also learned he is quite a risk taker. He tells me he had proposed this talk before
he had written any code that was going to be used as part of this talk. So with that
I will let him talk about Web performance testing and WebDriver. And as a shout out
to performance and security I am going to be giving out some Androidies for good questions.
So with that, I will hand over to you. >>Michael Klepikov: Thank you. And I would
like to especially thank the Chromium team and Ken Kania in particular whose excellent
talk you have heard yesterday, for working with me and taking patches.
So I am going to show you how you can use WebDriver to enable performance testing in
your existing functional WebDriver test. Now, why is performance important? I think
most people in this room, it's safe to say, care about performance of their apps. A faster
app obviously makes a much better user experience, and there are studies that performance is
money. And this is all exacerbated on mobile, on high latency networks where even though
mobile is becoming faster, mobile networks are becoming faster, latency is going to continue
to be a problem for the foreseeable future. So why test performance? Why not just go through
our pages and optimize them, then be done with it?
It turns out Web development is hard. Modern sites are complex. Browsers are complex. And
as people work on features, it's very, very easy to inadvertently introduce a performance
regression. And that's how most of them happen. And once the regression gets into production,
it's fairly expensive to get it fixed. Many people are involved, and you kind of spoil
the user experience. And performance problems are especially nasty
to debug. So it could get very expensive. As opposed to if you catch it close to where
it was introduced, it's a lot cheaper when only the developer is involved, ideally even
on presubmit before the regression ever hits the source tree.
So what tools are available to test performance? There are a number of tools, including, for
example, open source WebPageTest, which I happen to be working on. They have been
focused so far on page load performance testing where you submit the URL, and it runs it under various
conditions, various browsers and it gives you the results and tells you what can be improved,
what is fast and slow on the page. There has recently been WebDriver integration
in WebPageTest where you can submit a JavaScript WebDriver program it'll run it for you.
So that's all good. But it's expensive to integrate it into an existing tool chain that
you may have already as a continuous build and continuous testing, customized to your
organization's needs. WebPageTest, for example, has its own idea of what a test job
is, and if you want to start, for example, a web server as part of the test, that gets
tricky. That gets very nontrivial to integrate. So how can we make it easier? How can we spread
performance testing, make it east easy for organizations and for developers especially?
Why not integrate it into the WebDriver itself? Right, so it turns out WebDriver already has
Logging API, which fits nicely with that purpose. Basically, you ask when you create WebDriver,
you ask it to collect certain types of logs, and when you're done with the test, you ask
WebDriver -- give me back those logs. So why don't we enable performance instrumentation
of the pages under test as the test runs. That would fit very, very nicely into an existing
tool chain. You can inject it pretty much transparently for a test, and you can just
use any existing functional test that exercises whatever functionality you care about, and
instantly gives you performance instrumentation for that functionality.
Now, how does that work in WebDriver? There is a notion of capabilities that you pass
when you create a driver instance. Capabilities are essentially name value pairs and some
of them are designed to turn on certain types of logs.
So in the latest release of the ChromeDriver that Ken Kania has talked about yesterday,
there is the profiler log type that you can enable as you are creating the WebDriver instance.
And again, you can inject it as transparently to the test as you like. You can do it in
the setup method, you can do it through a WebDriver builder abstraction, you can even
proxy the WebDriver wire protocol and inject it that way, make it as transparent as makes
sense for your, again, custom tool chain. The test itself remains unchanged completely,
unless you want to get advanced and inject your own, you know, markers in the timeline
that you would ultimately get, right, to aid in future debugging.
So it's free for developers. Developers don't need to do anything. It's injected by the
framework. At the end of the test, again, injected, you
ask the driver for a certain log type, and the log entries, in ChromeDriver in particular,
are the DevTools or WebKit Remote Debugger Protocol JSON events for the domains Page,
Network, and Timeline, and that lets you track performance and memory consumption and things
like that that will tell you when it does reflows and runs JavaScript and so forth.
And that you can submit into any number of performance analysis tools, including
WebPageTest which I am going to use for that demo, but whatever fits your needs, whatever
your organization prefers to use, you can just use it on the side, run the regular tests,
and they submit performance results and they become automatically available to everyone.
Now let's show how that works. Let's switch over to the laptop. I'm going to start the ChromeDriver and I'm going to run the
test, which launches Chrome, loads Google News, does a search for GTAC 2013, and then
switches over to web search results for that same term.
And then at the end, it queries the performance results and submits them to WebPageTest.
And let's see what that looks like. So here you can see a network waterfall chart
of what was going on while the test was running. And you can open a Chrome developer tools timeline
which gives you a lot of info of what was happening.
So imagine you were debugging a performance regression in these tests. That's immediately
very powerful. Now, the neat thing about the new ChromeDriver
is it also works on Android. And I'll show you a few lines of code to run the ChromeDriver
on Android but the test is otherwise exactly the same. No change.
So let's switch over to the mobile projector, see what the device is doing while I do that.
It's launching Chrome. Loading news. Doing search for GTAC 2013.
Switching over to web search results, and let's switch back to the laptop.
It found 2000 DevTools events, submitted to WebPageTest, and this is what was happening
in the mobile browser while the test ran. Again, full timeline. A lot of detail on
what the mobile browser was doing while the test was running.
[ Applause ] >>Michael Klepikov: And again, huge kudos
to the Chromium team and the new ChromeDriver team for making that happen.
Now, this idea in itself is very powerful in the sense that with all the WebDriver-based
products that were very well presented at this conference, including native app testing,
you could enable performance instrumentation at the level of the framework and reuse existing
functional tests to get performance instrumentation for a fixed small cost without changing the
real tests. And, you know, presumably your developers or, worst case, test organization
produce tests all the time, and any of those tests as your functionality progresses, can
be used as performance tests with near zero incremental effort.
Now, as you want to get more advanced in the testing you may want to run the test for multiple
iterations again at the framework level so your results are more statistically valid.
And I'm going to slide back to that nice JavaScript API that exists in Chrome that lets you inject
custom markers into the DevTools timeline. So that's a great aid in debugging.
And there is no end to perfection. But the whole idea is integrate it into the
continuous build, make it part of your regular development process. Catch performance regressions
before they ever hit production or even on presubmit before they even hit the source
tree. Thank you.
[ Applause ] >>Tony Voellm: Yes, thank you. Thank you,
Michael. I'm clearly a perf nut if I call myself Perf Guy. I can spend hours with your
data, and those were very cool graphs. So we have a couple of questions up here on moderator.
If someone asks a live question, someone will earn themselves baby John-Oh up here, as my
kids call it. Otherwise, I'll try to throw it into the screen
and whoever asked this question can have it. I don't think that's going to work.
It says is this available -- is this only available in ChromeDriver? If yes, are any
plans for IE driver server? >>Michael Klepikov: Right. So the short answer
is send patches to Selenium. But the longer answer is there is already performance instrumentation
for IE. For example, as part of WebPageTest, IE test agents that could be integrated into
IE driver server and made available as a log type through WebDriver.
It's just a matter of someone sitting down and implementing it. It was easy to implement
for Chrome, so I sat down and did it. It's certainly possible to do for IE driver server
and it would be awesome if someone actually does it. All the pieces are there. It's just
a matter of sitting down and doing it. >>Tony Voellm: It sounds like a good opportunity
for someone. How about the live question here. >>> Alistair Scott from ThoughtWorks,
and the question was how is this different to the in-built performance measurement mechanisms
in most browsers like Firefox? It's got its own performance.
>>Michael Klepikov: Right. So how it's different from built-in performance? You mean like if
you open Chrome developer tools and then look interactively at what's going on? Yeah,
so that's what I was saying, like why don't we just go through all our pages, optimize
the heck out of them and be done with it. The whole idea is automation. You are using
the built-in browser's instrumentation. I am using Chrome's DevTools. I'm just integrating
it into the test tool chain. I'm showing you how you can take any existing custom test
tool chain that runs WebDriver test and enable it very easily in that tool chain so that
you automatically collect that information for all your tests that run, or for a subset
or what have you, right, and then you can go back and look at what was going on when
that test ran in the past. And you can automate post processing and automatically
catch performance regression, send alerts and so forth. But underneath, it's absolutely
taking advantage of whatever the underlying browser is.
>>Tony Voellm: Great. Thank you, Michael. And here you go.
All right, great. Thank you. >>Michael Klepikov: Thank you.
[ Applause ] >>Tony Voellm: Okay. Great. Thank you.
So -- [ Applause ]