>>Tony Voellm: We're going to go ahead and enter into our series of lightning talks.
Each one of these is going to be 15 minutes, including Q&A.
And so we're going to keep these pretty fast-paced.
First one up here is David Röthlisberger.
And I'm going to start a rumor that, like, he's related to someone on the Pittsburgh
Steelers, so he says he can get everybody free tickets.
So with that, he is from YouTube, and he is going to talk about automated set-top box
testing, which will be fascinating.
>>David Röthlisberger: Thank you.
So it's YouView, not YouTube.
>>Tony Voellm: Oh, sorry.
YouView.
>>David Röthlisberger: If -- So if we could have my laptop up on the screens, please.
Okay.
So at YouView in the U.K., we make set-top boxes.
And to test our set-top box, we have scripts running on a PC, sending infrared signals,
pretending to be a human user pressing buttons on a remote control.
And we check the video output from the set-top
box to ensure that it's done the right thing.
Of course, we have many other types of tests as well, but this is the most interesting
one to talk at a conference like this.
So today, what I'm going to show you is how you might go about implementing such a system.
The command line we're looking at here runs gst-launch, which is provided by GStreamer.
GStreamer is a set of libraries and utilities for handling audiovisual media.
If you use Linux on your desktop, your media player is likely to be Totem, which is built
on GStreamer, just to give you one example.
So let's run this.
Gst-launch takes its command line argument a GStreamer pipeline.
So these exclamation marks here, GStream is equivalent to the pipes.
So this pipeline has a video test source element, generating a video stream with this
test pattern.
And this output is piped to this color space converter element, which converts each frame
to a format understood by the next element in the pipeline, which is X image sync.
A sync is an element that consumes video.
And this particular one renders it to an X11 display.
GStreamer ships with a ton of different elements, such as this one.
V4l stands for video for Linux.
It's an API for drivers.
And v4 all true source supports any device with video for Linux drivers, such as my laptop
webcam here.
Or it could also be an external video capture device.
We can customize the behavior of individual elements by setting their properties.
So if I go back to video test source, I can set its pattern property here to something
different, and we can see it's generating a different video pattern.
Now, here's where it gets interesting.
I'm going to add this template match element.
And template match is a thin wrapper around an open source image processing library called
open CV.
The CV stands for computer vision.
Open CV is originally by Intel.
And template match takes a path to an image.
And I'll show you what this image looks like.
It's this white circle on the black background.
And when I run this, we can see that template match is drawing a red border wherever it
finds a match in the video stream.
Now, the CPU overhead of the image processing is introducing a little bit of jitter into
the video stream, which causes some time stamping problems for the sync.
So just tell the sync to ignore timing information in the stream itself and just display frames
as soon as they arrive and it's a little bit smoother.
Right.
Each element can pass messages to the GStreamer bus.
So I'll ask gst-launch to print those messages.
And here we can see that template match has posted a message for every frame that it's
processed, showing us whether or not it found a match, and the X and Y coordinates of that
match.
So I hope that you can see how all these pieces fit together and how easy it is to develop
a video capture image matching system of this type.
All these components are open source, so we can really take control of our test infrastructure,
we're no longer beholden to proprietary systems.
Now, I consult to a company in the U.K. called YouView.
YouView is a joint venture by the BBC and other major U.K. broadcasters and ISPs.
And YouView makes a set-top box.
To test the set-top box, we have developed STB tester, and we've open sourced it.
So it's built on these components that I've just demoed.
So it supports any video capture hardware with video (indiscernible) drivers, or, indeed,
any GStreamer source element.
And it uses GStream as Python bindings.
It allows you to write your test scripts in Python, and they look like this.
Set-top box tester dot press sends an infrared signal.
Wait for match.
Searches for the specified image, and raises an exception if it doesn't find it.
So this script never gets to our network and Internet settings menu, and we've encapsulated
that into a separate Python function so it can be reused across scripts.
Then we press down until we find this image.
And I'll just ask my editor to display those images in line.
So I'll remind you, this is just Python, and those were just string literals.
It's my editor that knows how to display the images on line.
It's EMACS, before anyone asks.
[ Laughter ] >>David Röthlisberger: So press down
until the submenu is selected.
We press okay.
Then we press right until this automatic option is selected.
And then we press down and we assert that the next button has been highlighted.
We press okay, and within 30 seconds, we expect to find this image.
So as you can see, it's very simple.
It's very readable.
It's entirely procedural.
And it reads much like a manual test script written in English.
So non-test programmers on the test team have no trouble understanding this and relating
it to their own requirements, test coverage, matrices, and so forth.
But at the same time, it is Python, so it's very powerful, and you can do anything you
can imagine, really.
So just show you a couple of quick examples
of other stuff we've done.
This is a video I recorded earlier of an STB tester script that knows how to navigate through
this double carousel of players.
So I've told it to find BBC iPlayer.
And now that it's found it, I've told it to find 4OD.
The, again, red rectangle is showing us where STB-tester has found a match for what
it's looking for.
And now it's found 4OD, I've asked it to find Demand 5.
So it's looking for this long, thin, blue rectangle to identify the current selection.
It looks for an image of the unselected player to find where it needs to go, and it
figures out what buttons it need to press to accomplish that.
And I'll just play that again while I show you the source code implementing that logic.
I won't go through it, but just to show you it's fairly simple.
It's barely two pages of code.
And most of that's doc strings and doc tests.
Although I have removed error handling.
But if you're interested in seeing the full code, I've posted it on STB-tester.com as
an example.
So, essentially, your script would just go find player, pass in an image of the unselected
player to know what it needs to find, and an image of the selected player to know when
it's got there.
Here's another example.
This state machine is YouView setup wizard.
So each image in this diagram, each Node, represents a possible state that the UI can
be in.
And each edge represents an action that a user can do from that given state.
This diagram is generated entirely automatically from some Python code we've written to describe
the setup wizard.
Each image you see is exactly the same image that STB-tester uses to identify that the
UI is in that particular state.
And each edge, each user action, is a Python function that carries out that function.
And you can click on one of them, and it shows you the source code implementing that,
and that is all fully hyperlinked, so you can drill down as deep as you like.
That's very helpful to -- you know, nonprogrammers can still follow the flow of each of these
functions here.
So you can write a test script where you explicitly call each of these actions in the order you
want them, or you can do something smarter.
You know, we do, after all, have this representation of the state machine, so we can randomly generate
random walks through it.
Or you can have a test script that takes a given path and just tries cutting the power
to the set-top box at every possible stage just to see what happens, you know, this kind
of stuff that's incredibly tedious to do manually, especially for each software release.
I'll show you the full state machine there.
All done.
So if you look at this in a certain light, it almost looks like a wire framey kind of
UX design document.
And really what we're aiming for is fully machine checkable specifications.
And this is all generated from plain text files, so, you know, it's version controllable,
you know, all that good stuff that we're used to for our code as developers, we can now
have for our specifications as well.
It's very exciting.
Before I quickly take some questions, I'll just point out, the STB-tester.com has loads
of documentation.
We've got introductory material, some videos to help you get started.
There's a mailing list if you have any questions.
And we try to do the development of it as openly as possible.
So thank you for your time.
[ Applause ] >>Tony Voellm: Great.
Thank you, David.
All right.
We can go ahead and pull up the moderator.
If you have questions, you go to either side here, in terms of asking questions.
That was very fascinating the way you did the sort of image matching and so forth.
I'll start off with a question of my own: How complicated does the image logic have
to be?
Or is it just very pixel-perfect in terms of matching and looking for elements?
>>David Röthlisberger: So as we capture video directly from the set-top box, we're
not going through cameras or anything like that, we have fairly high fidelity images.
But we do have video capture hardware that does H.264 encoding in the hardware, so we
do get some artifacts from that.
So our image processing algorithm does handle that.
And it's got a few knobs that you can tune, either globally or for a specific image if
you'd like.
>>Tony Voellm: Very cool.
He also reinstilled my faith in EMACS as being is cool, the fact that it will embed the images
for me.
Okay.
I'll take a question off the moderator.
It says does having images and test scripts cause flaky tests when styles change?
>>David Röthlisberger: So I wouldn't call that flaky tests, if the style starts
changing.
So yes is the answer to that question.
As you hopefully gathered from some of my examples, most of the actual image matching
we actually pull into library functions, which the test scripts call themselves.
So very few test scripts actually read like the one full of images that I showed you at
the beginning.
So it's just a matter of changing the comment functions and -- or your scripts aren't working,
and you can do things like match any of these images so that your tests continue to work
across different software releases if you need to still be testing, you know, a version
that's in production that's got different images than a development version.
>>Tony Voellm: Clearly, this next question was written by somebody who has done this.
It says, how do you manage the image repository without sucking the soul out of some poor
test engineer?
[ Laughter ] >>David Röthlisberger: Well, again, everyone
seems to be fixating on these images.
To be honest, the main problem we've had is not with the image repository.
It's with flaky video capture hardware.
So I guess the answer to this question would be much like the previous one, where the image
repository's fairly small compared to the number of tests that use it.
And we also have a lot of unit tests of our tests.
We have captured a lot of screen shots, and anytime we want to make a change to the
image processing algorithm, we run all those unit tests against our collection of previously
captured images to make sure that everything continues running properly.
>>Tony Voellm: Great.
Is this a live question?
>>> Yeah.
>>Tony Voellm: Okay.
Sure.
Go ahead.
We have time for just one more.
>>> Okay.
So I was just wondering if you used your framework to measure performance, let's say,
such as latency, if you click on something, how long does it take to go to the other view,
such things like that?
>>David Röthlisberger: Okay.
So when we're doing the image processing on a full sized, you know, 720P or whatever,
full-sized video stream, even with a fairly powerful computer, we can't process -- do
the image processing in real time.
So what we do, generally, is we just drop frames if we can't process them.
But for testing performance, we flick a switch in the test script just for -- you
know, we use Python context manager to have this apply to just a portion of the script,
where we queue up every single frame, and then we can measure things like that, like
smoothness of animations or latencies.
Latency's a bit difficult, because we have to take into account all the various latencies
in the system: The video capture device, the encoder, the infrared emitter, and so
on.
So we're working on it.
We're getting there.
>>Tony Voellm: Great.
And with that, thank you, David.
>>David Röthlisberger: Thank you.
[ Applause ]