>>Yvette Nameth: And up next, we have Brian Gogan, talking about Chromecast testing.
>>Brian Gogan: Hi, guys. So after all that talk of delicious dumplings
and lunch, I'm sure there's some post-food coma setting in. I'll try to keep it fast.
My name is Brian Gogan, I'm a software engineer, tools and infrastructure, at Google. And I've
been lucky enough to be working on Chromecast for the last couple of years.
So I'm going to do a little bit of context-setting first, just, you know, to kind of set the
stage. For Chromecast, you know, it's part of, I
guess, a wider movement that's going on in the industry about Internet of things. So
everybody's kind of aware of this. We have lots more devices being released onto the
market, being connected to the Internet, and interoperating with each other.
So this is a big test challenge for us. The numbers of devices are pretty big. Right now,
we're somewhere around about 5 billion devices in the market. And by 2020, that's expected
to grow pretty hugely. So why do we care? Well, software really powers
all of these devices. You know, typically, there's pretty extensive software stacks running
on these devices. And as we all know, software needs testing. So lots of opportunity for
us. So Chromecast, just want to kind of give a
background on Chromecast. I think most people, hopefully, know what Chromecast is. But for
those of you that don't, it's a dongle that plugs into the back of your TV, and it allows
you to stream content from the Internet onto your TV. And that content can be movies, can
be TV shows, it can be videos. It can also be music content or photos.
So it's kind of a -- it's a challenging device in several facets. In terms of Internet of
things and how it all fits together, it's kind of a good example of some of the challenges
that we face. There are, you know, multiple devices talking to each other. In order to
get content onto your TV, you use the phone. Your phone is your remote control. Therefore,
there is communication between the phone and the Chromecast device itself. So you've got
a lot of asynchronous communication going on, a lot of, you know, challenging things
to test. The device itself has been pretty successful.
We've sold about 20 million of them. We've got over 1,000 apps in the market, which means
it's, you know, more and more challenging for us to release new features and make sure
that we maintain quality of the ecosystem. So I'm going to talk a little bit about test
automation for Chromecast, talk about some of the challenges that we, you know, have
faced, and kind of map out some of the future work that we have, not just for Chromecast,
but for Internet of things in general. I'm going to focus mostly on device testing,
because that's kind of probably the most fun stuff that we do. But, you know, one question
that always pops up is, do we use emulators. And the answer is, absolutely. You know, virtualized
devices running in the cloud, it's a scalable solution, allows us to do end-to-end testing
from, for example, Android phones to Chromecast devices. We put some plumbing together to
make sure that they can communicate with each other, and then that gives us a very nice,
clean, hermetic test environment. So you get a lot of benefits there, like flakiness elimination.
And it's scalable in a couple of senses. One, it's scalable that you can just run, you know,
as many -- as many servers as you have in the cloud, you can run tests. And then also,
it enables internal partners that we have at Google, like YouTube, Play, Photos, to
use that infrastructure in their end-to-end testing.
Then the question comes up, why would we test on real devices at all. So there are classes
of tests where you just have to test on real devices. For example, performance benchmarking,
interoperability testing, you know, we have stacks of AV and DRM on the device that we
can only test on the actual hardware itself. And there's certain types of bugs, as well,
and issues that will only be found on the device. Largely, these are kind of timing-related
issues. So the challenge, though, is getting a very
good, solid signal-to-noise ratio from our test results. We have a lot of hardware variability,
a lot of network variability. And, you know, the end result of that is that we have a problem
of device management that we need to solve. So I'll talk a little bit about some of those
challenges. First of all, we -- our device lab for Chromecast,
just kind of put some numbers and context behind it, 450 devices we have under test.
That kind of spans the range of different cast-enabled devices that are in the market,
including Android TV, cast for audio and various different generations of Chromecast. Lots
of builds coming out per day, millions of test runs, all that good stuff. Lots of Gangnam
Style playbacks. We contributed pretty heavily to that.
So managing all these devices, what we did for this was to build a tool, our device manager.
Now, there are quite a few different tools at Google kind of working towards solving
this. We found we had to build one ourselves for a couple of reasons. One, we have some
pretty custom setups, of which the other solutions in the market or at Google weren't supporting.
The other one is that the grouping of devices, we have fairly specific groupings of devices
that are necessary for specific types of tests. And we needed to be able to support that.
So that's just -- you can kind of see there the UI of what the device manager looks like.
Kind of gives you an idea. So what this device manager does is, when
a test starts, this manager acts as a broker. It allocates a device to a test, and then
tests will run on that device for a specific period of time.
In terms of stuff that goes wrong, lots of stuff goes wrong. Kind of basic, obvious ones
are, you know, we don't have enough devices to run tests, so standard capacity issue.
You know, we kind of solve that by throwing more hardware at the problem.
There's also the variation of test results across devices. We've found that there were,
you know -- even between different revs of the hardware, we would find varying results.
So that was really tricky to kind of debug in some cases. You know, we kind of built
a lot of traceability of devices into the -- into the system so that we could really
nail down exactly where differences were and isolate issues.
Stuff that goes really wrong, unreliable devices, mainly. We've -- a lot of hardware in the
loop. And, you know, hardware likes to fry itself or overheat. It likes to disconnect
from the network for arbitrary reasons. We also get stuff like kernel bugs, which mean
it's really hard to recover the devices. We've also had cases where the MAC addresses of
the devices just changed for arbitrary reasons, probably breaking some IEEE rules. Just happens.
So countermeasures there, lots of monitoring, lots of logging. Redundancy as well. We threw
a lot of hardware at the problem, so if a device does go down, we have redundant backups.
And also, before we allocate a device for a test, we'll do some kind of basic sanity
checks to validate that, you know, this is a reasonable device to use.
And then kind of one of the worst-case scenarios that we sometimes run into is the kind of
bad build effect where a device gets a bad build and it breaks the device. And, you know,
this becomes a pretty virulent cascading effect and pretty quickly takes out the entire lab.
Really painful, horrible stuff. You've got to go in there and recover all these devices
manually. Not fun, especially when you've got hundreds of devices, and trying to trace
down the ones that are actually broken. So we've kind of established a quarantine
system. Before we push builds onto a wider population of devices, we will push them onto
a smaller subset of devices which are kind of our taste testers of sort.
So I think just kind of to summarize in terms of the tooling requirements for just general
device management and Internet of things testing. So device allocation and brokering mechanisms,
that's kind of a given. We also need a system to be somewhat device type agnostic, as we're
working with various devices, interoperating with each other. And plenty of monitoring.
Monitoring is super important. So the other kind of major challenge that
we face with Chromecast is Wi-Fi. So I'll talk a little bit about some of the challenges
here. So the requirements for a good Wi-Fi system
or test system are kind of somewhat, you know, counter to each other. Obviously you want
reliable test signal. You want to isolate away any Wi-Fi interference so that the test
is somewhat robust. At the same time, you want to have some level of controlled interference
so that when you do run a test, it's actually exercising some of the logic that's on the
device for robustness. At the same time, you want to do that in a
realistic end user environment, which is somewhat challenging to generate in a lab.
And at the same time, you want to test across a whole bunch of APs. There's hundreds of
APs out there that people are using. And then the kind of holy grail is also to
be able to reproduce Wi-Fi issues that happen in the field. That's kind of an overall industry
challenge that we have. So like I say, some of those may be mutually
exclusive. So our strategy, really, for this, for attacking this problem, has been to implement
various different test beds. We have test beds for these various different environments.
And I'll just kind of quickly walk through these.
First one is, these are RF isolation boxes that we use. In our test lab, one of the challenges
that we have is the Wi-Fi interference from, you know, various other devices that are around.
So even if we -- you know, if we just have a test running on a device in the lab, it's
probably going to be super flaky. So we put the devices into these boxes. These provide
about 80DBs of attenuation and give us pretty clean test signal.
One of the cool things about working at Google is we have lots of other teams working on
similar problems. So the Chrome OS guys have done an amazing job in putting together an
interoperability environment where there's 150-plus of the most popular APs in the market
in a fully automated setup. So we can drop our Chromecast devices into this environment.
The APs are turned on one hot so you can communicate with just one AP at a time. And this is what
we use for setup testing, for example, on the Chromecast to make sure that it interoperates
with these various different APs. This has turned out to be, you know, probably one of
our cornerstones of our entire test strategy for Wi-Fi on Chromecast.
Let's see. Another kind of environment that we have.
So this is less on the automated side, more on the manual test side, is a set of shield
rooms that we built. So these are effectively faraday cages about the size of an office.
So we have people in there testing. We have some amount of automation that we do in there,
mirroring, specifically where we are mirroring from an Android device to a Chromecast. As
far as Wi-Fi's concerned, that's a pretty challenging setup. There's multiple streams
going from device to device. So a shield room is very, very useful there.
One kind of interesting stress test we can do there is, we start off a mirroring session
and then slowly open the door and just watch the mirroring session degrade. It's kind of
fun to watch. Another kind of more recent addition to our
lab is this octobox. So this our effort towards a very controlled environment. This equipment
gives us a couple of different levers. One is an attenuator. So we'll put a router in
one of the chambers and the device owner test in a second chamber, and we can vary the attenuation
in a very controlled matter. It also has a multipath emulator which kind of simulates
the real-world interference or real-world echoing that happens in a room from an AP
to a device. And we -- it also allows us some level of interference generation to inject
that into the box. So this is -- you know, we do some level of
automated testing in this. But it's primarily, you know, an experimental setup which kind
of allows us to kind of solve -- you know, when we get a new Wi-Fi driver, to validate
that the quality is there. And the final one that we kind of set up is
an emulated network environment. So this has been very useful, for example, with mirroring,
where we want to have a very controlled set of environments where we inject a lot of latency
or packet loss. And it gives us, you know, good metrics on the quality of the mirroring
session. So the device lab. I have some photos to show.
When I was talking a little bit earlier about, you know, if the -- we get a bad build on
a bunch of devices, it's pretty painful to go through.
So here is the ugly previous situation. So you can imagine trying to find device number
22 in that mess. Not a lot of fun. We kind of had this ugly beast of a thing which we
anthropomorphized into an animal we call Casty, nasty Casty.
So we spent a lot of time on designing a better setup. And this is kind of what we have now.
So lots of, you know, neat features here. A couple of -- like, obviously, we've got
nice labeling so you can figure out which device is which. It's all logically laid out.
Cool thing here, you can see -- on the second from top shelf, you can see that we have switches
there. Those switches allow us to view the HDMI output of any of the Chromecasts on the
rack. So each rack holds 72 Chromecast devices, all running continuous integration testing.
Closeup. There you can see some of our gen 1 devices. See you can see how the ATMI connection
is there. So that's routed under the shelf and then up to the top screens via the switches.
Here you can see kind of the back side of it, where we have Ethernet connectivity to
all of the devices. That's one other point. All these devices under test are hard connected
via Ethernet, which, obviously, cleans up -- makes for more reliable test results.
Here you can see the serial connections coming from the bottom of the devices. We have a
class of tests as well that require HDMI syncs to be connected to. So it's kind of fun to
watch tests running on all these different screens. Here you can see our next-generation
Chromecast devices connected up. So I have some quick photos of the buildout
of the lab rack. We kind of came up with this concept, sourced these components. And, you
know, kind of built out the -- the lab rack as was. So we started off with some cardboard,
went from here all the way to a fully fleshed-out prototype. And then started building up on
a rack. And then ended up with what we have today.
So opportunities. I think, you know, Internet of things, like I say, it's a nascent area,
especially as far as test engineering is concerned. I think there's a lot of us in this room here
who are working on various different approaches to this. But I think it's fair to say that
there's no broadly adopted frameworks in the market at the moment. So lots of opportunities
for testing in the cloud, for virtualization, interoperability labs. We have emerging standards
coming out with Thread, Weave, and various different standards like that. So there are
-- there is convergence happening, but kind of, like I say to my team, you know, we have
probably decades of work ahead of us in order to solve these problems.
So that's it. [ Applause ]
>>Yvette Nameth: Thank you. And sadly, we don't have time for questions
after this one. >>Brian Gogan: I used it all up.
>>Yvette Nameth: Yeah, he decided to use it all up showing us all some pictures.