>>Tony Voellm: So next up here, we have Patrik Hoglund, who is an avid drummer, which probably
comes in quite handy when he's banging his head against this automated video quality
measurement system.
So with that, Patrik.
>>Patrik Hoglund: Thank you.
All right.
So I'm going to talk about automated video quality tests.
That is a problem we run into with the WebRTC project.
First, how many have heard about WebRTC before?
There's a couple.
Okay.
Wow.
WebRTC is a Web standard that is being developed right now for video and (indiscernible) built
into the browser.
So we are implementing this right now in Chrome, and also Firefox and Opera are implementing
this Web standard here.
So WebRTC gives you the possibility to, in a Web page, with JavaScript, acquire a user's
webcam and microphone, with their consent, of course, and send it over the Net directly
between the participants.
So it uses peer-to-peer technology.
And all that stuff is built into the standard, so it's real easy to write a video chat application
that works, regardless of (indiscernible), firewalls, and other things that can be in
the way.
Okay?
So the Chrome implementation of this is done in Stockholm and Mountain View and in Kirkland.
And I work in the Stockholm office with this project.
So we wanted to test this.
But to explain what we need to test, I'm going to explain briefly how WebRTC code works.
All right.
So this diagram here, as you can see, is a very simplified description.
So what happens here is that we can imagine we have two browsers on two different machines
somewhere.
And then it is up to the Web page to implement that application part there, as you can see.
And implement a installing.
is to figure out first who you would like to call with WebRTC, for instance.
And that can be implemented, like, a contact list or random or whatever.
And also the signal resolution exchanges the information that is needed to set up the code,
for instance, what codecs they support.
We're going to gloss over that part completely and instead focus on number two you can see
down there, which is media, when media starts flowing.
So given that we set up a call, how do we test it?
So one of the goals we set out to do here was to set up a call and measure the video
degradation.
Because, obviously, when we send video over the Internet, we're going to have to compress
it.
Right?
And also in WebRTC, there's a ton of built-in algorithms for bandwidth estimation, and like
adapting the code width parameters to how much bandwidth you have or how lossy your
connection is.
Or, as I like to say it as a test engineer, a lot of things that can go wrong, a lot of
things that can break.
So we wanted to measure the video quality that comes out at the other end of the call.
And we wanted those measurements to correspond to user perception as much as possible.
What do I mean by that?
Well, basically, if a user would think it looks bad, it should get a low score.
And if it's artifacts that a user doesn't really notice that much, it shouldn't affect
the score too much.
We wanted this thing to run continuously, we wanted it to be a reusable tool chain with
small tools that have well-defined tasks as much as possible, so it makes it more usable.
And we wanted the implementation work to be a sort of self-contained unit so we can hand
it off to an intern.
And that is precisely what we did.
We handed it off to our intern, Veli Spasova, from Bulgaria, and she implemented it, and
it worked great.
That's why I'm here to share with you an overview of what goes into making a tool like this.
So we're going to obviously gloss over a lot of details, but hopefully give you an idea
of what we need to do.
Okay.
So when we started this thing, we already had a test which could launch a browser, open
two tabs, launch a small C++ binary, which is a signaling solution, and have the two
tabs set up a WebRTC call.
And so the output video gets fed into a regular video tag, HTML video tag.
We had some fake webcam drivers on all major platforms so that we can basically feed a
known input video into the webcam and have it look like the webcam to WebRTC.
We also had implementations of the peak signal-to-noise ratio and structural similarity algorithms.
And I'm not going to go into detail what those are.
But supposedly, together, they give a pretty good picture of the video quality.
All right.
So the first problem to solve was how do we record what goes into the -- what is coming
out on the video tag on the other side?
First we considered screen scraping solutions, where you scrape the pixels on the scream,
but, unfortunately, we were able to avoid that, because you would have to make a platform-specific
implementation of that.
And, you know....
Instead, you can take a camera stack and tell the camera stack, capture the pictures that
are displaying in that video tag right now.
And then hooked that into a JavaScript timer that tries to capture at 30 fps or as fast
as it can, really.
Then we ran into some other problems, like we can't write the frames to disk from the
JavaScript environment.
So our intern wrote a Python server that we talked to using Web sockets, and then the
Python server writes on the disk.
That is a bit involved there.
So after running this, you -- you end up, basically, with a couple of hundred of image
files on disk.
And then we run a small tool to consolidate those into one YUV video that is raw video.
All right.
But there were still some problems, like we had no idea where the video starts.
Like, when we run the test, we don't know where we are starting an input video, because
the driver is just going to loop it over and over; right?
And since the algorithms I talked about earlier compare frame to frame and give you a score,
you need to compare the right frame in the output video with the right frame in the input
video, otherwise, you're going to get the wrong score.
Even if you solve that synchronization problem with a camera driver, what if we dropped the
first two frames?
Then we would compare frame in output video and frame minus 2 in input video, and the
score would be very much punished for something you wouldn't notice, so that's not good.
If the JavaScript does not capture all of the frames, because I sure don't know how
to predict how fast JavaScript will run.
So the trick is to encode bar codes into the input video.
Right?
So you can see that bar code there is going to move with the video.
You can see three examples of the video here, because this is what our test looks like,
what's coming in, what's coming out from the other side, and the cameras, what it's picking
up, capturing the stuff there.
And so how will that help us?
Well, it will make it able for us to encode a unique ID in each frame of the input video,
and then later find out which frame we actually captured on the other side.
So the encoder uses -- I'm going to go quickly through this.
The encoder uses a library called Zebra Crossing.
It can generate bar codes for us.
So we generate a bunch of bar codes in rising numbers, like 1, 2, whatever.
Then we use F and MPEG to convert those to YUV.
And then we combine the YUVs to bar code movies.
So that's going to be a very narrow movie with a moving bar code in it.
And then we can stitch that together with an input file of our choosing, and it creates
a video such as the one we saw over here.
Yeah?
So our setup ends up being feed that video into the fake webcam, have WebRTC acquire
that webcam, then we also test that, you know, our device acquisition code and all that stuff
works for low (indiscernible) systems.
We capture what we get on the receiving side with the video camera thing I talked about,
decode the bar codes, and then we analyze it.
So the analysis step is the next slide that I'm going to talk about.
First the bar code decoder.
So what does the bar code decoder do?
So it takes the YUV movie we produced in the capturing step, splits that into a bunch of
PNG files, because that is the Zebra Crossing (indiscernible), again, using SSIM MPEG.
Run it through Zebra Crossing, and then we get a bunch of numbers.
So our tools is going to produce a file we call stats.txt.
And that is a mapping for each frame in the output video what is the corresponding frame
in the input video.
And that is used in the final analysis step here.
So then we have this thing that takes the original video, the capture video, and that
mapping, and then it is going to run PSNR and SSIM on each frame using the mapping stats.txt.
And what we end up with is something like this.
Okay?
So it's a bit hard to see here in the graph, but what you can see in the graph is that,
first those lines that you see there, that is the distribution of scores.
So each frame is scored individually.
And then a big thick line in the middle, that is the average for that particular run.
So what you can also see, or maybe not, is that we are around 36-decibel peak signal-to-noise
ratio, which I would say is reasonable for Internet transmitted video.
That big dip on the right there, by the way, is our regression, looks like.
And when it goes back up again, that is where the regression was rolled back.
And for the structural similarity, similarly, we get consistent now I think about 0.96 out
of one in this quarter.
When we put the whole thing together, we end up with that.
So as you can see, it's a fair amount of stuff in there.
These are the parts I mentioned earlier, like you have your input video that you just need
to prepare and encode once.
And then when you run your test continuously, you feed the same video into the webcam, and
you run the test, as in our case.
And then you -- you do the necessary conversions so that you get YUV file, and you decode the
bar codes, and you end up with something we can analyze and produce a score from that.
So that concludes my talk.
Thank you for your time.
[ Applause ] >>Tony Voellm: Thank you, Patrik.
So if you have a live question, you can go to the mikes.
I see somebody up there.
And then we can also pull from the Dory.
Why don't we go live first.
Please.
>>> Hi.
A question about the output frames.
Does it ever happen that you lose output frames because you couldn't decode the bar code,
in other words, that the video quality is so low that you can't decode it?
>>Patrik Hoglund: Yeah.
So it hasn't happened very much for us.
When it has happened, it has been because WebRTC has been completely broken and is rendering
black or garbage or whatever.
So if the image gets so distorted that not even the bar code library can decode the bar
codes, then the image is probably completely broken.
So that is the conclusion you can draw from that, really.
>>> Thanks.
>>Tony Voellm: All right.
Let's pull one from the moderator here.
Do you test AV sync issues?
If yes, how do you do it?
>>Patrik Hoglund: No, we don't.
But -- so I guess that answers that question.
No.
But, I mean, that is obviously maybe something you can extend the test to.
And, by the way, that is a tape for all of you, that all of this code we have written
for this is open source and checked into the Chrome repository.
So if you want to use it, knock yourself out.
So you want to do something like that, you can probably do it.
>>Tony Voellm: Let's take another one here.
Are you able to measure the video quality while streaming at various network conditions?
>>Patrik Hoglund: No.
But we would love to be able to do that.
I mean, presumably, when -- when you have a lot of packet loss or a burst of packet
loss or a lot of delay or whatever, you would expect the bandwidth estimation algorithm
to adjust accordingly.
And then you should see probably that, okay, 2% packet loss, but then we have maybe 34
PSNR or whatever, and then you can keep your stuff working for all customers.
And I think this is something people don't do enough of, you know, verifying that, okay,
it looks great on Google's office network, but it will not work on an ADSL line in India
for instance.
>>Tony Voellm: We have time for one live question.
I think you got up here?
Or did you?
Toss up.
All right.
What's your question?
>>> I'm Narindra (phonetic).
I'm from Amazon.
The question I had was, why bar codes?
You might as well have written numbers in there.
>>Patrik Hoglund: Absolutely.
We had a library for it, like I said, Zebra Crossing.
I think bar codes are relatively resistant to normal artifacts when you encode and decode
a video.
So, I mean, we don't want that to be flaky.
And it has turned out to work very well.
>>Tony Voellm: If they had used numbers and they would have had a Cookie Monster instead
of zebra, so -- Okay.
All right.
Thank you, Ken.
Or Patrik.
Thank you, Patrik.
[ Applause ] >>Patrik Hoglund: Thank you.