Hi.
I'm Rick Viscomi, and I make YouTube fast.
In this video, I'm going to share
how we test web performance and a few tips
for doing the same on your own website.
The two test methodologies that we're going to look at
are real user monitoring, or RUM, and synthetic testing.
Real user monitoring, just as it sounds,
gathers performance data from real users
as they browse the site.
For YouTube, this provides us with the ground truth
about how users are experiencing the site.
Synthetic testing, on the other hand, is more like a laboratory
and is commonly used on demand to analyze our particular use
case.
Both methodologies collect performance data,
but given these distinctions, RUM
is especially useful for identifying
how fast the page is, and synthetic for identifying
how to make it faster.
The simplest way to gather and analyze RUM data
is with one of the existing analytics products.
In fact, Google Analytics can do this for you.
At Google, we use a custom solution
called client side instrumentation, or CSI.
RUM works on three basic levels.
First, we instrument our client side code
to provide timing and page state data to the RUM JavaScript
library.
Then, the library packages up the data
and sends it off to the Analytics server for later use.
And finally, the Analytics server
provides developers with a dashboard to view the results.
Out of the box, most RUM tools measure the general page events
published by their navigation timing API.
But for us, one of the most important performance metrics
is how quickly a video plays.
The metrics you collect should be customized
based on what your application does
and what your users are actually waiting for.
It's also important to include any contextual information that
might have an effect on performance.
For example, YouTube users may be
exposed to experimental features,
and we would need to know what effect that
might have had on performance.
By keeping track of these variables,
we're better able to get insights
into the performance of particular demographics.
Also, note that the CSI beacon is sent unobtrusively
after the critical path and video playback.
This helps to ensure that testing does not
interfere with the user experience.
To analyze the performance data, we turn to the CSI dashboard.
Here, we're able to look at the change in performance over time
and compare experiments against controls.
For example, in this chart, we're
comparing the results of an experiment against the baseline
control.
The experiment times are lower or faster
until this last point.
We can use the CSI dashboard to slice data
by various demographics to understand
why this is happening, and when we break the data down
by browser, it becomes clear that browser B
is responsible for the regression.
It's also worth mentioning that, because we gather data
from so many users-- all of whom have
different devices on different networks--
we typically look at the 50th percentile,
or the median value, to give us a representative view
of the performance data.
OK.
That's it for RUM.
Now let's compare it to synthetic testing.
For that, we use WebPagetest, which is a free public tool.
And instead of gathering data from real users,
we configure synthetic tests to a particular demographic.
The tests run in a laboratory-like environment
to mitigate the effective network anomalies,
and this is achieved through repetition.
So tests that are run many times are considered to be reliable.
And, finally, when the results are in,
we're able to analyze our application
specific metrics along with the test network log.
WebPagetest gives you full control
over the type of use case for which you want to test.
For example, we may be interested in the Watch page
performance in Chrome over a DSL connection.
And being a free public tool, WebPagetests
are run on ordinary computers connected to the internet.
To test YouTube, however, we need more privileged access
to beta versions of the site.
So for this reason, we run our own private instances
of WebPagetest inside of Google.
The results are shown on a waterfall diagram, which
is a visualization of the network activity.
In this view, we're able to precisely see the activity
on the critical path to video playback.
And, finally, WebPagetest also annotates the waterfall
with our application specific metrics.
This gives us the necessary context
to better understand what exactly
is contributing to slow values.
So that's it for our look at synthetic testing.
Remember that RUM is only as useful as the metrics
you provide.
CSI is just one of many tools that does it,
but your solution should be just as unobtrusive and lightweight.
Synthetic tests should be configured
to represent common use cases, like a particular browser,
connection, or page demographic.
Also, by locating synthetic test machines inside
of your own network, you can validate performance
on a staging server before launching.
To learn more about the NavigationTiming API,
check out the Mozilla Developer Network article at this URL.
Or for more information on using WebPagetest,
check out the link over here, and also
look out for my book coming later this year.
Thanks for watching.