[MUSIC PLAYING]
JASON O'GRADY: Hello, and welcome
to Optimize, our free website testing and personalization
product.
I'm Jason, a technical writer here at Google.
PRAVEEN KRISHNAN: I'm Praveen, product manager on Optimize.
JASON O'GRADY: And today, we want
to show you how to analyze Optimize reports.
So we're going to show you where to find your experiment
results, and most importantly, how to analyze and interpret
those results.
So, Praveen, can you tell us a little bit more
about why reports are important?
PRAVEEN KRISHNAN: Absolutely.
Happy to.
Analysis and reporting in Optimize is one of those things
that we are really quite excited about,
and actually very proud about as well.
Optimize offers a robust statistical engine
that makes it really easy and intuitive to make decisions
on experiments.
To begin, in Optimize, we use a statistical technique
known as Bayesian analysis for determining
experiment outcomes.
Now, there's a lot of interesting benefits
with this method of analysis.
But without getting too deep into the math behind it,
the biggest advantage is that it gives you answers
that are intuitive to grasp.
For example, you can answer the question,
what is the probability that one of your variants
is better than all the others?
And if you went with that variant, what
would the improvement be?
Now, this is a basic question users
of any system that does experiments
should be able to answer.
Note that this is not what a p-value
or a statistical significance number
will give you in a straightforward way.
Jason, most people, even scientists,
have trouble understanding what statistical significance
actually does.
And there's a lot of material about this online.
But we're not going to get into all that detail today.
JASON O'GRADY: Sounds like a topic for another day--
PRAVEEN KRISHNAN: Absolutely.
JASON O'GRADY: --especially given the pushback
from the scientific community.
Well, do you want to show us around the reporting
UI a little bit?
PRAVEEN KRISHNAN: Absolutely.
Let's dive right in.
So in the Optimize report, you will actually
see a lot of detail that really help you understand
what your experiment is doing.
So we're going to go through it section by section.
And we'll start with the top section, which is showing you
a recommendation based on the data
that Optimize has seen so far.
So if your experiment is going really well,
and there is some clear winners, or leaders,
among your variants, Optimize will call that out,
at a certain point, and will give you
what the probability is that that variant will do really
well.
In the case where there isn't a clear leader,
Optimize will also give you a recommendation on what
you can do at that point.
In some cases, Optimize will suggest
that you end the experiment.
In other cases, it'll tell you to keep it running for longer.
Now, between those two options, there is a lot of opportunity
for you to make decisions on the experiment yourself.
And Optimize is continuously estimating the probabilities
and the conversion rates so that you now
have the ability to make those decisions on your own.
Now, this is one of the advantages with the Bayesian
analysis.
JASON O'GRADY: So in Optimize, I can both see and take action
on those results at any time.
PRAVEEN KRISHNAN: Exactly.
And so that's what makes it really interesting,
because, as data comes in, we're continuously
evaluating the probabilities and the conversion rates
so you can actually make those decisions quickly.
JASON O'GRADY: Sounds great.
PRAVEEN KRISHNAN: So let's go a little bit deeper and talk
about, what are some of those data points
that you actually have in your hand to make decisions?
So below the recommendation section
is what we call the objective card.
And in the objective card, based on the objective
that you picked, which is at the top corner right over here,
you will see the data related to that objective
by different variants.
So there's two sections here.
The first section is observed data.
Now, observed data is the data that Google Analytics
is collecting from your experiment.
So you have, over here, experiment sessions,
which is telling you how many sessions
are in each specific variant.
And then here, it's telling you how many of those
sessions actually have conversions in them.
And then the last column here is telling you
a straightforward conversion rate for those variants.
Now, this is the source data that Optimize
uses to do its analysis.
Optimize takes a snapshot of this data multiple times a day.
And up here in the screen, you will see when
that last snapshot was taken.
The next section is the analysis section.
Now, this is where a lot of the key value
is that Optimize provides.
So the first column here is the probability to be best.
Now, what this is telling you is,
for any given variant, what is the probability
that that variant is going to be better than every other variant
that you have, including the original?
So what you see here is it's saying,
there's a 67% probability that this variant
is going to be the best.
The second column here, which is the modeled conversion rate--
or, depending on the objective that you selected,
the modeled objective value--
is telling you what Optimize's own model estimates
the conversion rate is going to be
in the long run for that particular variant.
And it's giving you a range as well.
So if you moused over this particular column--
we call this our TIE Fighter graph for you
"Star Wars" fans out there--
you're going to see three more values.
In Optimize, we love to give you as much data as you need.
And what this shows you is there's
a 95% chance that the conversion rate
for that particular variant is going
to be between 50.9 and 54.5.
The second line is going to tell you
that there's a 50% probability that the conversion
rate is going to fall between these two numbers.
And then the third one is Optimize's best estimate
of what the conversion rate for that variant
is going to be in the long run--
in this case, 52.7.
So now you can see how, as data is coming in,
you can quickly look at this and say,
this is what your conversion rate is
expected to be in the long run for each specific variant.
JASON O'GRADY: Nice.
That's a lot of valuable data there,
and it's very glanceable and easy to take action on.
But what about that third column on the right, the modeled
improvement column?
PRAVEEN KRISHNAN: Yes.
So the modeled improvement column
is a really valuable piece of information.
What the modeled improvement column does
is it gives you, quantitatively, the lower and upper bounds
of the improvement that you could expect
to see for a given variant.
So in this case, what it's telling me
is that this specific variant--
the best case, it's going to do 6% better
conversion than the baseline.
Now, this is not an absolute difference.
It's a relative difference.
So it's 6% of the conversion rate of the baseline.
So it's going to be better.
And in the worst case, it's going to be negative 4%
compared to the baseline.
So looking at this, I get a clear indication
of what my upside and downsides are if I
went with a specific variant.
So I can use the modeled improvement,
along with the probability to be best,
to make a really good decision on whether I should deploy
that variant or not.
Now, the last part of this screen
is a graph, which is the modeled conversion
rate over time graph.
And what it shows you is how Optimize has been estimating
the conversion rate over time.
So you saw here, in this table, the modeled conversion rate,
which is a snapshot in time.
And the modeled conversion rate over time graph
is simply showing that particular conversion
rate progressing over time.
So as you can see, there's a lot of information
here all packaged together in a fairly intuitive way for you
to understand and really help you make decisions
on experiments quickly.
JASON O'GRADY: Thanks, Praveen.
That was great.
For more information, check out our other videos.
And learn more at the Optimize Resource Hub.