SPEAKER: Hey there, app developers.
Have you been running an A/B test with our fancy new A/B
testing feature and are dying to find out what all those numbers
and charts actually mean?
Well, you've come to the right place.
We're going to take a closer look
at how to analyze the results of your A/B test
in the Firebase Console.
Are you excited?
I am-- you should be too.
Come on.
[MUSIC PLAYING]
So if you've been following along with this video series,
you've probably got your app ready to perform A/B tests
by adding a Remote Config and Analytics to your app,
then you've run your first A/B test
by creating an experiment in the Firebase Console.
And now you're probably wondering,
OK, so, how can I get my results?
So you'll probably start seeing results trickle in
after about half a day.
But here's the thing.
While these results will probably be super interesting
to see-- and I totally recommend you check them out--
don't go making any changes to your app
until the experiment is complete and A/B testing has told you
that it's found a winner.
See, whenever you make a user-visible change,
your users are going to notice it and they're going to act
on it--
not because it's necessarily better,
but because it's different and it's grabbed their attention.
But that doesn't mean it's going to be
the right move in the long run.
So looking at my experiment, if I change my background color
from orange to green, it's totally possible
that, in the long run, this doesn't work out.
Maybe we find that people just prefer that orange color
better.
But it's also quite likely that in an experiment where
I tried changing the background color to green,
people will notice the new panel color and click on that Sign
In button, again, not because it's a better change,
but because it's different.
So to avoid giving you misleading results like this,
Firebase has generally determined
that it needs to wait about two weeks for this short-term bump
to level off.
It also helps make sure we account
for things like variances within the week,
as far as how people interact with your app.
And that's generally why, even if your experiment results seem
to dramatically favor one variant over the others,
A/B testing might tell you it's not
ready to declare a leader just yet.
You really should wait until A/B testing has told you
it's found a clear winner in your experiment
before you start making changes.
Now, luckily, it's been at least two weeks since I first
started my A/B test, so I think at this point
it's safe to check in and see how things are going.
So I'll jump into the Firebase Console, select my app,
and go to the Remote Config panel.
And then I'll select the A/B Testing tab.
Now I can select my experiment from the Running Experiment
section, and, well, I basically have zero results here.
And that makes sense because I was
testing this on a sample app that's
got a monthly active user count of 1.
But even if I were to try this with a dozen test devices,
that still wouldn't be enough for a meaningful test.
See, it's not enough for Firebase A/B testing
to tell you, hey, this group got better average results
than this other group--
it's got to tell you, yes, this group got better results
than this other group, and I'm also
reasonably sure that these differences are actually
due to the changes you made in your app
and not just random chance.
Now it does this through a lot of fancy math called
Bayesian statistics that, maybe, I
think I understood at one point back in college
and have totally forgotten about it by now.
But trust me when I say it's very fancy.
And one big factor driving these fancy statistics
is making sure that it's had the opportunity
to measure these changes in front
of a large group of people.
The larger the group of people, the more random chance
gets factored out of the equation.
So if you're going to try and run an experiment with a few
dozen QA folks-- or, in my case, a handful of test devices--
that's not really going to be enough to get
you meaningful results.
You're going to need to try this with an actual app
in production with an actual audience.
And I don't want to shock you, but it turns out
that my app-- it isn't real.
I know, surprising, right?
I'm going to need to turn to an app that has actual users
out there in the real world.
Incidentally, if you are testing this
with a real app in the real world
and you're still not seeing any results,
there is a good chance you forgot to update your Remote
Config libraries in your Podfile or your [? Gradle ?] file,
like I was telling you to do in the last two videos.
Remember, you've got to do that so that Remote Config knows
enough to tell Analytics about the experiments that it's in.
Anyway, I'm lucky enough to have access to a real production app
where we were able to run some experiments.
This is a game called Bingo Blast that gets a few thousand
active users per month, which should be enough
for them to run a real A/B test and get
some meaningful results.
So, in fact, the Bingo Blast folks
recently tried an experiment where
they wanted to find out if they disabled ads in their game,
would that improve their retention
and keep people coming back into their app?
Well, that seems like a worthwhile experiment
to try, so let's take a look at the results in the Firebase
Console and find out what some actual real-world results look
like.
So I'm now looking at the Remote Config panel for Bingo Blast.
And over here in the Experiment section,
we've got this experiment called Ads Impact on Churn.
This is the one we're interested in checking out,
so let's take a look at the results.
So right up here, this is the most important line
of the entire experiment.
It basically gives you the summary
of your experiment in plain English.
And this is telling me that, as far as it can tell,
stopping ads didn't seem to affect retention one
way or another.
And it's run this experiment enough
that it's pretty confident it won't
find a meaningful difference in the future.
If it were still running and not yet ready to present results,
you'd probably see the word yet in there somewhere--
like we haven't found a clear leader yet.
But, in our case, it's saying there's
no clear leader in achieving our goal,
which, one, is actually kind of normal.
A lot of experiments you run will probably
end up without A/B testing being able to declare
a definitive winner.
But, two, is actually still kind of interesting
when you think about it--
it lets me know that my interstitial ads aren't
driving away customers, which is valuable info.
You don't have to have a clear winner for you
to learn something from your experiment.
But let's take a closer look at our data
because I bet you there are still
some interesting insights we can get out of this experiment
that we run.
So, first, if I click on this Details dropdown,
I can see more details about the experiment.
This really just gives us a summary
of what our variants consisted of
and what changes we made for each one.
And this basically covers everything we talked about
in the previous video.
And that's great and all, but I'm
more interested in our actual results.
So let's take a look at the next section.
Now this Improvement Overview section
gives us a summary on how each of our experiment variants
did compared to our control group,
based on whatever factors we chose
to measure in our experiment.
Now this first column will always
be the primary goal of our experiment,
and, in Bingo Blast's case, it was
their app's one-day retention.
So right here, we can see the range of improvement
this variation has over our control group.
Now you might ask yourself, why is
this a range and not like a single number telling us
how well this variant did?
And that's because it turns out that predicting the future,
it's a little tricky.
I mean, just because some variant saw a 5% improvement
in retention on average during the experiment, that
doesn't mean that's what you're going to get in the future--
that depends a lot on the sample size,
the consistency of the results that
were observed throughout the experiment,
and a whole bunch of other factors
that go into that whole fancy mathematics thing I
was talking about earlier.
So, instead, A/B testing generally
prefers to give us a range of values
that it's fairly confident you will see in the future
based on what it's calculated so far.
Sometimes these ranges can be very wide,
other times they can be quite narrow.
It's one thing to tell you, hey, I
think these results will get you between a 4% and 6%
improvement, and I think these results will get you
between a 35% and negative 25% improvement.
Sure they both average out to a 5% improvement,
but only one of these seems certain enough
that I'd be OK changing my app based on those results.
So going back to the results in our Bingo Blast app,
what A/B testing is telling me is,
I'm reasonably confident that, based on the results
I've seen so far, this variant will give you
somewhere between a 7% decrease and a 13% increase
in your user retention.
Well, that's a pretty big range, which
is really why it's telling me there's no clear leader here.
In cases where it was more confident,
I would expect this range to narrow considerably.
That's also why our results are in this unimpressive looking
gray.
If we had results where A/B testing was more confident they
were going to be better or worse,
you'd see these values in green or red,
along with some kind of arrow.
Now I can hover over this range and get some more details
like the actual median value, and some other ranges,
but, honestly, I tend not to use these too much.
And then in these columns here, I've
got improvement stats for all the other metrics
I chose to measure.
You can see I'm also measuring User Engagement, Crashes,
Purchase Revenue, and so on.
And it's important to look at these numbers in addition
to the primary goal of your experiment.
As I mentioned in the last video,
if I end up improving the main goal of my experiment
but totally tanking my retention or something,
that's not really a successful experiment.
So you should make sure you view all these values together
to get sort of a more holistic view of all your results.
You might notice as I click on these metrics, by the way,
I can see more details about them in the section below.
So let's take a closer look at the details for our One-day
Retention metric.
Now on the very left, you can see
each of my variants-- starting with my Control group--
along with the number of users that were placed in each one.
And if everything's working as planned,
these should be roughly the same size.
In the next column over, I have the same stats as above,
the range of improvement that A/B testing
is pretty sure my actual results will fall into in the future.
Next to it, the Probability to Beat Baseline
column is basically how likely A/B testing
thinks this variant will perform better
than our control group for this particular measurement.
Now when you first start this experiment
and it knows nothing else about the results,
this will be 50%, because it's basically a coin flip, right?
And now that it's run awhile, it's
saying that my test case has roughly
a 71% chance of being the better option.
And, honestly, that's generally not enough of an indicator
to be something I would consider actionable.
I like to see this value either in the high 90s
or the low single digits before I'm confident enough to really
rely on these results.
Now Probability to Be the Best Variation is essentially
a way of saying, hey, out of all the variants
in your experiment, which of these is likely to do best?
Now, in our case, we have only one variant and the control
group, so these numbers will be exactly the same
as the Probability to Beat Baseline numbers.
But if I had multiple variants, it wouldn't be.
So take a look at this snippet from another experiment.
Here, notice that both of these variants
seem to perform better than our control group,
but this variant here is the most likely to be the best
out of all of them.
Now this Retention Rate column here
is looking at the absolute percentages instead
of the percent improvement numbers.
So, for instance, here it's saying
my control group will have a one-day retention
rate of roughly 27% to 31%, while in my variant,
it's roughly 28% to 32%.
Next to it, this last column is essentially
the cumulative value of the thing we're measuring.
For retention, it's the total number
of users in our experiment who came back the next day.
For a particular event, it would probably
be the number of times an event was triggered.
And for user engagement, it's the total time
spent within the app across all of our users.
I'll be honest, though, I usually
don't look at these last two columns
very much-- they're more just kind of interesting background
info than anything else.
And in some cases, you'll also see
a graph that shows off the same data,
if you happen to be more of a visual person--
because graphs are always pretty.
So these results are kind of interesting,
and it does help us answer the question of,
does removing ads improve retention?
In our case, A/B testing is telling us, nah, not really
in a way that I could measure.
So does this mean there's no difference
at all between the versions?
Not necessarily.
There is a subtle, but important, distinction
between "I can confidently tell you that both versions perform
similarly" and "I think some variant might perform better,
but I'm not confident enough to stake my reputation on it."
And in our case, honestly, it looks to be a little of both.
If there is a change, it's going to be minor.
And because it's minor, it's hard for A/B
testing to really tell me confidently
that one variant will be better.
But let's take a closer look at some
of these other measurements to see if any of those
give us more information.
Over here in Daily User Engagement,
it looks like people are generally
spending about the same amount of time in my app,
whether or not they're seeing ads.
I guess maybe they're spending about 30 seconds more
in my control group, which, honestly, might just
be the amount of time they're watching these interstitials.
Who knows?
Over in our Crash-free Users Report,
it looks like there's pretty much
no difference between the two groups, which
is kind of what you'd expect unless it turns out that AdMob
was crashing, or something.
Also, it looks like we've got a pretty solid app-- nice work
there, engineers.
For Purchase Revenue, it looks like it's nearly impossible
to tell whether or not there's a meaningful difference
between the two groups.
And if you're wondering why this range is so large,
it's probably due to the fact that only a small segment
of my population is a paying user--
so we're not going to have nearly as much data
as we would for other stats.
And that's something you should keep in mind
if the goal you're measuring is something
that doesn't happen very frequently within your app.
You may need to target a larger percentage of your audience
than usual to get the results you need.
So in my case, while it's showing
that our ad-free version might have performed better when
it comes to in-app purchases, that's a really big range
it has there.
And this percentage to be the best variant still
isn't at the point where I'd be confident
making decisions based on that.
Like I said, I'd really like to see this in the high 90s.
It looks like my longer-term retention
isn't that different either between the two variants.
Again, there's nothing here to suggest
that one version is definitively better than the other.
So when you're all done with your experiment,
you have some options up here around what
action to perform next.
You can stop the experiment, which
means you'll no longer be running this A/B test
or collecting any data.
You can also duplicate the experiment,
if you want to run it again or run it again with some changes.
Maybe try some more dramatic changes
and see how that affects your results.
You'll often see an option to increase
the distribution, which will push your experiment out
to a wider audience.
If your current experiment is targeting only 5%
of the population and you feel like you need more data,
you could alter it to target 15% or 20%--
something like this.
Your last option is to Roll Out to 100%.
Now rolling out to 100% doesn't mean
you're putting more people into your experiment--
it's not the same as increasing your distribution.
It basically means you're declaring a winner.
That is, you're going to stop this experiment,
take the settings from one of your variants,
and make those the official new values in Remote Config for all
your users moving forward.
If you have a winning variant in your experiment,
you'll probably see it selected by default in this dropdown
list, but you can go ahead and select any variant you want.
So should the makers of Bingo Blast
remove their interstitial ads from their game?
Well, I'm leaning towards no.
Yeah, there are some indications that
removing these ads has a minor positive impact
on the retention.
And if the Bingo Blast people didn't
have to give up all their ad revenue
to get these improvements, they might
want to make this change anyway even
though A/B testing couldn't definitively
say that it was better.
But the fact of the matter is that A/B testing
isn't confident enough in these results
that I would be OK saying, why, yes, I'm
happy to give up all of my in-app ad revenue
in favor of these changes.
Honestly, I think I'd want to see a corresponding uptick
in my purchase revenue before I really
turned off all those ads.
So if I were them, I'd probably stop the experiment
and not make any permanent changes to the app,
as far as advertising goes.
And, again, I think an important takeaway here
is that sometimes having no clear leader in an experiment
is just as valuable as having a winning variant.
So hopefully this gives you a better idea
of what kind of data you can get from an A/B test in Firebase,
and how you can best interpret these results to make
the smartest decisions within your app.
So, as usual, if you enjoyed this video,
be sure to subscribe to the Firebase YouTube channel
for more like it.
And if you've learned something interesting or surprising
about your own app by conducting an A/B test,
tell us in the comments below.
In the next episode, we'll describe
how you can use A/B testing to experiment
with your notifications too.
Should be pretty useful, so stick around.
[MUSIC PLAYING]