TODD KERPELMAN: So at this point,
maybe you've been running a few A/B tests with Firebase, gotten
the general hang of things, and now you
have a few follow-up questions.
Questions like, can I run multiple experiments at once?
Can I use A/B testing along with BigQuery
to analyze my results further?
And what does this activation event thing mean anyway?
Well, lucky for you, I've got some follow up answers.
Let's see if we can get to some of those
on this episode of "A/B Tests Like A Pro."
[MUSIC PLAYING]
OK, let's jump right into our first question.
Can you run more than one A/B test at once?
And if so, do you have to worry about people
getting put into multiple experiments at the same time?
So the answer is that, yes, you can run more than one A/B test
at the same time.
In fact, I think you can run up to five at once.
And the general idea is that for every A/B test that you're
running, Firebase will basically reroll the random numbers
that it uses to determine which individual app instances get
put into each experiment.
So if you have two different experiments running
on 5% of the population, you don't
need to worry that the exact same 5% of your users
are going to be in both experiments.
That's not going to happen.
On the other hand, you're not going
to get a completely separate group either.
When Firebase grabs 5% of your population
to put into experiment number two,
odds are that 5% of these people will already
be in experiment one.
If my hands were a Venn diagram, it looks like this.
Now, in most cases, these groups will still
be evenly distributed among your variants.
So this shouldn't be too much of a problem,
particularly if these experiments are unrelated.
Like if one experiment is testing
the flow of your sign-up for our newsletter page,
and another one is testing whether your game over screen
improves user engagement, you're probably fine here.
But if you are in a situation where
one variant of one experiment could strongly
influence one variant of another experiment,
and you're pushing your experiments out to a very
high proportion of your population,
it could mess with your results.
So to use a somewhat contrived example,
imagine I've got two experiments-- one
where I'm testing the background colors of my buttons,
and another where I'm testing the text color of my buttons.
Well, if I decide to push these experiments
to 100% of a population, about half the people
in my red background color experiment
are going to end up with unreadable buttons,
and that's going to skew my results.
I'm going to think, oh, no, nobody likes
this red background color!
And that might not normally be the case
if it weren't for the confounding influence
of that other experiment.
Now, like I said, most of the time this won't be an issue.
But if you do find yourself in a situation like this,
you're probably better off going with the multivariant trick we
talked about in our second video and just
make this a four variant experiment
or run the two experiments one after the other
instead of in parallel.
OK, next question, what's up with these activation events?
So you might have noticed that when I created a remote config
experiment, I skipped over this advanced option section
when selecting a goal.
Well, hidden inside the section is a dropdown list
that lets you select an activation event.
So what is an activation event?
Glad you were rhetorically asked.
The general idea here is that for the purpose of measuring
experiment, we won't count the user as being in the experiment
until they first triggered this activation event.
Yes, they'll still be receiving whatever
remote config parameters you've sent down to this variant,
but they won't be included in any of your results
until they first trigger the activation event.
So when would you want to use this?
Well, generally speaking, you want
to do this when you think there's only
a subset of your audience who's going to be exposed to your A/B
test in the first place.
For example, imagine that we're A/B
testing our in-app storefront, and we
want to find out which version is more likely to drive
in-app purchases.
But it also turns out that maybe a small percentage
of our users, maybe 5% or so, even
decide to visit this door page in the first place.
Well, in this kind of experiment,
you'd want to make the visit store event your activation
event so the users who never visit your store simply
wouldn't get counted in your experiment.
Why does this matter?
Well, suppose that due to some random chance, more
people in experiment variant A decide to visit our storefront.
Then out of these two groups, 50 people
will make a purchase in variant A, and 45 people
make a purchase in variant B.
Now, if we to just look at the conversion rate
from being in the general variant population to making
a purchase, you might conclude that variant A did better
by a little bit.
It's got a slightly better conversion rate.
But if we were to make the visit storefront event our activation
event and just look at the conversions from there,
you would come to a much different conclusion.
This time it is very clear variant B is the better one,
because it did a better job of converting just the people who
visited the store.
And those are really the only users who
count if you think about it.
Honestly, another nice side benefit
of adding these activation events
is that the results end up looking
a little more intuitive.
Even if we end up with exactly the same number of users
visiting our storefront, seeing our users without an activation
event shows us we have a conversion jump of 1/10
of a percent, which feels insignificant and lame.
But if we were to rephrase this to, well,
out of the people who were actually
exposed to your storefront, we have a bump of 2 and 1/2%.
Well, that feels like it's worth doing
even though proportionately it's still
the same amount of improvement.
Now, two caveats about these activation events.
First, it doesn't count if a user has triggered this event
some time in the past before you started your experiment.
So if you're experimenting on some advanced user feature,
you might be tempted to add completed tutorial
as an activation event to make sure you don't include
any newbies in your experiment.
But in practice, this wouldn't work, because it would only
count users who trigger the completed
tutorial event while your experiment is going on.
So all of your regular expert users
who have completed the tutorial weeks ago
won't trigger this event and won't
get included in your experiment even though they probably
should.
The second thing is that activation events
aren't a per session thing or a per hour thing.
They're a "for the lifetime of the experiment" thing.
Why does this matter?
Well, let's imagine I've got an app where
there are two ways to get users to sign up for my newsletter.
One way is through a Settings page,
and another way is through a pop-up dialog
that pops up occasionally asking our users to sign up.
So let's say I'm trying to optimize this pop-up dialog
through A/B testing, and I decide
to add our pop-up dialog displayed event
as an activation event.
Now, does this mean that my experiment is only
going to measure whether users have signed up
through this pop-up dialog?
Unfortunately, not quite.
If my user dismisses the pop-up dialog
and then later signs up through the Settings page,
they'll still be included in my experiment,
because they'll have encountered this activation event earlier.
Now, granted, this will still help
a bit, like if we have users who never encountered this dialog,
they'll never be included in our experiment,
and that does help eliminate some of our noise.
But don't expect this to be a perfect filter.
If I really just wanted to focus only
on users who signed up through this pop-up dialog,
I might want to record a completely separate
signed up for newsletter through dialog event
and look to optimize that.
OK, last question, what if we want
to optimize for a goal that can't be measured
through the Firebase console?
Like right now, we have some very nice high-level goals,
like optimizing user engagement or attention, as well as goals
where we're looking to maximize the occurrence of an event.
But not all experiment goals work that way.
For example, let's say we've got some kind of puzzle game,
and our lead game designer has determined
that we want a player to win a level about 60% of the time
to be a nice challenge.
Or maybe we want to make sure that each level in our game
pays out the right amount of in-game currency,
not too much and not too little.
Well, this does seem like the kind of thing
you would want to A/B test for by playing around
with some gameplay variables.
But the problem here is that there isn't a way
to specify average payout is 20 coins per level
as a goal in the Firebase console.
But there is an alternative, and that's analyzing our experiment
data in BigQuery.
Now, I'm not really able to give you an entire breakdown of how
to use BigQuery.
That would be an entirely different video series.
But you can at least check out this video somewhere
in one of these Info Cards to get started.
But for those of you who have a little BigQuery experience,
here's how you'd analyze your A/B test results.
When you create an experiment in A/B testing, behind the scenes,
Google Analytics or Firebase will store a user property
for this user called Firebase underscore exp underscore
n, where n is the number of your experiment,
and the value will be the index of the variant
that they are in.
So you can use this to analyze your results in BigQuery.
So first, to find out the number of your experiment,
go ahead and open up your original experiment
in the Firebase console.
So you see that number there at the end of the URL?
That is your experiment number.
Now, the value of this Firebase underscore exp
underscore experiment number property
will be set to 0 if your user's in the control group, 1
if they're in the next variant, 2
if they're in the variant after that, and so on and so forth.
And of course, it will be null if your user wasn't placed
into this experiment at all.
So for example, let me go back and take a look
at that Bingo Blast experiment from a few videos ago
where they disabled ads.
So I can see here at the top in this URL
that it was experiment number 34.
So I can query this experiment in BigQuery,
like, let's maybe run a simple query here
where you see how many display AdMob add
events were recorded for each group in my experiment.
So what I can do here is grab the Firebase underscore
exp underscore 34 user property and the event name.
And sure enough, it seems like I am getting mostly
display AdMob events when our user is in variant 0, which
is our control group.
Here I'll add a little case statement to make this clearer,
and we can count up all of our events.
And as you would expect, it looks
like the vast majority of display ad events
are happening for users who are in our control group,
not the ads disabled variant.
Now, why isn't this other count exactly 0?
To be honest, I'm not sure.
There seems to be one or two rogue devices out there that
are reporting showing ads but are still
also reporting that they're in the no ads variant.
They all seem to be running on Android Ice Cream Sandwich.
So I suspect there might be some weirdness going
on with some very, very old versions of Android.
But we're talking less than 1% of devices out there.
So I wouldn't worry too much here.
And I can do other kinds of analysis as well.
Here's a query where I'm breaking up
the average score by whether you are
in the no ads variant or not.
Huh, that seems like a pretty big difference.
And you might conclude that taking a break
to watch some ads might be better for your long term bingo
playing.
But it turns out with bingo scores,
they can vary quite wildly based on how lucky you are.
So in fact, if I were to switch to a different day,
I can see that, well, here the control group is still
doing better, but the difference is much smaller.
And maybe that emphasizes an important point here,
which is that a simple BigQuery query like this
can give you averages, but it's not
going to tell you if these differences are statistically
significant.
That's going to require a lot more math.
Math that I don't really know.
And so while all of this is super interesting,
before you start querying all sorts of custom goals using
BigQuery, you might want to take a step back and ask yourself,
is this actually the goal I'm most interested in?
Sure, our game designer has decreed
that our puzzle game would work best if players
win a level 60% of the time.
But if you think about it, that's
not our real goal, right?
Our game designer probably came up with this number
because they determined that at some point
that a 60% win rate is the win rate that feels nice--
tough enough to be challenging, but not so difficult
to be frustrating.
And if you think about it, what our game designer is really
looking to do here is improve our attention.
And they've made an assumption, one that may or may not
be true, that a 60% win rate is the best way to do it.
And so what we really might want to do in our experiment
is optimize for retention as our primary goal,
which we can do simply using the Firebase console.
And then, once we figured out which group has actually
retained better, maybe we go back to our results in BigQuery
and find out what the win rate of our users
are actually experiencing in the winning variant.
So there you go, folks, three questions
you might have had about A/B testing answered.
Got any more questions?
Add them to the comments below.
Maybe we'll do a follow-up video if we get some good ones.
But in the meantime, go out there
and have some fun with A/B testing in your apps
and start making them better one experiment at a time--
or two at once, because you can do that, yeah.
[MUSIC PLAYING]