[TYPING]
TODD KERPELMAN: Oh, hey there, app developers!
Didn't see you there.
So if you've been following along with our series,
you've already learned about how to wire up your app
to use remote configs that you can create in A/B Test,
as well as how to add Google Analytics for Firebase
into your app so that Firebase can measure
the results of that test.
So now that we've done all our prep work,
why don't we create an A/B test for real?
[MUSIC PLAYING]
Now, as you'll recall from our last video,
I've got this great little sample
app where I have this little panel that
slides up and asks our user whether or not
they'd like to sign in.
And I've set up my app so that I can
change a whole bunch of values through Remote Config--
everything from the content of this text to the color of these
labels to the speed of the slide-in animation.
And so I want to run an experiment
to see what's going to be most effective for getting
my users to sign in.
So let's head on over to the Firebase Console
and make this happen.
Also, I mentioned this in the last video,
but I'm going to want to make sure I have
a recent version of Remote Config installed so that it
knows to tell Analytics about the experiments it might
be running.
And that means running pod update on iOS
and updating your gradle file on Android.
And you can refer to the documentation
to find out the exact minimum versions if you need them.
So from the Firebase Console, I'm
going to select my project and head on over to the Remote
Config panel.
Now, keep in mind our UI is always being updated
and these screens might look a little different
than what you're seeing today.
But they should be similar enough
that you get the general idea.
Anyway, you might notice here on the right there's
a new A/B Testing button here.
So I'm going to click on it, and this panel slides in.
And it'll tell me what experiments I currently
have running, which ones are still in draft form,
and which ones I've completed.
Well, right now I don't have any experiments running anywhere.
So I have this nice little Create Experiment
button at the bottom.
So let's click that and start creating
our very first A/B test.
OK, so these first few fields-- they're pretty straightforward.
I'm going to call this our sign in panel test
and give it a short but relevant description.
There we go.
And now for our target users, I'm
going to need to select a specific app.
Note that I can't run the same experiment on, say, my iOS
and Android app at the same time.
But if you think about it, that makes sense, right?
These platforms have different use patterns.
Your app probably differs between platforms.
And what works well on one platform
won't necessarily work well on another.
So I'm going to select my iOS app here-- just
to run the experiment on iOS devices.
And now I'm going to click the And button to refine
my audience further.
I might want to limit my experiment
to only certain users, like those belonging
to a certain analytics audience or those
who have a particular version of my app installed.
In my specific case, since I'm going
to be trying out some text changes,
I'm going to limit this to users who have their device
language set to English.
You don't want, say, your French-speaking users
to start seeing these English labels
that you're running A/B tests with.
That would just mess up your results.
Now, it just so happens that my app is already
using Remote Config to power every variable that I
want to test.
But my guess is that there will be
times when you're going to be creating brand
new values in Remote Config for the specific purpose of running
an A/B test.
And in those cases, you'll probably
want to limit your audience only to users
who have the app version where you've
added in these new values.
And so you'd be doing that by selecting Version here
from the dropdown list.
Next I'm being asked what percentage
of my potential audience to put into this experiment.
Now, if I made this 100%, that would basically
put my entire app population into the experiment.
But it's usually best to keep these small.
If my experiment were to introduce major bugs or mess
up my in-app economy or trigger some major backlash
from my community, I'd want to make
sure it's only affecting a small portion of my user base.
So I'd recommend keeping this value in the 5% to 10% range,
depending on how risky you think your changes are.
OK, now we get into the real meat of the experiment,
and that's the variance.
This is where I decide, for each of the variants
in my experiment, what different Remote Config values
they're going to receive.
So to start, I'll click the Add Parameter button.
And now it's asking me what parameter I want
to change in our experiment.
I think the first thing I want to mess with is
this call to action label.
Maybe this Save your Data Across Multiple Devices
text isn't that compelling.
This might not be an important use case to my users,
or they might not even understand what it means.
Now, the name of this parameter in my Remote Config setup
is Sign In Request Text.
So I'm going to copy and paste that directly from the code
just to make sure I get it right.
Note that if you have used other parameters with Remote
Config in the past, they'll be shown here in a dropdown
list as suggestions.
But you can also go ahead and type in new values
that you haven't used before, and that's fine, too.
So I'll select Create Parameter.
And now I have these fields to fill in.
So for our control group, I'm going
to leave this as no value.
Now, this doesn't mean I'm going to be passing an empty string.
Don't worry that all of a sudden my text is going to be blank.
This is just a way of telling Remote Config, go ahead and use
whatever value you'd use as if you weren't in this experiment.
And so for us, that's going to be the default value that we've
supplied in the code.
And I do generally recommend going with this no value
setting for your control group.
Along those lines, I also recommend not
mucking with this value much in the normal Remote Config
console while your experiment is going on.
Because if your results are changing in your control group
while the experiment is happening,
that could definitely mess up your results.
As for our first variant, let's change our text
to maybe emphasize that by signing in you can still
get access to your data in the event
that you lose your device.
Maybe I'll say something like, keep your data safe
and secure by signing in.
Let's type that in.
OK.
We also get a chance to name our variant.
This is really just for our benefit
when we're looking at the results
later so we can remember what we changed in each variant.
And so I will call this Safe and Secure.
So this right here-- this is a basic A/B test.
I've got two test groups.
The first group is seeing the original call to action text,
and my test group is seeing a modified version of that text.
And honestly, this would be enough to start an experiment.
But I could go a little deeper than this.
For starters, I don't need just one variant.
If I wanted to test out several different versions of this call
to action text at once, I could just by adding new variations
like this.
But I could also change more than one variable at once.
For example, what if I decide to change
both the text and the background color of the panel?
Well, I could do that by adding a Slidey View Color parameter
and changing that, as well.
And while this could work, you need to be careful here.
If this variant ends up being more successful,
well, I still don't really know why, right?
Is it the new text or is it the background color?
I honestly don't know.
And in fact, maybe this change would
have worked better if I hadn't changed that background color.
So if you're going to change multiple parameters at once,
my recommendation is to do this when the variables are related
and you can't really change one without also
changing the other.
For instance, if I were to change the background color
to something dark, well, I'd probably
need to change my text color to something that's whitish so
that the text is still legible.
And in this case, it would definitely make sense to group
these two changes together in a dark color scheme.
Alternately, I can just make sure
that I have my bases covered by making
sure I've got all combinations covered here in my variants.
For example, in this experiment, you
can see that I'm changing the button
text in this first variation but then leaving my Slidey View
Color unchanged.
Then I'm changing the Slidey View Color
parameter in my second variation while keeping
the text unchanged.
And then finally, I'm changing both the background color
and the text in the third variation.
For those of you who aren't hip to your A/B testing lingo,
this is known as multi-variant testing.
And I think this might be a fun experiment to try.
So you know what?
I'm going to go with this.
Once setting our variants is done,
our last step is to tell A/B Testing what our goal is
for this experiment.
And you can see here at the top of this dropdown
list it's got a bunch of high-level goals-- things
like increasing engagement, the amount of time people
spend with my app, or in-app purchase revenue.
But if I were to scroll down further,
I could start to see some of the events
that I've been recording in Google Analytics for Firebase.
And right here is my custom user signed in event.
Since the point of my experiment is
to see if I can get more people to sign in,
I think I'm going to select that.
You can see that A/B testing also, by default,
includes some other metrics to track.
Looks like right now it's including engagement,
retention-- that is, how many people come back to our app
after several days--
and minimizing crashes.
And this is all good information to include in my experiment,
because I want to make sure I don't mess up
any of these other metrics when I'm optimizing for my sign in.
I mean, I'm sure I could get super in your face
and aggressive with my sign in messaging,
and that would probably give me a boost in sign in.
But if that change also ends up driving people away
from my app-- well, that probably
wouldn't be considered a successful experiment, right?
So it's good to make sure I'm not
hurting my app's fundamentals when
I initiate a test like this.
OK, so I'm going to click the Review button.
And I can see a summary of my experiment right here.
Everything's looking pretty good.
But how do I know for sure?
I should probably make sure my text fits
the space I have reserved for this label
and that this green color isn't too garish.
Luckily, A/B testing makes testing these variants easy.
What we need to do is grab the instance ID token
of our application-- that is, the unique identifier assigned
to an instance of our app running on a particular
device--
and then we can tell Remote Config
to deliver a specific variation to that specific app instance.
So then we can try out each variation
before we publish our experiment.
So getting our instance ID token requires a tiny bit of code.
It's basically this one line on iOS
and this one line in Android.
And so I'm going to jump into my app for a second,
and I'm going to stick this line of code into my AppDelegate
and then run it.
And now I can grab my instance ID token right here
from the console output.
It should be a giant string that looks
a little something like this.
Next I'll go back to my experiment in the Firebase
Console and click the Manage Test Devices link here
at the bottom.
Now, in this dialog, I'm going to take this giant string I
copied, and I'm going to paste it into the Instance ID Token
field.
And then I can decide which variant
I want to push to this device.
Maybe I will try out my safe and green variations
so I can test both the new text and the new green color
at once.
So click Add.
And then I do also need to make sure I click Save.
OK, now I'll run it on my device and--
well, depending on how things are set up,
it might look exactly the same as before.
And that's because-- if you remember from our earlier
Remote Config videos--
the Remote Config Library will cache
values it downloads from the Cloud for about 12 hours
by default before it fetches new ones.
Now, in a typical production app, this is just fine.
But for testing purposes, this can sometimes
make things a little more difficult.
So I'm going to go ahead and make
sure I have Developer Mode enabled and my cache expiration
time set to 0.
And I'm also going to add a giant warning here
to remind myself that I shouldn't ship any production
app with this code in it or my app
will get throttled by the Remote Config service.
In fact, I will probably git-reset all of this
as soon as I'm done testing.
Anyway, let me give this another try.
This time, my app is grabbing new values from the Cloud.
Remote Config is giving it our specific experiment variation.
And there we go.
Look at that.
There's our new background and our new test message.
Nice.
Well, how persuasive.
I think I do want to sign in.
And if I wanted, I could go ahead and test out
all my other variations through this same Manage Test Devices
dialog.
I can just select a different variation from this dropdown
right here, hit Save.
And just like that, I'm testing my variant
with the new message on the old orange background.
Look at that.
And I'm free to add more devices to that Manage Test
Devices dialog so I can make sure things
look good on all sorts of other types of devices,
or I could use that to see the different variations
side-by-side all at once.
By the way, do keep in mind that this instance ID
token can, in theory, change from time to time.
But it generally only happens every few months
or if you reinstall your app.
But if you suddenly find that none of your test devices
are getting the variations you're expecting, just double
check that their token hasn't changed.
So I've gone ahead and created my experiment.
I've added in my variants and determined
what Remote Config values to change
for each one of those variants.
We've defined our success metric for this experiment.
And then I verified that all my variants look good
through this Manage Test Devices dialog.
I think at this point I'm ready to run my experiment.
Should we go for it?
Let's go for it.
I should probably note that once we start an experiment,
we can't change any of the values for any of the variants.
We also can't push out specific variants to specific test
devices anymore.
So just make sure you've done enough testing
with your variants that you're happy with how they look.
But you know what?
At this point, I'm pretty happy.
So I'm going to start my experiment.
And once I've done that, I get this page where
it shows me my results so far.
And as you might expect, I don't have a whole lot of data yet.
So at this point, my next step is to sit back and wait.
We're going to need some time for this experiment to run--
both so that it can reach the right amount of users
and also that they have some time to get
used to this change.
I'll explain all of that in our next video
where we talk about analyzing the results of an experiment.
But for now, go ahead make your own.
Even if it's something silly or frivolous,
it'll be a nice way to get your feet wet.
And then sit back, have a cup of coffee,
and I will see you in a week or two.
[MUSIC PLAYING]