TODD KERPELMAN: Quick, which one of these buttons
do you think would get more people
to browse your in-app store?
Are you sure?
Would you bet your entire company's future on it?
Because, you know, you're kind of doing that every day
when you make decisions like this.
So let's find out how to make sure you're
making the most informed decisions with a little help
from A/B testing from Firebase.
[MUSIC PLAYING]
OK, so what is A/B testing?
And why should you care?
I may have been a little over-dramatic in our intro.
But you do make lots of decisions every day
about your app, whether it's the look
and feel of a button, the text that's
designed to get people to try out a new feature,
or how often you show ads, or how challenging
you make level 12 in your game.
And while, individually, these changes might seem small,
together, they can have a big impact on your app's success,
often meaning the difference between a killer app
that feels just right versus a mediocre app that
never really gains traction.
But how do you know you're making the right changes?
I mean, you're probably a little too vested in your own app
to see it from the perspective of a typical user.
And sure, you could ask your friends, your family
or various focus groups.
But that can be kind of inefficient.
And you're usually dealing with small and probably biased
sample sizes--
sorry, Mom.
Also, frankly, sometimes what people
say they're going to do within your app
doesn't actually match up with how they really behave.
So another option is to make a change in your app,
publish it to the world, wait a couple of weeks,
and see if your users behave any differently.
So this can work.
But it's also time consuming if you
wanted to, say, test three or four variations.
I mean, assuming you want a couple of weeks
to try out each change, that could be,
like, two months spent just trying out those variations.
But more importantly, how do you know
the change in behavior you're seeing really
resulted from the change you made in your app?
I mean, maybe you ran a different ad campaign that week
that brought in a different set of users
with different characteristics.
Maybe your app got a mention in a blog post that brought
in a whole new set of users.
Or maybe the weather was just super
nice that week and people decided to spend time outdoors
and not on their phones.
It can't happen.
And so that's where A/B tests come in.
The idea behind an A/B test is that you
take a subset of your users--
usually not all of them--
and break them up into randomly assigned groups.
Then each one of these groups sees a slightly different
version of your app.
And you can measure how your users interact with your app
in each one of these different groups.
Synthase different groups are all
using your app at the same time, any external change,
like a particularly sunny day, will affect all of your users
equally.
So if there's a noticeable change between the groups,
you can be pretty confident the change
in the behavior that you're seeing is due to the change
within your app and not, like, some external factor.
Now, Firebase recently made some major improvements
to their A/B testing framework.
First, they added a whole bunch of enhancements
upfront to make it easier for you to get started with an A/B
test.
And then they added a whole lot of fancy math
under the hood to help you measure the results
and let you know that if these results you're seeing
are truly meaningful.
And we're going to show you how to get your app up
and running with A/B tests on this new system over the course
of the next few videos.
Now, the first two touch points that the team has opened up
to A/B testing are Firebase Notifications
and Firebase Remote Config.
Notifications are great for testing, well, notifications.
And Remote Config is good for testing just
about anything else that happens within your app.
So we're going to focus on Remote Config
initially in our video series, but we
will touch on Notifications in a future video.
So the first thing you're going to need to do,
if you want to run a good A/B test,
is wire up your app to use Remote Config.
Now, if you're not familiar with Remote Config,
you can kind of think of it like a key value
store that lives in the cloud.
Your app can download this data and change its behavior
based on the values that you've specified here.
Now, we've already got some pretty good screencasts
on how to get started with Remote Config,
both for Android and iOS.
So if you've never used Remote Config before,
I recommend you check these out.
And you can find links to them in the description below.
If you are somewhat familiar with Remote Config,
you might remember the process works a little like this.
First, you supply Remote Config with a bunch of default values.
Then, usually when your user starts up your app,
the Remote Config service will go out onto the internet,
grab any new values that you've defined from the Firebase
Console, and apply these new values
on top of your old defaults.
Then, when you query Remote Config for particular values,
it will either give you an updated value that you've
specified from the cloud--
if one exists-- or a default value if it doesn't.
And so the strategy we generally recommend
is to take any value you might want
to change, like every bit of text, every color, every game
balancing variable, and wire them up to use Remote Config.
Remember, because you're only downloading the values
that you've changed, the actual network
calls you make will still be nice and small.
But you still have the freedom to change
any value you might want later.
This does mean that, for maximum A/B testing effectiveness,
you'll probably want to spend a little more time setting
your UI elements by code, instead of relying
on what you set within Xcode storyboards or your Android
layout files.
It does seem like many large organizations
prefer to do this anyway.
Because dealing with git merges is usually easier with code
than with the generated XML file.
But it turns out it also makes everything a little more
Remote Config friendly, too.
And as far as localization goes, you should still be fine.
Remember, you're defining your default values
through code or through a plist or resource file.
So you can go ahead and continue to use
your localization-friendly code to set these default strings
or make sure that this external file is
among the resources that gets sent off for translation.
Either way, you'll end up with default strings
that are properly localized at runtime
for your current location.
Incidentally, since you are going
to end up with a slew of different keys
for, like, every constant you're defining within your app,
I do recommend making sure you use something
like an enum or a bunch of static constants
to reference these keys.
Don't rely on hand-typed strings,
because one typo could really throw you off.
And also, you want to make sure you
can use your ID's auto completion to help you out
here.
But you probably already knew that.
So let's take a look at my sample app here.
This here is my front page.
Isn't it lovely?
And I've added in this little slide
up panel thingy that's going to ask users to sign in
if they haven't yet.
And this is probably something I want
to experiment with to see what works best for driving sign
ins.
So let's think about how we would want to wire up
Remote Config within this app for maximum A/B testing
effectiveness.
Well, it looks like there's a bunch of values
I might want to tweak later.
Obviously, I want the ability to change
the text of this call to action label,
as well as maybe the text of these buttons.
This is probably less critical, but I might also
want the ability to change colors,
both the background color of my view, the color of this label,
and maybe the color of these buttons.
And then just to be complete, I've
also got the content of these two text labels going
through Remote Config, as well as a general label color
variable that I'm using to set the colors of all
of my other labels.
And finally, I'm setting the content of this image
through a Remote Config, too.
To do that, I'm creating a tiny little object in JSON
that specifies if this image is local.
And if so, what the local image name is-- or remote, and if so,
what URL to use.
Keep in mind that Remote Config doesn't
support JSON objects natively.
But it can certainly send them across as strings
and decode it on the client.
And in practice, that is what a lot of developers
do when they're using Remote Config.
Now, obviously, I could go even further if I wanted.
Like, in theory, I could probably
get every aspect of my app to use Remote Config, everything
from the font type and size to the exact spacing and layout.
But I think, for my app, this probably isn't necessary.
But you should decide what makes sense
in your particular situation.
Now, I don't want to derail this video
by getting into all the code that I
put into getting the sample app up and running.
But honestly, it's nothing too fancy.
However, if you are interested in the implementation details,
I'll be making a follow-up video where
I walk you through what I did.
And you will eventually find that link in the description
below.
So I've got my app wired up for Remote Config, which gives me
the ability to change all sorts of aspects about my app later.
But I'm not quite ready to run an A/B test yet.
I also need to determine whether or not
my changes are successful, which means I need to measure what's
going on within my app.
And for that, I will need some analytics.
Now, if you don't know anything about Google Analytics
for Firebase, the product formerly known
as Firebase Analytics, we also have some videos for you
to check out.
And if you're only going to watch one,
I recommend checking out our video on Events,
because that's primarily what you're
going to be dealing with.
And yes, I know the video title for it says iOS,
but, like, 90% of the video is applicable to Android folks,
too.
So go ahead watch.
So Google Analytics for Firebase works by recording events
that happen within your app.
A number of these are recorded automatically
by the SDK, things like in-app purchases or ad clicks,
which it can use to determine revenue,
as well as events like your user bringing
your app into the foreground, which can be used to determine
retention and engagement.
But you're probably also going to want to record custom events
that happen within your app, partly
because, well, by recording these events, you
can get a better sense overall of how people are interacting
with your app, but also because you
might want to use some of these custom events
as a way of measuring success within your A/B test.
See, when you run A/B tests on Firebase,
part of running that experiment is
telling Firebase what you consider
to be a successful outcome.
In some cases, it might be earning more money
within your app.
In other cases, it might be your users
spending more time within your app
or coming back to it more often.
But in other cases, it might be maximizing the occurrence
of a specific event.
For instance, if you want to see whether a change in your app
does a better job of getting your users to complete
your in-app tutorial, you probably
want to add a tutorial start and a tutorial-completed event
and then see if you can maximize the occurrence
of that latter event.
If you're messing around with the look and feel of a Save
to My Favorites button, you'll probably
want to create an add it to favorites event
so you can maximize that.
So going back to my sample app, I
think I'm going to add a sign in panel
reveal event to record that I am presenting the user
with this little panel.
But then I will also record a sign in event
when my user is signed in.
Because that is an event I will probably
be interested in maximizing within my experiments.
Finally, I'm going to want to make
sure I have a fairly recent version of Remote
Config installed so that it knows to tell analytics
about the experiments it might be running.
That means running pod update on iOS
and updating your gradle file on Android.
And you can refer to the documentation
for the exact minimum version numbers that you need.
So I am now at the point where I can create an experiment
by changing up a whole bunch of parameters within my app
through a Remote Config.
I'm also now able to measure the results of an experiment
by going to Google Analytics for Firebase,
either by looking at some of the values it's
recording automatically or looking
at some of my specific events.
So I think I'm finally ready to run an experiment for real.
And we will look at doing that in the next video.
So stay tuned.
[MUSIC PLAYING]