SPEAKER: So up until now, we've really only focused
on using A/B testing in conjunction with Firebase
Remote Config to help you A/B test
things that are happening inside your app,
but it turns out there are also things outside your app
that you might want to A/B test as well.
For example, how about those notifications
that you're sending?
Wouldn't it be nice if you could A/B test those too--
not just to see which ones are getting people to tap on them,
but also persuading them to do important things
like spend quality time within your app
or make an in-app purchase?
Well, it turns out you can use A/B testing for that too, which
is a good thing because otherwise, this would
have been a terrible intro.
Let's find out how on this episode
of "A/B Test Like a Pro."
[MUSIC PLAYING]
So A/B testing works in conjunction with Firebase Cloud
Messaging to help you run experiments with
your notifications, and Firebase Cloud Messaging-- or FCM,
as the kids like to say--
consists of a few different pieces that all work together.
First, you have the back-end service itself.
Now, this records which device token corresponds
to what kind of device so it knows
whether or not to route your messages through APNs.
It's also got a few other nifty features
that I'm not going to get into now.
It's a secret.
Then there's the client library, which
helps register your device with Cloud Messaging
and sends information about the notifications you
receive to analytics without your needing
to do any extra coding.
Then you've got the notification section
of the Firebase console.
This basically is a tool that makes
it easy for anybody in your organization
to send a notification through FCM without your needing
to write any code to talk to the service.
So you can compose your whole message in there,
maybe determine what subset of your users
you want to send this message to,
and even do nice things like schedule
your notification to be delivered
at some point in the future.
It also provides you with some basic analytics
around your notifications.
Now, for ordinary notifications, you
don't have to use the Firebase console.
And in fact, many developers have written their own tools
to send messages over FCM.
But for A/B testing, you do.
So to get started with A/B testing with notifications,
you're going to need two libraries on your client.
First, not surprisingly, is Firebase Cloud Messaging,
which provides the foundation necessary to make sure
that Firebase can talk to your device through the FCM service,
as well as enabling some of the other analytics you
can use to see your results.
Now, we already have some helpful videos
on how to get started with FCM on Android and iOS, so go ahead
and check those out if you need help adding
those libraries to your app.
But the good news here is that there's very little client
code you need to get the basic FCM service up and running,
which is all you really need for A/B testing.
Now, even if you already have Cloud Messaging installed,
I would recommend doing a pod update
or putting the latest version into your Gradle file
because you are going to need a fairly recent version
of the library installed on your app.
This will include the logic needed for Cloud Messaging
to tell analytics about any A/B tests
that we're running on our notifications, which leads us
to the other library you're going to need,
and that's Analytics.
Now, Analytics is part of the core Firebase library, which
means that if you've got FCM installed,
you already have at least a simple implementation
of Analytics running within your app,
but you might not be using it to its fullest.
As I noted in our first video, you
can use events recorded by Analytics
as goals for your experiments.
Now, Analytics automatically records a few events
like making an in-app purchase and opening a notification,
but if there are any app-specific events that you
might want to record as goals, such as completing
a level in your game, using whatever new feature you're
promoting, or visiting your in-app store,
you should make sure those are also
recorded as Analytics events.
So with these prerequisites in mind,
let's take a look at how we'd set up
A/B tests for a notification.
Now, this is going to cover a lot of the same ground
that we covered in our Remote Config experiments,
so I'm not going to spend too much time going
over content that I've already covered in a previous video.
If you do want more information about any of this stuff,
be sure to check out our other videos
in the description below, or probably linked in one of them
fancy YouTube info cards.
So let's say I've added a brand new Rate That Houseplant
feature to my app to help us better
understand what houseplants you might like.
Well, I'm excited to get people to try it out,
and I think a notification might be the way to do it.
Now, using Firebase Analytics, I'd
probably want to add something like a houseplant rated event,
if I didn't have one already, because that's
going to be the main goal of my notification.
I want people to get into my app,
try out the new feature, and rate at least one houseplant.
So it turns out I already have the FCM library installed
in my app, so really, just adding this one Analytics event
is all I need to get going from a code standpoint.
Now, to start an experiment, we're
going to want to go to the Notifications
panel of the Firebase console.
From there, we can click on this tab to start a new experiment
or see the status of all our existing experiments,
or we can click this Create Experiment button to jump right
into creating a new one.
Now, you can see here we have a lot of the same setup fields
as creating experiments in Remote Config.
We can give our experiment a name and a description,
decide which app we want to target,
and perhaps pick a subset of our users
to target with this notification.
Note that we can't use FCM Topics here,
but we can target segments like user audiences,
app version, which would be important
if only the recent version of my app contained this new feature,
and language, which will probably be important
since I'm going to be sending this notification in English.
We can also define what percentage of our target users
we want to send this notification to.
Like with other experiments, I might
recommend keeping this small if you're at all worried
that you might turn off users or create some kind of backlash
with a poorly worded notification.
Don't worry.
You'll still be able to roll out your notification later
to the rest of your users once this experiment is done.
Next, we get to our variants.
Now, this time, there's only one variable
we're allowed to change here, and that's the notification
message itself.
So we can try a couple of variations
here and see what works best.
One note here is that up here in the Control Group,
you have the option to leave this field blank.
This will basically not send a notification at all.
And so that's if you want to A/B test
the difference between sending a notification
and just not sending one, which can also be useful.
OK, now for a goal, we want to determine,
what is the main point of sending this notification?
Now, it could just be getting our users
to tap on the notification, in which case
I might want to add notification open as our main goal.
But is that the real goal you're trying to get to here?
It's probably not.
Most likely, you want our user to do something else
within your app, whether that's spend more time with it
or perform a specific task.
And you have to remember that even if the user doesn't
tap on your notification, they've seen it,
and that might be enough to get them to respond to your app
differently.
So in my case, since we want to direct users to check out
our new Rate That Houseplant feature,
we'll want to make our goal the houseplant_rated Analytics
event.
And as you can see down here, A/B testing by default
will also include notification open
among the secondary goals of my experiment.
So I can still see what version of my notification
got more people to open it without making
that the primary goal.
OK, in this next section here, you
can decide when you want to send this notification.
Maybe it's immediately, but maybe it's
the next day or at a specific time.
And just like with other notifications
you've sent through Firebase, you
can choose to send this at a time that's
local to your recipient's time zone.
So you can send notifications all over the world
without waking up your users in Indonesia
at 3:00 in the morning.
Down here are some other advanced features
that might look familiar if you've used Firebase
Notifications in the past.
You've got things like adding custom data,
turning on a badge count, enabling sound, and so on.
Now, just note that none of these attributes can be A/B
tested-- only the notification message itself--
but they will be included with every variant.
Oh, and remember that if you're on iOS,
you probably want to keep this Priority field set to High.
After that, you can click Review,
and we are back in our Experiment Review panel.
Again, this should look pretty familiar to you
if you've watched our previous videos.
And just like our Remote Config experiments,
you can also test any of your variants
on any individual device by using this Manage Test Devices
panel.
As you'll recall, I do this by grabbing my Instance ID
token by using this line on iOS and this line on Android.
I would then copy whatever giant string I get into here,
select the notification variant I want to see,
click Add, click Save and Send, and the notification
will show up on my device.
Now, once you've tried out all your test devices
and like the way the notification looks,
go ahead and click Start experiment,
and that will run the experiment.
Now, once an experiment is started,
you'll start seeing results trickle in after a few hours,
but it usually takes about a week for A/B testing
to officially declare a winner.
Now, this is about half as long as the time it
takes to declare a winner in a Remote Config test
since you don't have the same kind of hey,
I'm going to pay more attention to the things that
are different bias as you do with Remote Config, right?
It's a notification.
It's always different.
So I'm not going to get any meaningful results
from my little test app, but let's
take a look at some experiment results
for an actual experiment in Bingo Blast.
They ran an experiment a week ago
to see if they could increase their daily user engagement,
so let's see what they found out.
So I think most of these results here
should look pretty similar to what
we had in our Remote Config experiment results,
and you can always check out that video
if you want more detailed information about any
of the stuff we're seeing here.
So up here, you have a brief summary of your experiment--
if it found a clear winner or not,
or if it still needs more data.
And you'll have all your detailed results
here in this expandable section.
Now, below that, you can see your most likely improvement
range for all the goals you're measuring.
That will include your primary goal as well as
the secondary goals-- things like the rate
at which the notification was opened,
your app's retention, stats like that.
Now, the Bingo Blast folks tried three different notification
variants.
In the control group, they didn't send a notification
at all.
They kept that message field blank.
Then for the other two variants, they
tried two different notifications.
One was a back-to-school kind of notification,
and the other was a level up and unlock new content
kind of notification.
And you can see here that both of these notifications
did a significantly better job of increasing our user
engagement than not sending a notification at all.
See how these improvement goals are in a nice, confident green
with this little upward arrow?
That's A/B testing's way of telling us
it's confident as that these variants will do better
than the control group.
They also did a significantly better job
of causing a notification open event,
but I guess that's to be expected,
since this control group didn't send a notification at all.
For most of these other measurements-- in-app revenue,
causing a level-up event, and so on--
there's a general sense that they may have performed better,
but A/B testing isn't confident enough to be sure.
We've got some pretty wide ranges here
on most of these things.
And just like before, if you click
on any one of these results, you can
see more detailed information about them.
Here we've got the overall improvement
range, the likelihood that this variant is better
than the control group, the likelihood
that this variant is the best one out of all of our variants,
our raw value ranges, and in some cases,
you'll get some pretty nice-looking bar graphs
below if you're more of a visual results kind of person.
Now, you'll notice that in our experiment,
A/B testing is very confident that both of these variants
will do better than our control group of not
sending a notification at all.
And while it seems like our level-up notification did
better than our back-to-school one,
it's not quite as confident about picking
that as the absolute best variant, and that's why it's
telling us up on top here that there isn't
a leader in our experiment.
Sure, it can't pick a winner out of these two,
but I think we can say with confidence that sending either
of these notifications will work better
than not sending one at all.
So if I were the Bingo Blast folks,
I'd be looking to send out that level-up notification
to the rest of the group.
And how would they do that?
Oh, glad you asked.
So back up here at the Results Summary section,
you'll have more options depending on the state
of your experiment.
If you're in the middle of an experiment
and you're still gathering information,
you'll be able to increase your distribution to a larger
percentage of your potential audience.
When you do, you'll be given another chance
to schedule these extra notifications,
since you might have scheduled your first batch
to have already happened at some specific time in the past.
You also have your choice of stopping
the experiment, which will primarily just stop
gathering information.
And you can duplicate the experiment
if you want to try running something similar.
But probably the option you're most interested in
is having the ability to push out
a variant to 100% of your population
as a real notification.
If you're in an experiment with a clear winner,
it'll probably show up here as a big button.
But even if you're not in an experiment with a clear winner,
it should be available through a menu option here.
And just because it's not in a big button
doesn't mean you shouldn't do it.
You probably just ended up in a situation like Bingo Blast
where you had several versions that worked equally well,
so you can just pick one, even if we
haven't been able to statistically prove
it's the best one.
Now, when you do roll out a message,
you'll get a dialog like this one, where we have a chance
to pick a variant.
And if your experiment has a winning variant,
it'll already be selected by default.
You can make any last-minute edits,
schedule a time to push this out.
And this will basically go out to the rest of your audience
that wasn't in your original experiment.
Hit Send, and there you go.
It's sent.
Now, once you've rolled out an experiment like this,
it'll graduate to official notification status,
and it'll show up here in the Firebase Notifications panel,
just like any other notification.
And you'll have your usual stats here-- notifications
sent and notifications opened, although we
don't record anything else here as a conversion event.
Also, keep in mind that these stats aren't retroactive,
meaning that if you run an experiment
on 10% of your population then roll it out to the remaining
90%, I would expect these absolute numbers here
to be 90% of your typical results.
So don't freak out if they seem a little lower than what
you're expecting.
That's normal.
And as a nice little bonus feature,
if you're ever looking at a notification
in the console that began its life as an experiment,
you'll see a nice little button here
that will take you to the original experiment, which
I think is kind of neat.
So there you go--
everything you've always wanted to know about A/B
testing your notifications with Firebase.
Now, if you're already using the FCM library in your app,
it should be pretty easy to get one of these up
and running, so give it a try.
And I will see you soon in a future episode of "A/B Test
Like a Pro."
[MUSIC PLAYING]