Hi, everyone. I’m Andrea Wu, a software engineer on the Firebase team.
As you may have heard, we just released Remote Config Personalization into Beta,
and I'm so excited to share more about it.
I’ll also talk about how it compares to Firebase A/B Testing,
a longer standing product of Firebase that you may have used before today.
Personalization is a powerful new feature
that enables you to create several user experience alternatives
and automatically provide the best one to each user
based on a metric goal that you’ll choose to maximize.
How will it accomplish that?
Well, it uses machine learning efficiently trading off
between learning the best experience for different types of users
and making use of that knowledge to maximize your objective.
This means not only will it find the best experience for each user.
It will also be able to learn and adjust to changing user behavior,
always finding the best experience at any given time for a particular user.
This may sound similar to what A/B Testing does too,
but there are some key differences.
As a reminder, A/B Testing enables you
to run experiments to test hypotheses of optimal experiences for your users,
essentially gathering data to see how your users respond to different experiences.
Like Personalization, A/B Testing is also trying to figure out
which options served to users will optimize your objectives.
So how are A/B Testing and Personalization different?
When should you use each one?
I’m glad you asked.
Both Personalization and A/B Testing
can be used to find optimal experiences that maximize an objective.
One major difference pertains to how that optimal value is calculated.
In Personalization, machine learning algorithms
predict the optimal alternative for each individual user
and future predictions are updated based on the actual results that follow.
A/B Testing, on the other hand, does not predict an optimal experience in advance.
It serves each experience to a different group of users
and measures the results.
Bayesian statistics provide additional insights about the observed data.
Based on the differences in how these systems operate,
Personalization is better for when you want to directly maximize your objective
and A/B Testing is preferred when you want to maximize your confidence in a decision.
The ability to build confidence in a change using A/B Testing was helpful for Vinwap,
an Android app that helps users personalize their Android devices with live wallpapers.
They wanted to increase revenue by adding more ads,
but were concerned about a significant drop in retention caused by ad fatigue.
Thus, they ran an A/B Test, and split users into two groups.
One of which saw ads in the usual number of places,
while the other saw ads on all screens.
After examining the results,
which showed an increase of ad revenue by nearly 30%,
they decided to roll out this change to their entire user base.
They felt confident in this decision because the supporting statistics within A/B Testing
showed that the retention impact would be negligible, if not zero.
Related to optimizing an objective and using the outcome,
both A/B Testing and Personalization have built-in and custom objectives
and secondary metrics to measure.
These secondary metrics just provide information
as they do not impact results.
A secondary metric measured for Personalization
has no impact on which alternative is served to a user,
and A/B Testing does not factor it in account when determining which variant is best.
Personalization currently has two predefined events to measure:
user engagement time and ad clicks, and more will be added over time.
A/B Testing has many predefined events from which you can choose.
A/B Testing and Personalization have another key difference.
Personalization is optimized for each user,
while A/B Tests are optimized for all users or a defined subset of users.
With personalization, you’ll provide different experiences
and have the Personalization system decide which experience to serve to each user.
On the other hand, when you run an A/B Test,
the result will be whichever experience was the most optimal across all users.
If you choose to implement that experience as a result of running the experiment,
all users in your predefined group will have the same experience.
Somewhat along the same lines,
Personalization is ever evolving and always adapting to user behavior.
What a user prefers now may not stay the same in the future,
and Personalization will serve a different experience to the user
if another experience optimizes your objective better.
For A/B Testing, the experiments to compare experiences are across a set amount of time.
Whatever outcome results from the experiment
are what’s true for users during the timeframe of the experiment.
If you wanted to check for the optimal experience in the future
and make sure it’s indeed still the optimal experience across your users,
you can run another A/B Test.
Let’s say you wanted to maximize the number of users
who rate your app in the Play Store when you prompt them to.
One factor that might contribute to success is the timing of your prompt.
Do you show it when the user opens your app for the first, second, or third time?
Or do you prompt them when they successfully complete certain tasks?
The ideal timing likely depends on the individual user.
Some users might be ready to rate your app right away,
while others might need more time.
Personalization is great for this, as we saw in the keynote.
Personalization is also helpful for more advanced use cases too,
like personalizing parameters that control difficulty settings in the game.
If you made a super fun game and wanted to keep your users engaged,
perhaps you’re thinking about adjusting game difficulty for each user
so they don’t stop playing your game due to the level being too easy or too hard.
Suppose game difficulty is based on how fast your character moves,
the number of retries as a user receives,
and frequency of power-ups to replenish damage.
Let’s see how we can use Firebase Console to set up this Personalization.
First, let’s go to Remote Config and create a new parameter.
Let’s name that parameter “game_difficulty” and add a description of:
“sets game difficulty based on character speed,
number of retries, and frequency of power-ups.”
Next, we can click on the “Add New” button and select “Personalization,”
which will open this panel here for us to fill out the next steps.
We’ll have to decide on combinations of speed, retries and power-up frequency
to serve to different users.
For our first alternative, suppose we want to make the game super easy
and make the character very slow with one step per second,
give unlimited retries, and give a power-up every five minutes,
or 300 seconds, of gameplay.
Let’s click on this button which will open our JSON editor,
and we can enter these values here.
Let’s enter some other alternatives with higher difficulty,
perhaps three steps per second, a finite number of 10 retries,
and a power-up every 10 minutes, or 600 seconds.
Our most difficult mode would have a very fast character
at eight steps per second, just three retries,
and a power-up every 20 minutes, or 1,200 seconds.
Next, we’ll choose the objective, which is what we want to maximize.
We see here that there is a predefined objective of user engagement time.
Let’s go ahead and pick that.
This is a metric that Google Analytics automatically measures.
We can also add up to two secondary metrics to follow in case we are curious
whether maximizing user engagement time affects these other metrics.
We may be curious on how often users level up and unlock secret side quests,
so we can choose those metrics as well.
These additional metrics don’t affect which alternative will be served to a user,
and they will simply be measurements for your knowledge
as the chosen objective, up here,
is the only deciding factor for which alternative a user experience is.
Let’s pick “level_up” as our secondary metric.
We’ll then have to choose the group of users who will get these personalized values,
and it’s important to note that a portion of the chosen group
will be reserved for the baseline or what we can think of as the control group.
They will receive random alternatives to act as a benchmark
for the performance of the Personalization.
Let’s say we’ll create this Personalization for all Android users of our app.
Let’s go ahead and create conditions to do so.
We’ll call the condition “Android users”
and for the audience, we’ll go ahead and pick “Platform”
and choose “Android” as a platform.
Or if you don't want to involve your entire user base
in receiving varied game difficulty, you can just target 10% of your users.
Since Personalization is a part of Remote Config,
you can use any targeting condition for choosing an audience
to receive personalized values.
Let’s save this condition,
which will then be applied to this Personalization.
After saving, we’ll see our new Personalization, here, and our parameter,
and we can click save and publish the template.
Now that we’ve created a new Personalization in Firebase Console,
let’s see how we can actually serve those different experiences to users.
We will install the Firebase SDK for the platform of your choice.
In this case, I’ll use Android in Kotlin.
After adding Firebase to our Android project,
let’s take a look at our code for the Android app.
To get our Personalization values,
we’ll create a singleton Remote Config object,
which is used to store parameter values.
Since we want to adjust the game difficulty depending on the value of this fetch,
let’s get the value of our parameter from Remote Config
so we can retrieve the values for speed, retries, and power-up frequency per user.
We’ll get the value of “game_difficulty” as a string from Firebase Remote Config
and convert it into JSON, then save it into a variable.
We can then get and store the speed, number of retries,
and power-up frequency into variables, as well,
to use wherever we need to use them throughout the app.
These constants now store the particular user’s alternative served by Personalization,
and we can use these variables, wherever in our app,
that change the speed, number of retries, and power-up frequency.
When we set up Personalization,
we also set up a secondary metric to see how often users level up
as a result of personalizing game difficulty.
In our game code, we need to make sure to use these analytics events
when a user levels up.
After a period of time, we can view Personalization details
and compare how effective Personalization is by looking at the graph here.
The graph shows how well the objective is being maximized
by comparing our baseline users who are receiving a random alternative
versus those who experience the app with Personalization.
There’s also another graph here, for our secondary metric,
of a user leveling up.
And, just as a reminder, the secondary metric
has no bearing on the algorithm deciding which alternative a user receives.
We can see here that Personalization is truly better than the baseline,
as the line for Personalization is above the baseline
and there’s a positive number here for the lift.
Over time these values may change, and we can keep checking these metrics
to see if Personalization’s effectiveness.
If the lift is still positive,
we can continue serving the same Personalization alternatives,
and if we’re not receiving the desired results,
we can consider using different values.
I hope this gives you a clearer understanding of Personalization and A/B Testing
as well as when to use each one.
Thanks for coming to the talk today,
and I hope you enjoy the rest of Firebase Summit.