ALEX LESTER: Hello.
Thanks so much for coming, and welcome to our presentation.
Today, we're going to talk about how
to achieve data-driven application growth using
Firebase.
My name's Alex, and I'm an engineering manager
on Firebase predictions.
YANLI LI: Hello.
I'm Yanli, a software engineer on Firebase predictions.
ALEX LESTER: So let's talk about how
Firebase can help you, as an application developer,
grow a successful business.
We all know that building a great app
is something important.
But it's only half the battle.
You want to grow your user base, retain your users,
monetize them, so you can grow your business.
So today, we're going to talk about a few key points
in your journey.
Firstly, how can we objectively measure your application
success, both in terms of industry standard metrics
and events that are unique to your application?
Secondly, we'll be looking at ways
to make changes to your app's look
and feel, as well as behavior, without having
to do a release to the various app stores.
Thirdly, we'll look at ways to predict user behavior
using machine learning.
And lastly, we'll be able to test those changes to ensure
that there's been significant impact
and that the changes we've made are not
having any unintended effects.
So let's start with a simple example of an application
that you could build, and then we'll
adapt it to some more sophisticated strategies.
So let's consider your new app, Sparky's Adventure.
In this app, you've decided to offer in-app purchases
to the user while you continue to develop features
to make the app more interesting and appealing
to a wider user base.
As your app develops and our team
becomes more sophisticated, we try
to run some tests to help improve this process,
or to ensure the changes that your team makes don't undo
any of your previous hard work.
But these tests can involve a lot
of one-off engineering time, which ends up
being difficult to scale.
But this is where Firebase can help.
Powerful Firebase back-ends like cloud functions,
cloud Firestore, and cloud storage
allow you to quickly build and scale your application.
Firebase also has great tools to make
sure your app stays performing and crash-free,
with products like Crashlytics, Test Lab, and Performance
Monitoring.
But Firebase can also provide you with some out-of-the-box
solutions that can help you grow your user base,
retain your users, and monetize them,
so you can maximize your business success.
So let's say one day Jane, your product manager,
wants to test the hypothesis that showing the most
expensive items first will lead to more users spending more
money per average user session.
Well, that's an interesting idea.
But it brings with it a lot of questions.
What other implications could this change have?
How would we measure success?
Let's start by using Firebase to get some key indicators
for this app.
First off, we need to add some analytics
to our app to measure the success of our app.
Now, the nice thing here is that you've already
integrated with Firebase.
Analytics is already up and running.
And if haven't, just by creating a project in the Firebase
console and adding the base SDK to your iOS or Android app,
you can get Analytics up and running
without any custom code.
You get a whole lot of automatic reports
right there in the Firebase dashboard on data
that's important to us--
things like in-app purchases, user engagement,
retention cohorts, daily and monthly active users,
and more, all of which lets us get
a good sense of the overall health of that app right out
of the box.
We will also go ahead and record any events that
are important to your application,
whether that's people finishing a round in your game,
visiting the in-app store, starting or completing
your in-app tutorial, inviting a friend, and so on.
This is all data that we can use later,
either in the Firebase dashboard or through BigQuery
to answer questions that we might have about our app,
like, what percentage of my users
are making it through the tutorial?
Is level five too hard?
Is level five too easy?
Adding these events needs generally one line
of code per event.
Back to our original idea.
Now that we know that we can measure the success metrics
we care about.
But how can we implement this change
to part of the audience in a controlled way
and roll back quickly in case it doesn't work?
How can we make these changes without an app release?
You can do it using Firebase Remote
Config as the primary driver of changes to your app.
Now, if you've never used Remote Config before,
you can think of it as like a key value
store that lives in the cloud.
And you can use these values to change the look and feel
or behavior of your app without needing
to publish an update to the various app stores.
The way Remote Config works is you set up a bunch of values
locally in a user's device.
Then as time goes on, when you want
to change one of these values, you
can do that using Remote Config, either in the Firebase console
or using our REST API.
Then, the next time the user starts up the app,
we will download these values and apply them
on top of the default values.
When we ask for a value, you'll likely
get the default value that we supplied initially,
or one of the new values we've downloaded from the cloud.
And this is nice, because it means
that we can have hundreds of values wired up and ready to go
through Remote Config.
But because you're only downloading the deltas--
the values that have changed--
you don't have to worry about excessive downloads or data
usage every time a user starts the app.
So for instance, if you've gone ahead and parameterized a sort
order of the items in the store as a variable in-app,
this is something we can update easily through Remote Config.
You can push our update to the world,
potentially only for a small fraction of our audience.
And if there's a problem, you can simply
come back to Remote Config and undo the change that we made.
OK.
So you have the change ready to ship.
But wait.
Jane has had a new idea.
Jane is wondering if you can change the sort
order to show the most expensive items first, but only do it
for the people who have spent money in our application
before.
Her hypothesis is that this will maximize revenue
among your spenders, while still enticing our non-spenders
to spend money by showing them more affordable options.
This is something we can do pretty easily with the systems
that we already have in place.
When you implement analytics, you
have the ability to create audiences.
These are users who meet some set of criteria in your app,
based on triggering events or being assigned to various user
properties.
And one audience that analytics creates automatically for you
is purchasers.
This consists of anybody who has ever spent in your app.
One of the nice features about Remote Config that I haven't
mentioned yet is that it contains
logic that allows you to send different values
to different users, depending on their situation.
So for instance, you can send different values
to your Canadian users, either for a nice localized
experience--
Sparky's Canadian Adventure-- or test a new feature in Canada
before rolling it out to the rest of the world.
But you can also deliver a different experiences
based on analytics data, like whether a user is
in a certain audience.
So you can tell Remote Config, hey,
let's create a special condition based on whether or not people
are in the purchasers audience.
And then you can tell Remote Config
to change your sort order only for people in this group.
And you get to implement Jane's new monetization strategy, all
without touching any of the code in your app.
While this makes Jane happy, it also gets her thinking.
What if our past purchasers tend not to spend money again?
Thus we should focus on racking expensive items more highly
only for future purchasers, meaning
those that have not purchased but will likely
purchase in the future.
Can we really predict the future?
Yes, you can, with a little help from machine learning.
And we're going to do this by using Firebase Predictions.
Let me hand it over to Yanli, who can teach us how.
YANLI LI: Thanks, Alex.
Yes.
With Firebase Predictions you can
apply the power of machine learning to segment
you are users based on their predicted future behaviors.
The general idea is that as your users are interacting
with your app, the day getting sent to Google Analytics
for Firebase.
Once enabled, Firebase Predictions
will look at our data and learn from the patterns
that your users exhibit before they perform certain actions.
So let's dig a little deeper into what we do.
It's actually-- this is a supervised learning problem.
The first thing is to come up with examples
to teach a model what it is exactly
that you want it to learn.
The motto is updated every day, so every day we
take the last 45 days of user activity data,
reserve the last 7 days as the labeling window,
and the past 38 days as the feature window.
In the labeling window, we figure out
what actually happened and give labels to the users,
such as churner, spender, reviewer, et cetera.
And in the future we know we [INAUDIBLE]
features that can represent the user's behavior patterns.
The data is separated into training set and evaluation
set.
We feed the training set into a neural network powered
by TensorFlow for training the model.
Then, we apply the model to the evaluation set,
to generate statistics about quality of the predictions,
such as precision, recall, accuracy, et cetera.
And then we expose some of these metrics on the UI.
Once the model quality is verified,
we are ready to apply the model on our latest user data
to generate prediction scores [INAUDIBLE]..
There, you can target them in the Firebase products
that you typically would use, like
for instance, Remote Config.
So how do you use Firebase Predictions?
Do you need to go back to the university
and spend several semesters taking on neural networks
for learning class?
No.
You just need to click this button here on the Predictions
console, and that's really it.
At this point, Predictions will start a model training
and evaluation process for the two [INAUDIBLE] predictions,
spend and churn.
So by default, we will predict who
will spend money in your app, or who is
likely to stop using your app.
But really, Firebase Predictions can
be set up to predict around any that you have in your app,
like who's going to create accounts or share
the app with their friends.
Creating customized predictions is very simple.
You just need to click the Create a Prediction box
in the Predictions console, and then
provide a name so you can easily find it later and choose
the specific analytics that you want to predict for.
Then, in a couple of days, you'll
start to get the predictions that you can work with.
Let's see.
Now the UI shows your submit a review
prediction is ready to use.
By clicking this Explore and use prediction button,
you'll be able to select a threshold
to control the audience size and prediction probability.
On this Select user segment page,
you can expand a graph of [INAUDIBLE] users.
They are sorted from left to right in how likely they
are to submit reviews.
The probability score is on the y-axis.
We can use these blue sliders to select a segment.
And as we move the sliders, you will see the bars and the data
change on the right to give us more
information about the [INAUDIBLE]
that we are selecting.
After selecting the segment, you'll
be able to choose where to use the prediction
with other Firebase tools, such as Remote Config, In-App
Messaging, and Cloud Messaging.
And that's how easy it is to use Firebase Predictions.
Now you can leverage the power of machine
learning in your app.
But Jane's just thinking, what if I
want to dive deep into my data?
Or combine with my in-house data source?
Or export it to a third party system?
Firebase and BigQuery have you covered.
Many of Firebase tools already integrate with BigQuery,
including Predictions.
To export your Predictions data to BigQuery,
just go into your settings and find BigQuery integration
[INAUDIBLE].
The BigQuery integrations page allows
you to easily manage all of your links in one place,
turn them on and off, and [INAUDIBLE] data set, easy.
After linking your app, data set will
be created for you are visible from the BigQuery UI.
From here, you can [INAUDIBLE] against your raw and sample
data you can also export or join your data
sets with external data sources to perform custom analysis.
So going back to Jane's request.
We are going to change that condition in Remote Config
to now include users who we think
will spend money in the future.
Let's change a condition to be based on spend predictions,
then further adjust the percentile of users
will be included.
For example, only the top 1% users
who are most likely to spend in the future.
Hit the Save button, and then change the sort_order parameter
value in a way that you think will
generate the most of that revenue for these users.
Great news.
The new sort order for future spender is ready to ship.
But Jane starts thinking again.
How can we be sure that this has the intended outcome?
Even if we saw an uptick in spending,
how do we know that it was due to the changes
we just made instead of some random chance
or some other change in our app or maybe
a promotion we were running at the time?
Many apps have a variety of success criteria.
Let's look at ways that you can measure
the impact of your actions accurately,
as well considering any unintended consequences
of your change.
This is what Firebase A/B testing is for.
A/B Testing helps make sure that all organizations in your app
are backed up by actual data and the statistical analysis.
So let's try to prove Jane's hypothesis by data
by creating our A/B test.
Let's go ahead and Create experiment
on the A/B Testing console.
Let's choose to use Remote Config.
In our case, their target audience
is going to be users who are predicted
to spend money in your app.
We can run experiment on, say, 30% of these users.
Then you can tell Firebase that you
want to maximize in-app purchase revenue as our goal.
Note that Firebase can also check
a number of secondary goals.
This is very useful for determining
the overall health of your app.
If you made a change that boosted your in-app purchase
revenue that was so aggressive that it lowered your user
retention, that would be important to know.
So you secondary goals can help you check that.
In the Variants section, you can add a few different parameters
values here.
You can have up to 70 different variants if you want.
Then you can check back in a couple of weeks
and see what happens.
This is what a sample report might look like.
These reports come with a lot of tables and charts.
But the most important takeaway is this line here
on top of the page.
It tells us that for this experiment,
we have 99.9% confidence that variant A, which
is ranking expensive items higher for future spenders,
will increase both estimated [INAUDIBLE] revenue and user
retention.
So this is great we tested our hypothesis
using experiments and data.
Now, let me hand back over to Alex to wrap up.
ALEX LESTER: Thanks, Yanli.
It's worth pointing out that just about everything
we talked about today--
whether it's using Remote Config to change how your app behaves
in different situations or changing how your app behaves
against users predicted to hit some conversion event,
running an A/B test to improve your purchase revenue,
or perhaps running a cloud messaging campaign
to tell users about your next big sale, all of this
can be done by anyone on your team
directly through the Firebase console,
as long as you grant them permission.
So once your engineering team has done the work
to set up each of these features, your growth
team, your game designers, or others
that want to help you succeed can
run all of these experiments or marketing campaigns
directly through the Firebase console,
while your engineering team can go off and work
on the next big thing.
And your new app's monetization strategy
gets to go from a very simple, provide in-app purchases
and keep pushing more features kind of a strategy,
to a data-driven, fully-optimized monetization
strategy.
It creates audiences to get metrics like Google Analytics,
applies different parameter values to different audiences,
applies different parameter values
based on predictive behavior, and maximizes in-app purchase
revenue through A/B Testing.
So I hope you've learned more today about data-driven app
development.
To sum up-- one, we measure our overall app health
and set custom measures.
We can use Google Analytics for Firebase.
Two, to make changes on device without pushing
a change through the app store, we
can use Firebase Remote Config.
Third, to predict user behavior, we
can leverage Firebase Predictions.
Four, to prove our success, we can
leverage Firebase A/B testing.
If you're interested in learning more about anything
we've talked about today or any other user application
growth-related questions, we'd love
to hear from you on the Dory.
Thank you.