- Hi everyone.
Today we'll discuss about how we increase the revenues
within our product called offer wall
using spark, XGBoost and statistics.
I'm Daniel.
I'm a data scientist here at Fyber.
Alongside me is Michael.
He's the data science and BI lead.
Let's go over to agenda.
We'll start with a brief overview about Fyber.
What we do.
Who we are.
Then we'll dive into the business use case.
Then we'll explore a solution,
both in terms of what we looked for
and what we actually implemented.
Then we'll walk you through our AB testing mechanism,
what we test and how we test things.
And then we'll serve some main insights that we had
and learnt during the journey.
Let's talk with a brief overview about Fyber.
Fyber is an IT company.
We sit in the domain between the advertiser
and the publisher, which is the app developer.
We're a global company.
As you can see,
we're spread all over the world.
We're publicly traded.
And most importantly,
we are hiring so you can check our careers
within fyber.com.
We're a big data company.
As you can see, we have tons of events being processed
and passed through our system each and every single moment.
For example, we have 25 billion auctions per day,
200 million DAUs, which are daily active users,
300 terabytes per day, which makes it a lot of data.
Why are we here?
Marshal, our algorithm which we'll discuss about today,
increase the offer wall of this product revenues by 11%.
In terms of absolute revenue, this means a lot of money.
Let's discuss about our business use case.
Offer wall is a product which is integrated
within a user's application.
Basically,
this product contains offers
in which user can perform,
execute and then proceed within some kind of a game.
So, let's say you are a user
playing some kind of an application,
and now you're stuck.
You don't have any currency,
any life in order to proceed to the next phase.
What you can do is actually complete some kind of an action.
And once you have done this action,
you can get your in-app currency
and thus proceed within the game.
So, what's the motivation within this kind of a project?
First, we'd like to increase our user engagement.
We can do it by offering or showing the relevant users
the relevant ads.
Second, we can maximize the revenues for our clients,
which are the publishers.
And of course, the Fyber itself.
What are the challenges in this kind of a project?
First, our data is too big
within this product for ordinary frameworks.
We're talking about the scale of hundreds
of millions of events.
An event can be, for example,
an impression, a click by some kind of a user, et cetera.
Also, another great challenge
is that we're living in a delayed feedback environment.
Meaning that a conversion,
the actual tests a user will perform
can take up to a few minutes or up to a few hours
or even few days.
So, that's another great challenge.
- Okay, cool.
Thank you very much, Daniel.
I'm gonna elaborate a bit more about the delayed feedback
or the delayed conversions,
how we call it in the auction.
So, let's take three scenarios.
Three users.
User A,
B,
and C.
All of them clicked on a task on January 1st.
User A took two days to complete the task
and the value for us of the task was $2.
User B took nine days to complete the task
and the value was three and a half dollars.
And user C actually took 30 days to complete the task.
However, it was worth a lot to us.
It was worth $40.
Cool.
So, now, we can actually try to think about it
in an algorithmic way.
So,
we think we are looking at this
as the multi arm bandit problem.
The multi arm bandit problem
is when you are in a casino
and you have several slots.
And you want to level the slot
that will maximize your level.
So, the basic strategy should be to pull a little bit
on each one of the slots
in order to understand which one of them
maximizes your revenue.
Cool.
So, we look at it in a similar way
in also one where we want to give the best task to the user.
However, it's a bit different
because let's say when you pull the lever,
it will take some time for you to know
whether you succeeded or not.
So, this is multi arm bandit
in the delayed feedback settings where we want to know
which task is the most appropriate to the user
but it makes, it might take us some time to estimate it.
Cool.
Let's talk about now the solution exploration.
So, small recap as Daniel mentioned.
Well, in a big data environment,
most of the data is tabular.
It is time serious oriented,
meaning any impression might come at time T.
A click might come at T plus one
and eventually a conversion might come at T plus T.
And again,
this is the delayed feedback we are talking about.
So, where we looked,
we looked at Kaggle,
which has awesome data science project there.
And we had a literature review
where we found a very good paper
by Olivier Chevelle, Chapelle,
sorry Olivier,
from (indistinct)
I think now he's at Google
last time that I look at C.V, but okay.
And we took some ideas out of this paper
called modeling delayed feedback in display advertising.
And one of the points that the paper says
is instead of directly calculating
the probability of conversion in case of an impression,
split these to two.
And first, calculate the probability
that a user will click in case he saw the ads.
This is P of click.
This is the first stage to complete the offer.
And only then, calculate P of conversion
in case a click has already occurred,
the conditional probability.
And afterwards, do a multiplication of both
to get to the original,
a probability that you're aiming for.
And why is it good?
Because
in that way,
we don't try to predict all at once.
We are actually using different tools
or more appropriate tools for different problems.
However, we do need a framework
that is able to deal with our needs which are
big data aggregation,
machine learning and modeling also in big data,
dashboarding,
visualization and testing,
and of course,
debugging and troubleshooting.
Cool.
So, now I will give it to Daniel.
- [Daniel] Thank you, Michael.
So, let's see what we actually chose to implement
on our end.
So, let's see architecture HL overview
of what we actually managed to do.
First, we have the row events from S3
and the row event can be for example,
an impression event,
a click event as I described earlier.
Then we have airflow scheduler who schedules two processes.
The first one is the CTR prediction model,
the machine learning model,
which tries to predict the CTR.
The second one is the conversion prediction model,
which Michael described,
which tries to predict the conversion probability.
Then, we unify those two outputs or predictions together
and we store all of those predictions within aerospike,
so that the application layer, meaning offer wall,
will be able to take those predictions,
query them fast
and present them to the relevant users
in the relevant order.
Let's talk about the CTR prediction model.
So, we used XGBoost as our model.
Due to the fact it has some main advantages.
First, it uses spark.
And this is of course great,
based on (indistinct) and our big data.
Second, it knows how to handle missing values
within the dataset.
I'll elaborate about it in a bit.
And third, it knows how to handle great performance
with tabular data.
So, let's see how it works with databricks.
Xgboost4J, that's the name of the project
that is being constantly updated
and stabilized between XGBoost and spark.
We use it, of course,
to perform what is known as distributed training.
I'll show an example in a bit.
It knows how to integrate easily
with any spark ML framework aka MLib.
And of course, databricks contains all of the above
which makes it pretty easily
and straightforward to use.
So, what you have to do basically,
is within the documentation
you can read more about it.
However, I added some code samples
in which you can have a look as well.
First, you'll need to import your relevant libraries.
Then, you will need to set whatever data processing
you do.
For example, here, we're doing string indexing
on some kind of a categorical code.
Then you will need to make sure that your data
is within a vector data structure
because that's what the ML model,
the XGBoost model, expects for it to be.
Then you can create yourself
your relevant model parameters
using a map of key value.
Then you can create or instantiate your relevant model.
Make sure you train it on your data set.
Once it has been trained,
and by the way, as you can see here,
we're training it in a distributed training.
Meaning we're training it on two specific machines
in this case.
And once it's done training,
you can transform it on your test dataset
in order to generate some kind of predictions.
One of the great advantages about XGBboost
is that it knows how to handle a missing value
within datasets.
In tree-based algorithms,
branch directions,
or what is known as default directions,
are being learned during the training.
You can actually signify XGBoost
the specific missing value in this case,
minus 999, as if it was some kind of a missing value.
Then, once XGBoost will see this value within the dataset,
it will know to handle it accordingly
winning this default direction logic.
I'd like to mention another word about MLeap.
One of our technical challenges was how to save our models
or pipelines using specifically models that were generated
by sparks specifically as discussed XGBoost4J.
We looked for a solution, which is actually able to do so.
However, it's seems it's not that trivial and easy.
However, MLeap provides all of the above.
Also, databricks contains great recommendation about it.
This has made, of course, the process much easier.
And I think that that's a great opportunity to share
that we wrote a short blog post
about how to create the synergy between spark and MLeap.
- [Michael] Thanks Daniel.
Okay.
So, then we talked about the click prediction
and now we're gonna talk about the convention prediction.
So, again, very small reminder,
conversions slash feedback will arrive with a delay
one hour, one day, one week.
What we want to do is we want to predict
the number of conversions that arrive
before they arrive
in order to make our model faster and better.
For this purpose,
we look at this flow as a poisson process.
And poisson process is mostly used
where we count the occurrences of events
that happen at a certain rate,
but at random.
In the offer wall example,
specific tasks happen at a certain rate
but because each users decides independently
whether to start the task or not,
it happens at random.
So, taking this a step forward,
a single event within the disposal process
can be more than using the exponential distribution
and actually doing the probability estimation
using this distribution
and apply or pay which I'll elaborate a bit more
is relatively easy.
You take the prior P and you multiply it
by the exponent of the parameters shown here.
So, let's talk about the parameters.
First parameter is (indistinct)
which is just one over the average time
to convert from the click.
The X is the elapsed time since the click
and the prior P is our past knowledge
on the probability to convert.
Using only these parameters,
we can actually calculate a probability
for each user's click to actually become a conversion.
Okay.
So, now we have a model for the click
and for the conversion
and we need to orchestrate everything together.
We chose airflow
and decide they can follow the documentation.
Airflow is a platform to programmatically author,
schedule and monitor a workflow.
Our data pipeline is relatively complex
and we have several dependencies
that are affecting each other.
However, databricks sends an operator,
which is relatively easy to use.
You need to configure some options,
but it's not that hard.
And also,
there is a good documentation about it
so you can just read it.
Yeah.
So, now we will move to the testing and analysis.
So, our best practices, what we decided to do,
is to choose one dominant KPI beforehand
and two supporting...
Two or three supporting ones.
We build an analysis tool for analyzing tests
and we run a small AB test
and later on analyzed it using databricks dashboards
and a scheduler.
And let's see how we did it.
We read all data events using spark.
We aggregated to intermediate results.
And then saved it on an hourly manner
using the scheduler.
Very easy.
We used afterwards SQL, the built-in widgets,
in that the bricks and libraries like bokeh
to build a one time dashboard.
And again, we used the databricks scheduler
to schedule it every couple of hours.
And we actually shared the link
with our colleagues so they can look as well.
So, as you can see here,
this is an example for the data pipeline.
So, you can see that we attached the notebook
and we decided on the scheduling
and let me and Daniel know if something happens.
It's very quick, very easy.
I cannot share the real data, unfortunately,
but it's something one of the tables
or something like that.
The summary statistics table
where you have variant A, B, C, et cetera.
And you can show what KPIs
under performance or over performance variant.
This is not a statistical test or hypothesis test.
This is just a summary statistics table.
Another cool thing that you can do
is actually plot a histogram
from the model prediction distribution.
And for example,
we plotted model A, B, and C.
And we noticed that in model A,
there is a slight bump between 0.2 and 0.3.
And we know from our domain knowledge
that this does not make sense that there is so much of them.
So, we actually took a look at the model
and we found out that we had the bag
in the missing flag feature that Daniel talked about.
We fixed it.
And it was a very nice analysis.
Everyone was saying it, we will say it as well.
Exploratory data analysis is crucial.
Real world data has so many pitfalls
and you need to be extra careful when there will be.
And the second thing is that there is a high chance
that the first experiment will go wrong.
And actually, our phase experiment
with the mouse shell algorithm.
This algorithm.
We hit about 25%
below the baseline so it's okay.
Keep on.
Another thing is that late conversions
or late feedback equals late results.
Don't try to rush into conclusions.
And in addition,
a work is not done once deployment is done.
Actually, once we deployed to full scale,
we got great feedbacks from stage engineers
to account executives,
which made our model even better.
And post-deployment tools are crucial,
especially if other teams are supporting your models
like sales engineers, or customer support.
The thing that we decided to do is me and Daniel
prepare a notebook that goes Marshal investigation tool.
We hold all of the code
and then we just recalled the session
with the relevant engineers.
We showed them step by step how to use the tool.
And then we released the tool so everyone can clone it
and use the same base for analysis
which makes lives easier.
So, a quick summary.
We went through Fyber overview
and offer wall product overview.
After walls, we talked about our use case.
Our solution.
AB testing and analysis.
And we shared some insights from our own experience.
Thank you very much for listening.
We really appreciate it.
- [Daniel] Thank you very much.