YAN SUN: Hello, everyone.
I'm Yan, product manager for the BigQuery ML.
Today, we will introduce a new feature
called multivariate time series forecasting.
So this feature provide our customer more
accurate and powerful results by incorporating
the covariate, also known as the site features,
beside the target time series.
For example, if we want to forecast the future ice cream
sales, we are not only using the past sales data as our target
time series but also the external covariance,
such as weather, location, et cetera.
In today's session, we invited our customer
Yohan Mathew, who is the senior data scientist from Headlight,
and Google Cloud software engineer Haoming Chen
to join us to provide you a real-life user use case demo
and a product feedback session.
So before I hand over to Yohan and Haoming,
I will provide a little bit background regarding
our customer, Headlight.
Headlight is the next-generation agency.
They are building tools to scale the next generation
of the consumer product.
The team brings together the performance marketing,
the design, and the data science in order
to unlock the long-term growth for the emerging
startups to the global brands.
So the current problem they are looking into
is to determine the aggregated conversion volume
for the downstream matrix, such as subscription
and cancellation.
Then I will pass this session to Yohan for a demo.
Yohan, over to you.
YOHAN MATHEW: Hello.
Today, we are going to be running a multivariate time
series analysis on some marketing data.
This is our data set, cohorts_public_age.
And it contains a number of marketing metrics.
This table has been cohorted by cohort age
or, more specifically, install age.
In this case, install age is the number of days
after a user has installed.
So for example, a cohort age of 0
means the actions that users took on the day they installed.
A cohort age of 101 indicates the actions
that users would have taken 101 days after they've installed.
So what we've done is we've averaged all these values
while grouping by cohort age.
We also have a cohort_date_pseudo metric.
Now, as we can see cohort age is an integer value.
But time series models generally take in a timestamp variable
as their time series variable.
So what we've done is we've created a new field
by adding an arbitrary date to the cohort date.
So for example, in this case, we've
taken the arbitrary date as 1 Jan 2020.
And to get this cohort_date_pseudo,
we add the cohort age value to that as number of days.
So for example, cohort age 0 corresponds to 1 Jan 2020
itself because you're adding zero days, whereas cohort
age 54 corresponds to 24 of Feb because you're
adding 54 days post-Jan 2020.
We then also have the eight time-dependent variables that
have been labeled input_metric_01 to 07,
and target metric.
Now, all of these are the variables
that actually change over time.
That is to say that they change as the number of days
passed since a user installed.
We have qualitatively separated these out into input metrics 01
to 07 and target_metric because input_metric_01 to 07 have come
earlier in the user funnel.
And as a result, we have accurate, reliable data
for that, whereas target_metric comes much later
in the user funnel.
And because of that and because of the way
that attribution networks work nowadays,
we very often do not get that data.
Or even if we do, it ends up not being very reliable.
So that's the one that we are trying to predict for any given
future cohort date.
So let's say we have data on how users
behaved in the first seven days or the first 30 days.
We try and predict how they would behave,
let's say, 60 days down the line or a year down the line.
Now, getting into the creation of the time series model,
we use the standard BQML model creation statement.
We specify the model type as ARIMA_PLUS_XREG,
feed in the timestamp column, which
is our cohort_date_pseudo, as well as the metric that we're
trying to predict, which is, in our case, target metric.
The training data that we use is cohort ages D0 to D400.
So we train the model on those 401 days, or 401 rows.
And we run the predictions for 30 days
after that, so D401 to D430.
We see that this ARIMA_PLUS_XREG model is created with features
cohort_age, which is the numerical value of cohort age,
as well as the input_metric_01 to 07.
The timestamp variable is not given here.
And because we specified-- if you see here,
we specified auto_arima=true, so it actually cycles through
different values of P, D, and Q and figures out the optimum
value.
Coming here and running an evaluate
to see how it performed on the test data,
we see that we get a mean squared error of 31
and a mean absolute percentage error of 2.48%.
So our business requirements generally
state that we need a mean absolute percentage
error for predictions to be less than 5%.
So in this case, it does satisfy our business constraints.
What we do now is we also compare it
against how a more basic multivariate linear regression
model would perform.
So in this case, without any time series analysis,
coming here, we quickly run the CREATE statement.
In this case, we specify the model type
as linear regression.
And we pass in the same training data,
which is D0 to D400 values of cohort age,
except the cohort_date_pseudo because that
was an artificial metric that was created purely for the time
series model.
In this case, we will not be needing that.
And then we predict the model for the same D401 to D430.
We see that the linear model has been created with the same set
of features, cohort_age, as well as input_metric_01 to 07.
And running the evaluate, we can see here
that the mean square error is 107,
which is more than three times higher than the one
we got with the time series model.
But more importantly, we can see there's a huge negative r
square, implying that the model has performed really
badly on the test data.
So for example, in this scenario,
the multivariate time series model
would be the only one that we'd be
able to use to actually predict values of the target metric
or, basically, down-funnel metrics a certain number
of days after the user has installed.
Thank you.
HAOMING CHEN: Thanks for demo, Yohan,
I have some questions related to our product.
So the first question I have is, based
on the results, in what area do you think a multivariate time
series will be helpful to you and your company?
|
YOHAN MATHEW: So the BQML multivariate time series model
helps us predict downstream behavior of user cohorts
in a privacy-safe manner.
And it helps us do so early on.
So like I mentioned in the demo, one of the issues
is that these downstream metrics are generally not reliably
captured by attribution output.
So this helps us capture this behavior early on
and gives us additional inputs while reviewing and adjusting
our performance.
The multivariate nature of the time series forecasting
helps us to capture the time dependence
a lot better than if it was just a traditional univariate time
series model.
HAOMING CHEN: I see.
Sounds great.
The second question I have is, anything
you think we should improve further?
YOHAN MATHEW: By and large, we're
quite happy with the BQML implementation
of the multivariate time series modeling, the fact
that they are written in SQL ensures
that they are, in general, easy to integrate.
In an ideal scenario, one thing we might like
is for all steps of creating and deploying these models
to come integrated with orchestration systems
like Vertex AI.
Because we already have an end-to-end analytics
pipeline to schedule and automate most
of our data science workflows.
So having this also be part of an automated pipeline
would be the ideal scenario for us.
HAOMING CHEN: I see.
Thanks.
So the third question I have is, any other forecasting
features you want?
YOHAN MATHEW: I think one nice feature that would maybe
reduce a couple of steps from our implementation
would be if we could use integer values in the time series
variable, as well, rather than just timestamp-based values.
So what this would do is help reduce
some of the additional data wrangling on our side.
HAOMING CHEN: I see.
Thanks for feedback.
So the last question I have is, regarding productivity,
is there any improvement, comparing
with your existing workflow, from
the following perspective-- like how long it takes,
how many resources, and the skill required,
like SQL, Python, R, et cetera?
YOHAN MATHEW: Absolutely.
I can say that it definitely helps in that regard.
Given that this is done in SQL rather than Python,
it helps us distribute a lot of the tasks among both data
scientists and data analysts and helps us be more flexible
in our resourcing.
Like I mentioned, in our analytics pipelines,
a lot of our core frameworks are already in SQL.
So BQML helps us to enhance those data transformation
frameworks to also include predictive modeling,
whereas, earlier, we would have had to implement
that in a Jupyter Notebook.
And then we would need additional MLOps
to handle the orchestration between BigQuery and Jupyter
Notebooks.
This makes the entire pipeline a lot more streamlined
for use cases like this.
HAOMING CHEN: Yeah.
Yeah, I'm so glad to hear that.
Well, thanks again for your time doing this demo.