[MUSIC PLAYING]
OLIVIER VAN GOETHEM: Hey, everyone.
Thanks for joining to listen to our talk,
which will be about how to analyze
gaming data using Google Cloud.
I will cover the theory behind some of the Google Cloud
products and services.
And I'll then hand over to Luke from Improbable, who'll
talk about what they've built in terms of real world data
analytics pipelines.
This solution that they built has also been open sourced,
so you can go check it out afterwards
In an era of free-to-play and online multiplayer games,
data has become crucial to understanding and improving
your game.
The ability to collect, analyze, and derive insights from data
has effects across the entire value chain of your game.
And this is everything from being
able to create better game levels through play
testing using AIs that have been trained
on the behavior of real world gamers
to improving monetization outcomes through better
customer segmentation and tailored offers.
Organizations also want to be able to share
data across the different units from the game developers
to the marketing teams.
And this needs to be done in a secure manner.
Data is typically stored in a data warehouse.
However, talking to various teams in the industry,
we've discovered that they struggle
to achieve their goals of analyzing the gaming events
and data that you collect.
This is mainly due to challenges around the scalability
and cost of legacy data warehouse solutions.
For example, those solutions struggle
to handle the exponential growth of data
that we're seeing in the industry.
It's also more and more difficult to integrate
between the various external sources that you pull from.
And with more and more games moving to real time telemetry
feeds, being able to analyze, ingest, and process those
becomes even harder.
So really the question is, is there a platform out there
that can scale to the needs of your game?
Hey, everyone.
My name is Olivier Van Goethem.
And I work a Google Cloud as a customer engineer.
We'll now dig a bit deeper into the topic of data analytics
and look at one of our key products in this area.
And that product is BigQuery, our Cloud native data
warehouse.
BigQuery is fully managed and serverless and offers
everything that you'd expect from a data warehouse.
It also has key unique capabilities,
such as the ability to ingest and analyze real time streaming
data.
BigQuery also has a high speed in-memory cache,
which we call BI Engine for faster reporting and analysis
using tools such as Tableau.
Security is also very important for us and our customers.
That is why all data is encrypted at rest by default.
However, one of the most exciting capabilities
of BigQuery is its built-in machine learning training
and prediction capabilities.
Let's have a look at those.
So BigQuery ML enables data analysts to build
and deploy machine learning models on massive data
sets directly inside BigQuery using just the power of SQL.
BigQuery ML has the capability of addressing various use
cases.
For example, you're able to segment customers using
K-means clustering models.
You can determine a player's lifetime value
using linear regression.
And you can definitely create more funky complex use cases
by training external TensorFlow models somewhere and directly
importing that into BigQuery and making use of it
to predict using BigQuery ML and SQL.
So let's have a look at it in practice.
So let's imagine a table in BigQuery
that contains game of purchase and playing time data.
Here we've aggregated these data points
to have a single row per user ID that
contains values for their total number of game purchases
and total hours of playing time.
We've been asked by the marketing team
to segment these users, so that we can identify
our most valuable customers.
To do this, I can use BigQuery ML's K-means clustering
model, all done, again, using just a simple SQL statement.
I configure the model to segment our user base
into four categories.
And then I pass in the data from the table
that I was showing previously.
Once the model has been trained, I
can check the model evaluation to validate its output.
BigQuery ML automatically splits the data
we feed into the training step, such
that 80% is used to train our model
and 20% is used to test it.
Here, we can see that we've identified
the key characteristics for our four segments.
So for example, we're going to have our first group.
They purchase a few games, but play a lot of them.
We see a group of people who purchase a lot of games
and then play a lot of those games.
Then we have a group who just dabble in video games,
purchase very few games and also play them very little.
And then the final group, which is the one that I probably
belong in, which is I buy a lot of games during sales
and never play them at all.
BigQuery is only one piece of Google Cloud's smart analytics
platform.
Google's platform enables you to build much broader data
pipelines by connecting, analyzing,
and visualizing your data across your applications and systems.
This means we bring you the best of Google in our solutions.
And we can help you create a more
innovative and collaborative culture in your organization.
A Google Cloud Analytics platform
can also be combined with compute services
to build out a full-blown gaming architecture.
For example, you can use App Engine and Firestore
to manage user profiles and handle log-ins and log-outs.
You can also use Kubernetes Engine to host a matchmaking
service and your game servers.
And finally you can, from those systems above,
send all of the events and telemetry directly
into BigQuery for analysis.
As I mentioned, for a real world example,
I'm now going to hand over to Loek,
who will talk about what they are doing at Improbable.
LOEK LANDZAAT: Hi, guys.
So as Olivier said, I'm Loek.
I'm a data scientist at Improbable.
So first, a little bit about us, who we are, and what we do.
We provide better ways to make multiplayer games
and help multiplayer developers meet any challenge.
So how do we do that?
We've got several services, some of which
include, for instance, managed hosting and orchestration
of the game servers.
We also have networking products that
could allow you to combine multiple game engines,
so you can do more than what a single engine can currently do.
We've got online services, such as analytics.
And I'm going to be talking about that one
in particular today.
And then we still have other development tools, as well as
everything from advice and support
to full code development of your game.
Finally, we now also have three internal game studios
that are building multiplayer games using the services
and products that I just outlined.
And our first internal game called
"Scavengers" is going to be out later this year.
So keep an eye out for that one.
So the Analytics Pipeline, what is it?
And why did we build it?
While out in the field talking to game studios,
we kind of noticed that there wasn't
an easy way for them to quickly get going with analytics.
So we kind of set out to solve that
by providing a reference architecture that
is just like deploying your own cloud model,
so you could quickly could get going with analytics.
And one of the key tenants that we
kept in the back of our heads was
that we wanted to give studios full access of their own data
without any form of lock-in.
So the key principle is you just deploying
this in your own cloud.
All the data comes into your own cloud,
hosting your own kind of like blob storage.
And as of that point, you can then
decide to do whatever you want with it, which
could be, for instance, using an off-the-shelf live ops or game
analytics provider.
But you still have the flexibility
then to un-live data, also utilize any of the ML or AI
tools provided by Google Cloud in this case.
So why Google Cloud?
They provide highly scalable analytics.
For instance, you can process millions of events per second
with Pub/Sub and Dataflow.
They're very easy to use.
So at Improbable, we're a big user of Kubernetes.
And you kind of have this first classic Kubernetes experience
when developing on Google Cloud.
They're also fully supported by Terraform, which is our DevOps
templating language of choice.
And they've got great documentation
for all the bricks that they provide.
Also with example code snippets in various languages.
So if you kind of want to prototype something quickly,
you can just copy, paste those, get it to work.
And then if you think there's something there,
you can afterwards then make the code your own.
And then in particular for analytics,
they've got a great stack of AI and machine learning tools.
So like Olivier just outlined before--
BigQuery ML.
But if you have many models that you want to train and deploy
in production, you could use the AI platform for that,
for instance.
So ingesting data using the analytics pipeline,
the big picture is as follows.
On the left-hand side, we have things that emit events.
They could be PCs, consoles, mobile devices,
game events from other sources, systems, or whatnot.
Those post or send all their events
to the analytics pipeline.
And the goal of the analytics pipeline
is just to capture and persist and organize all of that data
in your own cloud hosting.
And that then kind of makes it ready to be
used by any other service in the future, such as Cloud Dataprep,
to prepare for machine learning models, BigQuery ML, Cloud
AutoML, et cetera, or just to analyze that data with BigQuery
and feed it into reporting systems like Tableau, Looker,
et cetera.
So the architecture starts with events.
They're JSON events that are posted to a REST API.
That is a Python 3 application adhering
to the flash programming model.
It is deployed on Kubernetes Engine.
So that means that, if the influx of events
kind of enhances, Kubernetes can auto scale
copies of that application to however much it needs
to meet those requirements.
So it's horizontally scalable.
When those events arrive, it kind of
augments them a little bit between the received timestamp
and then writes a new line delimited JSON in blob storage.
So once all your data is neatly aggregated in that blob storage
bucket, you're kind of done.
Because you can then use BigQuery to actually query data
that's inside of that bucket in a federated way.
Federated means that the data doesn't
live inside of BigQuery, but somewhere else.
But you can still use SQL statements
on top of data that is in new line delimited
JSON in Cloud Storage.
Over time, you might notice that, OK, I actually
have a lot of events coming in.
Each post request becomes a file in Cloud storage.
So my queries are slowing down, because I have so many files.
So the first way you can solve for that
is by using a cloud function that basically gets invoked.
Whenever a file hits this bucket,
the function gets invoked.
It takes the contents of that file from blob storage
and then writes into native BigQuery storage.
So at that point, if you then query those tables that
are in native storage, all your queries kind of
benefit from the native performance of BigQuery.
So everything will be super again.
Another way, which isn't in the open source repo,
but we've added ourselves when load testing this,
is add in a batch job.
So if you use Dataflow for this, that comes in every hour
and picks up all the latest small files in the first bucket
and then aggregates them into a second final resting place,
where we then have far fewer, but much larger, files.
And if you then query those in a federated way with BigQuery,
everything will become much bigger again.
So we have successfully scaled this infrastructure
to 10,000 requests per second at an average of one
server or client request 30 seconds, which is something
that you can tweak yourself based on your capacity and peak
CCU.
This would support a peak CCU of about 300,000.
We also tested this with an average payload
of 10 events per request.
So this would equal 8.6 billion events per day.
So there's two caveats when using this at scale.
One is you probably really want to use a similar batching
setup as we have.
And the second is, in order to further optimize this,
we add a caching error at the end point,
so that not each post request becomes a file.
But every 10 seconds, it flushes out
all the post requests in a single file and Cloud storage.
So what are some of the strengths of this setup?
So as I outlined, it's designed in a modular way
to handle evolving business needs.
It's super simple to start.
You just start with an end point,
and then write into Cloud Storage.
Then if your demand overtime enhances,
because you run more play tests or you have more players,
you could, for instance, use the Cloud Function
to hop all of those events over into native BigQuery storage,
or you can add a batching step like we
did to concatenate these files into fewer, but larger, ones.
You have a flexible and custom REST endpoint
as an entry point, which is set up to handle,
for instance, any event, schema, or source.
So what we've done is we've got a PlayFab event schema
parser and just one for our own event schema.
And as events come in, it just figures out
which parser to use and then write everything
in the same format in cloud storage.
And we also preserve flexibility where possible.
So we use new line delimited JSON and blob storage.
That's really the key.
And we've got Kubernetes and Docker, for instance,
utilized where possible.
We've used a single language, Python 3,
which is very popular within the data community.
It will also help.
If you've got data analysts that are kind of engineering savvy,
they can also then look at the pipeline code
and understand what's going on.
So that's great.
It's affordable.
Because this is a deploying your own cloud kind of delivery
model, you avoid any margin that comes
with an off-the-shelf solution.
So what are some of the opportunities
that we still have?
We could reduce the maintenance overhead
by deploying the end point, for instance,
serverless on Knative or Cloud Run.
This way you don't have to manage your own Kubernetes
cluster and have the overhead associated with that.
You could also use the end point to forward events
to a messaging queue, for instance.
So you add a method.
So instead of a v1 event, which goes into Cloud Storage,
you do v1 stream.
And it goes either to Kafka or Pub/Sub.
And then you can use that as a source
for, for instance, a Dataflow streaming
job that can keep a rolling count of, say, kills.
And if that spikes or surges, then you
can trigger a blood moon in your world.
So that kind of then enables this interplay
between what the players do in your ecosystem
and how the ecosystem responds to that on a higher level.
So what are some of the weaknesses of this setup?
It's that it does require a data engineering
capacity to maintain.
There is some complexity.
This is not a fully managed service.
The end point will definitely benefit
if you would port it over to a language like Go,
but at the cost then, again, of interpretability for some
of the data folks out there.
At high scale, you will have a large number
of small files in cloud storage that
will then disable the live external storage functionality.
So at that point, you should either use a Cloud Function
at the batching step or at the caching layer
in order to solve for that.
So threats, we've got an ever-evolving data landscape.
So with new things that come out,
those might challenge some decisions
that we've made with this infrastructure.
So what are some of the lessons learned?
One of the things that we have right
now is our own kind of event schema.
So on the wire when you send these JSON events,
what is inside of each event?
I'm not going to list all of these out one by one.
You can look at the slides later and figure out what each means.
But one thing that I wanted to explain
is that we've kind of identified these
as the only keys that can be at the top level
of your dictionary.
Anything else would go into event attributes.
So for instance, if it's a match start event,
the match ID would sit inside of event attributes.
But by always having only these kind of keys at the root level,
you can safely unpack these, for instance,
in your first table in a data warehouse
when you ingest that somewhere.
And that will dramatically kind of like
simplify the ingestion frameworks
that you will have to use if you want to move that data, ongoing
in the data warehouse.
So about instrumentation, automatically
enrich incoming events where possible.
So capture things like the event time stamp, the event index all
automatically, only requiring a developer
to pass in what is unique to that particular event.
Batch your events.
Don't send single events.
If you can, always batch them up.
And then at what rate you can flush those out
is informed by, on the one hand, the capacity you've provisioned
for in terms of requests per second
versus the number of peaks you seem to expect.
You can kind of model out, OK, well, then I
can at most flush x amount of time per seconds.
Keep a local cache on the client side
in case of connectivity disruptions,
then write them to a file.
And then whenever a connection is re-established,
you can flush everything out.
Make your ingestion end point configurable.
So for instance, which host, headers, or query strings
to use for this particular post request?
And in particular, the end point reads the query parameters
of many of them.
And they kind of co-determine the ultimate file path
that the file's stored in in the Cloud Storage.
So you can then basically already organize and filter out
certain event types and put them in neat constructed folders
in Cloud Storage.
This can have beneficial downstream effects.
Because then I could write, for instance, all
my frames per second events to a particular directory
only containing such events.
And then I can aggregate those before I write them
into BigQuery-- for instance, in a [INAUDIBLE] table.
So we're using HTTP end points.
So it's always good to have retries with exponential
back off.
If you don't get a 200 code back,
just wait a particular period and then try again.
Some of the tips and tricks that we picked up on as well
when actually utilizing this data
is that you can import a Google spreadsheet as a live BigQuery
table.
Live-- meaning, if I make changes to the spreadsheet,
they're instantly visible in BigQuery as well.
And that can be great, for instance,
if you've got item ID to item name mapping tables.
Then vice versa, you can also run BigQuery queries
within Google Spreadsheets.
And a result of that becomes a spreadsheet as well.
So this is a great way for, for instance, business analysts
to still use production data in BigQuery,
but in a spreadsheeting system that they are familiar with.
You should also check out the BigQuery JSON function
to extract data out of event attributes.
Don't use any of the string lookup functions.
Just use those functions.
Because then you can just give it a parsing path
and cleanly and exactly parse the JSON in however you want.
Also, avoid non-alphabetic characters
in JSON keys, because that will trip up the federated query
mechanism that I outlined earlier.
So then finally, we use DBT to manage
and source control all of our SQL code
and to document everything as well.
DBT kind of out of the box makes every query a model.
And then you reference other models,
which means the entire lineage graph or lineage
system is preserved with DBT invisible, which
is very nice for maintaining all of these queries.
And also, if you use the schema.yaml file,
you can generate docs just out of the box.
And that will really allow new [INAUDIBLE] analysts
that are just coming in, and they
need to get to know your schema really well.
It will really help them to have that as a resource
to figure out how to run the queries, what query [INAUDIBLE]
tables to look at, et cetera.
Then finally, you should also definitely adopt
the GDPR strategy.
So two examples-- but it's really up to you,
ultimately, how you want to approach this.
But one could be you add a method to the end point,
and then you route all your events
that contain personal identifiable information
only to your data warehouse.
So you avoid blob storage.
Because at that point, all of that data that is sensitive
is in a SQL kind of like ecosystem.
So you can run select star from exports
and delete from for any forget me kind of requests.
Alternatively, if you use the batching setup,
what you could do is you set a retention policy
on the first bucket for four to eight weeks.
And then when the batching stuff comes in,
for instance with Dataflow, you have the data in memory again.
And that's where you can then do any scrubbing or anonymization
of that data before you then move it over
into its final resting place.
But then you can kind of cope with it like that.
So that is everything.
Thanks for watching the video.
I hope you find it informative.
And if you have any questions about it or suggestions,
then, yeah, feel free to go to our website,
find out the forms, leave a question there.
Or find me on LinkedIn and just ping me a message,
and I will get back to you.
[MUSIC PLAYING]