(upbeat music)
(upbeat music)

Hi, everybody! My name’s Patrick Martin,
Hi, everybody! My name’s Patrick Martin,

and I’m gonna give you a brief rundown of recent Firebase updates
and I’m gonna give you a brief rundown of recent Firebase updates

designed to make your LiveOps lives easier.
designed to make your LiveOps lives easier.

Now, I probably don’t need to tell y’all this,
Now, I probably don’t need to tell y’all this,

but mobile gaming has come a long way since the early days.
but mobile gaming has come a long way since the early days.

Initially, folks just tried to replicate the PC and console business model:
Initially, folks just tried to replicate the PC and console business model:

build a game, sell it, and move on, making your next hit.
build a game, sell it, and move on, making your next hit.

Repeat, as you build up a catalog.
Repeat, as you build up a catalog.

Now, many folks tend to build games that are free to start
Now, many folks tend to build games that are free to start

Driving revenue through post-install monetization.
Driving revenue through post-install monetization.

This also means that it’s often worth it to maintain and grow a game,
This also means that it’s often worth it to maintain and grow a game,

developing your current community
developing your current community

rather than searching for a new one every year or two.
rather than searching for a new one every year or two.

This is what I mean by LiveOps.
This is what I mean by LiveOps.

The continued investment you put into operating a live game
The continued investment you put into operating a live game

to keep it fun, stable, and profitable.
to keep it fun, stable, and profitable.

So, Firebase has had LiveOps tools for years.
So, Firebase has had LiveOps tools for years.

A great example is Crashlytics.
A great example is Crashlytics.

It may not be the first thing that comes to mind when many folks think of LiveOps,
It may not be the first thing that comes to mind when many folks think of LiveOps,

but it’s probably one of the most important tools for running a live game.
but it’s probably one of the most important tools for running a live game.

By that I mean, stability drives up game revenue and retention.
By that I mean, stability drives up game revenue and retention.

This should be intuitive, but if a game crashes on launch,
This should be intuitive, but if a game crashes on launch,

you've probably lost that customer forever.
you've probably lost that customer forever.

Once your game, or worse, your brand is associated with being buggy,
Once your game, or worse, your brand is associated with being buggy,

it becomes part of the zeitgeist.
it becomes part of the zeitgeist.

Gamers will start to say things like,
Gamers will start to say things like,

“That game looks cool, but that publisher is always buggy,”
“That game looks cool, but that publisher is always buggy,”

or "Yeah, I tried that game on release, but it kept crashing.
or "Yeah, I tried that game on release, but it kept crashing.

I probably wouldn't recommend checking it out."
I probably wouldn't recommend checking it out."

And don’t forget that if you used fixed interval incentives,
And don’t forget that if you used fixed interval incentives,

such as daily rewards or quests for checking in,
such as daily rewards or quests for checking in,

an unexpected crash is the worst way to break the cycle.
an unexpected crash is the worst way to break the cycle.

This is a long way of saying that from the very first moment
This is a long way of saying that from the very first moment

your player experiences a crash in your game,
your player experiences a crash in your game,

you want it to be reported, you want to triage it,
you want it to be reported, you want to triage it,

and when you have a fix, you want to know that it worked.
and when you have a fix, you want to know that it worked.

You also want to be absolutely sure
You also want to be absolutely sure

that by the end of your beta or soft launch period
that by the end of your beta or soft launch period

you have everything running as smoothly as possible
you have everything running as smoothly as possible

to avoid any day one bad reviews.
to avoid any day one bad reviews.

Crashlytics is a free integration with any Firebase product
Crashlytics is a free integration with any Firebase product

and can track crashes from the field amongst your actual live player base.
and can track crashes from the field amongst your actual live player base.

It can aggregate crash reports, provide associated device metrics,
It can aggregate crash reports, provide associated device metrics,

and give you some baseline tools for figuring out
and give you some baseline tools for figuring out

what was happening at the time of the crash.
what was happening at the time of the crash.

And it's gotten much better in recent months.
And it's gotten much better in recent months.

For example, we have worked hard to make Crashlytics
For example, we have worked hard to make Crashlytics

one of the fastest and most reliable crash aggregation systems
one of the fastest and most reliable crash aggregation systems

for mobile game developers.
for mobile game developers.

What does this mean?
What does this mean?

Let me give you a high-level overview of the changes,
Let me give you a high-level overview of the changes,

but check out the complexity of NDK crash reporting
but check out the complexity of NDK crash reporting

for a full breakdown of how we achieve this.
for a full breakdown of how we achieve this.

I want to start with some features for C++ developers.
I want to start with some features for C++ developers.

The number one thing a C++ developer will want to know
The number one thing a C++ developer will want to know

is whether this stack trace is from a release build.
is whether this stack trace is from a release build.

And there's a good reason for this.
And there's a good reason for this.

Compiler optimizations make every stack trace questionable.
Compiler optimizations make every stack trace questionable.

I’ve personally hit step next on a debugger
I’ve personally hit step next on a debugger

and watch the debug indicator go back up a line rather than down to the next.
and watch the debug indicator go back up a line rather than down to the next.

I even had a C++ compiler put a divide by zero before the if statement
I even had a C++ compiler put a divide by zero before the if statement

that checked if a value was zero.
that checked if a value was zero.

Nothing really makes sense here.
Nothing really makes sense here.

So Crashlytics has gotten really clever.
So Crashlytics has gotten really clever.

It came up with a list of fun error cases
It came up with a list of fun error cases

such as templated code, constexpr functions, and inlined constructors.
such as templated code, constexpr functions, and inlined constructors.

And in all of these cases,
And in all of these cases,

Crashlytics is able to figure out which line is breaking and symbolicate it.
Crashlytics is able to figure out which line is breaking and symbolicate it.

Take this function that’s breaking in a bunch of functional code.
Take this function that’s breaking in a bunch of functional code.

It's parsing all these wacky templates, giving me accurate line numbers,
It's parsing all these wacky templates, giving me accurate line numbers,

and dropping me right into this fast divide function that broke.
and dropping me right into this fast divide function that broke.

Just for this reason alone,
Just for this reason alone,

I would add Crashlytics to my C++ project as a backup debugger
I would add Crashlytics to my C++ project as a backup debugger

and as a way to capture bug reports from my QA team
and as a way to capture bug reports from my QA team

running without being attached to LLDB.
running without being attached to LLDB.

The rules around where to place symbols have also been relaxed.
The rules around where to place symbols have also been relaxed.

So if you have a custom build system
So if you have a custom build system

or are integrating third-party pre-compiled libraries,
or are integrating third-party pre-compiled libraries,

you don't have to be super specific with your symbol file locations anymore.
you don't have to be super specific with your symbol file locations anymore.

Now I can go on about C++ for days,
Now I can go on about C++ for days,

but I know a lot of you are using the Unity game engine,
but I know a lot of you are using the Unity game engine,

so we have some Unity-specific improvements as well.
so we have some Unity-specific improvements as well.

First, I’d like to highlight that crashes aren’t always simple coding issues.
First, I’d like to highlight that crashes aren’t always simple coding issues.

Games go through a lot of QA, testing, and often soft launches.
Games go through a lot of QA, testing, and often soft launches.

The straightforward crashes have hopefully already been eliminated,
The straightforward crashes have hopefully already been eliminated,

but games tend to run up against the limits of users’ hardware.
but games tend to run up against the limits of users’ hardware.

Whether you’re accidentally relying on an OpenGL extension
Whether you’re accidentally relying on an OpenGL extension

that’s not actually in the base spec,
that’s not actually in the base spec,

or are you using an unsupported compressed texture format,
or are you using an unsupported compressed texture format,

or simply are using too much RAM for typical devices in a region.
or simply are using too much RAM for typical devices in a region.

I’ve seen some of these in simple games
I’ve seen some of these in simple games

where I was never worried about dropping below my minimum frame rate.
where I was never worried about dropping below my minimum frame rate.

These are the crashes that have you saying,
These are the crashes that have you saying,

“Why is that broken?” or “That shouldn’t happen.”
“Why is that broken?” or “That shouldn’t happen.”

So, you might run to your local phone dealership
So, you might run to your local phone dealership

and pick up the phone, or phones, you see in a crash report.
and pick up the phone, or phones, you see in a crash report.

In some cases, maybe even your favorite secondhand electronics website.
In some cases, maybe even your favorite secondhand electronics website.

But these bugs aren't done with you yet.
But these bugs aren't done with you yet.

Some manufacturers even keep model names common in each market they sell in
Some manufacturers even keep model names common in each market they sell in

but will swap out the CPU or GPU based on regional needs.
but will swap out the CPU or GPU based on regional needs.

It can really mess with you if you see where a crash is happening in the wild
It can really mess with you if you see where a crash is happening in the wild

with a specific phone but no amount of QA effort ever gets it to happen in-house
with a specific phone but no amount of QA effort ever gets it to happen in-house

on what you think is identical hardware.
on what you think is identical hardware.

So, Crashlytics has really drilled into game-specific use cases
So, Crashlytics has really drilled into game-specific use cases

and evaluated developers’ needs.
and evaluated developers’ needs.

With the latest SDK for Unity,
With the latest SDK for Unity,

detailed hardware metrics are sent up with every crash report.
detailed hardware metrics are sent up with every crash report.

This includes the CPU and GPU your players have, driver versions, VRAM,
This includes the CPU and GPU your players have, driver versions, VRAM,

and anything else that could help narrow in on the issue.
and anything else that could help narrow in on the issue.

This results in less time spent on reproducing the issue
This results in less time spent on reproducing the issue

and more time fixing and verifying it.
and more time fixing and verifying it.

That isn’t the only Unity-specific fix.
That isn’t the only Unity-specific fix.

We found that often a crash occurs in common engine code
We found that often a crash occurs in common engine code

even if it originates in user logic.
even if it originates in user logic.

This means that you might have several different issues
This means that you might have several different issues

in individual MonoBehaviours
in individual MonoBehaviours

but Crashlytics would group all of these into one bug report
but Crashlytics would group all of these into one bug report

since they all ended in the same bit of engine logic,
since they all ended in the same bit of engine logic,

leading to a frustrating game of bug Whack-a-Mole with your live players.
leading to a frustrating game of bug Whack-a-Mole with your live players.

In the worst case, a customer had literally all of their bugs listed on one report.
In the worst case, a customer had literally all of their bugs listed on one report.

And I know that this screenshot might be a little confusing,
And I know that this screenshot might be a little confusing,

but notice that the crash is in Unity’s native store provider.
but notice that the crash is in Unity’s native store provider.

This probably isn't your actual bug since it's in the Unity engine.
This probably isn't your actual bug since it's in the Unity engine.

So the Crashlytics team has been studying Unity stack traces
So the Crashlytics team has been studying Unity stack traces

and generated custom heuristics for Unity based workflows.
and generated custom heuristics for Unity based workflows.

This includes knowledge of Unity engine stackframes
This includes knowledge of Unity engine stackframes

so they can be excluded either above or below your game logic.
so they can be excluded either above or below your game logic.

This means that you'll see more crashes than before
This means that you'll see more crashes than before

but each issue in Crashlytics is actually a different issue
but each issue in Crashlytics is actually a different issue

that you can triage and resolve.
that you can triage and resolve.

In this case, it was smart enough to figure out that the user code
In this case, it was smart enough to figure out that the user code

was the BingoBlast StoreManager and flagged that appropriately.
was the BingoBlast StoreManager and flagged that appropriately.

And with all of this, you’re still getting real bugs
And with all of this, you’re still getting real bugs

from your players in the field as they happen.
from your players in the field as they happen.

Combined with existing features like velocity alerts,
Combined with existing features like velocity alerts,

you’ll know about and hopefully fix issues
you’ll know about and hopefully fix issues

before those dreaded one-star reviews really start rolling in.
before those dreaded one-star reviews really start rolling in.

So I know that crash reporting
So I know that crash reporting

isn’t always what folks think of when they think LiveOps.
isn’t always what folks think of when they think LiveOps.

What you really want to do is monitor your player behavior
What you really want to do is monitor your player behavior

and react as rapidly as possible to your players’ needs.
and react as rapidly as possible to your players’ needs.

Once you have a stable game,
Once you have a stable game,

you want to iterate on your game’s unique feature set.
you want to iterate on your game’s unique feature set.

In my first mobile games, I would just compile a binary
In my first mobile games, I would just compile a binary

and throw it over the fence to the various app stores.
and throw it over the fence to the various app stores.

If I wanted to try out a new feature, I'd literally cook a new build, launch it,
If I wanted to try out a new feature, I'd literally cook a new build, launch it,

and just watch if my app store rating went up or down.
and just watch if my app store rating went up or down.

With app review times, this really took a while.
With app review times, this really took a while.

And when you messed it up, it was really hard to roll back changes quickly.
And when you messed it up, it was really hard to roll back changes quickly.

It was sometimes better to just abandon the game
It was sometimes better to just abandon the game

than deal with all this iteration.
than deal with all this iteration.

Now you might be saying,
Now you might be saying,

“but I already used Firebase Remote Config to test out new changes."
“but I already used Firebase Remote Config to test out new changes."

Which is a great start.
Which is a great start.

If you don’t know what this is,
If you don’t know what this is,

you can categorize players using audiences in Google Analytics
you can categorize players using audiences in Google Analytics

and deliver specific configurations to some or all of the members in a group.
and deliver specific configurations to some or all of the members in a group.

These changes are efficiently synchronized over the air to live games
These changes are efficiently synchronized over the air to live games

as small JSON configuration flags.
as small JSON configuration flags.

Client SDKs already handle all of the hard work
Client SDKs already handle all of the hard work

so you can just define what you want to configure.
so you can just define what you want to configure.

You might also already use Firebase A/B testing.
You might also already use Firebase A/B testing.

This lets you measure competing configurations among similar groups of users.
This lets you measure competing configurations among similar groups of users.

It’s a way to see what your player community is tolerant of with respect to monetization
It’s a way to see what your player community is tolerant of with respect to monetization

or to experiment with new features
or to experiment with new features

or configurations within a subset of your players.
or configurations within a subset of your players.

This stack is great.
This stack is great.

If you have a user experience or a game design team
If you have a user experience or a game design team

looking for some optimal layout
looking for some optimal layout

or trying to gauge the direction to go for your next update,
or trying to gauge the direction to go for your next update,

you’ll want to run an experiment and test the results,
you’ll want to run an experiment and test the results,

eventually promoting the best performing configuration to production.
eventually promoting the best performing configuration to production.

And all of this can occur between app store releases
And all of this can occur between app store releases

so you can iterate on your time.
so you can iterate on your time.

But some decisions are personal.
But some decisions are personal.

Maybe you want to feature different in-app purchases per player type,
Maybe you want to feature different in-app purchases per player type,

dynamically tune difficulty,
dynamically tune difficulty,

or just think different folks would like different color schemes.
or just think different folks would like different color schemes.

These are all decisions that come down to a coin flip even after user testing.
These are all decisions that come down to a coin flip even after user testing.

So we have Firebase Personalization.
So we have Firebase Personalization.

This builds off of remote config and A/B testing
This builds off of remote config and A/B testing

but leverages Google's AI expertise.
but leverages Google's AI expertise.

Where A/B testing alone might find the best experience for a group,
Where A/B testing alone might find the best experience for a group,

Personalization can build a customized experience for every player in your game.
Personalization can build a customized experience for every player in your game.

To start, you define a remote config parameter to control
To start, you define a remote config parameter to control

and then event or events to optimize for.
and then event or events to optimize for.

Events and parameters can be anything you think is important.
Events and parameters can be anything you think is important.

Maybe you want to change the layout of your store,
Maybe you want to change the layout of your store,

or the frequency at which you show ads to drive up revenue.
or the frequency at which you show ads to drive up revenue.

You could tune games difficulty, measuring session length or retention,
You could tune games difficulty, measuring session length or retention,

or even when to show a rating dialogue
or even when to show a rating dialogue

as measured by how often a player actually opts to provide a rating,
as measured by how often a player actually opts to provide a rating,

as shown here.
as shown here.

In this case, I have an experiment that tries to decide
In this case, I have an experiment that tries to decide

how many games I should wait for a player to complete
how many games I should wait for a player to complete

before showing the “Please rate our game” screen
before showing the “Please rate our game” screen

in order to get the player to actually leave a rating.
in order to get the player to actually leave a rating.

In this case one, four or eight games.
In this case one, four or eight games.

Once you start training a personalization,
Once you start training a personalization,

depending on the player base and how well tuned your configurations are
depending on the player base and how well tuned your configurations are

will take varying amounts of time to properly kick in.
will take varying amounts of time to properly kick in.

But you can often expect good results after 14 days.
But you can often expect good results after 14 days.

And remember that the AI model will constantly group and regroup your players,
And remember that the AI model will constantly group and regroup your players,

run experiments among small subsets of them, and tune the model automatically.
run experiments among small subsets of them, and tune the model automatically.

This means that your game never stops optimizing
This means that your game never stops optimizing

as your players’ preferences evolve.
as your players’ preferences evolve.

In a way, this becomes the most live version of LiveOps you could imagine.
In a way, this becomes the most live version of LiveOps you could imagine.

An AI monitoring your game 24/7,
An AI monitoring your game 24/7,

and ensuring that your game is always being the best version of itself for your players
and ensuring that your game is always being the best version of itself for your players

even if your players have different personal preferences.
even if your players have different personal preferences.

Remember, this is a tool designed to augment your team’s ability to test new features
Remember, this is a tool designed to augment your team’s ability to test new features

and react to a changing market, not to take away control.
and react to a changing market, not to take away control.

So, at any point, you can check in on your personalization,
So, at any point, you can check in on your personalization,

swap out configurations that don't work, add new configurations to test,
swap out configurations that don't work, add new configurations to test,

or just make a config the default if it’s taken a clear lead.
or just make a config the default if it’s taken a clear lead.

If you’d like to know more, check out
If you’d like to know more, check out

“Personalizing your app in real-time - powered by Firebase and ML”
“Personalizing your app in real-time - powered by Firebase and ML”

also in the Google for Game Developers Summit.
also in the Google for Game Developers Summit.

This’ll give you everything you need to know about how and where to use personalization.
This’ll give you everything you need to know about how and where to use personalization.

To wrap up, since last year we’ve made big improvements to Crashlytics.
To wrap up, since last year we’ve made big improvements to Crashlytics.

We capture more crashes than ever before with more useful data
We capture more crashes than ever before with more useful data

to help you get to the bottom of your issues wherever and whenever they occur.
to help you get to the bottom of your issues wherever and whenever they occur.

Then we revamped our LiveOps offering,
Then we revamped our LiveOps offering,

from which you can control and deploy game configurations
from which you can control and deploy game configurations

to portions of your player base,
to portions of your player base,

allowing you to react in real-time around the clock
allowing you to react in real-time around the clock

so your game can be the best version of itself.
so your game can be the best version of itself.

Remember that this was a brief overview of the new features.
Remember that this was a brief overview of the new features.

If these sound interesting to you,
If these sound interesting to you,

you should check out “The complexity of NDK Crash Reporting”
you should check out “The complexity of NDK Crash Reporting”

and “Personalizing your app in real-time - powered by Firebase and ML”.
and “Personalizing your app in real-time - powered by Firebase and ML”.

I hope y’all enjoy the rest of the Google for Game Developers Summit.
I hope y’all enjoy the rest of the Google for Game Developers Summit.

This is a great time to really get to know what Google is doing for game developers.
This is a great time to really get to know what Google is doing for game developers.

And you can always find me @pux0r3 on the Twitter.
And you can always find me @pux0r3 on the Twitter.

So long, and have a wonderful time.
So long, and have a wonderful time.

(upbeat music)
(upbeat music)