>> Hi, welcome to GDC.
We are live here again at the Azure Boot today.
And with me today, I have Brian Vanous from PlayFab.
>> Brendan Vanous.
>> Brendan Vanous, I'm so sorry.
>> Okay.
>> Hey, this got off to a great start.
So Brendan, you are from PlayFab.
>> Yes.
>> Tell me a little bit about the history
when you joined PlayFab
and kind of your tour being there.
>> Sure. So, funny thing
is I was actually in Xbox previously so,
I joined in order to help with
the run up of the Xbox 360.
I was part of the ATG team,
the Advanced Technology Group.
I was managing that team for about seven, eight years.
And my main thing was Xbox Live.
That was my main
surface area for working with Developers.
And because of that,
when PlayFab came into being back around January 2014,
they wanted somebody to come over to be
the person running Developer Relations for them.
They need to be somebody deeply technical,
somebody who has past experience
being a game developer or working
with game developers and
it just seemed like the perfect fit for me.
So I joined them.
They were targeting exactly what
I felt needed to happen for the industry, anyway,
which was a Ubiquitous Backend Service,
that's Platform Agnostic,
available to everybody everywhere all the time.
And yeah, we've been having a great time
building it out and of course,
our part of Microsoft again.
>> Yeah, that's right. And I understand too
that you were at Microsoft before, is that right?
>> That's what I was just saying, I mean
part of the Xbox ATG team.
>> Coming back, though after going away,
staying at PlayFab, do you
see any differences or what do you think about it now?
>> Oh, wow. The biggest change is
obviously that games is now a pillar of Microsoft.
I mean, Phil Spencer is now
reporting directly to such an Nadella, obviously.
And it demonstrates a very clear commitment
by Microsoft to games in general and very broadly,
not just games on Xbox or Windows,
but the entirety of the games ecosystem.
And then the acquisition of PlayFab being
that we are a Platform Agnostic on
every platform, it just reinforces that.
>> Yeah, totally.
There's a true commitment there to the gaming,
I mean you see here too,
the Azure Boot, the GDCs,
a lot of activity going on right around.
>> It's huge.
>> Everybody's surrounding.
So it's really good to be here.
PlayFab always talks about what's called "LiveOps".
>> Oh, yeah.
>> It's a sharp turnaround. I think a lot of
people probably are familiar with it.
So, how would you describe LiveOps to somebody?
>> So, "LiveOps" is live game operations.
And it's all the things that you can do to
manage your title once you launch it.
I mean games, it used to be were packaged goods right?
You just create the game,
ship it to market, then move on to your next game.
But these days when you ship a game,
you've got a commitment going forward to your players.
You're going to run that game and manage it,
you're going to monitor it and
make changes and adjustments
based upon the things that you're seeing.
>> Yeah, got you. And so
that concept is what PlayFab does primary, right?
>> Yes.
>> So, today you want to talk about a piece of it.
I know we've got up here on the screen.
The different components of PlayFab.
Today's topic is really around
the real-time analytics side.
>> Yes.
>> So, when we think about the analytics side,
and we think about data and games,
like how has that impacted game development
like just the data collection?
>> Sure. I've got a few minutes left,
so, what this represents
is basically the fundamentals of PlayFab.
We built out a lot of core services,
authentication systems, data systems,
in our purchasing catalog of goods, etc.
But the core then beyond that was
having everything be feeding into
a real-time analytics system
that then becomes this stream of
data that you can use because if you go all the
way back to late 2000-2001,
these are the top grossing games reached here.
And you can see that there was
only one game out of all of these.
>> Yeah I remember that.
>> Made the transition right, The Sims,
2000-2001 it was on both lists.
Every other game is different between the years.
But if you jump forward to just a
couple of years ago 2016,
and then moving into 2017,
70 percent of these games were in both years.
And that's because these games are being
managed as live entities,
they were being managed using good LiveOps techniques.
People in the development teams,
were working with their customers and monitoring them
because when you look at your forums for example,
you can work with your community there.
But realistically, you're only going to be
hearing from about 10 percent of them.
So, monitoring and the things
that they say might not necessarily sync
up to what they're actually expecting
or doing in the game,
because that's just what they're saying.
Monitoring the way that players
are actually interacting with the game,
is the way you get true data
about what you are doing and what you need to do for.
>> Got you. These particular games here,
do all that particular thing.
They monitor in real-time, were updated or?
>> Looking at the list,
I would say, yes.
Each one of these games is managing their,
yes managing their games using LiveOps technique.
>> But most of those, I can see
these are mostly mobile titles.
Is this also applied to like PC and everything else?
>> This applies to absolutely everything now.
But one of the reasons why you
see a lot of mobile titles in
the top grossing games list nowadays is just because
of the overall addressable market space.
There is a lot more mobile phones out there
than they are of just about anything else,
so possibly for PCs.
>> Got you. Okay.
>> But yeah, when you're managing this,
what you're effectively doing is you're
taking all the data coming in.
And you might have some ideas
about what you're seeing and why you're seeing it,
what changes you might want to make.
So you test those ideas out,
check your data, and then you iterate on it.
It's a very cyclical process.
So, as an example of that,
you might have a game where
you get in a bunch of players.
Now, initial acquisition of players,
your day one is not going to be 100 percent, it never is.
So, the day one might
or might not be unusual to look like that.
But you see the level one here,
that's after the players actually
invested some time playing in the game.
>> Right.
>> So, having a drop off that sharp,
would be kind of a warning sign.
So what you want to do is have
some views of the data
where you dig in and you look out and you say,
okay, why am I seeing this?
What might be the things causing this?
And in this particular example we're saying, okay,
there was an achievement early
on of the player defeating a boss.
And what I'm seeing is that a lot of people
who don't get that achievement
are dropping out of the game.
So maybe you might say okay,
well let's make an A/B experiment out of this.
And let's say you've got a control group,
but you're going to have in this case looked up to
four A/B treatment groups and
in each treatment group you're going
to vary the experience.
So in one of them you're going to say okay,
after failing to beat the boss for five times,
let's make it 80 percent strong
and then another one 60 percent strong.
And then the other two it's like okay,
maybe after ten times trying to beat him, I'm failing.
Think of maybe 80 percent strong, 60 percent strong.
>> So, you're doing actually experiment
to two different experiences with two different groups?
>> It's actually five different treatment groups.
>> Five, okay.
>> A control group and
then four variations that you're doing from
the play experience to see which one of those things get
you a better response rate in
terms of improving retention.
>> Right. Okay.
>> Because then, you can go back
then to your data and check it.
And if you compare this to the original slide it showed,
there is a significant improvement to
the retention on that Level One.
It's not everything you might want
but you're seeing some significant improvement.
So what you want to do is look at
the treatment group results and say okay,
it was one of them clearly
a leader ahead of the others in terms of improvement.
>> Got you.
>> Okay then let's collapse the game
into that everybody use that one.
And then start in on maybe
a new experiment trying some other data group.
>> Just got you. Oh, this is sufficient because I think
everyone's experience like some boss
or something is too hard, right?
You never pass, you get frustrated,
you just kind of whip the game.
>> Exactly. Yeah.
>> So you're saying you could actually test to see hey,
did I make it too easy or what's the exact right level?
>> Right.
>> And then can you just update your game like.
>> Yeah.
>> Okay.
>> That is specifically the idea.
So, games are managed life these days.
So, part of thing we do if you look
at all of the components that we have here,
content is certainly one of them.
So content includes things like Title Data.
Title Data is just, it's configuration data,
it lives in the Cloud and so,
it controls a huge range of things.
In a lot of games it might be
like the strength of monsters,
or the speed of fear, or things like that.
And as you tweak and tune these factors,
you're doing that based upon the data that you get in,
to try to improve retention rates,
spend rates, all kinds of factors.
And in particular, what you're really trying to go for,
I mean what it comes down to is
the players are enjoying it.
If the players are enjoying your game,
you are successful.
>> Sure.
>> That's what you're going for.
>> Right.
>> So, because this all lives in the Cloud,
you just make an adjustment to it there.
>> Yep.
>> You don't go through any kind of
certification process because you're not changing code.
You just make a change in
the Cloud immediately your players are
using the new data. And so they have a new experience.
>> Okay. So that's right.
So you're managing the game live
making changes and you can figure out real time.
So, real time sounds
like something a step beyond that even.
Like why is real time even more important in this case.
>> Yeah, because I mean the example that I showed you,
you're evaluating data over time
and looking at it but when you talk about
the data you really really need it to be real time
because a lot of the changes that
you're going to be making are going to be sensitive,
particularly if you're talking about a free to play
game where you have enough purchases.
So when you push out a change to the game,
if you see a sudden sharp drop in
purchases or a sudden sharp increase
in errors because of something you've done.
You don't want to have to wait an hour.
You don't won't have to wait five minutes.
I know about that. You need to know immediately.
So, whenever you make
a major change to your game or maybe even a minor change,
you're going to be sitting there looking at
the dashboard and you
know, refreshing, refreshing, seeing.
Okay. It looks like things are going well or well,
wait, something changed,
you do not like to look up, let's roll back.
>> Got you. Okay. So this
sounds like you have to track a lot of things.
How do you actually do that in a game?
Like how do you track all that stuff going on?
how does it work?
>> Yeah. We provide a number of dashboards.
So, on our main dashboard.
You get all of the essentials in terms of
tracking on API calls, error rates,
the purchases that are being made and you can also dig
into the event history
which is kept up-to-date real time as well.
And you can look at all of the details
of all of events as they're coming out.
So, basically you've a
full real time event system and you've got
a dashboard that gives you the ability to actually
watch it in real time and see,
you know, look for trending, look
for error rates, all of that.
>> Right. So those of,
you're talking about a bunch of events that happened.
Do you guys supply
events or like do I make events? How does that work?
>> Right. All of
those core events that we built up to start,
core services we built up.
So the statistics and data systems,
all of those automatically generated software logs in,
when a new statistic
is reported, when they make a purchase.
All of those things, generate events automatically.
You don't have to worry about any of them.
But, games need to have the ability to
generate custom events because
they do a lot of custom things.
So yeah, we also provide
API calls for you to generate those customer events.
Both of the player level and the title level.
Since there's a significant difference between the two.
But then those fill-in of the systems
which you can use those for your analytics as well.
>> I got you. So you
provide the basics of the system all around.
But say I'm going to have a custom event inside my game,
I can actually just create my own event that feeds
into your venting system?
>> Exactly.
>> Okay, that's very very interesting.
So I can extend the event system to do something.
So, when you do the analysis
over viewing of it like what is the visualization.
How's that work. Like how do you,
how do I see these events.
Is it all text like what's that look like?
>> There's a couple of different ways but
the most convenient way in the game manager,
if you're looking at the overall
event history for the game.
>> And the game manager is your-
>> Oh, I'm sorry, the game manager, yes,
we provide a GUI interface.
Basically, for managing your game
but actually it's everything.
It's visualization of your events.
It's the way that you make changes to your data.
Like I talked about the configuration that you can
make the title data changes right there in the service.
You can manage all your players that got
a complete detailed permission system
with a role model on top of that.
So you could play the role for customer service reps.
So, they can only see and modify players but not
change your configuration for
example or see your revenue information.
And so that way you can give everybody the right tools
that they need right there and
the game manager to manage a game.
You know in the ways that are appropriate to each of
the different roles that you have in your team.
>> Yeah.
>> But yeah, within the tool,
when you talk about data specifically,
you got the full stream where you can
see all the details coming through for all
of the data but you also got a visualizer where you can
see a overall graph of all of the events over time.
You can do summations on different types of events.
You could group them,
that sort of thing, so that you could see,
you could say "Show me just my log-in
events and group them by platform".
You can see the breakdown of
how many people you have
for on each of the platforms, over time.
>> So you can slice and dice that to get
the right view that you need to make a decision.
Can I create a custom view or do you
just use the other box. How does that work?
>> For deep dives on your data,
where you really digging in to get
into custom information about your data.
You want to be visualizing the data directly yourself and
that's something where we provide a number of
different ways to get to the data,
whether it's web hooks or being able
to send the data out to an archive store for you.
But we also provide some integrations with
backend data platforms so you can send
data to and putting segment,
Snowflake and soon to be doing our VIs as well
to give you the frank visualization
capabilities in that platform.
>> Yeah. Visualization sounds
like a very powerful part of this, too, right?
So this is super interesting.
So now I can create a game out there and I
can have a pulse of what's going on with it.
>> Exactly.
>> Right. That's the real plan aspect.
>> And that's what's so critical.
It's being able to see what your game is
doing in real time and respond to it effectively.
>> So you recommend people should change
the way they create games these days must have.
Is this a must have for people?
>> I would say so. Even the game
that is a single player game.
The point of
purchase single player game not a free to play,
no multiplayer nothing like that.
You're still going to want to have
the information about your game.
How players are playing it.
For purposes of making your game better.
And you're certainly going to be shipping
out DLC, new content.
And as you do that,
being able to have
still your configuration in
the cloud gives you the ability to ship out some content.
Look at how it's being played. Look at how
players are responding to it and fix
anything that you want to
without having to push out a whole new update.
>> Yeah. That's super powerful.
Super powerful. If you think about this.
If so you're collecting a lot of
data and things like that.
That data needs to be protected in some ways.
What it's like, what is Microsoft to PlayFab,
where do you guys focus on to keep make sure that data
is remains safe from it feels like that's ah--
>> We've been through pretty extensive security reviews
processes with way back,
even well before the acquisition by Microsoft.
So we got to say, as part of that,
we went through a deep dive security review
which was absolutely fantastic.
And what I can say is that,
our security reviews have
all passed with flying colors at this point effectively,
where the backend for a number of companies that provide.
Well, they provide games to a wide range of players and
they're very sensitive and you've
got considerations of COPPA,
the Children's Online Privacy Protection Act.
You've got the upcoming GDPR,
the General Data Protection Regulation.
>> Yeah, that's a big deal.
>> It's huge. All those things or things that we are
addressing in our backend or I
should say already have addressed
or are continuing to address.
And we're even going beyond to
the regulations in the GDPR for example.
To make sure that we have the highest level
of security around the data possible.
So that we're protecting player privacy,
protecting the data for your titles,
so that only the people within
your team and only the right people
within your team have access to it.
>> That's super important too.
So, this is all fascinating.
Can you give me
other examples of some games you might know,
people that have implemented this or done this.
>> Sure. I mean the best example right now
because they were here with us throughout GDC.
They did a great talk on
Wednesday afternoon, is Fluffy Fairy.
>> Yeah. Interviewed them yesterday.
>> Yeah. The German guys.
They're fantastic and for their game Idle Miner Tycoon.
They did this rapid prototyping
to get the game to the market literally eight weeks.
They spent eight weeks building up the game,
shipped it to iOS and Android and then just rated on it.
Did very organic user acquisition.
It rated on the game using player feedback.
And then after a while
they kicked in all of their life box was using
all the techniques we're talking about throughout
the conference and that they talked about on Wednesday.
They started this overall LiveOps campaign.
Managing the game in real time,
monitoring the behavior of users,
trying different experiments with
changes to the Gameplay.
And they've just been hugely successful.
If you look at
the classic old school metrics that you see
for games where a game would launch
and then drop off and usage and go on to the long tail.
Games these days and then the Fluffy Fairy
guys are the perfect example,
they're game launched and
then when they kicked in the LiveOps,
you see them growing and growing and
growing and still growing that game to this day.
>> Yeah. They say they had like 30 million users
now, that's crazy.
That's great man. I love to see,
I love the game too, by the way.
Oh yeah so far so good.
So probably a good place to kind of wrap up.
What do you guys are thinking when you look at
real time analytics and PlayFab,
where this is going to the future directions.
>> Oh, yeah. I mean
being now part of Microsoft
is just going to be one of the greatest things for this,
because it gives us access to so
many tools that we didn't have before.
Just the integration of power on its
own is going to give a wide range of
customization options and new options
for the ways you can slice and dice
your data and the system.
But also going beyond that,
there's machine learning that
we're going to be able to incorporate into the system
for doing things like automated segmentation
of users taking into account
not just Gameplay information
but let's say you got chat into that.
So we'll have, you know,
maybe real time Speech to Text,
Text to Speech and being able to evaluate that text.
>> Oh, wow!
>> And then being able to automatically
segment the users into appropriate groups.
You are playing with people
who just naturally you're
going to have a better experience with.
>> Yeah. Machine learning,
that's super interesting area like,
when you find machine learning to segmentation.
How would that work? What is that.
How would that work.
>> In machine learning
at its core what it's doing is it's taking in
a massive amount of data and it's looking for you know,
areas of significance within the data.
And then once it identifies as significance within
the data it can then take those and
use them in order to make decisions.
>> Right.
>> So if a decision in this case might be okay,
this is a player who you know,
the way they play
the game as they play in a more casual way.
It's might be a shooter or whatever but they don't
play it like 24 hours a day.
They're not playing for eight hours of time.
No this is somebody who plays
occasionally and when they play,
the pattern of behavior that is used,
if they're playing with people who
are the super hardcore best players,
very short play sessions.
But then, when they play with people
who are following the same pattern as they are,
they no longer play sessions.
Well clearly a longer play session
is kind of indicative of having a harder time.
So, I mean, that's just the most trivial examples.
That kind of thing.
>> Oh that's actually super helpful
because I mean you're using the power like what
a person would naturally figure out
using a machine to kind of figure those things out.
That's super interesting. It feels like to me,
this is an area for a lot of
growth and investment in the future.
It also sounds like it's changed the way game for me.
>> Oh, yes absolutely.
>> That's fantastic.
Well, hey, Brendan.
Got it right this time.
Thank you so much for joining me.
It has been a good day.
Hope you enjoy the rest of
GDC and for everyone out there.
Thank you for tuning in.
And we'll be back in just a little bit
with some more live sessions.
Thank you.