[MUSIC PLAYING]
PRIYANKA VERGADIA: Yep, I am catching some Pokemons.
Yeah, you know, "Pokemon Go" is crazy.
Millions of people play this game, and it scales like that.
Did you know that it was built on Google Cloud?
Well, let's talk to technical manager at Niantic Labs
to take a behind the scenes look at "Pokemon Go"'s architecture,
and how it scales for a large number of players.
Hi, James, welcome to the show.
JAMES PROMPANYA: Hi, thank you for having me.
I'm excited to be here.
PRIYANKA VERGADIA: Well, for those
who may not be familiar with "Pokemon Go,"
let's quickly go over what it is.
JAMES PROMPANYA: So "Pokemon Go" is not
your typical mobile game.
It's a game that involves walking around, catching
these little creatures around you called Pokemon that are
appearing in the real world.
It encourages you to go outside, explore, and discover
things using augmented reality.
A big part of that which wasn't included
in the game when it first came out
was this big community aspect of it.
Players would go out and meet with other players in real life
to play together, battle in gyms,
and even like point it out when there
was this rare, powerful Pokemon that would appear somewhere
in your map.
Everyone sees the same Pokemon.
So you even see this in like YouTube clips out there,
but sometimes somebody would just yell out,
hey, there's a Charizard here, and you
would see crowds of Pokemon running towards that direction.
So nowadays we really like this part of the game.
So we made it a major part by hosting regular live events,
such as community days, raid hours, Team Go Rocket
takeovers.
It all culminates in this big summer event
that we call Go Fest.
PRIYANKA VERGADIA: So it's clear that lots of people
are playing at any given time, especially during Go Fest.
How does "Pokemon Go" back end actually
scale to handle these peaks in traffic?
JAMES PROMPANYA: So we use Google Cloud.
We have lots of services that need scaling,
such as the Google Kubernetes Engine and Spanner.
Our front end service is hosted on GKE.
And it's pretty easy to scale the notes there.
Google Cloud provides us with all the tools
we need to manage the Kubernetes cluster.
The Cloud Console is just really easy to use
and comes with detailed monitoring, graphs, tools,
logging all included.
And it's just a few clicks away.
In the past Go Fest we had Google engineers
virtually sitting with us side by side
and ready to tackle any issues that could come from the event.
It was like having an extra support team working directly
for us.
To give an idea of scale, we have about 5,000 spanner nodes
handling the traffic, and thousands of Kubernetes
nodes running just for the "Pokemon Go" service.
We also have the various microservices
that host other parts of the game that help augment the game
experience.
All of these work together to support millions
of players playing the game all across the world at any given
moment.
And if you compare ourselves to like,
maybe "World of Warcraft," you'll
see that other massively multiplayer online games,
they split players into multiple realms.
But for us, all of our players, they reside on a single realm.
It was important for us that the players can always
interact with each other and share the same game
experience no matter where or when they're playing.
PRIYANKA VERGADIA: Yeah, it's like making
sure it's one same world, and one same similar
view that you're sharing with all the players, which is
what makes the game so amazing.
You mentioned Spanner and GKE.
Was it always Spanner and GKE?
Or did you decide to make that architectural decision
as the game got popular?
JAMES PROMPANYA: So we actually started off
with Google Data Store.
It was an easy way for us to just
get started without having to worry
about another piece of infrastructure
that we needed to manage, and scale,
and deal with all those things.
However, as the game matured, we decided
that we needed more control over the size, and the scale,
and performance of the database.
So when we were looking at different providers,
we saw that Spanner has consistent indexing that
allows us to do more complex database
schemas with these primary and secondary keys.
Data Store was also non-relational
with atomic and durable transactions.
So we needed a relational database with full consistency,
and made the choice to go to Spanner,
which gave us these global asset transactions.
PRIYANKA VERGADIA: Well, it's great to see that you started
off with a single back end that worked at the time,
and then quickly evolved that architecture to include
the components that made more sense as the game grew.
Now, let's say I am a player.
And I'm playing the game right now.
I open my app to catch a Pokemon.
What is happening behind the scenes?
How does the request flow work?
JAMES PROMPANYA: When the user catches a Pokemon,
we receive that request through the Google Cloud load balancer.
All of the static media is also stored on Google Cloud Storage
buckets and is downloaded to the phone when you first
start the app.
We also have caching enabled at the load balancer.
So it's all cached and served through Google Cloud SDN.
So when the player catches the Pokemon,
the GCLB sends a request to our GKE cluster.
And we have a front end service that sits
behind a Nginx reverse proxy.
The request goes from the user's phone to the reverse proxy,
to one of these player front end services.
We also have this thing called a spacial query back end.
It's a cache by location.
So that is where we store the information that determines
where a Pokemon is shown on the map, what
gyms and Pokestops are around you, what time zone you're in,
or any other feature that is location-based.
The way I like to think about it is the front end
manages the player and their interaction with the game,
while the spatial query back end handles the map.
The front end retrieves the information
from the [? squibby ?] to send back to the user.
PRIYANKA VERGADIA: So the spatial back end,
it sounds like it's a custom database of sorts that presents
the map and the Pokemons.
Now, what happens when I hunt a Pokemon down and catch it?
JAMES PROMPANYA: When you catch a Pokemon,
it sends the request to the front end to the Spanner
Database where your player entity is stored in the store
there.
For catching Pokemon, that actually
doesn't go to the spatial query back end.
If you were instead battling in a gym,
or adding lures to the Pokestop, that information
is stored on the spatial query back end.
And then the data gets eventually consistent,
at which point the players will receive the update.
And then it's used for all the other players
that are in your vicinity.
The front end at that point will retrieve the information
from the spatial query back end and send it back to the user.
We also store this information using Protobuf
and additionally, we also write this data
into Bigtable, which we also use for logging and tracking
purposes with very strict retention policies.
We also use Pub/Sub to send kind of a message
to a Pub/Sub topic, which is then
used for an analytics pipeline.
PRIYANKA VERGADIA: So how do you ensure that when
two people who are in the same geographic regions
see the same Pokemon data, and keep that relatively in sync,
especially during the events?
JAMES PROMPANYA: So it's actually pretty interesting.
Everything in our servers are deterministic.
So even if players are on different machines,
they are sent to different front end services,
if they're in the same physical location,
all the inputs would be the same.
And the same Pokemon would get returned to both users.
There's a lot of caching involved,
and as well as precise timing, especially when the settings
are changed and they need to be in sync across all the servers.
So it's very tricky, but it was very important for us
that all the players feel like they're
part of this one shared world.
PRIYANKA VERGADIA: Speaking of analytics,
we talked a little bit about Pub/Sub.
And you're sending these messages
into Pub/Sub for analytics.
Now, there's a massive amount of data that must
be generated during the game.
How does data analytics pipeline work?
And what are you analyzing?
JAMES PROMPANYA: Yeah, you are correct.
There's like 10 to 20 GB of data that
gets generated on a daily basis, and stores all of it
into BigQuery.
This game data is of interest to our data science team.
They use this for marketing purposes, for verification.
For example, we might want to say that this player or players
caught X millions amount of Pokemon
during our Saturday community day events.
So it's important that we have that information
in an easily accessible manner.
We also use Dataflow as our data processing engine
to run batch jobs to process the player logs
that are sitting in Bigtable.
One example of that is using cheat detection.
We might look through logs and notice
that a player used to be in Japan,
and then a minute later they were in Australia.
So we want to check for that, and make sure
that we're responding to improper player signals.
On the other side of things, we also
use Google Dataflow to set up Pokestops, and gyms,
and habitat information all over the world.
We take in information from various sources,
like OpenStreetMap, the US Geological Survey,
and Wayfarer, which is our crowdsourced website
for players to submit points of interests
in their neighborhood, and combine all that information
to build a living map of the world.
PRIYANKA VERGADIA: Now, as the events grow,
like you mentioned, in some of these events
the traffic grows to millions of users per second.
You also said 10 to 20 GB of data is generated.
How does the system scale when it comes to data pipeline?
JAMES PROMPANYA: Yeah, so with this increase of transactions
there is increase of load throughout the system
with a lot of our data pipelines.
For things like BigQuery, a lot of that just works.
Google Cloud will handle the increase in traffic
without any intervention.
Some of the other things we'll just
scale preemptively and make sure we
work with Google to make sure that we
can support that traffic.
PRIYANKA VERGADIA: With that much traffic,
obviously the health of the system is critical as well.
How do you monitor the health of the system
during these massive events?
JAMES PROMPANYA: So we use multiple monitoring systems,
including Prometheus, Grafana, and Google Cloud Monitoring.
On a personal level, I actually prefer
to Google Cloud Monitoring, because it's
very simple to use.
For me, it's very easy to set up graphs,
to set up alerts, and not have to get really
into the nitty gritty details of some
of these other implementations.
Everything just is very easy to use in Google Cloud and Google
Cloud Monitoring.
PRIYANKA VERGADIA: Well, it was awesome to hear
the entire implementation and the architecture details.
I am very sure you have something in mind,
or you're cooking up something to evolve this architecture
right now.
What are some of the expansions and improvements you're
looking to do in the future?
JAMES PROMPANYA: On the Google Cloud side of things,
we are looking into things such as Agones and Google Game
Servers, and using that to see how we
can make our game even better.
PRIYANKA VERGADIA: We wish you and the Niantic team best
of luck for these enhancements.
And this has obviously been super insightful for us, James.
Thanks for joining me today and sharing your architecture
with us.
JAMES PROMPANYA: Yeah, thanks for having me.
This was fun.
PRIYANKA VERGADIA: Well, we just took a behind
the scenes tour into "Pokemon Go"'s
architecture, how they use GKE and Spanner for scaling
to those high peaks, and how their data science team works
with BigQuery, Dataflow, and Pub/Sub for the data analytics.
Want to learn more?
Check out the links below.
Like and subscribe while I catch some more Pokemons.