ROB MARTIN: Hello, and welcome to "Solutions
for Scaling Massive Global Games in the Cloud".
I'm Rob Martin, Chief Architect for gaming at Google Cloud.
SEBASTIAN WEIGAND: And I'm Sebastian Weigand,
a planet scale systems architect on Rob's team.
ROB MARTIN: And today we're going
to be talking a lot about global games.
First, we're going to define what we mean by a global game.
Next, we're going to define all of the components that
are used to power a global game in the cloud.
Then we're going to do a demo, and finally, we'll
close out with some thoughts on a toolkit for building
your own global game in the cloud.
So let's start by talking about global games.
Gaming is the world's most popular form of entertainment,
with 2.7 billion players globally,
and that number is expected to increase
by another half-billion by 2023.
Let's talk about an example of a truly global game.
Apex Legends is a great example of a truly global game.
Apex Legends launched exclusively on GCP
with our partner, Multiplay.
Within the first eight hours after launch,
over a million players had played the game.
Within the first three days, that number 10xed
to 10 million players, and by the end of the first month,
over 50 million players had joined the fun in Apex Legends.
So let's talk about concurrency and scaling.
A CCU is defined as one concurrent player.
Peak CCU is the maximum number of players
during the day that will be playing your game all at one
time.
You can see many points of peak CCU illustrated in the diagrams
to the right.
As players join your game, you'll
need to scale up infrastructure to meet that demand.
So in some sense, as more CCUs join your game,
you'll need to scale up more CPUs in the cloud to power
the multiplayer experiences.
Let's take a look at a few different examples
of peak CCUs.
So the first example is a premium launch
where the game is launched perhaps at retail,
there's a very large peak CCU spike on the first day,
and then you settle into a predictable pattern
in the following days.
Or in the second example, you might
have a mobile launch where you have a limited beta,
and then over time, user acquisition
opens up to add more and more concurrent players.
What we're seeing more and more of today
with today's free-to-play games like Apex Legends
that incorporate content drops, seasons, and live events
is you're seeing peaks happening throughout the game's
lifecycle, so in today's world, it's very important
to be able to scale to these unpredictable demand
peaks that are generated by these new types of content
releases.
And so we've talked about concurrency and scaling.
Now let's talk about geography.
So using an example game with a million concurrent players,
those players are going to be distributed
all around the Earth.
Using this example of a typical AAA PC console game,
we would expect about 40% of those concurrent players
to be located in North America.
We'd see about 30% of those players in Europe, about 10%
in South America, and then the remaining 20% of players
distributed across the Middle East and Asia.
Now, because latency matters in gaming,
you're going to need to deploy your compute resources
all around the Earth to get them as close as
possible to your players.
So as Seb and I prepared for this demo,
we thought to ourselves, would it
be possible to scale up all of the compute infrastructure
required to run a global game for one
million concurrent players within the space
of this 25-minute demo?
What do you think, Seb?
SEBASTIAN WEIGAND: With the right tools,
anything is possible.
All right, Rob.
Let's dive in.
I've created these clusters already,
and I've installed Agones onto each of them.
Right now, these are essentially empty clusters
running only a few nodes each.
I've got one cluster in the US, another in the Netherlands,
and a third in Japan.
Because our cloud console aggregates information
across clusters, I can see that I don't have anything running
in these clusters yet besides the pods needed
to run Agones itself.
I've coded up a game server in Go, which simulates a battle
royale-style game.
It records the events which typically take place
into BigQuery, our data warehouse.
It also creates player information,
such as match rankings, and writes those
into Spanner, our horizontally scalable relational database.
To deploy this server worldwide, we
define it using an Agones game server fleet spec
combined with a Google Cloud Game Servers deployment.
The fleet specification looks just like any other Kubernetes
resource.
Agones provides an abstraction layer on top of Kubernetes
to understand the concept of a game server
or a collection of game servers called a fleet.
It does this because game servers
aren't your typical stateless workload.
You don't want your game to get killed in the middle of a match
and scheduled on another node, for example.
Likewise, Kubernetes needs to know more information
about the state that the game server is
in, such as total player count, to handle functions
like scaling properly.
Agones takes care of things like this for us
so that we don't have to.
Let's take a look at our current fleets
on each of the GKE clusters.
As you can see, we don't have any fleets deployed.
Let's go ahead and create a ton of game servers.
To do this, I'm going to ask GCGS
to apply a new rollout using the configs I've specified.
What it'll do is connect to every GKE cluster
in each of the realms and create the Agones fleets
based upon our fleet spec.
In short, we'll go from no pods to thousands
across multiple clusters in multiple continents.
Here you can see the number of servers
that we want, the number which we are booting up,
and the number which are ready.
The allocated column would update once players
start connecting to them.
Rob, you asked about scaling up enough resources for a million
concurrent players.
Deploying thousands of these servers
across 10,000 CPU-optimized cores ought to do the trick.
Let's check those GKE clusters.
Remember, we only had one node in each zone.
Let's see what it's up to now that the Cluster Autoscaler has
seen our requests spike.
This is great.
The Cluster Autoscaler has already
begun adding additional nodes to these clusters based
upon the resources that we've requested.
OK, more and more game servers are coming up.
Agones and GKE are working together
to spin up additional virtual machines to handle this load.
This will take just a couple of minutes to scale up.
Let's check back in on these later.
Back to you, Rob.
ROB MARTIN: Thanks, Seb.
It'll be interesting to watch that scale up.
And while that's happening, let's
talk a little bit about the components that are required
to power a global game.
So first, you need a global network
to connect with your players.
Next, you need high-performance compute to run your simulations
and your multiplayer matches.
Then you need a global database instance
to track player progress throughout the game.
You need to monetize your game, so you'll
need planet scale analytics that can absorb all the events
you're going to be posting.
And then finally, you need a fleet orchestrator to bring it
all together, and to scale this infrastructure up
as players join your game, and to scale it down as players
move on to other activities.
Let's take a look at each one of these components
in more detail.
Let's start with our global network.
The first thing that you'll notice on this diagram
is the 24 regions distributed around the globe.
These are represented as colored circles
and are the geographies where you
can run your compute resources to power your matches.
The next thing you'll notice is our 144 edge points of presence
that are located as close as possible to your players.
Now, the unique and differentiated thing
on this diagram is the fiber optic network
that ties our edge points of presence and all of our regions
together.
Only Google Cloud offers a global private fiber
optic network out of the box.
Next let's talk about compute, which
you'll need to power your multiplayer matches.
So we know that there are many different types of games,
from a casual, mobile title that you
play on your phone with friends to a powerful and high-fidelity
multiplayer battle royale match that you play on your PC
or console.
At Google Cloud, we've created a spectrum of compute
that you can tailor to the specific needs of your game.
So for that casual title, you can
run on our cost-optimized VMs, and for that high-fidelity
battle royal title, you can run it
on our compute-optimized VMs, which run at high clock speed
and we built specifically to run demanding workloads like games.
Next, let's talk about a scalable database.
As players progress through your game,
you'll need a way to capture updates like rankings
and updates to inventory.
The way that a database for a global game is typically built
is by utilizing a commodity database like MySQL
and running tens or even hundreds of instances to power
a global game.
This technique is not globally scalable
and, from an ops perspective, is not awesome.
At Google, we offer an alternative approach
with Spanner, a horizontally scalable relational database.
We offer the capability to run a single instance that
can serve all of your players globally without shards.
Now that's awesome.
So we've talked about a lot of different infrastructure
components that will be required to power your global game.
It's going to be important that you can effectively
monetize, particularly in today's free-to-play world.
BigQuery offers a globally scalable data warehouse
that can scale from launch to the world's biggest games
with zero ops for your team.
Candy Crush is probably the best example
of scaling running on BigQuery.
Candy Crush ingests over 70 billion events
per day seamlessly, and Google manages all the ops.
So to tie it all together, you'll
need to scale these resources up and down throughout the day
as players join the game and move on to other activities.
To manage the scaling of all of these resources deployed
globally, we built Google Cloud Game Servers, a global control
plane for running your game.
Google Cloud Game Servers is built
on top of open source technology,
including a Agones and Kubernetes, which
allows you to run your infrastructure
components wherever you need them, in GCP, on other clouds,
or on bare metal.
Let's take a closer look.
In this diagram, we're showing the Game Servers API,
which is your global API and control
plane for managing your game.
On the left, we have one Kubernetes cluster
that's running in Google Cloud platform on GKE.
On the right hand of the diagram,
we have another Kubernetes cluster
that's running on premise, on bare metal.
Both clusters are being centrally managed
by Google Cloud Game Servers, and all
of the operational events and log events
are being captured centrally.
Agones and GCGS offer a new cloud-native way
to build your game.
Let's talk about the roadmap for building your game
with this new technology.
First, you containerize your game
and implement the open source Agones SDK.
Next, you register each of your clusters
with the Game Services API control plane.
Then you begin organizing your clusters
into groups by geography that we call realms.
Next, you define configs for things like autoscaling
policies, and finally, you test and rollout your game server
binary to your global fleet of game servers
running on Kubernetes.
So let's check in on our scaling.
Seb, how do you think things are looking?
SEBASTIAN WEIGAND: Pretty good from here.
Let's take a look.
OK, Rob, let's take a look and see how
our servers are coming along.
So far, we've got 4,200 servers running in the US,
a little over 3,300 running in the EU,
and 500 running in Japan.
This is truly a global fleet, more than enough
to support one million concurrent players
as each server can handle at least 150 players.
It took under eight minutes to scale up
all the compute needed for a massive deployment
all around the world.
This really demonstrates just how easy and powerful managed
Agones really is.
Now let's take a look at the database.
Here we can see the number of nodes we're currently
using, along with the CPU utilization and number
of concurrent requests.
Wow, that's a lot of requests, and this number
should keep rising as more and more game servers come online.
We're running about 80,000 queries
per second, which is about what you would see
for a very large global game.
Should we need to handle more, rather than deal with database
sharding, we simply increase the node count in Spanner,
and it handles the rest.
You got to love the separation of compute and storage.
Now let's check BigQuery, our gaming data
warehouse where we've been ingesting
massive quantities of telemetry events
since we started our fleet.
So far, this table is just about 100 gigabytes,
with over 800 million rows.
Excellent.
But let's figure out the number of events processed per second.
Even though there are thousands of game servers inserting data
right now, let's ask BigQuery to take the number of events
between the last two minutes and one minute and then
divide by 60.
What I love about this is we can keep
inserting more and more data and let our scalable databases
handle the rest.
When your cloud can scale to ingest data like this,
your team can focus more on creating really great games
and less on managing your infrastructure.
Wow, over 800,000 rows per second, not bad.
So Rob, in less than 10 minutes, we
scaled up close to 10,000 cores of compute on three continents,
wrote over 80,000 inserts per second
to a global database instance, and ingested over 800 million
rows to our data warehouse.
Talk about scalability.
ROB MARTIN: Wow, those are some impressive numbers, Seb.
We were able to scale all of the infrastructure required
for a million concurrent players in about 10 minutes.
To recap, let's take a look at each component in this toolbox
for building massive global games on Google Cloud.
Our premium network offers a global fiber
optic network that extends your reach to wherever your players
are around the globe.
Compute-optimized VMs offer the high clock speeds
required for today's most demanding games.
Spanner provides a global database instance for your game
with no sharding.
BigQuery is a planet scale data warehouse
that can ingest millions or billions of events each day
for your game.
And finally, Google Cloud Game Servers, the fleet orchestrator
that can scale these resources up and down seamlessly
during the day as your players need them.
To learn more, join us on the web,
and we'd like to thank you for joining us today.
So Seb, that was a pretty cool demo.
We were able to scale up a million concurrent
player fleet in about 10 minutes.
So for our viewers, what would it
take to scale up a fleet for 10 million concurrent players?
SEBASTIAN WEIGAND: Well, because the infrastructure
and technology are sort of taking care of it for you,
the only thing that I could think of would be more quota.