[MUSIC PLAYING]
YUTTY KAWAHARA: Hello, everyone.
This session is to explain how Dragon Quest
Walk, a game currently very popular in Japan,
handles millions of players and how
Cloud Spanner is essential to managing
high volumes of requests.
My name is Yutty, customer engineer from Google Cloud,
working with gaming industry companies in Japan.
Today, I'm with Taka, who is an engineer from Colopl.
The most difficult part of launching a new game
is unpredictability.
Once the game gets popular, the server
often receives massive requests in the short-term.
Compute is very easy to scale.
If you need more resources, you simply add more servers.
But the database often [? begins ?]
a bottleneck because legacy databases like MySQL
are not designed to scale horizontally.
So when the database is overloaded,
this is what happens.
For game players, I think this is the most frustrating error
message.
If you are a loyal player, maybe it's not a big deal.
But if you are logging in for the first time,
it may be your last time.
Very few games can overcome this type of error at launch.
So how do game companies have the database scaling program?
Well, in general, gaming company need [INAUDIBLE]..
And they use a technique called sharding,
which creates a single database on different servers.
For example, sharding can be done
by players, by geolocations, or by gaming platforms.
This is a broad generalization.
Sharding might be a solution but can lead to hidden costs,
as you can see in this iceberg.
First, maintenance costs-- for example,
if single database server can handle 10,000 players,
that means you need 100 servers for every million players.
I think no one wants to manage hundreds of servers.
Second, upgradability-- players from all over the world
will play at all times of the day.
Therefore, the database has to function 24/7.
It is impossible to find hour of big maintenance window that
accommodates everyone.
[? About ?] downtime, traditional relational
databases offer high availability
using a failover mechanism.
This means if the database server fails, it will recover.
But it generates downtime.
Today, Google Cloud offers many types
of database, including Firestore, Cloud SQL,
and third-party partner solutions,
which are designed for various customer needs.
Dragon Quest Walk has chosen Cloud Spanner as their database
since it is the ideal, fully managed, scalable database.
Cloud Spanner combines the benefit of relational database
structure with nonrelational horizontal scale, which
means Spanner is [INAUDIBLE].
Spanner also provides horizontal scale in/scale out.
It is fully managed and has no planned downtime.
In addition, Spanner can provide up to [? 99.999% ?] SLA.
As I mentioned earlier, traditional database
have big hidden costs.
By deploying your game on Cloud Spanner,
you can [? draw ?] much [INAUDIBLE] these hidden costs.
Spanner can scale out with one click and upgrades
automatically without downtime.
Data is replicated across zones, which
result in high availability.
But how does this translate into numbers?
When Enterprise Strategy Group, which is a research firm,
did a three-year TCO analysis for Spanner against us
other traditional and cloud databases,
they found that you can save 37% of the cost,
compared to other cloud solution, and 78% of cost,
compared to other on-premise solution.
The savings mainly come from hardware investment, as well as
many administrative tasks that you
need to spend on to make sure your database stays up
and running.
Now, let's hear from our customer, Colopl,
and how they adopted Cloud Spanner in their game,
Dragon Quest Walk.
TAKAYASU OYAMA: Hi.
My name is Taka, and I'm a software developer
for a Japanese game company called Colopl.
I will take over from here and briefly
talk about how we use Cloud Spanner in Dragon Quest Walk.
Before I go into the details, allow
me to do a brief introduction about the game.
The game is called Dragon Quest Walk,
and it was developed by us and published by Square Enix.
The game is part of a long-running Dragon Quest
series owned by Square Enix.
So here are some screenshots of what
the actual game looks like.
The game is a merge between location-based AR
and a traditional, turn-based JRPG.
The game is for mobile, and it's available in Japan
for Android and iOS.
So here are some numbers to give you an insight of how much data
we process every day with Cloud Spanner.
So we released this game back in September of last year.
And I can't say the exact numbers,
but we had a lot of users register on the first day.
What I can tell you is that we had over 5 million downloads
on the first week.
Currently, we still have millions
of monthly active users, and we process thousands
of queries per second across hundreds of Spanner
nodes every day.
Here's the agenda for what I'm going
to be talking about today.
First, I'll start by talking about why we chose Spanner.
Then, I will talk about how we integrated Spanner
into our development.
After that, I will talk about some of the challenges
we faced with Spanner.
So here are some of the reasons why
we chose Spanner for this game, starting with the first one
on the list, scaling in and out.
So we used to use a self-managed MySQL to handle our user data,
and we scaled it by splitting data into horizontal shards.
In our past titles, this worked well for us at first.
But when the numbers of shards start
to grow into double digits, we started to notice some problems
with this approach.
The problem arose whenever we wanted to scale.
When we scaled these databases in or out,
we had to scale manually and we had
to scale once every few months.
We would scale out to handle the high traffic whenever
we released a new feature or whenever
we did a seasonal event, and then, we
would scale in to save costs once the hype cooled down
and the traffic eased.
Preparing and executing these tasks
took hours and sometimes days, depending on the scale,
and it was stressful and time-consuming.
By switching to Spanner, we were able to do this with one click
and scaling would complete within a few minutes.
So that was reason number one.
So the second reason why we chose Spanner
was for stability.
In our previous titles, as I've mentioned earlier,
we used a self-managed MySQL, and we
used to get unexpected crashes at least
once a month, mostly due to host errors.
With Spanner, everything is managed
and all the problems are handled in the background.
Now, we've been running this game for over six months now,
and we haven't had any major downtimes, just minor ones,
and it wasn't Spanner's fault. So I could confidently
say stability has been good.
So that was reason number two.
The third reason we chose Spanner
was so we could write a more readable and maintainable code.
Previously, when we had to write code for sharded databases,
our code had to be infrastructure-aware,
which made it hard to read and maintain.
One example of this would be something
like distributed transactions, where
you have to make transactions across different shards.
This was hard to understand, and it
was even harder to debug, especially for the parts
where we had to roll back.
Then, there are things like replication delays, where
data retrieved from the replica might not be up-to-date,
so we had to be careful about where we read our data from.
After switching to Spanner, we were
able to make inserts, updates, and deletes
in a single transaction, and replication delay
became a non-issue because consistency is guaranteed.
Now, I would like to talk about how we integrated Spanner
into our development.
When we first decided to use Spanner,
we had the feeling it was going to be a bumpy ride,
since no one at the company had any experience with it
and there was very little information
available online at the time.
So we did some preparations to make sure
that the transition from MySQL to Spanner
would be smooth and painless as possible.
So how did we do this?
Well, first, we created a small, dedicated team.
And by team, I mean me and my colleague.
So me and my colleague perform some basic benchmarks
to check the performance.
And once we confirm that the performance was good,
we put in some extra effort to making
a dedicated driver for a PHP framework
we were using called Laravel.
Doing so allowed us to use pre-existing [? ORM, ?] which,
in turn, allowed us to reuse existing libraries
we were using for existing games.
By the way, we've uploaded the driver we wrote on GitHub,
so please take a look at it if you're interested.
So once the driver was written, we tested the driver
by taking an existing game and replacing the MySQL driver that
was used with a Spanner driver.
This worked well because we get to see
how it performs on real scenarios
and allowed us to weed out bugs we couldn't
catch with our automated tests.
Doing this also helped us get familiar with the database much
quicker than trying to apply it to a game from scratch.
After all that was done, we integrated Spanner
into the main game development cycle.
By doing these preparations, we were
able to fully integrate Spanner into our game
in about two weeks.
Now I'll talk about some of the challenges
we faced with Spanner and also talk about how
we overcame those challenges.
First one on the list is interleaving.
Spanner has this concept called interleaving,
which is crucial for getting optimal performance out
of your queries.
Interleaving is a concept of having
parent-child relationships between tables so
that data for child tables are inserted in the same node
as the parent table.
This is essential because if you have a has-many relation
and you try to get the related rows from the children,
it's much more efficient to look it up within the same node
rather than looking it up across different nodes.
So in order to account for this new concept,
we scan through all the existing schemas we wrote for MySQL
and added definitions for interleaving
for tables that had a parent-child relationship.
At the same time, we made sure all the indexes
that mimic the relationship in MySQL
were removed, since they were unnecessary when the tables are
interleaved.
Another thing we made sure was that all
queries that scanned the interleave children included
the parent IDs when doing a lookup.
This was because omitting it caused Spanner
to scan the entire database, which degraded performance
tremendously.
To be absolutely sure that the parent ID was included
in all the queries, we added some code
that listened to all the query events
and threw a warning when the parent ID was omitted.
Next item on the list is scale-in hiccups.
So in this game, we scale in and out fairly often,
based on how much traffic we're expecting for the day.
Scaling out is simple, since all you have to do
is increase the number of nodes in the UI
and wait a few minutes for it to be deployed.
Scaling in, however, was a bit trickier.
The actual task is the same as scaling out,
but we noticed that when we reduce the number of nodes
too aggressively, there was a noticeable response delay.
These delays ranged from a few milliseconds
up to maybe something like 30 seconds.
Sometimes, this also caused the servers
to choke due to too many connections being open.
So we resolved this by forcing Spanner
to time out in three seconds and return
a response to the client, which asks the client
to retry the request after a few seconds.
In addition to that, we now try to decrease the nodes
in small increments, since the range of delay
seems to be proportional to the number of nodes.
we decrease.
From what we can observe, if we go easy and reduce
one node per five minutes or so, we
manage to minimize the delay enough that no retries occurred
on the client.
The next item is handling bursts of access to shared data.
So the game has some small sets of data, which
is shared across all users.
These are things like item properties
and monster map coordinates.
We wanted to replicate these data
so access to these data could be distributed.
The way to do it in Spanner was to add a column
with a shard number and insert copies of the data
with different shard numbers.
Then, what you can do is have the code select
one of the shards at random and add that to the query
and that will distribute the load.
We thought about doing it this way,
but we figured it was too difficult to implement
into the framework at the time.
So for shared data, we decided to use a self-managed MySQL
instead, since it can replicate and scale out easily if all you
have is a small set of data.
The last item on the list is handling launch day traffic.
In order to talk about this, I first
have to explain what a split is.
Internally, Spanner divides your data
into chunks called splits, where a split can move around
between servers.
And when there is high load on a split,
it tries to slice it up and add split boundaries
to distribute the load.
This works well in normal scenarios
where there is a gradual increase in users
with gradual growth on load.
But this was not what we expected
to happen on launch day.
What we expected to happen on launch day
was thousands of users registering
in a span of a few seconds.
And we knew splitting wasn't going
to be fast enough based on numerous launch day
simulations we performed.
So in order to make this work, we presplit the database
before the launch.
We called this process warming up the database.
The way we warmed it up was rather simple.
We just forced the splits to trigger
by slowly adding thousands of dummy users
until it split enough to handle the launch day traffic.
After the split was complete, we deleted the dummy users
since the split is retained even after the users are deleted.
So the method was simple, but there was one small caveat.
These splits were only sustained if there was access to them.
Otherwise, these splits start to merge again
after around 24 hours.
So we made sure to do the warm-up 24
hours before the actual launch.
The warm-up was a tedious and time-consuming task,
but we're glad we did it because the launch was a great success
with zero problems.
Before I end, I just wanted to show you
how our storage architecture looks like
and briefly explain how various services are used
in conjunction with Spanner.
All the storage we use is lined up on the right.
Going from the top, we have Spanner,
which holds all the user data, as you know.
Our analytical data, like user action logs,
is streamed to BigQuery, since BigQuery works better
for scanning large sums of data.
Our master data, like monsters, items, and location data,
is running on multiple replicated MySQL servers,
where the load is distributed using GKE service.
Then, we have Memorystore, which is
used for temporary caching of frequently accessed data
to ease the load on other storages.
And I completely forgot to add to this,
but we also use Elasticsearch to store parts of our user data
so it can be used to look up names
and other various attributes.
That's all from me today.
Thank you.
YUTTY KAWAHARA: Thank you, Taka.
Feel free to reach out to us to find out
how Cloud Spanner can [? erase ?] challenges
that you may be facing.
Visit URL to learn more.
Thank you for joining today's session.
[MUSIC PLAYING]