This is My Architecture
Akatsuki: Building Stable And Scalable Large Scale Game Servers With Amazon ECS
Welcome to "This is My Architecture".
Today, we have Yuto Komai from Akatsuki.
Thank you for coming.
My pleasure.
Could you start off by
introducing Akatsuki and your architecture?
Akatsuki is an entertainment company that develops
live experience business including mobile games development.
Today, I'll share with you our architecture
that constantly runs over 1000 containers.
This workload is a back-end system for mobile games.
We operate multiple game titles.
For example,
there is a game released by Square Enix
called Romancing SaGa Re: Universe.
We develop the server for this game.
This game has reached over 15 million downloads
in the Japanese mobile app store alone
and has been enjoyed by millions of users.
Yes, Re: Universe is popular.
Thank you.
So you mentioned this architecture is used for the back-end.
How does it handle user requests?
When it receives API requests from mobile devices,
it sends back the game logic.
When a command such as "attack" is run on a mobile,
the server calculates the appropriate amount of damage
and gives a response.
Requests are sent to EC2-based ECS via ALB.
Player data is stored on Aurora
and cache information is stored on ElastiCache.
Game logic is calculated by the app on ECS
and is sent back to the client.
Games are very...
When a new event begins,
very many players may access all at once.
It causes heavy traffic on the system.
So we link EC2 and ECS for auto scaling.
By auto scaling EC2 and scaling it out,
the startup event is detected on CloudWatch Events.
The number of ECS tasks is controlled by Lambda.
It allows EC2 and ECS to link
for auto scaling in about two minutes.
Yes, spike is a problem in mobile games.
It's great that you can scale within two minutes after detecting it.
What part of this architecture did you focus on?
We focused on increasing concurrent connectivity.
During peak periods, there may be more than a million requests per minute,
or over 700 containers to scale.
Even then, the operation was stable.
A million requests per minute and 700 containers.
That's a lot.
How did you manage to stabilize the operation?
We conduct five different kinds of stress test.
First is the app's performance test.
We conduct a stress test on the system
by mocking the clients' behavior.
The server users the app performance monitoring service
to identify the app's bottleneck.
This test is repeated multiple times.
Second is the overload test.
This test identifies the bottleneck by overloading EC2 and RDS
so the CPU would run at 100%.
Let's say EC2 is overloaded
and causes an error when the CPU usage is 70%.
Here, we check if any resource is left unused
or if there's any parameter restrictions
for parameter tuning of the OS,
database middleware or app.
Third is the scaling test.
We check whether the vertically and horizontally partitioned database is properly distributed,
or the scaling in and out works properly.
On top of that,
we also check if the system sizing is adequate
for the expected number of access to the system.
Fourth is the failure test.
We cause failures on EC2 and RDS on purpose to see if the service stops.
For example,
we cause failures on Aurora on purpose during a stress test
to see if the service would keep running.
If it stops, we would identify the cause
or create an auto-recovery system.
Fifth is the long-hour test.
We give stress for long hours to see if it causes any troubles
such as memory leaks.
We conduct these kinds of stress tests from different perspectives
to improve concurrent connectivity and reliability of the system.
I see. By conducting such extensive tests,
you realize stable operation and improve performance.
Great effort.
Thank you.
What are some of the things you've achieved from this architecture?
Yes.
When the service was first released there was much more traffic than we expected.
It was once ranked at the top in the smartphone app store
and speaking of the system,
it received more than a million requests per minute.
Even then, it remained stable.
This stress test,
including the auto-scaling architecture is also used on other games
and we have been running more than 1000 containers at Akatsuki.
That's great. So your company uses a lot of this architecture.
That's a great impact on your business, too.
Please let our audience know which areas you'd like to improve further
and some features you plan to develop.
Sure.
We're considering a system that automatically performs stress tests.
Game apps are evolving every day.
We are currently perfoming a stress test for every version update.
To detect a change in load characteristics faster we are considering a system
that automatically and constantly performs load tests.
That could lead to greater efficiency in development and operation.
Lastly, do you have any message to our audience?
An international version of Romancing SaGa Re: Universe is underway.
Please look forward to it.
Thank you. We look forward to it.
Today's architecture is used on mobile games where spikes are likely to occur.
It auto scales within two minutes when it detects a spike.
in order to stabilize the operation.
This would be the end of the lecture.
Thank you, Mr. Komai.
Thank you.
I hope this helps our audience.
This is Akatsuki's "This is My Architecture".
Thank you for watching
For more information visit aws.amazon.com/this-is-my-architecture