Hi, welcome 'This is My Architecture'.
My name is Andrea, and I'm here with Drew from Riot Games.
Hi, Drew, welcome to the show.
Thanks for having me.
So what do you guys do?
We're the game company behind League of Legends,
Legends of Runeterra, Valorant, and few other upcoming titles.
Wonderful.
So we are here to talk about Hono Over AP.
Yes, yes mouthful.
What is that?
So it is a new data ingest system for League of Legends.
That's interesting, so it sits on AWS.
So let's talk about the details.
I'm really interested.
I'm sure our viewers are interested to understand what's behind this.
So you have players, I see player icon,
so how do they interact with the back end?
So they interact in a couple different ways.
So players are connecting to our game servers and data centers around the world.
The game client is also talking to an API that's running in EKS.
I see, so you have an API in EKS, Elastic Kubernetes Service.
Talk about the number of players,
how many are we talking about here in terms of scale?
Millions around the world in 20 Plus Shards,
Okay.
And then how are they interacting with the back end?
Are they sending information, in what ways are they interacting with the EKS cluster?
Yeah.
So the game client and game server
are both sending information about that player experience.
Like, how are their games going, what champions are they playing?
What kind of things in the store, are they engaging with?
Okay, so you're collecting that information ongoing,
and can you give us a scale of like how much information are you gathering in a given day.
So we are pulling about eight terabytes of data a day,
the system peaks at about, somewhere over 500,000 events per second.
Oh, wow.
And what's the geographic reach?
I mean, I suspect this is global.
Yeah, global worldwide, all over the world.
Is 20 different shards or something like that
20 different shards.
So with that global presence,
I'm sure amount of data that you are gathering,
how do you deal with that level of flow
and to make sure performance is sufficient?
So it's challenging.
There's chances with network connectivity, and throughput.
These are all data centers that are hooked up
through our Riot Direct network up into kind of the nearest AWS region,
but that network can become congested at times.
Okay.
So what do you do, then, once you connect into EKS,
what's the next thing that happens?
So data flows from EKS into MSK here,
and we have a regional buffer where there are five different regions
where data flows in, data is collected, and it's prepared for processing.
Okay, so Managed Streaming for Kafka,
And then how do you deal with that level of data flow?
Can you talk about the dynamics, like how's that working?
Well, so MSK just kind of takes care of a lot of that for us.
Kafka is a very robust streaming data platform.
And we're able to scale up and out like as we need to,
we've kind of settled on a nice scaling pattern that's served as well.
That makes perfect sense.
And then what about the gaming service?
Do you have any level of connectivity into the backend as well?
We do.
So the data is buffered here locally, and then we use MirrorMaker
to replicate data out of our local Kafka cluster here,
into those regional MSK clusters.
Okay, so, MirrorMaker, is a component of Kafka.
And why do you do that?
Like I mentioned earlier, network connectivity can be a challenge.
So when we hit peak players,
it's a huge difference from when we're in off time
from what just kind of network is available.
And that player experience is the most important thing.
So I don't want to be degrading their ping
by trying to send a bunch of data back to our system.
I see, so you have a local copy, I mean, are you buffering data here?
And essentially, you're using MirrorMaker to replicate all that to MSK.
Interesting.
So now, what happens next?
So once everything lands in MSK, we have a Spark job, Spark streaming job,
which is writing the data out into data lake in S3.
And then thats your data store, data lake,
what do you do with that data?
So that serves everybody from...
So we have our data consumers at that point.
So everybody from analysts, designers, engineers, executives,
are running their queries against that data,
ETL is running against that data.
And its like kind of provides insight into what's happening.
So what would that give to the user?
What's the user experience?
So when you have this set of information, what do I get as a player,
say for example?
So something like,
see we roll out a new patch, and we've made some changes to champions.
It's kind of critical to see what effect did that have on gameplay.
Are players winning more with that champion now?
Has the length of games changed, or the towers falling earlier?
So this system moves kind of the time to get that information out
somewhere between like six and 24 hours, down to five minutes.
Oh, wow, okay, so that's pretty impressive.
And then, on the consumer side, I do see Glue.
Can you talk about the consumer side?
Yeah, so we have databricks running,
Glue is our meta store.
So Glue is kind of information about what logical tables are there,
where are those files in S3.
And it acts as our permission and access layer
between our users recording the data and kind of the raw stuff out.
Okay, so do you have consumers here using the Glue
to then access at least to view and see and search for the data?
So Drew, thanks for walking us through this architecture,
it's really a great example of hybrid architecture, right?
Using Kafka to buffer information on-prem,
and they've been able to move them into the back end AWS
using Managed Streaming for Kafka,
as well as providing enrichment and access to analyst and others
to view the data, and also better for the user experience all together.
So thanks for being here on the show.
Thanks for having me.
And thank you for watching 'This is My Architecture'.