Hello, and welcome to 'This Is My Architecture.'
Today with me, I have Vasilisa from MYTONA. Hi.
So, Vasilisa, can you tell us a little bit about MYTONA?
Sure, MYTONA is a game developer and publisher company,
We create mobile games in the time management and hidden object genres.
Okay, so mobile games, is it?
So, what's this interesting architecture that you have for us today?
Yeah, for our games, we are solving the use case of more personalized and relevant offers.
So me and my machine learning team,
we built an in-game purchase recommendation system, which is called ML Offer.
Okay, so the system with its own name, that's very interesting.
So, what does it actually do?
Yeah, our system has two parts,
the one that makes predictions and the other is responsible for model training.
So, when game servers request predictions,
the Lambda function gets triggered by an API involvement,
and takes on the role of a reverse proxy.
It allocates traffic between our recommender models,
which are placed in the ECS cluster.
We also have one container for bandit here.
Multi-armed bandit.
Okay, so that's very interesting piece.
I suggest it allocates the traffic dynamically, but how does it actually work?
Yeah. In our Lambda function, we implemented dynamic traffic allocation.
The multi-armed bandit collects data from Kafka,
on the way users react to it for different model predictions,
and periodically recounts weight.
Okay, so it assigns the different weights to the models, I guess. Yeah.
Yep. So basically, we have our inner continuous A/B testing between our models.
Okay. So there's an additional use case, which is A/B testing?
Very interesting. Yeah.
But for machine learning, there is an additional interesting thing,
which is model training. How do you actually train your models?
Sure. For model training we use SageMaker,
which gets daily triggered by
and manage workflow in an EC2 based Airflow.
Which is a workflow manager, I suggest?
Yeah, it collects data from Kafka,
and eventually, our model is stored in an S3 bucket,
from there that gets added to build model API containers.
And we also use ECR
to store our API images for different models.
Okay, so that's very interesting
but definitely while building the system,
you might have had some challenges and problems.
What were those?
Yeah, you see, the ECS cluster and Lambda were not our first choices.
As the first we implemented manually,
a Docker swarm on EC2 instances with traffic for traffic allocation.
And it brought a lot of problems with scaling and adding new versions of models.
Okay, so what were these issues?
You see, as the requests are not evenly distributed within a day,
so we had to manually add new replicas, create new instances,
or change the capacity to handle traffic spikes.
Okay, kind of a challenge. How did you actually solve it?
Yeah, actually switching to this version of the architecture of our solution.
Having our models as ECS containers
allows us to create triggers for auto-scaling,
and to safely add new versions for models.
So problem solved, is it? Yeah.
Okay, so you've been running this system for some time already.
So what were actually the performance and the outcomes of the system?
Yeah, according to test results, our system performed much better
than the one without the machine learning personalization.
We increased our conversion rate three times,
and our revenue was boosted four times for these type of offers.
Okay, but how actually many customers is this system used per day?
It's like, several hundreds of thousands.
Okay, that's impressive.
Well, thank you, Vasilisa, for sharing this information with us today.
Thank you for having me. And thank you for watching 'This Is My Architecture.'