Hi. I'm Taehyun Kim from 'This is My Architecture'.
Today, we invited Sungwon Lee from Buzzvil
to talk about Buzzvil's architecture.
- Hello. - Hi.
First, please introduce Buzzvil.
As an ad tech startup,
Buzzvil is creating innovation by using rewards in their advertising ecosystem with a long history
and a lot of legacies.
We are operating a rewarded advertisement platform that can maximize the efficiency of advertisements
by providing appropriate ads and appropriate benefits to users.
Today, I will explain our user segment system.
It helps advertisers freely configure highly detailed targeting
and recommendation algorithms based on users' activity data.
For example, there are users who are interested in games and have the highest purchasing power,
or users who viewed our advertisement but did not participate in.
We target them.
We help execute advertisements efficiently.
I'm curious about the user segment system that you mentioned.
Please explain in detail.
Yes, we receive user's information from a variety of sources with their consent.
By using SDK, we receive it through the app or web.
We can receive it from server to server or by dumping the data.
The user data gathered in this way comes on top of our MSA.
Each service drops users' activity data into logs.
The logs dropped in this way are gathered in S3
through Fleunt Bit and FluentD.
In fact, the collected data includes the request-logs and other logs.
So only the users' activity data that we need is extracted
through Lambda and then put into S3.
In addition, the collected user's attribute data
is also stored in DynamoDB.
The collected data is collected in S3 through DynamoDB Streams
and Kinesis Firehose.
You are already using Fluent D when collecting logs.
Is there another reason to use Lambda for added filtration?
Due to the nature of the advertising industry, we have a tremendous amount of traffic with logs.
However, when I made this to be handled by Fluent D,
the CPU usage increased too much.
The cost also increased a lot.
However, since this doesn't need the real-time processing,
we performed the batch work in the file unit.
By using Lambda, the configuration became simple
and cost was reduced.
It can be processed in real-time by using Fluent D, but because of the cost issues, you used Lambda.
Yes, that's right.
And the data gathered like this is actually from our partners.
So we need to add our internal information before we actually use it.
When it comes to missing data, for example, the information
such as mapping between the partner's user IDs and our Buzzvil's user IDs.
This information is added to the data through Athena.
The gold data is formed in this way.
Are you using only Athena to create the gold data?
No, because we only need to create the form that gold data needs,
we are configuring each due to the nature of the data source.
Among them, we are also using EMR.
In fact, Athena can be used in most cases.
And I use Athena a little more
because it is
much easier to configure and can run serverless.
- I see. - In order to utilize the collected data,
our advertiser creates our segment condition information through the admin system.
In order to create segments like that, the data catalogs
or segment information created by using the data catalog
also stored in RDS.
Yes, the Evaluation Job that runs periodically with the RDS information created in this way
builds the Athena query.
When the Athena query is built,
it makes a request for the gold data that was created earlier.
Then, the information of users belonging to each segment can be retrieved.
All user information for each segment collected in this way
is stored in DynamoDB.
When it is necessary to recommend an advertisement for specific users in the future,
this segment data is received from DynamoDB and the ad is recommended with the information.
You save the segment meta information in RDS. For the segmented result, you use DynamoDB.
Is there any special reason?
Ad requests must be processed very quickly,
between 100ms and 200ms.
Our segment system that we use
must also handle a lot of traffic quickly,
and must be highly available.
And due to the nature of our service, we provide different services,
but there are times when traffic is high in all services.
As I tried to handle those situations,
I thought that DynamoDB is the most suitable.
Thanks for your explanation today.
Finally, please tell us about Buzzvil's future plans.
First, we are thinking of automating the data crawling in the short term.
Because data is generated from so many data sources,
there are times when creating a data catalog.
I am trying to crawl this part
by using the AWS Glue crawler.
When we do that, even if any new data is added,
the advertiser will be able to use the data a little faster
to execute the advertisement.
In addition, we are planning the architecture that will use machine learning technology
to create more and more various segments and execute the advertisements.
I'm looking forward to seeing Buzzvil's future.
Thank you.
This has been 'This is My Architecture'.
Thank you.