Hi, welcome to another episode of 'This is My Architecture'.
I'm Jose, and today I'm with Lucas from Smadex. What's up, Lucas?
What's up, Jose? I'm delighted to be here, thank you for the invitation.
What is Smadex? What do you do there?
Smadex is a programmatic digital marketing company.
We help our clients, advertisers,
to grow their digital businesses with online publicity.
Basically what we like to say is that we show the correct ad,
to the correct person, at the correct moment.
So, programmatic advertising, what does that mean?
It's very interesting. It all starts with a user
who visits a website,
or it could be an app too, it's the same thing.
When the user visits this website, at the moment they visit it
and before the site is ready, that, is during the loading time,
an auction is triggered.
This auction is intended to define which ads
will be shown to this user for this particular website.
We received this auction request
and we have only 100 milliseconds to respond to this auction.
In those 100 milliseconds we have to decide
if we want to participate in the auction or not.
If we're going to participate, which ads to show.
And finally, how much to bid, which is very important
Because we want to win the auction, bidding the lowest possible amount.
This happens a million times per second.
Once we win the auction and the ad is shown,
the user later interacts with that ad.
They can click on it, download an app, fill out a form.
We track all those events in another application
which is another layer, all of which is associated
with the original auction through a unique auction identifier.
And I see you're using Spot Instances EC2.
I understand that's for cost optimization.
Yes, cost optimization is paramount in this industry,
because with the request volume we manage
and the latency that we have to respect,
we need to be in three different geographical zones
to be as close to the user as possible.
And we have, on average, 600 instances running at all times.
Therefore, cost is critical.
Tell me a little more.
The interesting thing —besides the challenge of latency and volume—
is how to store all those requests, right?
Where do we store them to read them later, when you have to make the relationship?
A few years ago, when we didn't have
a million requests per second, we used Redis.
So when the auction is generated, this bid is saved in a Redis,
with an ID, and later is read directly
from the tracking layer, with the same ID, in key-value format.
As Redis functions in memory, the latency is super low,
and everything goes well, or went really well, until we started to grow.
And what did you do to improve this?
When it started to grow, we realized
that the only way to store it was to linearly scale that Redis.
Scaling linearly also means scaling costs,
which is something we don't want.
So we did a dispersion analysis of the time that passes
between the auction and the event we want to track.
We realized that 90% of those events
happen within the first hour after the auction.
So what we said was, let's save an hour of events in Redis,
a super fast in-memory database,
and about the rest, when an event that is older than an hour arrives,
we queue it in an SQS queue so that with a Lambda,
which is here in the middle, we can run a query in Athena.
That is to say, we have a hot data layer that is super fast and more costly,
and a cold layer that is less costly, but a lot slower.
But as 90% is here, that's not a problem.
Then a request is queued in SQS, which triggers a query in Athena,
which connects to a bucket, which is over here.
This bucket contains the same information
as this Redis because it flushes every minute from the bidding layer.
This bucket is partitioned by the minute, so the query goes really well,
because we know what minute to go to based on the ID
and that lets us reduce costs
and have coverage for 100% of the events.
It doesn't matter how much time passes once the auction takes place.
I see here another Lambda.
This one, other than using it for Athena, what do you use it for?
This Lambda is super important because this information
that travels from the auction layer to the Bucket travels in JSON format.
As we want to reduce costs,
and Athena, the pricing model, is by scanned data,
it's important to scan the least amount of data possible.
Therefore, what we do is, this appearance of these files
triggers a Lambda automatically, which transforms these files into Parquet format.
Therefore, reading through Athena is efficient.
We scan very few data, and we find events very quickly.
Well, Lucas, I see a platform… cost optimization, low latency.
How can it be improved? Any improvement you see for short, medium, or long term?
We are working on two ideas today.
One is to avoid this intermediate step of saving the files in JSON
and saving them directly in a binary format, which could be Parquet,
to avoid this Lambda and save even more costs.
The other option is, instead of using Athena,
to use Strict Select and trigger the query directly from SQS
and with this Lambda with Strict Select above the bucket.
Thank you, Lucas, for this explanation.
-It's been a pleasure being here. -Thanks a lot.
We'll see you in the next episode of 'This is My Architecture'.