Welcome to 'This is My Architecture.'
I'm Boaz and I have here with me Opher from Nielsen Marketing Cloud.
Hi Opher. - Hi Boaz.
- Tell us about Nielsen Marketing Cloud.
- So Nielsen Marketing Cloud is a data management platform.
We prepare marketing segmentation data that can then be used for campaigns.
- So basically you're getting files from different resources
and then sending them to the different networks?
- Right, so we basically create all the data in another part of the system
and then my system which is called DataOut
gets these files on this side and then processes them
and uploads them to the ad networks that are partners of ours.
- Okay, so let's walk through the system, what it does.
- Okay, so I think the first thing I want to mention is that
this system runs a huge amount of data.
We process about 250 billion events a day.
- That's a huge scale.
- Right, and these events come in files
each file has a bunch of events in them, which are the segmentation data
it gets processed and gets sent to the ad platforms.
The files pretty much come into the system
and are written to an S3 bucket.
They get some processing like transformations and formatting
using a Spark EMR cluster.
Get written to another S3 bucket and from here,
they go to Lambda functions that do all the processing of last formatting
and sending it to the ad platform.
- So you have your process data and then you're just
taking it and splitting it to the Lambda functions?
- Yes, correct.
So the Lambda functions actually do the last part of the work of
uploading it to all the platforms.
There is an additional layer of data here which is all the data about the files
and how to manage them.
So this information goes through here and gets written to a Postgres RDS database
We have a little work manager Lambda that reads the information,
makes decisions which files need to process in which way,
sends it to the Spark cluster, that gets written back
to the database.
- So all your metadata is basically storing RDS,
and you keep updating it to the flow? - Yes, correct.
We have a second work manager Lambda that basically
reads information about the process files
and then does like a fanout architecture of invoking many, many Lambdas
that then upload it to the networks.
In total we have over 100 networks we upload data to.
Once these Lambdas are done, they update back through an SQS queue
and this gets updated back to the database.
- Some status like 'process this file.'
- Correct and if it needs to be retried and stuff like that.
- So this looks like a huge system, 250 billion requests per day
I assume you had some challenges with it?
- Yeah, so I think the main,
the three things that are really interesting about the system
are the following:
One is scale, the other one is rate limiting,
which is something surprising, we didn't expect that we'd have to do.
And the last one which everybody is worried about always, is cost.
- Okay, so obviously 250 billion events a day,
scale is very, very important. Let's talk about the scale of the system.
- Okay, so our top day ever, we had 55 terabytes of data.
On that day we got 17 million files
and we had about 30 million Lambda invocations.
- This is one day? - Yeah, in one day.
And the system basically managed it without any problems.
We were constantly following it and just surprised at how well it worked.
Some of the... in some minutes we had
something around like 3,000 Lambdas up in the air
and concurrently running. - Okay.
- The interesting thing about it is that on a normal day,
the lowest hour in the day has about one terabyte of data,
but the peak hours have about six terabytes of data.
So the system has to breath up and down as the data comes in
and basically scale up and down throughout the day constantly.
- Do you need to do anything to scale it up, scale it down?
- This is really the nice thing about it,
so because of the serverless Lambda function architecture,
the system basically does it on its own, we just invoke a lot of Lambdas
and it scales up and as the data dwindles down,
the amount of invocations go down. We pretty much get it for free.
- Out of the box? - Yeah.
- So I assume with 3,000 Lambdas invocated at the same time
rate limiting can be a challenge both internally and externally?
Can you tell us about what you did to prevent those challenges?
- So originally, we didn't think that's an issue,
but we got some emails from some of our partners that,
"Guys, what are you doing, you're killing our servers."
And we started checking and obviously,
we could handle any burst of data that would come in -
we'd automatically just handle it. But the world doesn't work like this.
So, some of the systems there that you send so much data,
it's kind of like running a DDoS attack against them.
So we had to do something and just to be good neighbors,
and we introduced a rate limiting mechanism.
So the way it works is, we have some smarts in this work manager Lambda,
it basically invokes a lot of Lambdas, so it needs to know...
Yeah.
So it makes some intelligent decisions based on the size of the files
and the more of the events in them and so forth.
And we could pretty much rate limit on anything we want
that really important to our partners. - Such as?
- We can limit on megabits per second,
the amount of http connections open, number of requests,
and a few other things like that.
- And can you actually limit those specific networks?
- Yeah, so that's pretty much configured per network
and then we adhere to the limits that we set.
So for example, if the limit, we have one network that limits...
we set a limit for 250 megabits per second
so that if we get more data than that it will just stay in the queue and wait
and then we'll just gradually send it out.
The other interesting thing about it is that we had our own DDoS attack.
So, as the number of invocations of Lambdas grew,
originally we were writing back to the Postgres database,
but we ran into a limit of the number of connections
you can open against a database and starting to have...
it became a choke point.
So we introduced another queue as a buffer
that all the Lambdas report back to the queue.
And we have one Lambda that picks it up in batches of events and writes it...
- And you don't need it in real-time, it can wait the for as long ...
- Usually it's a few seconds, but it's fine,
we just need an update if the file was successful or not.
- Okay, so let's talk about costs, so I assume that size of a system,
many of our customers are looking at it and say,
"This is pretty, this is probably pretty expensive."
- Right, so I think the main thing we worried about
as we were building the system is making sure
we have costs under control.
The system today costs us about $1,000 a day
so that's around $300,000 a year.
And as we were building this we really wanted to be in control of the costs
so we started measuring cost.
So the way we measure it, we started out by
the costs were about $7.7 per billion events
and we had a goal to reduce this. It took us about a year,
today we are at about $4.25 per billion events
and still going.
- So can you tell us a bit, how did you get from this $7.7 to $4.25?
- Okay, so the nice thing about Lambda as opposed to other architecture
where you're paying for the servers and then if you improve your code,
you're still paying for the servers.
With Lambdas, the minute you improve your code,
and the Lambdas are more efficient, it immediately translates to cost savings.
So we had a few initiatives to reduce cost,
one of them for example is that we realized there is a trade-off between
how long the Lambda function runs, and the memory footprint.
If you reduce the memory footprint, you save on the cost.
You might increase the length of the time it runs,
but it might still be worthwhile. doing this...right.
So it's kind of like an optimization problem.
So we wrote a simulator that could simulate different loads on the Lambdas
with different memory sizes.
We experimented with it and run a few hundred different combinations,
and found the exact sweet spot of where it's gonna be optimized
and then improved our code so it'll work exactly in that sweet spot,
and that drove the cost down.
Another thing we did is we played around with the number of http connections,
or the request we send in each connection.
And we found that with, you know, the sweet spot there
that, you know, if we send more connections
we're waiting for all of them to complete.
And it's gonna take maybe longer, but the total cost may be lower,
'cause we're sending more data at the same time.
- So we had a very interesting system here
with a huge scale of 250 billion events per day.
Rate limits that actually solve some of the problems you had
during the implementation of the system. And of course, the cost issue
that you solve very nicely. Thank you very much Opher
and thank you very much for watching 'This is My Architecture.'