Hello and welcome to this episode of 'This is My Architecture'.
I'm Kapil, and with me I have Gireesh from Eyeota.
Gireesh, welcome to the show.
Thanks, Kapil. It's good to be here.
Glad to have you.
And, Gireesh, can you tell us a little bit about Eyeota?
Eyeota is an audience data marketplace with a global footprint.
We provide a privacy compliant way for advertisers
to reach their target audiences with precision
and for data providers to monetize the data with transparency.
And, Gireesh, you've built a new data platform on AWS,
can you tell us what were the challenges you were trying to solve?
The primary business problem we are trying to address is data quality.
We are certified by both the IAB Tech Lab and by Neutronian the gold standards
in ad tech data quality certification.
The way we achieve this quality is by using our machine learning algorithms,
which are designed to optimize the trade-off between scale and data quality.
In technical terms it translates into a few different requirements,
one is scalability, given the volumes of data we process,
flexibility for our development teams,
extraction from the infrastructure layer
and transparency on unit cost economics,
so that all of this comes together to increase delivery velocity.
Great.
And could you walk us through the diagram?
I see a lot of moving parts here.
Sure.
So this is really, you can think about it as a model lifecycle,
and we articulate that in code...
In the form of Airflow DAGs,
PySpark scripts, and SageMaker notebooks.
We also have our raw data,
which comes to us from our supply side partners.
And both of these, we stored in S3.
From S3 we use Airflow to pick up...
To orchestrate our date pipelines.
So this then it goes into the top over here,
essentially what's going on is we're using Glue
for pre-processing and feature selection.
That's where the PySpark scripts come in.
From there the data flows into SageMaker
from model training and tuning,
and also from here data flows into SageMaker batch inference,
which is where we run...
Which we run on our held out data sets.
Right.
Down here on the inference side, again,
Airflow is triggering a pipeline with Glue doing pre-processing.
And from there the data is going into SageMaker
for batch inference.
Remember, this is unlabeled data,
so the generated labels are then stored in S3,
from there it is distributed downstream to the rest of our pipeline.
And in the middle over here we have model selection
and pipeline optimization,
so both of these pipelines they push their performance
and operational metrics into CloudWatch
from there we have an internal metrics visualization tool
which we run on EC2.
That helps our data science teams figure out what is the right model
to push into production for each use case.
The selected models are pushed into S3,
and that's what is used by the batch inference job
to run on the production workload.
Our data science team also uses visualization
to monitor the production pipelines for things like model drift,
at the same time our data engineering and infrastructure teams
they also use the metrics over here
to ensure pipeline uptime
and also look for optimization opportunities.
Right.
And I can see there are clearly two different types of pipelines here.
Can you tell us what's the typical scale of data processes by these pipelines?
So up here from model training and tuning,
we run a few hundreds of training jobs,
each of which is processing
anywhere from a few hundred megs of data
to a few hundred gigs of data.
Down here on inference we run a few hundred inference jobs
each of which is processing terabyte scale data.
Ultimately we are processing like
six or seven billion unique advertising ID's a month
and mapping these into 70,000 different attributes.
So all of this is running on a ginormous distributed sparce matrix.
Fantastic.
And you are doing all of this orchestration from Apache Airflow.
Can you tell us why Airflow?
Primarily, it gives us a lot of flexibility
for both our data science and data engineering teams.
So typically a data science team needs to work
on very experimental features and they can do that on Airflow,
but the data engineering team is optimizing production,
scale, performance, so they can do that as well on Airflow,
so it's a single platform that both of them can use
and it's great for our velocity.
And what was your experience like integrating from Airflow with AWS services?
It was quite straight forward to orchestrate the services.
It was mostly a question of following the dots,
but then when we got into final-grained configuration,
we faced a bit of a learning curve,
for example, we needed to figure out the right metrics to monitor here
so that we could figure out how many DPUs to process for Glue,
to configure for Glue,
for SageMaker we needed to play around a bit
with distributed learning and model averaging to get it to scale to our sizes of data.
Right.
That's a lot of fantastic work done by your team.
What would you say are the key outcomes of this project?
Team efficiency.
I mean, right now we get most training pipelines
configured within three to five working days,
which is half of what we used to do earlier.
And we get this with a lot more transparency on model performance
and also a lot more configurability across the board,
so it's a huge win for the team.
That's fantastic.
Gireesh, thanks a lot for sharing all these insights with our viewers.
And thank you for watching this episode of 'This is My Architecture'.