Welcome to 'This is My Architecture.'
Today I have here, Cem from Insider. Hello, Cem.
Hello, Kamran.
Can you tell me a little bit more about Insider?
Sure.
Insider is a multi-channel marketing automation platform
that enables brands to deliver personalized experiences
to their customers.
So we have a really interesting architecture here.
Can you tell me what challenge you were trying to solve?
Yeah, so before, we are big Spot users within Insider.
Before using EKS,
we were managing our Spot Instances via auto-scaling groups,
Spot fleets and Elastic Beanstalk.
So with EKS,
we have consolidated all that
compute architecture into one place,
and we are also solving the problem of data
flowing from multiple channels to the company.
That's great, so getting value out of the data.
- Yeah. - So let's dive in.
Sure.
So a typical, we have integrations with multiple channels, as I said,
so let's take a mobile, our Mobile SDK, for example.
Once the user uses the app,
we get the data through our public ELB, which is provisioned by EKS.
So, that ELB then forwards the traffic
to EKS, to one of,
we have two actually two microservices
that are in charge of receiving the incoming traffic.
So those services, they contain minimal business logic,
they just do simple validations and sanity checks
and then forward the data to our queue solution,
- which is NS Queue. - Okay.
And, within EKS, we use managed node groups.
So that lets us take advantage of Spot Instances very efficiently.
All right, that's great.
So what kind of scale are we talking about here?
So through our public ELB,
we have about 50,000 requests per second.
And this is for write.
This is our write workloads.
And for our read workload,
we have about 100,000 requests per second.
Okay, so you write it into a queue, and then what happens next?
So what happens next is, we have consumers within EKS, again,
and those consumers,
I tie to the depth of the queue via custom metrics,
so our HBase scale our consumers, depending on the depth of the queue.
So, once we write the data,
we read it back into EKS, to our consumers.
And we provision more consumers depending on how much data we have in the queue,
as I said.
And once we read that data, that piece of traffic actually,
we write it three ways.
We write it to DynamoDB, our OLAP DB solution, which is ClickHouse,
and then we write it to an S3 bucket.
Okay, why the segregation of data between Dynamo and ClickHouse and S3?
Well, we use S3 to troubleshoot our problems.
Okay.
What we do is we write every piece of traffic to S3 along with a lot of meta data.
So in case there is like a bad integration
or unwanted data for situations,
we just plug Athena in to S3
to find the root cause of that unwanted data.
We can easily identify it via the meta data that we write alongside the traffic.
So like we can do quick troubleshooting regarding the ingestion of data.
And another use case we have is,
if something goes wrong with our data sources,
we can just replay
the entire traffic between a specified amount of time
from Athena and we just read the traffic back and write it to a queue.
And then the process starts all over again.
It's really powerful use case with Athena as well.
I also see you have a private ELB here.
I'm assuming this is for your internal users.
That's correct. So we have many teams within the company.
And all of them need the data
that is stored within those data sources from us.
So this is how we communicate with the outside world, if you will.
So when a product wants a specific piece of data from us, I mean,
all the products within the company are basically our customers, right?
We don't have products ourselves in this architecture.
When they want data from us,
they communicate it to us via the private ELB,
which then goes to EKS,
which is where we have a lot of microservices
that serve different purposes.
So depending on the use case, the data can be read from DynamoDB or ClickHouse,
and then served back to whichever product requested the data
in a manner that's formatted,
consolidated and sanitized.
So we never allow direct access to data sources.
We do everything through our APIs.
To provide that security for data as well.
- Right, exactly. - That's great.
So, what kind of benefits have you realized from this architecture?
As I said, we're big Spot users.
So moving this architecture
to EKS enabled us to spend less time
on DevOps operations.
And the other thing is, since we have many products,
we have different data stores for all of our products, right?
And they all had their own data solutions at the time.
What we did here is once we centralized all the data processing into one place,
we effectively freed all the other product teams
to actually focus on creating product features
and create business value rather than spend time with data management.
That's great.
So you're getting the value out of the data that you're acquiring.
- All right. - Yeah.
So, Cem, thank you for sharing your work with me today.
Yeah, thank you for having me, Kamran.
And thank you for joining 'This is My Architecture.'