Hi, and welcome to 'This is My Architecture.'
Today, I'm with Andreas from 1plusX.
- Hi, Andreas. - Hi, Ross.
What can you tell us about 1plusX?
1plusX is a Swiss company who has a software-as-a-service product
which is a data management platform.
And this is a software or product that media companies, publishers, and marketers
can use to ingest, process, enrich, and activate the user data.
All right, and what part of 1plusX are we going to be talking about today?
So we're actually going to talk about the core of the data management platform
where we go from events data to user profiles.
All right, I'm looking forward to it.
Why don't you start taking us through what we see on the board here?
Yes, so we can try to follow an event as it flows through the system,
so it's a streaming based system that has Kafka as a backbone.
So when we start with an event,
where a user visits a website of our customers,
say the user reads an article about the Olympics,
then the event will come in here into our ingestion component
which runs on ECS.
Here, it will be cleaned, processed, and prepared for the internal processing.
And it will go here to Kafka and will also go to S3.
Then in the next step, we go to the aggregation and machine learning step.
So here we actually go from event-level data,
so data that says a user did something at a certain point in time,
to attributes for users, so making statements about users.
A simple attribute could be what is the total number of events
that this user had in the past,
but there are also more complicated attributes
and that's why I put the machine learning here.
For example, to predict the gender of the user based on their past behavior.
This is happening in this step.
After that, we have the data again in Kafka
so the outputs here are the user attributes.
We go to the next step, the profile step.
So, again, here we have a connection.
So here we basically combine all the user attribute data
that we have computed in this previous step.
Since we had, in the previous step, different components
that we're computing user attributes here, we combine them in that profile step.
And we actually also have a storage here, a Couchbase database
and also an Elasticsearch to represent the data.
What we also do here is we compute so-called audiences.
Audiences are groups of users that are customer-defined.
For example, the customer could define, "I want everybody who's interested in the Olympics."
And since our user that you are following through the system
here has now shown that they're interested in the Olympics,
this user will be part of that audience.
This information goes back to Kafka and then goes into the serving part of the system
which is again an ECS service with a Couchbase database.
And here the data, this audience membership data,
is indexed such that it can be accessed when the customer clears it.
Okay, so that provides some level of separation
between, we have, real-time streaming, essentially, here coming through Kafka
and you storing some sort of view of that data here.
How does that affect the design of the system?
What happens if your real-time streaming goes down?
What is the effect on how your users are able to continue using your system?
Yes, so in general we want, of course, be fault tolerant, we want to be highly available,
but we're also dealing with massive volumes of data.
So a typical customer would have multiple billions of events per month
and tens of millions of profiles.
And we want to do this whole thing cost-effectively.
So the solution that we came up with for this
is that we have two different fault-tolerant strategies
for the periphery of the system ingestion and serving part and the core.
So for the periphery where we have the interfaces to the customer,
we have distribution to multiple availability zones.
We have autoscaling, load balancers, everything.
We have replication in the database and so on,
so we try to keep this working at all times.
In the core, we actually embrace failure instead of trying to avoid failure.
So we have this Kafka, we have these services, and these databases
only in one availability zone.
So some of those databases are even not replicated.
You can say, "Okay, how can you do that?"
So the thing is we have mechanisms
to recover the data whenever we have corrupt data anywhere
that we can employ within minutes to hours.
Basically, recreate any state here in any part of the system.
Okay, I want to go back to a number you dropped there when you were describing that,
when you spoke about billions of events for a user.
What affect does that have on scaling the infrastructure that we see?
Yeah, so all the components that we use have to be inherently scalable.
So we use ECS Fargate.
We don't run our own clusters.
We think it's great that we can just run services as containers, basically,
and don't think of clusters,
but then we can use autoscaling also here to adjust to changing load.
For example, at night, of course, we have less load,
less traffic on websites than during the day.
We have EMR here which we use in combination with S3 actually.
So we don't have an HDFS.
We don't have one single EMR cluster.
We have the data in S3 and we dynamically spawn EMR clusters
based on the demand that we currently have.
And we can run hundreds of nodes if we need to for a big job
or just five nodes for some small experiment.
Okay, and then one other thing,
seeing you're a Swiss company, you have a lot of customers within Europe,
GDPR always comes up in these discussions.
How does that affect what you have?
Yes, that's a challenge. So under GDPR,
users have the right to delete their data at any point.
So when a user requests the deletion,
we have to make sure that it's gone from all these databases
within a reasonable amount of time.
So this conflicts with a typical pattern around Kafka here
like this event sourcing pattern where you would typically keep your event data for very long.
Such and in case of a failure or some kind of need to restore a system state.
You can go back, you can basically rewind and just replay everything.
Now, if this data is user data and this user has requested the deletion of their data,
you cannot do that. That's problematic.
In Kafka, you cannot really delete records by key.
So what we came up with here is mainly in these components,
so actually when we go from events to user level data,
we use a combination of batch and stream processing.
And we only keep a short history of events in Kafka
that automatically expire, so the user deletion is granted.
And we keep files for historical data in S3 where we run deletion jobs.
And we compute the result of the whole component of the whole aggregation,
maybe the event count that I mentioned before or the gender probability
as a combination of results of a batch job running on EMR.
And the computation that runs on ECS on the newest events.
That is great to hear.
It's always a challenge for customers to implement GDPR correctly.
Andreas, so I got to say thank you for joining us today
and sharing the architecture of 1plusX.
Thanks for having me.
And thank you for watching 'This is My Architecture.'