This is My Architecture GumGum Verity:
Low Latency Contextual Analysis and Data Privacy at Scale on AWS Outposts
Welcome to This is My Architecture.
My name is Andrea,
and I'm here with Florian from GumGum.
Welcome to the show.
-Thank you. -So what do you guys do?
Yes, GumGum is a technology company
that is on a mission to erase the use of personal data
in digital advertising.
And to do so, we power Verity,
our very own contextual intelligence platform
that provides the ability for clients to classify text,
webpages, images, video, and audio content.
Great. This is on AWS.
-So let's dive in. -Sure.
-And find out what it does. -Sure.
So most of the work happens in the region here.
So here our clients will go, and talk to our API, and say,
for example, I want to classify gumgum.com webpage,
-and know what's in that webpage. -Okay.
So the first flow is users interact with API Gateway
that will then communicate with our EKS cluster,
on which we run our main API RESTful API project.
And this API will do a database lookup on DynamoDB
to see if we have any information about gumgum.com.
I see. And if that information isn't there,
how do you do the contextual analysis?
So, as part of the EKS cluster,
we have another component that act as an orchestrator.
And this orchestrator will also initiate a classification request
back to our entire GumGum AI.
So the Gumgum AI stack here is made of
natural language processing microservices,
as well as computer vision microservices.
Wonderful. So you run your custom models to do the contextual,
and then do you store that information back in DynamoDB?
Yeah. So there's a whole back and fold here
in between ECS microservices and EKS.
And the component here will keep updating DynamoDB
until we get the classification fully complete
and return it back to the client.
Awesome. I'm super curious with Outpost, right?
And talk us through what was the requirement
that led you to pick Outpost for this use case?
Yeah. So as you guys know,
the AI is a process that is sometimes slow.
And achieving a high throughput,
and really low latency on AI inference is quite challenging.
The system here wasn't built with latency in mind,
but just serve different type of use cases, classifies video,
and a whole set of different things.
But certain clients here have just the need to classify webpages.
So the Outpost here, and for this specific client,
we needed to achieve a latency of less than ten milliseconds.
And that was our SLA. And this is doing 100K requests per second.
So this is really intensive.
Okay. So walk us through the flow rate.
So I see users clients say, "I need that sub ten millisecond."
That's low latency. What happens?
Yeah. So the interesting part here is
that we've been redesigning our API product that runs here,
and we rewrote it in Go so that we build it
with direct latency in mind.
So we have this API product here running on EKS.
So we can call it our Edge API.
And this Edge API here, running on EKS,
will simply do lookups in Elasticache.
-I see. Okay. For retrieval of information. -Yeah.
So essentially have your cluster API request coming in here.
Now, talk us through the rationale now,
what size is this, and what's your cache hit ratio?
So, part of the design of this whole outpost was
to really consider how do we want to leverage
those EC2 machines and to do what,
and how many of them we needed.
So, what we plan here is that the Elasticache cluster is 1TB big.
So we dedicate a bunch of nodes to run the Redis cluster.
But here, as you can see, there might be a challenge here
where we have 15TB of classification data in DynamoDB,
and basically, not all the data here fits in the cluster.
And if we can talk
-about eviction process, -Yeah.
what's your cache hitch ratio, and what's the eviction process?
How often do you store new information on it?
Yeah, so new information is being stored just in real time.
We keep adding new data because this client keeps asking for new pages.
And the web is dynamic by nature,
so we keep getting new information.
What matters for these clients is to get
as much contextual information as possible.
So the least frequently used page is not really useful
to be stored in this cache,
given that we have only 1 TB.
So we use a list recently used eviction policy,
which means that our Elasticache cluster is always full,
and always evicting data to store new contextual data coming.
And assume that you have not seen that webpage before,
so there's no contextual information here, what do you do?
Correct. So, to close the gap on the other side,
so here we have a replication process in between this region
that will sync up the data from Dynamo to Redis.
But now on the other side, this component here,
the Edge API, is also aware of,
where do I go if I need to initiate a new classification?
And this message will be sent back to the main region,
to another component here that we run on our EKS
so that we can just resume the first loop that we went through.
So you have an orchestration
essentially kicking off that whole AI/ML analytics,
-and then bringing back onto Elasticache. -That is correct.
Awesome, Florian. Thank you so much for walking us
through this contextual platform that you built
for low latency requirements.
And we truly have appreciated this overview.
Thank you.
Thank you for watching This is My Architecture.
Thank you for Watching