Welcome to another episode of `This is My Architecture.´
I'm Aarthi, and joining me today is Peter from Imperva.
Peter, thanks for joining us today.
It's a pleasure to be here.
So, Peter, tell us what Imperva does.
Imperva is an application and data security company.
We help organizations protect their applications, APIs and databases,
whether they're running on premises or in cloud environments like AWS.
Great, security is job zero for all of us, so I am really interested to see
what architecture we have here.
- Can we go get into the details? - Sure, absolutely.
So, on the left-hand side here,
we have Imperva's different products that we offer.
We have our DDoS Mitigation Capablities,
our Cloud WAF Network, our Bot Protection Solution,
and our Runtime Agents.
Okay, so what problem are we trying to solve here,
with, like, these Agents that are running on premises or in the cloud?
So, as you can imagine, there are lots of attacks out on the internet,
and every single month Imperva blocks trillions of attacks.
Trillions of attacks means lots of data,
and so, one of the big things we looked to AWS to help us solve
was how we move that data from all of our individual products and sensors
back to the relevant systems
where we need to display them back to our customers
or even to their own environments.
Okay, so we are looking at that data ingestion and moving it
into a place where your end-users can see it,
as well as Imperva can make use of that data.
Why don't you walk us through how that's done?
Sure. So, for customers that are leveraging
either our DDoS Mitigation or Cloud WAF Network,
what will happen is
both those solutions will be able to protect their applications.
If any sort of attacks come in, we basically go
and we'll see event data coming in,
to manage streaming for Kafka.
Now within here, all of this different event data
works through a couple different lifecycles.
So, the first one that we have to worry about is,
how do we go and normalize the events, enrich them,
and then transform them to be able to deliver to their final destinations?
So, you talked about normalization and enrichment, right?
Are these done by specific subscribers that are subscribed to these topics?
Yep, exactly, so we have certain subscribers that will go and subscribe to them,
all of those running over on EKS,
and ultimately our goal for it is to be able to enrich
and be able to add additional context,
as well as to be able to send it off to its downstream destinations.
You mentioned a large scale of this data from your producers, right?
How did you end up choosing manage streaming for Kafka?
Previously we were doing a lot of batch processing,
and, ultimately, one thing that we found
was that it was too slow for a lot of our use cases.
So if we take a look at the Cloud WAF instance,
anytime there's an attack,
you need to be able to quickly investigate that incident or event.
In our old systems this was taking upwards of 20 minutes
for us to be able to deliver it back to our customers,
on site or on premise analytic services, systems like Elasticsearch.
So, by moving to MSK,
we were able to drastically reduce that time
that it takes to move that back down to less than 3 minutes.
Previously, we were always looking at about 20 minutes
to be able to go through and do all the batch processing
and delivery of that data to their environment.
Got it, and that's a good reduction, right, like from 20 minutes to less than 3 minutes?
For end customers, that's always critical to be alerted immediately
as they see any security events.
On the MSK, do you have multiple pipelines for each of these subscribers?
Yeah, the way that we generally will look at it
is we have 2 main sets of pipelines that we have to work with.
The first one is for event data, so these are all the different security events.
One of the big concerns that we have for event data
is a lot of this data is also subject to GDPR and all sorts of privacy regulations.
So, there's lots of data sovereignty requirements for it.
So, we replicate the different event topics all around the globe,
and we're able to ensure that we're not actually impacting
the customer's privacy and regulations.
The second topic that we have here is really focused on metrics.
Now, metrics are all the different data
that each of these individual services and sensors produce
that make it easy for the organization
to be able to understand what's exactly happening.
Great. That's a very good use case for MSK.
You mentioned here that you input, ingest data
into a customer's Elasticsearch or a SIM environment, right?
From a customer's perspective, they've got this data.
What do you use the other environments here for?
Yep, so the next major use case that we have
is we send this data to our research data lake.
So our threat researchers take all of this data,
and they understand attack patterns on the internet.
So they understand how botnets are going and leveraging
new modern exploits to attack our customers.
The benefit by us processing and sending that to our research data lake
and using technologies like EMR to process it
is we're able to very quickly identify these new attackers,
push that information back to our customers,
and then they can automatically gain that protection
through our Cloud WAF Network and other solutions.
And that's a good level of analytics here, where you ingest data back
to say, "Hey, we've figured out that there's much more that we can find out,
even at the initial layer."
And I do see EKS at the bottom. What do you use EKS for?
We actually use EKS for pretty much everything,
but one of the biggest use cases that we have
is we have all the event data which goes into various different data stores.
But then we also have all the metric data.
The metric data we need to be able to use to display back to our customers
and their dashboards, as well as being able to pull it via our APIs.
And so, we use EKS and a set of time-series databases
that are hosted on EKS to store this data
and make it immediately available for our customers.
Okay. This is a neat architecture
of how you've actually moved from your earlier batch processing to MSK
and using different producers to make sure
that security of your customers is always the number-one priority.
Thank you for sharing this architecture with us today, Peter.
And thank you for watching `This is My Architecture.´
[For more information visit aws.amazon.com/this-is-my-architecture]