Hello and welcome to, ‘This is My Architecture.’
I'm here with Sripriya from CloudHealth by VMware.
- Hi Sripriya. - Hi, thanks for having me here.
Yeah, we're happy to have you. Can you tell me a little bit
about CloudHealth by VMware? - Sure.
So CloudHealth Secure State is a cloud security
and monitoring platform. We help our customers mitigate
security and compliance risks
through our intelligent real-time insights.
That's great, I'm really looking forward to diving in and learning a little
of that more. So can you start where we would do
the collection piece? - Sure.
So we have primarily two modes of collection.
The main aspect, like I mentioned, is the real-time detection
of security violations. So for that we leverage
the cloud provider's native security event.
You know, like event streams which is like CloudWatch
or activity log in case of Azure.
And once the events are coming into our system,
we have a collection framework which evaluates those events
and determines the current state of the resource
and publishes those resources into Kinesis Data Stream.
Awesome. So as an architect, what challenges
would you face in this architecture?
Right, so one of the main challenges as with any data pipeline is the scale
that we have to handle, right? So we are supporting
three major cloud providers today.
And each of these cloud providers have a lot of services that we also support.
So given that, we are talking
about billions of events per day, and that scale is pretty huge to handle.
And with that topped by that number of resources we collect and monitor,
and to provide real-time insights we have to exercise
a graph-based expiration on those resources.
So those are the primary challenges we have in our system.
Yeah, no, that's great. So you mentioned the graph based use.
I am curious to hear how Neptune plays in.
Sure, so once the cloud resources are ingested into our data stream,
we have the consumer which publishes
those resources into Amazon Neptune, right?
So the primary usage of Amazon Neptune is a graph database, like we all know.
And the cloud resource data is a highly interconnected data set,
and we wanted to provide our customers graph-based expiration capabilities.
Neptune allows us to do that efficiently, and on top of that it also provides
the read scale-out that we require for executing a lot of rules on them.
On the cloud resources. And we also chose Neptune
because it's KMS. So one of the key considerations
in our system is like, as we on-board
more cloud provider, support more resource type,
each of them have varied configurations, and we want our data layer
to be agnostic of those representations.
And that is another reason that we had to choose Neptune.
Yeah, no, that makes sense. How many assets are we talking
about here? - Yeah, I think in terms of, like I said,
we are handling billions of events which amounts
to about 50 million plus resources that we collect and monitor
in our pipeline. And this number is an ever growing number.
Yeah, wow, that's a lot. That's a lot of assets to process.
So after the data goes into Neptune what happens next?
Right, so I think we also wanted to provide our customers capability
to slice and dice this cloud resources data
in various forms and shapes. We wanted them to have reporting
and analytic capabilities, source integrations,
and also we want them to visualize how the resource configurations
have changed over time. So one thing we found out
is there's no one database that can perform
all these more efficiently, so apart from Neptune,
which is our primary for persistence, we do have Elasticsearch and DynamoDB
as our backing source to power some of the other use cases.
Yeah, no, that makes sense. Those are great to use to process that.
But I am very curious to hear more about this area over here.
- Why are we using Sharding? - Sure.
So I think Neptune provides the read scale-out that we require to exercise
and execute the rules, right? We also wanted a strategy
to scale our writes out, and the way we do that
is with our Sharder service.
So Sharder service basically provides us a logical grouping of data
that we want to store in our provision Neptune clusters.
Right? And the Sharder service is intelligent
in itself that it uses this CloudWatch metrics
from Neptune and Elasticsearch to kind of data mine the shard capacity
of these provision emptying clusters. So it uses CPU memory
and fuel the parameters to determine, okay given Neptune is full or not.
And that allows us to efficiently load balance
our customers data in the provisioned Neptune cluster.
So that information kind of flows back which allows us to make decisions
at this front. - Yeah, no, that's really smart use.
So once that data is sharded and put back into Neptune,
where does it go? - Sure, so once the data is all persisted,
the next thing in our pipeline is the rule executions, right?
So the incoming event along with the resource changes
are then propagated to a rule execution pipeline.
The rule engine that we have is smart enough to determine
what rules needs to be executed for a given event
and the resource changes.
And this could be like thousands of rules, right?
That gets executed. So we fronted
with a Kinesis Data Stream, and we exercise these
over the Neptune graph database to perform those security checks.
And if it detects any security violations those are again,
persisted in our DynamoDB, and also we provide
real-time notifications to our customers through various channels.
We have support for Slack, email, Splunk and the like.
That's awesome, that's really great for your customers.
Sripriya, thank you so much for joining us today.
I've learned a lot about CloudHealth Secure State.
- Thank you. - And thank you for watching,
'This is My Architecture’.