[MUSIC PLAYING]
ANTON CHUVAKIN: Hello.
This is Marshall Heilman from Mandiant
and Anton Chuvakin from Google Cloud.
Today, the topic is how to prepare
for threat detection and incident response in the cloud.
For many organizations, public cloud computing
brings up challenges that they don't really
know how to solve in terms of preparing for incidents,
in terms of even understanding what an incident means.
So I'm going to highlight some of the differences
and similarities between cloud incident
response and on-premise incident response.
So before we go there, we want to understand
the motivation, the why.
Of course, we all know public cloud computing is a big deal.
But the thing is, it's getting bigger.
More companies are transforming.
More organizations are building capabilities in this area.
And that means more attackers are interested.
Marshall?
MARSHALL HEILMAN: Yeah.
Thanks, Anton.
And from Mandiant's perspective, we
are absolutely seeing attackers follow this trend.
As more organizations are moving their critical assets
into the cloud, as more users are interacting
with cloud-based services like the G Suite or like M 365,
we are seeing attackers target those services
and target those assets because that is where users
and data are migrating to.
As a matter of fact, this problem
is becoming so pervasive, not just in the cloud,
that insurance companies have raised insurance premiums 79%
in 2021.
That was after doubling their rates in 2020.
So this is obviously a huge problem,
and attackers are understanding how to go and then get data
in the cloud.
Now, if you want to read a little bit more on the subject,
the Google GCAT team issues an annual report
called Threat Horizons.
And Mandiant issues in an annual report called M-Trends.
ANTON CHUVAKIN: Perfect.
But before we go further, we want
to understand a little bit what it means to have a cloud
security incident.
So to me, the critical dimensions
are it's a security incident that touches
cloud assets or happens inside the public cloud environment.
Data theft from cloud storage, theft of cloud compute
resources, using cloud infrastructure
for malicious purposes, these are
all clear and obvious examples of cloud security incidents.
And of course, you see a lot of these headlines in the news,
both on the data theft side and as well as on the theft
of computing resources.
Now, before we go further, I want
to understand what challenges organizations face with dealing
with such incidents.
Marshall?
MARSHALL HEILMAN: Yeah.
Thanks, Anton.
So the first challenge from a very high level
is simply skills.
Understanding how to investigate systems
that are on-premise or in a data center
is not exactly the same as understanding
how to investigate cloud-based assets,
where assets are a bit more ephemeral than they were.
You have organizations that have multiple cloud environments
running their systems and the data passing back and forth
between them.
There's also the joint nature of performing incident response
in the cloud.
As you can see from the graphic on the left,
you can see the shared responsibility
model between Google Cloud and their customers.
So as an investigator, you're not necessarily
going to have access to all assets and data from the victim
organization.
You're going to have to go to the CSP
to get access to those assets and those data.
That is a change.
And finally, there's the data itself.
In many cases, for a cloud-based compromise,
the amount of data that you have access to
is simply overwhelming.
There's just so much log-based data
generated from cloud assets.
Additionally, the type of telemetry that you see
is going to be very different in the cloud
than it's going to be in an on-premise environment.
And we're going to dive into some of these details
in the next couple of slides.
ANTON CHUVAKIN: Perfect.
And before we go there, we understand
that some of the same challenges affect us
at the stage before IR, when we are actually detecting threats.
To me, while a lot of people obsess
about the volumes of telemetry, it's
the alien context behind detections.
The IP address is not just an IP address.
It's not a server.
It's maybe an indicator of some ephemeral asset.
Or it may be an endpoint.
Or it may be an API endpoint.
To me, it's the difference in context,
lack of skills in some SOCs, and just lack of understanding
how cloud functions.
That's what makes detection response challenging
in the cloud.
However, basic machinery, to some extent, is the same.
So let's talk similarities first.
Marshall?
MARSHALL HEILMAN: Sure, Anton.
So when performing an investigation,
there are definitely some similarities
between an on-prem or data center
environment and the cloud.
First is data preservation requirements.
The way that you preserve data may be different,
but the fact that you have to preserve data,
so whether it's a container or an image or log data,
there are regulatory and legal requirements
that organizations have to follow
and investigators have to follow to prove
the accuracy of what they found at a later point in time.
That doesn't change whether you're
in an on-prem environment or cloud environment.
The second thing that stays the same is you
have to have an understanding of the environment
to know what you're investigating.
It's very hard to investigate what you don't understand.
And so whether it's on-prem or in the cloud, understanding
how technology works, understanding how systems
communicate with one another, understanding where
critical data is, all these types of understanding
are critical regardless of the environment
that you are investigating.
The third one is that standard investigative techniques
have not changed.
You start with a fact.
You start with something.
Whether it's a detection or a piece of information
that someone gave you, you know that something is wrong.
You start with that particular fact or that piece of data,
and you spiral the investigation out from there.
You don't go and create a couple hypotheses
on what could have happened and go investigate
all these different hypotheses.
You start with a fact, that one piece of data,
and you figure out everything that happened around the fact.
And you keep spiraling and spiraling
until you understand the totality of the incident
in that particular environment.
Fourth, log data.
Regardless of the environment you're in,
it needs to be retained, as I said,
from preservation requirements.
It needs to be normalized.
It needs to be correlated.
It needs to be analyzed.
You have to deal with a huge amount of data
regardless of the environment you're operating in.
I used to say, when I was an investigator,
that time is the bane of any investigator's existence.
And it is still true today.
You have to deal with time zone issues.
You need to make certain that you're paying attention
to the time zone that the log data entries are in so
that you're not off by a couple of hours.
You need to make certain that you pay attention
to time skew, which is where different assets may
have the time set incorrectly.
So maybe you're off by a couple seconds
or a couple minutes or, in one case,
you're off by a couple months.
And I mean, can you imagine the havoc on investigation
if it looks like the attacker is doing certain things
months apart when really, it was done seconds apart,
but you just didn't properly account for time skew?
So time continues to be an issue in any investigation regardless
of environment.
And then finally, each incident is different.
It doesn't matter if it's the exact same attack
group going after the same type of data
in two different companies.
The reality is, the investigation
is going to be different because the environment is different.
And even if the environment is the exact same, the company
or the business realities, the operational realities
for that particular company, they're going to be different.
And so how you handle it, how you
react to that particular incident
will be unique for every single one
regardless of whether you're on-prem or in the cloud.
ANTON CHUVAKIN: To me, the most fascinating part is exactly
the last point, is that the similarities
between on-premise IR and cloud IR
is that every incident is different.
But guess what?
If your incident involves multiple clouds
and on-premise environment, you can thank these challenges
for dealing with more time skews, more types of log data.
But relying on similar investigative techniques
is critical because that's what gives you
an island of sanity in all this multiple types of data,
multiple types of operational environments,
multiple companies, in case of a CSP.
So to me, this is an interesting bundle of things
if we apply it to multicloud and mix
of multicloud and on-premise environments.
But here, these can be used as five pillars
to understand that IR is ultimately the same.
But wait a second.
Aren't there also big differences?
Here, we want to highlight the fact
that while understanding the realm is important,
the realm itself is quite different.
And it's different not just because we
use containers versus servers or VMs hosted somewhere else.
It's really the tech but also the operational realities
of the cloud.
It's ephemeral nature.
It's autoscaling.
It's APIs.
All this means that the realm itself,
where we are investigating, where the attackers
act, where we are detecting threats,
where we trace or backtrace the steps of the attacker,
it is different.
It's built differently.
And sometimes I always want to say
that it's built on different principles,
that assumptions that were true in the data center just flat
out aren't true anymore.
Even if you are of the lift and shift variety, where you copied
your virtual machines to the cloud, still a lot of things
have changed significantly.
And again, the shared nature of it is there.
Now, the second point sounds kind of obvious.
Of course you need deep technical expertise.
But to me, the critical bit is not just
understanding how containers differ from VMs
or what storage as a service really means.
It's knowledge of technology but also
knowledge of the different practices,
different operational realities.
Sure, you may know how containers work,
but do you know how DevOps works?
Do you know what's an SRE, how they
do things, how they run things?
How modern IT, for the lack of a better term,
does things differently in cloud environments?
Why do some enterprises release software every year,
and some cloud companies release software every day?
That changes things in a deeper level.
And us, as investigators, will have to deal with that.
Now, different baselines and norms
is an expanded version of this item about similarities
and differences.
The baselines, the learning the environment,
sure, they're all useful items, as we highlighted.
But the machinery and the methods
and the technology for this is just different.
So when we say understand the environment,
it may not mean which server is located on what floor
like you have in a data center.
It may mean understanding how code is pushed,
understanding where data flows, understanding where what
is encrypted with what key.
All this matters, and what all that-- a lot of this
is different, even if you more or less copied your environment
to the cloud as a first step.
Now, we highlighted log retention as item number four,
and there's a similar item on our what's the same list.
But guess what?
As I highlighted on the detection slide,
the context for a lot of this is different.
You would not have user such-and-such
connected from IP address such-and-such
to server such-and-such.
That's out of the window.
You would have logs and activities coming from deep
inside the applications.
You'd have things mentioned in services, microservices,
connections, data flows.
And a lot of this would be referring to objects
that just flat out don't exist.
And what's in your logs?
Sure, you have logs.
You have a volume of logs.
But what it means, you'd have to learn.
So it connects your items one, two, and three together
because otherwise, unless you understand
one, two, and three, your four wouldn't
produce useful evidence for investigation.
Now, we do talk quite a bit about being reliant
on CSP, Cloud Service Provider.
You're not reliant for Cloud Service
Provider on all the tasks.
You can collect the logs.
You can pull things over APIs.
You can pull logs off instances.
Now, cloud in 2022 isn't what it was in 2012.
There is a lot more ways to collect useful telemetry.
There are many more methods, cloud-native
and more traditional, again, to get data.
But there will be things where you are reliant on a cloud
provider for some access.
Marshall, you had a fun view on the different norms
in the cloud.
Care to share your perspective here?
MARSHALL HEILMAN: Yeah, Anton.
Absolutely.
We've definitely had a couple situations
where when we were performing an investigation,
all the data that we needed to really piece together
what happened between the different assets in the cloud
wasn't available to us.
And we've seen this happen a couple times where that data is
actually contained in service logs that the CSP has access
to because they're providing that service,
whether it's infrastructure or platform
or software as a service.
And so we had to go to the CSP to actually get access
to that data.
So this wasn't customer data.
This was Cloud Service Provider data
that we needed access to to be able to completely piece
together the totality of what the attacker did
in the environment.
Not only that, when you find yourself
in situations like that, obviously, you
need a specific skill set and a specific understanding
of how those assets and how those services are operating
in that particular cloud environment.
And even for us, professional incidence response firm,
that was difficult.
We had to spend some time really learning,
how do these assets operate in the cloud?
Why is an attacker doing anything?
And why aren't we seeing access--
or why aren't we seeing some of the data
we would have expected to see?
Well, it's because in a cloud environment, it's different.
You have to go to a CSP to get access
to certain types of logs.
And you have to really understand
how some of these different platform and infrastructure
and software as a service services are working
and how they interact with customer.
ANTON CHUVAKIN: Yes.
And this, to me, is a useful item
that ultimately, a lot of this would
be a three-way dance, customer, CSP, and an IR firm or possibly
an MDR service provider.
We covered some of the key differences
and some of the similarities.
And hopefully you have a bit of a structure in your mind
about how to prepare for incident response.
But we want to highlight some of the specific items you
can do to prepare today.
So Marshall, care to start this one?
MARSHALL HEILMAN: Yep.
So the first thing you can do is baseline your environment.
Understand your norms.
What we mean by that is during an investigation,
time is critical.
You want to make certain that your investigators are
able to understand your environment
and get answers to their questions
as rapidly as possible.
That is so much easier if you, as an organization,
really understand how do your as--
where are your assets located?
How does data flow throughout your environment and so forth?
For example, if you've got a set of servers
that are ephemeral in nature, and maybe they get refreshed
every 24 hours, you don't want an investigator
running around and figuring that out a couple days
after the fact.
You want the investigator to know upfront
that you're simply never going to find that system.
They'll never be able to perform live response
or forensic analysis on that system
because it no longer exists.
However, because you have a well-set-up environment,
you've captured operating system and application
logs from that system.
So we know, as an investigator, to go and look
at the log data for that particular system.
The second aspect here is understanding your security
profile.
You want to make certain that you are protecting your crown
jewels.
You are protecting your assets in the best possible way
against attackers that you know are going after you.
Now, I want to give a little bit of an example here, Anton.
So there have been a number of times
where I've worked with usually smaller companies.
And when I'm talking to them about why an attacker is
in their environment, I will get answers like,
well, I don't know, or we don't have any critical data.
There's no reason for an attacker to be targeting me.
And in almost every single case, when
we are able to identify the data that the attacker was going
after and we show that to the customer
or to the victim organization, the responses we get back
are something like, oh, no.
That is who we are as a company.
That is our intellectual property.
I can't believe they took that.
And it's like, aha.
You do have some critical assets.
You do have a critical data.
And understanding that helps you, as an organization,
figure out, what type of security
do you need to put in place to protect those critical assets?
The third here is to prepare logs and telemetry.
The worst possible time to realize
that you need data you didn't save
is during an investigation.
You want to know ahead of time that you're
saving the majority of data that will
be needed in an investigation just in case.
Similarly, with technology and skills,
you want to make certain that your team is trained properly
to understand how your environment operates
and to answer those technical questions that an IR team is
going to have of you specific to your environment to save them
that critical time.
There's also conducting realistic red and blue team
exercises.
So one of the best things you can do as an organization
is bring in a--
bring in an external organization and say,
here are my crown jewels.
Here are my assets that I cannot have an attacker get access to.
See if you can do it.
And try and prove it.
And if someone is able to access those assets,
you want to go and implement security measures
that prevent them from doing whatever it was they
did to gain access to those assets
because again, the wrong time to learn that your assets are
vulnerable is when an attacker has access to your environment.
The right time is when you hire a reputable external firm
to come in and do that work for you.
And then you put appropriate security measures in place
to prevent it from happening again.
And finally, change your security program
to match cloud needs.
We oftentimes see organizations that
have a security posture that worked
for them in-- or for their on-prem environment.
But when they move their assets to the cloud,
especially if they do what Anton talked about earlier, that
lift and shift, they think that their security posture
remains the same.
But the reality is that it doesn't
because the cloud operates very differently than on-prem,
as we just spent the last few minutes talking about.
And we need to make certain that our security
profile appropriately matches our cloud
environment and those critical assets in the cloud.
Anton, do you have-- do you have more to add to that?
ANTON CHUVAKIN: To me, yes.
I think I almost want to double down on the last point,
is as a lot of people do show up in the cloud
and they have to not just learn cloud,
but they have to map their mental models
to what happens in the cloud and what are the cloud realities.
To me, this is probably one of the critical points
from the entire--
from the entire show is that they have to adjust.
There will be adjustments in what they do, how they do it,
what they learn, as well as other things.
So while we are at the end, we wanted
to highlight that in the near future,
a lot more exciting things in this area
would come from Mandiant and Google Cloud
doing things together.
This applies to both intelligence,
transforming operations, and of course,
utilizing some of the similar tooling for this.
Marshall, you had a point about GRR.
MARSHALL HEILMAN: Yeah.
So what I want to point out is that Mandiant is standardizing
on using the GRR rapid response open source utility that Google
publishes to make certain that all
of the forensic capabilities that we need
are going to be built into that tool going forward.
And we're going to be leveraging that
along many of the powerful EDR solutions
that we are partnering with to make certain
that there is one particular forensic tool that
has access or has every single capability that we
need as an investigative firm.
And I would be remiss if I didn't talk about some
of the great things that Google Cloud and Mandiant, now that we
were part of the same organization,
are going to be doing in the future.
So first is we're really looking at really transforming
how security operations and incident response
is done not just in the cloud but also
in on-premise environments by taking a lot of our technology
and combining it.
We are also going to be taking data
from two of the most intelligent powerhouses in the world,
Google Cloud and Mandiant, and combining it
so that our customers get the best benefit from the best
intelligence in the industry.
And finally, we are going to be supporting
the GRR rapid response open source forensic utility, which
applies to both on-premise systems as well
as cloud-based systems, to make certain
that all the capabilities that we need as an incident response
firm is built into this utility so that the entire community
benefits.
ANTON CHUVAKIN: Perfect.
And if you want to learn more about things
we covered, we highlight the links to Threat Horizons
reports, Google Cybersecurity Action Team, and, of course,
Mandiant blog and M-Trends resources.
Take a look.
Thank you very much for your attention.
[MUSIC PLAYING]