- So without further ado,
Brian Donahue from Red Canary.
- First off, like thank you Adam,
thank you to the ATT&CK team for organizing this conference
and for allowing me to come and speak at it.
Apologies in advance if my talk gets interrupted by
any dogs barking or the pitter patter of
toddlers running flat footed above my head,
but such as the reality of life in 2020.
So without wasting any more time,
my talk is starting over with sub-techniques and it's about
the lessons that we learned at Red Canary.
When we remapped from like the old precept techniques
version of Miter ATT&CK to the new sub techniques version of
ATT&CK.
So, before I go any further, a few things about myself,
I work at Red Canary.
I primarily either like support publication of content or
work on it myself, importantly, in the context of this talk,
I am one of the lead writers of our threat detection report,
which is like an effort to gather all of the threats
we detect that our customer's environments and sort of rank
and stack the Mitre attack techniques associated with them
for the purposes of sort of like determining which ATT&CK
techniques are most prevalent in like creating sort of a,
a ranking order to them.
Prior to joining Red Canary,
I've worked as a journalist and as a CTI analyst,
about 10 years ago now I was interviewing
for a job
as a social media editor at Threatpost.
It would have been my first job, It was my first job.
The editors there asked me how I had any experience with
social media, I lied.
I told them I did.
When in fact I did not,
they either believed me or didn't care, but they hired me.
And that's how my career started here.
So if you dislike me or my talk,
you can send your hate mail to Paul Roberts and or
Dennis Fisher.
Now that I've got that out of the way.
So we've already gone through who I am.
I'm gonna give you guys
just an absolute bare minimum background
on what red Canary does, so that you can sort of
understand the context around our remapping effort.
Then we'll run through the seven lessons we learned,
and then we'll close with a minute or two
question and answers.
So like one of the primary things Red Canary does
is manage, detection and response.
So we gathered telemetry from our customers and points.
We parsed that telemetry against like some thousand of
detection analytics to surface potential threats.
Then those potential threats are reviewed by a human
on our detection engineering team,
that detection engineer can
either suppress or hide that event,
or they can confirm that it's a threat
in which case they ship what we call
a confirmed threat detection to a customer
for the purposes of this talk,
that's effectively a customer notification.
It's more than that, but like for our purposes
just to understand that it includes
among many other things.
The Mitre ATT&CK techniques that
that threat is corresponded with.
So given the way our system operates,
we have a few different options for how we can handle
ATTA&CK mapping.
We can do it at the behavioral analytic
or the detector level.
The benefit of doing it at that level
is that it's highly scalable.
You get to make your mapping decisions
when you write your detection analytic.
So you make that decision once
and it's automated from there on out.
The problem with that is that it's bad at nuance.
So you might have a theory about what your detection
analytic is gonna catch.
And then in practice,
it might catch something altogether different.
A second option would be to do this at the detection level.
So if we go back that last slide,
the detection level would be at this human analysis level.
The benefits of doing it at the human analysis level
is that it is good at nuance, right.
You've got all of the context.
You need to make a determination about what you are,
what Mitre ATT&CK technique is associated with it.
However, it's difficult to scale
because it slows down your detection pipeline
because you have to make a decision and choose from 200,
cell mode techniques.
Every time you detect the threat,
the third potential option is like the why not both option.
And I'm actually gonna get into that
later on in this presentation.
So we actually map at the analytic level.
So we make the determinations about what our analytics,
what technique they should map
to like when we're developing them.
And the reason we do this is because
primarily Red Canary's exists to detect threats.
So when we adopted Mitre ATT&CK a few years ago,
we made the decision to optimize for threat detection
and speed since like, you often times we're detecting things
of high priority rather than like being
a 100% accurate in our mapping all the time.
So what this looks like in practice,
the red text at the top here is a detection analytic.
It basically is looking for a process that is an ltest
And to that also includes a command line with domain trusts.
They're all trust this maps to domain trust discovery.
And it's basically a technique that adversary used
to make determinations about domain trusts
within an environment.
This is irrelevant for the purposes of this talk,
but historically this detector has helped us catch a whole
lot of trick bot activity.
So, now that you understand sort of this
high level background,
the first lesson that I'd like to impart on you is like,
if you're going to map things,
or if you're gonna transition
from old Mitre to sub techniques,
like the first thing you need to figure out
is all of the things that you were mapping
to Mitre ATT&CK in the first place.
So for our purposes,
that would include behavioral detection analytics,
which I already sort of explained.
It would also include Atomic Red Team tests.
So Atomic Red Team is an open source
adversary simulation, platform that,
that we manage and maintain in all of the tests in it
are designed to simulate ATT&CK techniques
and they're map to ATT&CK techniques
for the purposes of this talk.
You can kind of think of these two things,
the behavioral analytics and the atomic tests
sort of interchangeably,
'cause like the atomic tests are
isolated scripts that map to ATT&CK and simulated behaviors.
The analytics are isolated bits of Ruby code
also mapped to attack, designed to catch techniques.
We also map a lot of our research and educational content
to Mitre ATT&CK,
but like a lot of it is derivative of our detection
and we didn't go back and retroactively map any of it.
So that didn't really involve any upfront work,
for your purposes you might be mapping
threat intelligence reporting to Mitre ATT&CK.
You might have like charts and tables that you're using
to determine things like detection coverage,
or like any variety of security controls, alerts, IFCs,
and the like.
So.
Sorry.
Having some issues here.
So, once you,
figure out what you need
to map to ATT&CK, it's time to do the actual mapping.
Sorry, I'm having a little bit of trouble controlling
the screen right now.
Alright, there we go.
So first things first, well I guess it's the second thing.
So let the code do the work for you, okay.
So, luckily a Miter ATT&CK created a useful little JSON
crosswalk script and effectively what that'll tell you is
like a tech, it'll tell you what a technique remaps through.
So maybe a technique becomes a para-technique
and it'll tell you the sub-techniques under it.
Maybe a technique becomes a sub-technique
and it'll tell you the para-technique above it
or possibly the technique if it's like scripting
has just deprecated.
So we took,
that JSON crosswalk script,
we built a little power shell script
that sort of helped it work specifically
with our environment and like for atomic Red Team,
we kind of naively thought that the code
would do all the work for us.
We were quickly disabused of that notion
for the behavioral analytics as I'll get to in a minute,
we sort of knew that we were gonna have to do
a good deal of manual review there.
So, once you get to the point where you realized
that you're gonna to have to do
a human review at some level,
you have to make a choice of whether or not you want to
divide and conquer or do it as a small team or individual.
So the benefits of doing it as a team obviously
are that it will get done faster,
although it might get done faster
at the cost of it being less consistent.
The benefit of assigning it to an individual
or a very small team, is that it might be more consistent.
And then the cost of that might be that it takes longer.
So if you have a reasonable number of things
you need to remap, then by all means,
keep it to a small team or an individual.
However, for our purposes,
we've got thousands of active detection analytics.
We've got 500 plus from atomic Red Team tests.
And like, I don't even know how many,
but there's some like 200 plus ATT&CK technique
and sub-technique combos.
So that was just too much work for any individual to do.
So we assembled a team consisting of our intelligence team,
our threat research team, our detection, engineering team.
And most importantly, not me.
So all credit goes to those three teams.
I stood back and watched and asked a lot of questions
and wrote a blog about it eventually.
So I stole this quote from my colleague, Katie Nickles,
but ATT&CK mapping is an art; It's not a science.
So particularly if you're gonna have a team
that's doing this work,
you should create a style guide,
because people interpret techniques in different ways
in a style guide.
Like the other thing to remember is that
we're kind of learning this on the fly, right.
Like these, these sub-techniques are new.
So a style guide will help us learn like how to map things.
It will keep some guardrails on how we map and it will help
enforce consistency, equally important
is to assign a review team.
So we assigned a team of senior detection, engineers,
and intelligence analysts to review every single mapping
decision that was made by the broader remapping team.
This also helped to enforce consistency
and it helped us catch errors that we may have made
in the mapping process.
Lesson five, fix, find them, fix your legacy mapping issues.
So I'm gonna tell a little story.
So when we were working on the threat detection report,
like the first thing we do is we pull the data and we see,
and we make a pivot table and we see like,
what are the top 10 Mitre ATT&CK techniques from last year?
The initial data poll revealed that masquerading
was the top technique.
So we then went and we made a sub list
of the detectors that converted to a threat
that was mapped to masquerading most often.
And these two detectors showed up.
The first one looks for untrusted binary, spawning,
suspicious windows service hosts.
The second one identifies SPC host
executing with a suspicious pair of process.
All told these two detectors on their own
or in conjunction with one another
triggered on more than 4,000 confirmed threats.
However, when my colleague and our
principal intelligence analyst,
Jeff Felling went and reviewed all the individual detections
associated with those to a behavioral analytics,
he made a startling discovery.
He discovered that those two analytics actually
were representative of Process Injection
and not of masquerading.
So we manually remapped them, we run in the numbers again.
We also manually remapped a few other analytics,
but they weren't nearly as dramatic as these ones,
and process injection felt to number one
and then masquerading became number six.
So this is a great lesson.
Like this is the perfect time to review
your behavioral analytics.
Also probably good to be reviewing them on a continual basis
just to make sure that nothing changes over time.
Lesson number six, considering the alternatives.
So I promised you that there was a third sort of why not
both option for where we could do our attack mapping
and here's why I'm gonna talk about that.
So again, mapping at the detection level
is the most accurate,
mapping at the analytic level is the fastest.
However, we're developing an idea called
conditional mapping.
We haven't really operationalized it yet,
but the idea is that we're gonna try and identify
ambiguous analytics.
Maybe they're just broad.
Maybe there's some sort of other problem going on,
but the idea is,
like you pick out the ones that don't always alert
on the thing you hypothesized, they would alert on,
and then you require a human review of those.
So you're still able
to keep the main benefit of mapping at the analytic level,
because most of the mapping is gonna get done there.
So you still have the speed.
And then you just identify that small subset
of problematic detectors and you require
a human review of them.
Now the seventh and final lesson is
just to have fun with it guys.
Just kidding, that's not the seventh lesson at all.
The seventh lesson is actually a lag back.
To give back to the Mitre ATT&CK community.
So the ATT&CK team is constantly looking for feedback.
They probably get less of it than they need.
Whoops,
so,
ways you can give back.
Obviously the most obvious way to give back is probably to
contribute techniques back to the Mitre ATT&CK.
But I think also like they want feedback on what's working
and what's not working.
And by all means you should reach out to them
and let them know, suggest improvements
and whatever else you can think of.
Finally, like if you learn something
while using a tech that might be useful to the rest of us,
write a blog about it, or give a talk about it.
And then while we're on the topic of blogs,
I know that I only had 14 minutes here and I'm talking
incredibly fast, like a mad man.
So if you wanna take more time with this talk,
we wrote a blog version, Matt Graeber and Katie Nickels.
And I wrote a blog version of this talk
maybe a couple of weeks ago.
The link is right there.
I'll try to remember to tweet it after this talk,
if any of you follow me on Twitter.
And I guess without further ado,
we can sort of jump into the Q and A.
- That was fantastic Brian, thank you.
That was, that was really great.
Like I said, i really appreciate
from coming from the ATT&CK side,
we did so much work to kind of produce
and publish these sub-techniques.
So we are really excited to see like an organization
as large as Red Canary adopt it
and actually make it their own.
Like you said, I love the quotes coming off that blog,
the art of this, as well as all of the nuance.
So definitely I've seen a lot of traction and kind of
activity going around about your talks.
So definitely stay queued up for some questions.
One thing I just wanted to ask you was,
you kinda mentioned how kind of artistic and free
kind of free spirited this mapping is.
And one of the things I was wondering is,
do you think that as advice to other organizations,
this is an all or not effort, or can you map, like,
can you map certain portions to sub techniques and then
maybe stay behind and other areas,
you mentioned how like nuanced Red Canary is with the,
you know, Atomic Red team and the detections
as well as looking at the matrix itself.
Like, do you think there's areas to carve that up,
or do you think it's the,
a binary yes or no kind of question.
- Yeah.
So I think that I'm interested in talking to other
organizations about this.
'Cause I actually think that like in a weird way,
Red Canary is like designed to be mapped to Miter ATT&CK.
And I think that there's a lot of people that would have a
lot of a harder time with that.
Like specifically if you're doing like a lot of IOC
in signature based stuff,
I can see how it would be difficult to,
to map things through ATT&CK.
And also if like you're dealing with a lot of alerts
that maybe are false positives,
that I could also understand how that could be a problem,
but yeah,
for us like the Atomic Red team stuff
and the detection analytics really, really teed us up
to have like a relatively easy job of, of mapping to ATT&CK.
- Yep, tracking.
Thank you and thanks again.
Excellent talk, like I said, I think the experienced wisdom
that you guys have got kind of leading the way
and doing that mapping is really greatly appreciated
and you know, big benefits of the community.
So thank you.
But with that, I think we're right on time
to turn it over back to Adam for
our introduction of our next speaker.
- Great, thanks so much.
- Yeah, thank you.
Both Jamie and Brian.