- [Announcer] Join me in welcoming
Jackie Lasky and Sarah Yoder.
(audience applauding)
- Do we each get one?
- Okay, okay.
- [Jackie] Hi, so I'm Jackie Lasky and this is...
- [Sarah] Sarah Yoder.
- So we're two cybersecurity engineers here at MITRE.
And today we're just going to share with you some of our
methodology behind TRAM.
Which is our project that we've been working on
to create an easier way to map threat reports to ATT&CK.
So we'll start with how does information
get into the ATT&CK site?
This usually starts with finding open source
threat reporting, so looking at common vendors like
FireEye, SecureList, Cybereason, those kind of things.
And basically, choosing reports based on
the content that's in them.
So we're always looking for new reports on new APT
groups, on malware, things like that.
And then when we have our report,
we'll finally go through and we'll start from
beginning to end reading the report and looking for
different attack techniques that are in there.
And some of you that may have been in the
CTI workshop yesterday have learned that this is a
really manual process and it takes a long time.
We usually start when we're going through a report with
looking for the tactics.
We'll kind of highlight along the way any verbs
or any tactics that we may think and obviously trying to do.
And then when we're done, we'll kind of repeat this process
and look for all the techniques in the report.
So it takes a really long time to be able to cross reference
the tactics, the sentences, and going and looking for the
techniques that they map to.
Yeah, so all the data that's added into our ATT&CK site
is done manually, which means it's hard to
keep it up to date.
We usually often have a huge backlog of reports
that are unanalyzed and we have to prioritize
some over others.
So we also have things like human error, so two analysts
may look at a report and have two different
interpretations of the same text.
This also introduces things like availability bias.
So basically what that means is that you're more likely
to think of the attack techniques that you're already
familiar with or that you already know of,
which can cause you to overlook some of the less common
techniques that are out there.
So it's also a challenge that we have.
And then also training new team members
is really hard to do two.
ATT&CK is a huge site and it's continuously growing
every day, so it's a lot to take in
when you first start here.
So that's where our solution called the threat report.
attack mapper comes in.
We call it TRAM for short, and basically, obviously
as Katie said, this project grew out of a need to be able
to automate some of the stuff that we do here.
So we're just going to give a quick little overview
of sort of the details to our technical approach
as we started this project.
We don't have a lot of time to get in too big of the
details, but you can always find this after.
So the first step in our process was to get data in and
how we did this is we use the stix and taxii server
and basically we just grabbed down all the example
data from each techniques page and we're going to use
that for our training dataset.
So before we could do that, a lot of the texts in the world
is unstructured raw text.
So in some human language. So we had to be able to get it
into some language that the computer can understand.
So this is where a lot of our data normalization
and natural language processing occurs.
So doing a lot of things like removing stop words,
doing our STEMI lemmatization, things like that.
That's all going to occur in this step.
And then once we've got the data in a good format,
We can start building and training our models.
So to do this, we use Python's logistic regression,
which is a supervised learning classification method
and essentially what each technique has is they have
their own model built for them.
So we'll have a positive and a negative class that we build
for each technique.
So the positive class consists of the examples from the
ATT&CK site, and then the negative class consists
of a variety of things, so maybe like true negatives
or sentences that don't relate to any technique all,
or maybe other techniques, examples, to also
differentiate between different techniques.
Yeah. So after we've trained and built our models,
we had to collect a good reliable test dataset that we
could use. So we did this using our own
Feedly CTI RSS feed.
They have a Python API where you can just kind of grab
a bunch of reports that you want to look at
and we use that to collect sort of the test dataset
that we used.
So once we had our test data, we were able to transform
our test data with the models that we built from
the previous step.
So this is where we can perform a lot of our
cross validation and checking how our models are
performing as we go through this process.
And then once we have ran our reports, our test reports
through the models, we get sort of a print out
that the analyst can look at and decide
whether they approve or reject the, basically the
techniques that were found in the model.
So they'll have a list of all the techniques that were found
in the report and they can go through and decide whether
they like them or not and then we can also add in ones that
are missed as well.
And to pull it all together, we have a feedback loop
to have to be able to improve our model as we go through.
So we use a database to be able to track the different true
positives, false positives, true negatives, things like
that, so that we can include it and make the model better
each time and we use a serialized format in Python called a
pickle file so that we can reload and rebuild the models
whenever new data gets added to the ATT&CK site.
So this is still... Along the way,
we had a lot of challenges that we still had to overcome.
But the most obvious challenge is extracting meaning from
text is still really hard to do.
Humans are still needed in this process.
We haven't got rid of us yet, we still are using us.
So we still need to be part of that process
and improving the feedback loop, improving
the models as we go.
But the problem almost always comes back
to not having enough data to train with.
So this can lead to things like imbalanced datasets.
So maybe having a really large negative class
of training examples, but a really small positive
class of examples that are good.
So that's one of the challenges we had
to face as we went through this.
We had some techniques in ATT&CK that don't have any
examples at all, or maybe they only have one or two
examples, so for those kinds of cases, we had to look at
them on a case by case basis and sort of build out regular
expressions and string matching kind of things
to be able to kind of handle those cases
that we couldn't build, build enough models for.
- [Sarah] Cool. So now I'm going to go over
just how it looks.
So, because we have a short time slot, a lot of this
demo we're just going to skip through.
So at this point in the tool,
we would assume that the analyst has gone online,
found a report that they're interested in,
they grabbed the URL and ran it through our tool TRAM.
So when they go to analyze that report, this is the view
that they would see and as you can see, if they click on
the highlighted sentence, it'll show you a couple of
techniques that TRAM found.
So in this example, a spearphishing link and attachment.
Going back to what Jackie said, sometimes the computer
still has trouble distinguishing very similar techniques.
From here, we go ahead and reject spearphishing link,
since that wasn't what the sentence was talking about,
and confirm or accept the spearfishing attachment
since that was correct and as you can see,
it got rid of the spearphishing link from the list
and add spearfishing attachment down
into the confirmed techniques.
So from here, as Jackie mentioned,
the models will be updated based on the accept
or rejects as an analyst goes through this report.
So you would follow this process for the rest of the report,
and you can then export it as a PDF.
So at the bottom of the PDF, you would get a chart that
looks like this, that has all of the confirmed techniques
from the report that you analyzed.
So we hope that this is a useful little snippet
that the analysts can use to share with maybe other
members on your team or for us of the use case.
We'll probably be feeding this into ATT&CK to get more
information into the site.
So why does this matter?
Other than our use case, which was to automate our jobs,
we hope that it's useful for you as well.
We know that mapping reports to ATT&CK can be
kind of overwhelming.
There's a lot of techniques to choose from.
And so we hope that this is a good starting place since
there's only a handful of techniques that the analyst needs
to review per sentence.
As Jackie mentioned, remembering 266
and growing, techniques, is hard.
And there's many techniques that sometimes I forget about
and so luckily the computer doesn't have that problem,
so it can remember them for us.
And next, we hope that you can use reporting that's
important to you. So whether that's an internal report
source or, if you feel like ATT&CK doesn't have enough
information on maybe the financial sector or healthcare,
whatever your use case may be, we would hope that you can
use this to get more data for the techniques
that you care about.
So some takeaways. Many people have talked about
ATT&CK helping to frame behaviors, moving away from
IOCs and we hope that you can use a lot of what other
people are discussing today.
Mainly our four main attack use cases,
which are detections, kind of this sock assessments
and gap analysis can track adversaries you
care about using a cyber threat intelligence.
And then potentially emulate those adversaries.
We acknowledged that mapping data to ATT&CK is hard,
but that's where we hope that TRAM can come in
and make that easier.
So with that, we have some time for questions.
Oh, most importantly, this will be open source and available
to the community, hopefully by the end of the year.
(audience applauding)
Yep? Microphone's coming.
- [Jackie] Okay.
- [Questioner 1] Hi ladies. This is fantastic.
I've been wondering if you've had any, I guess,
experience or if the community's had an experience with
overlapping this with like a tip, whether it's mis or a
commercial tool and, integrating those two things?
- [Sarah] Yeah, so we're still really early in this process
on (murmurs) but, I could see as it gets more mature
and if the community can help make it even better,
that maybe that definitely would be an option to be used.
- [Questioner 2] Just a real quick technical question
on your NLP.
Was that the only library or learning model
that you considered or were there maybe
a couple other candidates?
- [Jackie] We were mostly just using the natural
language tool kit in Python.
It's one that I was already familiar with
and we definitely, both of us
kind of have a small background.
I have a computer science major, but I've never, I don't
have a PhD in AI or machine learning, you know,
so, I just kind of went with what I was familiar with
and what she's was familiar with.
Using our backgrounds and that's kind of
how he went from there in the limited time that we had, so.
Any others?
- [Questioner 3] Sorry, I wasn't if it's my turn.
I got my PhD from YouTube, on machine learning
(audiences laughing)
so we're in good company.
When you do the release and opensource
Jupiter notebook, perhaps?
- [Jackie] Our old code used to be on Jupyter notebook,
but we're actually, we have it in a web app now,
so no more Jupyter notebooks. Unless people want them.
(Jackie laughs)
- Thank you. It's a great effort.
You did great job on this. I think that the community can
really run with this, so I appreciate it.
- [Jackie] Awesome.
Any others?
- [Sarah] We have one in the front here.
- [Announcer] Getting your exercise today, huh?
- [Questioner 4] So if I'm understanding it right,
basically it's going to be a web app
that's going to be hosted by you guys, correct?
- [Sarah] No, sir. It's actually all run locally, so.
- Okay.
'cause the next place I was going was how you were going to
de-conflict potentially if two people submit the same
report and then train it, but.
- [Sarah] Yeah, so right now it's run locally
on the user's system.
Looking, we have looked into having it in more of a
centralized location so that multiple analysts would be able
to use it at the same time, to kind of curb your thought
of deconfliction were we have some features that are in the
works to kind of tag reports that would say like, Sarah is
working on this one, Jackie's working on this other,
so that hopefully the same analyst isn't working
on the same report.
- [Questioner 4] Would you have the option
to send the reports back to you
guys to then update the ATT&CK matrix?
- [Sarah] Yeah. Through our normal
contribution methods, yep.
We would love to get more reports in
that we didn't have to do ourselves so yeah,
definitely interested there.
- [Jackie] Yeah, we think it'll definitely help
increase community contributions that way too.
Cause that'll give you guys a way to find reports, have
them mapped and give them to us as well, so. Great.
- [Sarah] Thanks.
- [Jackie] Yeah.
(murmurs)
- [Announcer] I think Sarah and Jackie have
generously offered to do live demos for folks,
not on stage, but folks in McLean, pull them aside.
It's, it's a pretty slick tool and I'm pretty excited
to hopefully opensource that in the near future,
so there are no other questions, catch them during break
during the reception tonight 'cause they've done
some great work here.
Please join me in thanking Sarah and Jackie.
(audience applauding)