- Returning the Att&ck Con speaker, Daniel Wyleczuk-Stern.
- All right, thanks everyone.
As Adam said, I'm Daniel,
and I'm gonna be talking to you all today
about how to use ATT&CK to measure what matters
and how to do things in the right order.
Anytime you work on something,
you're choosing to not work on something else.
Therefore, when you're working
in a cybersecurity organization
with limited resources and motivated adversaries,
it's important that you make deliberate decisions
to maximize your return on investments
for security projects.
And I believe you can use ATT&CK
to really help make those kinds of decisions.
So a little bit about me.
I'm Daniel Wyleczuk-Stern, currently in Washington, DC.
I currently am a Security Engineer at Snowflake,
before that Praetorian and before that the US Air Force.
I have quite a few cats
but I've managed to subdue them for this talk
so hopefully those shan't be bothering us.
So what am I gonna talk about?
We'll cover some fundamental Assumptions and principles
that I have about cybersecurity and risk.
I'll discuss some of The Problems that I've seen
when making decisions.
Some of the Books and Principles that I've read
that have helped me formulate my hypothesis
and the way I operate with risk.
So to talk about the Main Idea with ATT&CK and how it helps,
and then work through an Example
that really shows the power of using ATT&CK
for quantitative risk analysis.
And then also some Devil's Advocates and Disclaimers,
counter-arguments et cetera.
So what are these Assumptions that I'm operating from?
One, I believe security is technical
and applied risk management.
The goal isn't necessarily to risk elimination,
as a security leader,
your job is to spend just enough to reduce risk
to an appropriate level.
The hard part is that our field is not really good
at quantitative risk management just yet.
When I say quantitative risk management
I mean using numbers to drive your decisions
as opposed to things we've commonly seen
such as low, mediums, highs, et cetera.
This means using numbers and saying
this is measurably more effective
or measurably less effective than this,
and making your decisions based off of that.
Below you see some common formulas using risks.
They're not super complicated
but they can be pretty hard to actually figure out
how to actually calculate them.
So why is it so hard?
Well, one of the things that I've seen so often
is when people do risk calculations
or examples in books,
it comes up with a thing like,
risk it's likelihood times impact.
Well, how do you measure those kinds of things?
It's left as an exercise to the reader
and it's pretty hard to figure out.
So how do we actually go about estimating how much risk
we're actually gonna reduce by implementing a solution?
And then how do we ensure standardization across projects
so that we can measure risks across a suite of portfolios
or suite of products that we're working on,
and make strategic high-level decisions?
Luckily, I think these problems have all been solved
and it's just a matter of applying the theories
to the work that we're doing.
So these three books, I would highly recommend.
The first "How To Measure Anything",
also as a cybersecurity edition, talks just about that,
how to measure anything.
It talks about measurements, estimation,
sort of the intangibles of businesses as it says.
"Thinking, Fast and Slow" is another bestseller
that really talks a lot about, a lot of things.
But one of the things it highlights
is probability and bias,
and humans are naturally bad estimators.
And then "Engineering Trustworthy Systems",
talks about quantitative risk management and cybersecurity.
There's a number of books
on threat modeling and risk management, like FAIR
that I would recommend.
So from these, I drew out a couple of guiding principles.
A lot of these are summarized
from "How to Measure Anything".
Essentially we have more data than we think we do
and you need less than you think you do
in order to reduce uncertainty
and drive improvements in your quantitative risk analysis,
especially risk quantification is possible
through understanding your threats
and doing threat models and threat profiling.
So you see these kinds
of improvements with consistencies and frameworks
when it comes to Application Security.
They use Frameworks to STRIDE
and dread to drive actionable, repeatable
and defendable threat models that you can use to
reduce your risk and make strategic decisions.
However, how do we actually apply this
to enterprise security?
And now we get to the Main Idea,
which is use ATT&CK tactics techniques, and procedures
in your threat models.
Sounds really simple and really easy
but the rest of the sidewalls sort of talked
through how you actually do that.
And some of the benefits you'll get from that.
So how do we get started?
Well, one of the things that's very hard to do
with threat models is get those kinds of measurements
for likelihood and impact.
Luckily when using ATT&CK a whole world of data
opens up to us to get that level of granularity.
So one example is from sort of people
who have measured already, such as the folks at rapid7
who published the Att&ck Frequency diagram.
So if you're ever looking at a threat model
and you're wondering, I wonder what the likelihood
of an ATT&CKer using PowerShell is,
while you have a number right there.
And you may say, well, maybe that's not accurate
to my organization.
That's true most often,
but still one of the founding principles,
one of the principles of "How to Measure Anything"
is that some reduction in uncertainty
is likely better than where you were before.
Or if you do have a unique scenario,
luckily there's tools such as TRAM from MITRE
or rcATT, or CATT, I'm not quite sure how to pronounce it
which allow you to pull attack techniques
from intelligence reports,
and from there you can begin
to collect measurements yourself
and really start to get measurements
of likelihood and probability.
Similarly, for Impact,
somebody has already solved this problem.
And if you're looking at
well, what is the impact of something happening?
That kind of data already exists.
And it's not too hard to go find it.
Again, it's probably not gonna be accurate
to your specific organization,
but it's better than where you before,
where it was just a critical risk.
Now we actually have numbers
and we can begin to measure things
and calculate return on investments.
So I thought the best way to show this
would be with an example.
Let's say you're an organization
trying to look at deploying
a Privileged Access Management system
for your active Directory
or path for just log on's for your Single Sign-On solution.
The first cost 50K a month, and the second five K a month.
Additionally, you've recently had a pen test
where the auditors came in and said,
hey you had a lot of passwords.
You know, ATT&CKers are brute force
in Single Sign-On portals, we recommend this.
So your SSOS is looking
at the second option because it's cheaper
and you've got auditors who are saying, that's a good idea.
However, with threat modeling and measurements
we can use ATT&CK to help avoid these kinds
of recency biases.
So what we're looking at here is a tree threat model,
the kind of threat models
that you'll see in books like "Engineering Trusted Systems"
and a couple of other threat modeling solutions.
Essentially what we're saying here
is that the privileged access management system
would help us reduce the likelihood
of an ATT&CKer obtaining access to a privileged account.
From threat modeling, we decompose this
into four nodes or four leafs,
which we see at the bottom,
Brute Force, Credentialed Dumping,
Access Token Manipulation and Credentials and Files.
And I've correlated those
with their associated Att&ck technique.
Now, because we've done this
we're able to now get measurements from these
and therefore determine overall probability
that somebody would obtain access to a privileged account.
Well, how do we do this?
Some of it can be done from measurements,
such as from a pen test or from compliance audits,
and some of it can be done from estimates and research,
such as Credential Dumping,
and we can have our cyber threat intelligence folks
pull reports and collect data
and say, what is the likelihood of this happening?
From this, we're able to get the overall probability
of about 12%.
Again, this isn't going to be very,
it's not a hundred percent accurate,
but it's more accurate than we were before
where we were saying, this is a low risk or a medium risk.
We now have actual quantitative numbers
which we can use in our calculations.
So what about after we implement the system?
Whether it's through estimates or through a trial
we're able to remeasure and re-estimate
and, you know, through this simulated talk
we get a reduction down to about 2%.
So 12% to 2%, it's pretty good.
It's about a factor of 10.
But that doesn't really tell us the whole story
when it comes to risk.
We also need to know the impact and harm.
So obtaining access to a privileged account
isn't necessarily harmful in and of itself.
This requires us to build out our threat model
a little bit more and say,
what is the ATT&CKer's actual goal?
In this case, we're looking at a threat model
of Compromising PCI Data.
Now our leaf node obtaining privileged access is now a tree,
is now a, sorry, our root node is now a leaf node
in this threat model.
And we can see that Obtain Access to a Privileged Account
in the bottom of the screen.
From our threat model,
we've determined that there's a couple of ways
that somebody could obtain access to PCI Data,
whether that's compromising the PCI database
or compromising the website from Magecart style attack.
For those who don't know, the Magecart style attack
is a class of Att&cks where ATT&CKers
are able to insert malicious code into websites
and steal credit card data as customers place orders.
So from our additional measurements and estimates,
we calculate that without any systems,
without the PAM or the passwordless log-ons implemented,
we had about a 12% chance of a breach of PCI Data occurring.
Using, again the estimates from before
with about maybe 70,000 records
we're saying we have a 10 to $50 million harm
if this were to occur.
Now with the expected harm, which is harm times probability
and we have a range of about 1.3 to $6 million.
Now what happens after we implement
the Privileged Access Management system?
We redo our calculations
and we can see that the Expected Harm
has dropped to 200 to 500,000.
Assuming a cost of about $50,000 a month,
we now have a range for our return on investments.
This may seem like a pretty high and wide range,
and you're right.
However, what we're looking at
isn't necessarily what is the return on investment
but what is the return on investment
relative to other projects,
so that we can make decisions about what we're gonna work on
and what we're gonna work on first?
So now let's look at
the Passwordless Single Sign-On solution.
We have an end now
where an ATT&CKer has to obtain email addresses
and passwords and MFA.
Again, we're able to collect
these two measurements and estimation
and we say that the likelihood
of somebody obtaining access to an account
is 1.2 times 10 to the negative six.
So pretty small probability.
This occurs because they have to Bypass MFA.
And we also have some pretty good password requirements
in our scenario.
So what about afterwards?
Afterwards, we say that the likelihood
of Phishing and Brute Force being successful
probably have to find some sort of bypass
or maybe some kind of service accounts,
has dropped significantly.
And we went from 10 to negative six,
to 10 to the negative 4th,
which is a very very large reduction
in terms of, you know, factors, right?
But how does that affect the impact of the overall risk?
So now we've expanded the right side of the tree
compromising website (indistinct) Magecart style Att&cks.
In this scenario, we're saying about 15% are developers
and 5% are admins who would have the privileges
if they were at compromise
to be able to push malicious code in this scenario.
Now, afterwards, we see that the right side of the tree
has dropped from 10 to the negative 7th or so
to about 10 to the negative 11th.
However, because the left side of the tree
is so dominant in the probability calculation
this hasn't changed the Expected Harm at all.
And we're assuming it costs the 5K a month,
we'd actually have a negative return on investments
when we do these calculations.
So what was the sort of point of this exercise?
We were able to obtain pretty, not necessarily accurate
but pretty realistic numbers
in terms of probabilities and impacts
and make a decision relatively quickly about which project
would yield a higher return on investment
and really reduce risk in a quantifiable way
that we know now we're making the correct decisions
about where to allocate our funds and allocate our time.
And ATT&CK has helped us
in terms of pulling up those likelihoods
and the data is readily available to us.
So isn't this just threat modeling ATT&CK?
Yes, it is.
But ATT&CK really helps
with the threat modeling calculations
to make a quantitative risk analysis
and really help with those measurements
and calibrated estimates.
You may see, you can't just make up numbers.
Well, in this scenario, they (indistinct) measurements
and also calibrate estimates.
And I highly encourage you to read
"How to Measure Anything"
as it talks about how to do estimation.
And you may say, this example was too simple, and it was
but I was constrained on time.
And also without trying to get
an exact return on investment,
our goal is to compare projects
to determine where we would get
the most on return on investment
and allocate our resources there.
So this, you have to invest time in calibrating your people.
I say estimates, and I wanna separate that from guesses,
and "How to Measure Anything"
really talks a lot about how people are pretty bad
at estimating probability.
And we have to be trained on how to become good estimators.
And I recommend starting with
a focus like cyber threat intelligence,
'cause they're gonna be the ones who have to do
a lot of these kinds of estimates
and have to pull from reports to draw conclusions.
And this can also be pretty hard.
Threat modeling and (indistinct) of itself
is a task that a lot of mature organizations struggle with.
And when you yard out estimations and measurements
and calibrations, it can become even more difficult.
It's something that at my current organization
we're still learning and improving on.
So I think I'm good on time.
Are there any questions?
- Great presentation, Daniel.
That was fantastic.
You got a lot of interest building up in Slack
and as usual, it's welcome back.
It's always great to have a familiar face.
So, you know, big question here is,
you know you've done a lot of research
and a lot of good highlighting
of how you kind of forecast problems
and kind of like you said,
see opportunity, see, you know, problems and address them.
But how do you calibrate that later on with,
once you've instilled and kind of installed,
you know, these practices, these policies
that you've kind of forecasted,
how do you calibrate that back towards, you know
real world operations, comparing, you know
what you saw versus what's happening
to kind of, you know address some of those uncertainties
that you highlighted?
- So I think what you're asking
is how do we sort of keep our models up to date?
Again, that's one of the benefits of ATT&CK.
A lot of folks or organizations will sort of store like,
hey what's our detection probability for this technique?
What are our risk mitigation's that we have applied to this?
Along with that, we can keep our model
of what do we think is the likelihood for this occurring,
and what are the systems that we have in place for that?
So again, something like password brute force thing
where we can say what we have,
you know, our MFA solutions,
we've had our Passwordless log on solutions,
and we can keep our repository
of what do we think is the likelihood
for this technique occurring.
And even locking that back to certain systems
where maybe we have, you know an old legacy database
where when we apply,
when we master the technique of this database,
there's this probability,
and then when we match it with our Single Sign-On solution,
it does a lower probability
but then when we do these lookups, we can say sort of where,
what the probabilities are for those systems.
- Okay, that makes sense.
And then, you know,
just kind of, you mentioned
this was a very notional, very simplistic model
just for the sake of kind of highlighting this process,
but, you know for those starting out
that were really interested
in kind of pursuing a similar approach,
you know what is your advice towards,
you know, selecting, you know
the subset of all of ATT&CK or even beyond ATT&CK
to kind of build as a foundation for these models?
- Sure.
So this is more of a top down approach
versus a bottom up approach.
So I wouldn't say, start with ATT&CK,
I would say start with systems and risks.
So let's say you're looking at implementing a new website
or a new database,
those are the kinds of business decisions
that should drive this risk analysis.
You shouldn't say,
I wanna start with cloud TTPs and look at my cloud threats
because that's, the risk modeling should be driven
top down by your systems
versus bottom up by your threats
when it comes to determining sort of where to look first.
That's my belief.
- That makes a lot of sense.
And I think, like you said
ATT&CK gives you kind of the coverage
just to understand like where those problems may lie
as well as, you know
just kind of the perspective on the bigger picture there.
I think another great question,
as well has just come in from Slack was,
you know regarding your models,
is there any way to kind of materialize
and actually, you know, manifest tangible actionable results
from those uncertainties?
I think someone mentioned like Monte Carlo simulations
and things like that to actually quantify those.
- Yeah.
"How to Measure Anything" talks a lot about,
and I think also "Thinking, Fast and Slow"
talks about using Monte Carlo simulations.
So when you get these estimates
in these ranges of probabilities,
you should really start with a range.
So I said, 10 to the negative one.
You could maybe more accurately say,
well we think it's five to 15%.
Then when you plug these into an Excel Spreadsheet
then you can do some Monte Carlo simulations.
You'll get a range of risk that will sort of bottle up.
It can get a little complicated,
that's why like for this scenario
and when you're first starting,
I recommend just keeping it simple
and seeing if you can get a decision that is clear
without having to do those kinds of estimates.
And again, "How to Measure Anything"
talks about how to get the value of additional data
when it comes to doing your measurements.
Like these problems are solved,
like just read books on it.
I encourage you to explore the areas
of risk and risk quantification.
- Well, thank you very much.
And great presentation.
We definitely have a couple of questions
building up in Slack.
So definitely hop over there to address those
but with that imma turn over to Adam.
- Yeah, and thank you so much Jamie and Daniel.