Ladies and gentlemen, thank you for
watching our presentation about the
usability of playbook frameworks.
So to jump right in, playbooks at their
core are really just a series of
step-by-step instructions that allow
incident responders to gain momentum
during stressful situations. Whether a
security system, organization, or a network
has been compromised or breached, these are
the first steps that a security
professional is going to follow to
identify the risk, quarantine it, and
mitigate it.
So playbook frameworks are actually the
guidelines that allow technicians to
then create and implement the playbooks
within their own environments.
So across industry, playbooks and their
frameworks are in use, they're adopted,
they're popular, but really the usability
is unmeasured. And so what we wanted to
do in our study
was to look at it in two different parts.
First, ask a series of
individuals that deal with security day
in and day out to use a framework, build
playbooks based off of common scenarios
they'd likely encounter in their day
jobs.
And then the second part is, we
actually use the playbooks that we
developed
during a series of no-notice cyber
attacks on a network and see how useful
or not useful the playbooks are at
dealing with the threat.
So in the first study we recruited 12
participants
and gave a series of instructions on how
to use two popular frameworks. We then
asked them to design playbooks for two
scenarios, the first of which is brute
force attacks trying to guess the
username and password to gain
unauthorized access to a system. And the
second one is valid credential misuse,
where you have a username and password
already and you attempt to gain access
to it, but
you're not supposed to.
And so based off these 24 playbooks,
we exported those to a series of
industry professionals
that are, for many cases,
chief information security officers at
major organizations and ask them to
evaluate would the playbooks that were
developed actually work within their own
security environments?
If so, why? If not, why not? So some of the
feedback that we got from this study, we
saw that many of the participants that
we had recruited appreciated the
proactive nature of sitting down
thinking about problems and coming up
with solutions before they had to do it
on the fly.
Additionally many of them noted that
playbooks were easy to learn. However,
despite being easy to learn,
many of the playbooks that were produced
were deemed insufficient by our expert
panel.
The most common feedback that we got
from our experts were that incident
response in itself is not linear. You
need to have robust plans that account
for
various paths that an
incident response event may take.
Additionally,
let's say there's an inexperienced
individual that's supposed to
complete
a task,
or let's say it's just a really
stressful situation, and they
cannot recall how to complete a task.
Having playbooks with enough detail to
prompt to overcome that obstacle
was a key recommendation of our
experts.
So,
after looking at the 24 playbooks that
were produced in the first study, we
selected the the best playbooks and
adopted them in a real security
operations center.
And then we gave them to the
professionals that work there and
asked them to use them during a series
of three no-notice attacks on their
networks that were conducted by a
trusted insider.
So what we found,
of the five participants that we used
throughout the study,
two of the more seasoned professionals
said that playbooks were nice, but they
really relied more heavily on their past
experience
when dealing with the particular problems at
hand.
Additionally, some of the more junior
technicians identified that when they
were stressed, they found themselves
fumbling through
the playbooks, that there really wasn't a
comfort level there
and that they were losing their place,
they were going page to page, and
we understood that there was a key gap
between novice individuals and more
experienced individuals. So after two
iterations of a no notice attack on the
same organization,
we did a pause, and we identified where
individuals were failing to identify
problems, quarantine it, and mitigate it,
and we implemented a series of revamp
measures. I think the most important
one that was conducted
was assigning seasoned mentors to novice
technicians and having them coach them
through, on a regular basis,
how to use the playbooks, the ins and
outs of them. So if you spend 30
minutes, during lunch, sitting down
talking through it, and so that when you
had to go do it in real life,
they were more comfortable with it.
One interesting thing to note was during
the third no-notice event, attack, that we
had conducted,
one of the employees was brand new.
She had onboarded using the playbooks,
and she said that she was actually
confident in her ability to identify the
problems and to come up with a solution
because the playbooks helped her
understand what her role
was in the organization. And she also
felt empowered on how to effectively
communicate with her peers and
her supervisors during the event to
keep everyone informed on what was
happening.
So there are numerous recommendations
that we have in the paper, which I highly
recommend you go read.
But in short, jumping to the key findings,
what we found were playbooks and their
frameworks were useful,
but not in isolation. There had to be a
number of additional organizational
reinforcements that had to take place
for them to be effective,
in our observations.
And so
what this means is, for your
organization, you may or may not have any
success at all,
just going to download a playbook off
the internet and saying, today we're
going to use this playbook for incident
response. There's a series of tailors
that need to be made, or tailoring,
to make it more
in line with your
organization's behaviors and practices.
Additionally, taking the time for
training and resources is a must,
and additionally identify where the
other gaps might be. Are there any policy
or legal hurdles that are in play
preventing your technicians from having
a successful
response to these type of events?
So looking at it holistically,
playbooks can help,
but they're not going to help by
themselves.
So with that,
if there are any questions or comments, I
would love for you to reach out and I
really appreciate you all
watching this presentation. Thank you