Hi, my name's Daniel Votipka.
I'm a PhD student at the University of Maryland.
And in this video, I'm going to talk about the work my
colleagues and I and the Security Privacy and People Lab
did, building and validating a scale for secure software
development, self efficacy.
Producing secure software is important.
And developers play a critical role.
Unfortunately, we still see several solved vulnerabilities.
These are bugs attackers can use to malicious ends
crop up and real software.
These are vulnerabilities that have a known solution,
either an API, language, or best practice
a developer could have used to prevent them.
This chart shows the number of vulnerabilities reported each year.
Each line signifies a different type of vulnerability as
classified by NIST's common weakness
enumeration categorization.
Note that buffer errors still remain
in the top two spot; information leak disclosure,
improper access control, and cryptography
issues have barely or not at all decreased.
And yet they're supposed to be relatively solved issues.
Given current libraries and APIs, why are
we still seeing these vulnerabilities in the real world?
Well, one reason is that security is really complicated.
NIST's National Initiative for Cybersecurity Education
or NICE Framework identifies 44
unique tasks.
A developer must perform to ensure their software is secure.
This list here shows just a sample of that.
And you can see sort of the complexity inherent in this challenge.
And unfortunately, prior work has shown that many developers
do not believe they are qualified
to perform these tasks.
In fact, other prior research has shown this to be the
case when they actually had developers
either trying to write secure code
or trying to find vulnerabilities
in insecure code.
So how do we help developers write more secure code?
Well, there are several different methods currently in use.
One is teaching them through active practice and
other educational tools.
There are also several guidelines outlining the considerations
developers need to make when thinking about security.
And finally, there are analysis tools that developers
can use to alert them when they might
be doing something wrong.
And APIs to handle complex security concepts for them.
But do these actually improve
secure development practice?
How do you actually measure skill improvement
in this domain?
Many prior studies have used secure development assessments
where participants are given a small requirement specification
and asked to write a secure program or try
to find a vulnerability in known insecure code.
Unfortunately, this can take a large amount of time
for participants to complete and does not scale well.
We propose an alternative approach to measuring secure development
skill, specifically a validated
psychometric scale.
Human behavior researchers regularly develop scales
to measure elusive phenomenon that cannot be observed directly
due to cost or complexity.
Specifically, we propose measuring
developers' secure development self- efficacy,
or their belief in their ability to write secure code,
which correlates with actual skill
in several other contexts outside of secure development.
Now, the scale would measure developers belief
in their ability to complete secure development tasks,
such as identifying security problems
during software design or employing
secure programming languages.
We refer to the scale as SSD-SES for short.
So to develop the scale we followed Netemeyer's
four step process.
We began with construct definition and
defining the content domain.
And here we tried to clearly identify the
targeted construct scope.
The underlying idea.
And we were focused specifically on tasks related to secure
production or code production.
This meant we were focusing first on only development tasks.
Nothing related to deployment or maintenance
or operations, only that first phase.
And we also are only looking at common tasks,
things prescribed by widely accepted secure development
frameworks or experienced security experts.
Next, we tried to generate
and judge a set of items
creating an initial pool of candidate questions,
and then we wanted to ensure that they were relevant
to the construct, that is the scope that we
defined previously and that they were understandable
by our developers.
And for the first part of this, for the item generation, we looked at
five of the most popular secure development frameworks all
given here and for each, two
researchers independently reviewed each framework and all the
tasks identified or described by them.
And we identified fifty seven
unique tasks from these five frameworks.
Then we took those fifty seven tasks and we presented them
to a set of 22 security experts who had on average,
20 years of experience.
And here we asked them to rate
each item's appropriateness
to that specific content
definition that we we had defined previously.
And the goal here is to identify content validity,
to make sure that those tasks were associated with that construct
we're trying to measure.
And we also allow them to suggest additional items.
So after this phase, we ended up with
a total of 58 remaining items.
Some were removed, some are added.
And then we took those to a set of eight professional software developers.
And we had them actually go through each
task reading and thinking aloud
as they went through in front of the researchers
so that we could get an idea of how they were thinking and whether
or not there was any kind of confusion with the wording.
So we could get an idea of face validity.
And after this phase, we actually changed the wording
of a few of the questions.
But we maintain the same set of 58 items.
Then we moved into the developing and refining
the scale phase where we were using
exploratory factor analysis to identify
the underlying factors, structure of the subcomponents
of the targeted construct.
We also wanted to refine the items down
to its most efficient form, so that we only included
items or tasks with sufficient variance among respondents.
And also so that it maintained reliability.
So the scale was consistent or
consistently measured
these constructs across
runs of people taking
the survey.
So first, we began with a survey
of one-hundred fifty-seven software developers.
These were recruited through
freelancing services,
Prolific, as well as,
through forums related to software development.
Each participant was asked to rate on a five point Likert scale.
each of the 58 items and their confidence
in their ability to perform each task.
From "I am absolutely confident" to "I'm not confident at all."
We also gave them the option to state that they didn't understand the option.
And so based on these responses, we first checked for a low
item, total correlation.
So to see if there are any items that maybe
appeared to not be measuring the same thing as the other items
and not really be related to the construct.
We also looked for any ceiling or floor effects
which would indicate that we might not
be measuring the actual variation because it's either
all the responses are on one end of the scale or the other.
We didn't see any of these effects with any of our items.
We also looked for whether
a large portion of participants
responded that they didn't understand the questions.
And there were 13 items that had greater than
five percent of participants, so we removed those
from the pool of items.
With the remaining items, we performed a principal
component analysis. From this, we found that there
were two underlying factors.
And so for those sort of underlying factors
that made up SSD-SES.
We then wanted to figure out which
items were associated specifically with each of those
factors uniquely.
And so we only retained items that met these
three criteria that they loaded
at least 0.5 on.
One of the factors that the item
loading on the primary factor.
So the one that had the highest loading on exceeded two
times the other factors and that the items commonality
was greater than point four.
And this is just the variance accounted for by all the retained factors.
And these three criteria basically just telling us,
basically, allow us to make sure that our.
Items that remained specifically measure a
single underlying factor.
And so though we see a change in one of the item responses,
we know that changes that underlying factor.
And it's not sort of noise shared across the two.
And this left us with retaining 18
items from our total 58.
And then we also tested internal reliability and
item-total correlation of the remaining items,
both together as the full scale.
And then for each subscale and found that
it had a reasonable level of
reliability and item-total correlation.
So now to finalize the scale.
We wanted to confirm that the previous identified factor
structure held after
we ran with a new population and that it still
maintained those reliability measures that we wanted
to make sure to check for.
And so we did another survey.
One hundred forty six software developers.
And again, we had all the participants.
Respond with their level of confidence.
Performing those 18 tasks on a Likert scale, on the same Likert scale.
We removed the "Do not understand" option.
We also had them respond.
Each participant was shown two of the following
six additional psychometric scales.
First, we included the computer programming self-efficacy
scale, who's included for convergent validity.
The idea here is that we wanted to make sure
we expected that people's ability to write code
is associated with their ability to write secure code
and to see if this relationship held up.
We also wanted to test for divergent validity to
make sure that we were
actually measuring something different.
So here we looked at two scales related to end-user
security behaviors and beliefs,
as well as just general self-efficacy.
We also included two psychometric measures
related to personality factors the need for cognition,
which is associated with someone's curiosity or
their want to perform tasks
that require cognitive or require them to think basically.
And also the general decision making scale, which
measures how people go about making decisions and have
several different sub skills related to
different decision making styles.
And so for this, we repeated our PCA
on those hundred fourty-six responses and
we actually found that three of the items there switched factors or
didn't sufficiently load in the second round, so we
removed those because those didn't hold up
over multiple rounds of the remaining factors.
Still, met all of our criteria.
They also performed a confirmatory factor analysis,
which utilized several different goodness of fit metrics that are given in the paper.
And we found that it met all the requirements
for these sort of goodness of fit tests.
And so this left us with the following 15
items broken into these two factors.
And these were generally aligned with the
first being participants' ability to find
and fix vulnerabilities.
The specific skill and then also their
ability to communicate security concepts to others.
Now, we compared it to the other skills we found that
SSD-SES correlated with,
computer programming self-efficacy scale for both of our sub scales.
This is kind of what we expected.
We also didn't see any
correlation with any of the other
scales that we included for discriminant validity checks, which
is what we expected.
We also saw that specifically the vulnerability, identification
and mitigation subscale did correlate with the
need for cognition scale.
This suggests that developers who are more curious are more
likely to feel confident in their ability to search for, identify
and mitigate vulnerabilities.
This corroborates prior work, which observed developers
who were more open or curious were
more likely to find vulnerabilities in secure development "puzzles".
So stepping back, now that we have
this validated scale, how can we actually use
it? Well, one that we sort of talked about as we
were motivating the development of the scale
is we can use it for testing educational interventions.
You can use this pre and post to see which might be most
effective and see how students respond
It can also be used as covariate to
approximate secured development skill.
So, for example, in a usability study of a security API
is would be beneficial to include SSD-SES
as a covariate to ensure differences
in outcomes between participants occur because
of changes to the API and not differences in security skill
Finally you could also use it
to measure secure development culture.
A manager could survey their developers to identify who
might most benefit from additional security education.
So in summary, we developed SSD-SES
which has a 15-item scale that can be
divided into two factors
related to vulnerably identification mitigation and secure communication.
And it was developed and validated through a review of
five of the most popular secure development frameworks,
surveys of security experts and several software developers.
And we've also given several things that it could be used for.
If you have any questions about this research
in general, how SSD-SES can be used or how we went about this? I'm happy to answer those.
I've given my contact information at the top right corner of the slide.
And also if you're just interested in our research generally,
definitely check out our our website, sec-professionals.cs.umd.edu
Thank you for your time and interest in watching this
video and your interest in our work. Thanks.