Hello everyone, my name is Verena Distler, and I’m a researcher at University of Luxembourg.
Today, I’m presenting our work titled “A Systematic Literature Review of Empirical
Methods and Risk Representation in Usable Privacy and Security Research”.
I am presenting today also on behalf of my co-authors from University of Luxembourg,
Carnegie Mellon University, CISPA Helmholtz Center for Information Security, and Eindhoven
University of Technology.
When using technology, people are exposed to a large number and type of security and
privacy risks, and we hear reports of successful security breaches almost daily.
It is important to understand how people perceive and understand these risks, and to study their
reactions to potential security attacks.
This can help design interventions that will help people better understand and respond
to risks and attacks.
For researchers designing a study, it is difficult to recreate a realistic experience of security
and privacy risks in a research context.
Researchers need to navigate trade-offs between realistic risk representation, ethical and
legal concerns, while also considering logistical and privacy-related concerns.
These are just some of the challenges that researchers in usable privacy and security
face when designing user studies.
In light of these challenges, we investigated which methods researchers use for privacy
and security studies that include human participants, with a particular focus on the approaches
researchers use to represent risk.
We had three research objectives.
First, we wanted to understand which methods researchers in the UPS community used.
Second, we investigate how researchers represented risk to research participants.
Third, we wanted to understand how researchers in the UPS community used deception in their
user study protocols.
To address these objectives, we conducted a systematic literature review of papers from
five top-tier UPS venues; and we used search terms that combined security/privacy related
terms with user-related terms.
From an initial sample of 633 papers, we systematically selected 284 full-length papers that included
human subjects studies.
We provide the dataset as supplemental material.
We then analyzed this set of papers, and we focused on study methods, risk representation,
the use of prototypes, scenarios, educational intervention; the use of deception to simulate
risk; and types of participants.
I will now present a subset of results, but we invite you to read the paper for a full
picture of the selection and analysis procedure, and the results.
To describe the set of analyzed papers, most of the papers had an experimental or descriptive
objective, with only few relational studies.
The most common study methods were experiments, interviews and survey.
For participant recruitment, researchers frequently relied on crowdsourcing and non-representative
convenience samples.
Risk representation in the analyzed papers was mostly based on naturally occurring or
simulated risk, varying with study methods and research objectives of the paper.
An example for what we analyzed as naturally occurring risk would be when researchers sent
a password reset email to actual Linkedin users and evaluated the effectiveness.
Another example was when researchers introduced threat modeling in an enterprise setting.
The risk is not simulated for the purpose of the research.
Instead, it is naturally occurring.
An example for simulated risk representation would be an online experiment where the researchers
asked participants to imagine that they were creating a password for an account that they
care a lot about, such as their primary email account.
Another example is when developers were asked to roleplay and imagine that they were responsible
for creating the code for user registration.
In our sample, there were few studies relied only on mentioned risk or did not represent
risk at all.
It might seem surprising that there are studies that do not attempt to create a perception
of privacy and security risk.
These were typically studies that focused only on the usability of a privacy or security
tool, without attempting to recreate a perception of risk.
Researchers used creative combinations of approaches to represent risk to research participants,
for example assigned tasks, prototypes, scenarios, deception, educational interventions, or incentives
for secure behavior.
We give many examples of how researchers used these approaches in the paper.
For now, let’s have a closer look at deception.
In the paper, we describe how there is a grey area between deception and not priming participants
to think about security and privacy topics, which we did not classify as deception.
We thought that researchers might frequently use deception to simulate risk, but surprisingly
we found deception was used only in 5% of papers.
We categorized these papers to understand what type of deception they used.
Most commonly, these papers deceived participants about the objective of the study, others deceived
participants about the presence of risk, and some deceived their participants about the
study participation.
In the paper, we give examples and provide a much more detailed analysis of papers using
deception.
In the paper, we discuss the choice of methods and risk representation.
Overall, the choice of method usually coincided with certain types of risk representation.
When picking a method, researchers will thus often face tradeoffs with regards to the risk
representation they can possibly use in their study design.
We also point to methods that might be underexploited, for example diary studies, vignette studies,
and studies involving groups of people, for example focus groups or workshops.
In terms of participant recruitment, we found that certain populations were understudied.
When they were included, it was usually in descriptive studies, and not to assess the
suitability of security or privacy tools for these populations.
We also discuss the use of crowdworkers in the sample.
With regards to ethics, we discuss the use of deception.
Authors used different definitions for deception, and the community might profit from further
discussion of what types of deception are ethically acceptable in UPS studies, and under
which circumstances.
A small number of studies also used attackers as non-consenting human subjects, which might
also warrant more discussion.
To analyze how researchers represent risk to their participants, it is essential to
have a clear understanding of how the authors recruited participants, what the participants
were told or led to believe, and how tasks or questions were framed.
In some of the papers we reviewed, these details were not clear, and we provide a checklist
and guidelines to help report the relevant details for better replicability and understandability
of how their research induced a perception of risk.
Thank you, I encourage you to read our paper for more details.