Hello, my name is Florian Mathis from the University of Glasgow and it is my
great pleasure today to present on behalf of my co-authors and PhD supervisors Dr. Kami Vaniea
and Dr. Mohamed Khamis our work RepliCueAuth: Validating the Use of a Lab-based Virtual Reality
setup for Evaluating Authentication Systems. We investigated the suitability of
Virtual Reality as a test-bed for usable security research, more specifically for
authentication schemes. We aimed to understand to what extent results from a VR-based usability and security study
match with the previously reported findings in the real-world study.
Why would a researcher want to evaluate a system that is intended to be used in the real world in Virtual Reality?
I brought with me today some examples of study setups within the scope of usable security and privacy.
Corresponding research studies often require a significant amount of hardware such as cameras,
touchscreens, 3D printed prototyping systems, just to name a few.
As a result, conducting such studies is often challenging and costly.
And at this point, VR studies -- using VR as a test-bed for human-centred research
can be particularly helpful because they can reduce the costs of research involving physical components.
They can also enable recruting many and diverse participants, and they have the potential to reduce the need
for face-to-face user studies. To explore how such VR studies can be applied
for usable security research, we chose to study CueAuth, a real-world authentication system for public displays.
CueAuth covers three input methods that are frequently used in security research such as touch, mid-air, and eye gaze.
We replicated the original study setup and the authentication scheme fully in VR
to then conduct a conceptual replication study and compare our findings to those
reported in the real-world study. We conducted two user studies: a lab-based
in-VR usability study where participants entered a set of PINs with touch, mid-air, and eye gaze.
And a remote non-immersive VR study where participants in the role of attackers
observed authentications performed by an avatar in Virtual Reality. This depicts the scenario where
an attacker looks over the user's shoulder, which is also called shoulder-surfing.
When taking a look at our in-VR usability study, we found many similarities between our VR study and the real-world study; however,
we also found that authentication with touch input took significantly longer in our Virtual Reality study.
The other was found for eye gaze. Gaze-based interaction was significantly faster in our VR study
compared to the original real-world study. The original work used a stationary eye tracker attached to the authentication system, and this
likely resulted in the eye tracking quality being influenced by the user's height, distance to the display, and the ambient
and lighting conditions in the real-world study. Eye gaze also resulted in significantly lower physical workload,
less frustration, and less effort in our VR study compared to the real-world study.
For the remote security study, participants' successful attack rates followed a similar pattern as in the original
real-world study. Input with eye gaze was overall the most secure input method.
Repeated-video attacks on touch and mid-air resulted in a higher successful attack rate. This was not the case
when attacking eye gaze, which is inline with the original real-world study. Overall, we showed through our study that
observing a virtual avatar during authentications reveals insights about
how resistant an authentication scheme and also the different input methods are against observations.
To conclude, our research showed that the transferability of quantitative results from VR to the world
highly depends on how well reality and its limitations are emulated. VR studies are promising
for usable security research, but -- and I really want to emphasise this -- they should not replace traditional research paradigms,
but rather augment them. We as a research community just need to be very clear what we can
and also what we should expect from evaluations that are conducted in, for example, the lab, fully in VR, online, or in the field.
Last but not least, I would like to end my talk with a quote by Garfinkel and Lipford.
I strongly believe that VR studies as presented in our work, have huge potential to augment existing
research methodologies in usable security and privacy and help speed the discovery
and adoption of usable and secure techniques and systems.
Thanks a lot for listening and please find more details about our work in our paper.