Hi, this is Wen-Jie from telecom Paris, Institute Polytechnique de Paris.
I'm happy to present our work, the dark side of perceptual manipulations in VR. In this paper,
we collected scenarios where malicious actors use perceptual manipulations to provoke harm
to a VR user, and identify the potential attacks and characterize the potential physical harm.
We also propose mitigations and countermeasures to against the malicious use of perceptual
manipulations. Our work is motivated by virtual physical perceptual manipulations (VPPM),
leveraging human perception to direct the user's movements and enable techniques. For example,
redirected working, redirected haptics, and also pseudo-haptic illusions.
Perceptual manipulations is different from physical manipulations
because systems like electrical muscle stimulation may override the user's physical actions.
The user is to some degree aware of having consented to be manipulated. In the contrary,
VPPMs create movements by the reaction to the presented perceptual stimuli. Creating
movements using VPPMs raised a question, whether a user understands they are being manipulated.
So this may vary to their prior consent and knowledge. However most of the time VPPMs are
developed or designed to be imperceptible, which means it's below our perception
threshold so that we're not able to perceive the subtle difference of being manipulated.
Lastly, the intention behind using VPPM is open to abuse. For example,
recent work has started to explore the immersive VR attack
where they can direct or position a user inside the safety boundary and control their position.
We explore the dark side of perceptual manipulations where malicious actors use
imperceptible VPPMs to provoke physical harm to the VR user unknowingly, without consent.
We focus on the physical harm and action causing hurt or damage to the VR user's body,
for example, hitting bystanders or colliding with the environment.
We define the threat model that a malicious actor has intention to harm the user by
tricking them into install malware and having access to the environment and tracking data.
Therefore malicious actors are able to apply VPPMs to manipulate the user and provoke harm.
We want to explore how could malicious actors abuse VPPMs and what potential physical harm
could they provoke to the VR user by manipulating their body motions. To address these questions,
one challenge is that VPPMs are still not used by most of the applications in VR. Therefore we
choose speculative design workshop as our method to both critique current practices and reflect on
future technologies in VR. we invited VR experts and designers to participate in our study
after the instruction of the workshop, participants had to speculate
on the scenarios that manipulate VR users body motions to provoke physical harm using VPPMs.
19 scenarios were collected. For the detailed description please see our paper.
Here i will select two scenarios. The first one is called magic maze where a VR user explores a VR
maze inside the physical building and at one point they think they are going towards the door but
the system (directed them to) redirect them to a stairway and make them fall over.
Another example is start a fight where the head mounted display
incorporates bystanders as an enemy in VR so that the VR user (will hit)
might hit the bystanders because they believe (it's a VR avatar) it's an avatar in VR
next participants had to identify specific dimensions by positioning the presented scenarios
in each workshop, and after that they rated the most relevant ones. For example, the severity
of harm where we started from mild pain like eye strain and muscle issue to fall over get punched
and even to extreme cases like death. Another example is perceived agency where it states to
what degree a VR user believes the physical harm or physical action was caused by themselves.
if no agency they might think it's system's fault. However if we use VPPMs and make them consider
it's an action created by themselves they might think it's their fault and fall into a trick
again. So we applied a thematic analysis to find underlying themes in the transcribed data from
the workshop. We found two classes of attacks and a characterization of physical harm. I will focus
on the classification of attacks. We identify puppetry attacks that control physical actions for
of different body parts of a VR user, for example, working in an environment or arm movement.
The other class is called mismatching attack. It creates a difference of information between
the virtual and physical environment to make a misinterpretation for a VR user.
For example, in an one-to-one mapping environment we can impose a virtual chair (where) but without
the physical counterpart. (it's) We call it false positive attack so that the VR user might sit on
it and fall over. And the other example we call swapping. It's the scenario of startup fight
where if we incorporate a physical object with different virtual appearance.
We also developed two applications SteppingOn and HittingFace to demonstrate subverting
existing VPPMs in hci. These examples can create a missing step or collision between the headset
and controller and in order to show a potential risk of VPPMs. After exploring
the dark side of VPPMs, we propose mitigations and preventative recommendations against it.
We include five different levels from individual to platform and community. Here i
focus the role of our research community and for the rest of the discussion please see our paper.
We want to raise the awareness about the malicious use of VPPMs in our research community.
VPPMs in fact we are hacking human perceptions. The results like enabling
infinite walking haptics and higher presence are all positive outcomes.
Meanwhile we are also exposing the vulnerability of our human perception.
We suggest the future research should publish VPPMs with the potential threats and risk in mind.
Finally, here are the takeaway messages from our talk vpvms can lead to malicious use of our
perceptual vulnerabilities. We presented potential attacks of VPPMs and the physical harm for example
puppetry attack and the mismatching attack. And finally hci community should become more
cautious publishing VPPMs and also reflect on the potential for abuse. Thank you for your attention
and this is the end of my presentation,. I'm grateful to all of my collaborators Elise, Mark,
Mohamed, Eric, Sam and Jan, and our participants. Thanks for all of their time and contributions