Hi, I'm Cathy Fang, and I'm presenting Retargeted Self Haptics for Increased Immersion in VR
without Hand Instrumentation. This work was done at the Future Interfaces Group at Carnegie Mellon
with Prof. Chris Harrison. Current VR systems provide immersive graphics and audio, but physical
touch feedback remains very limited. Most consumer systems today use handheld controllers which
only provide vibration feedback. Some consumer systems provide controller-free interactions,
preferring not to encumber the user's hands, but as a consequence cannot deliver any haptic
feedback. Thus, a large body of research has developed solutions that are handheld,
such as haptic links, or directly instrument the hands, such as Wireality. Another body of research
looking to contactless haptics using air vortexes, such as AIREAL, or ultrasonic phased arrays,
such as Ultraleap's STRATOS explore, to deliver tactile sensation akin to a tap on the skin.
Prior work has also considered the use of manual labor or human-generated actions to
provide haptic feedback. These approaches either prevent freehand interactions,
or cannot stimulate feedback from grasps, or require other humans to achieve the haptic
feedback, which motivated our research question: Could we repurpose the user's
own body to provide haptic feedback and allow for free hand interactions?
We present our technique called Retargeted Self-Haptics, which I will explain with an
example. Here a user places their hand onto a palm scanner, which then causes a keypad to flip open.
In a typical VR experience, both hands will feel nothing interacting with these virtual elements.
However, by manipulating the positions of the virtual hands,
what is known as haptic retargeting, we can subtly guide the user's hands together.
Now when the user types on a keypad they feel an actual physical surface.
Haptic retargeting, also known as redirected touch, utilizes visual dominance and temporal
instance to change how virtual objects are perceived. For example, the user sees that the
virtual hand touches three separate virtual cubes when in reality they touch the same physical cube.
Most related to our work and at least explored is self-haptics, which is haptic feedback touching
one's own body. Kohli introduced the concept of the haptic hand, where a virtual touch interface
overlays on top of the non-dominant hand. In air haptics, Ban et al. pioneered the idea of warping
the thermal index finger to create pinching and pulling sensations for virtual objects.
We built our approach on top of the body warping retargeting techniques, but instead of using
physical props, we warp the hands of a user such that they come into contact with one another
in a purposeful way in order to achieve a design haptic feedback. Note that in the scanner keypad
example only one hand receives appropriate haptic feedback, even though this is a two-handed task.
We define appropriate haptic feedback as when the magnitude and direction of forces approximate the
real-world action. We use these two dimensions to form an interaction design space to guide our
explorations. The palm scanner and keypad example goes in the upper right quadrant.
Other examples in this category include: striking a match where only the hand holding the match
receives appropriate resistance, hammering a nail where the hammering hand is halted appropriately,
and cutting vegetables where again the cutting hand is stopped at the cutting board.
It is also possible to have two-handed tasks where both hands receive coordinated physical feedback.
We built a number of illustrated examples, including this bow and arrow demo where the user's
thumb is retargeted to the bowstring, allowing the user's hands to exert opposing pulling forces
as will be experienced when a real bow is under tension...here the user can physically feel when
the ammunition magazine has been fully inserted by warping the hand positions... in this example,
the palm is retargeted to act as a phone screen and taps to the phone screen exert a downward
force on the holding hand... similarly the impact from hit a baton in the hand can now be felt.
Not all interactions require both hands, and so we also explored self haptics for one-handed tasks.
As there is only one hand present to receive haptic feedback, the bottom left of our design
space is not applicable. While we did consider how a single hand could provide haptics for itself, we
found the design space to be much richer when the unused hand was appropriated as an accessory prop.
For example, here a user is presented with a red launch button. Typically in VR, the user
will just reach out and paul at it with no haptic feedback. With our approach, a translucent hand
is also rendered in a scene, which turns green when the user matches the position and pose.
Now with the dominant hand, the user can reach out to perform the one-handed interaction and receive
some haptic feedback. In this example interaction, a thumb is used as a prop for a mixing whisk.
Here the unused palm allows the user to rest their finger on a virtual fingerprint scanner,
and finally, the palm again allows the user to physically knock on this door.
In the paper, we provide a table of specific retargeting parameters for all our demos
as well as open-sourced several examples for people to try. We conducted a user study with
11 participants who tried two examples out of each category (where one or both hands receive
feedback for a one- or two-handed task) and in both conditions (no haptic feedback or with self
haptics). We found that all three categories in the self haptic condition (represented as
blue bars) were rated higher in tactility than in the no haptic condition (represented as red bars)
while having no significant effect on either ownership or agency. Similarly,
all three categories in the self-hate condition were rated higher for immersion realism and fun,
and among them, the leftmost blue bar, which represents the category of bimanual tasks
where both hands received feedback, was rated the highest. Overall, the category that participants
preferred the most is the one where both hands received feedback during a bimanual task, in which
appropriate feedback was provided to both hands while making minimal modifications to the action
itself. In addition to the questionnaire results, we gather insights from combining participants'
think-aloud and observations. For example, we observed that impact-oriented interactions
that are short and fast performed better, and scenarios involving a smaller contact area,
such as a fingertip landing on the palm, performed better than warping a hand around the thumb.
Our technique faces tracking and precision in the current computer vision-based approach,
which affects realism. To mitigate errors, we suggest designers choose target surfaces that
are harder to miss and more forgiving rather than smaller body features or surfaces with irregular
geometries. We also identified five factors that are key to identifying the body part
best suited to convey a haptic effect, such as size, contour, and anchoring. Anecdotally,
we also observed potential psychophysical effects such as haptic valence and somatosensory
attenuation, which poses interesting questions for future work. Our technique is not suitable
for all interactions, such as training tasks that require high position and haptic details.
Only using parts of the hand and arm has a limited surface area and a limited pallet of
textures and compliance, which inherently feels different from common objects. As mentioned in
the previous slide, our technique involves partial hand occlusion which causes poor hand tracking.
Thus we avoided heavy occlusion and freezes the hand in place in case of a tracking loss.
Our technique utilizes visual haptic mismatches, which is unexpected. And the category where the
undominant hand is used as a prop is inherently less intuitive. But we found all participants
were able to quickly acquaint themselves with the shifted position after a brief period. For
the scope of our project, we only looked into retargeting the arm and hands. Future research
could look into other body parts and features for repurposing as well as retargeting the fingers
such that the hand provides haptic for itself. Please see the paper for more details. Thank you!