Hi everyone, I'm Siyou pei, a PhD student at UCLA. I'm really happy to present this work: Hand Interfaces
- Using Hands to Imitate Objects in AR/VR for Expressive Interactions.
It is done in collaboration with Alexander Chen, Jaewook Lee and my advisor Yang Zhang.
In the digital reality, there are many objects to retrieve and interact with.
This diverse and rich set of objects contribute significantly to the user experience of Augmented and Virtual Reality.
One prevailing method demands users to hold hand controllers which could be cumbersome
In response, prior work investigated free-hand interactions.
One popular technique, demonstrated by prior work, has users’ hands perform a grasping gesture to retrieve virtual objects in the air
We propose a new free-hand interaction technique.
Before revealing our designs, please recall when you play the rock-paper-scissors game.
In this game, we use hands in different shapes to imitate rock, paper and scissors
It’s intuitive and self-revealing.
So we wonder, can we generalize this idea of embodiment to AR and VR, and to a wide range of interactions that involve objects such as props and tools?
Specifically, we propose an interaction technique that allows users to embody objects through imitating them with hands.
Our technique enables two common tasks in AR and VR: object retrieval [pause] and interactive control.
Examples of our designs include joystick emulated by a thumb up gesture
a kalimba by five fingers extended out
a switch by a loop formed by the index and the thumb
an inflator by a horizontal horn gesture
a wand by a pointing gesture with the index finger
A fishing rod by a pointing gesture with the thumb extended out.
A globe by a fist gesture.
A spray can by a fist gesture with the index finger slightly extended out.
a trumpet by a pinky finger up gesture
a pair of scissors by the scissors gesture
binoculars by touching two "O" hand gestures
we have also created designs for many other objects collectively we call our designs hand interfaces.
we envision hand interfaces being widely used in AR and VR applications
let me show two examples
The first one is an educational app in VR. Hand Interfaces allow a user to use index fingers to quickly imitate multimeter probes for examining the circuit.
After the source of the problem is located, the user imitates a pen to write down some notes, And take a picture with an imitated camera for documentation.
The second example is an AR-based ubiquitous control for smart environments.
with an AR goggle, the user can imitate the toggle switch to turn a light on and off
the user can also adjust the orientation of the robotic lighting system with the thumb up joystick
note that the joystick is automatically retrieved and rendered around the user's hand
then the user forms a fits gesture to imitate a spherical color palette to adjust the light color
now let me talk about our design process
let's take this spray can as an example
we first brainstorm a lot of ideas
then we apply the criteria below to sift out the best one
we consider shape similarity as a first criterion and remove those not similar in appearance
then we consider the kinematic similarity
this means good designs should have moving parts that move similarly as a target object
with this criterion, we removed another two designs
we also removed designs that were uncomfortable to perform
finally, we consider social acceptance
this means good designs must be socially acceptable to people
following these criteria we got our final design for the spray can
with this design process we created 28 hand interfaces
As I mentioned earlier, these hand interfaces can be manipulated for tasks that involve interactive control.
Hand interfaces support both interactive control and object retrieval.
Here is the implementation.
1. We implemented our idea with a state machine in Unity using Oculus Quest.
2. We use thresholds to transition between states.
3. To improve robustness to small changes of fingers and joints we use sensitivity matrices.
we also open-source our project on GitHub
please refer to our paper for more implementation details
now let me talk about our evaluation and findings
overall we ran two studies. one for object retrieval, the other for interactive control.
for object retrieval, we have two baselines.
The first baseline is the Drop-down menu, which is commonly used for controller-based object retrieval in contemporary AR/VR applications.
The second baseline is to have users’ hands performing a grasping gesture to retrieve virtual objects.
after all indirection techniques we conducted a semi-structured interview regarding five metrics which were rated on a seven-point Likert scale we analyzed the quantitative data using the significance analysis method and ran a semantic analysis with quotes from participants
here are our results in this figure one indicates strongly disagree seven indicates strongly agree
for these metrics we used fidelity refers to the degree of which the interaction technique
provided feedback such as the virtual experience was similar to retrieving the object in reality
hand interfaces are significantly more realistic than the drop down menu method
our second metric freedom of movement investigates how unstricted participants movement was
We can see that Hand Interfaces support comparably free movement as drop-down menu method.
We used the Swiftness of retrieval metric to quantify how quickly participants could
retrieve an object from the standby hand pose
With Hand Interfaces, users retrieve objects more quickly than that with grasping gestures.
Comfort of retrieval refers to how physically and mentally comfortable participants felt during retrieval.
We don’t see significant difference among techniques here.
finally, ease of recollection measures how easily participants could recall how to retrieve the object
drop-down menu shows its innate advantage here
we have a lot of insights from our thematic analysis
for time being let me share two overall findings
First, we found that “perception of realism is affected by real-world experience”.
it is something similar to the exposure effect
if participants had imitated these objects before
they tend to perceive it as being realistic
example includes scissors, wand and binoculars
we also found that the ease to imagine also benefits the perception of realism
when imitating an object is easy to imagine which often happens when there are some analogies that user can draw these designs were credited with more realism
second we find perception of realism is also affected by visual alignment
this means participants perceive designs as being more realistic if their hands or objects were well
aligned visually
we found something similar to the uncanny valley effect that says minute discrepancy can result in severe discomfort
however we noticed that participants were more tolerant with visual discrepancies in hand interfaces than in the baseline with grasping gestures
for their recognition that Hand Interfaces were not designed to imitate their reward experience
now let's move on to the study of interactive control
in both of these baseline techniques participants manipulated virtual objects with another hand in the air the study
procedure and result analysis were the same as a previous study
the figure shows our results Hand Interfaces were comparable with other techniques across all of metrics,
no significant difference were found
I want to talk about two overall findings from our thematic analysis
first, we found that tactile feedback contributed to the perception of realism too
the average fidelity score of handy physics in interactive control scenarios was significantly
higher than that in object retrieval scenarios
which we suspect was due to detective feedback
finally we find the tactile feedback is beneficial to interactions with virtual objects
it can not only help increase comfort but also correlates with joyfulness as many participants express excitement and amusement
thank you for listening. I'm happy to take questions