As human beings, our hands play an important role in how we interact with the world.
We use them to communicate, we use them to count, we use them to create.
When it comes to the digital world, they still play an essential role
as we use them to interact with mice, keyboards, and controllers.
Hand gestures and movements have been explored for a long time as they offer potential
for more direct and natural interaction.
Recently, new technology has brought the power of hand-based interaction
to the digital field.
For example, using sensors such as Leap Motion offers undeniable advantages
for gaming, creation and navigation.
Importantly, research shows that gestures that have a meaning in the physical world
should be picked over more abstract gestures.
At the same time, the physical world has been explored as a support for digital content.
For example, making use of virtual reality and projection mapping, Roo and colleagues
showed how a simple sandbox can become a tool
for meditation and search of mindfulness.
The body has also been used to support digital content,
for example for gaming or creating
as in this digital makeup application.
However, to our knowledge, using the hand itself has not yet been explored.
Ultimately, placing the body at the core of digital experiences is strongly recommended
in embodiment literature.
Reconnecting with the body can be helpful for learning,
for example in mathematics, teaching students how to perform
hand gestures can help them perform better proofs.
But such an approach can also help people experience their body as play
or benefit from somaesthetic experiences.
We present DigiGlo, a system where the hand is used both as a display and a controller.
In our paper, we implemented a Virtual Reality version of the system,
however the mechanism has also great value for Spatial Augmented Reality
in particular as it would not require to equip the person
with any hardware.
Relying strongly on embodiment, our system brings the body back at the center of the
digital activity.
Moreover, co-locating input and feedback, reduces split-attention effects.
To test and evaluate our system we implemented 3 playful activities.
Space Traveller is a self-contained pinball game making use of hand shape of the display and
the strong connection with the controls.
On top of the main pinball mechanics, the hand controller brings in an interesting twist
as it lets the player bend space by moving their fingers and influence gravity.
Marble Runner explores natural hand interaction specific to DigiGlo.
In this fast-paced game, the player’s dexterity is put to the test as they go along a twisted
path filled with obstacles.
Exploiting a viewport metaphor, the player can control the marble by simply moving their
hand around.
Finally, in our educational activity, Noelle’s Ark, we focus on embodiment for learning.
In this activity, the player has to compare the weight of two items by moving their hands
up and down, mimicking a twin pan balance.
Involving the body in such a way is not only good for interaction, it's also good for
learning and retention of information.
We performed two user studies to evaluate our system: first, a preliminary usability study
focusing on quantitative aspects followed by a more focused qualitative thematic analysis.
In the latter, we interviewed five participants, and we asked them questions about
embodiment, interaction, and the potential of our system for gaming and learning.
We aggregated our findings as design recommendations and I’ll present some of them now.
Confirming previous research, participants enjoyed more gestures that have a strong physical
connection with their effects.
They mentioned that simpler gestures were more intuitive.
They also found them more natural and easy to use.
Thus, we recommend to use simple and meaningful hand gestures, both from a
usability perspective and a learning perspective.
But what is interesting with our system is that some gestures that are not necessarily meaningful per se
become meaningful because of the close mapping between the display and the controls.
For example, pinching two fingers
to bend space.
We also noticed that participants had much stronger expectations with respect to physics.
In particular, in the digital world participants expected physics to behave exactly as it
would in the physical one, even if it resulted in more cumbersome and impractical interaction.
For example, in Space Traveller, participants mentioned that they wished gravity would listen
to them more.
Also, in Noelle’s Ark, when participants had to compare two items
they used their hands horizontally, even
though it would reduce the readability of the content.
This is a form of tactile illusion that is specific to our system and we would recommend to be mindful
of expectations people have from the physical world when connecting their body so strongly
to the digital experience.
Overall, we show that DigiGlo has great potential
for gaming and learning activities.
But participants our also suggested several other use cases and applications.
In particular, DigiGlo could be used as a mechanism to foster
creativity, body awareness and mindfulness.
Thank you for your attention!