This work examines gestures performed with a class of input devices with distinctive
quality properties in the wearables landscape, which we call "index-Finger Augmentation Devices"
or iFADs.
In the large family of gesture-sensing devices, iFADs, which are devices that instrument the
index finger, hold a privileged position due to the rich interactions they afford on and
with the index finger, yet the corresponding gestures have not been systematically examined
in the scientific literature.
In this work, we focus on iFAD gestures and make the following contributions:
We introduce a four-level taxonomy of iFAD gestures that is index-finger-centric, encompassing
of a large diversity of gesture types, and technology-agnostic to be used with a variety
of iFADs and gesture recognizers.
We conduct a controlled experiment with 40 iFAD gesture types informed by our taxonomy,
and analyze more than six thousand gestures collected from 20 participants using a low-cost,
custom-made iFAD with a 3-axis accelerometer.
We compute and report recognition accuracy rates for iFAD gestures by using popular gesture
recognizers, readily implementable on wearable platforms.
Shilkrot et al. defined FADs as "finger-worn devices with an additional augmentation other
than their form, that provide a supplemental capability for one or more fingers using the
finger itself as a central element."
This definition is useful as the starting point for specifying iFAD gestures since it
sets the overall conceptual context of devices designed to be worn on fingers.
Based on Shilkrot et al.'s generic specification of what a FAD is, we provide an operational
definition for iFAD gestures as gestures performed with or on the index finger, detectable by
a FAD that is worn on the index finger with the purpose of providing input to an interactive
system.
This definition is encompassing of a variety of gesture types, and we provide a dedicated
taxonomy, for which the integrating dimension is the body scale at which gestures are performed
with the index finger in the personal, peripersonal, and extrapersonal space of the user.
In deriving the categories of our taxonomy, we start from the smallest scale corresponding
to gestures performed at the level of the index finger, such as taps on the iFAD, and
we progressively consider larger scales and gestures enabled by correspondingly larger
body parts, such as the hand and the arm, up to gestures that are fully specified at
the scale of the whole body, such as on-body and body-referenced gestures.
We show in the paper how the iFAD taxonomy complements existing gesture classifications
from the scientific literature, which are either too generic or too specialized to capture
the nuances and diversity of gestures that can be performed with the index finger.
To gain insight into the user performance with and perception of iFAD gestures, we conducted
a controlled experiment with 20 participants and an iFAD prototype built with a simple
accelerometer.
We collected more than 6000 gestures of diverse complexity, from simple taps on the iFAD to
specific finger and hand poses, stroke-gesture articulations of shapes, letters, and symbols
in mid-air, and body-referenced gestures.
We found that iFAD gestures are fast, taking roughly between 1 and 2 and a half seconds,
and their articulation times correlate positively with their complexity.
Also, finger-level gestures require the least amount of mean acceleration to produce, while
hand and arm-level gestures the most.
We also found that iFAD gestures were generally perceived by our participants as very easy
or easy to articulate, and socially acceptable in many public contexts involving various
locations and audiences.
To find out how various gesture recognizers perform on our large iFAD dataset, we used
the Nearest-Neighbor approach, popularized in the HCI community by the $-family of gesture
recognizers, with several dissimilarity functions for comparing gestures.
The highest accuracy rate was delivered by Dynamic Time Warping, followed by the Euclidean,
Modified Hausdorff, and $P+ dissimilarity functions.
The category of iFAD gestures with the highest recognition rate was represented by arm-level
gestures with 98.7% accuracy.
However, the accuracy of the entire set of 40 iFAD gestures was just 83.8%, which is
low according to current standards.
To identify a subset of highly-recognizable iFAD gestures, we analyzed the confusion matrix
of the best recognizer and iteratively eliminated gestures with high error rates.
We arrived at a first subset of 34 gestures, for which we reconducted the recognition experiment
and found that accuracy increased to 89.3%; see condition "B" in the figure from this
slide.
We continued our analysis and arrived at 93.6% accuracy for a subset of 28 iFAD gestures,
95.3% for 25 iFAD gestures, and 97.1% for 20 iFAD gestures, respectively; see the conditions
"C," "D," and "E" from this figure.
We also reconducted the recognition experiment by randomly selecting 20 gesture types from
the set of 40, which is condition "F" in the figure from this slide, and found an average
recognition accuracy rate of 89.1%.
These results are very promising given that the gestures were acquired solely from the
vantage point of the index finger, without sensing any other part of the user's body.
We use our results to propose practical implications for researchers and practitioners interested
in using iFAD gestures in their own projects.
For example, the iFAD gesture taxonomy, as a structured space for systematizing interactions
based on the index finger, enables exploration of many possible gesture types for gesture
UI design, from microgestures to hand poses to mid-air swipes to on-body input.
Designers should capitalize on the diversity of iFAD gestures towards gesture-based UIs
featuring interactions that are expected to be fast, low effort, and highly acceptable
for users.
Also, by adopting the vantage point of the index finger, a diversity of gestures can
be reframed, such as an on-body gesture described as an iFAD gesture.
This reframing can influence the choice of technology to sense
gestures, for example, an electronic ring with motion sensing instead of a touch sensor
integrated in the pocket.
Please see our paper for more details.
Thank you.