We examine in this paper gesture input with devices designed to be worn on the upper-body,
such as smart watches, rings, and glasses.
Our investigation is motivated by the fact that wearables are surpassing smartphones
as today's fastest-growing technological innovation for mobile users.
However, gesture-based interactions with off-the-shelf wearables are based on certain assumptions
about their users' motor abilities.
For example, a smart ring assumes the motor ability to move the wearing finger, rotate
the wrist, make a hand pose, and possibly raise the forearm in the process.
Upper-body motor impairments cause challenges during input with upper-body wearables implementing
such assumptions.
In this picture, a person with spinal cord injury controls a smart ring with movements
of the arm, not the fingers.
To understand the gesture input articulation performance of users with upper-body motor
impairments with upper-body wearables, we conducted a controlled experiment with a device
worn on the wrist as a smart watch, on the finger as a ring, and on the head in the form
of a pair of glasses.
Our experiment involved 14 people with various types of upper-body motor impairments as well
as a control group of 14 people without impairments.
We had two within-subjects independent variables.
Wearable, nominal variable with three conditions: watch, ring, and glasses, representing devices
worn on the wrist, finger, and head.
And Modality, nominal variable with two conditions: stroke-gestures performed on the wearable
device and motion-gestures of the body part wearing the device.
Stroke-gestures are paths produced on a touchscreen, for which symbolic associations are created
with system functions, such as the letter "M" stands for "Menu."
We designed a set of twelve stroke-gestures for input on the watch, ring, and glasses.
We selected the gestures to balance a number of criteria, such as the number of strokes,
the complexity of the geometric shape of the gesture, and others.
Motion-gestures are movements of a body part, such as moving the finger wearing a smart
ring.
We designed three gesture sets for the finger, wrist, and head corresponding to the three
conditions of the Wearable independent variable by drawing from prior work on motion gesture
input with wearables and existing taxonomies for motion gestures.
We used the Samsung Gear Fit 2 to collect gestures in the smartwatch condition.
To make sure that gestures were collected with the same resolution across the glasses
and ring conditions, we repurposed the Gear Fit device, which we detached from its strap
and affixed with a 3D-printed support to a pair of glasses and to a ring form factor,
respectively.
We collected more than seven thousand stroke-gestures and nearly four thousand motion-gestures from
our participants.
The completion rate of the experiment for the participants with upper-body motor impairments
was above 90%.
We found that stroke-gestures produced by users with upper-body motor impairments on
wearable touchscreens at various locations on the upper-body were longer, wavier, less
consistent, with more strokes, and took twice as much time to articulate compared to the
same gesture types produced by the control group.
Also, stroke-gesture articulation performance was influenced by the location of the wearable.
Stroke-gestures in the watch condition were the fastest and had the highest geometric
consistency.
Unlike stroke-gestures, we found that motion-gestures were produced equally fast and with similar
kinematic characteristics by both the participants with and without motor impairments.
The location of the wearable did affect gesture articulation, but in a similar way in both
groups.
Motion-gestures performed with the finger and wrist were faster than head gestures,
but also less consistent.
Based on our findings, we propose implications for more accessible gesture input for upper-body
wearables by capitalizing on the principles of ability-based design.
The ability-based design approach emphasizes the importance of focusing on users' abilities
in context based on seven principles.
In our case, abilities refer to gross and fine motor skills of the finger, wrist, arm,
and neck to perform stroke-gestures and motion-gestures on and with rings, watches, and glasses.
Next, we discuss some of the implications of our findings.
The ability principle states that designers focus on users' abilities in a given context.
Our findings suggest designing gesture sets that include both stroke-gestures and motion-gestures
according to the motor ability of the user to move fingers, land the finger on a device,
maintain stable contact with the device, rotate the wrist, produce accelerated movement, raise
the hand, and control cervical muscles.
For example, participant fourteen from our experiment was not able to perform stroke-gestures
on the ring, but was able to move the finger wearing the ring.
Another implication of the ability principle is to design customizable devices in terms
of the location on the body where they are worn to enable comfortable reaching for stroke-gesture
input, but also motion-gesture input with less effort involving that part of the body.
An example is participant seven from our experiment that could not raise their hand to perform
stroke-gestures on the temple of the glasses, but was able to use the same type of gestures
when the sensor was worn in the form of a watch and a ring, respectively.
The availability principle states that designers use affordable and available software and
hardware.
Since smartwatches and fitness trackers are becoming mainstream, requirements regarding
their affordability and availability are implicitly met.
However, other wearables, such as rings and glasses, are still on their way to mass adoption.
A practical solution to affordability and availability is to reuse the touchscreen of
a smartwatch, a prevalent device, and affix it to various parts of the body for easier
and more convenient access with the finger during touch input, but also for instrumenting
those body parts for motion input.
The performance principle means that systems employ data about users' performance for best
match with abilities.
The implication in our case is to model users' gesture articulation behavior.
Aspects such as the amount of time to articulate a gesture, the number of strokes produced
to draw the geometric shape of a stroke-gesture, and the axes of movement along which motion-gestures
are articulated are examples of information that the wearable device could collect and
use to tune accessibility settings, such as how long to wait for a multi-stroke gesture
to be entered before calling the gesture recognizer.
Overall, we provide ten practical implications informed by our empirical findings to implement
the seven principles of ability-based design towards more accessible gesture input with
upper-body wearables.
Please see the "Discussion" section from our paper for more details and examples.
To enable future work in this direction, we release our two datasets composed of nearly
11,000 stroke-gestures and motion-gestures collected from 28 participants.
The datasets are freely available for research purposes and they come with C# source code
that computes the measures reported in our paper.
Thank you for watching and please see our paper for more information.