Welcome to my presentation.
I’m going to present our paper Keep the Phone in Your Pocket.
I'm Yiwen Yin from the Department of Computer science and Technology, Tsinghua University.
Nowadays, smartphones are an indispensable tool for visually impaired people.
This picture shows a typical posture when a visually impaired user interacts with a
smartphone via a screen reader.
However, previous studies have shown that they may feel inconvenient by getting their
phones out of their pockets, constrained by two-handed interaction, and suffer from privacy
concerns and social awkwardness.
So our goal is to provide a new wearable input system to enhance phone interaction.
There are many related works based on different wearable devices, such as wristbands, rings,
and white canes.
But these interactions either have constrained number of gestures, constrained body locations,
or require two-handed interaction.
Some of them are not designed for visually-impaired people.
So in our work we use a smart ring embedded with a single IMU to sense gesture inputs.
Wearing the ring on the middle phalanx of the index finger, users could perform 15 gestures
on any surface to operate their phones.
Overall, we designed a set of commands and gestures suitable for smartphone control,
designed a classification algorithm to recognize 15 gestures, and provided usability evaluation.
Why do we choose rings?
We believe that in one-handed interaction scenarios, a ring can sense subtle finger
gestures with greater accuracy, thus provides higher efficiency and social acceptance.
Our first study is to design a versatile command set for phones and a suitable gesture set
for ring, and then map them up.
For the commands, after brainstorming, visually impaired participants were required to rate
the necessity of the commands so that we could remove unnecessary ones.
Here is our result.
We class them into different types of commands.
For the gestures, we designed a gesture vocabulary based on our design space.
We defined two posture types: finger pulp and fingertip.
This is our vocabulary.
We then conducted an ease-of-use scoring study to explore their preferences.
For the mappings, we considered users' pre-existing vocabulary of idiomatic gestures, and explored
the semantic relationships.
Then this is our final mappings of gestures and commands.
Now here comes our second study: gesture recognition.
Our prototype used a 9-axis IMU, connecting to an Arduino.
For the data, we used Madgwick Filter to get the gravity and pitch angle.
And we have 20 visually impaired participants to perform 4500 samples in sum.
We followed a paper to recognize the touch down event.
The registration gesture is a triple tap, so the system reports a registration event
when three touch events occurred within a specific window.
Additionally, the registration collects the initial pitch angle.
Our system will adjust the pitch angle according to this initial data.
And for our 15 classification, we designed the features according to these 5 criteria:
tap or swipe, how many touches, finger pulp or fingertip, swiping left or right, up or
down.
Then we designed a multi-level SVM classifier.
Here is our confusion matrix.
Leave-one-out cross-validation showed an average accuracy of 95.5%.
After the implementation, our third study is to evaluate the usability.
We gave a verbal explanation of the commands and gestures to VI participants, and then
they were required to finish eight daily tasks using ring and phone in turn.
After that, they completed questionnaires, comparing these two interactions.
According to the results, users could learn the gestures and commands quickly, and the
accuracy is around 95%.
As for the efficiency and task-based efficiency comparison, our ring can significantly reduce
the amount of registration time.
Also, this table shows the subjective score for the experience of 3 surfaces.
As you can see, the ring interaction outperforms the phone interaction in the dimensions of
"fun-to-use", "private", and "facilitates for one-handed interaction".
So basically, it's an input interaction for visually-impaired people, but it's also beneficial
to general people.
Sometimes we find it difficult to operate phones with a hand, or we want to discreetly
use a phone in socially sensitive situations.
Also, there are some limitations.
For example, the registration gesture still has unintentional touches, so we plan to add
a physical button.
We also want to add Bluetooth and battery modules, and remove physical wire connections
for production.
In conclusion, we present a ring-based input interaction with a single IMU enabling in-pocket
smartphone operation on any surface for visually-impaired people.
In a broader sense, it is a promising study that expands the HCI community's work of using
wearable devices to address the issues of visually-impaired people.