hello
I am Syed Billah. I'm an assistant
professor at the Pennsylvania State
University
I'm presenting our work Tilt-Explore:
Making Tilt Gestures Usable for Low
Vision Smartphone Users.
This work is done with Farhani Momotaz
who is a graduate student at Penn state
A quick overview of our work:
Low-vision is broadly defined as a
vision impairment that cannot be fully
corrected even with glasses, medication,
or surgery.
People with low vision interact with
smartphones via assistive technologies like
screen magnifiers.
Screen magnifiers provide built-in
multi-finger multi-touch gesture to pan
and zoom on screen content.
Between panning and zooming operations,
panning is done more frequently than
zooming. In this work, we exclusively
focus on panning gestures.
To use built-in panning gestures, low
vision users need to engage their both
hands -- they need to hold the phone with
one hand and drag the magnified
content with two or three fingers using
the other hand, as shown in these
pictures.
This interaction is cumbersome,
tiring, and error prone.
Our paper addresses these issues with
built-in panning gestures. Our
contribution is Tilt-Explore, a
tilt-based, usable, single-handed, panning
technique for low-vision smartphone
users.
The design of tilt-explore was informed
by a study with 12 low vision
participant, and the final prototype was
evaluated by a second study with 16 low
vision users.
Tilt-explore adopts low vision users
interaction behavior. Compared to
built-in panning gestures, it is
significantly less error prone, and more
efficient at lower magnification scale.
Now I will demonstrate how tilt-explore
works.
This is an unmagnified screen.
After magnification, a low vision user
activates tilt-explore by resting one
finger on the screen for some time,
like 800 milliseconds, as shown in this
video.
content gliding under their fingertip
automatically gets selected.
The system provides various audio-haptic
feedback.
To deactivate tilt-explore, the user
simply lifts their finger off the
screen.
Here is a longer video.
Now that we know how Tilt-Explore works,
let's talk about its design process.
Drawing on the large body of tilt-based
research, we first identified key factors
influencing the performance of tilt
input. These factors are as follows:
we consider the rotation along x and
y-axis,
30 degree max.
As for tilt gesture delimiter, to
indicate an intended tilt motion from an
unintended ones, we considered four
different delimiters.
DoubleFlip, where
the user rotates the phone along its
long side
so that the phone screen is away and
then put it back quickly, as shown in
here.
Throw gesture, where the user abruptly
moves the phone back and forth by
a small margin while facing the screen,
as shown here.
Press and hold the clutch button, where
the user pressed and hold the clutch
button for the duration of tilt.
As shown in bottom-right, this clutch
could be an assistive-touch-like button
on iPhones
or a floating button.
With a toggle clutch button, the user
presses the clutch to turn the tilt
gesture on and presses it again to turn
it off.
For panning control, we chose
velocity-based control, where the
movement is proportionate to the amount
of tilt per second.
Finally, for the transfer function, we used
Hinckley’s transfer function, where
panning is restricted to either x and y
at a time.
Based on these initial parameters, we
developed a prototype of tilt-to-pan on
android, and conducted a formative study
with 12 low vision participants. As for
the task,
we asked them to execute tilt
delimiters, five times each,
find a target in two-dimensional grid
and read a row in one-dimensional
layouts.
We found that DoubleFlip and throw
delimiters were the least favorite
because the visual flow is interrupted
and the users lost their context.
Toggle clutch button was not favored
either because the user forgot to press
the clutch again to stop tilt motion.
Press and hold the clutch button was ok
but the floating clutch button was hard
to see or locate.
So, the participant recommended touch and
hold anywhere on the screen
as the tilt motion delimiter.
We also learned that a single-handed
panning gesture was not a priority for
some participant, but an alternative,
usable planning gesture was. As shown in
this figure, some participants brought
the phone very close to their eyes
leaving little space between the screen
and their eyes to issue panning gesture.
So, tilt-based panning gesture was
considered as an addition not a
substitution.
We also found that when making panning
gestures, not all three or two fingers
touch the screen simultaneously, like
here.
the finger that touches the screen right
before others registers an unintended
one-finger tap event, causing accidental
context switches.
So, our tilt-based panning should
prevent such context switches.
We observed that when using tilt-to-pan, some
participants tilted the phone along an
axis quickly to create a mental scope
then gradually
tilted the phone backward, causing the
panning to stop momentarily before
picking up speed in the opposite
direction;
as if they were buying some time to
process visual content. This process
looked like a wave of tilt motions.
So, we need to capture this phenomenon in
our transfer function.
Finally, most participants used a screen
reader to supplement their screen
magnifier, especially when reading texts.
So,
integrating screen reading features into
a screen magnifier is recommended.
From this study, we can conclude that
prior work on tilt-based interaction are
mostly unusable for low-vision users.
Based on the recommendation and our
observations, we revised our prototype as
follows: first, we borrowed touch
exploration feature from screen readers
that makes touching the screen risk-free
and enables limited screen reading
features into screen magnifiers.
So, our final tilt gesture diameter is
touching and holding anywhere on the
screen with one finger for some time,
like 800 millisecond, to turn on tilt-to-pan
and lifting the finger off the screen to
turn it off. This process, as shown before,
is right here.
We also provided various audio-haptic
feedback to notify different events, such
as when hitting an edge or a boundary.
Finally, we revised our transfer function
to incorporate waves of tilt motion.
To that end, we introduced a notion of
dynamic reference that updates in every
lambda seconds to capture the
orientation of the device at that time.
Pictorially, this is how it goes. This is
our static reference,
the blue line.
At time t,
delta theta
s
is the tilt angle with respect to our
static reference, so this is the tilt
angle with respect to our static
reference.
Let's say,
time t is a multiple of lambda and the
dynamic references is also set at time t.
Now, this green line is our dynamic
reference.
At t plus one, if the user tilted the
phone backward a little,
then delta theta D and delta theta s
represent the tilt angle with respect to
dynamic and static references. so these
two.
Finally, our ultimate tilt angle is a
linear combination of these two,
where eta is 0.8 and lambda is
5 seconds. Our hypothesis was the
performance of Tilt-Explorer would be as
good as the built-in planning gestures,
if not better.
We conducted a summative study with 16
location participants: 10 males 6 females, who
were around 50 years old on average.
The study was a within-subject design,
with two conditions: baseline and tilt
explore;
and two tasks per conditions. Task 1 was
a visual task,
which was finding a book in a 4x4 grid.
Here, this is before magnification, and this is
a magnified view.
Task 2 was reading a paragraph.
Unmagnified view and the magnified view.
We found that participant used almost 80
percent less panning gestures with tilt
explore.
In addition, the number of accidental
taps or context switches was reduced by
95 percent with tilt-explore. These
reductions were statistically
significant.
Regarding task completion time,
the participants took slightly more time
to complete visual searches with tilt
explore. But this difference was not
statistically significant.
Interestingly, participants who needed
lower magnification, such as less than 4x,
took significantly less time with tilt
explore. In practice, low vision users are
likely to take less time with tilt
explore, if the recovery time from
accidental contact switches are
considered.
For reading speed, we conclude that
panning is not suitable for reading
tasks -- We recorded the highest reading
speed of 44 words per minute, which is
still not comparable to that of people
without visual impairment, whose reading
speed is around 250 words per minutes. In
terms of subjective feedback,
participants who did not have age
related conditions, rated tilt-explore
highly favorably.
The analysis of NASA-TLX scores of both
conditions revealed that the difference
of overall scores
was not statistically significant, except
for those who were 45 or under.
With tilt-explore, the physical and
temporal workload were increased
significantly but the frustration was
reduced significantly.
In conclusion, tilt-explore is a usable,
single-handed, planning technique for
screen magnifiers, although most low
vision users are likely to use it bi-
manually and they are likely to consider
it as an addition to built-in panning
gestures.
Tilt-explore utilizes screen readers'
Touch-Exploration mode.
It reduces accidental context
switches during panning.
The risk of accidentally turning it on
is low.
It can work seamlessly with a screen
magnifier or a screen reader, although it
can be a standalone mode by its own
right.
The efficiency comes from the dexterity
of human wrist movement.
Unfortunately, this dependence is also a
limitation for tilt-explore because
low-vision users often develop difficulty
in tilting due to age-related conditions.
Finally, low-vision users who need lower
magnification,
such as less than 4x,
and who are 45 or under, are likely to
benefit from Tilt-explore the most.
Thank You.