Hi, my name is Danyang Fan and I am excited to present on behalf of my
collaborators: Slide-tone and Tilt-tone, one degree of freedom haptic techniques
for conveying shape characteristics of graphs to blind users.
Data visualization has become increasingly important in our society,
augmenting the way we learn, work, track personal health and finances, and
interpret current events. Many of these applications need to be
versatile to the many data tasks that users may want to do, refreshable, and
widely available to be useful. Gorlewicz and all describes that graphical
access represents one of the biggest challenges to the independence and
productivity of individuals who are blind and visually impaired.
Unfortunately several reports during the recent pandemic show that data access
needs are not being met. At present, haptic methods are often
regarded as the most suitable for conveying spatial information because
they can support active and direct exploration of spatial relationships.
Tactile graphics, shown in the top left image, support a variety of tasks, but are
static and not refreshable. Tactile displays, shown in the two images
on the bottom left, are refreshable, but are currently not widely available due
to prohibitive costs. Audio-based methods such as alternative
text, read aloud tables, and sonification which map data attributes to sound
characteristics, can leverage audio capabilities that are
ubiquitous to common computing devices. However, they often do not support
direct and active exploration of spatial relationships.
In this work we ask how haptic and audio-based methods together can better
support data exploration in more versatile, widely available, and
refreshable ways. We look towards augmenting sonification
and speech interactions with simple haptic cues that are natural to object
shape perception. The image on the right shows a child
exploring wooden polyhedrons through touch.
Work in haptic perception has identified two important cues humans use
to understand global shape, which include the position of the finger on the object,
and the contact angle and pressure between the finger and the object.
In a data visualization context, we can imagine someone using a finger to
explore a physical object that has an edge in the shape of a trendline.
One way users can explore that trend- line, as shown in this animation, is by
running their finger along that edge. In the first cue, the position of the
user's finger changes corresponding to the zeroth order height of the graph.
This cue is called the zeroth order position cue.
Shown is an animation that labels how the height of the finger representing
zeroth order cues changes as a finger moves along the graph edge.
In the second cue, the inclination of contact between the finger and the edge
of the object changes corresponding to the tangent of the graph.
This cue is called the first order inclination cue.
Shown is an animation that labels how the contact angle between the graph edge
and the finger, representing first order cues, changes as it moves along the
graph edge. In this work, we ask whether these cues
can help support shape understanding in data visualization contexts, and whether
there are important interactions to consider when using these cues for data
exploration. To answer these questions, we first ran a
series of design workshops with blind participants to inform the development
of two multimodal systems for exploring line graphs. The contents of the workshop
are described in our paper. We then ran a controlled experiment to
evaluate the two systems, which we call Slide-tone and Tilt-tone.
In Slide-tone, as shown in the following clip, a user slides a platform
horizontally along the rail that corresponds to the x-axis.
Meanwhile, a motorized fader moves the user's finger to provide zeroth order
position cues, effectively causing the user to trace out the shape of the graph.
Sonification of the graph is simultaneously played.
[Sonification of graph]
In Tilt-tone, the navigation strategy and use of sonification is the same as in
Slide-tone. However, unlike Slide-tone, a servo motor tilts a platform under the user's
finger to provide first order inclination cues corresponding to the
trend's local tangent, as shown in this following clip.
[Sonification of graph]
Based on workshop recommendations, in both systems, users can retrieve a spoken
gist, which includes the title axis, range, and corresponding minimum and maximum
pitches. Two buttons also allow participants to
retrieve specific x and corresponding y values based on the location of the
platform along the x-axis. To evaluate these systems, we ran a
controlled experiment with eight blind users, comparing Slide-tone and Tilt-tone with
sonification without the haptic cues, as well as with an audio supported tactile
graphic, which we call the interactive tactile graphics condition.
To keep interactions consistent across the conditions, in the interactive
tactile graphic condition, users could also retrieve x and y values by pressing
two retrieval buttons while an exploration finger is on the trendline,
as shown in the following clip.
[Text-to-speech system]: x equals 229 [Text-to-speech system]: y equals 389
We counter-balanced the four modal conditions with four relatively complex
real-world datasets. Shown are tactile graphics showing the datasets used for
the study, each of which contain multiple inflection points, global patterns, and
local oscillations. We measured both correctness and speed of
participants completing four data tasks, which are: maximum identification, value
comparison, value retrieval, and mean estimation.
We found that the number of participants who answered each task correctly was
similar across all conditions, suggesting that active sonification, which forms the
basis of Slide-tone and Tilt-tone, performs comparably well for the explored tasks.
However, significantly more time was spent completing tasks in the
interactive tactile graphics condition compared to the Tilt-tone condition.
We believe this is because unfamiliar participants needed a functional
understanding of the spatial layout of tactile graphics to effectively navigate
and retrieve information for that condition, while the other conditions
constrain users exploration only to places on the trendline.
Through qualitative observations and participant comments, we identified
additional differences of using the different modal cues to explore data.
Participants described how different modal cues seemed better for identifying
different shape features. For example, several participants felt that haptic cues
communicated curvature and changes in slope better than sonification.
Participants also used different strategies in the different conditions.
For example, a gif in the middle of the slide shows how one participant used
additional fingers to mark heights on the motorized fader to help with making
comparisons. Finally we observed cue specific
challenges. In Tilt-tone, several participants rested their finger on the
edge of the tilt pad which rose and fell while the tilt pad conveyed the angle of
the graph. This rising and falling motion produced a false zeroth order cue that
confused several participants. Six of eight participants supported the
use of haptic cues together with sonification.
Five participants felt the haptic cues reinforce their understanding of the
sonified trend, and one felt that receiving the haptic cues in conjunction
with sonification helped them understand how sonification works altogether.
Five participants also felt that complementary information provided by
the haptic cues and sonification improved versatility. One participant
likened the cues to a quiver of tools that they could use to support different
tasks. However, conflicting cues in multimodal
contexts may lead to confusion, such as when the haptic cue provided in Tilt-
tone drastically changed when sonification did not.
Overall our results suggest that providing simple haptic feedback, when
not conflicting, can improve people's experiences interacting with
ecologically valid data. Improving our understanding for using
different modal cues for a broad variety of benchmark and open-ended tasks,
evaluating these cues in ecologically valid settings, and exploring ways to
affordably support these cues in common devices can make data more accessible
and compelling for people who are blind and visually impaired.
Thank you for your time, and we look forward to continuing the data
accessibility conversation.