In this talk, I will introduce our research on comparing the effectiveness of
visual and audio wayfinding guidance on smartglasses for people with low vision.
Low vision is a visual impairment that falls short of blindness but cannot
be corrected by eye glasses or contact lenses. There are many different low
vision conditions, for example, people can have blurry vision, central
vision loss, having blind spots in their vision, peripheral vision loss, and so on. And
one critical but challenging daily task for people with low vision is wayfinding,
which means a person needs to plan and follow a route to navigate from one
place to another. Because of their vision loss, low vision people had difficulties
in reading signs, orienting themselves in the environment,
recognizing landmarks, so that it is very difficult for them to complete a
wayfinding task independently, especially in an unfamiliar environment.
To address the wayfinding problems, many navigation systems have been designed to support
wayfinding for people with visual impairments in general. For example, some
systems can provide turn-by-turn audio instructions, and also some systems for
example the Microsoft Soundscape generates spatial audio beacon to guide
the user to the specific destination. And all the prior research mainly focused on
generating audio feedback for people with little to no vision. However,
different from people who are completely blind, low vision people still have
residual vision and they still actively use their vision during the navigation
task. This means, besides the traditional audio feedback, low vision people can
also potentially benefit from visual feedback if the interface is accessibly designed.
And the smartglasses platform has shown its great potential
in addressing the wayfinding difficulties to people with low vision.
With multiple embedded sensors, such as the camera and GPS, to locate the users'
current position, smartglasses can immersively
generate corresponding audio and visual feedback as wayfinding guidance to
enable a lower user to get to her destination independently. We thus want
to seize this unique opportunity and design wayfinding technology on smartglasses
for people with low vision. But how can we design such technology?
What feedback should we provide?
Since low vision people's visual ability is very complex,
and it ranges between blindness and full sight. So it is even not clear
whether audio or visual feedback is more effective for this population in a
wayfinding task.
So, to inform the design of wayfinidng technology, our
research aims to address the fundamental question, which is, what feedback modality
on smartglasses can best support low vision people in a wayfinding task.
So, to answer our research question, we sought to conduct a study to compare the
effectiveness of video and audio wayfinding guidance on smartglasses for
people with low vision. And then what exact visual and audio wayfinding guidance should
we compared? And the good news is that there has been already the de facto
standard wayfinding approaches for both blind and sighted community. We thus
started our exploration with these de facto standard approaches.
So specifically, our audio guidance followed the de facto standard turn-by-turn audio
instructions for blind people. And the design is based on prior navigation
system for blind users, called NavCog. And inspired by NavCog, we provided audio
feedback for three types of wayfinding information, including first the
distance to the next turn every 10 feet, and then the action at the turn the
user should take, for example turn left or turn right, and also the destination
information, such as the arrival at the destination and also the exact position
of the destination.
And moreover, our audio guidance is generated via
spatial audio from the position of the next turn so that the user can
better sense the direction that they should follow.
For the visual guidance, we
adapted the standard interfaces of the iOS Google Maps to AR glasses.
So similar with the highlighted route in the Google Maps, we generated a virtual
Path along the specific route, so that the user can simply follow this virtual path
to her destination. And we also designed a yellow star to indicate the exact
position of the destination, and the star is large enough so that a low vision person can
notice the existence of the star even at a far distance.
And following the turn-by-turn instructions,
we also visualized the distance to the next
turn and also the actions at the turn on the smartglasses.
Since low vision people
have very different visual abilities, we also provided different design
alternatives to support their different needs.
One design is called Floating Window.
It is a direct mapping from Google Maps' direction instruction on the
top bar. And we attached the action icon, such as the turning arrow, and the
distance to the turn, to the front of the user's eyes at about
two meters ahead, like a floating window.
And the distance updated each feet.
We anticipated that this design would be very helpful for people with limited
field of view because they could see this head-attached information all the
time without scanning around.
But for users who have a relatively wide field of view
and do not want the virtual information to block their central vision,
we designed Anchored Signs, where we anchored the action icon to the specific
turn and also attached the distance information to the side of the path, so
it's more like a street sign. And we put the sign every 10 feet along the path to
help the user better understand their current position.
And the low vision user
can select one of these two design options to customize their visual
experience during our study.
Okay. So now we have designed
both the visual and the audio guidance to present the wayfinding instructions.
But how would these feedback modalities affect a a low vision user's
experience in the real world wayfinding task?
To answer this question,
we conducted a user study to compare the effect of the visual and audio guidance
on the low vision people's wayfinding experiences.
Then what should we compare?
What metics should we measure?
So first, of course we want to compare the most basic
wayfinding performance, which is the wayfinding time, to measure how fast a
user can arrive the destination using different modalities.
And also, we want to
compare how many errors people make during this navigation process since we
want to enable the user to always follow the right route when using the specific
wayfinding guidance.
But beyond these basic performance, there are some
other factors that are also very important for a real-world wayfinding
task. For example, a wayfinding task in the real life is not simply walking from
one place to another.
There are usually some distractions, for
example, people may need to look for landmarks, pay attention to the obstacles,
and also talk to others.
So we want to make sure that the user still has extra
cognitive capacity to deal with these distractions during the navigation.
We thus want to explore which feedback modality requires less
cognitive load, thus the user can pay more attention to their surrounding and
other tasks.
And moreover, in many situations people may want to memorize
the environment, especially the places that they would visit frequently so that
they could navigate the same area next time more easily and independently. So that
we also compare the users mental map of a route after one navigation to see which
feedback modality can enable the user to better understand and memorize the
specific environment.
While our study measured all of these metrics, for the
sake of time, I'll only focus on the wayfinding time and also cognitive load in
this talk.
So with these questions and metrics in mind, we conducted a user
study with 16 low vision participants.
We also introduced a secondary task as a
distraction to measure the cognitive load of the different feedback modalities.
And we used an N-back task, which is a
measure for people's working memory.
In an N-back task, participants need to
monitor a series of stimuli, and respond when the current stimulus matches the one
that was presented n steps before in the series.
And this type of task is often used
in prior HCI research as a secondary task to induce cognitive load.
Specifically in our study, we designed a tactile version of the N-back task on a
smartwatch. And the reason we used the tactile stimuli is because we want to
avoid conflicts with the audio and visual wayfinding guidance.
We designed three
vibration patterns as the stimuli in the N-back task. And in each trial, we generated
a series of vibration stimuli and each stimulus was randomly selected from the
three vibration patterns. And participants were asked to tap on the
smartwatch screen whenever the current vibration pattern matches the previous one,
which is a 1-back task.
We asked each participant to conduct wayfinding tasks
in four conditions, including video guidance without and with the secondary
task, and the audio guidance with and without the secondary task.
And participants conducted 2 trials for each condition, and for each trial we asked
the participants to follow the specific wayfinding guidance to get to their
destination as quickly as possible with their typical walking speed. And for the
conditions with the secondary task, if the participants could not complete the two tasks
at the same time, we asked them to prioritize the wayfinding task, which
is the primary task.
So based on our study, we actually found several very
interesting results to inspire the future navigation system design.
So first about the wayfinding time.
So in our study, while we did not find any
significant difference in the wayfinding time
between the two feedback modalities cross all participants, but we found that
for participants with full field of view in at least one of their eyes, they
actually completed the wayfinding tasks significantly faster when they used the
visual feedback as opposed to using the audio feedback.
This is because
as opposed to the audio guidance that provided instructions for just the
current step, the visual guidance provided an overview of future
directions, which helped the participants better plan their steps in
the navigation. But for participants who have limited field of view, they actually
could not see all the virtual feedback altogether, for example this virtual path,
so that they are not able to benefit from this overview of the whole route.
And in terms of the cognitive load, we found that participants had a significant
better performance on the secondary task when using the visual guidance as
opposed to the audio guidance. And this bar chart shows the participants' mean
hit accuracy for the 1-back task during the navigation. You can see that
participants actually had much higher accuracy for the second task when they used
visual feedback as opposed to they used the audio feedback, which means they had
more ability to complete the secondary task with the visual feedback, so that
visual feedback required less cognitive load as opposed to the audio guidance.
And moreover, at the end of our study, we also asked the participants whether and how they
wanted to combine the visual and audio feedback to optimize their visual, to
optimize their wayfinding experience. And this bar chart shows the
distribution of participants' preferred modality for different wayfinding
information, including the overview of the route, the specific action at each
turn, the distance to the next turn, and also the destination information.
And the blue represents the number of participants who selected the visual
feedback to present the specific information, and the yellow represents
the number of participants who selected the audio feedback. And we found that, for
most of wayfinding information, participants actually wanted to combine
both of the modalities to optimize their experience.
So, these are the key findings
from our study. And to answer our research question, we compared visual and
audio wayfinding guidance on the smartglasses for people with low vision in
different perspectives, such as the wayfinding time, the errors they made, and
also the cognitive load. While the visual feedback seems to be the winner in all
of these perspectives, but most participants still want to combine visual
and audio feedback since they are more familiar with the audio feedback. So it
is very important to combine these feedback and provide multimodal feedback
to facilitate wayfinding tasks for people with low visoin in the future navigation systems.
So this is the major takeaway from our study.
Thank you so much.