Hi, I am Florian and on behalf of my co-author Tonja Machulla, I would like to introduce
our work “Pressing a Button you cannot see: evaluating Visual Designs to Assist Persons
with Low Vision Through Augmented Reality.” Visual impairments can manifest in a large
variety of symptoms. From fully blind or only recognizing bright or dark over clouded spots,
which lower the visual acuity over the whole visual field, to central vision loss, where
a person has a scotoma in their central field of view.
There are hundreds of millions of people with visual impairment worldwide, some walk around
using a cane; others just need to go a bit closer if they want to read text. Many of
them have remaining vision, which they are happy to use whenever possible.
One of the tasks that can be challenging for users with visual impairment is interacting
with a device. Especially touchscreens without any haptic feedback and possibly a changing
interface layout pose a considerable challenge. Here, we see a ticket machine of a German
railway company. Unfortunately, buttons have little contrast to the background and are
positioned closely together. As a result, finding the right button and hitting it poses
a challenge to users with visual impairment. Some interfaces, here a different ticket machine,
are more accessible due to their large contrast-rich buttons.
However, many interfaces are not and companies often show little motivation to make them
accessible. This problem extends to many public devices and even private ones without the
possibility to customize the interface. To assist people with visual impairments during
an interaction with touchscreen devices, we now want to leverage the users remaining vision.
Augmented Reality glasses have proven helpful as assistive technology. For example, they
can support navigating stairs or finding the correct item when shopping.
They provide many different sensors, including a microphone for speech input and cameras
to map their surroundings and provide contextual information.
We envision technology to improve in the coming years and become more widespread, comfortable,
cheaper, and more computationally powerful. There is work by Guo et al. that leverages
crowdsourcing and computer vision to identify interface elements. Therefore, a device that
recognizes user intention and interface elements and then can provide assistance is possible.
In our work, we focus on designing and evaluating augmentations that such a device can show
to the user. We addressed three research questions regarding
these augmentations. First, Do augmentations help, so can users complete the interaction
faster or more accurately when an augmentation is present?
Second, are there specific augmentations with different advantages or disadvantages?
And third, does providing such an augmentation result in a lower perceived workload of the
user? Since perception differs and imagination and
simulation can never give an actual image of how something is perceived, we integrated
persons with visual impairment into our design process starting early. We created augmentations
based on related work. Then, we asked three participants for a preliminary
evaluation and conducted a pilot study with seven participants.
The user study for the final evaluation was done with 16 participants.
We always collected qualitative feedback and suggestions for our designs.
We designed our augmentations to be visually simple, bright, and contrast-rich. They are
placed above the target touch screen element. First, we designed screen spanning augmentations
that should guide the user towards their center, either using a simple cross or more complex
forms which provide directional information. Then, we added motion to some of these augmentations
to provide an additional cue towards the center. Finally, we create small augmentations, which
only locally highlight the target. During our first meetings and the pilot study,
we learned some valuable lessons. Mainly, that little is sometimes more, as augmentations
that covered too much of the visual field or were too bright blocked the real world
and were disliked. Therefore, we reduced the brightness, used
thinner lines and added an aperture to the augmentations to not occlude the target itself.
Further, users did not report much difference regarding the color of the augmentation, even
if they had a color deficiency, as long as there was sufficient contrast to the background.
Our final user study had two different scenarios. First, to test our designs with familiar layouts,
we rebuild the keyboard of the previously shown ticket machine.
The user then had to enter a word. In the second scenario, we randomized the
letters on the buttons and the user had to find all occurrences of one specific letter.
In each scenario, we tested one Baseline condition and five different augmented conditions. Sixteen
participants took part in our study. We collected three measures—first, the task
completion time for one button press. Second, the error rate or how many button presses
missed the target, and third, we used the NASA-TLX questionnaire to measure perceived
workload. Here, we see the users view through the AR
glasses. We blurred the video to simulate a visual impairment for the viewer. Even so
the letters cannot be read, the augmentation provides sufficient information to locate
the target interface element. Out of 16 participants, seven could not use
the interface at all without any support. If we augmented the touchscreen, four more
participants could interact with the device. Our data supports that “Yes, augmenting
the interface helps.” In the three plots, we compare the TCT, ER, and NASA-TLX scores
for the Baseline in teal against the average of all augmented conditions in red. Each plot
shows the data for familiar and unfamiliar interfaces.
We see the task completion time in the first plot. In the case of unfamiliar interfaces,
it decreased significantly. For the second graph, we can see a significant
reduction in the error rate for familiar Layouts. And, the last plot shows that the perceived
workload is lower if an augmentation is present. Again, for unfamiliar layouts, this decrease
is significant. There was no significant difference between
the augmentations, presumably due to the inter-individual difference between participants and personal
perception. Perception differs. Each user has their own
visual perception, personal preferences, and learned strategies to access visual information.
Some participants could not even perceive the interface itself, as P11 only recognized
the grey area of the keyboard after asking her directly.
However, they could interact with the augmented device.
Others could see the keyboard and single buttons but not see or read the letters, leading to
an improvement in task completion time when the interface was augmented.
Furthermore, some could even read the letters as they were used to using a keyboard and
were slightly hindered if an augmentation was present.
In general, personal perception and preferences diverged greatly. For example, the circle
was the best augmentation for some and one of the worst for others. This might have masked
some differences between augmentations. There are some limitations to our augmentations.
On the technical side, the narrow augmentable field of view was smaller than the central
vision loss of one participant. Therefore, the augmented reality glasses could not be
used comfortably. Another participant could only perceive a
small area at once and not recognize the large augmentations as a whole.
Further, one participant reported avoiding tasks requiring high visual acuity and rather
asking for help. In conclusion, assistive devices are personal
and need customization. Nevertheless, augmenting a touchscreen in the presented form can support
users with visual impairment to interact with the device faster, easier, and with more accuracy,
or even interact at all.
I am happy to answer your questions.