Hi! I am Jaewook Lee, and today, I will be presenting our work on how different image
exploration modalities designed for blind or visually impaired individuals can encourage
skepticism towards imperfect AI-generated image captions.
This work was done in collaboration with Jaylin Herskovitz, Yi-Hao Peng, and Anhong Guo.
Traditionally, blind or visually impaired individuals rely on alternative text, or alt-text,
to understand the contents of an image.
However, alt-text is often missing, leaving many images inaccessible.
Auto-generating alt-text is a scalable alternative, but AI-generated captions can be incorrect
or incomplete.
Even if the auto-generated captions are inaccurate, without the means to verify correctness, blind
or visually impaired individuals place a high degree of trust in them.
Image exploration systems attempt to resolve this problem by providing additional information
about an image beyond a single caption. In this work, we aim to understand which image
exploration modalities could best support BVI people in identifying errors in auto-generated
image captions.
To achieve this, we compared three image exploration systems:
Facebook’s Automatic Alt Text Microsoft’s Seeing AI
And our ImageExplorer. In the next few slides, we showcase how each of the three systems
work in greater detail.
First is Facebook’s Automatic Alt Text feature, which provides details about an image as a
list of text that users can swipe through. Information is divided into several categories
including position information, size information, and elements by category.
'Position information, heading' 'On the left, one person. In the center, one
person.' 'Size information, heading'
'Minor elements, two people.'
Second is Microsoft’s Seeing AI, which provides a touch-based interface, enabling users to
explore by dragging their fingers across the screen.
'Two items detected. Move your finger over the screen to explore.'
Seeing AI will play a melody if their fingers do not intersect with any elements of interest.
When they intersect, Seeing AI will play a “ding” and vocalize a description of the
intersected element.
'Cat. Cat. Dining table. Cat. Dining table. Cat'
Third is ImageExplorer, our design probe that provides a multi-layered touch-based interface.
ImageExplorer begins by vocalizing the auto-generated caption and the number of elements in the
image.
'Please view in portrait. A black box on a table. There are four main objects available.
Touch to explore.'
Users can touch to explore. When users’ fingers intersect with an element, ImageExplorer
will first vocalize a description of the intersected element, similar to Seeing AI. If the element
contains additional information in the next layer, ImageExplorer will say “double tap
to explore”.
'Dining table, double tap to explore. Book, double tap to explore.'
When this user double taps on “book,” a first layer element, ImageExplorer reveals
its second layer elements such as text and drawings.
'Viewing book 12 sub-objects available. Touch to explore. Double tap anywhere to exit'
'Milk and Honey Rupi Kaur New York Times Bestseller. The logo on the shirt. Milk. Rupi. Honey.
Honey. Milk. And. The shirt is white. Milk and Honey Rupi Kaur New York Times Bestseller.'
Once users are done exploring an element in more detail, they can double tap anywhere
on the screen to return to the first layer. ImageExplorer will vocalize the number of
elements that have yet to be explored.
'Going back to the whole image three objects remaining.'
To implement ImageExplorer, we used the Mask R-CNN model for the first layer because it
can generate polygonal boundaries. We also used Google Cloud Vision’s object, face,
and text detection and labeling models as well as the DenseCap model to generate a detail-rich
second layer.
Participants were asked to use the three systems to explore nine images of three different
caption qualities. Caption quality was decided based on detailedness and correctness of the
caption.
Caption quality A refers to captions that are mostly accurate,
B refers to captions that are partially inaccurate, and
C refers to captions that are inaccurate.
From our study, we observed that participants were initially unsuccessful in determining
which captions were accurate and which were not prior to exploring the corresponding images.
Overall, image exploration systems were successful in encouraging skepticism towards imperfect
captions. This is shown by how participants’ scores for images with lower quality captions
decreased while scores for images with higher quality captions did not significantly change.
For image quality A, although participants’ average rating decreased, this change was
not significant. Additionally, even with a slight decrease in average rating post-explorations,
it still remained in the slightly accurate range.
For captions of image qualities B and C, the average ratings decreased significantly, which
suggests that participants were able to correctly judge that those captions were inaccurate
post-explorations.
After exploring an image, participants relied heavily on information they gathered when
judging caption accuracy. For example, for image 3B, a cat sitting on a chair, some participants
were able to figure out that the cat is on top of a dining table and next to two chairs
when exploring using the two touch-based systems. They decreased their scores because touch
empowered them to understand the spatial relationship between the cat, the chairs, and the dining
table.
Although all image exploration modalities were successful in encouraging skepticism
towards imperfect captions, ImageExplorer elicited the most correct explanations across
all caption qualities due to greater amount of detail.
When comparing a text-based system to a touch-based system, there is a tension between mental
effort and image understanding.
Text-based systems are quicker and simpler to interact with, as they allow users to explore
the information in a linear manner using a simple swipe gesture.
However, touch-based systems are more helpful for image understanding because they are capable
of providing more information such as spatial relationships. Additionally, layers are helpful
because they not only present information in an organized, hierarchical manner, but
also give users the autonomy to choose whether to view the additional information.
This tension is reflected in the overall ranking, where approximately half of the participants
preferred Facebook, a text-based system, and others preferred ImageExplorer, a touch- and
layer-based system.
Based on these results, we believe that the next iteration of ImageExplorer should include
two parts: a Text Overview and Touch Details.
Such a system will enable users to quickly read through high level information, just
as they were able to when using Facebook’s automatic alt text feature.
Then, if they are interested in understanding an image further , they could choose to receive
more details about it through a touch interface and layers, as they did with ImageExplorer.
In conclusion, image explorations encourage skepticism towards imperfect AI-generated
captions. While a text-based system is easier and quicker to use, touch and layers increase
image understanding, and the additional information helps enable BVI users to assess errors in
captions. When choosing an image exploration modality, users should decide whether to sacrifice
mental effort or image understanding. Thank you for listening to this presentation.